Lecture 7 Data for ITS modeling and Design

1.

DIGITAL
TRANSPORTATION
TECHNOLOGIES AND
INTELLIGENT
TRANSPORTATION
SYSTEMS
Lecture 7.
Data for ITS modeling and Design
Prepared by Makhnach Ekaterina

2.

Intelligent Transportation System (ITS) – a system that collects, processes, and uses data to manage
traffic flows, improve safety, and increase comfort. Without data, ITS is just ordinary traffic lights and signs
with a fancy "smart" label.
Transport data is any information that describes the state of a transport network, the movement of
objects (people, vehicles, goods), and the performance of infrastructure in space and time.
Example:
The green light stays on for exactly 30 seconds, even though there are no cars on the right – but a huge
queue on the left. Because data on congestion wasn't collected or wasn't used. An ITS based on real data
would have adjusted the phases.

3.

2. Data Classification: Where to Get It and What to Do With It
By source:
Stationary (inductive loops, cameras, radars on poles, vehicle detectors)
• Mobile (GPS trackers, mobile operator data, Bluetooth/Wi-Fi sniffers)
• User-generated (crowdsourcing – navigation app traffic, fare payments, car-sharing telemetry)
By nature:
• Aggregated (average speed on a segment over 5 minutes)
• Detailed (each vehicle with a timestamp)
By update frequency:
• Real-time (less than 1 second – for traffic signal control)
• Historical (monthly archives for modeling)
Key insight: The same data can be used both for real-time control and long-term design. Example: GPS
traces from thousands of taxis in London – today they help build a congestion model, tomorrow they help
plan bus lanes.

4.

Fundamental Traffic Flow Parameters
Flow rate (q) – number of vehicles passing a point per hour. Unit: veh/hour.
Density (k) – number of vehicles per km of road. Unit: veh/km.
Speed (v) – average speed of the traffic stream.
The relationship is simple: q = k × v (like in fluid dynamics).
As density increases, speed decreases. At critical density, the flow "collapses" – a traffic jam appears. ITS
can predict this 15 minutes in advance using detector data.

5.

Where Traffic Jams Come From in Data
Inductive loop – a copper wire loop under the asphalt. A passing vehicle changes the inductance. The
loop reports: "a vehicle passed, here's its length, here's its speed." Downside: it sometimes misses
motorcycles or can't distinguish a car from a truck.
Cameras with license plate recognition (ANPR) – give exact routes and travel times between cameras.
But: they work only in daylight and with clean license plates.
Bluetooth/Wi-Fi scanners – capture unique MAC addresses from devices inside vehicles. Anonymous
and cheap. This lets you measure travel time between two points without cameras.
GPS from navigation apps (Yandex, Google Maps, Waze) – huge coverage, but the sample is biased:
drivers with navigators don't drive exactly like those without.
Example:
London used cameras at the Congestion Charge zone entrances and obtained data for a model that
reduced congestion by 30%. Those same cameras became a 10-year historical archive.

6.

The Data Quality Problem
Y our ITS model can be mathematically perfect, but if the input is wrong, the forecast will be harmful.
Typical defects:
• Missing data (a detector breaks for one hour)
• Outliers (GPS shows 300 km/h in the city center)
• Synchronization errors (data from different sensors has mismatched timestamps)
What engineers do:
• Kalman filters – for smoothing trajectories.
• Interpolation – filling missing values from neighbors.
• Heuristics – "If speed > 18 0 km/h, discard."
In the Beijing subway, they use weight sensors on escalators to estimate passenger flow in real time when
ticket gates are down.

7.

Data Wrangling
Before building a model, you must:
• Clean the data
• Convert to a uniform time format
• Snap to a digital road map (map-matching)
• Aggregate (e.g., into 5-minute intervals)
Example: A bus GPS track is just a list of points (lat, lon, time). Map-matching "snaps" these points to the
actual road. Without this, your model would show the bus driving through a residential building.
Key concept: The digital road network map is the skeleton of ITS. Data without a map is just numbers. A
map without data is a drawing.

8.

Introduction to Traffic Modeling
Modeling can be:
• Microscopic (each vehicle is an individual agent with acceleration, braking, lane changes)
• Mesoscopic (groups of vehicles with shared characteristics)
• Macroscopic (traffic as a fluid)
Each level needs different data:
• Micro: high-resolution GPS trajectories of every vehicle.
• Macro: aggregated flow counts from loop detectors.

9.

The Data Value Chain in ITS
Raw data (voltages from loops, GPS pings, camera images)
↓
Processed data (vehicle counts, speeds, travel times)
↓
Information (congestion level, incident location, predicted delay)
↓
Knowledge (pattern: every Friday at 5 PM, this intersection fails)
↓
Action (adjust signal timing, dispatch a tow truck, suggest an alternative route)
Y ou can stop at any level. Most bad ITS projects stop at "processed data" and never reach action. The
best ones close the loop.

10.

Shockwaves and Their Data Signature
A shockwave is a boundary between two traffic states (e.g., free flow and jam). It moves backward
through the traffic stream — even though cars move forward.
A crash happens at 3:00 PM. At 3:01, the cars just behind the crash stop. At 3:02, cars 500 meters back
stop. At 3:05, cars 2 km back stop. The jam front is moving backward at about 15–20 km/h, even though
each individual car is stationary or crawling.
How data detects shockwaves:
Place two detectors 500 meters apart. Detector A (upstream) shows high speed, low density. Detector B
(downstream) shows low speed, high density. The shockwave is between them. By cross-correlating the
time series, you can calculate the shockwave speed.
If you know a shockwave is approaching an upstream detector, you can warn drivers before they hit the
jam. Dynamic message signs that say "Congestion ahead, 2 km" are based on shockwave detection.

11.

Radar (Microwave) Detectors
Emit a continuous wave or pulsed microwave signal. The signal reflects off vehicles. The Doppler shift
gives speed. Multiple beams can detect presence and lane position.
Types:
• CW Doppler — speed only, not presence when stopped
• FMCW (Frequency Modulated Continuous Wave) — speed, range, and presence
• Radar with beam-forming — can track multiple vehicles in multiple lanes
Advantages:
• No road cutting — mounted on poles or gantries
• Works in all weather (rain, fog, snow)
• Measures speed directly, not derived
• Can see stopped vehicles (FMCW only)

12.

Radar (Microwave) Detectors
Disadvantages:
• Can be blocked by large vehicles (trucks shadowing smaller cars behind them)
• Less accurate at very close range (under 5 meters)
• Expensive compared to loops ($2,000–$ 5,000 per unit vs. $ 500 per loop)
The Dutch highway system uses radar detectors on gantries every 500 meters on major corridors. They
feed real-time speed data to dynamic speed limit signs. When average speed drops below 7 0 km/h, the
limit is reduced from 100 to 7 0 to smooth flow.

13.

Cameras and Video Detection
Cameras feed video to a processor that uses computer vision algorithms to detect vehicles in defined
zones. Modern systems use deep learning (Y OLO, MobileNet) for vehicle detection, classification, and
tracking.
What they can do:
• Count vehicles by type (car, bus, truck, motorcycle, bicycle)
• Measure speed (by tracking between zones)
• Detect incidents (stalled vehicle, wrong-way driver, debris)
• Read license plates (ANPR/LPR)
• Detect pedestrians and cyclists
• Recognize traffic light states (red/green)
Accuracy: Very high in good light (95%+ ). Drops significantly at night, in rain, snow, or shadow.

14.

Cameras and Video Detection
Five years ago, video detection was unreliable. Today, a $ 50 camera with a $ 100 processor can do realtime vehicle detection at 30 fps with 98% accuracy. Why? Because models like YOLOv8 are pre-trained on
millions of labeled images. You don't need to program rules — you just show the model examples.
ANPR systems record license plates, which can be linked to vehicle owners. Many cities now have
regulations: data must be anonymized after 24 hours, or stored only for specific purposes (e.g., tolling,
crime detection).
In Barcelona, cameras on bus lanes automatically detect unauthorized private vehicles. The system
sends a fine automatically. In the first month, violations dropped by 80%. The data also showed that most
violators were delivery drivers — so they created a permit system for them.

15.

Bluetooth and Wi-Fi MAC Scanners
Every Bluetooth and Wi-Fi device has a unique MAC address. When the device is in discoverable mode (or
even just scanning for networks), it periodically broadcasts this address. A roadside scanner listens for
these broadcasts and logs the MAC address with a timestamp.
What they measure:
• Travel time between two scanners (if the same MAC is seen at both)
• Average speed (travel time divided by distance)
• Origin-destination patterns
Advantages:
• Very cheap (scanners cost $200–$ 500 each)
• No calibration needed
• Works in any weather, day or night
• Anonymous by design (MAC addresses are not directly linked to identity, though they are unique)

16.

Bluetooth and Wi-Fi MAC Scanners
Disadvantages:
• Only a sample of vehicles (maybe 5–15% of passing vehicles have a discoverable device)
• A single person may have multiple devices (phone, tablet, laptop, smartwatch) — double-counting risk
• Devices can be turned off or have randomized MAC addresses (new iPhones and Android phones
randomize MAC for privacy)
Sample bias problem: Drivers with Bluetooth on in their car are not a random sample. They might be
younger, wealthier, more tech-savvy. Their travel behavior could differ from the general population.
Always validate Bluetooth data against loop detector counts.
Real-world application: The I-35 corridor in Austin, Texas has 50 Bluetooth scanners. The system
measures real-time travel time every 5 minutes and posts it on electronic signs. Drivers see "Airport to
Downtown: 18 minutes" and decide whether to take the highway or local streets.

17.

GPS and Mobile Phone Data
Smartphones (and fleet vehicles with GPS) send location data to apps. Navigation apps (Google Maps,
Waze, Apple Maps, Y andex) collect this data to provide traffic services. The data is anonymized and
aggregated.
What they provide:
• Speed and position of individual vehicles every 1–10 seconds
• Real-time congestion maps
• Historical speed profiles for every road segment, for every hour of every day
• Route choice data (which path drivers actually take)
Google Maps claims to have over 1 billion kilometers of driving data every day. That's more data than any
government agency could ever collect.
Not everyone uses navigation apps. The sample is biased toward drivers who are unfamiliar with the area
(they need directions) or who want traffic information. Local drivers who know shortcuts might not use
apps.

18.

Emerging Sensors
Radar-based occupancy sensors for parking: Small radar units above parking spaces detect if a space
is occupied. No camera, no privacy issues. Used in many smart parking systems.
LiDAR (Light Detection and Ranging): Uses laser pulses to create a 3D point cloud of vehicles. Can
classify vehicles with high accuracy (sedan vs. SUV vs. truck), detect pedestrians, and measure vehicle
dimensions. Expensive ($ 10,000+ per unit) but dropping in price.
Acoustic sensors (microphone arrays): Detect vehicles by sound. Can estimate speed from Doppler
shift of engine noise. Rarely used alone but can supplement cameras in tunnels where other sensors fail.
Road-embedded piezoelectric sensors: Generate a voltage when a vehicle drives over them. Very
accurate for weigh-in-motion (WIM) to catch overloaded trucks. Expensive and short lifespan (2–3 years
under heavy traffic).

19.

The Cost of Bad Data
Failure 1 — No data: A mid-sized city installed adaptive traffic signals for $ 2 million. They used default
settings from the manufacturer. After one year, travel times actually increased by 8 %. Why? The system
was reacting to phantom "noise" in the sensors because no one calibrated it with real local traffic
patterns.
Failure 2 — Wrong data: A European highway agency used only loop detector data to predict congestion.
They ignored mobile GPS data. The loops showed low flow — but that was because drivers had already
abandoned the highway for side roads after seeing Waze alerts. The model predicted "free flow" while the
side roads were gridlocked.
Failure 3 — Old data: A South American city designed a new BRT (bus rapid transit) corridor using traffic
counts from 2015. Construction finished in 2022. By then, the city had grown 30% and two shopping malls
had opened along the corridor. The BRT was overcrowded on day one and the adjacent streets collapsed.

20.

Types of Data Errors
Systematic errors — consistent bias. Example: A loop detector consistently counts 5% low because its
sensitivity is set too high (small vehicles not detected).
Random errors — unpredictable fluctuations. Example: A GPS point jumps 50 meters due to multipath
reflection off a building.
Gross errors (outliers) — completely wrong values. Example: A detector reports 10,000 vehicles in one
hour on a road that can only carry 2,000.

21.

Missing Data Mechanisms
Missing completely at random (MCAR): The missingness has no relationship to the data value. Example:
A detector's memory card fails randomly.
Missing at random (MAR): The missingness depends on observed data. Example: A camera fails more
often at night (time of day is observed).
Missing not at random (MNAR): The missingness depends on the unobserved value. Example: A loop
detector fails when traffic is heavy (the missing data itself is correlated with the cause of missingness).

22.

Practical Data Cleaning Rules
Rule 1 — Physical plausibility:
Speed must be between 0 and 200 km/h (or 250 for highways in Germany)
Flow cannot exceed 2,500 veh/h/lane (absolute physical maximum)
Occupancy cannot exceed 100% (obviously)
Density cannot exceed 250 veh/km (bumper-to-bumper)
Rule 2 — Temporal consistency:
Speed cannot change by more than 50 km/h from one 5-minute interval to the next (without a crash)
Flow cannot drop by 90% and then recover in 5 minutes (would require a magic portal)
Rule 3 — Spatial consistency:
Adjacent lanes on the same road should have similar speeds (unless there's a lane closure)
Detectors 1 km apart should not report radically different flows (conservation of vehicles)

23.

Data Aggregation and Temporal Scales
Raw detector data is often collected at very high frequency
• Loops: vehicle pulse every 0.1–0.5 seconds
• GPS: positions every 1 second
• Cameras: 30 frames per second
1-second: Only used for real-time adaptive signal control (e.g., SCATS system in Sydney). Each second,
the controller decides whether to extend the green.
5-minute: The industry standard for traffic operations. Short enough to capture congestion dynamics,
long enough to smooth out random fluctuations.
15-minute: Used for planning and performance reporting (e.g., FHWA's annual congestion reports).
1-hour: Used for long-term trend analysis and capacity planning.
https://www.youtube.com/watch? v= 9c7 M1pIxi-8
https://www.youtube.com/watch? v= NQ_ L5C1sQEA

24.

Data Storage and Management for ITS
Volume and Velocity
A typical mid-sized city (population 500,000) with 500 loop detectors, 200 cameras, and 50 Bluetooth
scanners generates:
Loops: 500 detectors × 1 record per 20 seconds = 2.16 million records/day
Cameras: 200 cameras × 1 image per second (but most images not stored) = 17 .28 million images/day
(impossible to store all) → only store metadata (counts, classifications)
Bluetooth: 50 scanners × 100 MACs per minute = 7 .2 million records/day
Total: ~10–30 million records per day. That's 3.6–11 billion records per year. This is big data territory.

25.

Data Storage and Management for ITS
Database Choices
Relational databases (PostgreSQL, MySQL): Good for structured data with complex queries. But slow for
very high write rates.
Time-series databases (InfluxDB, TimescaleDB): Optimized for timestamped data. Compression (10–20x
smaller than relational). Fast aggregations. Industry standard for ITS.
Key-value stores (Redis): In-memory, extremely fast. Used for real-time dashboards and caching.
Data warehouses (BigQuery, Snowflake): Cloud-based, good for historical analysis but not real-time.
The London Traffic Control Centre uses a hybrid: Redis for real-time (last 5 minutes), TimescaleDB for
operational (last 30 days), and BigQuery for historical archive (all data, 2003–present).

26.

Data Storage and Management for ITS
Data Retention Policies
Real-time control: Keep 1–5 minutes (enough for signal coordination)
Performance monitoring: Keep 1–3 years (trend detection)
Legal/liability: Keep 5–7 years (in case of lawsuits from crashes)
Research: Keep indefinitely (but anonymize after 5 years)
Storage is cheap, but not free. A city storing 20 million records/day for 10 years = 7 3 billion records. Even
at 50 bytes per record (compressed), that's 3.6 terabytes. Manageable. But if you store images or video,
it's petabytes. Most agencies delete video after 7–30 days.

27.

Data Storage and Management for ITS
Real-Time Data Streaming
For ITS, data is useful only if it arrives quickly. Latency is the delay between the event and when the system
knows about it.
Inductive loops: < 0.1 second (hardwired to controller)
Radar/Cameras: 0.5–2 seconds (processing time)
Bluetooth scanners: 5–15 seconds (device discovery + transmission)
GPS from fleet vehicles: 10–6 0 seconds (cellular upload delay)
Crowdsourced GPS (Waze): 1–5 minutes (app uploads in batches to save battery)
Streaming architectures:
MQTT (lightweight publish-subscribe) — used for sensor networks
Apache Kafka (high-throughput distributed streaming) — used by large traffic centers
WebSockets — used for real-time dashboards in browsers

28.

Thank you for the attention!