Data Overview¶
Background¶
The objective of the Traffic Monitor is to be used as a primary data collection tool to capture holistic roadway data. The architecture is privacy-focused, featuring onboard detection, processing, and local data storage for cloud-free capability. Data is processed, generated, and captured locally in real-time.
The system records granular, event-level records for all sensors including those for detected objects (e.g., bicycle, vehicle, pedestrian), radar readings (speed, magnitude, detected object velocity, etc.), environmental measurements (air quality, temp, pressure, humidity, particulate matter, etc.), deployment metadata, system metrics, and more including data from plug-ins and extensions.
Data privacy¶
We believe in the power of open data. Anonymized data are crucial for useful social research and a public good. Our system (the Traffic Monitor database) prioritizes privacy by capturing as little personally identifiable information (PII) data as possible. With the default settings, our dataset is fully de-identified so it is easier to share with mitigated risk to individuals and the public.
The Traffic Monitor may, via configuration file changes, capture PII such as license plates and images from persons, for example, via full roadway snapshots and clips from the camera.
Be aware that when enabled, additional applications and sensors may be captured both in the primary database, losing de-identified qualification, and elsewhere on the device storage; e.g. application-specific folders and databases that contain their own retention policies.
Warning
Applications, plugins, and sensors may also store data for their own operation data in their own structure and locations.
Storage formats¶
The Traffic Monitor database utilizes unencrypted, open formats by default to make them portable and easy to share.
- SQLite is the native database format and can be viewed with open source DB Browser for SQLite.
- JSON - The Database UI tab allows downloading data with specific filteres including by table, object(s), and/or time frames.
Database structure¶
The Traffic Monitor database contains all captured, combined, and processed data from sensors, including additional, optional applications.
For tables related to sensors that are not installed, they will simply contain no records.
| Table Name | Description | PII Included |
|---|---|---|
| deployment | History of location details for device and sensors: includes latitude, longitude, bearing. | No |
| events | Object Detection confirmed events. Generated by Frigate with required information attached. | Potential (if included in Frigate sub-label or attribute) |
| radar_dov | DetectedObjectVelocity (DOV) readings from Radar API, ON command | No |
| radar_timed_speed_counts | TimedSpeedCounts readings from Radar API @O command with `@ | No |
| radar_raw_speed_magnitude | Show speeds and magnitudes in descending order by magnitude. OS and OM commands with O3 | No |
| radar_raw_speed_magnitude_single | Same as above, but only the first speed and magnitude value, for easier analysis. OS and OM commands with O1 | No |
| radar_oc_payload | Object detection capability, including vehicle length. OC command, with OPS9243 firmware | No |
| comments | Operator-entered time stamped entries for Traffic Monitor status, deployment, location, conditions, construction, etc. Used for downstream analysis. | Potential (operator-entered free-text) |
| airquality | Environmental readings from the optional air quality sensor: gases, particulate matter, temperature, pressure, humidity, and light. | No |
| plate_recognizer | License plate and vehicle details from the optional Plate Recognizer application. Stays empty unless Plate Recognizer is enabled. | Yes (license plates) |
Schema versions¶
The table definitions live in the repo as numbered migrations in container/node-red-tm/schema/migrations/, and container/node-red-tm/schema/schema.sql shows the full current schema. Each time the Node-RED service starts, it first applies any migrations the database doesn't have yet and records the schema version in the database itself. You can read it with PRAGMA user_version; in DB Browser for SQLite.
Before applying new migrations, the device saves a copy of the database in {{ tmsetup_codedir }}/node-red-tm/db/backup/ and keeps only the most recent copy.
How to access data¶
- Traffic Monitor Database: On-device dashboard: Database UI tab
- Frigate data: Recording and Snapshots follow configuration policies at
{{ tmsetup_codedir }}/frigate/storage - Other apps, plugins, and sensors may also store data for their own operation