Data Standardization

Purpose

Normalize raw data into a unified format. This is part of what makes data useful at a large scale.

Current state

We're currently translating data from our Scrapers to a unified format in Dolt. We're translating the data at the point of collection.

Do I need to know how to write an ETL to help?

No, but someone will need to add one for us to use your data. If you need help, reach out in the #scrapers slack channel.

What are we translating to?

A unified format for each data_type. Each has its own table in data-intake. For example, to get the format of Incident Reports, run this in Dolt:

DESCRIBE `incident_reports`

Future state

OCR, broader data type support. Pull columns together in a unified table.