Search…
Process overview
We're constantly iterating on each of these components, and there's always work to do. If you have ideas, reach out in Discord!

Structure

These terms are defined here.
1
Agencies
2
Datasets
3
Scrapers
4
Extractions
Copied!
Agencies can contain many Datasets; each dataset has one Scraper; each time a scraper is run, it generates an Extraction.

Data lifecycle

There's a flowchart here which covers the end-to-end data submission process.

Scraping

The process for writing Data Scrapers is here. Writing a data scraper typically takes a handful of hours, but can be accelerated to just a few minutes by using common code to gather similar records.
Last modified 2d ago
Copy link
Edit on GitHub