Process overview

We're constantly iterating on each of these components, and there's always work to do. If you have ideas, reach out in Discord!
Structure
These terms are defined here.
1
Agencies
2
  Datasets
3
    Scrapers
4
      Extractions      
Copied!
Agencies can contain many Datasets; each dataset has one Scraper; each time a scraper is run, it generates an Extraction.
Data lifecycle
There's a flowchart here which covers the end-to-end data submission process.
Scraping
The process for writing Data Scrapers is here. Writing a data scraper typically takes a handful of hours, but can be accelerated to just a few minutes by using common code to gather similar records.

Welcome to PDAP!

Next - Volunteer Activities

Maintain Datasets

Last modified 5mo ago

Copy link

Edit on GitHub

Contents

Structure

Data lifecycle

Scraping