Scraper Structure

Overview
Scrapers are best understood by the Dataset they are scraping. The Datasets database is normalized, so it's not easy to tell at a glance what properties a dataset has (unless you have memorized what agency_id = 73e93439e6bf4ffc8b3f931a86fa3ad0 or data_type = 4 means.
Field
Description
agency_id
The agencies.id column
agency_info
Information changed here will be updated in the datasets table the next time the ETL script is run.
agency_name
The agencies.name column.
agency_coords
The latitude and longitude coordinates from the main agency building.
city
Not applicable for county or state agencies.
state
Two-letter ISO code (e.g. "IN" or "CA")
county_fips
If agency_coords is populated, this will be populated automatically. You can also look it up manually from the counties table.
data
Most agencies have multiple Datasets.
dataset_id
If you're scraping an existing dataset, add the id here.
url
The datasets_url column.
full_data_location
The location of the scraper Extraction. This is typically the data directory at the top level of the Scraper.
source_type
The source_types.id column.
data_type
The data_types.id column.
update_freq
The update_frequency.id column.
last_modified
In most cases, leave this blank and it will be filled in automatically. The script will update either the schema.json file or the datasets db if the value in the other location is more recent.
mapping
If you're scraping tabular data with a data_type that exists in the data_intake database, use these fields to map columns in your Extraction to columns in data_intake.__uuid__ , __dataset_id__ , and __date__ will be automatically overwritten by the script. 

Volunteer Activities - Previous

Write Scrapers

FOIA Requests

Last modified 1mo ago

Copy link

Edit on GitHub