Data Sources database

Use the API to find, use, and manage Data Sources.

Base URL

https://data-sources.pdap.io

Search Tokens

Generate API token for front end search

GET [base-url]/search-tokens

The search tokens endpoint is located in resources/SearchTokens.py. The search tokens endpoint generates an API token valid for 5 minutes and forwards the search parameters to the Quick Search endpoint. This endpoint is meant for use by the front end only.

Query Parameters

Name	Type	Description
arg1*	String	The first argument that will be forwarded on to the appropriate endpoint. Currently either "search" for quick-search or "id" for data-sources
arg2*	String	The second argument that will be forwarded on to the appropriate endpoint. Currently just used for "location" for quick-search
endpoint	String	The endpoint that will be accessed after a search token is generated

200: OK Successful operation

Schema

count int
data array[object]
    agency_name string
    municipality string
    state_iso string
    data_source_name string
    description string
    record_type string
    source_url string
    record_format string
    coverage_start string
    coverage_end string
    agency_supplied boolean

Example

{
	"count": 1,
	"data": [
		{
			"agency_name": "Allegheny County Police Department - PA",
			"municipality": "Pittsburgh",
			"state_iso": "PA",
			"data_source_name": "Allegheny County Police Review Board Transcripts",
			"description": null,
			"record_type": "Policies & Contracts",
			"source_url": "https://www.alleghenycounty.us/county-council/police-review-board-meetings.aspx",
			"record_format": "[\"PDF: Machine Created\"]",
			"coverage_start": "2018-08-29",
			"coverage_end": "2018-09-26",
			"agency_supplied": true
		}
	]
}

500: Internal Server Error Error

{
    "message": error
}

Search

Quick Search Data Sources by search term and location

GET [base-url]/quick-search/{search}/{location}

The quick search endpoint is located in resources/QuickSearch.py. The quick search endpoint executes its search using the agency_source_link table in the Data Sources database, which links each data source in the data_sources table with its associated agency in the agencies table. This endpoint is meant for use by the search tokens endpoint only.

Path Parameters

Name	Type	Description
search*	String	Checks partial matches on any of the following properties on the data_source table: "name", "description", "record_type", and "tags". The search term will is case insensitive and will match singular and pluralized versions of the term.
location*	String	Checks partial matches on any of the following properties on the agencies table: "county_name", "state_iso", "municipality", "agency_type", "jurisdiction_type", "name"

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

200: OK Successful operation

Schema

count int
data array[object]
    agency_name string
    municipality string
    state_iso string
    data_source_name string
    description string
    record_type string
    source_url string
    record_format string
    coverage_start string
    coverage_end string
    agency_supplied boolean

Example

{
	"count": 1,
	"data": [
		{
			"agency_name": "Allegheny County Police Department - PA",
			"municipality": "Pittsburgh",
			"state_iso": "PA",
			"data_source_name": "Allegheny County Police Review Board Transcripts",
			"description": null,
			"record_type": "Policies & Contracts",
			"source_url": "https://www.alleghenycounty.us/county-council/police-review-board-meetings.aspx",
			"record_format": "[\"PDF: Machine Created\"]",
			"coverage_start": "2018-08-29",
			"coverage_end": "2018-09-26",
			"agency_supplied": true
		}
	]
}

403: Forbidden Invalid API key

500: Internal Server Error Something went wrong

{
	"count": 0,
	"message": error
}

400: Bad Request Missing or bad API key

{
    // Response
}

Data Sources

Get all Data Sources

GET [base-url]/data-sources

The data sources endpoint is located in resources/DataSources.py. The data sources endpoint returns all approved rows in the corresponding Data Sources database table by default. An optional JSON object can be passed to get data sources needing identification instead.

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

Request Body

Name	Type	Description
Data	JSON	In order to get data sources needing identification: {"approved": False}

200: OK Successful operation

Schema

count int
data array[object]
    name string
    submitted_name string
    description string
    record_type string
    source_url string
    agency_supplied boolean
    supplying_entity string
    agency_originated string
    originating_entity string
    agency_aggregation string
    coverage_start string
    coverage_end string
    source_last_updated string
    retention_schedule string
    detail_level string
    number_of_records_available string
    size string
    access_type array
    data_portal_type string
    record_format array
    update_frequency string
    update_method string
    tags string
    readme_url string
    scraper_url string
    data_source_created string
    airtable_source_last_modified string
    url_status string
    rejection_note string
    last_approval_editor object
    agency_described_submitted string
    agency_described_not_in_database string
    approval_status string
    record_type_other string
    data_portal_type_other string
    records_not_online string
    data_source_request string
    url_button string
    tags_other string
    access_notes string
    last_cached string

Example

count: 1
[{'name': 'Calls for Service for Chicago Police Department - IL', 'submitted_name': None, 'description': None, 'record_type': 'Calls for Service', 'source_url': 'https://informationportal.igchicago.org/911-calls-for-cpd-service/', 'agency_supplied': True, 'supplying_entity': None, 'agency_originated': None, 'originating_entity': None, 'agency_aggregation': None, 'coverage_start': '2019-01-01', 'coverage_end': None, 'source_last_updated': None, 'retention_schedule': None, 'detail_level': 'Summarized totals', 'number_of_records_available': None, 'size': None, 'access_type': ['Web page'], 'data_portal_type': None, 'record_format': None, 'update_frequency': 'Bi-weekly', 'update_method': None, 'tags': None, 'readme_url': None, 'scraper_url': None, 'data_source_created': '2022-09-28', 'airtable_source_last_modified': '2023-11-08', 'url_status': 'ok', 'rejection_note': None, 'last_approval_editor': '{"id": "usrtLIB4Vr3jTH8Ro", "email": "[e-mail]", "name": "Josh Chamberlain"}', 'agency_described_submitted': 'Chicago city, Illinois', 'agency_described_not_in_database': None, 'approval_status': 'approved', 'record_type_other': None, 'data_portal_type_other': None, 'records_not_online': None, 'data_source_request': None, 'url_button': None, 'tags_other': None, 'access_notes': None, 'last_cached': '2024-01-30', 'agency_name': 'Chicago Police Department - IL'}]

400: Bad Request Missing or bad API key

{
    // Response
}

403: Forbidden Invalid API key

500: Internal Server Error Error

{
	"count": 0,
	"message": "There has been an error pulling data!"
}

Data Sources for Map

GET [base-url]/data-sources-map

Headers

The data sources for map endpoint is located in resources/DataSourcesMap.py. The data sources endpoint returns all approved rows in the corresponding Data Sources database table by default with only the columns relevant to mapping.

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

200: OK Successful operation

Schema

count int
data array[object]
    data_source_id string
    name string
    agency_id string
    agency_name string
    state_iso string
    municipality string
    county_name array[string]
    record_type string
    lat float
    lng float

Example

count: 1
[{'data_source_id': 'recisSIaKGoWWbC8y', 'name': 'Allegheny County Jail policies', 'agency_id': 'recNjgPW5kD573LeK', 'agency_name': 'Allegheny County Jail', 'state_iso': 'PA', 'municipality': None, 'county_name': '["Allegheny"]', 'record_type': 'Policies & Contracts', 'lat': 40.4345561, 'lng': -79.9959509}]

400: Bad Request Missing or bad API key

{
    // Response
}

403: Forbidden Invalid API key

500: Internal Server Error Error

{
	"count": 0,
	"message": "There has been an error pulling data!"
}

Get Data Source by Id

GET [base-url]/data-sources-by-id/[id]

The data sources endpoint is located in resources/DataSources.py. The data source by id endpoint returns just the row for the data source that corresponds to the id passed.

Path Parameters

Name	Type	Description
id*	String	Data source id

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

200: OK Successful operation

Schema

count int
data array[object]
    name string
    submitted_name string
    description string
    record_type string
    source_url string
    agency_supplied boolean
    supplying_entity string
    agency_originated string
    originating_entity string
    agency_aggregation string
    coverage_start string
    coverage_end string
    source_last_updated string
    retention_schedule string
    detail_level string
    number_of_records_available string
    size string
    access_type array
    data_portal_type string
    record_format array
    update_frequency string
    update_method string
    tags string
    readme_url string
    scraper_url string
    data_source_created string
    airtable_source_last_modified string
    url_status string
    rejection_note string
    last_approval_editor object
    agency_described_submitted string
    agency_described_not_in_database string
    approval_status string
    record_type_other string
    data_portal_type_other string
    records_not_online string
    data_source_request string
    url_button string
    tags_other string
    access_notes string
    last_cached string
    homepage_url string
    count_data_sources string
    agency_type string
    multi_agency string
    submitted_name string
    jurisdiction_type string
    state_iso string
    municipality string
    zip_code string
    county_fips string
    county_name array
    lat float
    lng float
    data_sources array
    no_web_presence string
    airtable_agency_last_modified string
    data_sources_last_updated string
    approved boolean
    rejection_reason string
    last_approval_editor object
    agency_created string
    county_airtable_uid string
    defunct_year string
    data_source_id string
    agency_id string
    agency_name string

Example

{
    "data": {'name': 'Calls for Service for Asheville Police Department - NC', 'submitted_name': 'Asheville Police Department', 'description': None, 'record_type': 'Calls for Service', 'source_url': 'https://services.arcgis.com/aJ16ENn1AaqdFlqx/arcgis/rest/services/APD_CAD_911_Calls_2006/FeatureServer/0', 'agency_supplied': True, 'supplying_entity': None, 'agency_originated': None, 'originating_entity': None, 'agency_aggregation': None, 'coverage_start': '2006-01-01', 'coverage_end': '2006-12-31', 'source_last_updated': None, 'retention_schedule': None, 'detail_level': None, 'number_of_records_available': None, 'size': None, 'access_type': ['API', 'Download'], 'data_portal_type': 'ArcGIS', 'record_format': ['GIS / Shapefile'], 'update_frequency': None, 'update_method': None, 'tags': None, 'readme_url': 'https://docs.google.com/document/d/143a0LoGwNwmmHxJu1msxjOFAfAXPk7otQSWkrLtUDk0/edit?usp=sharing', 'scraper_url': 'https://pypi.org/project/openpolicedata/', 'data_source_created': '2023-03-02', 'airtable_source_last_modified': '2023-11-08', 'url_status': 'ok', 'rejection_note': None, 'last_approval_editor': '{"id": "usrtLIB4Vr3jTH8Ro", "email": "[e-mail]", "name": "Josh Chamberlain"}', 'agency_described_submitted': None, 'agency_described_not_in_database': None, 'approval_status': 'approved', 'record_type_other': None, 'data_portal_type_other': None, 'records_not_online': None, 'data_source_request': None, 'url_button': None, 'tags_other': None, 'access_notes': None, 'last_cached': '0001-01-01', 'homepage_url': 'https://www.ashevillenc.gov/department/police', 'count_data_sources': 18, 'agency_type': 'law enforcement/police', 'multi_agency': None, 'jurisdiction_type': 'local', 'state_iso': 'NC', 'municipality': 'Asheville', 'zip_code': '28801', 'county_fips': '37021', 'county_name': ['Buncombe'], 'lat': 35.594677, 'lng': -82.54986, 'data_sources': ['recpWxJ9JVa6BtLi5', 'reczwxaH31Wf9gRjS', 'recBd7NrWsvfTyDk0', 'recy24QB6I8FCVrQr', 'recW6aQGuNyzedIDl', 'recmrXPvQn9Gtfpba', 'recsojoTxJ3g08qKl', 'recTUW1QUZpsGVxoJ', 'reckqhpMEvDgiDGXF', 'recjnLeosesVTaW2r', 'reccrwbvTL6Ttd8XT', 'reckiy2nuY5iRptBm', 'recyJMD6eYF9Ln98B', 'recLdzmQMXC3XuPWU', 'recRNvjKBo5LkUBS9', 'recKFoiPMOmWFZvqM', 'recordd7DcM3raQF7', 'recV35fyFlof4pQXP'], 'no_web_presence': None, 'airtable_agency_last_modified': '2023-05-16', 'data_sources_last_updated': '2023-03-02', 'approved': True, 'rejection_reason': None, 'agency_created': '2022-08-18', 'county_airtable_uid': 'recrCy8hHYuxC8ZhU', 'defunct_year': None, 'data_source_id': 'reczwxaH31Wf9gRjS', 'agency_id': 'recJDGmbd7UMFcfa0', 'agency_name': 'Asheville Police Department - NC'}
}

400: Bad Request Missing or bad API key

Schema

count int
data array[object]

Example

{
	"count": 0,
	"data": []
}

403: Forbidden Invalid API key

500: Internal Server Error Error

{
	"message": "There has been an error pulling data!"
}

404: Not Found Id not found

{
    "message": "Data source not found."
}

Create Data Source

POST [base-url]/data-sources

The data sources endpoint is located in resources/DataSources.py. The create data source endpoint posts a new data source to the database and returns True if successful and False if not.

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

Request Body

Name	Type	Description
Data	JSON	A JSON object of the data source information. Refer to the Data Source dictionary for available fields: https://docs.pdap.io/activities/data-dictionaries/data-sources-data-dictionary. However, the following fields cannot be edited: rejection_note, data_source_request, approval_status, airtable_uid, airtable_source_last_modified Below is an example of an acceptable body: { "name": "Calls for Service for Chicago Police Department - IL", "record_type": "Calls for Service", "source_url": "https://informationportal.igchicago.org/911-calls-for-cpd-service", "coverage_start": "2019-01-01" }

200: OK Successful operation

Schema

message string

Example

{"message": "Data source added successfully."}

403: Forbidden Invalid API key

500: Internal Server Error Error

{
    "message": "There has been an error adding the data source"
}

400: Bad Request Missing or bad API key

{
    // Response
}

Update Data Source by Id

PUT [base-url]/data-sources-by-id/[id]

The data sources endpoint is located in resources/DataSources.py. The update data source by id endpoint updates a data source and returns a status to confirm a successful update.

Path Parameters

Name	Type	Description
id*	String	Data source id

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

Request Body

Name	Type	Description
Data	JSON	A JSON object of the data to be updated. Refer to the Data Source dictionary for available fields: https://docs.pdap.io/activities/data-dictionaries/data-sources-data-dictionary. However, the following fields cannot be edited: data_source_request, airtable_uid, airtable_source_last_modified Below is an example of an acceptable body: { "name": "Calls for Service for Chicago Police Department - IL", "record_type": "Calls for Service", "source_url": "https://informationportal.igchicago.org/911-calls-for-cpd-service", "coverage_start": "2019-01-01" }

200: OK Successful operation

{"message": "Data source successfully updated."}

500: Internal Server Error Error

{
    "message": "There has been an error updating the data source"
}

403: Forbidden Invalid API key

{
    // Response
}

400: Bad Request Missing or bad API key

{
    // Response
}

Archives

Get all Archived urls

GET [base-url]/archives

The archives endpoint is located in resources/Archives.py. The get method on the archives endpoint returns all rows for urls that the automatic archives script has cached in the Internet Archive.

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

200: OK Successful operation

Schema

id string
source_url string
update_frequency string
last_cached string

Example

[{'id': 'rec0Fh8dMrlfxlpoH', 'source_url': 'https://www.portlandmaine.gov/999/Annual-Reports', 'update_frequency': 'Annually', 'last_cached': '2024-01-30'}]

400: Bad Request Missing or bad API key

Schema

string

Example

"There has been an error pulling data!"

403: Forbidden Invalid API key

Get all Archived urls

PUT [base-url]/archives

The archives endpoint is located in resources/Archives.py. The put method on the archives endpoint updates the data source matching the passed id, updating the last_cached date if it alone is passed, or it and the broken_source_url_as_of field and the url_status to 'broken'.

Headers

Name	Type	Description
Authorization*	String	Value formatted as "Bearer [access token/api key]”

Request Body

Name	Type	Description
id*	String	The airtable uid for the data source that was cached
broken_source_url_as_of*	Date	The current date if the url is no longer active, otherwise None
last_cached*	Date	The current date since the data source url was just cached in the Internet Archive

200: OK Successful operation

Schema

count int
data array[object]
    agency_name string
    municipality string
    state_iso string
    data_source_name string
    description string
    record_type string
    source_url string
    record_format string
    coverage_start string
    coverage_end string
    agency_supplied boolean

Example

{
	"count": 1,
	"data": [
		{
			"agency_name": "Allegheny County Police Department - PA",
			"municipality": "Pittsburgh",
			"state_iso": "PA",
			"data_source_name": "Allegheny County Police Review Board Transcripts",
			"description": null,
			"record_type": "Policies & Contracts",
			"source_url": "https://www.alleghenycounty.us/county-council/police-review-board-meetings.aspx",
			"record_format": "[\"PDF: Machine Created\"]",
			"coverage_start": "2018-08-29",
			"coverage_end": "2018-09-26",
			"agency_supplied": true
		}
	]
}

400: Bad Request Missing or bad API key

403: Forbidden Invalid API key

Agencies

Get all Agencies

GET [base-url]/agencies/{page}

The agencies endpoint is located in resources/Agencies.py. The agencies endpoint returns 1000 rows from the corresponding Data Sources database table offset by the page number passed.

Path Parameters

Name	Type	Description
page*	String	Passing 1 will return the first 1000 rows. Subsequent page number return subsequent results

Headers

Name	Type	Description
Authorization	String	Value formatted as "Bearer [access token/api key]”

200: OK Successful operation

Schema

count int
data array[object]
    agency_name string
    municipality string
    state_iso string
    data_source_name string
    description string
    record_type string
    source_url string
    record_format string
    coverage_start string
    coverage_end string
    agency_supplied boolean

Example

{
	"count": 1,
	"data": [
		{
			"agency_name": "Allegheny County Police Department - PA",
			"municipality": "Pittsburgh",
			"state_iso": "PA",
			"data_source_name": "Allegheny County Police Review Board Transcripts",
			"description": null,
			"record_type": "Policies & Contracts",
			"source_url": "https://www.alleghenycounty.us/county-council/police-review-board-meetings.aspx",
			"record_format": "[\"PDF: Machine Created\"]",
			"coverage_start": "2018-08-29",
			"coverage_end": "2018-09-26",
			"agency_supplied": true
		}
	]
}

400: Bad Request Missing or bad API key

Schema

count int
data array[object]

Example

{
	"count": 0,
	"data": []
}

403: Forbidden Invalid API key