πŸ“–
Police Data Access Point Docs
pdap.ioGitHub
  • πŸ‘‹Welcome
  • ⚑Activities
    • Label new Data Sources
      • Labeling events
    • Volunteer for Data Requests
    • Search for Data Sources
    • Publish data
    • Web scraping
    • FOIA requests
    • Advocate for open data
  • πŸ”¬About
    • Search the PDAP database
    • Terms & definitions
      • What is a Data Source?
      • Terminology
    • Database details
      • Data Sources data dictionary
      • Agencies data dictionary
      • Requests data dictionary
      • Record Types taxonomy
      • Hidden properties
    • GitHub
    • Hugging Face
  • πŸ“‘API
    • Introduction/Getting Started
  • πŸ› οΈTools & Resources
    • Related projects
    • Resources for using data
    • Using LLMs like ChatGPT
  • πŸ”Meta
    • Internal Tools (Retool)
    • Internal dev resources
      • GitHub issue template
      • GitHub pull request template
      • Product changes checklist
      • β˜‘οΈProduction QA Checklist
      • Retool
    • Operations
      • Staff resources
        • Meeting Minutes
          • 2021-07-14
          • 2021-06-16
          • 2021-03-14
          • 2020 11-21 Tech Stack discussion
          • 2020-09-30 Leadership Cadence
          • 2020-10-14 Leadership Cadence
          • 2020-10-21 Leadership Cadence
          • 2020-10-28 Leadership Cadence
          • 2020-11-04 Leadership Cadence
          • 2020-11-12 Leadership Cadence
          • 2020-11-18 Leadership Cadence
          • 2020-11-25 Leadership Cadence Notes
          • 2020-12-02 Leadership Cadence Notes
          • 2020-12-09 Leadership Cadence Notes
          • 2020-12-12 Working Session
          • 2020-12-16 Leadership Cadence
          • 2020-12-30 Leadership Cadence Notes
          • 2021-01-06 Leadership Cadence Notes
          • 2021-01-13 Leadership Cadence Notes
          • 2021-01-20 Leadership Cadence Notes
          • 2021-01-27 Leadership Cadence Notes
          • 2021-02-03 Leadership Cadence Notes
          • 2021-02-10 Leadership Cadence Notes
          • 2021-02-17 Leadership Cadence Notes
          • 2021-02-24 Leadership Cadence Notes
          • 2021-03-03 Leadership Cadence Notes
          • 2021-03-10 Leadership Cadence Notes
          • 2021-03-16 Leadership Cadence Notes
          • 2021-03-27 database working session
          • 2021-03-31
          • 2020-12-1
          • 2021-01-23
          • 2021-04-10 Meeting notes
          • 2021-04-17 Meeting notes
          • 2021-04-21 Leadership Cadence
          • 2021-04-28 Leadership Cadence
          • 2021-05-05 Leadership Cadence
          • 2021-05-12 Leadership Cadence
          • 2021-05-19 Leadership Cadence
          • 2021-05-26 Leadership Cadence
          • 2021-06-02 Leadership Cadence
          • Decision log
        • Brand assets
      • Legal
        • Public records access laws & precedent
        • Legal Data Scraping
        • State Computer Crimes laws
      • Policy
        • Impartiality resolution
        • PDAP Access
        • PDAP Privacy Policy
        • Password Management
        • Personally Identifiable Information
    • Community calls
      • October 17, 2023
      • February 22, 2023
      • February 1, 2023
      • January 20, 2023
      • January 5, 2023
      • October 25, 2022
      • September 22, 2022
      • August 23, 2022
      • October 2, 2021
      • September 25, 2021
      • September 11, 2021
      • September 4, 2021
      • August 7, 2021
      • July 27 Dolt Bounty retro
      • July 17, 2021
      • July 10, 2021
      • June 26, 2021
      • June 19, 2021
      • June 12, 2021
      • June 5, 2021
      • May 1, 2021
      • April 24, 2021
    • Newsletter
    • Join our Discord
Powered by GitBook
On this page
  • Parts
  • Core principles, aka value add
  • Insights
  • Workflows powered by Splunk
  • Specific abilities granted by Splunk
  • Workflows not supported yet

Was this helpful?

Edit on GitHub
  1. Meta
  2. Operations
  3. Staff resources
  4. Meeting Minutes

2020 11-21 Tech Stack discussion

Broad strokes: we determined that the MVP fulfills the pieces not currently achieved by Splunkβ€”which could serve as the entire front end at very small scale. This means data integrity and depth are the core of the value added by PDAP.

Parts

Ingestion β†’ Archival Storage β†’ Search & analysis

Core principles, aka value add

Data stewardship

Transparency

Discipline

β€œLibrarian”

Insights

"I want police data."

Make a query β†’ get information

Enter search via UI (selects) or code β†’ present specific data

api (json), chart, csv

"I want to be able to analyze the police data I found."

Analysis tools or analysis that is done for you

Find extremes in the data automatically

"PDAP needs to verify data."

  • Guard the submissions process

  • Credibility score for each type of data

"PDAP needs to be like a librarian."

  • Nonintrusive

  • Pedigree

    • Legally captured?

    • Multiple sources?

    • Anonymity?

"What does it mean to verify data?"

"We need to be able to get data out of cold storage."

  • Eventually it'll be too much data for Splunk

Workflows powered by Splunk

Query data β†’ Analyze data β†’ Export insights

Upload data β†’ Analyze data* *The user agrees that we can keep the data, and provides information or verification about it.

Save an Analysis β†’ Share the Analysis with someone else

Save an Analysis β†’ Revisit it with updated data Alert user if an analysis changes based on updated information

Specific abilities granted by Splunk

  • easily write regex

  • accept any type of data

    • oddly / non-delimited

    • many file types

  • faster analysis / searching on the server rather than locally

  • automatically find "interesting fields"

  • search

  • analysis

Workflows not supported yet

Supply data to PDAP by volunteering or other sources

Verify submitted data β†’ Request more info from a submitter

Provide an unprecedented breadth of data

Safely archive historic data for the foreseeable future

Understand the categorization structure We need to make sure the structure is future-proof, and establish policies for sortation that cannot easily be corrupted.

Previous2021-03-14Next2020-09-30 Leadership Cadence

Was this helpful?

πŸ”