# 2020 11-21 Tech Stack discussion

Broad strokes: we determined that the MVP fulfills the pieces not currently achieved by Splunk—which could serve as the entire front end at very small scale. This means **data integrity** and **depth** are the core of the value added by PDAP.

## Parts <a href="#id-202011-21techstackdiscussion-joshcsnotes-parts" id="id-202011-21techstackdiscussion-joshcsnotes-parts"></a>

**Ingestion** → **Archival Storage** → **Search & analysis**

## Core principles, aka value add <a href="#id-202011-21techstackdiscussion-joshcsnotes-coreprinciples-akavalueadd" id="id-202011-21techstackdiscussion-joshcsnotes-coreprinciples-akavalueadd"></a>

Data stewardship

Transparency

Discipline

“Librarian”

## Insights <a href="#id-202011-21techstackdiscussion-joshcsnotes-insights" id="id-202011-21techstackdiscussion-joshcsnotes-insights"></a>

"I want police data."

Make a query → **get information**

Enter search via UI (selects) or code → present specific data

api (json), chart, csv

"I want to be able to analyze the police data I found."

Analysis tools or analysis that is done for you

*Find extremes in the data automatically*

"PDAP needs to verify data."

* Guard the submissions process
* Credibility score for each type of data

"PDAP needs to be like a librarian."

* Nonintrusive
* **Pedigree**
  * Legally captured?
  * Multiple sources?
  * Anonymity?

"What does it mean to verify data?"

"We need to be able to get data out of cold storage."

* Eventually it'll be too much data for Splunk

## Workflows powered by Splunk <a href="#id-202011-21techstackdiscussion-joshcsnotes-workflowspoweredbysplunk" id="id-202011-21techstackdiscussion-joshcsnotes-workflowspoweredbysplunk"></a>

**Query data → Analyze data → Export insights**

***Upload data*** **→ Analyze data\***\
\**The user agrees that we can keep the data, and provides information or verification about it.*

**Save an Analysis → Share the Analysis with someone else**

**Save an Analysis → Revisit it with updated data**\
\&#xNAN;*Alert user if an analysis changes based on updated information*

### Specific abilities granted by Splunk <a href="#id-202011-21techstackdiscussion-joshcsnotes-specificabilitiesgrantedbysplunk" id="id-202011-21techstackdiscussion-joshcsnotes-specificabilitiesgrantedbysplunk"></a>

* easily write regex
* accept any type of data
  * oddly / non-delimited
  * many file types
* faster analysis / searching on the server rather than locally
* automatically find "interesting fields"
* search
* analysis

## Workflows not supported yet <a href="#id-202011-21techstackdiscussion-joshcsnotes-workflowsnotsupportedyet" id="id-202011-21techstackdiscussion-joshcsnotes-workflowsnotsupportedyet"></a>

**Supply data to PDAP by volunteering or other sources**

**Verify submitted data → Request more info from a submitter**

**Provide an unprecedented breadth of data**

**Safely archive historic data for the foreseeable future**

**Understand the categorization structure**\
\&#xNAN;*We need to make sure the structure is future-proof, and establish policies for sortation that cannot easily be corrupted.*


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.pdap.io/meta/operations/staff/meeting-minutes/project-home-2020-11-21-tech-stack-discussion-josh-cs-notes.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.