Splunk
Splunk is an intelligence analysis tool to make sense of vast amounts of data.
- can run python script to save sanitized snapshots on a recurring basis
- can make API requests to other APIs, and has an API to make requests against
- can save alerts for the appearance of specific data
Access it here. For this iteration, there’s a shared username and password. Reach out to Alec Akin for it. This is installed on a $10 Vultr instance with 2gb ram, 55gb ssd, and 1 core.
If you want to install a private instance for local use, it only takes about 10 minutes to set up on Mac or Windows. Here’s where to get started.
When fed a batch of 3,000,000 documents from NIBRS, it quickly revealed that the 8 days responsible for the most hate crime in the United States were those immediately following September 11. The 9th was the final day of the Rodney King riots. It’s also a way to power parts of the site: a front end app could consume parts of the Splunk API.
Query data → Analyze data → Export insights
Upload data → Analyze data*
*The user agrees that we can keep the data, and provides information or verification about it.
Save an Analysis → Share the Analysis with someone else
Save an Analysis → Revisit it with updated data
Alert user if an analysis changes based on updated information
Query data from many dolt repos with the same format
- easily write regex
- accept any type of data
- oddly / non-delimited
- many file types
- faster analysis / searching on the server rather than locally
- automatically find "interesting fields"
- search
- analysis
Here’s how to make queries in SPL, Splunk’s proprietary searching language.
index=nibrs source="hatecrimes.csv" incident_date="12-SEP-01"