2021-04-10 Meeting notes
Was this helpful?
Was this helpful?
10 Apr 2021
Eddie Brown
Alec Akin
Eric Turner
Stabs
Answer schema questions to unblock .
Decide whether tableplus or DBeaver would be useful in the dolt pipeline ().
Item
Notes
Datasets
UUIDs have been added in dolt
may want to remove hyphens to save space (currently mysql built in)
Once things are more stable it’s probably worth doing some views
Eric wrote a script to keep CityProtect datasets up to date as well as commit the dataset to the dolt db
Fork then merge is a necessity
We could put these on a chron job or just manually one them—once for each portal type
These update quarterly, so automating that is not our biggest problem.
355 agencies in cityprotect for bulk downloading, avg 10 CSV files
We’ll need to get data out of datasets
Clones and forks become problematic just to make simple additions
Dolt x Scale
Risk: dolt is young and missing advanced SQL features. Dolt is not our advanced analytics tool. Dolt is not our deep storage layer.
Databases can’t currently reference each other, and repos are 1:1 with databases.
Data is unlimited but current technical cap is ~200gb
Can Dolt be an intake tool and not grow?
This may not even be our presentation layer because of the low amount of data it can store
one way around this might be checking data out of one dolt repo into another, preserving paper trail. does dolt support this / is it reasonable?
Where are we storing our data properties?
Is breadth an issue in Dolt? Would this table be prohibitively gigantic?
SQL column limit ~1000, mysql 4000. we should be good.
Hosting
If we went with mongodb directly we could bolt it into our DigitalOcean
Tableplus / DBeaver
Tableplus free has some limitations (open tabs)
Scrapers
For now—people should write whatever they want / is most convenient for them. If we want to refactor, we can figure out what the priority is on it + do it later.
Server access
We don’t use passwords, just keys—send it to Alec if you need access.
OCR
Stabs tried ~5 different ones and none of them worked well enough. Alec’s tried tensorflow
Dolt is not our advanced analytics tool. Dolt is not our deep storage layer.
Alternative for down the road:
Otherwise priority ↑ for
proof of concept for separate dolt DB (check whether DBs can easily reference one another) Done
proof of concept for global_properties table referenced by data_types views - JIRA project doesn't exist or you don't have permission to view it.
to make request in dolthub discord for one:many repos:dbs
reach out to 3 major cloud providers + mongodb for sponsorships
Point database volunteers in the direction of
add scraper philosophies to readme (do what you want, we can always refactor later)