/common/etllibrary from a specific scraped agency dir, pass the
schema.jsonfile to it.
agency_idexists and will grab the appropriate record from the database
dataitems in the
schema.json. It will create any datasets where the id is null, otherwise it will find an existing dataset and sync the data from the table with the
schema.json. Whichever source has the most recent
last_modifiedtime will be the “Source of Truth” and that’s the data that will be used to sync.
mappingobject, it will search for a table in
pdap/data-intakewith the exact name of the
data_typeand then sync the columns so they are there. If a new column is in the database but not the schema.json file, it will add the missing column with a
mappingobject and insert the data into
pdap/data-intake. Erroneous records are skipped a message displayed to the console.