Labeling events

A guide for labeling events, or "annotation parties"

Goals of each event

• Train a small group on data labeling

• Add as many data sources to the database as we can

• Set people up to label on their own time

• Refine our data labeling tools

Structure of activities

In advance

1. Participants RSVP to the event, held on Discord or in-person.

2. Organizers reach out to participants: "Thank you for signing up to label data sources! If you have not already, please create a PDAP account here. You will also need to create a free Retool account to access our labeling interface. Please let me know if you would like to find data for a particular place or topic area, or if you have any questions. Otherwise, we'll see you at the event!"

3. Participants create PDAP and Retool accounts.

4. Organizers give participants the appropriate permissions in the back-end.

5. If anyone has location or subject matter interests, organizers give them instructions for following the appropriate search.

At the event

1. Organizers welcome participants and kick off popcorn-style introductions.

2. Organizers screen-share the process of:

   1. selecting a batch

   2. labeling for relevancy

   3. labeling for record type

   4. labeling for agency

3. Organizers open the floor for questions.

4. Participants are encouraged to begin labeling, and are free to ask questions about specific labels or parts of the process as they come up.

5. Labeling continues for an hour; participants are encouraged to continue after the event at their own pace.

6. Organizers use the Final Review screen to approve or adjust labels in real-time, giving feedback as necessary.