Ingest events
Status: Updated 2026-09-30
Events come only from organizations on an allowlist, scripts/data/event-sources.json in the API repo. Six scripts read it, one per route kind: ingest:events:ics (calendar feeds), ingest:events:jsonld (pages that embed schema.org Event data, including list pages that link to detail pages), ingest:events:mobilize (the Mobilize public API), ingest:events:tribe (The Events Calendar REST API on WordPress sites), ingest:events:json (a static JSON file behind an organization's own events page, one named mapper per site) and ingest:events:dxe (Direct Action Everywhere's chapter database). All six POST to POST /api/ingest/events under the contract; every outbound request carries the ingest User-Agent, honours robots.txt (a 403 or 404 answer counts as absent) and is paced to one request per second per host. An event listing is public data; who attends is not, and no attendee field is ever read.
The allowlist
[
{ "source": "ics:saleranch", "hostName": "Sale Ranch Animal Sanctuary", "ics": "https://example.org/events/?ical=1" },
{ "source": "jsonld:openbarn", "hostName": "The Open Barn", "url": "https://example.org/event", "detailLinkPattern": "^/event-details-registration/" },
{ "source": "mobilize:thehumaneleague", "kind": "mobilize", "hostName": "The Humane League", "organizationId": 26695, "sourceUrl": "https://www.mobilize.us/thehumaneleague/" },
{ "source": "tribe:plantbasedtreaty", "kind": "tribe", "hostName": "Plant Based Treaty", "url": "https://example.org/wp-json/tribe/events/v1/events", "sourceUrl": "https://example.org/events/" },
{ "source": "json:seashepherd", "kind": "json", "mapping": "seashepherd", "hostName": "Sea Shepherd Conservation Society", "url": "https://example.org/events.json", "sourceUrl": "https://example.org/events/" },
{ "source": "dxe:losangeles", "kind": "dxe", "hostName": "Direct Action Everywhere Los Angeles", "pageId": "153660568381244", "sourceUrl": "https://www.directactioneverywhere.com/events", "enabled": false, "decision": "Facebook-derived; awaiting the owner's call." }
]
The file is a flat array. source is the item's source (the value TRUSTED_SOURCES matches); hostName is the hostName of every item and never comes from the feed. kind picks the script (ics, jsonld, mobilize, tribe, json, dxe); the first entries have no kind and are routed by whether they carry ics or url.
| Field | Meaning |
|---|---|
ics | feed URL (ics) |
url | list page (jsonld), REST endpoint (tribe) or JSON file (json) |
detailLinkPattern | regex matched against the path of same-origin links on the list page; the matching pages carry the Event JSON-LD |
maxPages | detail pages fetched per list page; lowers the crawl cap of 50, never raises it |
titleFilter | case-insensitive regex an event title must match to be kept; every script applies it before mapping |
defaultType | the type used when no keyword matches; other when absent |
enabled | false parks the entry; every script skips it and logs source skipped with reason: disabled |
placeholder | a note that the source publishes nothing usable yet; skipped with reason: placeholder |
organizationId | Mobilize organization id (mobilize) |
pageId | DxE chapter page id, digits only (dxe) |
mapping | the named mapper for a static file (json); today seashepherd |
region | any keeps events outside California (json); the default keeps California only |
sourceUrl | the organization's public events page: an item's link when the feed gives none of its own |
trust, scope, decision, verified, notes | documentation only; the moderation tier is set by TRUSTED_SOURCES on the API |
Mapping to the event item
| Event item | ICS (RFC 5545) | JSON-LD Event |
|---|---|---|
sourceId | UID, plus / and the instance DTSTART when the event has an RRULE | @id, else the url |
title | SUMMARY | name |
type | inferred, else the source's defaultType | inferred, else defaultType |
startsAt | DTSTART as ISO 8601 with the TZID offset | startDate |
endsAt | DTEND, else DTSTART plus two hours | endDate, same fallback |
venueName | first line of LOCATION | location.name |
address | remaining lines of LOCATION | location.address joined |
location { lng, lat } | GEO, else absent (stored without a map pin) | location.geo, else absent |
description | DESCRIPTION, HTML stripped, at most 4000 characters | description, same |
hostName | the allowlist org | the allowlist org |
sourceUrl | URL, else the feed page from the allowlist | url |
visibility | public | public |
titleFilter, when set, is applied to the title before anything else is mapped. Detail links on a list page are followed with their query string stripped (aggregators add ?referrer= to every href); the ?format=ical and ?format=json variants that Squarespace's robots.txt disallows are never followed. An ICS URL that answers an empty or non-calendar body (The Events Calendar does this when nothing is upcoming) counts as zero events and does not fail the run.
Recurring events are expanded 90 days ahead. Events that already ended are not sent. Cancelled items (STATUS:CANCELLED, eventStatus: EventCancelled) are not sent; an event sent earlier that later disappears stays listed until its date passes, and an admin cancels it if needed.
ORGANIZER, ATTENDEE, a JSON-LD organizer that is a Person, and attendee are never read. hostName always comes from the allowlist, not from the feed.
An event without coordinates is stored without a point. It appears in lists but not on the map until an admin adds one.
Route kinds added in round 2
mobilize. GET https://api.mobilize.us/v1/organizations/<organizationId>/events?timeslot_start=gte_now&per_page=100, following next (same host only, at most 20 pages). Kept only when visibility is PUBLIC, approval_status is APPROVED, and location.region is CA or is_virtual is true. One item per timeslot that has not ended and starts within 90 days; sourceId is <event id>/<timeslot id>. Mapping: title; description, else summary; browser_url as sourceUrl (else the entry's sourceUrl); timeslots[].start_date and end_date (unix seconds) rendered in the event's timezone; location.venue as venueName; address_lines, locality, region and postal_code as address; location.location.latitude and longitude as the point; a virtual event with no venue shows venueName Online. Type: RALLY to protest, VISIBILITY_EVENT to outreach, MEETING, MEET_GREET and WORKSHOP to meeting; anything else (including COMMUNITY) by keywords, then defaultType. Never read: contact, created_by_volunteer_host, sponsor, any attendee or attendance field.
tribe. GET <url>?per_page=50&start_date=now&page=n up to total_pages (cap 10). { events: [] } and { events: null } are valid empty answers. Only status: publish. Mapping: title; description with HTML stripped; url as sourceUrl; utc_start_date and utc_end_date rendered in the event's timezone, else start_date and end_date read as wall clock in that timezone; venue.venue as venueName; venue.address, city, stateprovince and zip as address; venue.geo_lat and geo_lng as the point (an empty venue means no place); sourceId is global_id, else id, else url. Never read: organizer, venue.phone.
json (mapping: seashepherd). Reads upcoming[] only. Mapping: id as sourceId; title; start_iso (offset included) as startsAt, date as a fallback; end_time (a clock time) on the same local day as endsAt, dropped when it lands before the start; venue split into venueName and address with city_label appended; lat and lng as the point; bring as the description; type by title keywords first, else meetup to meeting and cleanup and benefit to other, else defaultType. Only items whose city_label or venue contains , CA or California are kept unless the entry says region: "any". sourceUrl is tickets only when it is on the organization's own host; otherwise the entry's sourceUrl, then maps_link. Never read: leader, chapter_email, sponsor_name, sponsor_url, past[].
dxe. GET https://adb.dxe.io/external_events/<pageId>?start_time=<now minus 1 hour>&end_time=<now plus 180 days> per chapter, the only URL the script can build (a non-numeric pageId is refused, so the chapter endpoint that returns internal fields cannot be called). { events: null } is a valid empty answer. Mapping: ID as sourceId; Name; Description with HTML stripped; StartTime and EndTime (UTC) rendered as Los Angeles; LocationName free text, first part as venueName and the rest as address; IsCanceled drops the event; sourceUrl is EventbriteURL only when it is an https eventbrite.com link, else the chapter's sourceUrl from the allowlist, never a Facebook URL. No coordinates are sent, so items land without a map pin for an admin to place. Never read: AttendingCount, InterestedCount, Cover, PageID, EventbriteID, Lat, Lng, LastUpdate.
Trust tiers
Auto-publish (in TRUSTED_SOURCES): the organization publishes its own feed or markup and the host is the organization: ics:saleranch, jsonld:kindredspirits, jsonld:openbarn, jsonld:marthasfarm, jsonld:pebbleranch, tribe:plantbasedtreaty, mobilize:thehumaneleague, json:seashepherd.
Pending (never in TRUSTED_SOURCES): jsonld:animalrightscalendar-la and -sd (activist-submitted, moderated by Anonymous for the Voiceless, no organizer in the markup, so every item carries the aggregator as host), every dxe:* entry (Facebook-derived, free-text addresses, no coordinates), jsonld:ourhonor once activated, and anything from Eventeny or Nsefu if approved.
Type inference
Keywords in the title, checked in this order. First match wins.
| Keywords | type |
|---|---|
| vigil | vigil |
| cube, outreach, leaflet, tabling | outreach |
| potluck, dinner, brunch | potluck |
| tour, volunteer, work day, sanctuary | sanctuary_day |
| screening, film, documentary | screening |
| protest, march, rally, demonstration, disruption | protest |
| meeting, orientation, training | meeting |
| none | the source's defaultType, else other |
Host resolution
hostName is slugified and matched to organizations.slug. A match sets hostType: organization and hostId. No match creates an organization { name, slug, type: org, verified: false } with the same source and sourceId equal to the slug, so the event still has a host and an admin can verify it later. Ingested events have no createdBy; provenance identifies them.
Forbidden
- Facebook, Instagram, Meetup, and Eventbrite HTML. Their terms forbid it and the pages carry personal accounts.
- Attendee data of any kind: counts, names, handles, RSVPs from the source.
- Sources not on the allowlist, even when they publish an ICS feed.
- Fetching Eventbrite or Facebook. A feed's own Eventbrite link may be stored as an event's
sourceUrl(it is never fetched); a Facebook link never is.
Sources beyond ICS and JSON-LD
A round of research on 2026-09-30 across some fifty national, global and Southern California organisations found that most publish nothing machine-readable (their events live on Facebook, Instagram, Meetup or Eventbrite, which are never read), and that the ones that do fall into five shapes. Two are the feeds above. The other three each get a small script: a public JSON API (Mobilize, used by The Humane League), The Events Calendar's REST endpoint (wp-json/tribe/events/v1/events, used by Plant Based Treaty), and a site's own static JSON file (Sea Shepherd). The Animal Rights Calendar, the aggregator behind the Cubes of Truth, is plain JSON-LD once the listing's tracking query strings are stripped from its links. The field mappings and the fields each fetcher must never read are documented with the scripts; the schedule is on the automations page.
The allowlist grew three optional fields: enabled (a source switched off stays documented), titleFilter (a case-insensitive pattern a title must match, for calendars that mix in unrelated events), and trust (a note on why the source is or is not on TRUSTED_SOURCES).
Decisions awaiting the owner
These sources are in the allowlist with enabled: false and do not run until a person decides.
| Source | Question |
|---|---|
| Direct Action Everywhere chapters (Los Angeles, Orange County, San Diego, Inland Empire) | DxE serves its chapters' events from its own API, but the rows are synced from Facebook events. The fetcher drops attendee counts and cover images by construction and never calls the chapter endpoint that exposes internal fields; whether Facebook-derived data is acceptable under this policy is the owner's call. If enabled, these land pending. |
| Vegan Street Fair on Eventeny | The organisation's own site has no event markup; its listing on a ticketing platform carries valid Festival JSON-LD. Whether a third-party platform page counts as the organisation's own site is undecided, and whether the series URL rolls to the next date is unverified. |
| San Diego VegFest (Nsefu Wildlife Conservation Foundation) | Valid JSON-LD, but the same calendar carries non-vegan events. Enabled only with the titleFilter `vegfest |
| Mercy For Animals, Farm Sanctuary | Events are sold through Eventbrite. Its official API is allowed by the rules but needs a token and an organisation id; neither is set up. |
| Animal Rights Calendar host resolution | One pending source under the aggregator's name (today), or one source per organization page on the calendar if those pages render links server-side (unverified). |
| Our Honor | Activate when its list shows a current event, and confirm that virtual webinars belong in a Southern California events list. |
Adding a source
The bot does not edit the allowlist at runtime. It opens a pull request to scripts/data/event-sources.json with the feed URL, the organization's name as written on its own site, and the --dry-run count in the PR description. A person merges it. The next daily run picks it up.
Cadence
Daily, in the data-refresh order: ingest:events:ics, ingest:events:jsonld, ingest:events:mobilize, ingest:events:tribe, ingest:events:json, ingest:events:dxe. Each run fetches every enabled source (skips are logged with a reason), POSTs in batches of at most 200, and reads the 207 rejections per the runbook. A new source runs --dry-run once first. The cron line lives on the automations page.