Ingest media
Status: Proposed 2026-09-29, Open Library books added 2026-10-24
The media library is seeded, not scraped. ingest:media:wikidata finds films about the cause, ingest:media:tmdb enriches them, and ingest:media:openlibrary adds books. All three POST to POST /api/ingest/media under the contract. Items land draft unless the source is trusted; an admin reads the synopsis, checks the links, and publishes. Nothing here hosts a film or a book.
Wikidata
Wikidata is CC0. The query finds films and documentaries whose main subject (P921) is veganism, animal rights, intensive animal farming, or animal welfare.
SELECT DISTINCT ?film ?filmLabel ?year ?tmdb ?imdb WHERE {
VALUES ?subject { wd:Q181138 wd:Q426 wd:Q912362 wd:Q459426 }
?film wdt:P921 ?subject .
?film wdt:P31/wdt:P279* wd:Q11424 .
OPTIONAL { ?film wdt:P577 ?date . BIND(YEAR(?date) AS ?year) }
OPTIONAL { ?film wdt:P4947 ?tmdb }
OPTIONAL { ?film wdt:P345 ?imdb }
SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . }
}
ORDER BY DESC(?year)
Q181138 is veganism, Q426 animal rights, Q912362 intensive animal farming, Q459426 animal welfare. Q11424 is film; documentary film (Q93204) is a subclass, so the P31/P279* path covers both. Send with Accept: application/sparql-results+json and a descriptive User-Agent; the query service throttles anonymous generic clients.
| Media item | From |
|---|---|
sourceId | the Q-id, Q1234567 |
title | filmLabel |
kind | documentary when P31 includes Q93204, else film |
year | year |
externalIds | { wikidata, tmdb, imdb } |
tags | environment for intensive animal farming, ethics for the other three; admins adjust |
sourceUrl | https://www.wikidata.org/wiki/ followed by the Q-id |
source is wikidata. Talks, shorts, and series are curated by hand; Wikidata coverage of those is thin.
TMDB enrichment
ingest:media:tmdb runs after the Wikidata step over items that have externalIds.tmdb, and needs TMDB_API_KEY. It re-sends each row under its existing key (source: wikidata, the same sourceId) with the fields below added, so the upsert updates the row instead of creating a second one. source: tmdb is used only for a title added by hand-picked TMDB id that Wikidata does not know.
| Field | From TMDB | Rule |
|---|---|---|
synopsis | overview | a draft only; admins rewrite it in the app's own words before publishing |
posterKey | poster_path | the script downloads, re-encodes, and uploads to the media bucket under media/posters/ plus the sourceId, then sends only the key. The API never fetches a remote image. |
watchLinks | watch providers for region US: flatrate, free, rent, buy | { provider, url } where url is the title's watch page from the same response, since providers expose no deep links |
backdropKey | backdrop_path | same rule as the poster, under media/backdrops/ |
trailerYoutubeId | videos of type Trailer on YouTube, official first, then any trailer, then a teaser | the client embeds through youtube-nocookie.com and loads on click only |
year, releaseDate | release_date | year only when Wikidata had none |
tagline | tagline | shown under the title on the item page |
runtimeMinutes | runtime | rounded |
directors | credits crew with job Director | at most ten, deduplicated |
featuring | credits cast in billing order | at most six names; documentaries list the people on screen |
genres | genres[].name | at most ten |
rating, ratingCount | vote_average, vote_count | one decimal; omitted when there are no votes |
contentRating | the US certification from release_dates | omitted when the US has none |
originalLanguage | original_language | ISO 639-1 |
All of these ride on one request per film (append_to_response=videos,credits,release_dates), then one more for watch providers. A curated year, synopsis or trailerYoutubeId is never overwritten; the rest is source-owned and refreshes on every run unless an admin edited it.
Watch providers require the line "Watch providers data by JustWatch" wherever watchLinks render. The client shows it on the item page; the ingest sets nothing for it.
Poster upload needs the media bucket's write role, which the bot never holds. When the enrichment runs where that role is absent it omits posterKey and sends the rest; a later run on the API host fills it in.
The editorial layer in the seed
scripts/data/media-seed.json is the curated catalogue and the editorial layer in one file. Beside the ids, each entry may carry:
| Field | Meaning |
|---|---|
tags | topic tags from the vocabulary ethics, health, environment, activism, investigation, plus free tags; the automatic home rows are built from the vocabulary |
kind | documentary (default), film, series, talk, short, or book |
contentWarnings | short phrases such as "graphic footage of animal suffering"; shown on the item page, never hidden behind a click |
officialSite | the film's own site |
watchLinks | only official free streams, with access: free; TMDB providers are appended after them, never in front |
actions | one or two { label, url, type, org } links to a well-known organisation: petition, donate, pledge, volunteer, guide, learn |
featured | read by the shelf seed on the host, never sent to the endpoint |
Every script that reads the file goes through one mapper (toItem in the seed script), so the Wikidata pass and the seed pass agree on sourceId and a title is never inserted twice.
The data refresh on the host
seed:media:collections runs on the API host (it needs the database), after the three ingest steps, as part of the data refresh. It upserts the shelves in scripts/data/media-collections.json by slug, resolves each shelf's titles to the rows the seed created, sets featured on the titles the seed flags, and with --prune-duplicates removes a second copy of a curated title left by an earlier mapper mismatch, keeping the row the seed owns. A title the library does not have yet is skipped with a log line and picked up on the next run. Bots never run this step.
Run the media steps together
ingest:media:seed, ingest:media:wikidata and ingest:media:tmdb write the same rows in turn, each owning different fields. The seed re-sends the editorial layer (tags, warnings, official site, the free official stream, actions) and so replaces watchLinks with the curated links and drops the two attribution tags until the TMDB step runs again. Run the three in that order, as the automation and the host's data refresh do; a seed run on its own leaves the library without providers until the next TMDB pass. A field a script does not send is never touched, so a run without a synopsis keeps the one TMDB wrote.
Open Library books
ingest:media:openlibrary searches the Open Library Search API for subject:veganism and subject:"animal rights". Catalogue metadata is CC0. Each work becomes one book row under source: openlibrary, upserted by the work id, and a work found under both subjects keeps both topic tags. openlibrary stays out of TRUSTED_SOURCES, so the rows land draft.
| Media item | From |
|---|---|
sourceId | the Open Library work id |
title | the work title |
kind | book |
authors | catalogue names, at most ten. These are not members |
year | first_publish_year, kept when it falls between 1000 and 2100 |
externalIds.isbn | one ISBN, an ISBN-13 when a valid one is present |
externalIds.openLibrary | the same work id |
tags | open-library, plus veganism or animal-rights |
sourceUrl | the Open Library work page |
posterKey | the cover, after it is downloaded, stripped of EXIF, XMP, and IPTC, and stored in the media bucket |
The script waits one second between Open Library calls, including cover downloads, and skips a cover already in the bucket. The API receives the object key. Covers are re-hosted rather than hotlinked. When the bucket role is absent, covers are skipped and the book rows still post. --dry-run searches, maps, and prints counts without posting.
The command runs monthly, on the automations page, and the host data refresh runs it after the TMDB step. It does not rewrite Wikidata or seed rows: those use other (source, sourceId) keys.
What is never done
- No film is hosted or proxied.
watchLinksleave the app. - Trailers are YouTube ids only, embedded through
youtube-nocookie.com, loaded on click. - No hotlinking of TMDB, Wikimedia, or Open Library images.
- No scraping of streaming sites or JustWatch pages. TMDB's API with a key is the only provider source.
Cadence
The film steps run weekly, as the media job on the automations page. The Wikidata query returns a few dozen rows, and the TMDB step makes two calls per item, well inside TMDB's limits at that size. Open Library books run monthly on that same page: about 1,600 veganism works and about 1,400 animal-rights works at the time of the change, with overlap, paced at one request per second. Admins verify watch links quarterly, per the media feature page. Books wait in draft until an admin publishes them.