iamnbutler/shelfgoblinPublic

Shelf Goblin — discover LitRPG audiobooks, follow series, and track your library.

Add backfill subcommand for historical year data (#33)

Adds a `backfill` CLI subcommand that exhaustively fetches audiobook data
for historical years (2020-present). Supports single year (--year),
range (--from/--to), and full backfill (--all) modes.

The backfill pipeline runs MIGRATE -> FETCH -> EXPORT stages with
non-incremental fetching (ignores cursors), a configurable delay
between years (default 5s), and per-year progress reporting. Data goes
into the same database and exports to the same static/data/{year}.json
format. Interrupted runs can be resumed since search cursors track
per (source, search_key, year) progress.
IAiamnbutler authored
bbf1ca9Patch
Parent0dea32e
3 files changed+260-9
StackedSplit
package.json+1 −0
scripts/backend/index.ts+102 −9
scripts/backend/pipeline.ts+157 −0