iamnbutler/shelfgoblinPublic

Shelf Goblin — discover LitRPG audiobooks, follow series, and track your library.

Add Hardcover enrichment and Royal Road discovery fetchers

Hardcover.app (enrichment):
- GraphQL API searches for books already in our DB
- Adds community genre tags and stores raw response
- 1 req/sec rate limit (well under their 60/min), 7-day cache
- Gracefully skips if HARDCOVER_API_TOKEN is not set
- Enriches up to 100 books per run, prioritized by rating count

Royal Road (discovery):
- Scrapes best-rated lists for 6 genre tags (litrpg, gamelit,
  progression, cultivation, isekai, dungeon core)
- 3-second delay between page fetches, 7-day cache per genre
- Only ~6 pages total per full scrape — very light footprint
- Discovers new series not in our Audible search config
- Polite User-Agent identifying the project

Pipeline integration:
- FETCH stage now runs: Royal Road discovery → Audible fetch → Hardcover enrichment
- Each source can be run independently via --source flag
- Failures in Royal Road/Hardcover don't block the pipeline

HTTP client:
- Added post() method to HttpClient interface for GraphQL APIs

Closes #34, closes #45
IAiamnbutler authored
18c3407Patch
Parent3fc642f
4 files changed+469-22
StackedSplit
scripts/backend/fetchers/hardcover.ts+210 −0
scripts/backend/fetchers/royalroad.ts+185 −0
scripts/backend/http.ts+18 −4
scripts/backend/pipeline.ts+56 −18