A 429 is how you get an anonymous bot's attention
Quote
only way I know how to handle that traffic... is to, um, is to rate limit it by hand. and start giving them a 429 because I don't know who this thing is.
Episode 262: Podcleanse (episode page, opens in a new tab) 01:34:01 (play audio from this timestamp, opens in a new tab)
Voice-matched
Summary
When unidentified traffic hammers the Podcast Index API (a vague user agent from some CDN), Dave's only tool is to rate-limit it by hand and return 429s. Usually the traffic just disappears. Sometimes he gets an email, explains what they were doing wrong, and offers a better way. One podcast app was trying to scrape every episode to backfill transcripts.
Not in the audio. Guest episode (John Spurlock; 0.35 voice threshold). The quote sits at the start of a 7-cue Dave run, in Dave's first-person account of running the Index API.
Also said
Episode 262: Podcleanse (episode page, opens in a new tab) 01:35:18 (play audio from this timestamp, opens in a new tab)
scrape you know every episode in the end you can't scrape 150 million episodes I will give you a better way let's work together because it's gonna it's going to degrade the platform for everybody else who also
Voice-matched