Archive.org integrations and automation

The Internet Archive serves two very different needs through separate interfaces. One is the collection of items — books, audio, video, software and datasets — each identified by a unique identifier and described by metadata. The other is the Wayback Machine, the archive of captured web pages. Automations that mix them up get confusing results, so decide first whether you are working with items or with captures of URLs.

For items, the Advanced Search and Scrape APIs find identifiers by query over metadata fields such as collection, creator, date and mediatype, and the Scrape API pages through large result sets with a cursor. The Metadata API returns an item’s full record and file list, and can also apply edits to metadata for items you have rights to. Uploading files uses the S3-like interface, authenticated with an access key and secret generated in your archive.org account settings; after upload the Archive runs derive tasks to produce formats such as text or thumbnails, which take time to finish.

For the web archive, the Wayback Availability API reports the closest capture of a URL to a given date, and the CDX server lists every capture with timestamps, status codes and content digests, which is how you notice a page actually changed. Requesting a new capture uses Save Page Now, which requires an account and is subject to the Archive’s own pacing; a burst of save requests will be slowed or refused. Comparing digests from CDX is a cheap way to see whether a page changed before asking for anything heavier.

The Archive is a nonprofit service and asks automated clients to be polite: identify yourself, avoid hammering endpoints, and back off when told to. Plan schedules accordingly, cache identifiers you have already processed, and keep large file downloads deliberate.