Recipes/Python requests

Python requests

A batch of videos, drained as each job finishes.

The code

import os, time, requests
 
API = "https://api.tubeextract.dev/v1"
H = {"X-API-Key": os.environ["TUBEEXTRACT_KEY"]}
 
schema = {
    "multiple": True,
    "columns": [
        {"name": "topic",
         "description": "A subject the host discusses at length"},
    ],
}
 
# one call for the whole list, up to 200 videos
job_id = requests.post(API, headers=H,
                       json={"videos": urls, "schema": schema}).json()["job_id"]
 
# results is None until every video has settled
while True:
    time.sleep(2)
    job = requests.get(f"{API}/{job_id}", headers=H).json()
    if job["results"] is not None:
        break
 
# handed over once, so persist before anything else
for video in job["results"]:
    if video["status"] == "done":
        save(video["video_id"], video["rows"])
    else:
        log(video["video_id"], video["error"]["code"])

Notes

For a large batch, poll /batches/:id for counts and only fetch the jobs it reports as finished, rather than asking after every job on every pass. Failures are logged and dropped from the set instead of retried, since most job errors will not succeed a second time.

Was this page useful?Tell us what was missing

Node fetch

Go net/http