IntermediateLesson 19 of 306 min read

Web requests and APIs

Call web APIs from Python, read JSON responses, check status codes and handle failures with timeouts and clear error handling.

In this lesson you will

  • Explain requests, responses and status codes
  • Read a JSON API response
  • Handle timeouts and errors
  • Keep API keys safe

About 22 minutes

Talking to the rest of the internet

Weather forecasts, exchange rates, news headlines, maps, payments: this information lives on other computers, and those computers will hand it to your program if you ask in the right way. The doorway is called an API (Application Programming Interface). A web API is simply a web address that returns data instead of a page, almost always as JSON, which you learned in the last lessons. This lesson shows how to call one, read the answer and handle the things that go wrong.

Good to know

The practice server in this lesson runs on your own computer so every example works without internet access. The code for calling real websites is the same; only the address changes.

How a web request works

  1. Your program sends a request to a URL, using a method such as GET (fetch data) or POST (send data).
  2. The server sends back a response: a status code, headers, and a body (often JSON).
  3. Your program checks the status code, then reads the body.
Status Meaning
200 OK, here is your data
201 Created (after a successful POST)
400 Bad request: you sent something wrong
401 / 403 Not logged in / not allowed
404 That address does not exist
429 Too many requests, slow down
500 The server had a problem

A tiny API on your own computer

To practise safely, this example starts a small web server inside the program, then calls it with Python’s built-in urllib. Focus on the second half, which is what a real client looks like.

import json, threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.request import urlopen

USERS = {"1": {"id": 1, "name": "Sara", "city": "Lahore"},
         "2": {"id": 2, "name": "Ali", "city": "Karachi"}}

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/users":
            self.send_json(200, list(USERS.values()))
        elif self.path.startswith("/users/") and self.path.split("/")[-1] in USERS:
            self.send_json(200, USERS[self.path.split("/")[-1]])
        else:
            self.send_json(404, {"error": "not found"})
    def send_json(self, code, data):
        body = json.dumps(data).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass

server = HTTPServer(("127.0.0.1", 0), Handler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

# --- the client side: what you will write against real APIs ---
with urlopen(f"http://127.0.0.1:{port}/users") as response:
    print("Status:", response.status)
    users = json.loads(response.read())

print(users)
for u in users:
    print(u["name"], "lives in", u["city"])
server.shutdown()
Status: 200
[{'id': 1, 'name': 'Sara', 'city': 'Lahore'}, {'id': 2, 'name': 'Ali', 'city': 'Karachi'}]
Sara lives in Lahore
Ali lives in Karachi

The requests package

The standard urllib works, but almost everyone installs the friendlier requests package (pip install requests). Here is how a real call looks. This code needs the internet, so it is shown as a pattern, not as run output.

import requests

response = requests.get("https://api.example.com/users", timeout=10)

print(response.status_code)      # 200
data = response.json()           # parse the JSON body into Python

for user in data:
    print(user["name"])

With requests, response.json() replaces the manual json.loads. Always pass timeout=: without it, a slow server can make your program wait forever.

import requests

# query parameters: ?city=Lahore&units=metric
params = {"city": "Lahore", "units": "metric"}
r = requests.get("https://api.example.com/weather", params=params, timeout=10)

# sending data with POST
payload = {"name": "Sara", "email": "sara@example.com"}
r = requests.post("https://api.example.com/users", json=payload, timeout=10)

# API keys usually go in headers
headers = {"Authorization": "Bearer YOUR_SECRET_KEY"}
r = requests.get("https://api.example.com/me", headers=headers, timeout=10)

Handling failures properly

Networks fail. Servers say no. A good client checks everything.

import json
from urllib.request import urlopen
from urllib.error import HTTPError, URLError
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        code = 200 if self.path == "/ok" else 404
        body = json.dumps({"path": self.path}).encode()
        self.send_response(code)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *a): pass

srv = HTTPServer(("127.0.0.1", 0), H)
port = srv.server_address[1]
threading.Thread(target=srv.serve_forever, daemon=True).start()

def fetch(path):
    url = f"http://127.0.0.1:{port}{path}"
    try:
        with urlopen(url, timeout=5) as r:
            return json.loads(r.read())
    except HTTPError as e:
        print(f"Server answered {e.code} for {path}")
    except URLError as e:
        print("Could not reach the server:", e.reason)
    return None

print(fetch("/ok"))
print(fetch("/missing"))
srv.shutdown()
{'path': '/ok'}
Server answered 404 for /missing
None
  • HTTPError: the server replied, but with an error status (404, 500).
  • URLError: the server could not be reached at all.
  • With requests, call response.raise_for_status() to turn bad statuses into exceptions, and catch requests.RequestException.
import requests

try:
    r = requests.get("https://api.example.com/data", timeout=10)
    r.raise_for_status()
    data = r.json()
except requests.Timeout:
    print("The server took too long")
except requests.HTTPError as e:
    print("Bad status:", e.response.status_code)
except requests.ConnectionError:
    print("No internet connection")

Working with the data you receive

import json

# the kind of text an API returns
raw = '''{"city": "Lahore", "current": {"temp": 31.5, "humidity": 40},
          "forecast": [{"day": "Mon", "high": 33}, {"day": "Tue", "high": 35}, {"day": "Wed", "high": 29}]}'''

data = json.loads(raw)
print(data["city"], data["current"]["temp"])

hottest = max(data["forecast"], key=lambda d: d["high"])
print("Hottest day:", hottest["day"], hottest["high"])

# be careful with missing keys
print(data.get("wind", "no wind data"))
print(data["current"].get("humidity"))
Lahore 31.5
Hottest day: Tue 35
no wind data
40

Use .get() when a key might be missing, so the program does not crash with a KeyError. Print the raw data once and look at its shape before writing code to dig into it.

Good manners and safety

  • Read the API’s documentation for its address, parameters and limits.
  • Respect rate limits; if you get 429, wait before retrying (time.sleep).
  • Keep secret keys out of your code. Store them in an environment variable and read them with os.environ.get("API_KEY"). Never upload keys to GitHub.
  • Use timeouts on every request.
  • Cache results you will reuse so you do not hammer the server.
  • Check the website’s terms of use before collecting data from it.
Common mistake

Pasting an API key straight into your code and then sharing the file. Bots scan public repositories for keys within minutes. If it happens, delete the key at the provider and create a new one.

Practice

From this response text, print every product name whose price is under 100: [{"name": "Pen", "price": 45}, {"name": "Bag", "price": 1250}, {"name": "Ruler", "price": 80}]

Show one possible answer
import json

raw = '[{"name": "Pen", "price": 45}, {"name": "Bag", "price": 1250}, {"name": "Ruler", "price": 80}]'
for item in json.loads(raw):
    if item["price"] < 100:
        print(item["name"])
Pen
Ruler
Practice

Write a function get_nested(data, *keys) that digs through nested dictionaries and returns None if any key is missing.

Hint: Loop through the keys, stepping into the data each time.

Show one possible answer
def get_nested(data, *keys):
    for key in keys:
        if not isinstance(data, dict) or key not in data:
            return None
        data = data[key]
    return data

info = {"a": {"b": {"c": 42}}}
print(get_nested(info, "a", "b", "c"))
print(get_nested(info, "a", "x", "c"))
42
None
Free practice APIs

Search for free public APIs (weather, jokes, country data, space photos). Many need no sign-up. Start with something fun, since you will enjoy the project more.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is an API?

  1. A kind of loop
  2. A web browser
  3. A database file
  4. A way for programs to request data from each other

A web API returns data, usually JSON.

What does status code 404 mean?

  1. Success
  2. The address was not found
  3. Server crashed
  4. Not allowed

404 = not found.

Why pass timeout= to requests?

  1. To speed up the server
  2. To log in
  3. To stop waiting forever on a slow server
  4. To cache data

Without it the call may hang indefinitely.

Where should you store an API key?

  1. In an environment variable, not in shared code
  2. In a public file
  3. In a comment
  4. In the URL

Keys are secrets.

What does response.json() do?

  1. Sends JSON
  2. Saves a file
  3. Parses the JSON body into Python data
  4. Opens a browser

It converts the JSON text into dicts and lists.

Summary

  • A web API returns data (usually JSON) from a URL.
  • Check status codes and use timeouts.
  • response.json() parses the body.
  • Keep API keys out of your code.

Next lesson

Next: your intermediate project, a command-line expense tracker that saves its data.

Questions

requests is not found. What now?

Install it with pip install requests, inside your virtual environment.

What is the difference between GET and POST?

GET fetches data; POST sends data to create something.

Why do some APIs need a key?

So the provider can identify you, limit usage and block abuse.

Progress is saved in this browser.