🕯️ Magic Note
HTTP (Hypertext Transfer Protocol) is the foundation of data communication on the web. Understanding HTTP requests is essential for any web scraping or API work. The requests library is the de facto standard for HTTP in Python, known for its simple and elegant API.
| HTTP Method | Purpose | Common Use |
|---|---|---|
| GET | Retrieve data | Fetching web pages, API data |
| POST | Submit data | Form submission, creating resources |
| PUT | Update entire resource | Replacing data |
| PATCH | Partial update | Modifying some fields |
| DELETE | Remove resource | Deleting data |
| HEAD | Retrieve headers only | Checking if a resource exists |
| Status Code Range | Meaning |
|---|---|
| 1xx | Informational (request received, continuing) |
| 2xx | Success (200 OK, 201 Created) |
| 3xx | Redirection (301 Moved Permanently, 302 Found) |
| 4xx | Client Error (404 Not Found, 403 Forbidden) |
| 5xx | Server Error (500 Internal Server Error) |
Bash
# Install the requests library
pip install requests
Python
import requests
Python
import requests
# Simple GET request
response = requests.get(“https://api.github.com”)
# Check status code
print(f”Status: {response.status_code}”)
# Get response as text
print(f”Text: {response.text[:200]}…”)
# Parse JSON response (if the server returns JSON)
data = response.json()
print(f”JSON: {data}”)
# Raise exception for bad status (4xx or 5xx)
response.raise_for_status() # Raises HTTPError if status >= 400
🕯️ Magic Note
The requests.get() function sends a GET request and returns a Response object. This object contains the server’s response: status code, headers, and content. The .json() method is particularly useful for API work.
Python
import requests
# Without params (manual URL building)
url = “https://api.github.com/search/repositories?q=python&sort=stars”
response = requests.get(url)
# With params (cleaner)
params = {
“q”: “python”,
“sort”: “stars”,
“order”: “desc”,
“per_page”: 5
}
response = requests.get(“https://api.github.com/search/repositories”, params=params)
# The actual URL used
print(f”URL: {response.url}”)
if response.status_code == 200:
data = response.json()
for repo in data.get(“items”, [])[:5]:
print(f”Repo: {repo[‘name’]} – Stars: {repo[‘stargazers_count’]}”)
Python
import requests
# Form data (application/x-www-form-urlencoded)
form_data = {
“username”: “feloriya”,
“password”: “securepass”,
“remember”: “true”
}
response = requests.post(“https://httpbin.org/post”, data=form_data)
print(f”Status: {response.status_code}”)
print(f”Response: {response.json()}”)
# JSON data (application/json)
json_data = {
“name”: “Feloriya”,
“age”: 25,
“skills”: [“Python”, “Web Design”]
}
response = requests.post(“https://httpbin.org/post”, json=json_data)
print(f”Sent JSON, received: {response.json()[‘json’]}”)
🕯️ Magic Note
Use the data parameter for form-encoded data (like HTML forms). Use the json parameter for JSON data. The json parameter automatically serializes your dictionary to JSON and sets the Content-Type header to application/json.
Python
import requests
# Basic headers
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36”,
“Accept”: “application/json”,
“Accept-Language”: “en-US,en;q=0.9”,
“Authorization”: “Bearer YOUR_TOKEN_HERE”
}
response = requests.get(“https://api.github.com/user”, headers=headers)
print(f”Status: {response.status_code}”)
# For web scraping, a realistic User-Agent is often necessary
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36”
}
response = requests.get(“https://example.com”, headers=headers)
Python
import requests
# Without session (each request is independent)
response1 = requests.get(“https://httpbin.org/cookies/set?name=value”)
response2 = requests.get(“https://httpbin.org/cookies”)
print(f”No session: {response2.json()}”) # Cookies may be missing
# With session (cookies persist)
session = requests.Session()
session.get(“https://httpbin.org/cookies/set?name=value”)
response = session.get(“https://httpbin.org/cookies”)
print(f”With session: {response.json()}”) # {‘cookies’: {‘name’: ‘value’}}
# Session also persists headers
session = requests.Session()
session.headers.update({“User-Agent”: “My Scraper/1.0”})
response = session.get(“https://httpbin.org/headers”)
print(“Headers from session persist across requests”)
🕯️ Magic Note
A Session object reuses the same TCP connection for multiple requests (improving performance) and persists cookies and headers. For web scraping multiple pages of the same site, always use a Session.
Python
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
# Basic timeout
try:
response = requests.get(“https://api.example.com”, timeout=5) # 5 seconds
except requests.exceptions.Timeout:
print(“Request timed out”)
# Separate connect and read timeouts
response = requests.get(“https://api.example.com”, timeout=(3.05, 10)) # (connect, read)
# Automatic retries with session
session = requests.Session()
retry_strategy = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=[“GET”, “POST”]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount(“http://”, adapter)
session.mount(“https://”, adapter)
response = session.get(“https://api.example.com”)
Python
import requests
# GET request to JSON API
response = requests.get(“https://jsonplaceholder.typicode.com/posts/1”)
if response.status_code == 200:
post = response.json() # Parses JSON into Python dict
print(f”Title: {post[‘title’]}”)
print(f”Body: {post[‘body’][:50]}…”)
# POST request to create a resource
new_post = {
“title”: “Python HTTP Requests”,
“body”: “Learning about requests is essential.”,
“userId”: 1
}
response = requests.post(“https://jsonplaceholder.typicode.com/posts”, json=new_post)
if response.status_code == 201: # Created
created = response.json()
print(f”Created post with ID: {created[‘id’]}”)
# Handle errors gracefully
try:
response = requests.get(“https://jsonplaceholder.typicode.com/posts/999999”)
response.raise_for_status() # Raises for 404, etc.
data = response.json()
except requests.exceptions.HTTPError as e:
print(f”HTTP error: {e}”)
except requests.exceptions.ConnectionError:
print(“Connection error (network problem)”)
except requests.exceptions.Timeout:
print(“Request timed out”)
except requests.exceptions.RequestException as e:
print(f”Request failed: {e}”)
Python
import requests
# Basic authentication (username and password)
response = requests.get(“https://api.example.com/private”, auth=(“username”, “password”))
# Bearer token (common for REST APIs)
headers = {“Authorization”: “Bearer YOUR_ACCESS_TOKEN”}
response = requests.get(“https://api.example.com/user”, headers=headers)
# API key as query parameter
params = {“api_key”: “YOUR_API_KEY”}
response = requests.get(“https://api.example.com/data”, params=params)
# API key as header
headers = {“X-API-Key”: “YOUR_API_KEY”}
response = requests.get(“https://api.example.com/data”, headers=headers)
Python
import requests
import sys
from pathlib import Path
def download_file(url, filename=None):
“””Download a file with progress indicator.”””
if filename is None:
filename = Path(url).name or “downloaded_file”
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36”
}
try:
# Stream the response to avoid loading entire file into memory
response = requests.get(url, headers=headers, stream=True, timeout=30)
response.raise_for_status()
# Get file size from headers (if available)
total_size = int(response.headers.get(“content-length”, 0))
downloaded = 0
with open(filename, “wb”) as f:
for chunk in response.iter_content(chunk_size=8192):
if chunk: # Filter out keep-alive chunks
f.write(chunk)
downloaded += len(chunk)
if total_size:
percent = (downloaded / total_size) * 100
sys.stdout.write(f”\rProgress: {percent:.1f}% ({downloaded}/{total_size} bytes)”)
sys.stdout.flush()
print(f”\nDownloaded to: {filename}”)
return True
except requests.exceptions.RequestException as e:
print(f”Download failed: {e}”)
return False
# Example usage
# download_file(“https://example.com/large-file.zip”)
🕯️ Magic Note
The stream=True parameter prevents loading the entire response into memory at once. This is essential for downloading large files. The iter_content() method yields chunks of data, allowing you to process them incrementally.
- Forgetting to handle exceptions (network errors, timeouts, HTTP errors)
- Not setting a timeout (request could hang forever)
- Assuming JSON response without checking response.headers[‘Content-Type’]
- Hard-coding sensitive data (API keys, passwords) in code (use environment variables)
- Not closing sessions (though with Session() as s: helps)
- Sending too many requests too quickly (getting rate-limited or banned)
- What is the difference between a GET request and a POST request?
- How do you add query parameters to a GET request using the requests library?
- What is the purpose of a Session object?
- How do you send JSON data in a POST request?
- Why should you set a timeout on requests?
- What does response.raise_for_status() do?
⚡ Whisper
The requests library is your telephone to the web. You dial a URL with get(). You send a message with post(). The server answers with a status code: 200 says “I am here”, 404 says “not found”, 500 says “I am broken”. The response comes back with headers (metadata) and a body (the content). You learn to read both. response.text for HTML. response.json() for API data. You learn to set headers to identify yourself. You manage cookies with sessions. You handle errors with try-except. You set timeouts so you do not wait forever. This is not magic. This is communication. Your program talks to servers around the world. The servers answer. Sometimes they give data. Sometimes they give errors. Your job is to ask politely, handle all answers gracefully, and respect the rules. Master requests, and you master conversation with the web. The web is listening. Ask your questions.
