🕯️ Magic Note
Web scraping walks a fine line between gathering public data and violating terms of service. Always check a website’s robots.txt file (e.g., https://example.com/robots.txt) and terms of service before scraping. Respect rate limits and identify your bot. Be a good citizen of the web.
Bash
# Install requests and beautifulsoup4
pip install requests beautifulsoup4
# Optional: lxml parser (faster than built-in)
pip install lxml
# Optional: for JavaScript-heavy sites
pip install selenium
Python
import requests
# GET request (most common)
response = requests.get(“https://example.com”)
# Check status code
print(f”Status code: {response.status_code}”) # 200 means OK
# Get the HTML content
html_content = response.text
print(f”Content length: {len(html_content)} characters”)
# Raise exception for bad status codes
response.raise_for_status() # Raises HTTPError for 4xx or 5xx
# Add headers to mimic a browser
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36”
}
response = requests.get(“https://example.com”, headers=headers)
Python
from bs4 import BeautifulSoup
html = “””
<html>
<body>
<h1 id=”title”>Welcome to My Site</h1>
<p class=”description”>This is a paragraph.</p>
<div class=”content”>
<a href=”https://example.com”>Click here</a>
<a href=”https://google.com”>Visit Google</a>
</div>
</body>
</html>
“””
# Parse the HTML
soup = BeautifulSoup(html, “html.parser”)
# Pretty print (makes HTML readable)
print(soup.prettify())
🕯️ Magic Note
BeautifulSoup can use different parsers: “html.parser” (built-in), “lxml” (faster, needs installation), and “html5lib” (most lenient). For most projects, “html.parser” is sufficient.
Python
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, “html.parser”)
# Get first h1 tag
h1 = soup.h1
print(f”Tag: {h1}”)
print(f”Text: {h1.text}”)
# Get first p tag
p = soup.p
print(f”Paragraph: {p.text}”)
# Get all a tags (returns a list)
all_links = soup.find_all(“a”)
for link in all_links:
print(f”Link text: {link.text}, href: {link.get(‘href’)}”)
Python
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, “html.parser”)
# Find by ID
title = soup.find(id=”title”)
print(f”By ID: {title.text}”)
# Find by class
description = soup.find(class_=”description”) # Note: class_ (underscore)
print(f”By class: {description.text}”)
# Find all elements with a specific class
content_divs = soup.find_all(class_=”content”)
# Using CSS selectors (more powerful)
links = soup.select(“div.content a”)
for link in links:
print(f”CSS selector: {link.get(‘href’)}”)
# CSS selector for ID
title = soup.select_one(“#title”)
print(f”CSS ID selector: {title.text}”)
Python
from bs4 import BeautifulSoup
html = “””
<div id=”container”>
<ul>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
</ul>
</div>
“””
soup = BeautifulSoup(html, “html.parser”)
# Get children
container = soup.find(id=”container”)
children = list(container.children) # Includes text nodes
# Get direct children (tags only)
for child in container.find_all(recursive=False):
print(f”Child: {child.name}”)
# Get parent
first_li = soup.find(“li”)
parent_ul = first_li.parent
print(f”Parent of li: {parent_ul.name}”)
# Get siblings
for sibling in first_li.next_siblings:
if sibling.name == “li”:
print(f”Sibling: {sibling.text}”)
Python
from bs4 import BeautifulSoup
html = “””
<div class=”product”>
<h2>Python Book</h2>
<p class=”price”>$29.99</p>
<a href=”/products/123″ class=”details” data-id=”123″>View Details</a>
</div>
“””
soup = BeautifulSoup(html, “html.parser”)
div = soup.find(“div”, class_=”product”)
# Extract text
title = div.h2.text
price = div.find(“p”, class_=”price”).text
print(f”Title: {title}”)
print(f”Price: {price}”)
# Extract attributes
link = div.find(“a”)
href = link.get(“href”)
data_id = link.get(“data-id”)
print(f”Link: {href}”)
print(f”Data ID: {data_id}”)
# get() with default if attribute missing
missing = link.get(“data-missing”, “default”)
print(f”Missing attribute: {missing}”)
Python
import requests
from bs4 import BeautifulSoup
import csv
# Example: Scraping a demo book catalog
# Note: This uses a fake URL for demonstration
url = “https://books.toscrape.com/” # Real demo site for scraping practice
def scrape_books():
headers = {
“User-Agent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36”
}
try:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f”Error fetching page: {e}”)
return []
soup = BeautifulSoup(response.text, “html.parser”)
books = []
# Find all book containers
for article in soup.find_all(“article”, class_=”product_pod”):
title_elem = article.find(“h3”).find(“a”)
title = title_elem.get(“title”, “No title”) if title_elem else “No title”
price_elem = article.find(“p”, class_=”price_color”)
price = price_elem.text if price_elem else “No price”
rating_elem = article.find(“p”, class_=”star-rating”)
rating = rating_elem.get(“class”)[1] if rating_elem else “No rating”
books.append({
“title”: title,
“price”: price,
“rating”: rating
})
return books
def save_to_csv(books, filename=”books.csv”):
if not books:
print(“No data to save”)
return
with open(filename, “w”, newline=””, encoding=”utf-8″) as f:
writer = csv.DictWriter(f, fieldnames=[“title”, “price”, “rating”])
writer.writeheader()
writer.writerows(books)
print(f”Saved {len(books)} books to {filename}”)
if __name__ == “__main__”:
books = scrape_books()
for book in books[:5]:
print(f”{book[‘title’]} – {book[‘price’]} – Rating: {book[‘rating’]}”)
save_to_csv(books)
🕯️ Magic Note
The site books.toscrape.com is specifically designed for practicing web scraping. It has no robots.txt restrictions and is safe to scrape. Always test your scrapers on such practice sites before targeting real websites.
Python
import requests
import time
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
# 1. Retry on failure
session = requests.Session()
retry = Retry(total=3, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount(“http://”, adapter)
session.mount(“https://”, adapter)
# 2. Rate limiting (be polite)
time.sleep(1) # Wait 1 second between requests
# 3. Handle different status codes
response = session.get(“https://example.com”)
if response.status_code == 200:
html = response.text
elif response.status_code == 404:
print(“Page not found”)
elif response.status_code == 429:
print(“Rate limited. Waiting…”)
time.sleep(60)
else:
print(f”HTTP {response.status_code}”)
# 4. Handle missing elements gracefully
element = soup.find(“div”, class_=”maybe-missing”)
if element:
text = element.text
else:
text = “Not found”
- Check robots.txt (e.g., https://example.com/robots.txt) and respect it
- Read the website’s terms of service
- Identify yourself with a descriptive User-Agent header
- Implement rate limiting (delay between requests)
- Scrape during off-peak hours if possible
- Cache responses to avoid repeated requests
- Do not bypass login systems or access non-public data
- Do not overload the server (use reasonable concurrency)
- Check for an official API (most ethical and reliable)
- Check for RSS feeds or structured data exports
- Look for JSON data embedded in the page (many sites include JSON in <script> tags)
- Contact the website owner about data access
- Not handling missing elements (leading to AttributeError)
- Not using rate limits (getting IP banned)
- Parsing JavaScript-rendered content with BeautifulSoup (use Selenium for dynamic sites)
- Hard-coding selectors that may change (build resilient scrapers)
- Not checking robots.txt
- Scraping too aggressively (overwhelming the server)
- What is the purpose of the requests library?
- How do you find all elements with a specific class using BeautifulSoup?
- What is the difference between find() and find_all()?
- Why should you add a delay between requests?
- What is robots.txt and why should you check it?
- How do you extract the href attribute from an <a> tag?
⚡ Whisper
The web is an ocean of data. Web scraping is your fishing rod. With requests you cast your line. With BeautifulSoup you examine what you catch. You find the .title and .price hidden among the HTML. You extract them carefully, handling the ones that are missing, ignoring the ones you do not need. But fishing has rules. Respect the robots.txt signs. Do not fish in protected waters. Do not fish too fast. Leave some for others. Identify yourself. Be polite. And remember: sometimes the website offers an API, a cleaner way to get data. Check for it before you scrape. But when there is no API, and the data is public, and the rules allow it, scraping is your tool. Use it wisely. Respect the web. It gives you data. Give it space in return.
