These sites cannot be scraped with requests alone. The HTML you download is empty. The data you want is added later by JavaScript.
Selenium is the solution. It automates real web browsers (Chrome, Firefox, Edge). It loads pages fully, executes JavaScript, waits for content, and lets you interact with pages as a real user would. You can click buttons, fill forms, scroll, hover, and extract data after dynamic content has loaded.
This lesson covers Selenium WebDriver, locating elements, waiting for content, handling alerts and popups, and scraping dynamic websites. Selenium is your tool for the modern, interactive web.
🕯️ Magic Note
Selenium is not a parser like BeautifulSoup. It is a browser automation tool. It launches a real browser window (visible or headless), navigates to pages, and lets you control the browser programmatically. This makes it slower than requests, but it can handle anything a human can do in a browser.
Bash
# Install Selenium
pip install selenium
# For Chrome: Download chromedriver from https://chromedriver.chromium.org/
# Or use webdriver-manager (auto-manages drivers)
pip install webdriver-manager
Python
# Using webdriver-manager (recommended – no manual driver setup)
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
# Automatically downloads and sets up ChromeDriver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
# For Firefox
from webdriver_manager.firefox import GeckoDriverManager
driver = webdriver.Firefox(service=Service(GeckoDriverManager().install()))
Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
# Setup driver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
try:
# Navigate to a page
driver.get(“https://example.com”)
# Get page title
print(f”Title: {driver.title}”)
# Find elements
heading = driver.find_element(By.TAG_NAME, “h1”)
print(f”Heading: {heading.text}”)
# Get page source (for BeautifulSoup if needed)
page_source = driver.page_source
finally:
# Always close the browser
driver.quit()
| Method | Description | Example |
|---|---|---|
| By.ID | Find by ID attribute | driver.find_element(By.ID, “main”) |
| By.CLASS_NAME | Find by class name | driver.find_element(By.CLASS_NAME, “product”) |
| By.TAG_NAME | Find by tag name | driver.find_element(By.TAG_NAME, “h1”) |
| By.NAME | Find by name attribute | driver.find_element(By.NAME, “email”) |
| By.CSS_SELECTOR | CSS selector | driver.find_element(By.CSS_SELECTOR, “#main .product”) |
| By.XPATH | XPath expression | driver.find_element(By.XPATH, “//div[@class=’product’]”) |
| By.LINK_TEXT | Exact link text | driver.find_element(By.LINK_TEXT, “Click Here”) |
| By.PARTIAL_LINK_TEXT | Partial link text | driver.find_element(By.PARTIAL_LINK_TEXT, “Click”) |
Python
from selenium import webdriver
from selenium.webdriver.common.by import By
# Assuming driver is already initialized
# Find by ID
header = driver.find_element(By.ID, “header”)
# Find by class name (note: returns first element with that class)
product = driver.find_element(By.CLASS_NAME, “product-item”)
# Find by CSS selector
price = driver.find_element(By.CSS_SELECTOR, “.product .price”)
# Find multiple elements
all_products = driver.find_elements(By.CLASS_NAME, “product-item”)
print(f”Found {len(all_products)} products”)
# Find by XPath (powerful but complex)
element = driver.find_element(By.XPATH, “//div[@data-id=’123′]”)
🕯️ Magic Note
CSS selectors are generally preferred over XPath for their readability. However, XPath can be more powerful for complex navigation (like finding elements by text content or parent-child relationships).
Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Implicit wait (global, applies to all find_element calls)
driver.implicitly_wait(10) # Wait up to 10 seconds for elements to appear
# Explicit wait (specific condition for a specific element)
wait = WebDriverWait(driver, 10)
# Wait for element to be present
element = wait.until(EC.presence_of_element_located((By.ID, “dynamic-content”)))
# Wait for element to be clickable
button = wait.until(EC.element_to_be_clickable((By.ID, “submit-btn”)))
# Wait for text to be present in element
wait.until(EC.text_to_be_present_in_element((By.ID, “status”), “Complete”))
# Common expected conditions
# EC.presence_of_element_located – element is in DOM
# EC.visibility_of_element_located – element is visible
# EC.element_to_be_clickable – element is visible and enabled
# EC.invisibility_of_element_located – element disappears
# EC.alert_is_present – alert dialog appears
Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import time
# Fill a text input
search_box = driver.find_element(By.NAME, “q”)
search_box.clear() # Clear existing text
search_box.send_keys(“Python web scraping”)
# Press Enter key
search_box.send_keys(Keys.RETURN)
# Click a button or link
submit_button = driver.find_element(By.ID, “submit”)
submit_button.click()
# Click a link by text
next_link = driver.find_element(By.LINK_TEXT, “Next Page”)
next_link.click()
# Select from dropdown
from selenium.webdriver.support.ui import Select
dropdown = Select(driver.find_element(By.ID, “options”))
dropdown.select_by_visible_text(“Option 2”)
dropdown.select_by_value(“option2”)
dropdown.select_by_index(1)
# Scroll to element
element = driver.find_element(By.ID, “footer”)
driver.execute_script(“arguments[0].scrollIntoView();”, element)
# Scroll to bottom of page
driver.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
# Scroll by pixels
driver.execute_script(“window.scrollBy(0, 500);”)
# Hover over an element (ActionChains)
menu = driver.find_element(By.ID, “menu”)
hover = ActionChains(driver).move_to_element(menu)
hover.perform()
Python
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Wait for alert to appear
wait = WebDriverWait(driver, 5)
alert = wait.until(EC.alert_is_present())
# Get alert text
alert_text = alert.text
print(f”Alert says: {alert_text}”)
# Accept (click OK)
alert.accept()
# Dismiss (click Cancel)
alert.dismiss()
# For prompt with input
alert.send_keys(“User input”)
alert.accept()
Python
# Get current window handle
main_window = driver.current_window_handle
# Open a link in a new tab (simulate Ctrl+Click)
link = driver.find_element(By.LINK_TEXT, “Open New Tab”)
link.send_keys(Keys.CONTROL + Keys.RETURN)
# Get all window handles
all_windows = driver.window_handles
# Switch to new window
for handle in all_windows:
if handle != main_window:
driver.switch_to.window(handle)
break
# Now work with the new window
print(f”New window title: {driver.title}”)
# Close current window and switch back
driver.close()
driver.switch_to.window(main_window)
Python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Configure Chrome options
options = Options()
options.add_argument(“–headless”) # Run in headless mode
options.add_argument(“–no-sandbox”)
options.add_argument(“–disable-dev-shm-usage”)
options.add_argument(“–window-size=1920,1080”)
# Create headless driver
driver = webdriver.Chrome(options=options)
driver.get(“https://example.com”)
print(f”Title: {driver.title}”) # Works without visible browser
driver.quit()
🕯️ Magic Note
Headless mode is essential for running scrapers on servers without a graphical interface (like cloud VMs). It also makes scraping faster because rendering is skipped.
Python
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def scroll_infinite_page(url, scroll_pause=2, max_scrolls=20):
driver = webdriver.Chrome()
driver.get(url)
items = []
last_height = driver.execute_script(“return document.body.scrollHeight”)
scroll_count = 0
while scroll_count < max_scrolls:
# Scroll to bottom
driver.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
# Wait for new content to load
time.sleep(scroll_pause)
# Check if we have reached the end
new_height = driver.execute_script(“return document.body.scrollHeight”)
if new_height == last_height:
print(“Reached end of page”)
break
last_height = new_height
scroll_count += 1
print(f”Scrolled {scroll_count} times”)
# Extract data after all scrolling
item_elements = driver.find_elements(By.CSS_SELECTOR, “.item-class”)
for elem in item_elements:
items.append({
“title”: elem.find_element(By.CSS_SELECTOR, “.title”).text,
“price”: elem.find_element(By.CSS_SELECTOR, “.price”).text
})
driver.quit()
return items
Python
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def login_to_site(username, password):
driver = webdriver.Chrome()
try:
# Navigate to login page
driver.get(“https://example.com/login”)
# Wait for username field
username_field = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, “username”))
)
# Fill form
username_field.send_keys(username)
driver.find_element(By.ID, “password”).send_keys(password)
# Click login button
driver.find_element(By.ID, “login-btn”).click()
# Wait for login to complete (e.g., wait for logout link)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.LINK_TEXT, “Logout”))
)
print(“Login successful!”)
# Now we can scrape protected data
# …
return driver # Return driver for further scraping
except Exception as e:
print(f”Login failed: {e}”)
driver.quit()
return None
Python
from selenium import webdriver
from bs4 import BeautifulSoup
# Use Selenium to load dynamic content
driver = webdriver.Chrome()
driver.get(“https://example.com”)
# Wait for dynamic content to load
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, “dynamic-content”))
)
# Get the fully rendered HTML
html = driver.page_source
# Parse with BeautifulSoup
soup = BeautifulSoup(html, “html.parser”)
# Extract data with BeautifulSoup (easier syntax)
items = soup.find_all(“div”, class_=”item”)
for item in items:
print(item.text)
driver.quit()
🕯️ Magic Note
This combination is powerful: Selenium handles JavaScript and dynamic loading, BeautifulSoup handles parsing and data extraction. Best of both worlds.
- Forgetting to wait for elements (causes NoSuchElementException)
- Using time.sleep() instead of explicit waits (inefficient and unreliable)
- Not closing the driver (processes stay open)
- Running visible browser on servers (use headless mode)
- Hard-coding driver paths (use webdriver-manager)
- Scraping too fast (getting blocked; add delays between actions)
- When should you use Selenium instead of requests + BeautifulSoup?
- What is the difference between implicit and explicit waits?
- How do you run Selenium in headless mode?
- Write code to fill a search form and click the search button.
- How do you scroll to the bottom of an infinite scroll page?
- What is webdriver-manager and why is it useful?
⚡ Whisper
The modern web is alive. JavaScript moves elements, loads content, responds to clicks. Static scrapers see only the skeleton. Selenium sees the living page. It opens a real browser, waits for content, clicks buttons, fills forms, scrolls to infinity. It is slower. It is heavier. But it works where other tools fail. Use it when you need it. For static pages, stick with requests and BeautifulSoup—they are faster and lighter. For dynamic pages, infinite scroll, login-required content, or JavaScript-rendered data, reach for Selenium. It is your key to the interactive web. Remember to wait. Remember to close the browser. Remember to use headless mode on servers. And when you combine Selenium with BeautifulSoup, you have the ultimate scraping toolkit. The web is dynamic. Your scrapers can be too.
