Web Scraping with Beautiful Soup

Python
from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three sisters:
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>.</p>
"""

soup = BeautifulSoup(html_doc, "lxml")  # prefer lxml for speed

# Navigate
soup.title.string          # "The Dormouse's story"
soup.p["class"]             # ['title']

# Search
soup.find_all("a")                      # all <a> tags
soup.find(id="link3")                   # single tag by attribute
[a["href"] for a in soup.find_all("a")]  # extract all URLs

# Extract text
soup.get_text()

Progress checklist for a typical scraping task:

• Choose a parser (lxml for speed/leniency, html.parser for no dependencies, html5lib for browser-accurate parsing of broken HTML)
• Load markup into BeautifulSoup(markup, parser_name)
• Inspect structure with soup.prettify()
• Navigate (.parent, .children, .next_sibling, dot-access like soup.div.p) or search (find, find_all, CSS select)
• Extract data (.string, .get_text(), tag['attr'], tag.get('attr'))
• Modify if needed (.replace_with(), .decompose(), .unwrap(), attribute assignment)
• Serialize back with str(soup) or soup.prettify() if output is required

1. Pick the right parser

Always pass the parser name explicitly — never rely on autodetection in production code, since different installs may have different parsers available, producing inconsistent trees.

2. Navigating vs. Searching

• Use dot notation (soup.title, soup.body.p) only for quick, unambiguous access to the first matching tag.
• Use find/find_all for anything conditional: by tag name, id, class_, attrs dict, string content, or a function predicate.
• Use CSS selectors (soup.select("p.story a"), soup.select_one(...)) when the target is easier to express as a selector than a filter.

3. Extracting data safely

Python
tag.get("href")        # None if missing — safe
tag["href"]             # KeyError if missing — use only when attribute is guaranteed
tag.get_text(strip=True)  # text without leading/trailing whitespace

4. Modifying the tree

Python
tag.string.replace_with("new text")
tag.decompose()          # remove tag and its contents entirely
tag.unwrap()              # remove tag but keep its contents
tag["class"] = ["new-class"]
new_tag = soup.new_tag("a", href="http://example.com")
tag.append(new_tag)

Example 1 — Extract all links: Input: HTML page with multiple <a> tags. Output:

Python
links = [a.get("href") for a in soup.find_all("a") if a.get("href")]

Example 2 — Extract table rows into dicts: Input: <table> with header row and data rows. Output:

Python
table = soup.find("table")
headers = [th.get_text(strip=True) for th in table.find("tr").find_all("th")]
rows = []
for tr in table.find_all("tr")[1:]:
    cells = [td.get_text(strip=True) for td in tr.find_all("td")]
    rows.append(dict(zip(headers, cells)))

Example 3 — Remove scripts/styles before text extraction: Input: page with <script> and <style> tags polluting get_text(). Output:

Python
for tag in soup(["script", "style"]):
    tag.decompose()
clean_text = soup.get_text(separator=" ", strip=True)

Example 4 — Find by multiple attributes: Input: need <div class="product" data-id="42">. Output:

Python
soup.find("div", class_="product", attrs={"data-id": "42"})