The 11 fields and concepts you'll meet in the response — defined in plain English, each with a real example value.
The process of analyzing HTML document structure to extract specific data elements.
Parsers build a DOM (Document Object Model) tree from HTML, then use selectors (CSS, XPath) to navigate and extract data. Libraries like BeautifulSoup (Python), Cheerio (Node.js), and native browser APIs provide parsing capabilities.
ExampleParse HTML, find all <a> tags, extract href attributes
Patterns used to select HTML elements based on their tag, class, ID, attributes, or hierarchy.
CSS selectors are used both for styling and data extraction. Common selectors: tag (a, div), class (.link), ID (#header), attribute ([href]), descendants (div a). Selectors enable precise targeting of data within HTML structure.
Examplea.external-link selects all <a> tags with class="external-link"
A web browser without a graphical interface, used for automated testing and scraping JavaScript-rendered sites.
Headless browsers (Puppeteer, Playwright, Selenium) execute JavaScript and render pages like regular browsers but run in the background. They handle dynamic content, form submissions, and user interactions programmatically. Essential for scraping modern SPAs.
ExamplePuppeteer renders React app, waits for content, extracts data
Document Object Model—a tree structure representation of HTML that allows programmatic access to page elements.
Browsers parse HTML into a DOM tree. JavaScript and scraping tools navigate the DOM to find, read, and manipulate elements. DOM selectors (getElementById, querySelector) enable precise data extraction from HTML structure.
Exampledocument.querySelector("a.link") finds first link with class="link"
A protocol that enables websites to control how their content appears when shared on social media.
Open Graph tags (og:title, og:description, og:image, og:url) are meta tags in the HTML <head>. Facebook created the protocol, but most platforms (LinkedIn, Slack, Discord) support it. Proper Open Graph tags ensure attractive link previews.
Example<meta property="og:title" content="Page Title" />
HTML elements in the <head> section that provide metadata about the page for browsers, search engines, and social media.
Meta tags include title, description, keywords (deprecated), viewport (for responsive design), robots (indexing directives), and Open Graph/Twitter Cards. Search engines use meta description for snippets. Social platforms use og: tags for link previews.
Example<meta name="description" content="Page summary for search results" />
Meta tags that control how URLs appear when shared on Twitter.
Twitter Cards (twitter:card, twitter:title, twitter:description, twitter:image) customize link previews on Twitter. Types include summary, summary_large_image, and player. Twitter falls back to Open Graph tags if Twitter Cards aren't specified.
Example<meta name="twitter:card" content="summary_large_image" />
Hyperlinks that point to other pages within the same website domain.
Internal links help users navigate, distribute PageRank across your site, and establish site hierarchy. Orphan pages (no internal links pointing to them) are hard for users and search engines to find. Good internal linking improves SEO and user experience.
Example<a href="/about">About Us</a> (links within same domain)
Hyperlinks that point to pages on different websites.
External links show topic relationships and can pass SEO authority to the linked site. Linking to authoritative sources can boost your credibility. Too many external links may dilute your authority. Use rel="nofollow" for untrusted or paid links.
Example<a href="https://example.com">External Site</a>
A link attribute (rel="nofollow") that tells search engines not to pass authority or "link juice" through the link.
Use nofollow for: user-generated content, paid/sponsored links, untrusted sites, or when you don't want to vouch for the destination. Google introduced "sponsored" and "ugc" as more specific alternatives, but nofollow still works.
Example<a href="https://example.com" rel="nofollow">Link</a>
A file at the root of a website that tells web crawlers which pages they can or cannot access.
Robots.txt uses directives like "Disallow: /admin/" to block crawler access. Crawlers check robots.txt before scraping. It's advisory, not enforced—malicious bots may ignore it. Blocking important pages in robots.txt prevents search engine indexing.
ExampleUser-agent: * / Disallow: /private/
