Working draft. Every code snippet here is illustrative, a generic version of the pattern rather than the production module. No employer I scrape is named anywhere in this post. The only services I name are hiring platforms that publish their own public job-board APIs. You will also see in the preview image that some failed. This is normal, expired certificates occur sporadically and tend to be self resolving.
The problem
Applying to engineering jobs in Chicagoland is a paperwork loop disguised as a search. The postings are public, but they sit across a few hundred company career pages, each one paginated, slow, and structured just differently enough that you cannot hold them all in your head. What I wanted was a single list: every relevant engineering and data role within commuting distance of Chicago, updated nightly, sortable and filterable without opening twenty tabs.
A career page is fine if you want to check one company today. It is less fine if you want to ask what is new this week across 150 employers, narrow it to titles that fit a chemical engineering plus CS background, and avoid re-reading the same posting you already dismissed on Tuesday.
So I built a nightly scraper and a static dashboard. The scraping itself turned out to be the easy part. What surprised me was how little of the work actually depends on the individual company.
The naive version, and why it falls over
The obvious design is one scraper per employer. You open a company’s careers page, find the listings in the HTML, write the selectors, and move on to the next company. That is 150 companies and 150 little scrapers.
I wrote a handful of these before the pattern started to annoy me. Each one feels locally reasonable, since it is only a few CSS selectors. The total is 150 brittle files that break in different ways on different weeks. A careers page redesign on Monday becomes a silent failure on Tuesday, and at 150 sites something is always redesigning. The maintenance cost of that is the whole project.
What finally fixed it had nothing to do with writing the scrapers faster. I was building the wrong number of scrapers to begin with.
The real shape: one adapter per hiring platform
Almost nobody builds their own careers page anymore. They rent one. Underneath the company logo and the custom font, the listings come from one of a small set of hosted applicant tracking systems. The branding changes from company to company, but the plumbing underneath is shared.
That plumbing is the part that matters, because most of these platforms render their listings from a JSON endpoint that the page calls in your browser. The careers page you see is a thin client over an API. Open the network tab on a typical hosted board and the job data arrives as structured JSON before any of it becomes HTML, so “scraping” it is often just reading the same response the page already reads, with no HTML parsing involved. Several of these platforms even publish that endpoint as a documented public job-board API on purpose, because employers want their openings syndicated. Greenhouse, Lever, Ashby, SmartRecruiters, and other similar platforms do exactly that, serving their public postings as JSON with no authentication required.
That reframes the architecture. Instead of one scraper per company, you write one adapter per platform, plus a small config file for each company that names the platform it uses.
scraping/
adapters/ # one module per ATS platform; the real logic lives here
companies/ # one tiny config per employer: which adapter, which board id
utils.py # shared HTTP, title matching, normalization
runner.py # orchestrates: for each company, dispatch to its adapterA platform adapter has one job. Given a company’s identifier on that platform, it returns a list of normalized postings. Generically it looks like this:
def fetch(board_id: str) -> list[Posting]:
data = get_json(PLATFORM_ENDPOINT.format(board_id=board_id))
return [
Posting(
title=row["title"],
location=row["location"],
url=row["absolute_url"],
posted_at=parse_date(row.get("updated_at")),
)
for row in data["jobs"]
]With the adapters in place, adding a company is a few lines of config:
{
"name": "A Large Industrial Manufacturer",
"platform": "greenhouse",
"board_id": "redacted",
"enabled": true
}Twenty adapters ended up covering well over a hundred employers. That ratio, a handful of platforms behind a large pile of companies, is the reason one person can maintain this at all. When a platform changes its response shape, I fix one adapter and every company sitting on that platform comes back in the same commit. I stopped writing a scraper for every logo, and that turned out to be the whole trick.
When there is no JSON: the browser fallback
Not every site is cooperative. A few render their listings server-side, or bury them under enough client-side machinery that there is no clean endpoint to call. For those I keep a Playwright fallback: a real headless browser that loads the page, lets its scripts run, and reads the result off the rendered DOM.
A headless browser is slower, heavier, and more fragile than a plain HTTP request, so it stays a last resort, reserved for the sites that give me no other way in. Out of the whole roster, only a handful need it.
The bug that taught me to distrust “it didn’t crash”
The failure mode that cost me the most went like this. A scraper runs, raises no exception, returns zero jobs, and the pipeline concludes the company has no openings and retires every posting it had stored.
A site redesign, a renamed field, or a rate-limit page returned with a 200 status all produce a clean run with empty results. If “did not crash” is your definition of success, you will quietly delete real data the first time a site hiccups.
So every company gets a per-run health flag, and an empty result does not count as success:
pass: the source returned at least one raw posting.needs_review: the run completed without error but returned zero postings. Existing jobs are left in place rather than retired.failed: the run raised an exception. Existing jobs are left in place here too.
Retiring a posting now requires positive evidence that the source is alive. An empty scrape gets investigated before anything downstream may act on it, and that single change removed nearly every case of the dashboard losing jobs it should have kept.
The rest of the pipeline
The scraper writes normalized postings into SQLite. A small set of scripts turns that into the dashboard: archive postings that have been inactive long enough, export the active set to JSON, and rank that JSON against a search profile. Title filtering starts at scrape time with wildcard patterns like *engineer* and *scientist*, and the ranking pass then scores what survives with weighted title terms plus location rules covering the city, the suburbs, the state, and remote or hybrid roles. A TypeScript frontend reads the JSON and renders a static page. A scheduled task on a machine I own runs the whole chain nightly and pushes the result to a static host. There is no server to keep alive, since the dashboard is just files, and the deployed page sits behind an access gate. Its entire user base is me.
I am saving the resume and cover letter tailoring half of this project for a follow-up. That piece takes a posting the scraper found and drafts application materials for it, which opens a messier set of questions about honesty and authorship, so it gets its own post.
The part everyone skips: is this even okay?
A personal scraping project lives in a legal gray area, and I would rather address that directly than hope nobody asks. I am an engineer, not a lawyer, and nothing here is legal advice. I did do the reading, though, and the landscape is clearer than the folklore around scraping suggests.
The headline fear is the Computer Fraud and Abuse Act, the federal “hacking” statute. After Van Buren v. United States (Supreme Court, 2021) [1] and hiQ Labs v. LinkedIn (Ninth Circuit, 2022) [2], the governing reading is that collecting publicly available data, the kind served to anyone with a browser and no login, does not count as “unauthorized access” under the CFAA. If the data is handed to everyone who asks, there is no authorization gate left to exceed.
The detail that actually matters is how hiQ ended. After winning the CFAA point, hiQ still lost the war and settled with a judgment against it on breach of contract and common-law torts, because it had used fake accounts to reach data sitting behind LinkedIn’s login [3]. That is the line worth remembering. Public data that anyone can load without signing in stays on the safe side of it. The moment you create an account, click through a login, or work around an access control to reach something that is not public, you are in the territory that sank hiQ, which is contract and trespass law, and those do not care what the CFAA says.
A few more things I keep in view:
- Terms of service are a contract. Violating a site’s ToS is a contract question, not a hacking one. The realistic exposure for a personal, public-data project is small, but it is not zero.
- Don’t impair the service. The common-law theory that does reach scrapers is trespass to chattels, which turns on whether you degraded the system you accessed. Hammering a server is how a defensible read of public data turns into a real grievance. Rate limiting is partly courtesy and partly self-protection, since a slow, light scraper never creates the impairment this tort is about.
- Facts are not copyrightable [5]. A job title, a location, and a link are facts. Storing those and linking back to the source is a long way from republishing someone’s full posting copy as my own. I keep the facts and point at the original.
I am also in Illinois, in the Seventh Circuit rather than the Ninth, so the hiQ reasoning is persuasive here without being binding. Illinois has its own computer statute, the Computer Tampering law (720 ILCS 5/17-51) [4], which turns on access “without authorization.” The same public versus private distinction decides it. Accessing a page that the operator serves to the general public, without defeating any access control, is not the unauthorized access these statutes were written to punish. I would not lean on any of this the moment a login or a paywall enters the picture.
None of this is a guarantee, and a blog post does not settle it. What the reading gives me is a clear sense of where the bright lines are, and the engineering follows from them:
- Public endpoints only. No login, no account, no access control defeated, ever. This is the single most important rule, and it is exactly what hiQ got wrong.
- Prefer the documented API. When a platform publishes a public job-board endpoint, I use it. A surprising amount of “scraping” here is just reading a feed the vendor built to be read.
- Respect
robots.txtand rate limit hard. Low concurrency, real delays, honest retries with backoff. The aim is to be indistinguishable from light, ordinary traffic, because that is what it is. - Identify honestly and stay small. A real user agent, a modest request volume, and no attempt to look like something I am not. Anything that amounts to evading detection points the wrong way, so I leave it alone.
- Store facts, link back, keep it personal. This is one person’s job search rather than a commercial data product, and the footprint reflects that.
The short version: scraping public data is broadly defensible, the danger lives almost entirely in logins and load, and most of the legal safety comes from the same discipline that makes the scraper a polite guest on someone else’s server.
What I took away
The most useful lesson is about counting. The hard problem looked like writing 150 scrapers, and it only became tractable once I saw that the real count was twenty. The variation I had been fighting was branding, and the structure I needed was sitting underneath it, shared across companies. The second lesson is that the health-flag design and the rate-limiting discipline were never separate from the legal reasoning. Being careful with other people’s servers and staying on the right side of the law turned out to require the same code.
References
- Van Buren v. United States, 593 U.S. 374 (2021). https://supreme.justia.com/cases/federal/us/593/19-783/
- hiQ Labs, Inc. v. LinkedIn Corp., 31 F.4th 1180 (9th Cir. 2022). Plain-language summary of the narrow CFAA reading: Jenner & Block client alert. https://www.jenner.com/en/news-insights/publications/client-alert-data-scraping-in-hiq-v-linkedin-the-ninth-circuit-reaffirms-narrow-interpretation-of-cfaa
- Morgan Lewis, “LinkedIn v. hiQ: Landmark Data Scraping Suit Provides Guidance,” on the December 2022 settlement and the contract and tort liability. https://www.morganlewis.com/blogs/sourcingatmorganlewis/2022/12/linkedin-v-hiq-landmark-data-scraping-suit-provides-guidance-to-data-scrapers-and-web-operators
- Illinois Computer Tampering statute, 720 ILCS 5/17-51. http://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=072000050K17-51
- Feist Publications, Inc. v. Rural Telephone Service Co., 499 U.S. 340 (1991). https://supreme.justia.com/cases/federal/us/499/340/
Comments