Company registry scraper

Overview

A government company registry holding roughly 150,000 records, wanted as a local database. No listing endpoint, no export, and a search that had to be exhausted by querying it repeatedly.

This one is here because it did not work.

What happened

The search half was straightforward. Sessions, two-factor login, a keyword strategy that pulled from a seed list and kept going until queries stopped producing new records, and a schema that generated identifiers and timestamps on insert.

The results table was the problem. It rendered twenty-five rows at a time and unmounted everything outside the viewport, a technique variously called windowing or virtual scrolling, hardened further against automation. Rows that were not on screen did not exist in the page to be read. I found and drove the internal counters far enough to prove the approach, and could not get it to hold at scale.

After several days I told the client I could not solve it, offered the work at a reduced price rather than the agreed milestone, and suggested they take that specific piece to someone else. The handover included the working scraper, the keyword file, the requirements, the setup notes, and a separate file containing every scrolling approach I had tried and why each failed, so the next developer would not spend the same days rediscovering it.

The client accepted, paid the reduced amount, and closed the contract. Their review reads: “reliable and transparent about the challenges the task involved.”

Why it is published

A portfolio of only successes is a portfolio with a survivorship problem, and anyone who has done this work knows that scraping runs into walls. What is worth judging is not whether a job was ever hard, it is what happened when it was.

I did not quietly burn the budget, deliver something that half worked and call it done, or hand back a script with the broken part unmarked. I said which part was unsolved, priced myself down for it, and left a written record of the dead ends so the money the client spent next would go further.

One thing I would do differently. I priced this before opening the site properly. The quote had to move once I understood the volume, and the actual blocker was not visible at all until I was inside the search results. On anything involving a page I have not driven myself, the estimate should follow the reconnaissance rather than lead it.

Stack

Python with Selenium against an authenticated session, SQLite for the output schema, a keyword-exhaustion strategy over the search.