Linked Company Scraper

Overview

Logs in, runs a search, and collects the full details of every company in the results one by one, writing to CSV as it goes. It ran to a couple of hundred thousand companies.

The contract asked for the About, People and Insights data across an initial list of a thousand company pages, with more to follow if it worked.

Complexities tackled

Resume is not a feature, it is the requirement. A run of that size will be interrupted, by a session expiring, a rate limit or a machine restarting. Without the ability to pick up where it stopped, every interruption costs the whole run and the job is impossible rather than slow. Writing progressively and tracking what has already been done is what makes the scale reachable at all.

Three tabs, three page shapes. About, People and Insights are different structures with different fields and different loading behaviour, so each needs its own extraction rather than one selector set applied three times.

Pacing against a platform that does not want this. The constraint is not how fast the code can go, it is how fast it can go without the session being stopped. That makes the deliberate waiting part of the design rather than an inefficiency to remove.

Stack

Python with Selenium, progressive CSV output with resume support.