Beatport Scraper

Overview

Collects from a music catalogue that requires a login, working through charts, playlists, releases and hold bins according to a sources file, then formats what it finds into post templates and publishes to several WordPress sites.

This contract ended on the same day the long-running platform engagement with the same client began. The relationship did not end with it: months later I was still on calls with him, and that is the part of this job I would point at rather than the scraper.

What I own

All of it, collection through publishing.

Complexities tackled

A login changes the whole shape of a scraper. Session state has to be established, kept alive across a long run and re-established when it lapses, and every request has to carry it. That is the difference between a script that works once while you watch it and one that runs unattended.

The sources file is the product. Charts, playlists, releases and hold bins are different page shapes with different fields, and the client needed to add and remove sources without a developer. Driving the whole run from a configuration file rather than from code is what made it theirs rather than mine.

It is not one store. The collector reaches a range of specialist music retailers, each with its own page structure, its own naming and its own idea of what a release record contains. A separate module per source behind one shared extraction contract is what keeps that tractable: adding a store is writing one module, not editing a large one.

The same release is described differently by every source. Titles carry remix suffixes, label names, feature credits and punctuation that no two stores agree on, and dates arrive in whatever format the page happened to use. Matching them needs fuzzy string comparison and tolerant date parsing rather than equality checks, because exact matching on this data silently produces duplicates of records that are the same thing.

Two browser strategies, because one is not enough. A conventional driver handles most of it, with a second, lower level browser automation approach available for the sources where the first is detected and blocked. Keeping both behind the same interface means the choice is a per-source setting rather than a rewrite.

The output has to be publishable, not just correct. Posts are generated with titles, meta descriptions and heading variants rather than dumped as fields, so what lands is a finished post. Four separate template generators cover four different publication styles from one collection run.

Collection and publishing are separate concerns. Scraping produces records; publishing turns records into formatted posts across multiple destinations, each with its own credentials. Keeping those two halves apart means a publishing failure does not cost the collection, and a template change does not risk the scraper.

It had to run on his machine, not mine. Months after the contract closed I was on a call walking the client through installing Python and a version manager on his own laptop, so he could run the collector himself rather than wait for me to run it. That is a different standard from “it works on my machine”, and it is the standard that decides whether a tool actually gets used after the developer leaves. The configuration-driven design above is what made that possible; the call is what made it happen.

The work was done on shared screens, not in a ticket queue. From the first evening the pattern was a call, a shared screen and a decision, which is faster than written specification for a job where the client knows the domain and the developer knows the code. It leaves almost no paper trail, which is a real cost, and it was still the right way to run this one.

Stack

Python for the whole pipeline. Authenticated collection through browser automation with a second low level driver as a fallback, HTML parsing, fuzzy string matching and tolerant date parsing for reconciliation, configuration-driven sources, and WordPress publishing across multiple accounts with per-brand templates and generated SEO metadata.