Url Authenticator

Overview

Reads a list of URLs from a spreadsheet, checks whether each one is actually reachable, groups them by image type, and writes the good ones out per group with everything broken or duplicated sent to a separate file.

The list ran to tens of thousands of links, which is what turns a one-line check into a real job.

Complexities tackled

At this volume, sequential checking is not an option. Requests have to run concurrently, with timeouts that fail fast rather than hanging the run on one dead host, and with pacing that does not look like an attack to the servers being checked.

Broken is not one condition. A URL can be a redirect, a timeout, a wrong content type, a soft 404 that returns a valid page saying nothing is here, or a duplicate of one already checked. Collapsing all of those into a boolean throws away exactly the information the client needs to act.

Separating the failures is the deliverable. The valid links go where they are useful; the failures go somewhere they can be reviewed. A single output file with a status column reads as one problem rather than two lists with two different next actions.

Stack

Python with Requests for the checking, Pandas for the grouping and output, spreadsheet input.