The Analyst's Stack
Automation and AI for Investigators
If you check the same sources by hand every week and still hear about things late, this series is for you. It shows how to make a machine do the looking, so you find out in 15 minutes instead of next Tuesday and spend your attention on what a thing means.
The idea that holds it together: every collection system I have built, and every one I have taken apart, is the same 6 stages in a row. Collect, normalize, store, analyze, alert, present. Once you can see them, an unfamiliar tool stops being a mystery. You are working out which stages it covers and which it leaves to you.
There are 22 posts, of 2 kinds.
The guide — 10 teaching posts. Five foundations lay out those stages and where AI genuinely earns a place inside them. Five methods cover the parts people get wrong: running models on your own machine, reaching sources that block you, measuring whether an AI step works, automating without writing code, and storing what you collect so you can search it.
The showcase — 12 short posts, one per working system. Why it was built, what it solves, how it fits together, and what it cannot do.
Start with part 1, even if you have never written a line of code. The hard part of this work is knowing which source is worth watching, and you are already good at that.
Not AI, not robots. Six stages that every collection system turns out to be a version of — and the 4 cases where the right answer is to do it by hand.
One small real problem, walked end to end: fetching, a stable ID, storage, notification, scheduling, and the failure mode that looks exactly like good news.
One stage out of 6, and a short list of jobs inside it. The test that sorts the narrowing work a model can take on from the verification and attribution work it can only draft — and why that draft is a lead, not a finding, until a person has checked it.
Six rules for putting a model anywhere near an investigation: assume it will be confidently wrong, keep every claim traceable to a fetched artifact, and treat a page you collected as something that might be talking to your model.
You have automated the act of looking, so you have automated the act of being seen. What a server actually learns from a monitor that runs for a year, and why the fix is usually to slow down rather than to hide better.
Poll a source every 15 minutes, hash the path for a stable ID, keep the state in SQLite, archive what you find, say one thing in a chat channel. Six stages, no platform.
A public feed with no export button and a rate limit. Walking it politely with rotating exits, collapsing 37 spellings of city names into one, and turning the result into map layers by city and by area plus category counts over time.
A ship that switches off its transponder disappears from the data you have. Satellite detection plus an AIS cross-check gives you a second look that does not depend on the ship's cooperation.
Two files, no shared identifier, and one question: which rows describe the same real-world thing? About 60 lines of Python, 2 arbitrary decisions, and one bug that quietly inflated the answer.
The same 6 stages as a one-file watcher, except every one of them has grown into a component: continuous collection, cross-posting detection, machine translation, an AI pass, a map and a dashboard.
Testing whether a point falls inside an area is one line of code. Deciding whether an object was actually there takes the whole sequence of observations, and a threshold you have to pick yourself.
Some sources have no export button and no intention of building one. Record what the source draws as it draws it, give every object a stable key, and you own a dataset the source never published.
Six megabytes of exported messages, labelled and filtered by a few dozen keyword rules. No model, no API bill, nothing leaving the laptop — and every label traceable to the line that produced it.
Arabic, English and Hebrew, collected and enriched by a worker with a queue per stage. The same 6 stages as everything else in this series, except a shift depends on them being up.
A corpus of 340,920 observations from 18 catalogues, collapsed into 120,063 canonical events across 90,565 victims. Deciding which records describe the same incident is the whole product — and a 3-year rebuild of a 2023 original.
Running 7 scanners is a thread pool and an afternoon. Making them agree on what a finding is — one shape, one severity scale, one row per problem — is the whole product.
A folder of documents becomes rows, the rows are processed in batches rather than one at a time, and nothing leaves until a person has marked it reviewed. The gate is the product.