Seven government recall feeds, and what it takes to make them agree
Every developed market publishes its product recalls. The EU has Safety Gate, France runs RappelConso on top of it, the US splits the job across the CPSC, NHTSA and three separate FDA enforcement databases. All of it is free, all of it is public, and most of it has an API.
So checking whether a product has been recalled should be easy.
It is not, and the reasons are more interesting than "the schemas differ." Here is what came out of normalising 176,000 recalls from seven of these sources into a single queryable corpus.
The schemas differ, and that is the easy part
Safety Gate publishes XML per weekly report. RappelConso is an OpenDataSoft instance with a JSON API. NHTSA ships a ZIP containing a pipe-delimited flat file. openFDA gives you clean JSON with proper pagination. The CPSC has a REST API that mostly behaves.
Mapping five shapes onto one schema is a week of tedious work with a clear finish line. You write a normaliser per source, you write tests, you are done. Nobody's project fails here.
Where it actually gets hard is that the sources disagree about what a recall is.
An openFDA enforcement report is one row per *product* in a recall event: a single event with forty affected lot numbers is forty rows sharing an event_id. A Safety Gate notification is one row per *notification*, which may cover several products from one manufacturer. NHTSA is one row per *campaign*, which may cover multiple vehicle models across several years.
Deduplicate naively across sources and you undercount, because the same physical hazard legitimately appears as three different record types. Do not deduplicate at all and a customer searching a barcode gets forty near-identical hits for one event. Neither is what anybody wants.
We keep records at source granularity and reconcile at the *identifier* level instead. If a GTIN appears in an openFDA row and a Safety Gate notification, the lookup returns both, and it is obvious from the payload that they are two agencies describing one hazard rather than two hazards. That decision is unglamorous and it is the single most consequential one in the project.
Identifiers live in prose, not in fields
The thing you actually want to search by — a barcode, a VIN, a model number — is frequently not a field.
Here is a real openFDA code_info value, lightly trimmed:
Lot #1234567, Exp 03/2027; UPC 0 36000 29145 2; also lots 1234568,
1234569 (UPC 036000291452) distributed to CA, NV, AZThat contains one GTIN written two ways and three lot codes, inside a free-text field with no consistent delimiter. The FDA is not being careless. This is a field designed for a human reading a recall notice, and it does that job well.
So identifiers have to be mined out of prose, which means a regex pass, and a regex pass over free text means false positives. 0 36000 29145 2 is a GTIN. So is a 12-digit order reference that happens to sit next to the word "UPC." The distinguishing feature is the check digit: a real GTIN's last digit is a mod-10 checksum over the others, and a random 12-digit number passes that test one time in ten.
Validating the check digit takes about fifteen lines and throws out roughly a fifth of the candidates a naive regex finds. Skip it and your API confidently reports recalls against identifiers that were never barcodes.
The corollary matters more: an invalid identifier must be rejected, never answered. If someone queries a malformed GTIN and you return "no recalls found," you have told them their product is clear when in fact you did not look it up. For a compliance tool that is the worst possible failure: it is wrong in the direction the customer will not check. We return a 400.
The pagination caps nobody documents prominently
openFDA's enforcement endpoints paginate with limit and skip. The limit maxes at 1,000, which is generous.
skip refuses anything above 25,000.
skip=25000 -> HTTP 200
skip=25001 -> HTTP 400 "Skip value must 25000 or less."The missing word is theirs. I have grown attached to it.
The device enforcement dataset holds 39,538 records. Sort by report date, page through, and you hit a wall 14,538 records short of the end, with a 200 OK on the last successful page and nothing anywhere telling you the history is truncated. Your corpus is quietly incomplete and every downstream number is confidently wrong.
I assumed an API key would raise the cap. It does not. A key raises the daily *request* quota from 1,000 to 120,000 and leaves skip exactly where it was. Registering one would not have recovered a single record.
The route past it is date windows. Constrain the search to one year and no window comes close to the cap. The densest year measured is 3,757 records, so skip never exceeds 4,000 and the ceiling stops binding. Sum the per-year totals and you get exactly 39,538. Full coverage, no key.
RappelConso has the same shape of limit with a different number, and fails more honestly: it returns an error rather than a truncated success.
The 252 MiB surprise
This one cost a production outage, so it is worth being specific.
The pipeline archives every raw payload before parsing it, so a parser bug never means re-hitting seven government endpoints to recover. Good rule. It assumes the payload fits in memory.
Profiling the cold path per source:
wall cpu payload
openfda-device 46,889ms 20,672ms 251.9 MiB
openfda-food 16,027ms 4,406ms 27.4 MiB
openfda-drug 13,010ms 2,891ms 21.6 MiB
cpsc 16,729ms 2,734ms 26.1 MiB
nhtsa 5,206ms 4,515ms 14.1 MiB
rappelconso 29,466ms 2,078ms 22.2 MiB
eu-safety-gate 36,083ms 1,313ms 6.4 MiB
─────────
38,609msDevice enforcement records carry long product descriptions listing every affected catalogue number, so 25,000 records come to 252 MiB. This ran on Cloudflare Workers, where an isolate gets 128 MB and 30 seconds of CPU by default. The payload cannot exist. Gzipping it alone — a step that produces no data — costs 13.6 seconds, 35% of the entire budget.
The failure mode is what makes it worth writing down. A Worker killed for exceeding its limits does not unwind. No exception, no catch, no finally. The row we had written to say "this source is running" stayed saying that forever, the failure counter stayed at zero, and the alert that fires after three consecutive failures never fired, because as far as the database was concerned nothing had failed.
Six of seven sources stopped ingesting and every dashboard reported healthy. It was caught by building a status page and noticing it said "all sources current" beside a tile reading "1 / 7."
Two lessons, both cheap:
Reap your own abandoned work. When a run starts, close out anything still marked running from last time and count it as a failure. That converts an invisible kill into a countable event, and it is about eight lines.
Do not let independent jobs share one budget. Seven sources ran in one invocation, so the most expensive decided whether the cheapest ran at all. One trigger per source costs nothing and removes the entire failure class.
Streaming is not a memory fix
Look again at the profiling table. NHTSA sits near the bottom at 14.1 MiB, one of the cheapest sources there. That number is a lie, and it is a lie of my own making: it measures the payload as served, and NHTSA serves a ZIP.
A profiler that records what arrives on the wire reports this source as harmless. The number that matters is not visible until something tries to decompress it:
$ ls -l FLAT_RCL_POST_2010.zip
14713512 FLAT_RCL_POST_2010.zip # 14 MB. Cheapest source in the table.
$ unzip -l FLAT_RCL_POST_2010.zip
308496873 FLAT_RCL_POST_2010.txt # 294 MiB. Hm.
$ # isolate memory limit: 128 MB
$ #
$ # ah.The isolate found out before I did:
RangeError: Invalid typed array length: 308496873That is not an approximation of the problem, it *is* the uncompressed size. The error message and the file listing agree to the byte, which was the most helpful thing that happened that morning.
unzipSync allocates the decompressed member as a single buffer, in a 128 MB isolate. Deterministic, every run, at the same instruction. It never looks intermittent once you have the number in front of you.
The obvious fix is to stream: inflate through DecompressionStream, and collapse rows as they arrive rather than materialising the file. That reduction is real: the flat file carries one row per campaign *per model*, so 243,199 rows collapse to 15,133 campaigns, and a campaign covering eight models arrives as eight rows carrying identical defect prose with only the model name differing. Fold rows in as they arrive and each one becomes garbage immediately.
I wrote that, measured the retained text at 15.4 MiB, and it still ran out of memory at a 192 MB heap.
The reason is worth knowing, because it is invisible in the code. In V8, String.prototype.split and slice return sliced strings: a length, an offset, and a pointer to the string they were cut from. The substring does not own its characters. So retaining one short field per campaign kept alive every 1 MB decoded chunk that any field had been cut from, and all 294 MiB stayed reachable. Streaming had moved the allocation without removing it. The heap profile showed 15 MiB of strings I wanted and 200 MiB of parents I did not know I was holding.
The fix is to decode each field from the line's bytes with TextDecoder, which returns a fresh string that owns its own characters and points at nothing:
unzipSync + split killed at 192 MB heap
stream + split killed at 192 MB heap
stream + decode per field 67.1 MiB peak, 1.6s parseTwo things I would take to the next project. "Stream it" is not a memory fix by itself. Retention is decided by what the surviving objects point at, and in V8 a substring points at its parent. And profile the size you process, not the size you fetch. Every compressed source is a payload figure hiding a multiplier, and this one was 21×.
The amendments nobody announces
The part I did not expect to matter most.
Agencies revise recalls after publication. A recall issued in March covering three lot codes covers eleven by June. The URL does not change. There is no changelog, no revised_at field, no notification. The page just says something different than it did.
If you check a product against a recall in April, clear it, and the recall widens in June to include it, nothing tells you. You checked, you have a record of checking, and your record is now wrong.
The only way to see this is to diff. Store a content hash per record, compare on every ingest, and write a field-level delta when it moves. Across this corpus that produces a steady stream of amendments: models added, date ranges extended, hazard descriptions sharpened. None of it was announced anywhere.
For anyone doing this for compliance rather than curiosity, that stream is the actual product. Under the EU's General Product Safety Regulation the obligation is continuous diligence, and "we subscribed to a feed" is not evidence of it. What you need is a dated record of what you checked, against a corpus of known freshness, with the result frozen as it stood, so that a later amendment does not silently rewrite your own audit trail.
If you are building this yourself
Worth doing. It is a genuinely interesting normalisation problem and the data is free. Budget for these, which is where the time actually goes:
- Reconciling record granularity across sources — days, and the decision is irreversible
- Mining and check-digit validating identifiers out of prose — a week, and skimping shows
- Finding every undocumented pagination cap — you find these by auditing totals, not by reading docs
- Amendment detection — cheap to build, and worth more than the rest combined
- Not letting one oversized source take down the run — one afternoon, after it bites you once
- Measuring decompressed sizes, not download sizes — five minutes, and it would have saved me two
The schemas are the part that looks hard and isn't.
---
*The corpus described here is available as an API and as a dated, checksummed export at recallproven.com. Every source is under an open licence permitting commercial reuse; the attribution obligations are published at /v1/sources and travel with every export.*
RecallProven turns these seven feeds into one API with a consistent schema, stable identifiers and a per-check audit trail. See the pricing or read the documentation.