The Regulator Does Not Ask About Wednesday
An SEC examination team writes to a broker-dealer asking for the disclosures displayed on a specific product page between the tenth and the seventeenth of a month eighteen months ago. The firm’s compliance team pulls up the archive. There is a capture from the seventh and another from the twenty-first. Between those two captures, nine days of the site’s actual state are missing.
If nothing changed during those nine days, the missing captures do not matter. If something did change – an update to a fee, a revision to a disclosure, a promotional banner added and removed – the firm can produce the version from the seventh and the version from the twenty-first, but not the version that was actually live during the window the examiner asked about. That gap is the material the examiner most wants to see.
Periodic snapshot archiving – weekly, monthly, quarterly – is the model most firms deployed when they first took website recordkeeping seriously. It was better than nothing, and for many years it was adequate to satisfy general awareness of the recordkeeping obligation. It is no longer adequate to satisfy how regulators and courts now use website records. The gap between what the archive holds and what visitors actually saw is where enforcement, litigation, and courtroom challenges land.
This article is about what continuous archiving actually is, why the difference from periodic snapshots is not academic, and what an organisation building a defensible programme should look for.
What Periodic Snapshot Archiving Misses
A snapshot programme runs on a schedule – once a week, once a month, once a quarter – and captures the site as it exists at that moment. It is the simplest archiving model to design, and its failure mode is easy to describe: it does not capture anything that happens between snapshots.
Website content between snapshots is not stable in the way a slow reader might imagine. Modern websites change constantly. Marketing publishes updates. Product teams push new copy. Legal revises disclosures. A/B tests swap variants for hours or days. Promotional banners appear for a campaign and disappear when the campaign ends. Third-party services – embedded videos, chat widgets, tag managers, analytics scripts, personalisation engines – inject or modify content on every page load. A page rendered on Monday is not the page rendered on Wednesday.
The consequences of missing this activity are not evenly distributed. A snapshot programme reliably preserves the state of pages that do not change and reliably misses the state of pages that do change. Since changing pages are precisely the ones a regulator, plaintiff, or auditor is most likely to ask about, the archive tends to have coverage exactly where coverage is not needed and gaps exactly where gaps hurt.
There is a related failure that snapshot programmes share regardless of frequency. A snapshot captures what was live when the crawler happened to run. It does not capture what was published, briefly seen, and taken down between two crawler runs. A promotional page live for six hours during a snapshot’s off-cycle simply never enters the archive. If that page later becomes the subject of a customer complaint or an inquiry, the firm has no record that it ever existed.
What Continuous Archiving Actually Means
Continuous archiving is not “snapshots, but more often.” It is a different design.
A continuous programme detects change and captures on change, rather than capturing on a wall-clock schedule irrespective of whether anything has moved. The archiving system runs against the site frequently – often multiple times a day – comparing what it observes against what it has already preserved. When it identifies a change, it captures the new state and preserves it alongside the earlier state. When it identifies no change, it records that the state has not moved. The resulting archive is a chronologically complete record of every version of every page the archiving system has observed, not a set of disconnected snapshots.
The properties this produces are:
Change resolution close to the change itself. If a page is updated at 10:15 and the next crawl runs at 10:30, the archive records the new state within fifteen minutes of the change. The window during which the archive is behind reality is measured in tens of minutes, not days or weeks. For content that changes hourly, a continuous programme keeps up. A snapshot programme does not.
Coverage of ephemeral content. A page that goes live at 09:00 and comes down at 14:00 is captured by a continuous programme’s next crawl within its normal cadence. That five-hour existence lives permanently in the archive. A weekly snapshot has a one-in-thirty-three chance of catching it. A monthly snapshot has essentially none.
Version chains, not disconnected states. Because each capture is linked to the state that preceded it, the archive can present a chain of versions for any given page: this was the state at 08:00, this at 10:15 after the fee schedule was updated, this at 15:40 after the disclaimer was revised. That version chain is what supports change-history analysis, side-by-side comparison, and answers to questions like “when did the language on this page first say X.”
No dependency on human triage. A continuous programme captures what changes. It does not require anyone to decide in advance which pages are important. This matters because the pages that turn out to be important are frequently not the pages anyone would have flagged in advance.
Why This Matters Under Regulation and in Court
The move from snapshot to continuous archiving is not cosmetic. It changes what the archive can answer.
A snapshot archive can answer “what was on the site the day the snapshot ran.” A continuous archive can answer “what was on the site at any moment we crawled it,” which – for a well-designed programme – means at any moment relevant to a regulatory or litigation request.
Under SEC Rule 17a-4(b)(4) and FINRA Rule 4511, broker-dealers are required to preserve business communications, including website content within the scope of communications-with-the-public rules, for at least three years, the first two in an easily accessible place. The rules do not require the firm to have captured a specific moment; they require the firm to have preserved the records. When an examiner asks for the state of a page during a defined window, the firm is expected to produce those records. A snapshot programme forces the firm to argue, sometimes credibly and sometimes not, that the closest snapshot is a reasonable stand-in for the actual state. A continuous programme sidesteps that argument by having preserved the actual state.
Under FRCP Rule 26, in United States civil discovery, the historical state of a party’s website is electronically stored information within the scope of the rule. A continuous archive makes producing that ESI a lookup against material that already exists. A snapshot archive turns it into a debate about coverage. Under FRCP Rule 37(e), which governs sanctions for lost ESI, the reasonable-steps standard for preservation is easier to satisfy when the archiving programme was designed to capture change than when it was designed to sample.
Under FRE 902(13) and 902(14), self-authenticating electronic records rely on a written certification about a reliable process. A continuous programme – documented, scheduled, auditable, hash-verified at each capture – is a process that a certifier can attest to. A programme built around ad hoc captures triggered by human attention is not.
What a Continuous Programme Actually Requires
The technical properties that support continuous archiving are not exotic, but they need to be present.
Change detection. The archiving system needs to know when to capture, which means it needs to be able to compare the current state of a page against what it has already preserved without recapturing every resource on every crawl. Standard techniques – HTTP conditional requests, content hashing, header inspection – do this efficiently. The alternative, recapturing everything on every visit, wastes storage and bandwidth without producing a better record.
Delta capture and full-record storage. When change is detected, the system captures the new state in full and preserves it alongside the earlier state. Compact representations of the change (a diff) are useful for presentation but do not replace the underlying full record; a regulator or court asking for the state of a page wants the state, not a delta against a state.
WARC per version. Each captured state is stored as a WARC file aligned with ISO 28500. That is the standard format for web archives and the one that lets any conformant tool replay the state. A proprietary internal representation may be efficient but it is not portable, and portability matters when the archive has to be produced.
Cryptographic hash at capture. Each capture carries a hash computed at the moment of capture – SHA-256 is the standard – and the hash is preserved with the capture. This is what supports self-authentication and gives the certifier something specific to attest to.
Immutable storage. Captures are written to storage that either does not allow modification (WORM) or produces a complete audit trail of every attempted modification. This meets the format requirement inherited from SEC Rule 17a-4(f) and prevents the archive itself from becoming a spoliation concern during litigation.
Retention policy that reflects the underlying obligations. Different classes of content have different retention windows. A single blanket policy is simpler but often wrong. The system supports differentiated retention so that customer complaint materials, communications concerning registered investment companies, and general marketing content are each held for the correct period.
Auditable operations. The archiving programme leaves its own audit trail: what was crawled, when, with what result, whether any error was encountered. That operational trail is what a certifier or an examiner uses to distinguish a continuous programme that actually ran continuously from one that ran continuously in principle but stopped running in practice.
The Cost of Getting This Wrong Later
Continuous archiving cannot be retrofitted onto a historical window. If a firm ran a monthly snapshot programme from 2022 to 2025 and switches to continuous archiving in 2026, the archive for 2022-2025 stays what it was: monthly snapshots. The versions of pages that lived between those snapshots are not recoverable. If a regulatory inquiry, a class action, or a customer dispute reaches back into that window, the firm’s archive is only as good as what the snapshot programme preserved.
The right time to move to continuous archiving is before that inquiry. The archive built this year is the archive that answers questions asked next year. The archive not built this year is a gap next year’s inquiry will land in.
About Aleph Archives
Aleph Archives is a continuous web archiving platform. Sites are crawled on a change-driven cadence, each observed state is preserved as a WARC file aligned with ISO 28500, and every capture carries a SHA-256 hash and precise timestamp recorded at the moment of capture. Storage is immutable, retention is configurable per content class, and audit trails support SEC Rule 17a-4 and FINRA Rule 4511 examinations, FRCP Rule 26 production, and FRE 902 self-authentication. Aleph Archives is focused exclusively on web archiving.
