[snapshot-hasher, vc] Hash snapshots in a dedicated service (#803)
Type of change

New feature
Improvement (improvement to code, performance, etc)
Test update
Documentation update

Description

Add service/snapshothasher, a new single-instance service that owns
snapshot hashing end to end: it polls the latest durable _snapshot
record every poll-interval and hashes the record’s clone inline, so
one path covers a fresh snapshot, a resubmitted one, and one orphaned
mid-hash by a restart. Start it with committer start snapshot-hasher
(gRPC 3001, monitoring 2120); it serves only health checks, since work
arrives through the state database, so WaitForReady is unconditionally
true – the readiness handshake exists to hold back a gRPC server that
would otherwise answer requests before it can. It is named for hashing
rather than for snapshots because creating a snapshot and its clone
stays in the VC.
Move the hasher (table discovery, keyset-paginated scans,
deterministic per-table and root digests) out of service/vc into that
service, and remove snapshot hashing from the VC entirely. The VC’s only
snapshot duty is now to make the record durable atomically with its
clone.
Because exactly one scheduler exists and hashing runs inline, no
cross-process exclusion is introduced at all: no lease, no ownership
token, no fenced state transitions, no leader election.
Extract the durable _snapshot record contract into
utils/statedb/snapshot_state.go as a SnapshotStateManager
(latest-record pointer, encoding, FOR UPDATE read/update) and cover it
with unit tests. It lives in statedb, next to the schema it reads,
rather than in a package of its own; utils/statedb therefore joins
REQUIRES_DB_PACKAGES in the Makefile, since its tests now need a
database but not a particular one. Both processes share it: the VC reads
it to gate a new snapshot request and will write CHECKPOINTED through it
when checkpointing lands, while the snapshot service drives PENDING ->
IN_PROGRESS -> COMPLETED. Update requires a status, writes the error
text unconditionally so a record cannot report both COMPLETED and a
previous attempt’s failure, and treats a missing or undecodable row as
terminal instead of spending the retry budget on it; retry.Execute
takes terminal errors for that, matching ExecuteWithResult.
Encode tx_status as int4send(coalesce(status, -1)) || height,
because status is nullable and NULL || height is NULL: without the
coalesce a single NULL status drops that row’s height from the digest,
so two rows differing only in height hash identically.
Open the clone pool before the record is marked IN_PROGRESS, so a
committed record whose clone is not there to hash is left exactly as it
was found. A clone is created before its snapshot transaction commits,
so a committed record always names a clone that exists; neither a
missing clone_database nor a missing database can be produced by this
system, so both are reported as ErrCorruptSnapshotState, stop the
service, and leave the record untouched as evidence.
Make statedb.NewPool treat SQLSTATE 3D000 (invalid_catalog_name)
as terminal instead of retrying it, export ErrDatabaseNotFound so a
caller can tell an absent database from a connection failure, and close
a pool whose ping failed rather than leaking one per attempt. The
database is provisioned before any service starts, so 3D000 means the
configured name is wrong or the cluster was never initialized; retrying
spent the whole retry budget before reporting a condition that was
decidable on the first attempt.
Replace the VC’s max-workers-for-snapshot-hash and
snapshot-hash-batch-size with the new service’s poll-interval,
resource-limits.max-workers-for-hash, and
resource-limits.hash-batch-size; add
cmd/config/samples/snapshot-hasher.yaml and its test template, under
the SC_SNAPSHOT_HASHER_ env prefix.
Export snapshot/state test fixtures from service/vc so the new
service’s tests seed records through the real commit path, and add
docs/snapshot-hasher.md plus snapshothasher_* metrics for
completed/failed jobs, hash duration, poll errors, when the running hash
started, and the shared gRPC server metrics every other service reports.

Additional details (Optional)

Running a second instance of the snapshot hasher is a deployment error
rather than a correctness hazard: a clone is immutable and the digest
deterministic, so the symptom is duplicate work writing the same digest.
This is the trade that keeps the design free of exclusion machinery.
A hash failure stays retryable and is recorded on the record, because
the record is the only queue: nothing re-enqueues it, so a terminal
status would strand the snapshot on one transient failure, and the VC
would then reject every later snapshot. Retrying is cheap and safe,
since a clone is immutable and the digest deterministic.
snapshothasher_poll_errors_total counts ticks that could not
determine whether there is work (state database unreachable, a pointer
naming no row, a record that does not decode). It is incremented where
the failure is classified rather than in the poll loop, since only there
is a failed poll distinguishable from a hash that started and then
failed; the hash-job counters cannot express the former, so without it a
service whose database is down looks like an idle one.
snapshothasher_hash_started_timestamp_seconds describes a hash while
it runs, which nothing else here can: hash_duration_seconds is
observed only once a hash returns, so for the many minutes a full clone
scan takes, a busy service and an idle one publish identical numbers,
and this service has no RPC or queue depth to read instead. Alert on the
elapsed time rather than on the value being non-zero, since a stuck job
holds it forever and never reaches the histogram.
Each hash job opens its own pool on the clone with exactly
max-workers-for-hash connections, since at most one connection per
worker is in use while it pages through its table;
database.max-connections sizes only the record poll, whose sample
default drops to 2.
Hashing latency is bounded by one poll-interval (default 1m):
nothing notifies the service, so that interval is both the scheduling
delay and the restart resume delay.
Planned per-table digests and a Merkle Patricia trie for localizing
divergence between organizations land in this service; both are
finer-grained hashing, so they fit the name rather than outgrowing it.
Upgrading a deployment that already took state snapshots: deploy the
hasher as a new unit and delete the two removed keys from vc.yaml,
which are ignored rather than rejected if left behind. In-flight records
need no migration — the new service picks up whatever state the old VC
left, including a record left IN_PROGRESS by the upgrade itself.
Left as a TODO on hashSnapshot: no hash failure is classified as
permanent. The hasher’s per-page and table-discovery retries pass no
terminal errors, so a permanent failure (a permission change, a dropped
table) would be retried for the whole budget and then again on the next
tick. No operational path reaches that today, so it is recorded rather
than fixed here.

Related issues

resolves #663


Signed-off-by: Senthilnathan cendhu@gmail.com