Continuous inference provenance
for public sector AI

Budget controls and compliance controls are different conversations

Public sector AI teams typically approach governance in two phases. The first phase is cost visibility: how many tokens, which models, at what cost per query. That conversation belongs to finance and procurement. It is tractable and well-served by existing observability tooling.

The second phase is harder. Compliance and FedRAMP stakeholders do not ask about token counts. They ask about audit artifacts: what model version processed each request, what input it received, what decision it produced, and whether the chain of custody is complete from inference to archival. Those questions require provenance, and provenance is not a subset of cost monitoring. Provenance is a prerequisite for FedRAMP-oriented AI governance; it is not a replacement for the broader authorization boundary, control implementation, or assessment process.

Most governance frameworks stop short of end-to-end lineage. They capture model version at registration time, but not at inference time. They log routing decisions in application code, but not as durable, hash-anchored records delivered to an audit sink. They track downstream consumers in documentation, not in the record itself. StreamKernel was built to close every one of those gaps in a single JVM runtime, delivering SHA-256 input, output, model-reference, and configuration fingerprints on every event.

What continuous provenance actually means

"Continuous" means provenance is not sampled, not reconstructed after the fact, and not stored only in a side-channel log. Every inference event carries its own complete audit record as a first-class field of the delivered message. The record cannot be separated from the payload in transit or at rest.

StreamKernel's public sector provenance transform stamps each event with five distinct layers of lineage before delivery:

Each layer contributes distinct fields to the audit record. A gap at any layer produces an incomplete lineage chain that cannot satisfy audit requirements. StreamKernel enforces completeness at the transform level: if any required provenance field cannot be populated, the record is routed to the dead-letter queue rather than delivered as a partial audit artifact.

Transform chain

The public sector provenance pipeline executes three transforms in sequence per event: STRING_TO_WIREEVENT normalizes the inbound event to StreamKernel's internal wire format, DJL_EMBEDDING runs in-process semantic embedding inference using ONNX Runtime against the registered champion model, and PUBLIC_SECTOR_PROVENANCE stamps the complete audit envelope before Kafka delivery.

Inference runs in-process. There is no model server, no sidecar, no network hop between the pipeline and the embedding model. The inference timestamp, input hash, output hash, and model reference hash are computed inside the same JVM invocation that processes the event.

What each record carries

The post-run verifier sampled records directly from the delivered Kafka topic and confirmed all required fields in the JSON payload and all required headers. The full set is reproduced below; this is not aspirational — it was confirmed present in every sampled record from the 5-minute proof run.

Annotated record sample

The following is a representative record from the proof run, condensed to the lineage-relevant fields. The lineage_chain array documents the full provenance path; audit_record is a self-contained evidence object that can be extracted independently.

"model_registry": {
  "name":       "streamkernel-minilm-onnx",
  "alias":      "champion",
  "version":    "4",
  "run_id":     "060b0c4bdfb14898b7dafd0d7123fe1c"
},
"inference_runtime": {
  "engine":              "OnnxRuntime",
  "execution_provider":  "CPU",
  "inference_timestamp": "2026-05-28T02:10:45.982557300Z"
},
"routing_layer": {
  "policy_version": "public-sector-ai-routing-policy-v1",
  "decision":      "APPROVE",
  "allowed_routes": ["databricks_delta_audit"],
  "denied_routes":  []
},
"input_sha256":  "b5d0a1f06e979f62cf1d66976a92432f...",
"output_sha256": "7ef293c6bcf6d40934ef9013d897b547...",
"lineage_chain": [
  "model_registry", "inference_runtime",
  "routing_layer",   "sink_delivery",
  "downstream_consumers"
],
"downstream_consumers": [
  "auditor_workbench",
  "fedramp_evidence_export",
  "databricks_delta_lineage_loader"
]

An escalation record — classification CUI, FedRAMP impact moderate — carries expanded reason codes (CUI_CLASSIFICATION, SENSITIVE_PERSON_DATA, FEDRAMP_MODERATE_IMPACT, MODEL_CONFIDENCE_REVIEW) and routes to compliance_escalation with public_dashboard and external_ai listed as denied routes. The denied-route list is part of the durable record, not a transient routing decision.

5-minute benchmark: 28,304 records, zero loss

The proof run executed a 5-minute FedRAMP-oriented inference provenance benchmark against a 12-partition Kafka topic following a code review. The unit test for the public sector provenance transformer passed, the JAR was rebuilt, and all provenance contract fields were verified by a post-run sampler reading directly from the delivered topic.

Decision coverage

The post-run verifier confirmed all four routing decisions were present across 28,304 records. The distribution is weighted, not equal — REVIEW is the most common outcome at 37.5%, ESCALATE the least at 12.5%, reflecting a realistic risk profile rather than synthetic uniform splitting. All four decision paths exercised the full provenance contract.

Pre-run model resolution

The MLflow registry alias was resolved before the pipeline started, not lazily at first inference. The champion model version, run ID, and artifact URI were confirmed present in the preflight and stamped into every event at the transform level — establishing a verifiable chain of custody from registry registration to delivered audit artifact. The post-run verifier confirmed the model identity in sampled records read directly from the delivered Kafka topic.

mTLS, OIDC, OPA, and retention governance — all confirmed under load

The baseline run in Section 04 validated the provenance contract and throughput floor under PERMIT_ALL security. This section documents a second, independent 5-minute run executed after a targeted code review and SSL fix, with every security layer active and verified. The provenance contract was unchanged. The security stack was additive.

Security stack confirmed before the run started

Hardened run results

Decision coverage — hardened run

Decision counts were drawn from an authoritative Kafka topic replay after pipeline shutdown, not from in-flight metrics. The weighted distribution was preserved from the baseline run.

What the sampled records show

The post-run verifier sampled one record for each decision type. The four records together demonstrate the full enforcement range of the hardened policy in a single evidence snapshot:

Record 1  UNCLASSIFIED / FedRAMP moderate / APPROVE
  agency: civilian-benefits-agency  |  mission: benefits-case-triage
  route: databricks_delta_audit     |  risk_score: 0.1536
  reason_codes: [FEDRAMP_MODERATE_IMPACT]
  denied_routes: []
  sink.auth: MTLS  |  security.type: OPA_SIDECAR

Record 2  CUI / FedRAMP moderate / REVIEW
  agency: state-transportation-office  |  mission: permit-review-assistant
  route: human_review_queue            |  risk_score: 0.5253
  reason_codes: [CUI_CLASSIFICATION, SENSITIVE_PERSON_DATA, FEDRAMP_MODERATE_IMPACT,
                 SOURCE_REQUESTED_HUMAN_REVIEW]
  denied_routes: [public_dashboard, external_ai]
  data_categories: [PII, CUI]

Record 3  CUI / FedRAMP HIGH / ESCALATE
  agency: public-health-agency  |  mission: population-risk-screening
  route: compliance_escalation   |  risk_score: 0.6784
  reason_codes: [CUI_CLASSIFICATION, SENSITIVE_PERSON_DATA, FEDRAMP_HIGH_IMPACT,
                 LOW_MODEL_CONFIDENCE, SOURCE_REQUESTED_HUMAN_REVIEW]
  denied_routes: [public_dashboard, external_ai]
  data_categories: [PII, PHI, CUI]

Record 4  CONTROLLED / FedRAMP HIGH / DENY
  agency: justice-modernization-program  |  mission: case-search-assistant
  route: dlq                              |  risk_score: 0.8414
  reason_codes: [CONTROLLED_CLASSIFICATION, REGULATED_GOVERNMENT_DATA, SENSITIVE_PERSON_DATA,
                 FEDRAMP_HIGH_IMPACT, SOURCE_REQUESTED_HUMAN_REVIEW,
                 PUBLIC_OR_EXTERNAL_SINK_REQUESTED, SENSITIVE_DATA_TO_PUBLIC_OR_EXTERNAL_SINK]
  denied_routes: [public_dashboard, external_ai, databricks_delta_audit]
  data_categories: [PII, CJIS, CUI]

Every record in this sample carried streamkernel.provenance.sink.auth:MTLS, streamkernel.provenance.security.type:OPA_SIDECAR, the policy reference SHA-256, the config SHA-256, the model reference SHA-256, the transform chain, and the full model registry identity — in the Kafka headers, not just the JSON payload. The denied-route list is part of the durable record. A compliance reviewer can determine not only what was approved, but what was explicitly blocked and why.

The performance knob that was actually a compliance knob

Tuning this pipeline produced an unexpected finding that deserves its own section because it illustrates exactly how provenance requirements interact with performance optimization in practice.

Early tuning attempts explored reducing the ai.embedding.input.max.chars setting to increase throughput. The "wide" and "balanced" profiles set this value aggressively. Both produced zero completed records despite high attempted ingestion:

The failure mode was not a performance failure — it was a provenance failure. Setting ai.embedding.input.max.chars to 512 caused public sector JSON records to be truncated before the audit envelope could be fully formed. The hash of a truncated input is not the hash of the original input. Records were dropped rather than delivered with a compromised audit chain.

The winning configuration preserved full 1024-character input admission and reduced tokenizer sequence length from 64 to 32. Throughput nearly doubled compared to the baseline. The compliance contract remained intact.

What both runs demonstrate for FedRAMP-oriented deployments

The following proof points are supported by artifacts from both the baseline run (Section 04) and the hardened run (Section 05). Where a point is strengthened by the hardened run specifically, it is noted.

JVM-native, air-gap capable, security-stack validated

StreamKernel runs as a single fat JAR. Inference executes in-process via ONNX Runtime. There is no separate model serving infrastructure, no network dependency between the pipeline and the inference layer, and no mandatory cloud connectivity. Both runs in this paper executed entirely on a CPU-only workstation with no external model service.

The hardened run confirmed that the full security stack — mTLS Kafka transport, Keycloak OIDC identity plane, fail-closed OPA policy sidecar, authenticated Prometheus metrics, and environment-specific retention manifests — operates within the same single-JVM deployment model. There is no additional infrastructure tier required to activate the security profile. The difference between the baseline and hardened profiles is configuration, not architecture.

GPU-backed inference can be evaluated separately for higher-throughput deployment profiles. The CPU-only numbers in this paper represent the minimum viable floor for constrained or air-gapped environments. For agencies operating in air-gapped or constrained-connectivity environments, the in-process inference architecture is not a workaround — it is the architecture.

The transform chain is pipeline-configurable. The provenance fields, routing policy, security type, and downstream consumer contract are all driven by the pipeline properties file. New agencies, mission systems, classification levels, and security profiles can be onboarded by configuration rather than code change.

StreamKernel as a governed ingestion layer, not a replacement

For teams building on Databricks, the value of StreamKernel is not replacing Delta Lake, MLflow, or Unity Catalog. The value is creating a governed ingestion and inference layer that emits audit-ready records before they land downstream. StreamKernel operates at the event boundary: it resolves the model registry alias, runs in-process inference, stamps the routing decision and lineage chain into the record, and delivers it to Kafka with the full provenance envelope intact.

That record then becomes the input to downstream consumers — an audit workbench, a Delta Lake lineage loader, a FedRAMP evidence export pipeline. The governance artifact exists at the moment of delivery, not as a reconstruction from logs after the fact. Compliance reviewers get a durable, self-describing record. Delta, MLflow, and Unity Catalog get a richer, pre-governed event to work with.

This is the gap in most public sector AI governance architectures today: budget controls tell finance what the model cost; end-to-end provenance tells compliance what actually happened. StreamKernel is designed to produce the second kind of record as a first-class output of every pipeline execution.

The end-to-end lineage is the hard part most frameworks stop short of

Governance frameworks that treat provenance as a logging concern rather than a delivery concern will produce audit trails that cannot satisfy regulators. The gap is not in intention — it is in architecture. Logs can be lost, filtered, or delayed. A durable Kafka record with SHA-256 hashes in its headers and a complete lineage chain in its payload is evidence that survives the pipeline.

StreamKernel demonstrated continuous inference provenance across all five lineage layers — model registry, inference runtime, routing layer, sink delivery, and downstream consumers — in two independent validated runs. The baseline run delivered 28,304 records at 94.0 avg EPS with zero drops under a synthetic workload. The hardened run delivered 17,808 records at 59.1 avg EPS with the same zero-drop result, adding mTLS Kafka delivery, Keycloak OIDC token exchange, fail-closed OPA policy enforcement, authenticated observability, and retention governance — all confirmed in the artifacts before and after the run.

The measured throughput difference between the two profiles — 94.0 vs 59.1 avg EPS — is the honest, documented cost of the security stack on a single-node CPU deployment. It is not hidden. It is the kind of number a public sector architect needs to plan with, and it is available because both runs were instrumented and retained.

The 46,112 records across both runs are not benchmark numbers. Each one is a proof that a specific model version, at a specific moment in time, received a specific input, produced a specific output, made a specific routing decision, and delivered that record to specific authorized consumers under a versioned consumer contract. In the hardened run, the transport was mutually authenticated, the policy engine was fail-closed, and the security mode is stamped into the record itself.

That is the compliance conversation.