tools/list · tools/call · resources/read · audit · mutation gate
StreamKernel is a JVM-native event pipeline kernel designed to run governed data movement, transformation, inference, observability, and delivery inside one process. The architectural premise is direct: for many real-time AI and operational data pipelines, splitting ingestion, transformation, inference, policy, and delivery across separate services creates unnecessary network hops, serialization boundaries, failure surfaces, and operational drag.
The embedded MCP control plane extends that runtime with an opt-in, agent-addressable interface. When enabled, a local or authorized MCP-compatible client can inspect the active pipeline, validate configuration, check dependency health, query Kafka offsets, inspect DLQ state, read benchmark summaries, and dry-run or execute guarded operational mutations through JSON-RPC.
Core idea: this is not an external agent gateway, sidecar, or monitoring proxy. The MCP server is embedded inside the same JVM as the active StreamKernel pipeline.
Most real-time AI systems distribute responsibility across a broker, stream processor, transformation service, model server, sink connector, observability stack, and control plane. That model works, but each boundary adds deployment surface, security policy, serialization overhead, network latency, credentials, monitoring, and failure modes.
StreamKernel’s MCP control plane exists because once the pipeline runtime becomes the place where state, policy, inference, and delivery converge, agents and operators need a safe way to ask operational questions and preflight actions against the live runtime.
At runtime, StreamKernel loads a pipeline from configuration. A profile declares source type, sink type, transformer chain, cache behavior, metrics provider, security provider, DLQ behavior, and optional MCP settings.
The MCP server runs inside the same JVM boundary as the active pipeline rather than in a sidecar, gateway, or external operator service.
stdio and HTTP transports expose the same catalog, schema contract, audit behavior, and mutation gates.
Guarded tools support dry-run behavior and fail closed unless mutations are explicitly enabled.
The current implementation exposes one active JVM pipeline and does not yet include a full long-running profile manager.
The embedded server supports the expected MCP interaction model: initialize, list tools, call tools, list resources, read resources, and ping the server. The resource catalog exposes bounded read-only snapshots of pipeline, model, health, and metrics state.
| Method | Purpose |
|---|---|
| initialize | Returns server metadata and capabilities. |
| tools/list | Lists StreamKernel MCP tools and input schemas. |
| tools/call | Invokes a named operational tool. |
| resources/list | Lists JSON resources for pipeline, model, health, and metrics. |
| resources/read | Reads a StreamKernel JSON resource. |
| ping | Lightweight health response. |
| Resource URI | Resource |
|---|---|
| streamkernel://pipeline/current | Current pipeline status snapshot. |
| streamkernel://model/status | Current model status snapshot. |
| streamkernel://health/summary | Pipeline health summary. |
| streamkernel://metrics/snapshot | Point-in-time metric values. |
The tool catalog exposes 15 read-oriented operations and 5 guarded mutations. This split matters because the operational surface is useful to agents even when mutation execution remains disabled.
| Tool | Type | Engineering purpose |
|---|---|---|
| streamkernel.pipeline.list | read | Returns active pipeline profiles and sink configuration. |
| streamkernel.pipeline.status | read | Returns source, transformer, sink, and security status for the active pipeline. |
| streamkernel.model.status | read | Reports loaded model and inference health state. |
| streamkernel.model.history | read | Reads recent model transition records from MCP audit history. |
| streamkernel.health.summary | read | Reports health, DLQ depth, record loss counters, and uptime. |
| streamkernel.sink.throughput | read | Reports average records per second and cumulative sink count. |
| streamkernel.dlq.inspect | read | Reports DLQ counts and error classifications. |
| streamkernel.lineage.query | read | Queries MCP control-plane audit records by tool and time range. |
| streamkernel.metrics.snapshot | read | Returns a point-in-time metrics snapshot. |
| streamkernel.auth.status | read | Reports security identity, auth cache state, and control posture. |
| streamkernel.config.validate | read | Validates the active profile and MCP hardening posture. |
| streamkernel.dependency.health | read | Checks TEI-compatible embedding endpoint health and dimensionality. |
| streamkernel.kafka.topic.counts | read | Uses Kafka AdminClient to estimate topic record counts from offsets. |
| streamkernel.benchmark.matrix.discover | read | Discovers benchmark matrix CSV entries. |
| streamkernel.benchmark.last_run.summary | read | Summarizes latest benchmark CSV evidence. |
| streamkernel.pipeline.start | mutate | Starts a named pipeline profile when a profile manager is available. |
| streamkernel.pipeline.stop | mutate | Requests graceful shutdown of the active pipeline. |
| streamkernel.model.promote | mutate | Updates MLflow alias or stage for a model version. |
| streamkernel.model.rollback | mutate | Rolls back MLflow alias or stage to a prior model version. |
| streamkernel.auth.refresh | mutate | Expires the authorization cache so checks refresh through the security provider. |
Every tool response uses a common envelope with ok, tool, timestamp, and correlation_id. Errors return typed StreamKernel-style codes instead of exposing raw JVM stack traces.
Why architects should care: stable envelopes and typed errors make agent workflows deterministic. Agents can branch on response fields instead of scraping logs or parsing exception strings.
The embedded MCP control plane is opt-in and intentionally conservative. The local smoke profile is evidence-focused, not a production security template. It binds MCP HTTP and Prometheus to loopback, disables mutations, and enables audit.
mcp.server.enabled=true.127.0.0.1 by default.mcp.http.max.request.bytes.mcp.mutations.enabled=true.Production note: before exposing MCP outside local-only usage, enable HTTP auth, tool-level authorization, bounded request sizes, audit retention, and network controls.
The final local smoke run established the current MCP evidence baseline. This is a control-plane smoke test, not a long-duration load test or production resilience claim.
| Evidence item | Observed value |
|---|---|
| Status | PASS |
| Run ID | mcp-smoke-20260524-231344 |
| Tool count | 20 |
| Pipeline status | RUNNING |
| Config valid | true |
| Dependency health | true |
| Kafka health | true |
| Kafka topic | arena-bench-test |
| Kafka estimated records | 816 |
| Benchmark matrix entries | 15 |
| Latest result file | results_20260524_1828.csv |
| Blocked mutation code | SK_MCP_MUTATIONS_DISABLED |
An agent can read health, status, config validation, dependency health, Kafka topic counts, and benchmark summaries before a demo, deployment, or benchmark.
Benchmark matrix discovery and latest-run summaries expose engineering evidence through a formal tool instead of manual file copying.
A release agent can dry-run model promotion or rollback before a human-approved mutation path is enabled.
Platform teams can validate whether profiles are loopback-bound, whether request caps exist, and whether mutation posture fits the environment.
The current implementation exposes one active JVM pipeline. It does not yet include a general profile manager capable of cold-starting arbitrary profiles inside one long-running process. Because of that, streamkernel.pipeline.start fails explicitly with a single-process unsupported error unless a future profile manager is registered.
streamkernel.lineage.query currently reads MCP control-plane audit records. It does not query full data-plane lineage records. Data-plane lineage remains stamped onto emitted records as part of the pipeline, but it is not indexed by the embedded MCP server in this implementation.
Evidence boundary: the smoke run proves MCP readiness and operational behavior. It does not claim long-duration resilience, high-throughput control-plane performance, multi-tenant isolation, or production-grade remote exposure.
For production or shared environments, harden beyond the local smoke profile. Keep MCP local unless remote access is explicitly required, protect HTTP when enabled, keep mutations closed by default, and enable tool-level authorization in controlled environments.
StreamKernel’s embedded MCP control plane is architecturally interesting because it places a standard agent interface directly inside a governed event and inference runtime. The control surface is bounded, schema-described, auditable, and mutation-gated.
| Question | Answer |
|---|---|
| Is this a sidecar? | No. The MCP server is embedded inside the StreamKernel JVM. |
| Can agents inspect runtime state? | Yes, through read-oriented MCP tools and JSON resources. |
| Can agents mutate production state by default? | No. Mutations are disabled unless explicitly enabled, and guarded tools support dry runs. |
| Is this a replacement for every stream processor? | No. It is strongest for governed, model-aware, policy-aware real-time pipelines where fewer runtime boundaries matter. |
| What is proven? | A control-plane smoke baseline: tool exposure, runtime status, dependency checks, Kafka counts, benchmark discovery, dry-run mutation, and blocked mutation path. |
StreamKernel’s embedded MCP control plane turns a single-process event and inference runtime into an agent-addressable operational surface. It allows agents and automation to observe health, inspect configuration, check dependencies, query Kafka state, read evidence, and safely preflight operational changes without deploying a separate control service.
The design is most compelling for teams that care about governed real-time AI, low-latency in-process inference, air-gapped deployment, and operational auditability. Its value is not only that agents can call tools. Its value is that those tools are close to the runtime they control.
Final takeaway: in systems where every additional service is another security review, failure boundary, and operational burden, embedding the control plane inside the pipeline kernel is a pragmatic architectural tradeoff.