Research library / Post
Stream. Infer. Persist. Prove.
Introducing StreamKernel's Delta Lake + MLflow profile — a single Java process that pulls a model from a registry, runs AI inference inline, and commits enriched records to a cloud-native Delta table with full Spark read-back verification. Zero record loss. Zero glue code.
- Java 21
- ONNX / DJL
- Delta Lake
- MLflow Registry
- MinIO / S3
- Apache Spark
// Run summary — STREAMKERNEL_DELTA_MLFLOW_LOCAL_5M
- Records processed
- 3,360
- Delta rows written
- 3,360
- DLQ leakage
- 0
- ACID Delta commits
- 105
- Wall-clock runtime
- 5.32min
- Avg ONNX / record
- 74.5ms
demo_after_delta_mlflow.ps1
✓ MLflow alias resolved streamkernel-minilm-onnx / champion → v1
✓ Delta table s3a://streamkernel-delta/enriched-tickets
✓ Spark read-back 3,360 rows confirmed — all queryable
✓ Parity check processed = embedded = out = delta rowsProof
An independent Spark session read back exactly 3,360 rows from the Delta table — confirming ACID durability end-to-end, from ONNX inference to cloud object storage.
// Why this matters
01 No inference sidecar, no separate embedding service. StreamKernel runs the MiniLM model in-process via ONNX Runtime — the same JVM that consumes, enriches, and persists the stream. That's the whole point.
02 MLflow as the model authority. The engine resolves a named alias (champion) at startup — no hardcoded paths, no manual artifact management. Promote a new model in the registry and the next run picks it up automatically.
03 Delta Lake as a first-class sink. Every enriched record lands in a versioned, ACID-compliant Parquet table on S3-compatible storage — ready for Spark, Databricks, or any lakehouse query engine. 105 commits. 20 checkpoints. No data loss.
04 This is early — and that's the point. The benchmark suite is running, the integration contracts are proven, and the architecture is holding. The throughput numbers will come. The foundation is already real.