Architecture Overview
KalamDB is built as a modular system with clear separation between the query engine, storage layer, and real-time pipeline.
Component Diagram
Data Directory Layout
data/
├── rocksdb/ # Hot storage (RocksDB column families)
│ ├── system_* # System tables
│ └── user_* / shared_* # Hot buffers per table
└── storage/ # Cold storage (Parquet segments)
├── user/{user_id}/{table}/
│ ├── manifest.json # Schema + segment index
│ └── batch-<index>.parquet # Flushed segments
└── shared/{table}/
├── manifest.json
└── batch-<index>.parquetkalamdb-core orchestrates everything and never talks to RocksDB or the filesystem directly — it goes through kalamdb-store (key/value hot path) and kalamdb-filestore (Parquet + manifest management).
Technology Stack
| Component | Technology | Purpose |
|---|---|---|
| Language | Rust 1.92+ | Performance, safety, concurrency |
| Storage (Hot) | RocksDB 0.24 | Fast buffered writes (<1ms latency) |
| Storage (Cold) | Apache Parquet 57.0 | Compressed columnar analytics |
| Query Engine | Apache DataFusion 51.0 | SQL across hot+cold storage |
| In-Memory | Apache Arrow 57.0 | Zero-copy data structures |
| API Server | Actix-web 4.12 | REST + WebSocket endpoints |
| Auth | bcrypt + JWT | Password hashing + token auth |
| Deployment | Docker | Production containerization |
Last updated on