Advanced Configuration

This guide explains every configuration option in plain language.

Use this page when you want to move beyond defaults and tune KalamDB for your workload.

Before You Start

• If you are new to setup, start with Getting Started: Configuration.
• If you are not sure about a setting, keep the default value first and only tune after observing behavior.
• Prefer changing one section at a time, then test.

Download Full Sample Config

Use this file when you want all available settings documented in one place with comments:

• server.example.toml (full commented sample) 

Quick download command:

curl -L https://raw.githubusercontent.com/jamals86/KalamDB/refs/heads/main/backend/server.example.toml -o server.sample.toml

Loading Order

1. Load TOML config file (--config /path/to/server.toml)
2. Apply environment variable overrides (KALAMDB_*)
3. Normalize paths and validate configuration

Full Configuration Reference (User-Friendly)

[server]

• host: Network interface to bind the server to. Use 127.0.0.1 for local-only access, or 0.0.0.0 to accept remote connections.
• port: HTTP port clients connect to.
• workers: Number of worker threads for request handling. 0 usually means “auto based on CPU”.
• api_version: URL prefix version for API routes (for example v1).
• enable_http2: Enables HTTP/2 support. Keep enabled unless you have specific proxy compatibility issues.
• ui_path: Optional path to Admin UI static files. Leave empty/none to disable serving UI from the server.

[storage]

• data_path: Base directory where local data is stored.
• shared_tables_template: Folder structure template for shared tables.
• user_tables_template: Folder structure template for user-isolated tables.

[storage.rocksdb]

• write_buffer_size: Memory per write buffer. Increase for heavier write throughput, decrease for lower memory usage.
• max_write_buffers: How many write buffers can accumulate before flush pressure increases.
• block_cache_size: Read cache size for RocksDB blocks. Higher values improve repeated reads but consume more RAM.
• max_background_jobs: Number of background compaction/flush workers.
• max_open_files: Maximum number of files RocksDB keeps open. Lower to reduce file-descriptor pressure.
• sync_writes: Forces WAL fsync on writes. Improves durability but can significantly reduce write speed.
• disable_wal: Disables write-ahead log. Faster but unsafe for crash recovery.
• compact_on_startup: Runs compaction when server starts. Helps with fragmentation but increases startup time.

[storage.remote_timeouts]

• request_timeout_secs: Maximum time for a remote storage request to complete.
• connect_timeout_secs: Maximum time allowed to establish a remote storage connection.

[limits]

• max_message_size: Maximum incoming payload size accepted by the server.
• max_query_limit: Hard upper bound for LIMIT in queries.
• default_query_limit: Default LIMIT used when query does not specify one.

[logging]

• level: Global logging verbosity (error, warn, info, debug, trace).
• logs_path: Directory where log files are written.
• log_to_console: If enabled, logs are also printed to stdout/stderr.
• format: Log style (compact, pretty, json). Use json in production log pipelines.
• targets: Per-component log overrides (for example quieter datafusion, noisier app code).
• slow_query_threshold_ms: Queries slower than this are marked/logged as slow.

[logging.otlp]

• enabled: Turns OpenTelemetry trace export on/off.
• endpoint: OTLP collector endpoint.
• protocol: OTLP transport protocol (grpc or http).
• service_name: Name shown in tracing tools (Jaeger/Tempo/etc.).
• timeout_ms: Maximum export timeout for telemetry batches.

[performance]

• request_timeout: Max time for full request handling.
• keepalive_timeout: Idle keep-alive connection timeout.
• max_connections: Max simultaneous open client connections.
• backlog: Pending TCP connection queue depth.
• worker_max_blocking_threads: Limit for blocking tasks per worker; protects server under heavy sync operations.
• client_request_timeout: Time allowed for client to send request headers.
• client_disconnect_timeout: Time allowed for graceful client disconnect.
• max_header_size: Max HTTP header size (important for large JWT/cookies/custom headers).

[datafusion]

• memory_limit: Memory budget for query execution.
• query_parallelism: Number of parallel query execution threads.
• max_partitions: Upper bound of partitions used in distributed query planning.
• batch_size: Record batch size for processing pipelines.

[flush]

• default_row_limit: Flush to durable storage after this many rows.
• default_time_interval: Flush after this many seconds even if row limit is not reached.
• flush_batch_size: Rows processed per batch during flush job.

[manifest_cache]

• eviction_interval_seconds: How often cache cleanup runs.
• max_entries: Maximum number of manifest entries kept in memory.
• eviction_ttl_days: Remove entries not accessed within this many days.
• user_table_weight_factor: Eviction bias; higher values evict user-table entries sooner than shared-table entries.

[retention]

• default_deleted_retention_hours: Default soft-delete retention window before hard cleanup.

[stream]

• default_ttl_seconds: Default lifetime of stream events.
• default_max_buffer: Max in-memory buffer size for stream rows/events.
• eviction_interval_seconds: How often expired stream data is removed.

[websocket]

• client_timeout_secs: Disconnect clients that stop responding past this timeout.
• auth_timeout_secs: Time allowed to authenticate after opening WebSocket.
• heartbeat_interval_secs: Frequency for heartbeat checks/pings.

[rate_limit]

• max_queries_per_sec: Per-user query rate cap.
• max_messages_per_sec: Per-WebSocket message rate cap.
• max_subscriptions_per_user: Maximum concurrent subscriptions per user.
• max_auth_requests_per_ip_per_sec: Protects login/setup endpoints from brute-force or bursts.
• max_connections_per_ip: Caps concurrent sockets per client IP.
• max_requests_per_ip_per_sec: Caps unauthenticated/early request flood rate.
• request_body_limit_bytes: Hard cap for request payload body size.
• ban_duration_seconds: Temporary ban duration after abuse thresholds are exceeded.
• enable_connection_protection: Master switch for connection abuse protection middleware.
• cache_max_entries: Max in-memory entries for rate-limiter state.
• cache_ttl_seconds: Idle expiration for rate-limit cache entries.

[auth]

• root_password: Optional root password in config file (you can also provide by env var).
• jwt_secret: Secret key used to sign/verify JWTs. Must be strong and private.
• jwt_trusted_issuers: Restricts accepted token issuer(s).
• jwt_expiry_hours: Token expiration lifetime.
• cookie_secure: Sends auth cookies only over HTTPS when true.
• min_password_length: Minimum accepted user password length.
• max_password_length: Maximum accepted user password length.
• bcrypt_cost: Password hash work factor; higher = stronger but slower.
• enforce_password_complexity: Requires mixed character classes in passwords.
• allow_remote_setup: Allows first-time setup from non-localhost clients.

[oauth]

• enabled: Enables OAuth login flow.
• auto_provision: Automatically creates user on first successful OAuth login.
• default_role: Role assigned to newly auto-provisioned OAuth users.

[oauth.providers.google], [oauth.providers.github], [oauth.providers.azure]

• enabled: Enables this provider specifically.
• issuer: Expected token issuer URL.
• jwks_uri: URL to provider public signing keys.
• client_id: OAuth app client ID.
• client_secret: OAuth app client secret (required for some flows/providers).
• tenant: Tenant identifier (mainly used for Azure).

[user_management]

• deletion_grace_period_days: Days to wait before permanently deleting soft-deleted users.
• cleanup_job_schedule: Cron schedule for user cleanup background task.

[shutdown]

[shutdown.flush]

• timeout: Maximum wait time (seconds) for flush jobs during graceful shutdown.

[jobs]

• max_concurrent: Max parallel background jobs.
• max_retries: Retry attempts before marking job failed.
• retry_backoff_ms: Initial retry delay between attempts.

[execution]

• handler_timeout_seconds: Max duration for execution handlers.
• max_parameters: Maximum number of SQL parameters in a request.
• max_parameter_size_bytes: Max size for a single SQL parameter value.
• sql_plan_cache_max_entries: Max number of cached query plans.
• sql_plan_cache_ttl_seconds: Idle TTL before cached plans are evicted.

[security]

• allowed_ws_origins: Allowed origins for WebSocket connections.
• max_ws_message_size: Max allowed WebSocket message size.
• strict_ws_origin_check: If true, rejects WS connections without valid Origin.
• max_request_body_size: Global HTTP request size limit.

[security.cors]

• allowed_origins: Allowed browser origins for cross-origin requests.
• allowed_methods: Allowed HTTP methods for CORS.
• allowed_headers: Allowed request headers from browsers.
• expose_headers: Response headers exposed to browser JavaScript.
• allow_credentials: Allows cookies/authorization with CORS requests.
• max_age: Browser cache duration for preflight responses.
• allow_private_network: Allows private-network CORS requests where supported.

[files]

• max_size_bytes: Maximum file size per uploaded file.
• max_files_per_request: Maximum number of files accepted in one request.
• max_files_per_folder: File count threshold before folder rotation.
• staging_path: Temporary path for upload staging before final persistence.
• allowed_mime_types: Restricts MIME types; empty list means all types are accepted.

[cluster] (optional)

• cluster_id: Logical identifier for this cluster.
• node_id: Unique ID of this node inside the cluster.
• rpc_addr: Internal Raft RPC address.
• api_addr: Public/HTTP API address for this node.
• user_shards: Number of shards for user table data.
• shared_shards: Number of shards for shared table data.
• heartbeat_interval_ms: Raft leader heartbeat interval.
• election_timeout_ms: Min/max timeout range for elections.
• snapshot_policy: Controls Raft snapshot frequency and strategy.
• max_snapshots_to_keep: Number of snapshots retained on disk.
• replication_timeout_ms: Timeout for learner/follower catch-up steps.
• reconnect_interval_ms: Delay between reconnect attempts to peers.
• peer_wait_max_retries: Max retries while waiting peers during startup.
• peer_wait_initial_delay_ms: Initial peer-wait backoff delay.
• peer_wait_max_delay_ms: Maximum peer-wait backoff delay.

[[cluster.peers]]

• node_id: Peer node unique ID.
• rpc_addr: Peer internal RPC endpoint.
• api_addr: Peer API endpoint.
• rpc_server_name: TLS server-name override for peer certificate validation.

[cluster.rpc_tls]

• enabled: Enables TLS/mTLS on internal cluster RPC.
• ca_cert_path: CA certificate used to validate peer certificates.
• node_cert_path: This node certificate path.
• node_key_path: Private key path for this node certificate.

Environment Variable Overrides

Environment variables are helpful in containers and CI/CD, where secrets/hosts/ports differ per environment.

Common override groups:

• KALAMDB_SERVER_HOST, KALAMDB_SERVER_PORT
• KALAMDB_LOG_LEVEL, KALAMDB_LOGS_DIR, KALAMDB_LOG_TO_CONSOLE
• KALAMDB_OTLP_ENABLED, KALAMDB_OTLP_ENDPOINT, KALAMDB_OTLP_PROTOCOL, KALAMDB_OTLP_SERVICE_NAME, KALAMDB_OTLP_TIMEOUT_MS
• KALAMDB_DATA_DIR
• KALAMDB_JWT_SECRET, KALAMDB_JWT_TRUSTED_ISSUERS, KALAMDB_JWT_EXPIRY_HOURS, KALAMDB_COOKIE_SECURE, KALAMDB_ALLOW_REMOTE_SETUP
• KALAMDB_RATE_LIMIT_AUTH_REQUESTS_PER_IP_PER_SEC
• KALAMDB_CLUSTER_ID, KALAMDB_NODE_ID, KALAMDB_CLUSTER_NODE_ID, KALAMDB_CLUSTER_RPC_ADDR, KALAMDB_CLUSTER_API_ADDR, KALAMDB_CLUSTER_PEERS
• KALAMDB_CLUSTER_RPC_TLS_ENABLED, KALAMDB_CLUSTER_RPC_TLS_CA_CERT_PATH, KALAMDB_CLUSTER_RPC_TLS_NODE_CERT_PATH, KALAMDB_CLUSTER_RPC_TLS_NODE_KEY_PATH

Legacy variables are still recognized (KALAMDB_HOST, KALAMDB_PORT, KALAMDB_LOG_FILE, KALAMDB_LOG_FILE_PATH, KALAMDB_ROCKSDB_PATH).

Practical Tuning Order

For most user deployments, tune in this order:

1. auth.jwt_secret, security.cors, rate_limit.* (security first)
2. storage.data_path, logging.* (operational basics)
3. performance.*, datafusion.*, flush.* (throughput and latency)
4. cluster.* only when moving to multi-node deployments

Example: Production-Oriented server.toml

[server]
host = "0.0.0.0"
port = 8080
workers = 0
enable_http2 = true
 
[storage]
data_path = "./data"
 
[logging]
level = "info"
format = "json"
logs_path = "./logs"
 
[performance]
max_connections = 25000
 
[auth]
jwt_secret = "REPLACE_WITH_STRONG_SECRET_AT_LEAST_32_CHARS"
cookie_secure = true
enforce_password_complexity = true
 
[rate_limit]
enable_connection_protection = true
max_queries_per_sec = 100
 
[security]
max_request_body_size = 10485760
 
[security.cors]
allowed_origins = ["https://app.example.com"]
allow_credentials = true
 
[files]
max_size_bytes = 26214400

For quick-start defaults and first setup, see Getting Started: Configuration.