Kafka Cluster Operations: Day-2 Management for Enterprise Deployments

Day-2 Kafka: Beyond the Initial Deployment

Kafka is relatively straightforward to deploy and hard to operate well at scale. The initial deployment — configure brokers, create topics, deploy producers and consumers — is well-documented and tooled. The challenge is everything that comes after: managing consumer group lag as throughput grows, handling broker storage as data accumulates, rebalancing partitions after a broker failure, and maintaining performance as topic counts and partition counts scale beyond initial estimates.

This is a practical guide to the operational discipline that keeps enterprise Kafka clusters healthy over the long term.

Consumer Group Lag: The Primary Health Signal

Consumer group lag — the difference between the latest offset produced to a topic partition and the latest offset consumed by a consumer group from that partition — is the most important Kafka operational metric. Growing lag means consumers are not keeping up with producers; if lag grows without bound, the consumer is effectively non-functional for real-time use cases.

Monitoring Consumer Lag

Kafka Lag Exporter (open source) and AKHQ both provide consumer group lag metrics compatible with Prometheus. The key metrics are:

• kafka_consumer_group_lag: lag per partition per consumer group
• kafka_consumer_group_lag_sum: total lag across all partitions for a consumer group

Alert thresholds depend on your use case latency requirements: for real-time processing, alert when lag exceeds 1,000 messages; for batch processing, the threshold might be 1,000,000. The critical alert should trigger when lag is growing — even a small lag that is consistently increasing indicates a consumer that will eventually fall behind completely.

Diagnosing Consumer Lag

• Consumer is slow: Processing time per message exceeds the produce rate. Profile the consumer for blocking operations, slow downstream calls, or insufficient parallelism.
• Consumer is under-partitioned: More partitions allow more consumer threads; if lag appears evenly across partitions, adding partitions (and consumers) increases throughput. Note that adding partitions to an existing topic requires a partition rebalance.
• Consumer is failing and rebalancing: Repeated consumer group rebalances (due to consumer crashes, long processing times exceeding max.poll.interval.ms) create lag spikes. Look for frequent rebalance log entries in consumer application logs.

Broker Storage Management

Kafka retains messages on disk for a configured retention period (default 7 days for time-based retention, or a configured size limit for size-based retention). For high-throughput topics, broker storage can grow rapidly. Key practices:

• Monitor disk usage per broker and alert at 70% of configured disk capacity, not 90% — Kafka performance degrades before disks are full.
• Use log compaction for changelog topics (state stores, CDC topics) rather than time-based retention. Compacted topics retain only the latest value per key, bounding storage independent of write rate.
• Set topic-level retention configurations (retention.ms, retention.bytes) explicitly. Topics without explicit configuration inherit broker defaults, and broker defaults may be inappropriate for specific use cases.
• Monitor the total number of log segments per broker. The Kafka controller uses partition metadata in memory; very large partition counts (tens of thousands per broker) can create controller performance issues.

Partition Rebalancing

Partition leadership is distributed across brokers but can become unbalanced over time — particularly after a broker failure and recovery, where the recovering broker returns without its previously-led partitions. The Kafka preferred replica election mechanism re-elects preferred leaders, but it must be triggered or configured for automatic preferred leader election.

• Enable auto.leader.rebalance.enable=true and set leader.imbalance.check.interval.seconds=300 to automatically trigger rebalancing.
• Use Cruise Control (open source, by LinkedIn) for data-aware partition rebalancing that balances both leadership count and byte-in/byte-out rates across brokers.
• After a broker failure and recovery, explicitly run a preferred replica election (kafka-preferred-replica-election.sh) to return partition leadership to the recovered broker.

KRaft Mode Operations

Kafka clusters migrated to KRaft mode (ZooKeeper-free, available from Kafka 3.3+) have a different operational model for metadata management. The KRaft controller quorum replaces ZooKeeper; controller health monitoring and quorum majority requirements replace ZooKeeper ensemble health. In KRaft mode, monitor controller quorum health via the kafka.controller:type=KafkaController,name=ActiveControllerCount JMX metric and ensure the controller quorum maintains majority (3 of 5, or 2 of 3 controller nodes, depending on configuration).

OSSeva Operate for Kafka

OSSeva Operate for Kafka clusters includes pre-deployed consumer lag monitoring with per-group alerting, broker storage management with automated retention review, and 24/7 response to partition leadership failures and consumer group incidents. Quarterly reviews include capacity planning for partition and storage growth based on trending data.

Conclusion

Kafka day-2 operations are primarily a monitoring and response discipline. The failure modes are deterministic once you understand the data model; the operational challenge is instrumenting the right metrics, setting thresholds that match your use case requirements, and responding to degradation before it escalates. Teams that invest in this operational foundation find Kafka reliable at enterprise scale; teams that don't encounter the same failure modes repeatedly without a systematic path to resolution.

What actually pages you

Day-two Kafka problems are rarely the ones the getting-started guide prepares you for. Four account for most incidents we are called into.

Under-replicated partitions that never recover

A broker restarts, partitions go under-replicated, and the count does not return to zero. The usual cause is that the replica cannot catch up within replica.lag.time.max.ms because it is competing for the same disk the leader is writing to, so it is repeatedly dropped from the ISR and re-added. The symptom looks like a broker problem and is almost always an IO contention problem. Check disk utilisation on the lagging broker before touching replication settings.

Consumer group rebalance storms

A consumer group that rebalances continuously, processing nothing between rebalances. The trigger is usually max.poll.interval.ms being shorter than the real worst-case processing time for a batch — one slow downstream call and the consumer is evicted, which triggers a rebalance, which delays everyone else, which evicts more consumers. It is self-sustaining once started. The fix is either a longer interval or a smaller max.poll.records, and the diagnosis is in the consumer logs rather than the broker's.

The disk that fills because retention is not what you think

retention.ms is evaluated per segment, and a segment is only eligible for deletion once it is closed. A low-throughput topic with a large segment.bytes can hold data far past its retention window simply because the active segment has not rolled. Teams discover this when a partition that should hold a day holds a month.

Unclean leader election, which is a data-loss switch

unclean.leader.election.enable decides whether an out-of-sync replica may become leader when no in-sync replica is available. Enabled, you keep availability and silently lose whatever the new leader never received. Disabled, the partition goes offline and waits. Both are defensible; what is not defensible is not having made the choice deliberately per topic, which is the common state.

The upgrade path is the real constraint

Kafka upgrades are gated by two properties that are easy to set once and then forget: inter.broker.protocol.version and log.message.format.version. A rolling upgrade moves the binaries first and the protocol version second, and the second step is the irreversible one — once brokers are speaking the newer protocol, rolling back means downgrading through a format the older binaries cannot read.

The practical consequence is that estates get stuck mid-upgrade: new binaries, old protocol version, indefinitely, because nobody wants to take the step that cannot be undone. That is a supported state and a reasonable place to pause, but it should be a decision with a date attached rather than a position you discover you have been in for two years.