Kafka vs Kinesis: Shards, Partitions, Retention, MSK and When to Choose Each

The short answer

Kinesis Data Streams is an AWS service; Apache Kafka is open-source software. Both store ordered streams of records that several applications can read independently and replay. Kinesis splits a stream into shards, keeps data for 24 hours by default and up to 365 days, and has no servers for you to run, but it uses its own API and lives only in AWS. Kafka splits topics into partitions, keeps data for as long as you configure, and brings a large open-source ecosystem, Kafka Connect and Kafka Streams among it, that runs on any infrastructure. Amazon MSK sits between the two: real Apache Kafka, operated by AWS. AWS's own guidance is short: if you are new to streaming, use Kinesis; if you already have Kafka applications, or prefer open source, use MSK.

Kafka vs Kinesis vs MSK at a glance

How Kinesis Data Streams works

A Kinesis data stream is a set of shards. Every record carries a partition key, which Kinesis hashes with MD5 to pick a shard, and a sequence number that Kinesis assigns on write. Records with the same partition key go to the same shard, so ordering holds per key, much as it does per partition in Kafka. AWS says the delay between putting a record and being able to read it is typically under a second.

Capacity comes in three modes. In Provisioned mode you choose the shard count, and each shard takes up to 1 MB per second or 1,000 records per second of writes and serves up to 2 MB per second of reads. On-demand Standard manages shards for you and accommodates up to double the peak write throughput of the previous 30 days. On-demand Advantage is an account-level mode that lets you pre-warm write capacity and register more enhanced fan-out consumers. Those are AWS's published quotas, not benchmarks.

Consumers either share a shard's read capacity through GetRecords or register for enhanced fan-out, where Kinesis pushes records to each consumer with up to 2 MB per second per shard of dedicated throughput. A stream can have 20 enhanced fan-out consumers, or 50 in On-demand Advantage. Most consumers use the Kinesis Client Library, which runs a record processor per shard and stores its checkpoints in DynamoDB tables. Retention starts at 24 hours and can be raised to 8,760 hours, or 365 days. Kinesis documents that producer and consumer retries can deliver a record more than once, so consumers have to tolerate duplicates.

How Kafka differs

Kafka's unit is the partition. Producers append records to partitions of a topic, records with the same key land in the same partition, and consumers in a group divide the partitions between them and record their offsets in Kafka itself. Retention is set per topic by time or size, log compaction can keep the latest value for every key with no time limit, and tiered storage can move completed segments to object storage. Since Kafka 4.0 the cluster runs in KRaft mode only, and Kafka 4.2 made share groups production-ready for queue-style consumption with per-record acknowledgement. Our Kafka vs RabbitMQ post explains the model in more depth.

Two things in Kafka have no direct Kinesis equivalent. Transactions let a Kafka Streams application read from one topic and write to another with exactly-once processing. Kafka Connect and Kafka Streams are part of the Apache project, and a large set of third-party tools speak the Kafka protocol. Kinesis has its own integrations instead: AWS lists Firehose, Kinesis Video Streams and Managed Service for Apache Flink as part of the same platform.

Where Amazon MSK fits

Amazon MSK runs open-source Apache Kafka, so existing Kafka clients and tools work against it unchanged. MSK Provisioned offers Standard brokers and Express brokers; Express brokers manage storage for you, but AWS notes they do not yet fully support the Kafka Streams API and do not yet support KIP-932 share groups. MSK Serverless scales capacity automatically and requires IAM access control rather than Kafka ACLs. MSK Connect runs Kafka Connect connectors, and MSK Replicator copies data between clusters. MSK supports Kafka versions up to 4.2.x, with 3.9.x listed as recommended; our post on MSK version end of support covers the upgrade calendar.

MSK removes broker operations but keeps you on AWS's version schedule. Self-managed Kafka keeps the schedule yours and moves the operations, and the patching, back in-house.

Which should you choose, Kafka or Kinesis?

• Choose Kinesis Data Streams when everything lives in AWS, the team is new to streaming, you want no clusters or partitions to plan, and 365 days of retention is enough. Kinesis is the simpler service to start with, and AWS recommends it for that case.
• Choose Amazon MSK when you want Kafka's API, clients and ecosystem on AWS without running brokers yourself, or you are moving existing Kafka applications to AWS.
• Choose self-managed Apache Kafka when you run in more than one cloud or on premises, want control over versions and upgrade timing, need retention or compaction beyond what Kinesis offers, or want to avoid tying your event backbone to one provider's API.
• Do not switch only for a small difference in features. Kinesis and Kafka do not share an API, so moving means changing producers, consumers and monitoring.

Where OSSeva fits

OSSeva does not support Kinesis Data Streams or operate MSK; AWS runs both. OSSeva supports Apache Kafka you run yourself, on EC2, EKS, other clouds or your own hardware. OSSeva for Apache Kafka ships patched builds from 2.8 through 3.9, including 3.8 and 3.9 after upstream archived them, and covers 4.x in KRaft mode, so a team leaving MSK's version schedule, or staying on 3.x while it plans the move to 4.x, has a supported path. See Kafka and ZooKeeper extended support. Kafka sits under one contract with your databases and other messaging, priced per cluster, not per GB of throughput. Book a discovery call for a quote. For the wider field, see our list of Kafka support providers.

Frequently asked questions

Kafka vs Kinesis vs SQS: which should I use?

SQS is a queue: each message is processed by one consumer, acknowledged or retried after a visibility timeout, and deleted. Standard queues deliver at least once, FIFO queues give exactly-once processing, and messages are kept for four days by default and fourteen at most. Kinesis and Kafka are streams: records stay for the retention period, keep their order per key, and several applications read the same data. AWS recommends Kinesis over SQS when you need ordering, routing by key, several consumers or replay, and SQS when you need per-message acknowledgement and delays.

Is Kinesis Data Streams the same as Kafka?

No. Kinesis Data Streams is a separate AWS service with its own API and Kinesis Client Library. The concepts are similar: shards correspond roughly to partitions and sequence numbers to offsets. Kafka clients cannot write to a Kinesis stream; for Kafka on AWS, use Amazon MSK or run Kafka yourself.

MSK vs Kinesis: which does AWS recommend?

AWS's Kinesis FAQ recommends Kinesis Data Streams if you are new to streaming, and MSK if you already run Kafka applications or prefer open-source technology, since MSK and MSK Connect are compatible with open-source Kafka and Kafka Connect.

Kafka vs Kinesis latency: which is faster?

AWS documents Kinesis put-to-get delay as typically under one second, and enhanced fan-out pushes records to consumers rather than waiting for them to poll. Kafka latency depends on batching, acknowledgement settings, replication and client configuration. Test with your own record sizes and durability settings rather than relying on published benchmarks.

Is Kinesis cheaper than Kafka?

The pricing models differ. AWS bills provisioned Kinesis streams per shard hour and on-demand streams per GB written and read; MSK Serverless uses throughput-based pricing; self-managed Kafka costs whatever infrastructure and people you put behind it. Which is cheaper depends on volume, retention and how many consumers read each stream. Model your own workload against the current AWS pricing pages.

How long can Kinesis keep data compared with Kafka?

Kinesis keeps records for 24 hours by default and up to 365 days. Kafka retention is set per topic with no fixed ceiling, compacted topics keep the latest value per key indefinitely, and tiered storage moves older segments to object storage.