You've enabled idempotent producers and wrapped your writes in Kafka transactions. Your producer retries are safe. But your consumer? It's still processing the same message twice after a crash mid-handling. Exactly-once semantics don't stop at the broker — they end at your business logic.
The Gap Between Producer and Consumer Guarantees
Kafka's transactional API ensures atomic writes across partitions. The consumer side, however, only guarantees at-least-once delivery by default. If your service crashes after updating the database but before committing the offset, the rebalance replays the message. You get duplicate orders, double charges, or corrupted state.
"Idempotency is not a broker feature. It's a contract your consumer honors.
— Confluent Engineering Blog
Pattern: Deterministic ID + Fast Deduplication Store
Generate a stable idempotency key per message. Hash the message key, payload, and partition — not the offset, which changes on replay. Check Redis (or a local LRU with TTL) before executing business logic. Only commit the offset after the key is stored and the transaction succeeds.
💡
TipUse Redis SET NX with TTL. It's atomic, fast, and self-expiring. No cleanup jobs needed.
Integrating with Spring Kafka Transactions
Spring's @Transactional on the listener method wraps the entire processing in a database transaction. But Kafka offset commits happen outside that transaction by default. You need KafkaTransactionManager chained with your DataSourceTransactionManager via ChainedTransactionManager — or use the outbox pattern.
⚠️
WarningManual ack mode + chained transactions = complex failure modes. Prefer the outbox pattern for critical paths.
When to Use Kafka Transactions on the Consumer
Consumer-side transactions (read-process-write in one atomic unit) only work if your output is also Kafka. Writing to a database and Kafka atomically requires the transactional outbox pattern: write events to an outbox table in the same DB transaction as your business logic, then publish via Kafka Connect or a relay.
| Approach | Complexity | Latency | Use When |
|---|---|---|---|
| Deduplication Key + Redis | Low | < 1ms | Most services, external APIs, non-Kafka sinks |
| Chained TX Manager | Medium | DB roundtrip | Simple DB writes, Spring Boot, low throughput |
| Transactional Outbox | High | Async (ms-sec) | Critical consistency, microservices, audit trails |
Checklist Before You Ship
- Idempotency key includes partition — not offset
- Dedupe store TTL > max reprocessing window (7 days default)
- Offset committed after business logic + key storage
- Duplicate path commits offset without reprocessing
- Metrics on duplicate rate, dedupe hits, processing latency
✦
ℹ️
NoteStart with deterministic keys + Redis. It solves 90% of duplicate cases with 10% of the complexity. Add transactions only when your sink demands it.