Confluent Cluster types
Confluent Cloud offers these Kafka cluster types:
- Basic clusters - Used for experimentation, early development and basic use cases.
- Standard clusters - Used for production-ready features and functionality.
- Enterprise clusters - Used for production-ready functionality that requires private networking capabilities.
- Dedicated clusters - Used for critical production workloads with high traffic or private networking requirements.
- Freight clusters - Used for high-throughput, relaxed latency workloads that are less expensive than self-managed open source Kafka.
Cluster provisioning and scaling
Confluent uses billing units to provision and scale clusters.
eCKU - Elastic Confluent Unit for Kafka
Elastic scaling
Basic, Standard, Enterprise, and Freight clusters are elastic, shrinking and expanding automatically based on load. You don’t resize these clusters (unlike Dedicated clusters). When you need more capacity, your cluster expands up to the fixed ceiling. If you’re not using capacity above the minimum, you’re not paying for it. If you’re at zero capacity, you don’t pay for anything. For more information, see Elastic Confluent Unit for Kafka and eCKU/CKU comparison.
Freight clusters scaling considerations
Freight clusters are elastic, scaling automatically based on load, so you only pay for the eCKUs you use. However, Freight clusters may not scale as quickly as other eCKU clusters. In general, Freight clusters can support an additional 4 eCKU of capacity (240 MBps of ingress / 720 MBps of egress) every 10 minutes. If you workload grows faster than this, you may experience higher latency or failed requests. As your workload decreases, you pay for your actual eCKU usage.
In scenarios where you expect a large, rapid increase in traffic, consider contacting your account team to get Confluent to work with you to meet your request.
Fast scaling for Enterprise clusters
All Enterprise clusters support fast scaling up to 10 eCKUs, which is similar to how elastic scaling has worked in the past. Beyond 10 eCKUs, Enterprise clusters support on-demand scaling, which may be limited to a growth rate of approximately 20 minutes per eCKU.
Considerations
- To provision Enterprise clusters with a maximum of 32 eCKU on AWS, your cluster networking must use Private Network Interface (PNI).
- Enterprise clusters that use PrivateLink networking on AWS are limited to a maximum of 10 eCKU.
- If you have workloads that require fast scaling beyond 10 eCKU or workloads larger than 32 eCKU, reach out to Confluent to request to have your cluster enabled.
Manual scaling
Dedicated clusters are provisioned and billed in terms of Confluent Unit for Kafka (CKU). CKUs are a unit of horizontal scalability in Confluent Cloud that provide a preallocated amount of resources. How much you can ingest and stream per CKU depends on a variety of factors including client application design and partitioning strategy. For more information, see Monitor Dedicated Clusters in Confluent Cloud and Dedicated Cluster Performance and Expansion in Confluent Cloud.
Features
All clusters have the following features:
- Kafka ACLs
- Fully-managed replica placement
- User interface to manage consumer lag
- Topic management
- Fully-Managed Connectors
- View and consume Connect logs
- Stream Governance
- Stream Catalog
- Stream Lineage
- Encryption-at-rest
- TLS for data in transit
- Role-based Access Control (RBAC) (Basic clusters do not support RBAC roles for resources within the Kafka cluster)
Comparison
Feature comparison table
| Feature | Basic | Standard | Enterprise | Freight | Dedicated |
|---|---|---|---|---|---|
| Exactly Once Semantics | ✓ | ✓ | ✓ | ✓ | |
| Key based compacted storage | ✓ | ✓ | ✓ | ✓ | |
| Custom Connectors | ✓ | ✓ | ✓ | ||
| Flink | ✓ | ✓ | ✓ | ✓ | |
| ksqlDB | ✓ | ✓ | ✓ | ||
| Schema validation | ✓ | ||||
| Public networking | ✓ | ✓ | ✓ | ||
| Private networking | ✓ | ✓ | ✓ | ||
| OAuth | ✓ | ✓ | ✓ | ✓ | |
| Mutual TLS (mTLS) | ✓ | ||||
| Audit logs | ✓ | ✓ | ✓ | ✓ | |
| Self-managed encryption keys | ✓ | ✓ | |||
| Automatic Elastic scaling | ✓ | ✓ | ✓ | ✓ | |
| Stream Sharing | ✓ | ✓ | ✓ (*) | ||
| Client Quotas | ✓ | ||||
| Access Transparency | ✓ | ||||
| SLAs | 99.5% | 99.9%, 99.99% (requires 2 eCKU) | 99.9%, 99.99% (requires 2 eCKU) | 99.99% | 99.95% (SZ), 99.99% (MZ) |
* Stream sharing doesn’t support all private networking options.
eCKU/CKU comparison
| Dimension | Basic | Standard | Enterprise | Dedicated | Freight |
|---|---|---|---|---|---|
| Maximum eCKU/CKU | 50 | 10 | 32 | 152 | 152 |
| Fast scaling | n/a | n/a | 10 | n/a | n/a |
| Ingress (MBps) | 250 | 250 | 1,920 | 9,120 | 9,120 |
| Egress (MBps) | 750 | 750 | 5,760 | 27,360 | 27,360 |
| Partitions (pre-replication) | 1500 | 2500 | 96,000 | 100,000 | 50,000 |
| Number of partitions you can compact | 1500 | 2500 | 11,520 | 100,000 | None |
| Total client connections | 1000 | 10,000 | 576,000 | 2,736,000 | 2,736,000 |
| Connection attempts (per second) | 80 | 800 | 16,000 | 76,000 | 76,000 |
| Requests (per second) | 15,000 | 15,000 | 240,000 | 2,280,000 | 2,280,000 |
| Message size (MB) | 8 | 8 | 20 | 20 | 20 |
| Client version (minimum) | 0.11.0 | 0.11.0 | 0.11.0 | 0.11.0 | 0.11.0 |
| Request size (MB) | 100 | 100 | 100 | 100 | 100 |
| Fetch bytes (MB) | 55 | 55 | 55 | 55 | 55 |
| API keys | 50 | 100 | 500 | 2,000 | 500 |
| Partition creation and deletion (per five minute period) | 250 | 500 | 500 | 5,000 | 500 |
| Connector tasks per Kafka cluster | 250 | 250 | 250 | 250 | N/A |
| ACLs | 1,000 | 1,000 | 4,000 | 10,000 | 10,000 |
| Kafka REST Produce v3 - Max throughput (MBps): | 10 | 10 | 10 | 7,600 | 10 |
| Kafka REST Produce v3 - Max connection requests (per second): | 25 | 25 | 25 | 45,600 | 25 |
| Kafka REST Produce v3 - Max streamed requests (per second): | 1000 | 1000 | 1000 | 456,000 | 1000 |
| Kafka REST Produce v3 - Max message size for Kafka REST Produce API (MB): | 8 | 8 | 8 | 20 | 8 |
| Kafka REST Admin v3 - Max connection requests (per second): | 25 | 25 | 25 | 45,600 | 25 |
| Autoscaling | Yes | Yes | Yes | Yes | No – Programmatic scaling required |
| Use Cases | Start small with no risk on your test cases | Production-ready for most applications | Enhanced security with private networking | Customizable for use cases with distinct networking or throughput requirements | Cost-efficient for high scale use cases with relaxed latency requirements |
| Networking options | Internet | Internet | Private Network Interface, Private Link/PSC | Internet, Private Link/PSC, VPC/VNet Peering, AWS Transit Gateway | Private Network Interface (internal only link) |
| Starting from | ~$0/month | ~$385/month | ~$895/month | ~$1,950/month | ~$2,300/month |
- Min 2 eCKUs/CKUs required for 99.99% uptime SLA
- eCKU-based clusters scale to zero if no dimensions are being used – starting price assumes clusters are up on average ~70% of the time
- Higher cost driven by 2 eCKU minimum for Freight, which is targeted for large-scale logging and observability use-cases
Considerations
- To provision Enterprise clusters with a maximum of 32 eCKU on AWS, your cluster networking must use Private Network Interface (PNI).
- Enterprise clusters that use PrivateLink networking on AWS are limited to a maximum of 10 eCKU.
- Basic, Standard, Enterprise, and Freight cluster limits are based on maximum eCKU for the cluster type. For more information, see Elastic Confluent Unit for Kafka.
- Dedicated Kafka cluster limits are based on 152 CKU. For more information, see CKU purchase limits and Confluent Unit for Kafka.
- For connector tasks per cluster, Basic clusters are limited to one task per connector. You can deploy 250 connectors to a Basic cluster but each connector can only have one task. If you need more than one task, upgrade your cluster.
- Enterprise clusters support fast scaling up to 10 eCKUs and on-demand scaling beyond 10 eCKU. On-demand scaling may be limited to a growth rate of approximately 20 minutes per eCKU. For more information, see Fast scaling for Enterprise clusters.
- The maximum limits for requests, total client connections, and connection attempts are currently not strictly enforced for Basic, Standard, Enterprise, and Freight clusters, whether you are using a cluster with the default capacity, or a maximum capacity you configured.
- You can lower maximum eCKU to reduce the capacity of your cluster but you must also manage your workload, especially for limits not currently strictly enforced. Confluent will contact you if your clusters with managed maximum eCKU repeatedly exceed the user-defined reduced capacity for requests, total client connections, and connection attempts. You should be prepared to discuss mitigation strategies, such as reduction of workload to stay within the limits of your reduced capacity, or an increase to the maximum eCKU configuration to accommodate the workload.
Configurations
- You cannot edit cluster settings on Confluent Cloud on Basic, Standard, Enterprise, and Freight clusters, but many configuration settings are available at the topic level instead.
- You can change some configuration settings on Dedicated clusters using the Confluent CLI or REST API.
- The default maximum timeout for registered consumers is different for Confluent Cloud Kafka clusters than for Confluent Platform clusters and cannot be changed.
group.max.session.timeout.msdefault is 1200000 ms (20 minutes)
The following table lists editable cluster settings for Dedicated clusters and their default parameter values.
| Parameter Name | Default | Editable | More Info |
|---|---|---|---|
| auto.create.topics.enable | false | Yes | |
| ssl.enabled.protocols | TLSv1.2 | Yes | Options: TLSv1.2, TLSv1.3, or both. |
| ssl.cipher.suites | “” | Yes | |
| num.partitions | 6 | Yes | Limits vary, see: Kafka Cluster Types in Confluent Cloud |
| log.cleaner.max.compaction.lag.ms | 9223372036854775807 | Yes | Min: 21600000 ms |
| log.retention.ms | 604800000 (168 hours = 7 days) | Yes | Set to -1 for Infinite Storage |
Manage Kafka Cluster Configuration Settings in Confluent Cloud | Confluent Documentation
Links
Kafka Cluster Types in Confluent Cloud | Confluent Documentation
Networking Comparison - Networking on Confluent Cloud | Confluent Documentation