Confluent Cloud Connect
Connect to External Services
- Overview
- Install Connectors
- ActiveMQ Source
- AlloyDB Sink
- Amazon CloudWatch Logs Source
- Amazon CloudWatch Metrics Sink
- Amazon DynamoDB CDC Source
- Amazon DynamoDB Sink
- Amazon Kinesis Source
- Amazon Redshift Sink
- Amazon S3 Sink
- Amazon S3 Source
- Amazon SQS Source
- AWS Lambda Sink
- Azure Blob Storage Sink
- Azure Blob Storage Source
- Azure Cognitive Search Sink
- Azure Cosmos DB Sink
- Azure Cosmos DB Sink V2
- Azure Cosmos DB Source
- Azure Cosmos DB Source V2
- Azure Data Lake Storage Gen2 Sink
- Azure Event Hubs Source
- Azure Functions Sink
- Azure Log Analytics Sink
- Azure Service Bus Source
- Azure Synapse Analytics Sink
- ClickHouse Sink
- Couchbase Source
- Couchbase Sink
- Databricks Delta Lake Sink
- Datadog Metrics Sink
- Datagen Source (development and testing)
- Elasticsearch Service Sink
- Elasticsearch Sink V2 Connector for Confluent Cloud Quick Start | Confluent Documentation
- GitHub Source
- Google BigQuery Sink [Deprecated]
- Google BigQuery Sink V2
- Google Cloud BigTable Sink
- Google Cloud Dataproc Sink [Deprecated]
- Google Cloud Functions Gen 2 Sink
- Google Cloud Functions Sink [Deprecated]
- Google Cloud Pub/Sub Source
- Google Cloud Spanner Sink
- Google Cloud Storage Sink
- Google Cloud Storage Source
- HTTP Sink
- HTTP Sink V2
- HTTP Source
- HTTP Source V2
- IBM MQ Source
- InfluxDB 2 Sink
- InfluxDB 2 Source
- InfluxDB 3 Sink
- Jira Source
- MariaDB CDC Source
- Microsoft SQL Server CDC Source (Debezium) [Deprecated]
- Microsoft SQL Server CDC Source V2 (Debezium)
- Microsoft SQL Server Sink (JDBC)
- Microsoft SQL Server Source (JDBC)
- MongoDB Atlas Sink
- MongoDB Atlas Source
- MQTT Sink
- MQTT Source
- MySQL CDC Source (Debezium) [Deprecated]
- MySQL CDC Source V2 (Debezium)
- MySQL Sink (JDBC)
- MySQL Source (JDBC)
- Neo4j Sink
- New Relic Metrics Sink
- OpenSearch Sink
- Oracle Database Sink (JDBC)
- Oracle Database Source (JDBC)
- PagerDuty Sink [Deprecated]
- Pinecone Sink
- PostgreSQL CDC Source (Debezium) [Deprecated]
- PostgreSQL CDC Source V2 (Debezium)
- PostgreSQL Sink (JDBC)
- PostgreSQL Source (JDBC)
- RabbitMQ Sink
- RabbitMQ Source
- Redis Sink
- Redis Kafka Sink
- Redis Kafka Source
- Salesforce Bulk API 2.0 Sink
- Salesforce Bulk API 2.0 Source
- Salesforce Bulk API Source
- Salesforce CDC Source
- Salesforce Platform Event Sink
- Salesforce Platform Event Source
- Salesforce PushTopic Source
- Salesforce SObject Sink
- ServiceNow Sink
- ServiceNow Source [Legacy]
- ServiceNow Source V2
- SFTP Sink
- SFTP Source
- Snowflake Sink
- Snowflake Source
- Solace Sink
- Splunk Sink
- Zendesk Source
- Confluent Hub
- Install Custom Plugins and Custom Connectors
- Manage CSFLE
- Manage Provider Integration
- Networking and DNS
- Overview
- AWS Egress PrivateLink Endpoints for First-Party Services
- AWS Egress PrivateLink Endpoints for Self-Managed Services
- AWS Egress PrivateLink Endpoints for Amazon RDS
- Azure Egress Private Link Endpoints for First-Party Services
- Azure Egress Private Link Endpoints for Self-Managed Services
- Google Cloud Private Service Connect Endpoints for First-Party Services
- Connect API Usage
- Manage Public Egress IP Addresses
- Sample Connector Output
- Configure Single Message Transforms
- Configure Custom SMTs
- View Connector Events
- Interpret Connector Statuses
- Manage Service Accounts
- Configure RBAC
- View Errors in the Dead Letter Queue
- Connector Limits
- Manage Offsets
- Transforms List
- Overview
- Cast
- Drop
- DropHeaders
- EventRouter
- ExtractField
- ExtractTopic
- Filter (Kafka)
- Filter (Confluent)
- Flatten (Kafka)
- Flatten (Confluent)
- FromXML
- GzipDecompress
- HeaderFrom
- HeaderToValue
- HoistField
- InsertField
- InsertHeader
- KeyToValue
- MaskField
- MessageTimestampRouter
- RegexRouter
- ReplaceField (Kafka)
- ReplaceField (Confluent)
- SetSchemaMetadata
- TimestampConverter
- TimestampRouter
- TombstoneHandler
- TopicRegexRouter
- ValueToKey
- Reference
Elasticsearch Sink V2 Connector for Confluent Cloud
What V2 unlocks?
Elasticsearch V2 removes version and deployment constraints that existed in V1:
- Elasticsearch 9.x Support: Compatible with Elasticsearch 9.x servers.
- Elasticsearch Serverless Support: Supports integration with Elasticsearch Serverless deployments.
- Topic-to-Resource Mapping: Allows flexible mapping of Kafka topics to Elasticsearch indexes or data streams.
- API Key Authentication: Supports secure authentication using Elasticsearch API keys.
- Alias Support: Enables writing to Elasticsearch index aliases for seamless index rollover and management.
Other Features of Elasticsearch V2 Sink
- Exactly-once Delivery: Guarantees records are delivered exactly once to Elasticsearch indexes or data streams, preventing duplicates.
- Private Networking: Supports private connectivity to Elasticsearch clusters across AWS, Azure, and GCP.
- Client-side Field-Level Encryption (CSFLE): Enables encryption of sensitive fields before data leaves Kafka.
- Input Data Formats: Supports AVRO, JSON_SR, PROTOBUF, and JSON.
- Offset Management: Provides robust offset management for reliable and predictable data delivery.
- Fail-fast Validations: Performs early validation of configuration and connectivity to catch issues before runtime.
Elasticsearch Sink V2 Connector for Confluent Cloud Quick Start | Confluent Documentation
Datagen Source Connector
The fully-managed Confluent Cloud Datagen Source connector for Confluent Cloud is used to generate mock data for development and testing. The connector supports Avro, JSON Schema, Protobuf, and JSON (schemaless) output formats.
max.interval - Set the maximum interval (in milliseconds) between each message.
- Type: int
- Default: 1000 (1 message per second)
- Valid Values: 10,… for non-dedicated clusters and 5,… for dedicated clusters
- Importance: high
If you are on a non-dedicated cluster where the minimum interval is capped at 10ms (100 msg/sec), you will need 10 tasks to reach a total of 1000 msg/sec.
- To hit 10,000 msg/sec, you will need 100 tasks.
Datagen Source Connector for Confluent Cloud Quick Start | Confluent Documentation