Upstash Kafka with Decodable
This tutorial shows how to integrate Upstash Kafka with Decodable
Decodable is a platform which enables developers to build data pipelines using SQL. It is built on Apache Flink under the hood to provide a seamless experience, while abstracting away the underlying complexity. In this post, we will show how to connect an Upstash Kafka topic to Decodable to streamline messages from Kafka to Decodable.
Upstash Kafka Setup
Create a Kafka cluster using Upstash Console or Upstash CLI by following Getting Started.
Decodable Setup
Just like Upstash, Decodable is a managed service that means you do not need to host or provision anything. You can easily register for free and start using it.
After creating your account, click on Connections
and New Connection
. Select
Apache Kafka
. Then:
- Select Source as connection type.
- Select SASL_SSL as security protocol and SCRAM-SHA-256 as SASL mechanism.
- Enter your topic, SASL username, SASL password. You can find all those from Upstash console.
- Value format should be JSON.
In the next step, click on New Stream
and give a name to it.
In the schema screen, add country
, city
, region
and url
with string
type.
Give a name to your connection and click Create Connection
. In the next screen
click on Start.
Test the Setup
Now, let’s some events to our Kafka topic. Go to Upstash console, click on your
cluster then Topics
, click mytopic
. Select Messages
tab then click
Produce a new message
. Send a message in JSON format like the below:
{
"country": "US",
"city": "San Jose",
"region": "CA",
"url": "https://upstash.com"
}
Now, go back to Decodable console, click Streams and select the one you have
created. Then click Run Preview
. You should see something like:
Links
Was this page helpful?