Collect Cloudflare Workers Logs to Serverless Kafka

In this tutorial, we will show how to collect logs from Cloudflare Workers to Serverless Kafka.

Kafka Setup

Create a Kafka cluster and topic using Upstash Console or Upstash CLI. Copy the Webhook URL to be used in the next steps.

Project Setup

We will use Wrangler 2 for deployment, so please install (or upgrade) Wrangler 2.

Create a folder for your project and run wrangler init. Select js:

➜  quickstart-cloudflare-workers wrangler init
 ⛅️ wrangler 2.0.7
-------------------
Using npm as package manager.
✨ Created wrangler.toml
Would you like to use git to manage this Worker? (y/n)
✨ Initialized git repository
No package.json found. Would you like to create one? (y/n)
✨ Created package.json
Would you like to use TypeScript? (y/n)
Would you like to create a Worker at src/index.js? (y/n)
✨ Created src/index.js

The Code

Update src/index.js as below:

export default {
  async fetch(request, env, context) {
    context.waitUntil(postLog("some log from cf workers!"))
    return new Response("Hello World!");
  },
};

function postLog(data) {
  return fetch("REPLACE_UPSTASH_WEBHOOK_URL", {
    method: "POST",
    body: data,
  });
}

Copy/paste the webhook URL from Upstash Console. Webhook URL simply pushes everything posted to the Kafka. You can also use Upstash Kafka SDK or directly the REST API.

Test and Deploy

You can test the function locally with wrangler dev.

Deploy your function to Cloudflare with wrangler publish

The endpoint of the function will be printed. You can check if logs are collected in Kafka by copying the curl expression from the console:

curl https://definite-goldfish-14184-us1-rest-kafka.upstash.io/consume/GROUP_NAME/GROUP_INSTANCE_NAME/mytopic -H "Kafka-Auto-Offset-Reset: earliest" -u \
  REPLACE_HERE