Console

·

4 min read

Get Started with Sidekiq and Upstash Redis on Fly.io

Burak Yılmaz

Site Reliability Engineer @Upstash

Fly.io Setup
Upstash Redis Setup on Fly
Application Code
Launch App
App Deployed
Closing Words

Sidekiq is a popular background processing library in Ruby language. With this library, we can distribute our processes to what is called Sidekiq workers instead of doing the heavy work on the server itself.

What these workers do basically is they fetch the enqueued or scheduled tasks from the database. Then, execute them whenever the time comes and the worker is idle.

This way, we can make sure that our server remains responsive for incoming requests, without being blocked by heavy or recurring processes.

We will create a sample app using Sidekiq on Fly.io using Upstash Redis. It will enqueue and/or schedule some notification for users, depending on the inputs provided.

schema

Fly.io Setup

  • Create a Fly.io account.
  • Install flyctl and run flyctl auth login

Upstash Redis Setup on Fly

  • Run flyctl redis create.
  • Set your connection link (after the .toml creation): flyctl secrets set UPSTASH_REDIS_LINK=<upstash-redis-connection-link>

Application Code

  • bundle init and bundle add sidekiq sinatra to initiate Gemfile and install dependencies. `
  • Create sendEmail.rb file:
require "sidekiq"
require "sidekiq/api"

connection_url = ENV['UPSTASH_REDIS_LINK']

Sidekiq.configure_client do |config|
    config.redis = {url: connection_url}
end

Sidekiq.configure_server do |config|
    config.redis = {url: connection_url}
end

class EmailService
    include Sidekiq::Worker
    def perform(id, type)
        # Logic goes here. Let's assume sending email by printing to console.
        puts "Emailed to: " +  id + ": " + "'Congrats on " + type + " plan.'"
    end
end

def updateEmail(id, newType)
    jobFound = false
    
    a = Sidekiq::ScheduledSet.new
    a.each do |job|
        if job.args[0] == id
            job.delete
            jobFound = true
        end
    end

    if jobFound
        EmailService.perform_async(id, ("starting using our service and upgrading it to " + newType))
    else
        EmailService.perform_async(id, ("upgrading to " + newType))
    end
end

def sendEmail(id, type)
    case type
    when "free"
        # if free, delay for 10 seconds.
        EmailService.perform_in("10", id, "free")
    when "paid"
        # if paid, delay for 5 seconds.
        EmailService.perform_in("5", id, "paid")
    when "enterprise"
        # if enterprise, immediately queue.
        EmailService.perform_async(id, "enterprise")
    when "enterprise10k"
        EmailService.perform_async(id, "enterprise10k")
    else
        puts "Only plans are: `free`, `paid` and `enterprise`"
    end
end

def clearSchedules()
    Sidekiq::ScheduledSet.new.clear
    Sidekiq::Queue.new.clear
end
  • Create a simple Sinatra server app.rb:
#!/usr/bin/env ruby

require 'rubygems'
require 'bundler/setup'
require 'sinatra'
require './sendEmail.rb'

get '/register/:id/:plan' do
  sendEmail(params[:id], params[:plan])
  "<h1>Email is queued.</h1>"
end

get '/update/:id/:plan' do
  updateEmail(params[:id], params[:plan])
  "<h1>Updated Email is queued.</h1>"
end

get '/clear' do
  clearSchedules()
  "<h1>Queue and the schedules are cleared.</h1>"
end
  • Create config.ru file:
# config.ru

require './app.rb'
require 'sidekiq'
require "sidekiq/api"

run Sinatra::Application

Launch App

  • First, run flyctl launch to create .toml template.
  • There, update internal_port = 9292 under [[services]]. It should look something like this:
app = "sidekiq-example"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  builder = "heroku/buildpacks:20"

[env]
  PORT = "8080"

[[services]]
  internal_port = 9292
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443
  • Create a file named Procfile so that we can run both the Sinatra server and the Sidekiq Worker:
web: bundle exec rackup & bundle exec sidekiq -r ./sendEmail.rb
  • run flyctl deploy to deploy.

App Deployed

  • send get register/id1/enterprise via curl <fly-domain>/register/:id/:plan enterprise

  • send get register/id2/paid via curl <fly-domain>/register/:id/:plan paid

  • send get register/id3/free and then get update/id3/paid via curl <fly-domain>/register/:id/:plan and curl <fly-domain>/update/:id/:newPlan register-update

Closing Words

In this example, we have implemented an app, using Sidekiq background processing library. In the sample, we have demonstrated a simple usecase which you can customize depending on your needs. This way, we have decreased the workload of our main server and with that enabled for more responsive experience for users.

For the complete source code, you can look at our repo.

You can give your feedbacks on any of our services via twitter or discord, which will be greatly appreciated.



© 2022 Upstash, Inc. Based in California.

* Redis is a trademark of Redis Labs Ltd. Any rights therein are reserved to Redis Ltd. Any use by Upstash is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and Upstash.

** Cloudflare, the Cloudflare logo, and Cloudflare Workers are trademarks and/or registered trademarks of Cloudflare, Inc. in the United States and other jurisdictions.