Upstash
·4 min read

Get Started with Sidekiq and Upstash Redis on Fly.io

Burak YılmazBurak YılmazSite Reliability Engineer @Upstash

Sidekiq is a popular background processing library in Ruby language. With this library, we can distribute our processes to what is called Sidekiq workers instead of doing the heavy work on the server itself.

What these workers do basically is they fetch the enqueued or scheduled tasks from the database. Then, execute them whenever the time comes and the worker is idle.

This way, we can make sure that our server remains responsive for incoming requests, without being blocked by heavy or recurring processes.

We will create a sample app using Sidekiq on Fly.io using Upstash Redis. It will enqueue and/or schedule some notification for users, depending on the inputs provided.

schema

Fly.io Setup

  • Create a Fly.io account.
  • Install flyctl and run flyctl auth login

Upstash Redis Setup on Fly

  • Run flyctl redis create.
  • Set your connection link (after the .toml creation): flyctl secrets set UPSTASH_REDIS_LINK=<upstash-redis-connection-link>

Application Code

  • bundle init and bundle add sidekiq sinatra to initiate Gemfile and install dependencies. `
  • Create sendEmail.rb file:
require "sidekiq"
require "sidekiq/api"
 
connection_url = ENV['UPSTASH_REDIS_LINK']
 
Sidekiq.configure_client do |config|
    config.redis = {url: connection_url}
end
 
Sidekiq.configure_server do |config|
    config.redis = {url: connection_url}
end
 
class EmailService
    include Sidekiq::Worker
    def perform(id, type)
        # Logic goes here. Let's assume sending email by printing to console.
        puts "Emailed to: " +  id + ": " + "'Congrats on " + type + " plan.'"
    end
end
 
def updateEmail(id, newType)
    jobFound = false
 
    a = Sidekiq::ScheduledSet.new
    a.each do |job|
        if job.args[0] == id
            job.delete
            jobFound = true
        end
    end
 
    if jobFound
        EmailService.perform_async(id, ("starting using our service and upgrading it to " + newType))
    else
        EmailService.perform_async(id, ("upgrading to " + newType))
    end
end
 
def sendEmail(id, type)
    case type
    when "free"
        # if free, delay for 10 seconds.
        EmailService.perform_in("10", id, "free")
    when "paid"
        # if paid, delay for 5 seconds.
        EmailService.perform_in("5", id, "paid")
    when "enterprise"
        # if enterprise, immediately queue.
        EmailService.perform_async(id, "enterprise")
    when "enterprise10k"
        EmailService.perform_async(id, "enterprise10k")
    else
        puts "Only plans are: `free`, `paid` and `enterprise`"
    end
end
 
def clearSchedules()
    Sidekiq::ScheduledSet.new.clear
    Sidekiq::Queue.new.clear
end
require "sidekiq"
require "sidekiq/api"
 
connection_url = ENV['UPSTASH_REDIS_LINK']
 
Sidekiq.configure_client do |config|
    config.redis = {url: connection_url}
end
 
Sidekiq.configure_server do |config|
    config.redis = {url: connection_url}
end
 
class EmailService
    include Sidekiq::Worker
    def perform(id, type)
        # Logic goes here. Let's assume sending email by printing to console.
        puts "Emailed to: " +  id + ": " + "'Congrats on " + type + " plan.'"
    end
end
 
def updateEmail(id, newType)
    jobFound = false
 
    a = Sidekiq::ScheduledSet.new
    a.each do |job|
        if job.args[0] == id
            job.delete
            jobFound = true
        end
    end
 
    if jobFound
        EmailService.perform_async(id, ("starting using our service and upgrading it to " + newType))
    else
        EmailService.perform_async(id, ("upgrading to " + newType))
    end
end
 
def sendEmail(id, type)
    case type
    when "free"
        # if free, delay for 10 seconds.
        EmailService.perform_in("10", id, "free")
    when "paid"
        # if paid, delay for 5 seconds.
        EmailService.perform_in("5", id, "paid")
    when "enterprise"
        # if enterprise, immediately queue.
        EmailService.perform_async(id, "enterprise")
    when "enterprise10k"
        EmailService.perform_async(id, "enterprise10k")
    else
        puts "Only plans are: `free`, `paid` and `enterprise`"
    end
end
 
def clearSchedules()
    Sidekiq::ScheduledSet.new.clear
    Sidekiq::Queue.new.clear
end
  • Create a simple Sinatra server app.rb:
#!/usr/bin/env ruby
 
require 'rubygems'
require 'bundler/setup'
require 'sinatra'
require './sendEmail.rb'
 
get '/register/:id/:plan' do
  sendEmail(params[:id], params[:plan])
  "<h1>Email is queued.</h1>"
end
 
get '/update/:id/:plan' do
  updateEmail(params[:id], params[:plan])
  "<h1>Updated Email is queued.</h1>"
end
 
get '/clear' do
  clearSchedules()
  "<h1>Queue and the schedules are cleared.</h1>"
end
#!/usr/bin/env ruby
 
require 'rubygems'
require 'bundler/setup'
require 'sinatra'
require './sendEmail.rb'
 
get '/register/:id/:plan' do
  sendEmail(params[:id], params[:plan])
  "<h1>Email is queued.</h1>"
end
 
get '/update/:id/:plan' do
  updateEmail(params[:id], params[:plan])
  "<h1>Updated Email is queued.</h1>"
end
 
get '/clear' do
  clearSchedules()
  "<h1>Queue and the schedules are cleared.</h1>"
end
  • Create config.ru file:
# config.ru
 
require './app.rb'
require 'sidekiq'
require "sidekiq/api"
 
run Sinatra::Application
# config.ru
 
require './app.rb'
require 'sidekiq'
require "sidekiq/api"
 
run Sinatra::Application

Launch App

  • First, run flyctl launch to create .toml template.
  • There, update internal_port = 9292 under [[services]]. It should look something like this:
app = "sidekiq-example"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
 
[build]
  builder = "heroku/buildpacks:20"
 
[env]
  PORT = "8080"
 
[[services]]
  internal_port = 9292
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"
 
  [[services.ports]]
    handlers = ["http"]
    port = 80
 
  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443
app = "sidekiq-example"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
 
[build]
  builder = "heroku/buildpacks:20"
 
[env]
  PORT = "8080"
 
[[services]]
  internal_port = 9292
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"
 
  [[services.ports]]
    handlers = ["http"]
    port = 80
 
  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443
  • Create a file named Procfile so that we can run both the Sinatra server and the Sidekiq Worker:
web: bundle exec rackup & bundle exec sidekiq -r ./sendEmail.rb
web: bundle exec rackup & bundle exec sidekiq -r ./sendEmail.rb
  • run flyctl deploy to deploy.

App Deployed

  • send get register/id1/enterprise via curl <fly-domain>/register/:id/:plan enterprise

  • send get register/id2/paid via curl <fly-domain>/register/:id/:plan paid

  • send get register/id3/free and then get update/id3/paid via curl <fly-domain>/register/:id/:plan and curl <fly-domain>/update/:id/:newPlan register-update

Closing Words

In this example, we have implemented an app, using Sidekiq background processing library. In the sample, we have demonstrated a simple usecase which you can customize depending on your needs. This way, we have decreased the workload of our main server and with that enabled for more responsive experience for users.

For the complete source code, you can look at our repo.

You can give your feedbacks on any of our services via twitter or discord, which will be greatly appreciated.

sidekiqflyflyctlworkerbackgroundupstash
Previous post

Using LaunchDarkly with Redis

Next post

Announcing Vercel Integration v2