Ruby

Re-Implementing Sinatra: Params, Status Code, Headers, ERB and Automated Testing

Forrest Ye

May 7, 2019 • 26 min read

This post continues the quest we started in a previous post: to re-implement our own almost-compatible Sinatra clone. If you haven't read it, I suggest you at least give a quick skim.

In this post, we will tackle:

supporting POST, PATCH, PUT and DELETE HTTP methods
parsing query params and named parameters
specifying response status code and response headers
HTTP redirection
using ERB as the template engine
custom-built automated testing

In the end of this post our mini framework will be able to handle most HTTP methods, parsing named parameters like get '/articles/:id', specify custom status code and headers, handle redirection, rendering HTML with some erb nicety, and have our own mini automated test set.

So let's begin.

Graceful Exiting with Signal Handlers

At the end of our previous post, we reached at a bare-minimum Sinatra implementation like this:

# fake_sinatra.rb
require 'webrick'

class FakeSinatra
  def self.get_instance(server=nil)
    @instance ||= new
  end

  def service(req, res)
    res.body = "#{@routes[req.path].call} - served by FakeSinatra"
  end

  def register(path, block)
    @routes ||= {}
    @routes[path] = block
  end

  module Helpers
    def get(path, &block)
      FakeSinatra.get_instance.register(path, block)
    end
    # TODO: implement more HTTP methods
  end
end

# so the user can call `get` etc
include FakeSinatra::Helpers

at_exit {
  server = WEBrick::HTTPServer.new(Port: 4567)
  server.mount('/', FakeSinatra)
  server.start
}

It allow us to register GET requests and will return plaintext responses. Before we start to add functionalities, there is a small annoying issue we want to fix: when we hit CTRL-C to kill the server, it displays some annoying stack trace messages:

➜  mini-sinatra git:(master) ruby app.rb
[2019-05-06 15:27:37] INFO  WEBrick 1.4.2
[2019-05-06 15:27:37] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
[2019-05-06 15:27:37] INFO  WEBrick::HTTPServer#start: pid=9646 port=4567
^C[2019-05-06 15:27:38] FATAL Interrupt:
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:170:in `select'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:170:in `block in start'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:32:in `start'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:157:in `start'
	/Users/forresty/workspace/theseus/mini-sinatra/fake_sinatra.rb:32:in `block in <top (required)>'
[2019-05-06 15:27:38] INFO  going to shutdown ...
[2019-05-06 15:27:38] INFO  WEBrick::HTTPServer#start done.
Traceback (most recent call last):
	4: from /Users/forresty/workspace/theseus/mini-sinatra/fake_sinatra.rb:32:in `block in <top (required)>'
	3: from /Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:157:in `start'
	2: from /Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:32:in `start'
	1: from /Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:170:in `block in start'
/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:170:in `select': Interrupt

Not very elegant. Since we love Ruby and its beauty, we want to make sure our Sinatra clone can exit gracefully as well, like the original Sinatra did:

➜  mini-sinatra git:(master) bundle exec ruby app.rb
[2019-05-06 15:33:48] INFO  WEBrick 1.4.2
[2019-05-06 15:33:48] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
== Sinatra (v2.0.5) has taken the stage on 4567 for development with backup from WEBrick
[2019-05-06 15:33:48] INFO  WEBrick::HTTPServer#start: pid=9922 port=4567
^C== Sinatra has ended his set (crowd applauds)
[2019-05-06 15:33:52] INFO  going to shutdown ...
[2019-05-06 15:33:52] INFO  WEBrick::HTTPServer#start done.

But how does it work?

Process, Process ID, Parent Process and Signals

Actually, all *nix systems such as MacOS and Linux support a mechanism called Signal Handling. Basically it's a way for a process to send messages to another process. Pressing CTRL-C in the terminal is basically asking the terminal process to send SIGINT (which means the Interrupt Signal) to its foreground child process: the running Ruby process.

a normal Sinatra app start and terminate flow

But wait. What is a process? According to Wikipedia:

In computing, a process is the instance of a computer program that is being executed.

Whenever you opened your favorite web browser, or your code editor, or a music app, or any other apps, you started at least one process. Each process has a unique integer id (usually called PID, aka process id), a parent process and potentially any number of child processes. The reason I said "at least one" is because those processes might start child and grandchild processes as well.

If you are on the Mac you can open the Activity Monitor app to inspect all running processes. Following screenshot shows that WeChat is running with its PID set to 4848.

Sorry for the Japanese screenshot because I am learning the language

If you use a Mac, a regular process that's an instance of a Mac Application will have the parent set to launchd, which has ID 1.

this WeChat process has launchd as its parent

If you start a program from the terminal, for example the htop, which is a nice way to view your system resource usage, the situation is a little bit more complicated than a standard Mac app:

Since every process has one and only one parent, we can use a nice little tool called pstree to display the process tree:

$ pstree -p 13954
-+= 00001 root /sbin/launchd
 \-+= 13643 forresty /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal
   \-+= 13644 root login -pfl forresty /bin/bash -c exec -la zsh /bin/zsh
     \-+= 13645 forresty -zsh
       \--= 13954 forresty htop

Here 13954 is the ID of htop process, and it's run under the zsh shell under the Terminal.app under the launchd which is the parent of all Mac processes.

If we run pstree without any argument we get a full list of all running processes on our computer, here is a truncated version for my laptop:

➜  mini-sinatra git:(master) pstree
-+= 00001 root /sbin/launchd
 |--= 00077 root /Library/Application Support/iStat Menus 6/iStatMenusDaemon
 |--= 00521 forresty /usr/local/opt/redis/bin/redis-server *:6379
 |--= 00528 forresty /Library/Application Support/iStat Menus 6/iStatMenusAgent.app/Contents/MacOS/iStatMenusAgent
 |-+= 00533 forresty /usr/local/opt/postgresql/bin/postgres -D /usr/local/var/postgres
 | |--= 00628 forresty postgres: checkpointer process
 | |--= 00629 forresty postgres: writer process
 | |--= 00630 forresty postgres: wal writer process
 | |--= 00631 forresty postgres: autovacuum launcher process
 | |--= 00632 forresty postgres: stats collector process
 | \--= 00633 forresty postgres: bgworker: logical replication launcher
 |--= 00541 forresty /Library/Application Support/iStat Menus 6/iStat Menus Status.app/Contents/MacOS/iStat Menus Status
 |--= 00673 forresty /Library/Little Snitch/Little Snitch Network Monitor.app/Contents/MacOS/Little Snitch Network Monitor -psn_0
 |--= 00675 forresty /Applications/Bartender 3.app/Contents/MacOS/Bartender 3
 |-+= 00687 forresty /Applications/Docker.app/Contents/MacOS/Docker
 | \-+= 00780 forresty /Applications/Docker.app/Contents/MacOS/com.docker.supervisor -watchdog fd:0
 |   |--- 00781 forresty com.docker.osxfs serve --address fd:3 --connect vms/0/connect --control fd:4 --log-destination asl
 |   |-+- 00782 forresty com.docker.vpnkit --ethernet fd:3 --port vpnkit.port.sock --port hyperkit://:62373/./vms/0 --diagnostics
 |   | \--- 00788 forresty (uname)
 |   \-+- 00783 forresty com.docker.driver.amd64-linux -addr fd:3 -debug
 |     \--- 00793 forresty com.docker.hyperkit -A -u -F vms/0/hyperkit.pid -c 2 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virti
 |--= 01373 forresty /Library/Input Methods/Squirrel.app/Contents/MacOS/Squirrel
 |-+= 01489 forresty /Applications/iTerm.app/Contents/MacOS/iTerm2
 | |-+= 08553 forresty /Applications/iTerm.app/Contents/MacOS/iTerm2 --server /usr/bin/login -fpl forresty /Applications/iTerm.ap
 | | \-+= 08554 root /usr/bin/login -fpl forresty /Applications/iTerm.app/Contents/MacOS/iTerm2 --launch_shell
 | |   \--= 08555 forresty -zsh
 | \-+= 70635 forresty /Applications/iTerm.app/Contents/MacOS/iTerm2 --server /usr/bin/login -fpl forresty /Applications/iTerm.ap
 |   \-+= 70639 root /usr/bin/login -fpl forresty /Applications/iTerm.app/Contents/MacOS/iTerm2 --launch_shell
 |     \-+= 70643 forresty -zsh
 |       \-+= 10728 forresty pstree
 |         \--- 10729 root ps -axwwo user,pid,ppid,pgid,command
 |-+= 04353 forresty /Applications/Visual Studio Code.app/Contents/MacOS/Electron .
 | |--- 04361 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --type
 | |--- 04367 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --type
 | |-+- 08186 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --type
 | | |--- 08188 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --no
 | | |--- 08189 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /App
 | | \--- 08192 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /App
 | \-+- 69508 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --type
 |   |-+- 69510 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper --no
 |   | |--- 09396 forresty /usr/local/bin/git fetch
 |   | \--- 73961 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /U
 |   \--- 69511 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper.app/Contents/MacOS/Code Helper /App
 |--- 04366 forresty /Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Resources/crashpad_han
 |-+= 54409 forresty /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
 | |--- 08843 forresty /Applications/Google Chrome.app/Contents/Versions/76.0.3783.0/Google Chrome Helper.app/Contents/MacOS/Goog
 | |--- 08924 forresty /Applications/Google Chrome.app/Contents/Versions/76.0.3783.0/Google Chrome Helper.app/Contents/MacOS/Goog
 | \--- 66645 forresty /Applications/Google Chrome.app/Contents/Versions/76.0.3783.0/Google Chrome Helper.app/Contents/MacOS/Goog
 |--- 54417 forresty /Applications/Google Chrome.app/Contents/Versions/76.0.3783.0/Google Chrome Framework.framework/Helpers/chro
 \--= 63121 forresty /Applications/Preview.app/Contents/MacOS/Preview -psn_0_2794154

You will see that I am running redis, docker, PostgreSQL, iTerm2, VSCode, Chrome, Preview.app among other apps. Some apps by its nature will have multiple child processes, for example PostgreSQL need that since it is a complicated RDBMS; VSCode relies on Electron for rapid development with web technologies; Chrome loads every tab and every extension in a separated child process so it's fast and won't crash the whole app if one tab or one extension goes wrong.

Running processes consume memory and CPU, drains your battery, that's why it is a good idea to always keep an eye on what processes your computer is currently running, and don't open too many apps / chrome tabs and install too many chrome extensions.

We are not going to discuss processes further here. Jesse Storimer has a wonderful little book called Working with Unix Processes, it uses Ruby as the example language to explore the Unix Process land, I highly recommend you go buy it. I did it several years ago and it was a wonderful read.

It's a Trap: Catching the Signals

Okay, so much about the background knowledge of processes. Let's see how Sinatra handles the CTRL-C event (to be exact, it should be how it handles the SIGINT / SIGTERM signals):

# https://github.com/sinatra/sinatra/blob/v2.0.5/lib/sinatra/base.rb#L1543-L1556

def setup_traps
  if traps?
    at_exit { quit! }

    [:INT, :TERM].each do |signal|
      old_handler = trap(signal) do
        quit!
        old_handler.call if old_handler.respond_to?(:call)
      end
    end

    set :traps, false
  end
end

The method we are looking for is Kernel#trap, it takes a signal and a block, register the block to be called when the signal arrives. Here Sinatra also handles SIGTERM because by default when we use the kill command or Activity Monitor to stop the process, we are sending the terminate signal.

So let's improve our code to incorporate signal handling mechanism:

# fake_sinatra.rb
require 'webrick'

class FakeSinatra
  def self.get_instance(server=nil)
    @instance ||= new
  end

  def start
    @server = WEBrick::HTTPServer.new(Port: 4567)
    @server.mount('/', self.class)
    @server.start
  end

  def stop
    @server.stop
  end

  def service(req, res)
    res.body = "#{@routes[req.path].call} - served by FakeSinatra"
  end

  def register(path, block)
    @routes ||= {}
    @routes[path] = block
  end

  module Helpers
    def get(path, &block)
      FakeSinatra.get_instance.register(path, block)
    end
    # TODO: implement more HTTP methods
  end
end

# so the user can call `get` etc
include FakeSinatra::Helpers

at_exit { FakeSinatra.get_instance.start }

[:INT, :TERM].each do |signal|
  trap(signal) {
    puts "SIG#{signal} caught, shutting down gracefully..."
    FakeSinatra.get_instance.stop
  }
end

Try it out:

$ ruby app.rb
[2019-05-06 17:11:14] INFO  WEBrick 1.4.2
[2019-05-06 17:11:14] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
[2019-05-06 17:11:14] INFO  WEBrick::HTTPServer#start: pid=14835 port=4567
^CSIGINT caught, shutting down gracefully...
[2019-05-06 17:11:14] INFO  going to shutdown ...
[2019-05-06 17:11:14] INFO  WEBrick::HTTPServer#start done.

Awesome.

Additional HTTP Methods

Supporting additional HTTP methods is actually easy. Let's rush out to create a test app that registers a POST route first:

# app.rb
require_relative 'fake_sinatra'

get '/articles' do
  'list all articles'
end

post '/articles' do
  'creating a new article'
end

If we run the program, it should fail immediately since the post method is not implemented yet. But weird things happened:

$ ruby app.rb
[2019-05-06 17:39:26] INFO  WEBrick 1.4.2
[2019-05-06 17:39:26] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
[2019-05-06 17:39:26] INFO  WEBrick::HTTPServer#start: pid=16718 port=4567

Nothing.

But when we CTRL-C:

^CSIGINT caught, shutting down gracefully...
[2019-05-06 17:40:00] INFO  going to shutdown ...
[2019-05-06 17:40:00] INFO  WEBrick::HTTPServer#start done.
Traceback (most recent call last):
app.rb:8:in `<main>': undefined method `post' for main:Object (NoMethodError)

The reason is when the NoMethodError is thrown, since no exception rescue is in place the program will crash, then the at_exit block is triggered. Remember what we did in at_exit block? we starts the webrick server. Therefore webrick will take over no matter what, only until CTRL-C is caught the server got shutdown and finally the error is shown.

$! – Global Variables in Ruby

Here we can use a Ruby trick: the global variable $! to handle the situation. Ruby has many built-in global variables, $! holds the current error object if it exists. Most of the time you don't need to use it, but in this case it is very handy. What we need to do is to add one line to the at_exit block:

at_exit {
  exit if $! # really exit
  FakeSinatra.get_instance.start
}

With that in place our program crashes as expected (sometimes crashing is a good thing!):

$ ruby app.rb
Traceback (most recent call last):
app.rb:8:in `<main>': undefined method `post' for main:Object (NoMethodError)

For reference, here's a list of relatively important global variables:

var	meaning
$!	latest error message
$_	string last read by gets
$&	string last matched by regexp
$~	the last regexp match, as an array of subexpressions
$n	the nth subexpression in the last match <- we will be using this later
$0	the name of the ruby script file
$*	the command line arguments
$$	interpreter's process ID
$?	exit status of last executed child process

Let's implement post support:

# fake_sinatra.rb
class FakeSinatra
  # ...omitted
  def service(req, res)
    method = req.request_method.to_sym
    res.body = "#{@routes[[method, req.path]].call} - served by FakeSinatra"
  end

  def register(method, path, block)
    @routes ||= {}
    @routes[[method, path]] = block
  end

  module Helpers
    def get(path, &block)
      FakeSinatra.get_instance.register(:GET, path, block)
    end
    def post(path, &block)
      FakeSinatra.get_instance.register(:POST, path, block)
    end
  end
end
# ...omitted

Here we use WEBrick::HTTPRequest#request_method to detect the HTTP method used. Using a two element array [method, path] as key makes me really miss tuples in Python. But hey, you can't always get what you want.

We love DRY. With a little Ruby trick we can implement all other methods we want to support:

# snippet of fake_sinatra.rb
module Helpers
  %w{ get post patch put delete }.each do |method|
    define_method(method) do |path, &block|
      FakeSinatra.get_instance.register(method.upcase.to_sym, path, block)
    end
  end
end

Module#define_method allows us to create methods dynamically. It takes an argument which becomes the method name, and a block which becomes the method parameters and body. Ruby is awesome.

Add a small test route:

# snippet of app.rb

# TODO: use pattern matching
delete '/articles/1' do
  'deleting a article'
end

We can test our app with an awesome HTTP client called HTTPie:

It works handsomely.

Query Parameters

Our application is flawed. Very flawed. It ignores all query parameters and doesn't understand routes with named parameters such as get '/articles/:id'. Let's fix this.

First we need to provide the params method. Let's first try to puts a p in the block:

# app.rb
require_relative 'fake_sinatra'

get '/articles' do
  p params
  'list all articles'
end

and test:

[2019-05-06 21:39:27] ERROR NameError: undefined local variable or method `params' for main:Object
	app.rb:5:in `block in <main>'
	/Users/forresty/workspace/theseus/mini-sinatra/fake_sinatra.rb:21:in `service'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/httpserver.rb:140:in `service'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/httpserver.rb:96:in `run'
	/Users/forresty/.rbenv/versions/2.6.2/lib/ruby/2.6.0/webrick/server.rb:307:in `block in start_thread'
::1 - - [06/May/2019:21:39:27 CST] "GET /articles HTTP/1.1" 500 339
- -> /articles

Look at this error:

ERROR NameError: undefined local variable or method `params' for main:Object

That means, we are trying to call params method on the main object. What's this main object?

The "main" Object: Secrets of "self"

The truth is: Ruby is so Object-Oriented that every method we call, we call it on an object. That means every method has an owner, there are no such things as free methods.

To be more technically correct, every time we call a method, we call the method on an object by sending that object a message, and the object that's receiving the message is called the receiver. This pattern is usually referred to as Message Passing.

If we do not specify a receiver for that method, the implied receiver will be self, the current object.

So in our code:

get '/articles' do
  p params
  'list all articles'
end

is equivalent to:

get '/articles' do
  p self.params # or self.__send__(:params) if :params is a private method
  'list all articles'
end

Ruby's main object is the top level object, it's the current object if you are outside any class and method body. Inside a class, self will be bound to the class object; inside a method, self will be bound to the same object that's calling the method; inside a block, it will be bound to the same context when the block is evaluated.

To reiterate:

inside class context, self is bound to the class object
inside method context, self is bound to the same object that's calling the method
2.1. if it is an instance method, self is the instance
2.2. if it is a class method, self is the class
outside of any class or method, bound to the top level object main
inside a block, bound to the same context specified in rule 1, 2 and 3

So if we don't do anything to change the self binding, it will be bound to the top level main for our app.rb since it's outside any class and method when we call the get method with our block:

# app.rb
require_relative 'fake_sinatra'

get '/articles' do
  p params
  'list all articles'
end

We want the params to work, that is, we want to change the context of @routes[[method, req.path]].call to the FakeSinatra instance in following code:

def service(req, res)
  method = req.request_method.to_sym
  # the "served by FakeSinatra" signature is removed for brevity
  res.body = @routes[[method, req.path]].call
end

We can achieve this with BasicObject#instance_eval. According to the documentation:

instance_eval {|obj| block } → obj: Evaluates the given block within the context of the receiver (obj). In order to set the context, the variable self is set to obj while the code is executing, giving the code access to obj's instance variables and private methods.

If we change the implementation of FakeSinatra#service to:

class FakeSinatra
  # ...omitted
  def service(req, res)
    method = req.request_method.to_sym
    block = @routes[[method, req.path]]
    res.body = FakeSinatra.get_instance.instance_eval(&block)
  end

  def params
    { answer: 42 } # dummy object, act as a placeholder
  end
  # ...omitted
end

We got the dummy params object:

$ ruby app.rb
[2019-05-06 22:40:47] INFO  WEBrick 1.4.2
[2019-05-06 22:40:47] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
[2019-05-06 22:40:47] INFO  WEBrick::HTTPServer#start: pid=26138 port=4567
{:answer=>42}
::1 - - [06/May/2019:22:40:49 CST] "GET /articles HTTP/1.1" 200 17
- -> /articles

Here with the magic of instance_eval we changed the self inside get, post and all other http method blocks:

# snippet of app.rb
get '/self' do
  self
end

$ http localhost:4567/self
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 33
Date: Tue, 07 May 2019 09:53:03 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

#<FakeSinatra:0x00007fb753ab1c80>

So what's left is to get the real params. Again after some source digging on the webrick we find that WEBrick::HTTPRequest#query is (almost) what we want. So let's finish this:

# fake_sinatra.rb
def service(req, res)
  @params = req.query
  method = req.request_method.to_sym
  block = @routes[[method, req.path]]
  res.body = FakeSinatra.get_instance.instance_eval(&block).to_s
end

def params; @params; end

# app.rb
get '/inspect' do
  params
end

$ http localhost:4567/inspect\?foo=bar
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 14
Date: Mon, 06 May 2019 14:52:49 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

{"foo"=>"bar"}

$ http localhost:4567/inspect
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 2
Date: Mon, 06 May 2019 14:52:57 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

{}

Looking good. So what we need is to enable the wildcard-fu on our routes.

By the way, if you need more information on self or just want to clear the FUD, go read Metaprogramming Ruby. Trust me, it is awesome.

Take a Walk on the Wild Side: Wildcard Routes (aka Named Params)

Not a big deal. Just some regular-expression-fu. Basically, what we want is to turn '/articles/:id' into %r{\A/articles/([^/]+)/?\z}, and '/restaurants/:id/comments' into %r{\A/restaurants/([^/]+)/comments/?\z}, while keeping '/articles/' almost intact as %r{\A/articles/?\z}.

For simplicity, we are not going to allow using more than 1 named parameter in the route. Most of the time, it's not a good pattern anyway.

# fake_sinatra.rb
def service(req, res)
  @params = req.query
  method = req.request_method.to_sym

  named_param = nil
  block = @routes.find { |path, _|
    pattern, named_param = to_pattern(path)
    pattern =~ req.path
  }[1][method] rescue nil

  if block
    # $1 will be the named param value
    @params.merge!({ named_param => $1 }) if named_param
    res.body = FakeSinatra.get_instance.instance_eval(&block).to_s
  else
    res.body = "unknown route: #{method} #{req.path}"
  end
end

def register(method, path, block)
  @routes ||= {}
  @routes[path] ||= {}
  @routes[path][method] = block
end

private

def to_pattern(path)
  # '/articles/' => %r{\A/articles/?\z}
  # '/articles/:id' => %r{\A/articles/([^/]+)/?\z}
  # '/restaurants/:id/comments' => %r{\A/restaurants/([^/]+)/comments/?\z}

  # remove trailing slashes then add named capture group
  path = path.gsub(/\/+\z/, '').gsub(/\:([^\/]+)/, '([^/]+)')

  # $1 will be the matched named param key if present
  [%r{\A#{path}/?\z}, $1]
end

Here in service we use inline rescue to quickly bypass many different situations where nil values will cause us troubles:

block = @routes.find { |path, _|
  pattern, named_param = to_pattern(path)
  pattern =~ req.path
}[1][method] rescue nil

This basically mean if anything wrong happens, set value of block to nil.

Remember the list of global variables? Now $1 really comes handy. In #to_pattern $1 is the name of the params, so '/articles/:id' will give us 'id'; and in #service it will be the value from the captured group of pattern =~ req.path (for example, %r{\A/articles/([^/]+)/?\z}), therefore the value of the param.

Note that obviously our implementation here is not performant at all: we are calling to_pattern multiple times for every incoming request.

But optimization will degrade the readability of our code, also it is not the focus at the moment, since we are still very early in development. As the wise said, premature optimization is the root of all evil.

Ready for test drive:

# app.rb
require_relative 'fake_sinatra'

get '/articles/' do
  params
end

get '/articles/:id' do
  params
end

get '/restaurants/:id/comments' do
  params
end

$ http -b localhost:4567/articles
{}

$ http -b localhost:4567/articles/1
{"id"=>"1"}

$ http -b localhost:4567/articles/1\?foo=bar
{"foo"=>"bar", "id"=>"1"}

$ http localhost:4567/restaurants
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 31
Date: Mon, 06 May 2019 15:42:25 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

unknown route: GET /restaurants

It was one hell of a hack, but fortunately it works.

HTTP Status Code, or How I Became a Teapot

Our Sinatra clone is taking shape. But now it is a bit boring: it only outputs plaintext, and even when we got unknown route, it still says HTTP/1.1 200 OK.

I mean, how can that be ok? It should have said 404 Not Found!

Also sometimes, when our app server become sentient, it might want declare itself as 418 I'm a teapot. I mean why not? Aren't we all created equal? We should have prepared for that.

If you don't already know, per MDN:

HTTP response status codes indicate whether a specific HTTP request has been successfully completed. Responses are grouped in five classes: informational responses, successful responses, redirects, client errors, and servers errors.

Basically 2xx means ok, 3xx means redirection, 4xx means you are wrong and 5xx means it is server's fault. Here is a very nice website that shows you different status codes and their meanings: https://httpstatuses.com

In Sinatra we can set custom status code like this:

post '/articles' do
  status 201
  'a new is article created.'
end

Then we have:

$ http post localhost:4567/articles
HTTP/1.1 201 Created
Connection: Keep-Alive
Content-Length: 0
Content-Type: text/html;charset=utf-8
Date: Tue, 07 May 2019 04:43:14 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

Should be easy. Let's go ahead and implement it:

# snippet of fake_sinatra.rb

def service(req, res)
  @status = 200 # reset to default
  # ...omitted

  if block
    # ...omitted
    res.body = FakeSinatra.get_instance.instance_eval(&block).to_s
    res.status = @status # need to do this after evaluating the block
  else
    # ...omitted
    res.status = 404
  end
end

def status(code)
  @status = code
end

And it works:

$ http post localhost:4567/articles
HTTP/1.1 201 Created
Connection: Keep-Alive
Content-Length: 2
Date: Tue, 07 May 2019 04:52:18 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

{}

$ http localhost:4567/articles/1
HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Length: 11
Date: Tue, 07 May 2019 04:52:11 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

{"id"=>"1"}

$ http post localhost:4567/non-existent
HTTP/1.1 404 Not Found
Connection: Keep-Alive
Content-Length: 33
Date: Tue, 07 May 2019 04:52:14 GMT
Server: WEBrick/1.4.2 (Ruby/2.6.2/2019-03-13)

unknown route: POST /non-existent

Our #service method is getting a bit complicated, let's apply some refactoring:

# snippet of fake_sinatra.rb

def service(req, res)
  @status = 200 # reset to default

  block, path_params = match_route(req)

  return not_found!(req, res) if block.nil?

  @params = req.query.merge(path_params)
  res.body = FakeSinatra.get_instance.instance_eval(&block).to_s
  res.status = @status # need to do this after evaluating the block
end

private

def not_found!(req, res)
  method = req.request_method.to_sym

  res.status = 404
  res.body = "unknown route: #{method} #{req.path}"
end

def match_route(req)
  method = req.request_method.to_sym

  named_param = nil

  block = @routes.find { |path, _|
    pattern, named_param = to_pattern(path)
    pattern =~ req.path
  }[1][method] rescue nil

  # $1 will be the named param value
  [block, { named_param => $1 }.compact]
end

We Write Tests Because We Are Lazy

As we keep improving our mini-framework it's getting more and more tedious to manually run tests after each modification. But if we do not run tests, there might be bugs. As a great hacker once said, there are three virtues of a programmer:

laziness
impatience
hubris

If we have bugs in our code it will hurt our hubris, but we are also too lazy to manually run tests.

Hence we write automated tests.

Since our codebase is so simple now, let's don't introduce a test framework, but instead just roll our own.

Process#fork

Remember previously we talked about child processes? It's very easy to start one ourselves:

# test.rb

app_pid = fork {
  # will be run in the child process
  # parent never reaches here
  load 'app.rb'
}

# will be run in the parent process
# child never reaches here

10.times { |i| puts "shutting down in #{10-i}"; sleep 1 }

Process.kill(:INT, app_pid)

Here we used Process#fork to create a new child process, the child process will have a copy of everything the parent process has, and continues the execution in the block provided to the fork call. The only line of code inside the block is the load method call, which is basically running every line of code in app.rb. So the role of the child process is to achieve the same effect as running ruby app.rb, only we don't need to do that manually.

The Process#fork will return the PID of the child process, therefore later in the parent process we can kill it with Process.kill.

Test it:

$ ruby test.rb
shutting down in 10
[2019-05-07 13:24:30] INFO  WEBrick 1.4.2
[2019-05-07 13:24:30] INFO  ruby 2.6.2 (2019-03-13) [x86_64-darwin18]
[2019-05-07 13:24:30] INFO  WEBrick::HTTPServer#start: pid=5410 port=4567
shutting down in 9
shutting down in 8
shutting down in 7
shutting down in 6
shutting down in 5
shutting down in 4
shutting down in 3
shutting down in 2
shutting down in 1
SIGINT caught, shutting down gracefully...
[2019-05-07 13:24:40] INFO  going to shutdown ...
[2019-05-07 13:24:40] INFO  WEBrick::HTTPServer#start done.

Note that the PID shown by webrick in above example is 5410, which is the PID of the child process, since the webrick server is started in the child process. If we are fast enough to identify the PID of the parent process before time runs out:

# 1. print all process info
# 2. find ruby ones
# 3. exclude ones that contains 5410 which is the child pid
# 4. print the second column
$ ps aux | grep ruby | grep -v 5410 | awk '{print $2}'
5382

Don't worry too much if you are not fully comfortable with above shell script, later we will have a post discuss how to write our own shell.

In this case, the parent PID is 5382, then we can run pstree:

$ pstree 5382
-+= 05382 forresty ruby test.rb
 \--- 05410 forresty ruby test.rb

We are able to visualize the tree we started (or planted?). Again, if you are interested in how processes work, I highly recommend you go read the Working With Unix Processes book.

Testing with Real HTTP Requests

Now what we need to do is to replace the countdown with some real test code. Later when our fake Sinatra implementation is Rack-compatible we can use libraries like rake-test to fake HTTP requests, but now let's simply send real ones:

# test.rb

app_pid = fork {
  # will be run in the child process
  # parent never reaches here

  # silence webrick
  [$stdout, $stderr].each { |f| f.reopen("/dev/null", "w") }

  # start the app
  load 'app.rb'
}

# will be run in the parent process
# child never reaches here

# ensure killing webrick even when errors are raised
at_exit { Process.kill(:INT, app_pid) }

require 'net/http'

def request(method, path, body={})
  uri = URI("http://localhost:4567#{path}")
  res = case method
  when :get
    Net::HTTP.get_response(uri)
  when :post
    Net::HTTP.post_form(uri, body)
  else
    raise "method '#{method}' is not supported yet"
  end

  # status, headers, body
  [ res.code.to_i, Hash[res.each.to_a], res.body ]
rescue Errno::ECONNREFUSED
  sleep 1 # if webrick is not ready we try again later
  retry
end

def get(path)
  request(:get, path)
end

def post(path, body={})
  request(:post, path, body)
end

def assert_equal(expected, actual)
  assert expected == actual, "expecting #{expected.inspect}, got #{actual.inspect}"
end

def assert_match(pattern, actual)
  assert pattern =~ actual, "expecting #{actual.inspect} to match #{pattern.inspect}"
end

def assert(condition, message)
  raise message unless condition
  print '.'
end

status, headers, body = get('/articles/1')
assert_equal 200, status
assert_equal %[{"id"=>"1"}], body

status, headers, body = get('/articles/1?foo=bar')
assert_equal 200, status
assert_equal %[{"foo"=>"bar", "id"=>"1"}], body

status, headers, body = get('/restaurants/233/comments')
assert_equal 200, status
assert_equal %[{"id"=>"233"}], body

status, headers, body = post('/articles')
assert_equal 201, status
assert_equal %[{}], body

status, headers, body = get('/non-existent')
assert_equal 404, status
assert_match /unknown/, body

puts
puts
puts "all passed."

Here $stdout, $stderr again are two important global variables in Ruby. Actually, every *nix process will have 3 special files open, they are called the standard streams:

Standard Input: $stdin
Standard Output: $stdout
Standard Error: $stderr

When we run a program and see results printing out to the terminal, that's because the program is printing to either stdout or stderr. And since /dev/null is a special location, which will discard anything that got written to it (like a blackhole! Wait. More like a trash bin with a portal at the bottom), we can silence the webrick child process by pointing its stdout and stderr to /dev/null.

Another trick we did is we register the at_exit callback on the parent side immediately after the fork call, so that any bad things happen in the future we can ensure the child process will be clean up.

Also we want to rescue Errno::ECONNREFUSED and sleep then retry in #request method because the HTTP requests may be send before webrick is ready to receive them.

Note that we don't need a third party HTTP library, we can just use net/http.

And it works.

$ ruby test.rb
..........

all passed.

Setting Response Headers

Now with testing in place we are much more confident to move forward quickly. Let's implement the ability to set headers.

In Sinatra it works like this:

get '/header' do
  headers "X-Custom-Value" => "This is a custom HTTP header."
  'ok'
end

get '/headers' do
  headers "X-Custom-Value" => "foo", "X-Custom-Value-2" => "bar"
  'ok'
end

So after checking the source code of webrick we go ahead and implement it like this:

# snippet of fake_sinatra.rb
def service(req, res)
  @status, @headers = 200, {} # reset to default

  # ...omitted
  res.body = FakeSinatra.get_instance.instance_eval(&block).to_s
  # ...omitted
  @headers.each { |k, v| res[k] = v }
end

private

def headers(additional_headers)
  @headers = additional_headers
end

Add some test:

_, headers, _ = get('/header')
# webrick turn headers into lowercase form
assert_match /custom/, headers['x-custom-value']

_, headers, _ = get('/headers')
assert_equal 'foo', headers['x-custom-value']
assert_equal 'bar', headers['x-custom-value-2']

And run:

$ ruby test.rb
.............

all passed.

Redirection: HTTP 301 / 302 and Location Header

HTTP redirection is actually very simple. The client sends a request to a URL, the server says, "no, what you are looking for is not here, but I know where it is". The server tells the client this with HTTP status code 3xx, where most of the time is either 301 or 302, and a "Location" header, so the client can decide whether it wants to follow the redirection. The redirection part is actually optional.

Sinatra's API for redirection is this:

get '/temp-redirect' do
  redirect 'http://google.com'
end

get '/perm-redirect' do
  redirect 'http://google.com', 301 # permanent redirection
end

Since we already have a mechanism to set status code and header, let's implement redirection:

private

def redirect(target, code=302)
  status(code)
  headers 'Location' => target
end

Add more tests:

status, _, _ = get('/temp-redirect')
assert_equal 302, status

status, headers, _ = get('/perm-redirect')
assert_equal 301, status
assert_equal 'http://google.com', headers['location']

When we run automated tests they all passed, but we are cautious, so we manually try them. Here we use -L option of curl so that it will follow redirection like our browsers do:

$ curl -L http://localhost:4567/temp-redirect
... a long HTML from google.com

$ curl -L http://localhost:4567/perm-redirect
... a long HTML from google.com

It works.

ERB

Let's add templating capabilities and call it a day. There are many template engines, but ERB is the built-in one and the chef's choice in Rails's omakase. Later we will add support for other template engines, for now let's just implement erb.

The simplest use case maybe this:

# snippet of app.rb
get '/index' do
  erb :index
end

<!-- ./views/index.erb -->
<h1>it works</h1>

Per erb documentation, we implement the erb method as this:

def erb(view)
  path = File.expand_path("./views/#{view}.erb", __dir__)
  template = ERB.new(File.read(path))

  template.result(binding)
end

__dir__ is a special method that always return the current file's directory. Here the binding is a special Ruby method, basically it represents the context of current object self, and in this case, the FakeSinatra instance.

Objects of class Binding encapsulate the execution context at some particular place in the code and retain this context for future use. The variables, methods, value of self, and possibly an iterator block that can be accessed in this context are all retained. Binding objects can be created usingKernel#binding

Test it out:

Voila.

Even better, since we are using the same binding, we will be able to setup instance variables and the view will pick them up immediately:

get '/time' do
  @time = Time.now
  erb :time
end

<!-- views/time.erb -->
<h1><%= @time %></h1>

Just be careful not to incidentally overwrite the internal ivars.

Wrapping Up

So that's it for today. The source code of this post lives in the same repository as the previous one, go check it out.

As of now our mini-framework has 83 lines of code and 67 lines of test.

$ cloc fake_sinatra.rb
       1 text file.
       1 unique file.
       0 files ignored.

github.com/AlDanial/cloc v 1.80  T=0.02 s (63.8 files/s, 6505.6 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Ruby                             1             28              8             83
-------------------------------------------------------------------------------
$ cloc test.rb
       1 text file.
       1 unique file.
       0 files ignored.

github.com/AlDanial/cloc v 1.80  T=0.01 s (84.4 files/s, 7003.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Ruby                             1             24             10             67

Later we will have one or more follow up posts to discuss how we can improve our mini framework to:

understand request headers
process HTTP request body
support other template engines
support before and after filters
Middleware and Rack compatibility
...etc.

As usual, welcome to leave me a message on the comment area or drop me an email at [email protected] Thanks for reading.