sensible.io blog

SSH Tunnel - Local and Remote Port Forwarding Explained With Examples

2014-05-17T17:57:00Z

There are two ways to create an SSH tunnel, local and remote port forwarding (there’s also dynamic forwarding, but we won’t cover that here). The best way to understand these is by an example, let’s start with local port forwarding.

Imagine you’re on a private network which doesn’t allow connections to a specific server. Let’s say you’re at work and imgur.com is being blocked. To get around this we can create a tunnel through a server which isn’t on our network and thus can access Imgur.

$ ssh -L 9000:imgur.com:80 user@example.com

The key here is -L which says we’re doing local port forwarding. Then it says we’re forwarding our local port 9000 to imgur.com:80, which is the default port for HTTP. Now open your browser and go to http://localhost:9000.

The awesome thing about SSH tunnels is that they are encrypted. Nobody is going to see what sites you’re visiting, they’ll only see an SSH connection to your server.

Connecting to a database behind a firewall

Another good example is if you need to access a port on your server which can only be accessed from localhost and not remotely.

An example here is when you need to connect to a database console, which only allows local connection for security reasons. Let’s say you’re running PostgreSQL on your server, which by default listens on the port 5432.

$ ssh -L 9000:localhost:5432 user@example.com

The part that changed here is the localhost:5432, which says to forward connections from your local port 9000 to localhost:5432 on your server. Now we can simply connect to our database.

$ psql -h localhost -p 9000

Now let’s stop here for a little bit an explain what is actually going on. In the first example the 9000:imgur.com:80 is actually saying forward my local port 9000 to imgur.com at port 80. You can imagine SSH on your server actually making a connection (a tunnel) between those two ports, one on your local machine, and one on the target destination.

If we however say something like 9000:localhost:5432, it means localhost from the server’s perspective, not localhost on your machine. This means forward my local port 9000 to port 5432 on the server, because when you’re on the server, localhost means the server itself.

This might be a bit confusing, but it is important to understand what the syntax actually means here.

Remote port forwarding

Now comes the second part of this tutorial, which is remote port forwarding. This is again best to explain with an example.

Say that you’re developing a Rails application on your local machine, and you’d like to show it to a friend. Unfortunately your ISP didn’t provide you with a public IP address, so it’s not possible to connect to your machine directly via the internet.

Sometimes this can be solved by configuring NAT (Network Address Translation) on your router, but this doesn’t always work, and it requires you to change the configuration on your router, which isn’t always desirable. This solution also doesn’t work when you don’t have admin access on your network.

To fix this problem you need to have another computer, which is publicly accessible and have SSH access to it. It can be any server on the internet, as long as you can connect to it. We’ll tell SSH to make a tunnel that opens up a new port on the server, and connects it to a local port on your machine.

$ ssh -R 3000:localhost:9000 user@example.com

The syntax here is very similar to local port forwarding, with a single change of -L for -R. But as with local port forwarding, the syntax remains the same.

First you need to specify a local port on your machine which you wish to expose publicly, which in this case is 3000. Next there is the address on which the server should listen to, and a port on which the server should listen to, which in this case is 9000.

There is one more thing you need to do to enable this. SSH doesn’t by default allow remote hosts to forwarded ports. To enable this open /etc/ssh/sshd_config and add the following line somewhere in that config file.

GatewayPorts yes

Make sure you add it only once!

$ sudo vim /etc/ssh/sshd_config

And restart SSH

$ sudo service ssh restart

After this you should be able to connect to the server remotely, even from your local machine. The way this would work is that you would first create an SSH tunnel that forwards traffic from the server on port 9000 to your local machine on port 3000. This means that if you connect to the server on port 9000 from your local machine, you’ll actually make a request to your machine through the SSH tunnel.

A few closing tips

You might have noticed that every time we create a tunnel you also SSH into the server and get a shell. This isn’t usually necessary, as you’re just trying to create a tunnel. To avoid this we can run SSH with the -nNT flags, such as the following, which will cause SSH to not allocate a tty and only do the port forwarding.

$ ssh -nNT -L 9000:imgur.com:80 user@example.com

SSH has a huge number of features, so I’d recommend you to checkout the manual page at man ssh, which contains even more tips.

There’s also an amazing talk called The Black Magic of SSH / SSH Can Do That?, which I really recommend you to watch.

Supercharge your VIM into IDE with CTags

2014-05-09T00:00:00Z

CTags generates index file of all your classes, methods and all other identifiers. You can use that index in your editor to jump straight to the methods you’re interested in. In this article, I’ll show you how to use them with Vim and Rails.

Installation

You need to install Exuberant CTags, on OSX just run

$ brew install ctags

or on Ubuntu/Debian

$ sudo apt-get install exuberant-ctags

Generating CTags manually

In your rails project you can generate CTags for you project with

$ ctags -R --languages=ruby --exclude=.git --exclude=log .

But what about generating CTags for our bundled libraries too? Easy task, let’s add bundle paths

$ ctags -R --languages=ruby --exclude=.git --exclude=log . $(bundle list --paths)

Usage

You can jump into the method with

:ta attr_accessor

or using regular expressions like that

:ta /^before_*

If you position cursor over the method and hit CTRL+] it will take you into the method.
CTRL-T will take you back from that method.
CTRL-I and CTRL-O will take you In and Out from the method.

:ts [expr]  # Lists tags matching expression
:[count]tn  # Jumps to the next matching tag
:[count]tp  # Jumps to the previous matching tag
:[count]tf  # Jumps to the first matching tag
:[count]tl  # Jumps to the last matching tag

CtrlP

If you’re using CtrlP, you can use CtrlPTag to browser your tags. You can bind that command to a key and add this line to your .vimrc

nnoremap <leader>. :CtrlPTag<cr>

Getting Advanced

Since we want to DRY this workflow a little, we’ll install Ruby gem to auto-generate those tags for us.

We’ll be using Guard-CTags-Bundler

Install the gem:

$ gem install guard-ctags-bundler

Add it to your Gemfile (inside development group):

group :development do
  gem 'guard-ctags-bundler'
end

Now add it to your Guardfile with

$ guard init ctags-bundler

Now you can run guard and it will start watching your files will generate tags and gems.tags files. Since Vim is not looking for gems.tags by default, you’ll need to edit your .vimrc and add set tags+=gems.tags line, then restart Vim.

.gitignore

Since you don’t want to commit those files into the GIT index, add them to .gitignore or I like to add them to my global ~/.gitignore.

$ cat <<EOT >> ~/.gitignore
tags
gems.tags
EOT

Path depends on your configuration git config --global core.excludesfile "~/.gitignore"

Questions?

Vim has very nice build-in help that can be accessed by

:help tags

Bonus: Inspecting Gems

I like to inspect Gems a lot. Just set your $EDITOR variable and then run command bundle open rails to open it in your favourite editor. I recommend you to try Tim Pope’s automatic RubyGems CTags invoker gem-ctags.

Just install the gem

$ gem install gem-ctags

and generate CTags for already installed Gems (needs to be run only first time).

$ gem ctags

Don't just dump code into your models

2014-04-19T16:05:00Z

A few years ago there used to be a motto saying Skinny controllers, fat models. It was good at the time, because people thought all of the application code belonged into the controllers, and it helped them realize that it’s good to have just a simple controller layer and push things down. The problem with this motto is that it’s wrong. Just because you should have skinny controllers it doesn’t mean you should have fat models. In fact, none of your classes should be fat.

The reason we have MVC is that it is a good starting point for separating the concerns in your application. It makes it more obvious where you should put your code when you’re starting out with your application. But this doesn’t mean your application should be limited to just these three layers, quite the opposite.

Every time you need to add a new service to your application, you should think about how you’re going to be using it, and then build a sufficient abstraction on top of that.

For example let’s say that your application needs to automatically sign up newly registered users to a mailing list. You’ve even chosen a provider, say AWeber or Mailchimp, as both of these have a nice Ruby gem wrapping up their API, allowing you to subscribe the user with only a few lines of code.

Here’s how the naive solution might look like

class User
  after_create :subscribe_to_mailchimp

  def subscribe_to_mailchimp
    list_id = ENV["MAILCHIMP_LIST_ID"]
    Mailchimp.new(ENV["MAILCHIMP_API_KEY"].subscribe(list_id, self.email)
  end
end

If you happen to like YAML files more than environment variables, you might write something like this instead

def subscribe_to_mailchimp
  config = SomeGlobalYamlConfigReader.mailchimp
  Mailchimp.new(config["api_key"].subscribe(config["list_id"], self.email)
end

While this might make sense the first time you try this, there are actually many things wrong with this approach. Let’s list a few:

If the Mailchimp API is unavailable during the time you decide to create a new user, your after_save callback will raise an exception.
If an exception is raised, ActiveRecord will rollback the transaction and your user won’t get created.
Even if you choose to handle the exception in the callback, you’ll fail to subscribe the user to the mailing list.
Your User model becomes coupled with the Mailchimp gem. If for some reason you later decide to switch to AWeber, you’ll have much harder time doing so (probably because you have Mailchimp in more than one place).

These are just a few problems. There are many more, for example you probably don’t even want to run this code in development/test environment, which means the method should be changed to something like

def subscribe_to_mailchimp
  if Rails.env.production?
    config = SomeGlobalYamlConfigReader.mailchimp
    Mailchimp.new(config["api_key"].subscribe(config["list_id"], self.email)
  end
end

but this is still wrong, because you might have a staging server where you do want to run this callback, making the method even more complicated.

But wait, there’s more. Let’s say you magically manage to overcome all the above issues and actually decide to test this method. A thought pops up in your head and you think I should probably be able to handle if I get a bad `APIKEYor aLISTID` which doesn’t exist on Mailchimp. But you have no easy way of testing that, because your callback is tightly coupled to the global configuration (or ENV variables). Wouldn’t it be much nicer if you could just inject the API_KEY and the LIST_ID somehow? Maybe even because you later decide that you need to subscribe different users to different mailing lists.

Here’s a little less wrong approach

# notice there's no mailchimp in the name
class EmailMarketing::Subscription
  attr_reader :api_key

  def initialize(api_key)
    @api_key = api_key
  end

  def subscribe(user, list_id)
    Mailchimp.new(api_key).subscribe(list_id, user.email)
  end
end

# now whenever you need to subscribe a new user, you'd just write
EmailMarketing::Subscription.new(API_KEY).subscribe(user, LIST_ID)

Where do you get the API_KEY and LIST_ID from? It doesn’t really matter, as long as it’s not in the Subscription or User classes that fetch it. Every class should have a single reason to change, and if you need to change your User model because you changed the way you store your email marketing API key, you’re doing it wrong.

Where do you put this class? Again, it doesn’t really matter. Maybe something like lib/email_marketing/subscription might make sense. When you later decide to add lib/email_marketing/campaign, it won’t look as stupid as having all that code in the models.

Why isn’t subscribe a class method? There are many reasons why you should use instance methods and save yourself some trouble.

Because of the problems mentioned earlier, the only way to solve this issue correctly is to use a background queue such as Sidekiq or Resque. There is one thing that’s always true for interacting with 3rd party services, they will fail occasionally. It’s not a question of if, it’s a question of when, and you have to be ready for it.

It’s completely normal that a service goes down for a few seconds or even minutes every now and then. If you have the subscription logic in your controller when the user is signing up, he might end up registering an account but because the request to Mailchimp fails he actually won’t get added to the mailing list.

This is why you have to use a background service that will try to add the user to the list, and if that fails it will automatically re-schedule a retry.

class UsersController < ApplicationController
  def create
    @user = User.new(user_params)
    if @user.save
      EmailSubscriptionWorker.perform_async(@user.id)
      redirect_to root_path, notice: "Welcome!"
    else
      render :new
    end
  end
end

class EmailSubscriptionWorker
  include Sidekiq::Worker

  def perform(user_id)
    api_key, list_id = # read this from some config

    user = User.find(user_id)
    EmailMarketing::Subscription.new(api_key).subscribe(user.email, list_id)
  end
end

You could also have a cron job that periodically checks if all users are signed up for the service, as an alternative to the background worker, but using a queue is a far superior solution.

Getting started with Ember App Kit

2014-03-07T00:00:00Z

In the following tutorial we’ll create a simple Ember.js app for logging your water intake. Besides learning the basics of Ember.js, you will also learn how to use Ember App Kit.

You can see the final application right below:

Link to Demo

About Ember App Kit

EAK is the basic building block of your app. It helps you to rapidly start the development with features such as the ES6 module transpiler, automatic reloading, the Grunt task runner, the Bower package manager…

EAK uses ECMAScript 6 Modules, which means you can keep your namespace clean and only import objects you need to the current scope. Thanks to es6-module-transpiler you can use the current ES6 module syntax and it will be automatically compiled into today’s JavaScript.

Creating a project

Create your project by cloning EAK

➤ git clone git@github.com:stefanpenner/ember-app-kit.git waterize
➤ cd waterize

Optionally you can get rid of the Git history and start fresh

➤ rm -rf .git
➤ git init

Install required node binaries

This only needs to be done once. Install Grunt, Bower and Loom (if you want to use generate command to easily generate controllers/models etc.).

➤ npm install -g grunt-cli bower loom

Starting a development server

Install the required node packages and start the server

➤ npm install
➤ grunt server

Finally open our project in the web browser (port defaults to 8000).

open http://localhost:8000

Generating basic routes and controllers

We’ll use day controller for storing our water intake per that day. Now let’s generate an object controller with Loom

In Ember.js, controllers allow you to decorate your models with display logic. In general, your models will have properties that are saved to the server, while controllers will have properties that your app does not need to save to the server.

Ember.js guide - Controllers

➤ generate controller day

Select o since we want an object controller, then generate a route for the controller

When your application starts, the router is responsible for displaying templates, loading data, and otherwise setting up application state. It does so by matching the current URL to the routes that you’ve defined.

Ember.js guide - Defining Your Routes

➤ generate route day
generating from:
  - node_modules/loom-generators-ember-appkit/loom
   created:     app/routes/day.js

Change app/templates/application.hbs to

<h2 id='title'>Waterize</h2>

{{outlet}}

Edit app/router.js and the following at the top of the route list

// We'll catch / and redirect user to current day
this.route("index", { path: "/"} );
// Catch /#2014-03-03
this.resource("day", { path: "/:formatted" });

Edit app/routes/index.js, remove model and add

beforeModel: function() {
  this.transitionTo("day", "2033-12-21");
}

Inside DayRoute in the app/routes/day.js add snippet

model: function(params) {
  return Ember.Object.create();
}

Now generate our main template showing the bottle.

➤ generate template day
generating from:
  - node_modules/loom-generators-ember-appkit/loom
   created:     app/templates/day.hbs

Bottle component

Generate a bottle component holding a bottle SVG dynamic image & percentage logic.

➤ generate component bottle-image
    generating from:
      - node_modules/loom-generators-ember-appkit/loom
       created:     app/components/bottle-image.js
       created:     app/templates/components/bottle-image.hbs
       created:     tests/unit/components/bottle-image-tests.js

Remove test for the component, since we won’t be using it now.

➤ rm tests/unit/components/bottle-image-tests.js

Edit app/templates/components/bottle-image.hbs and replace everything with SVG of a bottle. It’s simple path of the bottle, with the same clip path and a blue rectangle that represents the water level.

<svg width="169px" height="441px" viewBox="0 0 169 441" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    <title>bottle</title>
    <g id="Page-1" stroke="none" stroke-width="0" fill="none" fill-rule="evenodd">
      <clipPath id="ClipBottle">
        <path d="M114.611325,50.6484375 L48.5449223,50.6484375 L48.5449223,70.0136719 C48.5449223,70.0136719 22.787939,81.0417179 15.7187505,87.0292969 C8.15080179,93.4393231 4.63351106,107.20649 4.63351106,107.20649 L4.63351106,403.066413 C4.63351106,403.066413 17.3756986,431.116187 81.3929257,431.116211 C145.410153,431.116241 158.15234,403.066406 158.15234,403.066406 L158.15234,107.20649 C158.15234,107.20649 153.633342,93.2511626 145.410153,87.0292969 C137.247926,80.8535568 114.611325,70.0136719 114.611325,70.0136719 L114.611325,50.6484375 Z M114.611325,50.6484375" stroke="#1EB5E8" stroke-width="8"></path>
      </clipPath>

      <rect d="M10,206 L10,432 L161,432 L161,206 L10,206 Z M10,206" id="Water" stroke="#979797" fill="#1EB5E8" x="8" {{bind-attr y="y"}} width="151" {{bind-attr height="height"}} clip-path="url(#ClipBottle)"></rect>
      <path d="M114.611325,50.6484375 L48.5449223,50.6484375 L48.5449223,70.0136719 C48.5449223,70.0136719 22.787939,81.0417179 15.7187505,87.0292969 C8.15080179,93.4393231 4.63351106,107.20649 4.63351106,107.20649 L4.63351106,403.066413 C4.63351106,403.066413 17.3756986,431.116187 81.3929257,431.116211 C145.410153,431.116241 158.15234,403.066406 158.15234,403.066406 L158.15234,107.20649 C158.15234,107.20649 153.633342,93.2511626 145.410153,87.0292969 C137.247926,80.8535568 114.611325,70.0136719 114.611325,70.0136719 L114.611325,50.6484375 Z M114.611325,50.6484375" id="Bottle" stroke="#1EB5E8" stroke-width="8"></path>
      <rect d="M47.99606,0 C43.579958,0 40,3.58204887 40,7.99357759 L40,36.0064224 C40,40.4211534 43.5797154,44 47.99606,44 L115.00394,44 C119.420042,44 123,40.4179511 123,36.0064224 L123,7.99357759 C123,3.57884659 119.420285,0 115.00394,0 L47.99606,0 Z M47.99606,0" id="Cap" fill="#1EB5E8" x="40" y="0" width="83" height="44" rx="8"></rect>

      <text fill="#000000" font-family="Arial" font-size="32" font-weight="bold" letter-spacing="0">
        <tspan x="43" y="233">{{percent}} %</tspan>
      </text>
    </g>
</svg>

Edit app/components/bottle-image.js and add code for handling percentage, water levels.

var BottleImageComponent = Ember.Component.extend({
  goal: 2000, // in ml

  percent: function() {
    return Math.floor(this.get("drunk") / this.get("goal") * 100); // Calculate percentage of a goal (can be 100%+)
  }.property("drunk"),

  y: function() {
    return 430 - this.get("height"); // Offset of the rectangle
  }.property("height"),

  height: function() {
    return 3.75 * this.get("percent"); // Height of the rectangle
  }.property("percent")
});

export default BottleImageComponent;

Edit app/templates/day.hbs and replace everything with a bottle component, we’ll use a fixed drunk value for now.

{{bottle-image drunk=500}}

A model is a class that defines the properties and behavior of the data that you present to the user. Anything that the user expects to see if they leave your app and come back later (or if they refresh the page) should be represented by a model.

Ember.js guide - Defining Models

Create app/models/day.js and fill it with basic model that will save our stats

var Day = Ember.Object.extend({
  drunk: function() {
    return 0;
  }.property(),

  save: function() {
    // Implement me
  }
});

Day.reopenClass({
  query: function(conditions) {
    return Day.create();
  }
});

export default Day;

Adding model to the equation

We can now use our model in our routes, so edit app/routes/day.js and at the top add

import Day from "appkit/models/day"; // ES6 import, you need to import model, so it's accessible inside your route

to import our model and replace

return Ember.Object.create();

with

return Day.query({formatted: null});

Edit app/templates/day.hbs and replace fixed number 500 to drunk so our day.hbs will look like this now

{{bottle-image drunk=drunk}}

Drinking a cup!

Now edit app/controllers/day.js so it looks like this chunk of code

var DayController = Ember.ObjectController.extend({
  cups: [
    { ml: 250, name: "250 ml / 8.45 oz" },
    { ml: 500, name: "500 ml / 16.9 oz" }
  ],

  selectedCup: 250,

  actions: {
    drinkCup: function() {
      var model = this.get("model");
      var ml = this.get("selectedCup");
      model.incrementProperty("drunk", ml);
      model.save();
    },

    emptyBottle: function() {
      var model = this.get("model");
      model.set("drunk", 0);
      model.save();
    }
  }
});

export default DayController;

Now we’ll add some HTML controls. Edit app/templates/day.hbs and under the component add

{{view Ember.Select
       content=cups
       optionValuePath="content.ml"
       optionLabelPath="content.name"
       value=selectedCup
}}

<button {{action 'drinkCup'}}>Drink!</button>
<button {{action 'emptyBottle'}}>X</button>

Making our data persistent

To easily persist our data, we’ll use localStorage with a hash of date and ml drunk as keys. When it comes to time handling, the easiest way is to use Moment.js. Let’s install it with Bower

➤ bower install moment --save
bower cached        git://github.com/moment/moment.git#2.5.1
bower validate      2.5.1 against git://github.com/moment/moment.git#*
bower install       moment#2.5.1
moment#2.5.1 vendor/moment

Now we have MomentJS under vendor/moment, let’s include it on our HTML page.

Edit app/index.html and under jquery.js add line

<script src="/vendor/moment/moment.js"></script>

Now it’s time to implement the persistence part in app/models/day.js. We’ll replace everything with our model implementation

var Day = Ember.Object.extend({
  // Returns instance of moment for the given timestamp
  moment: function() {
    return window.moment(this.get("timestamp"));
  }.property("timestamp"),

  // Return formatted date
  formatted: function() {
    return this.get("moment").format("YYYY-MM-DD");
  }.property("moment"),

  // We'll use this as a key for the storage
  storageKey: function() {
    return "%@|%@".fmt(Day.namespace, this.get("formatted"));
  }.property("formatted"),

  // ml drunk
  drunk: function() {
    var drunk = window.localStorage[this.get("storageKey")];
    drunk = parseInt(drunk || 0, 10);

    return drunk;
  }.property("storageKey"),

  // Write value into localStorage
  save: function() {
    window.localStorage[this.get("storageKey")] = this.get("drunk");
  }
});

Day.reopenClass({
  namespace: "waterize", // Namespace in the localStorage

  // Query by timestamp
  query: function(conditions) {
    var timestamp = window.moment(conditions.formatted).startOf("day").valueOf();

    return Day.create({ timestamp: timestamp });
  },

  today: function() {
    return Day.query({ formatted: window.moment() });
  }
});

export default Day;

Wire those model changes up into the routes

Edit app/routes/index.js and at the top of route add

import Day from "appkit/models/day";

and change transitionTo to

this.transitionTo("day", Day.today());

so when we’re on the / page, it will redirect us to the current day.

Then edit app/routes/day.js and change that query part to

return Day.query({formatted: params.formatted});

this will return the day based on the url params.

Now visit http://localhost:8000 and it should redirect you to the current day, perfect! If you change the date, the bottle percentage should automatically update.

Switching between the days

We’ll need to add 2 more instance methods into our model app/models/day.js

  // Return next day
  tomorrow: function() {
    var tommorow = this.get("moment").clone().add(1, "day");

    if(window.moment().isAfter(tommorow)) {
      return Day.create({ timestamp: tommorow.valueOf() });
    }
  }.property("moment"),

  // Return previous day
  yesterday: function() {
    return Day.create({ timestamp: this.get("moment").clone().subtract(1, "day").valueOf() });
  }.property("moment")

so our app/models/day.js looks like this

var Day = Ember.Object.extend({
  // Returns instance of moment for the given timestamp
  moment: function() {
    return window.moment(this.get("timestamp"));
  }.property("timestamp"),

  // Return formatted date
  formatted: function() {
    return this.get("moment").format("YYYY-MM-DD");
  }.property("moment"),

  // We'll use this as a key for the storage
  storageKey: function() {
    return "%@|%@".fmt(Day.namespace, this.get("formatted"));
  }.property("formatted"),

  // ml drunk
  drunk: function() {
    var drunk = window.localStorage[this.get("storageKey")];
    drunk = parseInt(drunk || 0, 10);

    return drunk;
  }.property("storageKey"),

  // Return next day
  tomorrow: function() {
    var tommorow = this.get("moment").clone().add(1, "day");

    if(window.moment().isAfter(tommorow)) {
      return Day.create({ timestamp: tommorow.valueOf() });
    }
  }.property("moment"),

  // Return previous day
  yesterday: function() {
    return Day.create({ timestamp: this.get("moment").clone().subtract(1, "day").valueOf() });
  }.property("moment"),

  // Write value into localStorage
  save: function() {
    window.localStorage[this.get("storageKey")] = this.get("drunk");
  }
});

Day.reopenClass({
  namespace: "waterize", // Namespace in the localStorage

  // Query by timestamp
  query: function(conditions) {
    var timestamp = window.moment(conditions.formatted).startOf("day").valueOf();

    return Day.create({ timestamp: timestamp });
  },

  today: function() {
    return Day.query({ formatted: window.moment() });
  }
});

export default Day;

Now we need some links to take us to previous/next days, so let’s edit app/templates/day.hbs and at the top add

<p>{{#link-to 'index' classNames="today"}}Today{{/link-to}}</p>

<p>
{{#link-to 'day' yesterday}}&laquo;{{/link-to}}
Date: {{formatted}}
{{#if tomorrow}}
{{#link-to 'day' tomorrow}}&raquo;{{/link-to}}
{{/if}}
</p>

Testing

Let’s add only very simple acceptance test

Remove route test, because we’ve changed that route a little and it would raise an test error.

➤ rm tests/unit/routes/index-test.js

Now edit tests/acceptance/index-test.js and change that test part so it looks like this chunk of code

var App;

module('Acceptances - Index', {
  setup: function(){
    App = startApp();
  },
  teardown: function() {
    Ember.run(App, 'destroy');
  }
});

test('index renders', function(){
  expect(1);

  visit('/').then(function(){
    var today_link = find('a.today');
    equal(today_link.text(), 'Today');
  });
});

Now run the tests with

➤ grunt test

BONUS for Free: Adding off-line cache manifest

➤ npm install grunt-manifest -—save-dev

Create Grunt task in tasks/custom-options/manifest.js

module.exports = function(grunt) {
  grunt.loadNpmTasks('grunt-manifest');

  return {
    generate: {
      options: {
        basePath: 'dist',
        network: ['http://*', 'https://*'],
        preferOnline: true,
        verbose: true,
        timestamp: true,
        hash: true,
        master: ['index.html']
      },
      src: [
          'assets/*.min.js',
          'assets/*.min.css'
      ],
      dest: 'dist/manifest.cache'
    }
  };
};

…and change html line in app/index.html to

<html <!-- @if dist=true --> manifest="/manifest.cache"<!-- @endif -->>

We’ll use the manifest file only for distribution builds and disable it for development.

Edit Gruntfile.js and add our ‘manifest’ task to the bottom of the createDistVersion so the end of the task looks like this chunk

grunt.registerTask('createDistVersion', filterAvailable([
...
    'htmlmin:dist', // Removes comments and whitespace
    'manifest'
    ]));

Now kill your development server and run

➤ grunt server:dist

…to start a server with your distribution build. Browse your code, terminate the server and reload the page. Everything should work offline now.

Source of the final product can be found here: github.com/cyner/waterize

Updated 2014-03-17 - Fix typos

4 Tips for Working with Dates in PostgreSQL

2013-08-26T11:47:00Z

Those of us who come from Rails aren’t surprised when we see something like 5.weeks.from_now or 3.days.ago + 2.hours, which makes working with dates much easier. But PostgreSQL got your back on this as well, you can just use the builtin functions and get most of the same functionality.

Current Time/Date/Timestamp

There are many ways of getting a current time, but first we need to distinguish between two types

always returns current value (clock_timestamp())
always returns current value, unless in a transaction, in which case it returns the value from the beginning of the transaction (now())

Let’s take a look at an example

postgres=# BEGIN;
postgres=# SELECT now();
              now
-------------------------------
 2013-08-26 12:17:43.182331+02

postgres=# SELECT now();
              now
-------------------------------
 2013-08-26 12:17:43.182331+02

postgres=# SELECT clock_timestamp();
        clock_timestamp
-------------------------------
 2013-08-26 12:17:50.698413+02

postgres=# SELECT clock_timestamp();
        clock_timestamp
-------------------------------
 2013-08-26 12:17:51.123905+02

As you can see, clock_timestamp() changes every time the statement is executed, but now() always returns the same value. It’s also worth noting that both of these functions take timezone into account.

Time interval, aka `3.days.ago`

You can easily create time intervals using the interval operator, for example

interval '1 day'
interval '5 days'
interval '5 days' + interval '3 hours'
interval '5 days 3 hours'

As you can see, we can do simple math using the interval operator, which makes it very easy to construct things like 3.days.ago just by doing the following

postgres=# SELECT now() - interval '3 days';
           ?column?
-------------------------------
 2013-08-23 12:23:40.069717+02

Extracting the day of the week and more

Sometimes you just want to know the day of the week for a given date, or the century, or just the day. PostgreSQL has an extract() function which does just this.

Just to put this into context the examples were executed on Monday, August 26.

postgres=# SELECT extract(DAY FROM now());
 date_part
-----------
        26

postgres=# SELECT extract(DOW FROM now());
 date_part
-----------
         1

You can do much more with extract(), for a complete list of examples take a look at the official documentation. Here’s just a few

day
century
dow (day of week)
doy (day of year)
minute
month
year

Converting between timezones

Sometimes it is useful to show a specific date in a different timezone, which is exactly what the AT TIME ZONE construct is for. Let’s take a look at how it works. We’ll do this in a transaction so that the now() function always returns a same value and we can easily see the difference in hours.

postgres=# BEGIN;
BEGIN
postgres=# SELECT now();
              now
-------------------------------
 2013-08-26 12:39:39.122218+02

postgres=# SELECT now() AT TIME ZONE 'GMT';
          timezone
----------------------------
 2013-08-26 10:39:39.122218

postgres=# SELECT now() AT TIME ZONE 'GMT+1';
          timezone
----------------------------
 2013-08-26 09:39:39.122218

postgres=# SELECT now() AT TIME ZONE 'PST';
          timezone
----------------------------
 2013-08-26 02:39:39.122218

PostgreSQL Sequences and Array Column Types

2013-08-24T14:21:00Z

This article is a sequel to our PostgreSQL series, which is aimed to teach you how to get the most out of your database. You might have been led by Active Record (or Rails in general) that it is a good idea to completely abstract away the database, but that it’s hardly ever possible.

If you plan on switching databases on any larger-than-small application, you’re going to have to do some manual work anyway. Every database is special and you should use it as such, instead of just settling for the lowest common denominator.

There are many features in PostgreSQL that can help you develop web applications, such as the array & json column types, hstore, PostGIS and much more. In this article we’re going to take a look at arrays.

Every one of us remembers the moment when you first learned about databases and you were told that if you want to store multiple values in one column you have to split that in a 1:N relationship, because each column can only hold one value. Well, that’s not true in the world of PostgreSQL. You are free to create array columns with arbitrary length. You can even perform array-like queries on them.

But first we need to start by creating a table. We’ll do this with the most obvious example - posts with multiple tags. Each post has a title, content and an arbitrary number of tags. We also want to be able to select all posts with a specific tag.

Let’s first create a new database so that we can play around and drop it at the end of out session, keeping our machine clean.

$ psql -U darth postgres
postgres=# CREATE DATABASE test TEMPLATE template0;
CREATE DATABASE
postgres=# \c test;

Now we can create our posts table.

CREATE TABLE posts (
  id SERIAL PRIMARY KEY,
  title VARCHAR(255) NOT NULL,
  content TEXT NOT NULL,
  tags VARCHAR(255)[] DEFAULT '{}'
);

test=# \d posts
                                  Table "public.posts"
 Column  |           Type           |                     Modifiers
---------+--------------------------+----------------------------------------------------
 id      | integer                  | not null default nextval('posts_id_seq'::regclass)
 title   | character varying(255)   | not null
 content | text                     | not null
 tags    | character varying(255)[] | default '{}'::character varying[]
Indexes:
    "posts_pkey" PRIMARY KEY, btree (id)

But let’s stop here for a moment and take a look at our CREATE TABLE query in more depth. First we’re defining our id column as SERIAL PRIMARY KEY. The PRIMARY KEY is pretty self explanatory, but what about the SERIAL attribute? If you come from the world of MySQL you probably know this as AUTO INCREMENT, but there’s more to it.

CREATE TABLE posts (
  id SERIAL PRIMARY KEY,
  title VARCHAR(255) NOT NULL,
  content TEXT NOT NULL,
  tags VARCHAR(255)[] DEFAULT '{}'
);

PostgreSQL will actually create a SEQUENCE for the auto increment key and use it to generate the default value. The SERIAL keyword here is merely an alias of the following.

test=# DROP TABLE posts;
DROP TABLE

test=# CREATE SEQUENCE posts_id_seq;
CREATE SEQUENCE

test=# CREATE TABLE posts (
  id INT PRIMARY KEY DEFAULT nextval('posts_id_seq'),
  title VARCHAR(255) NOT NULL,
  content TEXT NOT NULL,
  tags VARCHAR(255)[] DEFAULT '{}'
);
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "posts_pkey" for table "posts"
CREATE TABLE

test=# ALTER SEQUENCE posts_id_seq OWNED BY posts.id;
ALTER SEQUENCE

Now you might be asking why in the world would you want to do it the more complicated way, instead of using just a simple SERIAL attribute. What if you wanted the id to increment by 2? Or start at a different index than 0. Well that’s easy to do just by creating a custom SEQUENCE.

test=# CREATE SEQUENCE derp START 1024 INCREMENT 16;
CREATE SEQUENCE

test=# SELECT nextval('derp');
 nextval
---------
    1024
(1 row)

test=# SELECT nextval('derp');
 nextval
---------
    1040
(1 row)

There are many more options you can pass in here, which is why I’d recommend you to take a look at the official documentation for PostgreSQL SEQUENCEs if this sounds interesting. But now back to array columns.

Defining array column types

The way we tell PostgreSQL that a column type is an array is by simply appending [] at the end of the type, such as the following:

CREATE TABLE things (
  id SERIAL PRIMARY KEY,
  names VARCHAR(255)[],
  numbers INT[],
  descriptions TEXT[]
);

In our example code we’re also defining a default value, in this case an empty array, which is defined as '{}' (the single quotes are important here). If we wanted a different default value, we can simply put it in the curly brackets '{thing}'. Note that these arrays columns can also be indexed and thus lighting fast:

CREATE INDEX idx_test on things USING GIN(names);

Now that we know the syntax, we can actually start inserting some data.

test=# INSERT INTO posts (title, content, tags) VALUES ('sample post', 'lorem ipsum', '{apple,orange}');
INSERT 0 1

test=# INSERT INTO posts (title, content, tags) VALUES ('another post', 'fruits are bad', '{apple,hamburger}');
INSERT 0 1

test=# SELECT * FROM posts;
 id |    title     |    content     |       tags
----+--------------+----------------+-------------------
  1 | sample post  | lorem ipsum    | {apple,orange}
  2 | another post | fruits are bad | {apple,hamburger}
(2 rows)

The advantage of doing this over just serializing the tags as a string (for example “apple,hamburger”) is that PostgreSQL knows it’s an array and it can perform operations on it. We can easily select posts with a single tag.

test=# SELECT title FROM posts WHERE 'apple' = ANY(tags);
    title
--------------
 sample post
 another post
(2 rows)

We can also update a specific element of an array (notice that the indexing starts at 1 by default).

test=# UPDATE posts SET tags[1] = 'potato' WHERE id = 1;
UPDATE 1

test=# SELECT * FROM posts;
 id |    title     |    content     |       tags
----+--------------+----------------+-------------------
  2 | another post | fruits are bad | {apple,hamburger}
  1 | sample post  | lorem ipsum    | {potato,orange}
(2 rows)

There’s a lot operations that are supported right of the box, you can see the full list here.

This all about covers the basics of using array columns. There are more things which you can do, for example multi-dimensional arrays, but I would encourage you to take a look at the official PostgreSQL documentation, as it explains everything in details.

If you’re using Rails 4, you can now make use of the array columns without the need for gems like postgres_ext:

class CreateUsers < ActiveRecord::Migration
  def change
    create_table :users do |t|
      t.string :username
      t.string :roles, array: true

      t.timestamps
    end
  end
end

One closing thought, it is important to think before using an array column. This is not a replacement for normalization and you should really think about the way you plan on using the data. If, for example, you wanted to create a dedicated page where you manage all of your tags, assign them to posts and so on, using an array column is probably not the best choice.

The following articles will cover things like JSON column types, hstore, migrating data from one type to another and much more.