How to Deploy a Rails App To an Elastic Beanstalk Web Server and Worker Tier

technology advice from elevatepoint
Learn more about how to deploy a Rails app to an Elastic Beanstalk web server and worker tier.

Share This Post

Share on facebook
Share on linkedin
Share on twitter
Share on email

[vc_row][vc_column][vc_column_text]We recently shipped a Rails SAAS product hosted on Elastic Beanstalk, Amazon’s 1-Click Web App Deployment service. While I had worked with EC2 instances several times on previous applications, setting up and configuring a load balanced, scalable environment was something I personally hadn’t done. Elastic Beanstalk aims to take a fair amount of the setup of scalable environments out of the way, to allow for more focus on the software development. But there were a number of steps I had to figure out on my own. So I wanted to share the majority of the steps we took to deploy a Rails app to both a web server and worker tier.

Our needs

Not only did we want the ability to scale up application servers as needed, we also needed the ability to process jobs in the background as well as facilitate recurring tasks, such as billing.

For recurring tasks in a Unix environment, cron is the way to go. The problem is that cron jobs do not work in a distributed environment. If you added a task to an EC2 instance, and it scaled, you’d suddenly have multiple tasks running. Apparently customers don’t appreciate being billed multiple times.

To meet those needs, we were able to setup a Web Server tier that would serve public requests to the application, and a Worker tier that would facilitate periodic tasks as well receive messages from an Amazon SQS message queue to process tasks in the background on demand.

Tools

AWS provides a CLI client for Elastic Beanstalk as well as a web UI through console.aws.amazon.com to provision and configure Elastic Beanstalk.

We opted to use the eb_deployer Ruby gem. It provides the ability to provision multiple environments, configure Blue-Green deployments, rake tasks to execute deployments, and to create CloudFormation templates. Additionally, it didn’t require the entire dev team to install additional clients, and all the configuration would be checked into source control.

Step 1: Set up your AWS account

I won’t go through all the details here, but you’ll need to sign up for an AWS account.  Once your account is created, create a separate IAM account and generate an Access Key.  You can store your Access Key ID and Secret Access Key in .bashrc, .zshrc, or whatever your shell’s configuration file is as environment variables like this:

     export AWS_ACCESS_KEY_ID=MYACCESSKEYID
     export AWS_SECRET_ACCESS_KEY=MYSECRETKEY

If you don’t want to put those keys in your config file, you can install and use the AWS CLI to store these creds as well.

$     aws configure

This will prompt you for your keys.

With your creds in place, you won’t have to reference them in code anywhere.

Step 2: Install eb_deployer

Add the eb_deployer to your app’s Gemfile, and run:

 $    rails generate eb_deployer:install

This creates a few boilerplate templates that are to be customized as well as an .ebextensions directory.

The two most important files generated are config/eb_deployer.yml and config/rds.json which we’ll discuss in the next step.

Step 3: Configure eb_deployer.yml

eb_deployer.yml has a number of settings, and is well documented.

The main areas to modify are the option_settings, resources, and environments.

Environments

This is a pretty straight forward section. As many environments as you want (i.e. production, ci, staging, qa, dev) you can declare here. Each environment can override any of the general option settings set in the rest of the file.

Option_settings

This is where you can set the type of EC2 instance, set environment variables, configure HTTP ports, and pretty much set any option available via Amazon’s API.

I recommend creating and setting an IAM Role as well as setting up a Key Pair so you can access the instances once they are running:

    - namespace: aws:autoscaling:launchconfiguration
      option_name: EC2KeyName
      value: MyKeyName
    - namespace: aws:autoscaling:launchconfiguration
      option_name: IamInstanceProfile
      value: MyRoleName

If you are going to be using SSL, add your SSL cert to AWS, and you can use settings like this:

    - namespace: aws:elb:loadbalancer
      option_name: LoadBalancerHTTPPort
      value: "443"
    - namespace: aws:elb:loadbalancer
      option_name: LoadBalancerPortProtocol
      value: HTTPS
    - namespace: aws:elb:loadbalancer
      option_name: SSLCertificateId
      value: arn:aws:iam::AccountID:server-certificate/MySSL.crt

 Resources

This section is about setting input values that will be used in the CloudFormation template (by default the config/rds.json file) and capturing output values.

eb_deployer creates a separate RDS instance for each environment. This means we can build up and tear down any Elastic Beanstalk environment without losing our persistence layer.

Out of the box, eb_deployer will create a Postgresql RDS instance. We’ll also need a few other resources along with the database layer, namely two SQS queues (a message queue and a dead letter queue), and a DynamoDB table.

The message queue is how we will send messages to the worker tier to process various jobs. The DynamoDB table is what will store our periodic tasks.

The entire resources section should look like this:

  resources:
    # Creating a RDS instance by CloudFormation, so that database will not hook up with any ElasticBeanstalk environment
    template: config/rds.json
    capabilities:
      - CAPABILITY_IAM
    inputs:
      env: <%= environment %>
      QueuePrefix: MyMessageQueue
      DBName: db_name
      DBUser: db_user_name
      DBPassword: db_password
    outputs:
      RDSPassSecurityGroup:
        namespace: aws:autoscaling:launchconfiguration
        option_name: SecurityGroups
      RDSHost:
        namespace: aws:elasticbeanstalk:application:environment
        option_name: DATABASE_HOST
      RDSPort:
        namespace: aws:elasticbeanstalk:application:environment
        option_name: DATABASE_PORT
      MessageQueueName:
        namespace: aws:elasticbeanstalk:application:environment
        option_name: MESSAGE_SQS_NAME
      MessageQueueUrl:
        namespace: aws:elasticbeanstalk:application:environment
        option_name: MESSAGE_SQS_URL
      DeadLetterQueueName:
        namespace: aws:elasticbeanstalk:application:environment
        option_name: DEAD_LETTER_SQS_NAME

Elastic Beanstalk Worker Tiers will create a queue by default unless you specify one. I like to create them so I can have meaningful names associated with them. In order to specify the name, we need to add the inputs for the environment (env) and the queue prefix (QueuePrefix). Then, in the outputs section, we capture the queue names and urls which will be stored as environment variables.

One last useful modification was to set the version_label. Since we use Git tags for each release, it made sense to reference those tags as the version being deployed.

version_label: <%= `git describe`.strip %>

 

Step 4: Update config/rds.json

This file can be named anything, and doens’t have to be limited to just RDS instance creation.

This JSON file will be the basis for the CloudFormation template. This is where eb_deployer is better than simply using the eb CLI or web interface. It allows us to store the resource settings in a template that can be used to provision and manage the resources for each environment.

CloudFormation templates can be pretty involved, but the have a few main sections.

  1. Resources – this is where you specify the various AWS products and settings (i.e. RDS, SQS, EC2, ELB)
  2. Parameters – These relate to the input parameters we set in eb_deployer.yml
  3. Outputs – These are outputs that are also referenced in eb_deployer.yml
  4. Mappings – When you need to specify values conditionally, you create mappings to inform CloudFormation.

Click here for a gist of the rds.json file.

The primary thing to note is that we generate queues based on the input values and provide output values that get attributes such as the queue name

Step 5: Specify Components in eb_deployer.yml

Usually Elastic Beanstalk applications have separate application code in the Web Server tier from the Worker tier, which makes good sense. In our situation, we didn’t want to invest in making architectural decisions to split out services. It was more important to ship the product and get feedback to begin iterating on the features. So we have a monolithic Rails application that needs to be deployed to both the web server and worker tiers.

To do that, we took advantage of components in eb_deployer. Components are an undocumented feature of eb_deployer. They allow you to identify components that need to be deployed to a given environment. You might have a web component and an API component, or in our case a bg component. Each component will deploy the same code to their respective containers. But components also allow additional customization beyond the global and environmental options.

As a top level section we added the following components:

  components:
    - name: web
    - name: bg
      tier: Worke
      option_settings:
        - namespace: aws:elasticbeanstalk:application:environment
          option_name: APP_MODE
          value: background
        - namespace: aws:elasticbeanstalk:sqsd
          option_name: HttpPath
          value: /messages
        - namespace: aws:elasticbeanstalk:sqsd
          option_name: WorkerQueueURL
          value: <%= "https://sqs.us-east-1.amazonaws.com/MyAccountId/MyQueue-#{environment}" %>

For the web component, we didn’t need to change any settings, other than to provide a name. For the bg component, we needed to indicate the tier, set an environment variable so we could know in application code if this was a background environment or not. Lastly, we need to tell Elastic Beanstalk the end point that the message queue would post messages to, and the queue url.

Step 6: Add cron.yaml

At the root of your Rails app, create a cron.yaml file. This file will store all the periodic tasks you want to schedule. The format is pretty simple.

cron:
  - name: “first-periodic-task“
    url: “/periodic-tasks/first-task"
    schedule: "*/15 * * * *"
  - name: “second-periodic-task“
    url: “/periodic-tasks/second-task"
    schedule: "0 0 * * *"

Based on the schedule indicated for each task, a POST request will be sent to the Worker tier at the indicated end point.

The name and urls need to be unique across this file.

Step 7: Install the Postgres development headers

In addition to install the postgres lib on each EC2 instance, we’ll need to install the development headers so the pg gem will install.

Simply edit .ebextensions/01_postgres_packages.config to look like:

---
packages:
  yum:
   postgresql94-libs: []
   postgresql94-devel: []

 

Step 8: Deploy

With this in place, commit your changes, and run

$     rake eb:deploy

This will package up the application, send it to S3, and begin to provision all the resources indicated. It will take a good amount of time on the first deploy.

Once it’s finished, it will have created a single EC2 instance for the web and bg components, configured the web and ruby server, load balancers, scaling triggers, RDS instance and migrated the database to the latest schema, scheduled the periodic tasks

You should be able to log into the AWS web console and see the application and various environments running.

To deploy to a different environment (e.g. production) simply run:

$     EB_DEPLOYER_ENV=production rake eb:deploy

Concluding Thoughts

  1. Elastic Beanstalk w/ eb_deployer can create a scalable environment fairly quickly
  2. Like any “1-click” solution, it’s not exactly 1-click, and there are some limitations that are to be expected for a pre-configured environment
  3. We don’t plan on utilizing Elastic Beanstalk forever. As our service matures, and hopefully grows, we’ll undoubtedly move to a more customized, sophisticated environment.
  4. Elastic Beanstalk along with eb_deployer provided just enough customization with enough convention to allow to focus on developing the application and have enough confidence in the infrastructure to handle the early growth stages of our service.

[/vc_column_text][/vc_column][vc_column][us_cta title=”Contact us about more tech help” btn_link=”url:%2Fcontact-us|||” btn_label=”Get Help” btn_style=”4″ btn_size=”15px”][/us_cta][/vc_column][/vc_row]

More To Explore

remote worker
Digital Workplace

How to Work Remotely

Working remotely is hard for some people. We have ideas to make it easier.

We can help

teams and teamwork