Webfaction Elasticsearch Server Step-by-Step Tutorial

Elasticsearch server on Webfaction using NGINX with basic authorization and HTTPS protocol

The following step-by-step instructions will set you up with a secure standalone elasticsearch server installed on Webfaction. When we’re done you will have:

  • Webfaction custom application (~/webapps/elasticsearch) with elasticsearch downloaded and configured.
  • A cronjob that keeps elasticsearch up and running.
  • Webfaction NGIX application (~/webapps/elastic_access) that puts basic authorization in front of the elasticsearch index.
  • SSL certificate security on the access point application to encrypt the data.
  • Webfaction custom application (~/webapps/redirect_to_https) to redirect HTTP traffic to the HTTPS site.

This is going to be a long post with lots of steps but it’s useful to have them all laid out in one place.

 

Webfaction plan

Make sure the plan you have with Webfaction has enough resources. I added a new 1G shared plan just for elasticsearch. This was after blowing through the memory allocation on an existing plan when I started loading data. (Whoops!) For the purposes of this tutorial, I’m going to refer to this as the “elasticsearch server”.

A note about shared plans. Rather than letting elasticsearch use the defaults it detects for the entire server, you may need to configure settings like the number of available processors to fit within the plan’s limits. Specific example below.

 

Install elasticsearch

The steps to installing elasticsearch on Webfaction are:

  1. install/update the Java JRE
  2. add a Webfaction custom application with an open port (PORTNO)
  3. download and install elasticsearch in this new application
  4. set-up a cron job to keep elasticsearch up and running.

This agileadam post runs through the steps in detail so I’m not going to repeat them here.

Some notes on my experience:

  • Java is now on version 8 so I downloaded jre-8u102-linux-x64.tar.gz using

    wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn/java/jdk/8u102-b14/jre-8u102-linux-x64.tar.gz

    By the time you’re reading this, the version will probably have updated again. The key is to remember that you want the linux-x64 version and to double check that you have the the minimum version required by elasticsearch. You might find this page or this page on Oracle helpful in your hunt for the right version.

  • For this tutorial we’re working with elasticsearch 2.x which as of this writing is 2.4.1

    wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.1/elasticsearch-2.4.1.tar.gz

  • The changes I made to ~/webapps/elasticsearch/config/elasticsearch.yml are:

    This gives the cluster a more descriptive name and places the actual index files in their own location.

  • Depending on your Webfaction plan, you many need to configure the number of available processors in the ~/webapps/elasticsearch/config/elasticsearch.yml file. (For now, I have the number set to 2 which probably lower that it needs to be.)

  • After you set-up the crontab entry as Adam describes, make sure it’s running and then you’ll probably want to comment out the “elasticsearch is running.” line before you get flooded with messages.

 

Verify elasticsearch is okay

On the elasticsearch server, run:

Where PORTNO is the elasticsearch http.port number.

You should see something like this:

 

Add outside access using NGINX and basic authorization

Since this is as a standalone elasticsearch server, we need to set-up access from the outside using NGINX. For security we’re also going to add basic authorization.

  • A Passenger application seems to be the cleanest way to create an NGINX application. Make sure you’re adding the application to the correct Webfaction web server. And the Passenger source can just be left there for now since it’s not in the way.
    1. On webfaction.com, create a new Passenger application named elastic_access
      (~/webapps/elastic_access).
    2. Assign this to a domain and website. For this tutorial, this will be es.example.com
    3. Review the cronjob line added for this application. It should be something like:
      2,22,42 * * * * /home/<username>/webapps/elastic_access/bin/start
  • Add a user and password that will be allowed access. I would suggest a more obscure username but for now we’re going to use elastic_user

    From ~/webapps/elastic_access:

    NOTE: Make the password unique and really secure and then write it down on a scrap of paper if you need to. No point in setting this up if the username and password are easy to guess!

  • Edit ./nginx/conf/nginx.conf to pass the user’s request to ~/webapps/elasticsearch

    Where PORTNO is the elasticsearch http.port number.

  • Make sure to restart NGINX every time you edit the conf file

    ./bin/restart from ~/webapps/elastic_access

 

Test the outside access

From a server other than the elasticsearch server, run:

You should get an authorization error something like this:

Next run:

Enter the password and you should see the elasticsearch cluster info.

 

Add an SSL certificate to for encryption

If you’ve never used an SSL certificate before, I’d recommend spending some time in Google researching just what it is and how it’s used. The documentation will make more sense if you understand what they’re referring to.

This refactorsaurus rex post is a good starting point but there have been some changes at Webfaction since it was written. For example, no need to open a support ticket!

  • Purchase an SSL certificate

    NOTE: I ended up using SSLMate to buy the certificate since it seemed reasonable. At this point I don’t have enough experience with SSL certificates to say if that’s a good thing or not. There are other vendors and ways to acquire a certificates. The point is to get one for the elasticsearch server and then move on to the next step, adding the certificate to Webfaction.

    Since I was starting with a new set-up, first I needed to install SSLMate. Mostly I followed along with Nick’s blog post and SSLMate’s docs and in the end I had sslmate installed in ~/share/sslmate with a certificate purchased for es.example.com. When I was done, running sslmate list listed my new certificate and sslmate test es.example.com returned a status of ‘Good’.

    Be sure to keep track of the private key since SSLMate (or whichever vendor you use) is not going to save it (and if they do, find another vendor!)

  • Add the certificate to your Webfaction account under Domains/Websites/SSL Certificates. Webfaction provides some pretty detailed documentation.

    I used the Cut and Paste method and I have to confess it took me a few tries to get the right data in the right field. For example, what SSLMate labels as ‘chain’ is the ‘intermediates/bundle’ that Webfaction is looking for. And this is when you’ll be happy you have your private key handy (don’t ask me how I know!)

  • On webfaction.com, update the es.example.com website to use HTTPS and assign the correct certificate.

    In the security section, click on Encrypted website (https) and select the certificate you just added.

    In the domains field, you’ll probably want to add www.es.example.com as well as es.example.com.

  • Since there is no HTTP access, only HTTPS, we need to add a redirect.

    Follow Webfaction’s step-by-step directions for redirecting from HTTP to HTTPS.

    1. Create a static application named redirect_to_https
    2. Create a website for http://es.example.com that connects to the redirect_to_https application
    3. In ~/webapps/redirect_to_https, add the .htaccess file:
    4. Add the file ~/webapps/redirect_to_https/index.html containing the line “hello world” to act as a default.

 

Test that HTTPS and the redirect are working

To test the redirect, from a server other than the elasticsearch server, run:

You should get a 301 redirect message something like this:

And running this:

Enter the password and you should see the elasticsearch cluster info.

 

(bonus) Accessing the elasticsearch server from a Rails application

  • Set the environment variables needed to define the elasticsearch server. If you’re using the figaro gem for example, in config/application.yml, you would have:

  • Create an initializer to make the connection on start-up. Create the file ./config/initializers/elasticsearch.rb:

  • Try it out with the Rails console:

 

And we have a standalone elasticsearch server running on Webfaction!

  • If you run into problems, some helpful queries to get started trouble shooting: