Scott Frederick's humble blog

on sofware development and technology

Cloud Foundry and Logstash

Update 2014-4-15: Changed ELK installation instructions to install logstash 1.4, elasticsearch 1.1, and kibana 3.

Cloud Foundry has the ability to capture logs from several platform components and from applications running on the platform, aggregate these logs, and export the data to an external log management and analysis system. You can read more about this in the Cloud Foundry documentation.

Logstash is one of the log management systems that Cloud Foundry can work with, but some configuration is required to get logstash to consume and understand Cloud Foundry logs. This post shows how to configure logstash for use with Cloud Foundry. Detailed instructions for installing, configuring, and using logstash as well as elasticsearch and kibana (collectively known as ELK) on Ubuntu Linux are also provided.

Configuring logstash for Cloud Foundry

The Cloud Foundry Loggregator component formats logs according to the syslog standard as defined in RFC5424. The logstash cookbook includes an example configuration for syslog consumption, but that configuration follows an older RFC3164 syslog standard.

Here is a logstash configuration that works with RFC5424 output, with some additional changes for Cloud Foundry:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}" }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
  }
}

output {
  elasticsearch { }
}

The important difference between this configuration and the one in the logstash cookbook is in line 15, which configures the parsing of log records according to RFC5424. With this configuration, these logstash event fields will be populated:

  • syslog5424_host will always be loggregator, indicating that the log entry came from Cloud Foundry
  • syslog5424_app will contain the GUID for the Cloud Foundry application that generated the log entry
  • syslog5424_proc will contain the Cloud Foundry component that generated the log entry, with values like:
    • [DEA] – logs from the DEA
    • [STG] – logs from the appliation staging process
    • [RTR] – logs from the Router
    • [App/n] – logs from an application, with n designating the instance index
  • syslog5424_msg will contain the log message text

This parsing configuration is almost the same as the built-in RFC5424 parser for logstash, with a few important differences:

  • the default logstash parsing for syslog5424_app allows only alpha, numeric, and underscore characters, but Cloud Foundry sets this field to a GUID which contains - characters
  • the default logstash parsing for syslog5424_proc allows only alpha, numeric, and underscore characters, but Cloud Foundry can include a / character

With this configuration, you can follow the instructions in the Cloud Foundry documentation to create a user-provided log draining service and bind the service to an application. The configuration above tells logstash to listen on port 5000, so the user-provided service creation and binding might look something like this:

1
2
3
$ cf cups logstash-drain -l syslog://[logserver]:5000
$ cf bind-service [app-name] logstash-drain
$ cf restart [app-name]

where [logserver] is the name or IP address of the server where logstash is running and [app-name] is the name of an application running on Cloud Foundry.

If you are knowledgable in logstash installation and configuration, this should be enough to allow your logstash system to consume logs from Cloud Foundry. For a quick and easy setup of logstash, elasticsearch, and kibana for log management and analysis, read on.

Installing ELK on Ubuntu 12.04

Downloading and installation instructions for the ELK stack can be found here. Detailed instructions for Ubuntu 12.04 are provided below for the impatient. Adjust as necessary for other operating systems or Linux distributions by following the instructions for installing and configuring each component.

The easiest way to install elasticsearch, logstash, and kibana on Ubunutu is to add the ELK repositories to your local apt configuration and install using apt-get.

Configuring apt

First, add the elasticsearch public GPG keys to your local apt configuration:

1
$ wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -

Edit your local /etc/apt/sources.list file and add these two lines to the end of the file to make the elasticsearch apt repositories visible to your system:

1
2
deb http://packages.elasticsearch.org/elasticsearch/1.1/debian stable main
deb http://packages.elasticsearch.org/logstash/1.4/debian stable main

Now update apt with the changes:

1
$ sudo apt-get update

apt should now be updated and ready to install ELK components.

Installing elasticsearch

Installing and starting elasticsearch is very simple. elasticsearch requires a Java Runtime Environment, so install that before elasticsearch:

1
2
3
$ sudo apt-get install openjdk-7-jre
$ sudo apt-get install elasticsearch
$ sudo service elasticsearch start

The default elasticsearch configuration should work without modification.

Installing and configuring logstash

Installing logstash is just as simple:

1
$ sudo apt-get install logstash

logstash needs a configuration file that instructs it where to get its inputs from and where to send its outputs. Copy the contents of the logstash configuration file shown at the beginning of this post and paste it into a file named /etc/logstash/conf.d/syslog.conf.

Then start the logstash process:

1
$ sudo service logstash start

Installing and configuring kibana

The kibana web application runs in an Apache httpd web server. Install Apache httpd, download the kibana HTML and JavaScript files, and copy them into the httpd document root directory:

1
2
3
4
5
6
$ sudo apt-get install apache2
$ cd /var/www
$ sudo wget https://download.elasticsearch.org/kibana/kibana/kibana-latest.tar.gz
$ sudo tar zxf kibana-latest.tar.gz
$ sudo mv kibana-latest kibana
$ sudo service apache2 start

You should now be able to point your browser to a URL like http://logserver/index.html#/dashboard/file/logstash.json, where logserver is the name or IP address of the server where kibana was installed. You should see a logstash dashboard running in kibana. The dashboard will be empty until you create a user-provided log drain service and bind it to an application.

Creating and binding a log drain service

As stated earlier in this poast, you can follow the instructions in the Cloud Foundry documentation to create a user-provided log draining service and bind the service to an application using commands like these:

1
2
3
$ cf cups logstash-drain -l syslog://[logserver]:5000
$ cf bind-service [app-name] logstash-drain
$ cf restart [app-name]

where [logserver] is the name or IP address of the server where logstash is running and [app-name] is the name of an application running on Cloud Foundry.

Custom PropertySource in Spring 3.1 - Part 3

This post is the third in a series of posts showing examples of registering a custom property source in Spring 3.1, where the property source requires non-trivial initialization of its own. Part 1 showed initialization of a stand-alone application, and Part 2 showed initialization of integration tests. This part will show how to initialize a custom property source in a Spring MVC web application.

Custom PropertySource in Spring 3.1 - Part 1

One of the improvements made to the Spring Framework in the 3.1 release is in the area of property placeholders. Chris Beams explains how the new unified property management system works in a blog posting on the SpringSource community site.

Several implementations of the new PropertySource interface are provided with the framework. PropertySource implementations backed by Java system properties and operating system environment variables are registered by default. If you specify one or more properties files using the <context:property-placeholder> tag in an XML configuration file or using the @PropertySource annotation on a Java config @Configuration class, the framework registers a PropertySource implementation backed by the specified files.