Why New Relic?

Purely due to random circumstances at work (and on the side), I hadn't used New Relic in quite some time (~3 years). Recently I took another look and found that their Server monitoring tier was separated from their other offerings - APM, Insights, Mobile, Browser and others.

This isn't how I remembered it, but perhaps the distinction was just in the messaging/marketing. In any case, the exciting part is that the Server monitoring tier is free!

While in terms of your web application, you'll get the most value out of New Relic with their paid offerings, I've found the free Server monitoring to still be an instant life saver, especially when monitoring many (70+) servers.

The free server monitoring tier is a lead-in to make it easier on your to decide to buy into the paid products, of course. Whether or not that stays free remains to be seen!

At work, we run customer trials and paid hosting through AWS. Since the main application is an on-premise application (and not the typical multi-tenant code/database structure), we have a server per application instance. That makes for a lot of running servers!

We've setup Cloudwatch with SNS for alarms and notifications, but it's still hard to use Cloudwatch to get a good overview of all the servers. We decided to try out New Relic on a subset of the servers to see what insights we could find.

We were immediately happy that we did.

Install

Installing the server monitoring daemon is really easy, particularly because they hand you OS-specific instructions on a silver platter when you choose to add a new server within their web console.

For Debian/Ubuntu servers, these instructions are as follows (only really slightly tweaked for those not running as user root):

# Add Repository and Signing Key
echo deb http://apt.newrelic.com/debian/ newrelic non-free | sudo tee /etc/apt/sources.list.d/newrelic.list
wget -O- https://download.newrelic.com/548C16BF.gpg | sudo apt-key add -

# Update repos and install their package
sudo apt-get update
sudo apt-get install newrelic-sysmond

# Configure the local install with your license key:
sudo nrsysmond-config --set license_key=1234567890abcdefghijklmnopqrstuvwxyz

# Start the service:
sudo service newrelic-sysmond start

Their instructions will conveniently give you your license key. Don't blindly copy and paste the license key above ?.

Configure

You technically don't need any more configuration, but I do like to change one more thing. The server name as shown in New Relic will match the server's host name, which you can find by running the command hostname.

This might not be the way you want to identify it, so you can either wait for the server to appear in New Relic, or configure the hostname ahead of time in the New Relic configuration installed on your server. I choose the latter approach.

Edit the /etc/newrelic/nrsysmond.cfg file and change the hostname section to whatever is appropriate for your server.

# You'll find your license key in here, which you can set manually
# instead of running the `nrsysmond-config` command
license_key=1234567890abcdefghijklmnopqrstuvwxyz

# Find the hostname entry and set it
hostname=some-server.example.com

After making any changes, start (or restart) the newrelic-sysmond service:

sudo service newrelic-sysmond restart

You'll soon see that server appear in your New Relic account under the Server section, without any further work on your part.

What We Watch

This monitoring on New Relic isn't super advanced, but it has some really nice features:

Processes and process count.

Note that PHP is a bit out of control there - that was one thing that we were alerted to, as the server has a high CPU usage over their default threshold of 15 minutes.

We found some zombie PHP processes, which we killed off quickly. We immediately saw CPU reduction back to normal levels.

# Find a process
$ ps aux | grep php
www-data  5889  0.0  3.3 438856 34004 ?        S    Jan12   0:17 php: /path/to/file.php

# Kill it based on process ID
$ sudo kill 5889

We can get a more detailed view of each process type as well, sorting by memory and cpu usage:

Disk Usage

New Relic also gave us alerts about disk usage. One server had excess files taking up a lot of space. We were able to reduce the disk spaced used by manually deleting extra media files, giving us time to provision more hard drive space.

Alerts

Alerts may not send you alerts off the bat, but can be easily configured.

In any case, when viewing a list of all your servers, you will see some on the sidebar worth paying attention to.

Automated with Ansible

I automated this installation with a super simple Ansible Playbook (specific to Debian/Ubuntu). The basics are as follows:

File tasks/main.yml:

---
- name: Add New Relic Repository
  apt_repository:
    repo: deb http://apt.newrelic.com/debian/ newrelic non-free
    state: present

- name: Add New Relic Signing Key
  apt_key:
    url: https://download.newrelic.com/548C16BF.gpg
    state: present

- name: Update Repositories
  apt:
    name: newrelic-sysmond
    update_cache: yes
    state: present

- name: Add New Relic Config
  template:
    src: nrsysmond.cfg.j2
    dest: /etc/newrelic/nrsysmond.cfg
    owner: newrelic
    group: newrelic

- name: Restart New Relic
  service:
    name: newrelic-sysmond
    state: restarted

File templates/nrsysmond.cfg.j2, with 2 variables:

#
# Option : license_key
# Value  : 40-character hexadecimal string provided by New Relic. This is
#          required in order for the server monitor to start.
# Default: none
#
license_key={{ new_relic_key }}

# Much taken out for brevity....

#
# Setting: hostname
# Type   : string
# Purpose: Sets the name of the host (max 64 characters) that you wish to use
#          for reporting. This is usually determined automatically on startup
#          but you may want to change it if, for example, you have machine
#          generated hostnames that are not visually useful (for example, the
#          names generated by Docker containers).
# Default: The system configured host name
#
hostname={{ domain }}

# Much taken out for brevity....

Then you can set your defaults or vars/main.yml file with your variable definitions:

new_relic_key: 1234567890abcdefghijklmnopqrstuvwxyz
domain: whatever.example.com

When using Ansible for this, I actually provided the domain variable has a host variable, included in the hosts inventory file for each host being updated.