Introduction

Performance Co-Pilot or PCP is an open source toolkit designed for monitoring and managing system-level performance. These services are distributed and scalable to accommodate the most complex system cofigurations and performance issues.

PCP consists of two components :

• PCP Collectors, this component collects and extracts performance data from various sources, e.g: kernel and database.
• PCP Monitors, this component displays data collected from hosts or archive that have PCP collector installed. Some monitoring tools are available as part of PCP.

You can learn more about pcp from PCP website

Vector is an open source on-host performance monitoring framework which exposes hand picked high resolution system and application metrics to every engineer’s browser. Having the right metrics available on-demand and at a high resolution is key to understand how a system behaves and correctly troubleshoot performance issues.

Basically we will use pcp to gather metrics for us, let pcpweb expose the data and use Vector to show us nice graph of server performance so we can monitor server performance in real time. This is really useful especially when troubleshooting a problem on server. You can learn more about Vector from https://github.com/Netflix/vector.

In this tutorial we’ll learn how-to install PCP to gather system performance metrics and use Netfix’s Vector to graph the data. We will install the tools on Ubuntu 14.04 LTS.

Network Requirements

PCP components will listen on ports below :
* 44321, used by pmcd
* 44322, used by pmproxy
* 44323, used by pmwebd
* 4430, used by pmlogger

Update Base Systems

If this is a new server make sure that you update the system to latest system by running command below :

$ sudo apt-get update
$ sudo apt-get -y upgrade

If this is a production machine, please recheck whether it’s ok to update the system to get the latest update because sometimes updating packages can break applications running on the server.

There are two repositories that we can use to install PCP. The first repository is from PCP repository and the second repository is from netflix repository. We’ll learn how-to install from both method but we need to choose and use only one of them.

Install PCP from PCP Repository

In this section we’ll learn how-to install PCP from PCP repository. Let’s add PCP package key so apt can verify package integrity.

$ curl --silent 'https://bintray.com/user/downloadSubjectPublicKey?username=pcp' | sudo apt-key add -

Add PCP Repository

$ echo "deb https://dl.bintray.com/pcp/trusty trusty main" | sudo tee -a /etc/apt/sources.list.d/pcp.list > /dev/null

Update apt metadata

$ sudo apt-get update

Install PCP and pcp-webapi

$ sudo apt-get install pcp pcp-webapi

To check pcp service status we can run command below :

$ sudo service pcp status
Checking for pmcd: running
Checking for pmlogger: running

From output above we see that pmcd and pmlogger is running. You can also check status for pmcd itself using command below :

$ sudo service pmcd status
Checking for pmcd: running

To check pmlogger status :

$ sudo service pmlogger status
Checking for pmlogger: running

To check pmproxy service status you can run command below:

$ sudo service pmproxy status
Checking for pmproxy: running

The last service that you need to check is pmwebd that run pcp web api. You can check this process using command below.

$ sudo service pmwebd status
Checking for pmwebd: running

Let’s also check whether pcp related service already listen to required ports :

$ sudo netstat -ltp | grep "/p"
tcp        0      0 *:44321                 *:*                     LISTEN      1169/pmcd       
tcp        0      0 *:44322                 *:*                     LISTEN      1375/pmproxy    
tcp        0      0 *:44329                 *:*                     LISTEN      14692/pmwebd    
tcp        0      0 *:4330                  *:*                     LISTEN      13181/pmlogger  
tcp6       0      0 [::]:44321              [::]:*                  LISTEN      1169/pmcd       
tcp6       0      0 [::]:44322              [::]:*                  LISTEN      1375/pmproxy    
tcp6       0      0 [::]:44329              [::]:*                  LISTEN      14692/pmwebd    
tcp6       0      0 [::]:4330               [::]:*                  LISTEN      13181/pmlogger

Change PCP pmwebd port

Vector will connect to the pmwebd port to get data from pcp. You might need to change the port to match network policy on your environment. In this section we’ll learn how-to change pmwebd port.
Open /etc/pcp/pmwebd/pmwebd.options file. find line below :

OPTIONS="$OPTIONS -v"

Below the line that add verbosity above, add these lines :

PORT=44329
OPTIONS="$OPTIONS -p $PORT"

After setting the port, restart pmwebd service

$ sudo service pmwebd restart

to check whether the new port already applied or not we can use netstat :

$ sudo netstat -naptu | grep pmwebd
tcp        0      0 0.0.0.0:44329           0.0.0.0:*               LISTEN      14692/pmwebd    
tcp6       0      0 :::44329                :::*                    LISTEN      14692/pmwebd    

Install PCP using Netflix repository (alternative method)

This section covers an alternative method to install PCP using the Netflix repository. If you already followed the steps on the previous section you don’t need to follow the steps in this section.

Add Netfix package key for package verification –

$ curl 'https://bintray.com/user/downloadSubjectPublicKey?username=netflixoss' | sudo apt-key add -

Add Netflix repository

$ echo "deb https://dl.bintray.com/netflixoss/ubuntu trusty main" | sudo tee -a /etc/apt/sources.list.d/pcp.list

Update apt metadata

$ sudo apt-get update

Install PCP and pcp-webapi

$ sudo apt-get install pcp pcp-webapi

You can see section above on how-to check pcp related services.

How-to Install Vector On Local Computer

We will use Vector that’s already built. If you want to learn how to build vector from source code you can read Netflix blog post about Vector. Vector is simply a html+js+css app that you can just run from your browser. Every engineer can download Vector archive and run it on their workstation, but you can also serve Vector from a webserver like Nginx or Apache httpd.

In this tutorial we’ll only learn how-to use Vector from local computer.

Download Vector archive from netfixoss repository, at the time of this writing the latest version of Vector is 1.0.3. You might want to check Vector site on GitHub to get the latest version of Vector.

Download Vector archive.

$ wget https://bintray.com/artifact/download/netflixoss/downloads/1.0.3/vector.tar.gz

Create a directory to extract vector, we need to create this since vector build is not placed in a directory.

$ mdkir vector

Extract vector archive to the folder that we created previously.

$ tar xzf vector.tar.gz -C vector

Go to vector folder and open index.html using your browser and Vector is ready. Since we haven’t connected to any pcpweb api, Vector will give blank graphs.

Enter hostname or ip address of the server that you want to monitor. If you change pmwebd port you need to specify hostname:port or ipaddress:port. The performance graphs will start showing the current condition of your system

Please note that the connection between Vector and pmwebd is from your computer to the server that you want to monitor even if you host Vector on a webserver. Make sure that your server allow pmwebd port 44323

Summary

In this tutorial we covered how to install Performance Co-Pilot (PCP) on Ubuntu Server 14.04 and also install Vector on our local computer. We then used Vector to connect to pcpwebapi so that we can start monitoring our server.