How to Securely Monitor Remote Servers Using Zabbix on CentOS 7

Zabbix is a free and open source enterprise-ready software used to monitor the availability of systems and network components. Zabbix can monitor thousands of servers, virtual machines or network components simultaneously. Zabbix can monitor almost everything related to a system such as CPU, memory, disk space and IO, processes, network, databases, virtual machines, and web services. If IPMI access is provided to Zabbix then it can also monitor the hardware such as temperature, voltage and so on.

Prerequisites

• A Vultr CentOS 7 server instance.
• A sudo user.

For this tutorial, we will use 192.0.2.1 as the public IP address of the Zabbix server and 192.0.2.2 as the public IP address of a Zabbix host which we will monitor remotely. Please make sure to replace all occurrences of the example IP address with your actual public IP addresses.

Update your base system using the guide How to Update CentOS 7. Once your system has been updated, proceed to install the dependencies.

Install Apache and PHP

Upon installation of Zabbix web, it automatically creates the configuration for Apache.

Install Apache to serve the Zabbix front-end or the web UI.

sudo yum -y install httpd

Start Apache and enable it to start at boot automatically.

sudo systemctl start httpd
sudo systemctl enable httpd

Add and enable the Remi repository, as the default YUM repository contains an older version of PHP.

sudo rpm -Uvh http://rpms.remirepo.net/enterprise/remi-release-7.rpm
sudo yum -y install yum-utils
sudo yum-config-manager --enable remi-php71

Install the latest version of PHP along with the modules required by Zabbix.

sudo yum -y install php php-cli php-gd php-bcmath php-ctype php-xml php-xmlreader php-xmlwriter php-session php-sockets php-mbstring php-gettext php-ldap php-pgsql php-pear-Net-Socket

Install and Configure PostgreSQL

PostgreSQL is an object-relational database system. You will need to add the PostgreSQL repository in your system, as the default YUM repository contains an older version of PostgreSQL.

sudo rpm -Uvh https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Install the PostgreSQL database server.

sudo yum -y install postgresql96-server postgresql96-contrib

Initialize the database.

sudo /usr/pgsql-9.6/bin/postgresql96-setup initdb

initdb creates a new database cluster, which is a group of databases managed by a single server.

Edit the pg_hba.conf to enable MD5 based authentication.

sudo nano /var/lib/pgsql/9.6/data/pg_hba.conf

Find the following lines and change peer to trust and idnet to md5.

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            idnet
# IPv6 local connections:
host    all             all             ::1/128                 idnet

Once updated, the configuration should look like shown below.

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

Start the PostgreSQL server and enable it to start automatically at boot.

sudo systemctl start postgresql-9.6
sudo systemctl enable postgresql-9.6

Change the password for the default PostgreSQL user.

sudo passwd postgres

Login as the PostgreSQL user.

sudo su - postgres

Create a new PostgreSQL user for Zabbix.

createuser zabbix

Switch to the PostgreSQL shell.

psql

Set a password for the newly created database user for the Zabbix database.

ALTER USER zabbix WITH ENCRYPTED password 'StrongPassword';

Create a new database for Zabbix.

CREATE DATABASE zabbix OWNER zabbix;

Exit from the psql shell.

\q

Switch to the sudo user from the current postgres user.

exit

Install Zabbix

Zabbix provides binaries for CentOS, which can be installed directly from the Zabbix repository. Add the Zabbix repository to your system.

sudo rpm -ivh http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-1.el7.centos.noarch.rpm

Install Zabbix server and Zabbix web.

sudo yum -y install zabbix-server-pgsql zabbix-web-pgsql 

Import the PostgreSQL database.

zcat /usr/share/doc/zabbix-server-pgsql-3.4.*/create.sql.gz | psql -U zabbix zabbix

You should see something similar to the following at the end of the output.

...
INSERT 0 1
INSERT 0 1
COMMIT

Open the Zabbix configuration file to update the database details.

sudo nano /etc/zabbix/zabbix_server.conf

Find the following lines and update the values according to your database configuration. You will need to uncomment the DBHost and DBPort lines.

DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=StrongPassword
DBPort=5432

Zabbix automatically installs the virtual host file for Apache. We will need to configure the virtual host to update the timezone and PHP version.

sudo nano /etc/httpd/conf.d/zabbix.conf

Find the following lines.

<IfModule mod_php5.c>
...
#php_value date.timezone Europe/Riga

Since we are using PHP version 7, you will also need to update the mod_php version. Update the lines according to your timezone as shown below.

<IfModule mod_php7.c>
...
php_value date.timezone Asia/Kolkata

Now restart Apache to apply these changes in configuration.

sudo systemctl restart httpd

Start the Zabbix server and enable it to automatically start at boot.

sudo systemctl start zabbix-server
sudo systemctl enable zabbix-server

You should have the Zabbix server running now. You can check the status of the process by running this.

sudo systemctl status zabbix-server

Modify the firewall to allow the standard HTTP and HTTPS port. You will also need to allow port 10051 through the firewall, which will be used by Zabbix to obtain the events from Zabbix agent running on remote machines.

sudo firewall-cmd --zone=public --permanent --add-service=http
sudo firewall-cmd --zone=public --permanent --add-service=https
sudo firewall-cmd --zone=public --permanent --add-port=10051/tcp
sudo firewall-cmd --reload

To access the administration dashboard, you can open http://192.0.2.1/zabbix using your favorite browser. You will see a welcome message. You should have all the prerequisites satisfied on the next interface. Follow the instructions on the installer page to install the software. Once the software has been installed, login using the username Admin and password zabbix. Zabbix is now installed and ready to collect the data from the Zabbix agent.

Setup a Zabbix Agent on the Server

To monitor the server on which Zabbix is installed, you can set up the agent on the server. The Zabbix agent will gather the event data from the Linux server to send it to the Zabbix server. By default, port 10050 is used to send the events and data to the server.

Install the Zabbix agent.

sudo yum -y install zabbix-agent

Start the agent and enable it to automatically start at boot.

sudo systemctl start zabbix-agent
sudo systemctl enable zabbix-agent

The communication between the Zabbix agent and the Zabbix server is done locally, thus there is no need to set up any encryption.

Before the Zabbix server can receive any data, you need to enable the host. Login to the web administration dashboard of the Zabbix server and go to Configuration >> Host. You will see a disabled entry of the Zabbix server host. Select the entry and click on the "Enable" button to enable the monitoring of the Zabbix server application and the base CentOS system on which the Zabbix server is installed.

Setup the Agent on Remote Linux Machines

There are three methods by which a remote Zabbix agent can send events to the Zabbix server. The first method is to use an unencrypted connection, and the second is using a secured pre-shared key. The third and most secure way is to encrypt the transmission using RSA certificates.

Before we proceed to install and configure the Zabbix agent on the remote machine, we need to generate the certificates on the Zabbix server system. We will use self-signed certificates.

Run the following commands on the Zabbix server as a sudo user.

Create a new directory to store Zabbix keys and generate the private key for the CA.

mkdir ~/zabbix-keys && cd ~/zabbix-keys
openssl genrsa -aes256 -out zabbix-ca.key 4096

It will ask you for a passphrase to protect the private key. Once the private key has been generated, proceed to generate the certificate for the CA.

openssl req -x509 -new -key zabbix-ca.key -sha256 -days 3560 -out zabbix-ca.crt

Provide the passphrase of the private key. It will ask you for a few details about your country, state, organization. Provide the details accordingly.

[user@vultr zabbix-keys]$ openssl req -x509 -new -key zabbix-ca.key -sha256 -days 3560 -out zabbix-ca.crt
Enter passphrase for `zabbix-ca.key`:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:IN
State or Province Name (full name) []:My State
Locality Name (eg, city) [Default City]:My City
Organization Name (eg, company) [Default Company Ltd]:My Organization
Organizational Unit Name (eg, section) []:My Unit
Common Name (eg, your name or your server's hostname) []:Zabbix CA
Email Address []:mail@example.com

We have successfully generated the CA certificate. Generate the private key and CSR for Zabbix server.

openssl genrsa -out zabbix-server.key 2048
openssl req -new -key zabbix-server.key -out zabbix-server.csr

Please do not provide a passphrase to encrypt the private key when running the above command. Using the CSR, generate the certificate for the Zabbix server.

openssl x509 -req -in zabbix-server.csr -CA zabbix-ca.crt -CAkey zabbix-ca.key -CAcreateserial -out zabbix-server.crt -days 1825 -sha256

Similarly, generate the private key and CSR for the Zabbix host or agent.

openssl genrsa -out zabbix-host1.key 2048
openssl req -new -key zabbix-host1.key -out zabbix-host1.csr

Now generate the certificate.

openssl x509 -req -in zabbix-host1.csr -CA zabbix-ca.crt -CAkey zabbix-ca.key -CAcreateserial -out zabbix-host1.crt -days 1460 -sha256

Copy the certificates into the Zabbix configuration directory.

sudo mkdir /etc/zabbix/keys
sudo cp zabbix-ca.* zabbix-server.* /etc/zabbix/keys

Provide the ownership of the certificates to the Zabbix user.

sudo chown -R zabbix: /etc/zabbix/keys

Open the configuration file of the Zabbix server to update the path of the certificates.

sudo nano /etc/zabbix/zabbix_server.conf

Find these lines in the configuration file and change them as shown.

TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-server.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-server.key

Save the file and exit from the editor. Restart the Zabbix server so that the change in configuration can take effect.

sudo systemctl restart zabbix-server

Copy the certificates using the scp command to the host computer which you wish to monitor.

cd ~/zabbix-keys
scp zabbix-ca.crt zabbix-host1.* user@192.0.2.2:~

Make sure that you replace 192.0.2.2 with the actual IP address of the remote host on which you want to install the Zabbix agent.

Install the Zabbix Host

Now that we have copied the certificates to the host system, we are ready to install the Zabbix agent.

From now on, all the commands need to be executed on the host which you wish to monitor.

Add the Zabbix repository into the system.

sudo rpm -ivh http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-1.el7.centos.noarch.rpm

Install the Zabbix agent into the system.

sudo yum -y install zabbix-agent

Move the key and certificates to the Zabbix configuration directory.

sudo mkdir /etc/zabbix/keys
sudo mv ~/zabbix-ca.crt ~/zabbix-host1.* /etc/zabbix/keys/

Provide ownership of the certificates to the Zabbix user.

sudo chown -R zabbix: /etc/zabbix/keys

Open the configuration file of the Zabbix agent to update the server IP address and the path to the key and certificates.

sudo nano /etc/zabbix/zabbix_agentd.conf

Find the following line and make the necessary changes to make them look like shown below.

Server=192.0.2.1                    # Replace with actual Zabbix server IP
ServerActive=192.0.2.1              # Replace with actual Zabbix server IP
Hostname=Zabbix host1               # Provide a appropriate name or hostname

The hostname must be a unique string which is not specified for any other host system. Please, make a note of the hostname as we will need to set the exact hostname in the Zabbix server.

Furthur, update the values of these parameters.

TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/keys/zabbix-ca.crt
TLSCertFile=/etc/zabbix/keys/zabbix-host1.crt
TLSKeyFile=/etc/zabbix/keys/zabbix-host1.key

Now, restart the Zabbix agent and enable it to automatically start at boot.

sudo systemctl restart zabbix-agent
sudo systemctl enable zabbix-agent

You have successfully configured the Zabbix agent on the host system. Browse the Zabbix administration dashboard at https://192.0.2.1/zabbix to add the newly configured host.

Go to Configuration >> Hosts and click on the Create Host button at the top right corner.

On the create host interface, provide the hostname, which must be exactly similar to the hostname configured in the host system's Zabbix agent configuration file. For the display name, you are free to choose anything you want.

Select the group in which you want to add the system. Since we are running the Zabbix agent to monitor a basic Linux server, we can choose the Linux server group. You are also allowed to create a new group if you want.

Provide the IP address of the host on which the Zabbix agent is running. You can provide an optional description of the host.

Now, go to the Template tab of the new host interface and search for the template according to the application. In our case, we need to find the Template OS Linux. Click on the Add link to assign the template to the host.

Finally, go to the Encryption tab and choose the Certificate for both the connections to the host and the connections from the host. This way, the communication from both sides will be encrypted using the RSA certificates. The Issuer and Subject fields are optional.

Once done, you can click on the Add button to add the new host to the Zabbix server. After a few minutes, the ZBX availability will turn green. This signifies that the host is available using the Zabbix agent and actively sending events from the host computer to the Zabbix server.

If the ZBX availability turns red in colour instead of green, it means there was some error and the host is unable to send data to the server. In that case, look for the error in log files and troubleshoot the error accordingly. The path to the Zabbix server log and Zabbix agent log is /var/log/zabbix/zabbix_server.log, and /var/log/zabbix/zabbix_agentd.log.

You can now monitor the host by going to monitoring tab. You can view problems, received events, live graphs and much more information on the monitoring pages.

Conclusion

Congratulations, you have successfully deployed a Zabbix server instance and enabled monitoring on a remote host.

You can monitor the whole infrastructure of your small or midsize company using a single Zabbix instance.