Linux Blimp

MySQL is a powerful open source Relational Database Management System or in short RDBMS. It was released back in 1995 (20 years old). It uses Structured Query Language which is probably the most popular choice for managing content within a database. The latest MySQL version is 5.6.25 and was released on 29 May 2015.

An interesting fact about MySQL is the fact that the name comes from Michael Widenius’s (MySQL’s creator) daughter My. Even though there are plenty of interesting facts about MySQL, this article is meant to show you some useful practices to help you manage your MySQL server.

In April 2009 the MySQL project was bought by Oracle. As a result a MySQL community fork called MariaDB was created. The main reason for creating the fork was to keep the project free under the General Public License.

Today MySQL and MariaDB are one of the most (if not the most) frequently used RDBMS used for web applications such as WordPress, Joomla, Magento and others.

This article will show you some basic, yet useful tips how to optimize the fine tune the performance of MySQL/MariaDB. Please keep in mind that this article assumes that you already have MySQL or MariaDB installed. If you are still wondering how to install them on your system, you can follow our extensive guides here:

1. Installing LAMP on RHEL/CentOS 7
2. Installing LAMP on Fedora 22
3. Setting Up LAMP on Ubuntu 15.04
4. Installing MariaDB on Debian 8
5. Install MariaDB on Gentoo Linux
6. Install MariaDB on Arch Linux

Important: Before we start – do not accept this suggestions blindly. Each MySQL setup is unique and requires additional thought, before making any changes.

Things you need to know:

1. MySQL/MariaDB configuration file is located in /etc/my.cnf. Every time you modify this file you will need to restart the MySQL service so the new changes can take effect.
2. For writing this article MySQL version 5.6 has been used as template.

1. Enable InnoDB file-per-table

First it is important to explain that InnoDB is a storage engine. MySQL and MariaDB use InnoDB as default storage engine. In the past MySQL used to keep database tables and indexes in a system tablespace. This approach was meant for servers which sole purpose is database processing and their storage disk is not used for any other purposes.

The InnoDB provides more flexible approach and each database information is kept in a .ibd data file. Each .ibd file represents a tablespace of its own. That way database operations such as “TRUNCATE” can be completed faster and you may also reclaim unused space when dropping or truncating a database table.

Another benefit of this configuration is the fact that you can keep some of the database tables in a separate storage device. This can greatly improve the I/O load on your disks.

The innodb_file_per_table is enabled by default in MySQL 5.6 and above. You can see that in /etc/my.cnf file. The directive looks like this:

innodb_file_per_table=1

2. Store MySQL Database Data on Separate Partition

Note: This setup only works with MySQL, but not with MariaDB.

Sometimes OS read/writes can slow down the performance of your MySQL server, especially if located on same hard drive. Instead, I would recommend using separate hard drive (preferably SSD) for the MySQL service.

To complete, this you will need to attach the new drive to your computer/server. For the purpose of this article, I will assume that the drive will be under /dev/sdb.

The next step is to prepare the new drive:

# fdisk /dev/sdb

Now press “n” to create new partition. Next press “p” to make the new partition primary. After that, set the partition number from 1-4. After that you will select the partition size. Press enter here. On the next step you will need to configure the size of the partition.

If you wish to use the entire disk press enter once more. Otherwise you can manually set the size of the new partition. When ready press “w” to write the changes. Now we will need to create a filesystem for our new partition. This can be easily done with:

# mkfs.ext4 /dev/sdb1

Now we will mount our new partition in a folder. I have named my folder “ssd” and created in the root directory:

# mkdir /ssd/

We are ready to mount the new partition we just made in the new folder:

# mount /dev/sdb1  /ssd/

You can perform the mount at startup by adding the following line in /etc/fstab file.

/dev/sdb1 /ssd ext3 defaults 0 0

Now you are ready to move MySQL to the new disk. First stop the MySQL service with:

# service mysqld stop

I would recommend you stopping Apache/nginx as well to prevent any attempts to write in the databases:

# service httpd stop
# service nginx stop

Now copy the entire MySQL directory in the new drive:

# cp /var/lib/mysql /ssd/ -Rp

This may take a while depending on the site of your MySQL databases. Once this process is complete rename the MySQL directory:

# mv /var/lib/mysql /var/lib/mysql-backup

Next we will create a symlink.

# ln -s /ssd/mysql /var/lib/mysql

Now you are ready to start your MySQL and web service:

# service mysqld start
# service httpd start
# service nginx start

At this point your MySQL databases will be accessed from the new drive.

3. Optimizing InnoDB buffer pool Usage

The InnoDB engine has a buffer pool used for caching data and indexes in memory. This of course will help your MySQL/MariaDB queries be executed significantly faster. Choosing the proper size here requires some very important decisions and good knowledge on your system’s memory consumption.

Here is what you need to consider:

1. How much memory you need for other processes. This includes your system processes, page tables, socket buffers.
2. Is your server dedicated for MySQL or you will be running other memory hungry services.

On a dedicated box, you would probably want to give about 60-70% of the memory to the innodb_buffer_pool_size. If you plan on running more services on a single box, you should re-consider the amount of memory you dedicate for your innodb_buffer_pool_size.

The value that you should edit in my.cnf is:

innodb_buffer_pool_size

4. Avoid Swappiness in MySQL

Swapping is process that occurs when system moves part of memory to a special disk space called “swap”. The event usually appears when your system runs out of physical memory and instead of freeing up some RAM, the system pushed the information into disk. As you might have guess the disk is much slower than your RAM.

By default the option is enabled:

# sysctl vm.swappiness 

vm.swappiness = 60

To disable swappiness, run the following command:

# sysctl -w vm.swappiness=0

5. Set MySQL Max Connections

The max_connections directive tells your server how many concurrent connections are permitted. The MySQL/MariaDB server allows the value given in max_connections + 1 for user with SUPER privileges. The connection is opened only for the time MySQL query is executed – after that it is closed and new connection can take its place.

Keep in mind that too many connections can cause high RAM usage and lock up your MySQL server. Usually small websites will require between 100-200 connections while larger may require 500-800 or even more. The value you apply here strongly depends on your particular MySQL/MariaDB usage.

You can dynamically change the value of max_connections, without having to restart the MySQL service by running:

# mysql -u root -p
mysql> set global max_connections := 300;

6. Configure MySQL thread_cache_size

The thread_cache_size directive sets the amount of threads that your server should cache. As the client disconnects, his threads are put in the cache if they are less than the thread_cache_size. Further requests are completed by using the threads stored in the cache.

To improve your performance you can set the thread_cache_size to a relatively high number. To find the thread cache hit rate, you can use the following technique:

mysql> show status like 'Threads_created';
mysql> show status like 'Connections';

Now use the following formula to calculate the thread cache hit rate percentage:

100 - ((Threads_created / Connections) * 100)

If you get a low number, it means that most of the new mysql connections are starting new thread instead of loading from cache. You will surely want to increase the thread_cache_size in such cases.

The good thing here is that the thread_cache_size can be dynamically changed without having to restart the MySQL service. You can achieve this by running:

mysql> set global thread_cache_size = 16;

7. Disable MySQL Reverse DNS Lookups

By default MySQL/MariaDB perform a DNS lookup of the user’s IP address/Hostname from which the connection is coming. For each client connection, the IP address is checked by resolving it to a host name. After that the host name is resolved back to an IP to verify that both match.

This unfortunately may cause delays in case of badly configured DNS or problems with DNS server. This is why you can disable the reverse DNS lookup by adding the following in your configuration file:

[mysqld]
# Skip reverse DNS lookup of clients
skip-name-resolve

You will have to restart the MySQL service after applying these changes.

8. Configure MySQL query_cache_size

If you have many repetitive queries and your data does not change often – use query cache. People often do not understand the concept behind the query_cache_size and set this value to gigabytes, which can actually cause degradation in the performance.

The reason behind that is the fact that threads need to lock the cache during updates. Usually value of 200-300 MB should be more than enough. If your website is relatively small, you can try giving the value of 64M and increase in time.

You will have to add the following settings in the MySQL configuration file:

query_cache_type = 1
query_cache_limit = 256K
query_cache_min_res_unit = 2k
query_cache_size = 80M

9. Configure tmp_table_size and max_heap_table_size

Both directives should have the same size and will help you prevent disk writes. The tmp_table_size is the maximum amount of size of internal in-memory tables. In case the limit in question is exceeded the table will be converted to on-disk MyISAM table.

[mysqld]
tmp_table_size= 64M
max_heap_table_size= 64M

10. Enable MySQL Slow query Logs

Logging slow queries can help you determine issues with your database and help you debug them. This can be easily enabled by adding the following values in your MySQL configuration file:

slow-query-log = 1
slow-query-log-file = /var/lib/mysql/mysql-slow.log
long_query_time = 1

The first directive enables the logging of slow queries, while the second one tells MySQL where to store the actual log file. Use long_query_time to define the amount of time that is considered long for MySQL query to be completed.

11. Check for MySQL idle Connections

Idle connections consume resources and should be interrupted or refreshed when possible. Such connections are in “sleep” state and usually stay that way for long period of time. To look for idled connections you can run the following command:

# mysqladmin processlist -u root -p | grep “Sleep”

This will show you list of processes that are in sleep state. The event appears when the code is using persistent connection to the database. When using PHP this event can appear when using mysql_pconnect which opens the connection, after that executes queries, removes the authentication and leaves the connection open. This will cause any per-thread buffers to be kept in memory until the thread dies.

The first thing you would do here is to check the code and fix it. If you don’t have access to the code that is being ran, you can change the wait_timeout directive. The default value is 28800 seconds, while you can safely decrease it to something like 60:

wait_timeout=60

12. Choosing Right MySQL Filesystem

Choosing the right filesystem is vital for your databases. Most important things you need to consider here are – data integrity, performance and ease of administration.

As per MariaDB’s recommendations, the best file systems are XFS, Ext4 and Btrfs. All of them are enterprise journaling filesystems that can be used with very large files and large storage volumes.

Below you can find some useful information about the three filesystems:

The pros and cons of the Linux filesystems have been extensively covered in our article:

1. Linux Filesystem Explained

13. Set MySQL max_allowed_packet

MySQL splits data into packets. Usually a single packet is considered a row that is sent to a client. The max_allowed_packet directive defines the maximum size of packet that can be sent.

Setting this value too low can cause a query to stall and you will receive an error in your MySQL error log. It is recommended to set the value to the size of your largest packet.

14. Check MySQL Performance Tuning

Measuring your MySQL/MariaDB performance is something that you should do on regular basis. This will help you see if something in the resource usage changes or needs to be improved.

There are plenty of tools available for benchmarking, but I would like to suggest you one that is simple and easy to use. The tool is called mysqltuner.

To download and run it, use the following set of commands:

# wget https://github.com/major/MySQLTuner-perl/tarball/master
# tar xf master
# cd major-MySQLTuner-perl-993bc18/
# ./mysqltuner.pl 

You will receive a detailed report about your MySQL usage and recommendation tips. Here is a sample output of default MariaDB installation:

15. Optimize and Repair MySQL Databases

Sometimes MySQL/MariaDB database tables get crashed quite easily, especially when unexpected server shut down, sudden file system corruption or during copy operation, when database is still accessed. Surprisingly, there is a free open source tool called ‘mysqlcheck‘, which automatically check, repair and optimize databases of all tables in Linux.

# mysqlcheck -u root -p --auto-repair --check --optimize --all-databases
# mysqlcheck -u root -p --auto-repair --check --optimize databasename

That’s it! I hope you have found the above article useful and help you tune up your MySQL server. As always if you have any further questions or comments, please submit them in the comment section below.

Read Also:

1. 20 MySQL Commands for Database Administration
2. 4 Useful Tools to Monitor MySQL Performance
3. MySQL Backup and Restore Commands

Source

MySQL is the world’s most popular open source database system and MariaDB (a fork of MySQL) is the world’s fastest growing open source database system. After installing MySQL server, it is insecure in it’s default configuration, and securing it is one of the essential tasks in general database management.

Read Also: Learn MySQL/MariaDB for Beginners – Part 1

This will contribute to hardening and boosting of overall Linux server security, as attackers always scan vulnerabilities in any part of a system, and databases have in the past been key target areas. A common example is the brute-forcing of the root password for the MySQL database.

In this guide, we will explain useful MySQL/MariaDB security best practice for Linux.

1. Secure MySQL Installation

This is the first recommended step after installing MySQL server, towards securing the database server. This script facilitates in improving the security of your MySQL server by asking you to:

• set a password for the root account, if you didn’t set it during installation.
• disable remote root user login by removing root accounts that are accessible from outside the local host.
• remove anonymous-user accounts and test database which by default can be accessed by all users, even anonymous users.

# mysql_secure_installation

After running it, set the root password and answer the series of questions by entering [Yes/Y] and press [Enter].

2. Bind Database Server To Loopback Address

This configuration will restrict access from remote machines, it tells the MySQL server to only accept connections from within the localhost. You can set it in main configuration file.

# vi /etc/my.cnf	                   [RHEL/CentOS]	
# vi /etc/mysql/my.conf                    [Debian/Ubuntu] 
OR
# vi /etc/mysql/mysql.conf.d/mysqld.cnf    [Debian/Ubuntu] 

Add the following line below under [mysqld] section.

bind-address = 127.0.0.1

3. Disable LOCAL INFILE in MySQL

As part of security hardening, you need to disable local_infile to prevent access to the underlying filesystem from within MySQL using the following directive under [mysqld] section.

local-infile=0

4. Change MYSQL Default Port

The Port variable sets the MySQL port number that will be used to listen on TCP/ IP connections. The default port number is 3306 but you can change it under the [mysqld] section as shown.

Port=5000

5. Enable MySQL Logging

Logs are one of the best ways to understand what happens on a server, in case of any attacks, you can easily see any intrusion-related activities from log files. You can enable MySQL logging by adding the following variable under the [mysqld] section.

log=/var/log/mysql.log

6. Set Appropriate Permission on MySQL Files

Ensure that you have appropriate permissions set for all mysql server files and data directories. The /etc/my.conf file should only be writeable to root. This blocks other users from changing database server configurations.

# chmod 644 /etc/my.cnf

7. Delete MySQL Shell History

All commands you execute on MySQL shell are stored by the mysql client in a history file: ~/.mysql_history. This can be dangerous, because for any user accounts that you will create, all usernames and passwords typed on the shell will recorded in the history file.

# cat /dev/null > ~/.mysql_history

8. Don’t Run MySQL Commands from Commandline

As you already know, all commands you type on the terminal are stored in a history file, depending on the shell you are using (for example ~/.bash_history for bash). An attacker who manages to gain access to this history file can easily see any passwords recorded there.

It is strongly not recommended to type passwords on the command line, something like this:

# mysql -u root -ppassword_

When you check the last section of the command history file, you will see the password typed above.

# history 

The appropriate way to connect MySQL is.

# mysql -u root -p
Enter password:

9. Define Application-Specific Database Users

For each application running on the server, only give access to a user who is in charge of a database for a given application. For example, if you have a wordpress site, create a specific user for the wordpress site database as follows.

# mysql -u root -p
MariaDB [(none)]> CREATE DATABASE osclass_db;
MariaDB [(none)]> CREATE USER 'osclassdmin'@'localhost' IDENTIFIED BY 'osclass@dmin%!2';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON osclass_db.* TO 'osclassdmin'@'localhost';
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> exit

and remember to always remove user accounts that are no longer managing any application database on the server.

10. Use Additional Security Plugins and Libraries

MySQL includes a number of security plugins for: authenticating attempts by clients to connect to mysql server, password-validation and securing storage for sensitive information, which are all available in the free version.

You can find more here: https://dev.mysql.com/doc/refman/5.7/en/security-plugins.html

11. Change MySQL Passwords Regularly

This is a common piece of information/application/system security advice. How often you do this will entirely depend on your internal security policy. However, it can prevent “snoopers” who might have been tracking your activity over an long period of time, from gaining access to your mysql server.

MariaDB [(none)]> USE mysql;
MariaDB [(none)]> UPDATE user SET password=PASSWORD('YourPasswordHere') WHERE User='root' AND Host = 'localhost';
MariaDB [(none)]> FLUSH PRIVILEGES;

12. Update MySQL Server Package Regularly

It is highly recommended to upgrade mysql/mariadb packages regularly to keep up with security updates and bug fixes, from the vendor’s repository. Normally packages in default operating system repositories are outdated.

# yum update
# apt update

After making any changes to the mysql/mariadb server, always restart the service.

# systemctl restart mariadb		#RHEL/CentOS
# systemctl restart mysql		#Debian/Ubuntu

Read Also: 15 Useful MySQL/MariaDB Performance Tuning and Optimization Tips

That’s all! We love to hear from you via the comment form below. Do share with us any MySQL/MariaDB security tips missing in the above list.

Source

Normally while installing MySQL/MariaDB database server on Linux, it’s recommended to set a MySQL root user password to secure it, and this password is required to access the database server with root user privileges.

Suggested Read: How to Install and Secure MariaDB 10 in CentOS 7

In this guide, we will show you how to connect and run MySQL commands without entering a password (mysql passwordless root login) on the Linux terminal.

How to Set MySQL Root Password

In case you have freshly installed the MySQL/MariaDB server, then it doesn’t require any password to connect to it as root user. To secure it, set the MySQL/MariaDB password for root user with the following command.

Note that this command is just one of the many MySQL (Mysqladmin) Commands for Database Administrationin Linux.

# mysqladmin -u root password YOURNEWPASSWORD

How to Connect or Run MySQL Without Root Password

To run MySQL commands without entering password on the terminal, you can store your user and password in the ~/.my.cnf user specific configuration file in user’s home directory as described below.

Now create the config file ~/.my.cnf and add configurations below in it (remember to replace mysqluser and mysqlpasswd with your own values).

[mysql]
user=user
password=password

Save and close the file. Then set the suitable permissions on it, to make it only readable and writable by you.

# chmod 0600 .my.cnf

Once you have set user and password in the Mysql configuration file, from now on when you run mysql commands such as mysql, mysqladmin etc, they will read the mysqluser and mysqlpasswd from the above file.

# mysql 
# mysql -u root 

You may also like to read these related articles about MySQL/MariaDB:

1. 1. 20 MySQL (Mysqladmin) Commands for Database Administration in Linux
   2. How to Change Root Password of MySQL or MariaDB in Linux
   3. How to Reset MySQL or MariaDB Root Password in Linux
   4. 15 Useful MySQL/MariaDB Performance Tuning and Optimization Tips
   5. 4 Useful Commandline Tools to Monitor MySQL Performance in Linux

In this guide, we showed how to run MySQL commands without entering root password on the terminal. If you have any queries or thoughts to share, do hit us up via the feedback form below.

Source

Following previous tutorials concerning Zabbix series, this article describes how to install and setup an instance of Zabbix agent to run as a service on Microsoft Windows systems in order to monitor your infrastructure windows environments, especially server machines.

Requirements

 Installing and Configuring Zabbix on Debian 8 and RHEL/CentOS 7 – Part 1

Related Guides

 How to Configure ‘Zabbix Monitoring’ to Send Email Alerts to Gmail Account – Part 2

 How to Install and Configure Zabbix Agents on Remote Linux Systems – Part 3

Step 1: Download and Install Zabbix Agent on Windows

1. The pre-compiled zip agents for Windows environments can be obtained from official Zabbix download page and manually installed and started on the system using windows Command Prompt as in the following example:

C:\Users\caezsar><full system path to zabbix_agentd.exe> --config <full system path to zabbix_agentd.win.conf> --install

Example, suppose you’ve downloaded and extracted the Zabbix agent zip archive to D:\Downloads\zabbix_agents_2.4.4.win\, run the following command to install the service:

C:\Users\caezsar>D:\Downloads\zabbix_agents_2.4.4.win\bin\win32\zabbix_agentd.exe --config D:\Downloads\zabbix_agents_2.4.4.win\conf\zabbix_agentd.win.conf --install

2. After the service has been installed on your windows host, open zabbix_agentd.win.conf file and manually edit the following parameters:

Server=IP of Zabbix Server
ServerActive=IP of Zabbix Server
Hostname=use the FQDN of your windows host

3. To start the service just type:

C:\Users\caezsar>D:\Downloads\zabbix_agents_2.4.4.win\bin\win32\zabbix_agentd.exe --start

To stop the service run the same command as above with --stop argument and to uninstall the service use the --uninstall argument.

C:\Users\caezsar>D:\Downloads\zabbix_agents_2.4.4.win\bin\win32\zabbix_agentd.exe --stop
C:\Users\caezsar>D:\Downloads\zabbix_agents_2.4.4.win\bin\win32\zabbix_agentd.exe --uninstall

4. A second and more convenient method to install and automatically configure Zabbix agent on Windows environments is by downloading Zabbix Agent installer msi package specific for your system architecture by visiting the link: http://www.suiviperf.com/zabbix/index.php.

5. Once the Zabbix agent msi file has been downloaded on your system, run it and supply the required information in order to configure and install the agent on the target monitored host as it follows:

Hostname: use the FQDN of your windows host (the hostname value should match the “Full Computer name” configured for your machine)
Zabbix server Name: use the IP of the Zabbix Server
Agent Port: 10050 
Remote Command: check this value
Active Server: IP of Zabbix Server

If you need to modify Zabbix configuration file with other custom values at a later date, the conf file can be found on %programfiles%\Zabbix Agent\ path.

6. After you’ve finished the setup, open a windows Command Prompt with Administrator privileges, run services.msc command in order to open Windows Services utility and locate Zabbix Agent service to check if the service is running and automatically started after reboot.

services.msc

From this console you can manage the service (start, stop, pause, resume, enable or disable).

Step 2: Configure Windows Firewall and Test Zabbix Agent

7. Almost all Windows based systems have Windows Firewall active and running, therefore zabbix agent port must be opened in firewall in order to communicate with zabbix server.

In order to open Zabbix agent port in windows firewall, open Control Panel -> System and Security – > Windows Firewall and hit on Allow an app through Windows Firewall.

8. Next, click on Allow another app button and a new window should open. Use the Browse button to navigate and add Zabbix agent executable file (usually found in %programfiles%\Zabbix Agent\ if you installed it using msi program), then hit on Add button to add the service.

9. Next, make sure you check and open the firewall rule on the network segment where zabbix server is located in your network and hit OK button to finish and apply configuration.

10. In order to test if Zabbix agent running on windows is reachable from Zabbix server side, use telnet or netcat command on zabbix server against windows agent IP-Port and a Connected message should appear. Hit Enter key to generate an error message and automatically disconnect from agent:

telnet <Windows_agent IP Address> 10050

Step 3: Add Zabbix Agent Monitored Windows Host to Zabbix Server

11. Once the windows agent has been tested from command line and everything looks fine, go to Zabbix Serverweb interface, move to Configuration tab -> Hosts and hit on Create Host button in order to add the Windows monitored host.

12. On Host window add the FQDN of your windows agent machine in Host name filed, add an arbitrary name to Visible name filed in order to easily identify the monitored machine on Zabbix panel, make sure the host is included into a Group Servers and add the IP Address of your windows host in Agent interfaces filed. The Portvalue leave it unchanged.

13. Next, go to Template tab and hit on Select button. A new window with Zabbix Templates should appear. Navigate through this window, check Template OS Windows and hit on Select button to add the template.

14. Once the Template OS Windows appears on Link new templates filed, hit on Add button in order to link this template to the windows host configuration.

Finally, after the Template OS Windows is visible in Linked Templates filed hit on below Add button to complete the process and add the entire Windows host configuration.

15. After your monitored windows machine has been added return to Configuration -> Hosts and the windows Host should now be present in this window as illustrated on below screenshot.

That’s all! Just assure that your windows host Status is set to Enabled and wait a few minutes in order for Zabbix server to contact the windows agent side and process the received remote data.

As an example, to get a graphical inside of the CPU load on the monitored Windows machine go to Zabbix web console Monitoring tab -> Graphs, select the windows machine host name and CPU load Graph and all the collected data so far should be presented into a nice graphical chart.

Continuing Zabbix series, this tutorial will guide you on how you can install and configure Zabbix agents on Linux (Debian based systems and CentOS) in order to actively monitor local resources on remote systems.

The main job of zabbix agents consists in gathering local information from the targets where they run and send the data to a central Zabbix server to be further processed and analyzed.

Requirements

Install and Configure Zabbix 2.4.5 on Debian 8 and RHEL/CentOS 7.

1. Setting Up Zabbix Monitoring on Debian and CentOS based Systems

Step 1: Install Zabbix Agents in Linux Systems

1. Depending on the Linux distribution you are running, go to Zabbix download page, grab the latest version of the available agents binary packages using a tool such as wget or curl and install it on your machine using the distribution specific package manager – Yum, Rpm or Dpkg.

On Debian based Systems

For Debian systems (including latest release – Debian 8 Jessie) use the following steps to download and install Zabbbix Agent:

$ wget http://repo.zabbix.com/zabbix/2.4/debian/pool/main/z/zabbix/zabbix-agent_2.4.0-1+wheezy_amd64.deb  
$ sudo dpkg -i zabbix-agent_2.4.0-1+wheezy_amd64.deb

On CentOS based Systems

For CentOS alike systems, download the .rpm packaged for the distribution specific release number, using the same page as above, and install it using rpm package manager.

In order to automatically manage missing dependency issues and install the agent using one shot use the yumcommand followed by the binary package download link, as in the example below used for installing the agent on CentOS 7:

# rpm -Uvh http://repo.zabbix.com/zabbix/2.4/rhel/7/x86_64/zabbix-agent-2.4.1-2.el7.x86_64.rpm

Step 2: Configure and Test Zabbix Agent in Linux

2. The next logical step after installing the packages on the system is to open Zabbix agent configuration file located in /etc/zabbix/ system path on both major distributions and instruct the program to send all the collected information to Zabbix server in order to be analyzed and processed.

Therefore, open zabbix_agentd.conf file with your favorite text editor, find the below lines (use the screenshots as a guide), uncomment them and make the following changes:

# nano /etc/zabbix/zabbix_agentd.conf

add zabbix server IP address and hostname as shown below.

Server=IP of Zabbix Server
ServerActive=IP of Zabbix Server
Hostname=use the FQDN of the node where the agent runs

3. Once you’ve finished editing the Zabbix agent configuration file with the required values, restart the daemon using the following command, then use netstat command to verify if the daemon has been started and operates on the specific port – 10050/tcp:

$ sudo systemctl restart zabbix-agent
$ sudo netstat -tulpn|grep zabbix

For older distributions use the service command to manage zabbix agent daemon:

$ sudo service zabbix-agent restart
$ sudo netstat -tulpn|grep zabbix

4. If your system is behind a firewall then you need to open 10050/tcp port on the system in order to reach through Zabbix server.

For Debian based systems, including Ubuntu, you can use ufw tool to open the port and on CentOS 7 you can use Firewalld utility to manage the firewall rules as the below examples:

$ sudo ufw allow 10050/tcp  [On Debian based systems]

$ sudo firewall-cmd --add-port=10050/tcp                [For centOS 7 on-fly rule]
$ sudo firewall-cmd --add-port=10050/tcp --permanent    [For centOS 7 permanent rule]

For older distributions such as centOS 6 or unmanaged firewalls through specific utilities use the powerful iptables command to open ports:

# iptables -A INPUT -p tcp -m tcp --dport 10050 -j ACCEPT

5. Finally, in order to test if you can reach Zabbix Agent from Zabbix Server, use Telnet command from Zabbix server machine to the IP addresses of the machines that run the agents, as illustrated below (don’t worry about the thrown error from agents):

# telnet zabbix_agent_IP 10050

Step 3: Add Zabbix Agent Monitored Host to Zabbix Server

6. On the next step it’s time to move to Zabbix server web console and start adding the hosts which run zabbix agent in order to be monitored by the server.

Go to Configuration -> Hosts -> Create Host -> Host tab and fill the Host name field with the FQDN of the monitored zabbix agent machine, use the same value as above for Visible name field.

Next, add this host to a group of monitored servers and use the IP Address of the monitored machine at Agent interfaces field – alternatively you can also use DNS resolution if it’s the case. Use the below screenshots as a guide.

7. Next, move to Templates tab and hit Select. A new window with templates should open. Choose Template OS Linux then scroll down and hit on Select button to add it and automatically close the window.

8. Once the template appears to Link new template box, hit on Add text to link it to zabbix server, then hit on the lower Add button to finish the process and completely add the monitored host. The visible name of the monitored host should now appear hosts window.

That’s all! Just assure that the host Status is set to Enabled and wait a few minutes in order for Zabbix server to contact the agent, process the received data and inform or eventually alert you if something goes bad on the monitored target.

Source

If you are using Zabbix to monitor your infrastructure you might want to receive email alerts from your local domain somewhere on public internet domain, even if you don’t own a valid registered internet domain name with a mail server which you can configure on your own.

This tutorial will briefly discuss you on how to setup Zabbix server to send mail reports to a Gmail address by using SSMTP program, without the need to install and configure any local MTA daemon, such as Postfix, Eximetc.

Requirements

1. Install Zabbix Monitoring Server

Step 1: Install and Configure SSMTP

1. SSMTP is a small software, which does not fulfill any of the functionality of a mail server, but only delivers emails from a local machine to an external email address on a mailhub.

To install SSMTP program alongside with mailutils package that you will use it to send mails, issue the following command on your RedHat and Debian like server:

# yum install ssmtp mailx                    [On RHEL/CentOS 7] 
$ sudo apt-get install ssmtp mailutils       [On Debian 8]

# vi /etc/ssmtp/ssmtp.conf                   [On RHEL/CentOS 7]
$ sudo nano /etc/ssmtp/ssmtp.conf            [On Debian 8]

SSMTP settings for GMAIL account

root=gmail-username@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=your_local_domain
hostname=your_local_FQDN
UseTLS=Yes
UseSTARTTLS=Yes
AuthUser=Gmail_username
AuthPass=Gmail_password
FromLineOverride=YES

Step 2: Gmail Tests for Zabbix Email Alerts

3. On the next step it’s time to send a local generated email to Gmail account by issuing the below command.

# echo "Body test email from 'hostname -f' "| mail -s "subject here" gmail_user@gmail.com

4. Normally, Gmail prevents different types of authentications to their servers from your account, so, in case you get the error “mail: cannot send message: Process exited with non-zero status”, then login to your Gmail account from browser and navigate to the following link https://www.google.com/settings/security/lesssecureapps in order to allow access for less secure apps as in the following screen.

5. After you have turned on Less Secure Apps feature on your Gmail account, run the above mail command again and verify your Inbox after a few seconds to check if the locally generated email has been successfully delivered – you should normally see the email has incoming from Gmail.

Step 3: Configure Zabbix Sendmail Script

6. Further, based on the $(which mail) command create the following Bash script to Zabbix alertscriptsdirectory with the following content and give it execute permissions:

# vi /usr/local/share/zabbix/alertscripts/zabbix-sendmail            [On RHEL/CentOS 7]
$ sudo nano /usr/local/share/zabbix/alertscripts/zabbix-sendmail     [On Debian 8]

Script content:

#!/bin/bash
echo "$3" | /usr/bin/mail -s "$2" $1

Next, set the execute permission on the script file.

# chmod +x /usr/local/share/zabbix/alertscripts/zabbix-sendmail

7. Next, as previously, test the script functionality by sending a local email to Gmail account. The way to run the script with positional parameters is explained above:

# /usr/local/share/zabbix/alertscripts/zabbix-sendmail gmail_username@gmail.com "Subject here" "Body of the message here"

Afterwards, verify Gmail Inbox and check if the new local message has arrived.

Step 4: Configure Zabbix to Send Alerts to Gmail

8. If the tests so far were successful, then you can move to next step and setup Zabbix to send generated email alerts to Gmail. First, login to Zabbix web interface and navigate to the following menu: Administration -> Media types -> Create media type.

9. On the next screen enter an arbitrary Name to uniquely identify for the script in the Zabbix configurations (in this example Send-Email-Script is used), choose Script as Type from the list and enter the name of the Bash script created earlier (zabbix-sendmail used in this tutorial) to send email from command line (don’t use the path for the script, only the script name). When you’re done, hit the Add button below to reflect changes.

10. Further, let’s configure an email address to which you will send Zabbix alerts. Go to Profile -> Media -> Addand a new pop-up window should appear.

Here, select the name of the script that you have earlier named (in this example Send-Email-Script is used) for Type, enter the Gmail address to which you will send emails, choose the time period (week, hours) when email reports should be active for sending, choose the severity of the messages that you want to receive on your Gmail address, select Enabled as Status and hit the Add button to add the media. Finally hit the Update button to apply configuration.

11. On the next step, enable the defult zabbix alerts by navigating to Configuration -> Actions, select as the Event Source – > Triggers from the right menu and hit on Disabled Status in order to enable it. Repeat the step for Event Source – > Internal or other custom created Actions and you’re done.

Wait for a while for Zabbix to start gather information and generate some reports, then verify your Gmail Inboxand you should see some Zabbix alerts submitted so far.

That’s all! Although this guide was mainly focused on sending Zabbix alerts to a Gmail account using Gmail SMTP server as a mailhub, using the same configuration you can, also, push zabbix email alerts further to other valid internet email accounts by relying on Gmail to route your emails through SMTP servers.

Source

Zabbix is an Open Source, high-level enterprise software designed to monitor and keep track of networks, servers and applications in real time. Build in a server-client model, Zabbix can collect different type of data than are used to create historical graphics and output performance or load trends of the monitored targets.

The server has the ability to check standard networking services (HTTP, FTP, SMTP, IMAP etc) without the need to install extra software on the monitored hosts.

However, in order to gather data and create statistics about local services or other specific system resources that run on remote instances, such as CPU, disks, internal system process, RAM, etc, you need to install and configure a Zabbix agent.

Following are the 4-article series about Zabbix Monitoring application:

--------------- On Debian/Ubuntu --------------- 
$ sudo apt-get install apache2 php7.0 php7.0-mysql php7.0-ldap php7.0-bcmath php7.0-mbstring php7.0-gd php7.0-xml php7.0-mcrypt

--------------- On CentOS/RHEL ---------------
# yum -y install epel-release
# yum install httpd php php-mysql php-ldap php-bcmath- php-mbstring php-gd php-xml php-mcrypt		

$ sudo nano /etc/php/7.0/apache2/php.ini 	[On Debian/Ubuntu] 
# vi /etc/php.ini				[On RHEL/CentOS 7]

max_execution_time 300
memory_limit 128M
always_populate_raw_post_data -1
session.auto_start = 0
mbstring.func_overload = 0
date.timezone = Europe/Bucharest

$ sudo systemctl restart apache2.service	 [On Debian/Ubuntu] 
# systemctl restart httpd.service		 [On RHEL/CentOS 7]

$ sudo apt-get install mariadb-server mariadb-client libmysqld-dev	 [On Debian/Ubuntu] 
# yum install mariadb-server mariadb-client mariadb-devel	         [On RHEL/CentOS 7]

# mysql_secure_installation

# mysql -u root -p
MariaDB [(none)]> create database zabbixdb character set utf8 collate utf8_bin;
MariaDB [(none)]> grant all privileges on zabbixdb.* to 'zabbixuser'@'localhost' identified by 'password1';
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> exit

# wget http://repo.zabbix.com/zabbix/3.4/debian/pool/main/z/zabbix-release/zabbix-release_3.4-1+stretch_all.deb 
# dpkg -i zabbix-release_3.4-1+stretch_all.deb
# apt update
# apt install zabbix-server-mysql zabbix-frontend-php 

# wget http://repo.zabbix.com/zabbix/3.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.4-1+xenial_all.deb 
# dpkg -i zabbix-release_3.4-1+xenial_all.deb
# apt-get update
# apt install zabbix-server-mysql zabbix-frontend-php 

# rpm -ivh http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm
# yum install zabbix-server-mysql zabbix-web-mysql

# apt install zabbix-agent    [On Debian/Ubuntu] 
# yum install zabbix-agent    [On RHEL/CentOS 7] 

# systemctl restart apache2   [On Debian/Ubuntu] 
# systemctl restart httpd     [On RHEL/CentOS 7] 
# setenforce 0                [Disable SELinux on RHEL/CentOS 7] 

# zcat /usr/share/doc/zabbix-server-mysql/create.sql.gz | mysql -u zabbixuser zabbixdb -p

# nano /etc/zabbix/zabbix_server.conf

DBHost=localhost
DBName=zabbixdb
DBUser=zabbixuser
DBPassword=password1

# systemctl restart zabbix-server.service

# nano /etc/zabbix/zabbix_agentd.conf 

Server=127.0.0.1
ListenPort=10050

# systemctl restart zabbix-agent.service 

http://192.168.1.151/zabbix/setup.php
OR
https://192.168.1.151/zabbix/setup.php

# chown -R www-data /var/www/html/conf/		[on Debian 8]
# chown -R apache /var/www/html/conf/		[on RHEL/CentOS 7]

https://your_domain.tld/zabbix/ 
Username: Admin
Password: zabbix

Cockpit is an easy-to-use, lightweight and simple yet powerful remote manager for GNU/Linux servers, it’s an interactive server administration user interface that offers a live Linux session via a web browser.

It can run on several Linux distributions including Debian, Ubuntu, Fedora, CentOS, RHEL, Arch Linux among others.

Cockpit makes Linux discoverable thereby enabling system administrators to easily and reliably carry out tasks such as starting containers, managing storage, network configurations, log inspections coupled with several others.

Suggested Read: 20 Command Line Tools to Monitor Linux Performance

While using it, users can easily switch between the Linux terminal and web browser without any hustles. Importantly, when a user starts a service via Cockpit, it can be stopped via the terminal, and just in case of an error that occurs in the terminal, it is shown in the Cockpit journal interface.

Features of Cockpit:

1. Enables managing of multiple servers in one Cockpit session.
2. Offers a web-based shell in a terminal window.
3. Containers can be managed via Docker.
4. Supports efficient management of system user accounts.
5. Collects system performance information using Performance Co-Pilot framework and displays it in a graph.
6. Supports gathering of system configuration and diagnostic information using sos-report.
7. Also supports Kubernetes cluster or an Openshift v3 cluster.
8. Allows modification of network settings and many more.

How to Install Cockpit in Linux Systems

You can install Cockpit in all Linux distributions from their default official repositories as shown:

Install Cockpit on Fedora and CentOS

To install and enable Cockpit on Fedora distributions, use the following commands.

# yum install cockpit
# systemctl enable --now cockpit.socket
# firewall-cmd --add-service=cockpit
# firewall-cmd --add-service=cockpit --permanent

Install Cockpit on RHEL

Cockpit is added to the Red Hat Enterprise Linux Extras repository from versions 7.1 and later:

# subscription-manager repos --enable rhel-7-server-extras-rpms
# systemctl enable --now cockpit.socket
# firewall-cmd --add-service=cockpit
# firewall-cmd --add-service=cockpit --permanent

Install Cockpit on Debian

Cockpit is not included in Debian official repositories, but you install it using following repository that contains weekly builds specially for Debian unstable:

First add the following repository to /etc/apt/sources.list file.

deb https://fedorapeople.org/groups/cockpit/debian unstable main

Next, import Cockpit’s signing key and then run the following series of commands to install it.

$ sudo apt-key adv --keyserver sks-keyservers.net --recv-keys F1BAA57C
$ sudo apt-get update
$ sudo apt-get install cockpit
$ sudo systemctl enable --now cockpit.socket

Install Cockpit on Ubuntu and Linux Mint

In Ubuntu and Linux Mint distributions, Cockpit is not included, but you can install it from official Cockpit PPA by executing the following commands:

$ sudo add-apt-repository ppa:cockpit-project/cockpit
$ sudo apt-get update
$ sudo apt-get install cockpit
$ sudo systemctl enable --now cockpit.socket

Install Cockpit on Arch Linux

Arch Linux users can install Cockpit from the Arch User Repository using following command.

# yaourt cockpit
# systemctl start cockpit
# systemctl enable cockpit.socket

How to Use Cockpit in Linux

After Cockpit installed successfully, you can access it using a web browser at the following locations.

https://ip-address:9090
OR
https://server.domain.com:9090

Enter system username and password to login in the interface below:

After logging in, you will be presented with a summary of your system information and performance graphs for CPU, Memory, Disk I/O, and Network traffic as seen in the next image:

Next on the dashboard menu, is Services. Here you can view Targets, System Services, Sockets, Timers and Paths pages.

The interface below shows running services on your system.

You can click on a single service to manage it. Simply click on the drop down menus to get the functionality you want.

The Logs menu item displays the logs page which allows for logs inspection. The logs are categorized into Errors, Warnings, Notices and All as in the image below.

Additionally you can as well view logs based on time such as logs for the last 24HRs or 7 days.

Suggested Read: 4 Best Log Monitoring and Management Tools for Linux

To inspect a single log entry, simply click on it.

Cockpit also enables you to manage user accounts on the system, go to Tools and click on Accounts. Clicking on a user account allows you to view the users account details.

To add a system user, click on “Create New Account” button and enter the necessary user information in the interface below.

To get a terminal window, go to Tools → Terminal.

How to Add Linux Server to Cockpit

Important: Be aware that you must install Cockpit on all remote Linux servers in order to monitor them on Cockpit dashboard. So, please install it before adding any new server to Cockpit..

To add another server, click on dashboard, you will see the screen below. Click on the (+) sign and enter the server IP address. Remember that information for each server you add is displayed in Cockpit using a distinct color.

Same way, you can add many Linux servers under Cockpit and manage it efficiently without any trouble..

That is it for now, however, you can explore more in case you have installed this simple and wonderful server remote manager.

Cockpit Official Documentation: http://cockpit-project.org/guide/latest/

For any questions or suggestions as well as feedback on the topic, do not hesitate to use the comment section below to get back to us.

Source

Web-Vmstat it’s a small application written in Java and HTML which displays live Linux system statistics, such as Memory, CPU, I/O, Processes, etc. taken over vmstat monitoring command line in a pretty Web page with charts (SmoothieCharts) and diagrams through WebSocket streams using websocketd program.

I’ve recorded a quick video review of what the application can do on a Gentoo system.

Requirements

On a Linux system the following utilities must be installed.

1. A wget for retrieving files using HTTP, HTTPS and FTP protocols.
2. Nano or VI CLI Text Editor.
3. Unzip Archive Extractor.

This tutorial will guide you through installing Web-Vmstat application on CentOS 6.5, but the procedure is valid for all Linux distributions, the only things that differ are just the init scripts (optional), which helps you manage more easy the entire process.

Read Also: Monitor Linux Performance using Vmstat Commands

Step 1: Install Web-Vmstat

1. Before proceeding with installing Web-Vmstat, make sure you have all the above required commands installed on your system. You can use package manager such as yum, apt-get, etc command to install it. For example, under CentOS systems, we use yum command to install it.

# yum install wget nano unzip

2. Now go to Veb-Vmstat official web page at and download the latest version using Download ZIP button or use wget to download from command line.

# wget https://github.com/joewalnes/web-vmstats/archive/master.zip

3. Extract the downloaded master.zip archive using unzip utility and enter to extracted folder.

# unzip master.zip
# cd web-vmstats-master

4. Web directory holds the HTML and Java files needed for the application to run in a Web environment. Create a directory under your system where you want to host the Web files and move all web content to that directory.

This tutorial uses /opt/web_vmstats/ to host all application web files, but you can create any arbitrary path on your system you like it, just assure you retain the absolute web path.

# mkdir /opt/web_vmstats
# cp -r web/* /opt/web_vmstats/

5. Next step is to download and install websocketd streaming program. Go to the official WebSocket page and download the package to match your system architecture (Linux 64-bit, 32-bit or ARM).

On 32-bit System

# wget https://github.com/joewalnes/websocketd/releases/download/v0.2.9/websocketd-0.2.9-linux_386.zip

On 64-bit System

# wget https://github.com/joewalnes/websocketd/releases/download/v0.2.9/websocketd-0.2.9-linux_amd64.zip

6. Extract the WebSocket archive with unzip command and copy websocketd binary to a system executable path to make it available system-wide.

# unzip websocketd-0.2.9-linux_amd64.zip
# cp websocketd /usr/local/bin/

7. Now you can test it by running websocketd command using the following command syntax.

# websocketd --port=8080 --staticdir=/opt/web_vmstats/ /usr/bin/vmstat -n 1

Description of the each parameter explained below.

1. –port=8080: A port used to connect on HTTP protocol – you can use any port number you want.
2. –staticdir=/opt/web_vmstats/: The path where all Web-Vmstat web files are hosted.
3. /usr/bin/vmstat -n 1: A Linux Vmstat command which updates its status every second.

Step 2: Create Init File

8. This step is optional and only works with init script supported systems. To manage WebSocket process as a system daemon create a init service file on /etc/init.d/ path with the following content.

# nano /etc/init.d/web-vmstats

Add the following content.

#!/bin/sh
# source function library
. /etc/rc.d/init.d/functions
start() {
                echo "Starting webvmstats process..."

/usr/local/bin/websocketd --port=8080 --staticdir=/opt/web_vmstats/ /usr/bin/vmstat -n 1 &
}

stop() {
                echo "Stopping webvmstats process..."
                killall websocketd
}

case "$1" in
    start)
       start
        ;;
    stop)
       stop
        ;;
    *)
        echo "Usage: stop start"
        ;;
esac

9. After the file has been created, append execution permissions and manage the process using start or stopswitches.

# chmod +x /etc/init.d/web-vmstats
# /etc/init.d/web-vmstats start

10. If your Firewall is active edit /etc/sysconfig/iptables firewall file and open the port used by websocketd process to make it available for outside connections.

# nano /etc/sysconfig/iptables

If you use port 8080 as in this tutorial add the following line to iptables file after the rule that opens port 22.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

11. To finalize the whole process restart iptables service to apply the new rule.

# service iptables restart
# service web-vmstats start

Open a browser and use the following URL to display Vmstats system statistics.

http://system_IP:8080

12. To display name, version and other details about your current machine and the operating system running on it. Go to Web-Vmstat files path and run the following commands.

# cd /opt/web_vmstats
# cat /etc/issue.net | head -1 > version.txt
# cat /proc/version >> version.txt

13. Then open index.html file and add the following javascript code before <main id=”charts”> line.

# nano index.html

Use the following JavaScript code.

<div align='center'><h3><pre id="contents"></pre></h3></div>
<script>
function populatePre(url) {
    var xhr = new XMLHttpRequest();
    xhr.onload = function () {
        document.getElementById('contents').textContent = this.responseText;
    };
    xhr.open('GET', url);
    xhr.send();
}
populatePre('version.txt');
                </script>

14. To view the final result refresh http://system_IP:8080 web page and you should see information and live statistics about your current machine as in the screenshots below.

TCPflow is a free, open source, powerful command line based tool for analyzing network traffic on Unix-like systems such as Linux. It captures data received or transferred over TCP connections, and stores it in a file for later analysis, in a useful format that allows for protocol analysis and debugging.

Read Also: 16 Best Bandwidth Monitoring Tools to Analyze Network Usage in Linux

It is actually a tcpdump-like tools as it processes packets from the wire or from a stored file. It supports the same powerful filtering expressions supported by its counterpart. The only difference is that tcpflow puts all the TCP packets into order and assembles each flow in a separate file (a file for each direction of flow) for later analysis.

Its feature set includes an advanced plug-in system for decompressing compressed HTTP connections, undoing MIME encoding, or invoking third-party programs for post-processing and much more.

There are many use cases for tcpflow which include to understand network packet flows and also supports for performing network forensics and divulge the contents of HTTP sessions.

How to Install TCPflow in Linux Systems

TCPflow is available in the official repositories of mainstream GNU/Linux distributions, you can install it using your package manager as shown.

$ sudo apt install tcpflow	#Debian/Ubuntu
$ sudo yum install tcpflow	#CentOS/RHEL
$ sudo dnf install tcpflow	#Fedora 22+

After installing tcpflow, you can run it with superuser privileges, otherwise use the sudo command. Note that it listens on the active network interface (for instance enp0s3).

$ sudo tcpflow

tcpflow: listening on enp0s3

By default tcpflow stores all captured data in files that have names in the form (this may be different if you use certain options such as timestamp).

sourceip.sourceport-destip.destport
192.168.043.031.52920-216.058.210.034.00443

Now let’s do a directory listing to see if tcp flow has been captured in any files.

$ ls -1

total 20
-rw-r--r--. 1 root    root     808 Sep 19 12:49 192.168.043.031.52920-216.058.210.034.00443
-rw-r--r--. 1 root    root      59 Sep 19 12:49 216.058.210.034.00443-192.168.043.031.52920

As we mentioned earlier on, each TCP flow is stored in its own file. From the output above, you can see that there are three transcript file, which indicate tcpflow in two opposite directions, where the source IP in the first file and the destination IP in the second file and vice versa.

The first file 192.168.043.031.52920-216.058.210.034.00443 contains data transfered from host 192.168.043.031 (the localhost on which tcpflow was run) via port 52920, to host 216.058.210.034 (the remote host) via port 443.

And the second file 216.058.210.034.00443-192.168.043.031.52920 contains data sent from host 216.058.210.034 (the remote host) via port 443 to host 192.168.043.031 (the localhost on which tcpflow was run) via port 52920.

There is also an XML report generated, which contains information about the program such as how it was compiled, and the computer it was run on and a record of every tcp connection.

As you may have noticed, tcpflow stores the transcript files in the current directory by default. The -o option can help you specify the output directory where the transcript files will be written.

$ sudo tcpflow -o tcpflow_files
$ sudo ls -l tcpflow_files

total 32
-rw-r--r--. 1 root root 1665 Sep 19 12:56 157.240.016.035.00443-192.168.000.103.45986
-rw-r--r--. 1 root root   45 Sep 19 12:56 169.044.082.101.00443-192.168.000.103.55496
-rw-r--r--. 1 root root 2738 Sep 19 12:56 172.217.166.046.00443-192.168.000.103.39954
-rw-r--r--. 1 root root   68 Sep 19 12:56 192.168.000.102.00022-192.168.000.103.42436
-rw-r--r--. 1 root root  573 Sep 19 12:56 192.168.000.103.39954-172.217.166.046.00443
-rw-r--r--. 1 root root 4067 Sep 19 12:56 192.168.000.103.45986-157.240.016.035.00443
-rw-r--r--. 1 root root   38 Sep 19 12:56 192.168.000.103.55496-169.044.082.101.00443
-rw-r--r--. 1 root root 3159 Sep 19 12:56 report.xml

You can also print the contents of packets to stdout as they are received, without storing any captured data to files, using the -c flag as follows.

To test this effectively, open a second terminal and run a ping, or browse the internet. You should be able to see the ping details or your browsing details being captured by tcpflow.

$ sudo tcpflow -c

It is possible to capture all traffic on a particular port, for example port 80 (HTTP). In the case of HTTP traffic, you will be able to see the HTTP Headers followed by the content all on the stdout or in one file if the -c switch is removed.

$ sudo tcpflow port 80

To capture packets from a specific network interface, use the -i flag to specify the interface name.

$ sudo tcpflow -i eth0 port 80

You can also specify a target host (accepted values are IP address, hostname or domains), as shown.

$ sudo tcpflow -c host 192.68.43.1
OR
$ sudo tcpflow -c host www.google.com 

You can enable all processing using all scanners with the -a flag, this is equivalent to the -e all switch.

$ sudo tcpflow -a  
OR
$ sudo tcpflow -e all

A specific scanner can also be activated; the available scanners include md5, http, netviz, tcpdemux and wifiviz (run tcpflow -H to view detailed information about each scanner).

$ sudo tcpflow -e http
OR
$ sudo tcpflow -e md5
OR
$ sudo tcpflow -e netviz
OR
$ sudo tcpflow -e tcpdemux
OR
$ sudo tcpflow -e wifiviz

The following example show how to enable all scanners except tcpdemux.

$ sudo tcpflow -a -x tcpdemux 

TCPflow usually tries to put the network interface into promiscuous mode before capturing packets. You can prevent this using the -p flag as shown.

$ sudo tcpflow -p -i eth0

To read packets from a tcpdump pcap file, use the -r flag.

$ sudo tcpflow -f file.pcap

You can enable verbose mode using the -v or -d 10 options.

$ sudo tcpflow -v
OR
$ sudo tcpflow -d 10

Important: One limitation of tcpflow is that, at the present time it does not understand IP fragments, thus data transmitted as part of TCP connections containing IP fragments will not be properly captured.

For more information and usage options, see the tcpflow man page.

$ man tcpflow 

TCPflow Github repository: https://github.com/simsong/tcpflow

That’s all for now! TCPflow is a powerful TCP flow recorder which is useful for understanding network packet flows and performing network forensics, and so much more. Try it out and share your thoughts about it with us in the comments.

Source