{"id":12357,"date":"2019-03-26T21:24:19","date_gmt":"2019-03-26T21:24:19","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=12357"},"modified":"2019-03-26T21:24:19","modified_gmt":"2019-03-26T21:24:19","slug":"manage-log-messages-under-systemd-using-journalctl-comprehensive-guide","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/03\/26\/manage-log-messages-under-systemd-using-journalctl-comprehensive-guide\/","title":{"rendered":"Manage Log Messages Under Systemd Using Journalctl [Comprehensive Guide]"},"content":{"rendered":"

Systemd<\/strong>\u00a0is a cutting-edge system and service manager for Linux systems: an init daemon replacement intended to start processes in parallel at system boot. It is now supported in a number of current mainstream distribution including Fedora, Debian, Ubuntu, OpenSuSE, Arch, RHEL, CentOS, etc.<\/p>\n

Earlier on, we explained the\u00a0story behind \u2018init\u2019 and \u2018systemd\u2019<\/a>; where we discussed what the two daemons are, why \u2018init<\/strong>\u2019 technically needed to be replaced with \u2018systemd<\/strong>\u2019 as well as the main features of systemd.<\/p>\n

One of the main advantages of systemd over other common init systems is, support for centralized management of system and processes logging using a journal. In this article, we will learn how to manage and view log messages under systemd using\u00a0journalctl<\/strong>\u00a0command in Linux.<\/p>\n

Important<\/strong>: Before moving further in this guide, you may want to learn how to manage\u00a0\u2018Systemd\u2019 services and units using \u2018Systemctl\u2019<\/a>\u00a0command, and also\u00a0create and run new service units in systemd<\/a>\u00a0using shell scripts in Linux. However, if you are okay with all the above, continue reading through.<\/p>\n

Configuring Journald for Collecting Log Messages Under Systemd<\/h3>\n

journald<\/strong>\u00a0is a daemon which gathers and writes journal entries from the entire system; these are essentially boot messages, messages from kernel and from syslog or various applications and it stores all the messages in a central location \u2013 journal file.<\/p>\n

You can control the behavior of\u00a0journald<\/strong>\u00a0via its default configuration file:\u00a0\/etc\/systemd\/journald.conf<\/strong>\u00a0which is generated at compile time. This file contains options whose values you may change to suite your local environment requirements.<\/p>\n

Below is a sample of what the file looks like, viewed using the\u00a0cat command<\/a>.<\/p>\n

$ cat \/etc\/systemd\/journald.conf \r\n<\/pre>\n
Journald Configuration File<\/div>\n
# See journald.conf(5) for details.\r\n\r\n[Journal]\r\n#Storage=auto\r\n#Compress=yes\r\n#Seal=yes\r\n#SplitMode=uid\r\n#SyncIntervalSec=5m\r\n#RateLimitInterval=30s\r\n#RateLimitBurst=1000\r\n#SystemMaxUse=\r\n#SystemKeepFree=\r\n#SystemMaxFileSize=\r\n#SystemMaxFiles=100\r\n#RuntimeMaxUse=\r\n#RuntimeKeepFree=\r\n#RuntimeMaxFileSize=\r\n#RuntimeMaxFiles=100\r\n#MaxRetentionSec=\r\n#MaxFileSec=1month\r\n#ForwardToSyslog=yes\r\n#ForwardToKMsg=no\r\n#ForwardToConsole=no\r\n#ForwardToWall=yes\r\n#TTYPath=\/dev\/console\r\n#MaxLevelStore=debug\r\n#MaxLevelSyslog=debug\r\n#MaxLevelKMsg=notice\r\n#MaxLevelConsole=info\r\n#MaxLevelWall=emerg\r\n<\/pre>\n
Note that various package installs and use configuration extracts in\u00a0\/usr\/lib\/systemd\/*.conf.d\/<\/strong>\u00a0and run time configurations can be found in\u00a0\/run\/systemd\/journald.conf.d\/*.conf<\/strong>\u00a0which you may not necessarily use.<\/p>\n
Enable Journal Data Storage On Disk<\/h4>\n
A number of Linux distributions including Ubuntu and it\u2019s derivatives like Linux Mint do not enable persistent storage of boot messages on disk by default.<\/p>\n
It is possible to enable this by setting the \u201cStorage<\/strong>\u201d option to \u201cpersistent<\/strong>\u201d as shown below. This will create the\u00a0\/var\/log\/journal<\/strong>\u00a0directory and all journal files will be stored under it.<\/p>\n
$ sudo vi \/etc\/systemd\/journald.conf \r\nOR\r\n$ sudo nano \/etc\/systemd\/journald.conf \r\n<\/pre>\n
[Journal]\r\nStorage=persistent\r\n<\/pre>\n
For additional settings, find the meaning of all options which are supposed to be configured under the\u00a0\u201c[Journal]\u201d<\/strong>\u00a0section by typing.<\/p>\n
$ man journald.conf\r\n<\/pre>\n
Setting Correct System Time Using Timedatectl Command<\/h4>\n
For reliable log management under systemd using journald service, ensure that the time settings including the timezone is correct on the system.<\/p>\n
In order to view the current date and time settings on your system, type.<\/p>\n
$ timedatectl \r\nOR\r\n$ timedatectl status\r\n\r\nLocal time: Thu 2017-06-15 13:29:09 EAT\r\nUniversal time: Thu 2017-06-15 10:29:09 UTC\r\nRTC time: Thu 2017-06-15 10:29:09\r\nTime zone: Africa\/Kampala (EAT, +0300)\r\nNetwork time on: yes\r\nNTP synchronized: yes\r\n RTC in local TZ: no\r\n<\/pre>\n
To set the correct timezone and possibly system time, use the commands below.<\/p>\n
$ sudo timedatectl set-timezone  Africa\/Kampala\r\n$ sudo timedatectl set-time \u201c13:50:00\u201d\r\n<\/pre>\n
Viewing Log Messages Using Journalctl Command<\/h4>\n
journalctl<\/strong>\u00a0is a utility used to view the contents of the systemd journal (which is written by journald service).<\/p>\n
To show all collected logs without any filtering, type.<\/p>\n
$ journalctl\r\n<\/pre>\n
View Log Messages<\/div>\n
-- Logs begin at Wed 2017-06-14 21:56:43 EAT, end at Thu 2017-06-15 12:28:19 EAT\r\nJun 14 21:56:43 tecmint systemd-journald[336]: Runtime journal (\/run\/log\/journal\r\nJun 14 21:56:43 tecmint kernel: Initializing cgroup subsys cpuset\r\nJun 14 21:56:43 tecmint kernel: Initializing cgroup subsys cpu\r\nJun 14 21:56:43 tecmint kernel: Initializing cgroup subsys cpuacct\r\nJun 14 21:56:43 tecmint kernel: Linux version 4.4.0-21-generic (buildd@lgw01-21)\r\nJun 14 21:56:43 tecmint kernel: Command line: BOOT_IMAGE=\/boot\/vmlinuz-4.4.0-21-\r\nJun 14 21:56:43 tecmint kernel: KERNEL supported cpus:\r\nJun 14 21:56:43 tecmint kernel:   Intel GenuineIntel\r\nJun 14 21:56:43 tecmint kernel:   AMD AuthenticAMD\r\nJun 14 21:56:43 tecmint kernel:   Centaur CentaurHauls\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: xstate_offset[2]:  576, xstate_sizes[2]\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: Supporting XSAVE feature 0x01: 'x87 flo\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: Supporting XSAVE feature 0x02: 'SSE reg\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: Supporting XSAVE feature 0x04: 'AVX reg\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: Enabled xstate features 0x7, context si\r\nJun 14 21:56:43 tecmint kernel: x86\/fpu: Using 'eager' FPU context switches.\r\nJun 14 21:56:43 tecmint kernel: e820: BIOS-provided physical RAM map:\r\nJun 14 21:56:43 tecmint kernel: BIOS-e820: [mem 0x0000000000000000-0x00000000000\r\nJun 14 21:56:43 tecmint kernel: BIOS-e820: [mem 0x0000000000090000-0x00000000000\r\nJun 14 21:56:43 tecmint kernel: BIOS-e820: [mem 0x0000000000100000-0x000000001ff\r\nJun 14 21:56:43 tecmint kernel: BIOS-e820: [mem 0x0000000020000000-0x00000000201\r\nJun 14 21:56:43 tecmint kernel: BIOS-e820: [mem 0x0000000020200000-0x00000000400\r\n<\/pre>\n
View Log messages Based On Boots<\/h4>\n
You can display a list of boot numbers (relative to the current boot), their IDs, and the timestamps of the first and last message corresponding to the boot with the\u00a0--list-boots<\/code>\u00a0option.<\/p>\n
$ journalctl --list-boots\r\n\r\n-1 9fb590b48e1242f58c2579defdbbddc9 Thu 2017-06-15 16:43:36 EAT\u2014Thu 2017-06-15 1\r\n 0 464ae35c6e264a4ca087949936be434a Thu 2017-06-15 16:47:36 EAT\u2014Thu 2017-06-15 1 \r\n\r\n<\/pre>\n
To view the journal entries from the current boot (number 0), use the\u00a0-b<\/code>\u00a0switch like this (same as the sample output above).<\/p>\n
$ journalctl -b\r\n<\/pre>\n
and to see a journal from the previous boot, use the\u00a0-1<\/code>\u00a0relative pointer with the\u00a0-b<\/code>\u00a0option as below.<\/p>\n
$ journalctl -b -1\r\n<\/pre>\n
Alternatively, use the boot ID like this.<\/p>\n
$ journalctl -b 9fb590b48e1242f58c2579defdbbddc9\r\n<\/pre>\n
Filtering Log Messages Based On Time<\/h4>\n
To use time in Coordinated Universal Time (UTC) format, add the\u00a0--utc<\/code>\u00a0options as follows.<\/p>\n
$ journalctl --utc\r\n<\/pre>\n
To see all of the entries since a particular date and time, e.g. June 15th, 2017 at 8:15 AM, type this command.<\/p>\n
$ journalctl --since \"2017-06-15 08:15:00\"\r\n$ journalctl --since today\r\n$ journalctl --since yesterday\r\n<\/pre>\n
Viewing Recent Log Messages<\/h4>\n
To view recent log messages (10 by default), use the\u00a0-n<\/code>\u00a0flag as shown below.<\/p>\n
$ journalctl -n\r\n$ journalctl -n 20 \r\n<\/pre>\n
Viewing Log Messages Generated By Kernel<\/h4>\n
To see only kernel messages, similar to the\u00a0dmesg command<\/a>\u00a0output, you can use the\u00a0-k<\/code>\u00a0flag.<\/p>\n
$ journalctl -k \r\n$ journalctl -k -b \r\n$ journalctl -k -b 9fb590b48e1242f58c2579defdbbddc9\r\n<\/pre>\n
Viewing Log Messages Generated By Units<\/h4>\n
To can view all journal entries for a particular unit, use the\u00a0-u<\/code>\u00a0switch as follows.<\/p>\n
$ journalctl -u apache2.service\r\n<\/pre>\n
To zero down to the current boot, type this command.<\/p>\n
$ journalctl -b -u apache2.service\r\n<\/pre>\n
To show logs from the previous boot, use this.<\/p>\n
$ journalctl -b -1 -u apache2.service\r\n<\/pre>\n
Below are some other useful commands:<\/p>\n
$ journalctl -u apache2.service  \r\n$ journalctl -u apache2.service --since today\r\n$ journalctl -u apache2.service -u nagios.service --since yesterday\r\n<\/pre>\n
Viewing Log Messages Generated By Processes<\/h4>\n
To view logs generated by a specific process, specify it\u2019s PID like this.<\/p>\n
$ journalctl _PID=19487\r\n$ journalctl _PID=19487 --since today\r\n$ journalctl _PID=19487 --since yesterday\r\n<\/pre>\n
Viewing Log Messages Generated By User or Group ID<\/h4>\n
To view logs generated by a specific user or group, specify it\u2019s user or group ID like this.<\/p>\n
$ journalctl _UID=1000\r\n$ journalctl _UID=1000 --since today\r\n$ journalctl _UID=1000 -b -1 --since today\r\n<\/pre>\n
Viewing Logs Generated By a File<\/h4>\n
To show all logs generated by a file (possibly an executable), such as the D-Bus executable or bash executables, simply type.<\/p>\n
$ journalctl \/usr\/bin\/dbus-daemon\r\n$ journalctl \/usr\/bin\/bash\r\n<\/pre>\n
Viewing Log Messages By Priority<\/h4>\n
You can also filter output based on message priorities or priority ranges using the\u00a0-p<\/code>\u00a0flag. The possible values are: 0 \u2013 emerg, 1 \u2013 alert, 2 \u2013 crit, 3 \u2013 err, 4 \u2013 warning, 5 \u2013 notice, 6 \u2013 info, 7 \u2013 debug):<\/p>\n
$ journalctl -p err\r\n<\/pre>\n
To specify a range, use the format below (emerg to warning).<\/p>\n
$ journalctl -p 1..4\r\nOR\r\n$ journalctl -p emerg..warning\r\n<\/pre>\n
View Log Messages in Real-Time<\/h4>\n
You can practically watch logs as they are being written with the\u00a0-f<\/code>\u00a0option (similar to\u00a0tail -f<\/strong>\u00a0functionality).<\/p>\n
$ journalctl -f\r\n<\/pre>\n
Handling Journal Display Formatting<\/h4>\n
If you want to control the output formatting of the journal entries, add the\u00a0-o<\/code>\u00a0flag and use these options: cat, export, json, json-pretty, json-sse, short, short-iso, short-monotonic, short-precise and verbose(check meaning of options in the man page:<\/p>\n
The\u00a0cat<\/strong>\u00a0option shows the actual message of each journal entry without any metadata (timestamp and so on).<\/p>\n
$ journalctl -b -u apache2.service -o cat\r\n<\/pre>\n
Managing Journals On a System<\/h4>\n
To check the journal file for internal consistency, use the\u00a0--verify<\/code>\u00a0option. If all is well, the output should indicate a PASS.<\/p>\n
$ journalctl --verify\r\n\r\nPASS: \/run\/log\/journal\/2a5d5f96ef9147c0b35535562b32d0ff\/system.journal                               \r\n491f68: Unused data (entry_offset==0)                                                                \r\nPASS: \/run\/log\/journal\/2a5d5f96ef9147c0b35535562b32d0ff\/system@816533ecd00843c4a877a0a962e124f2-0000000000003184-000551f9866c3d4d.journal\r\nPASS: \/run\/log\/journal\/2a5d5f96ef9147c0b35535562b32d0ff\/system@816533ecd00843c4a877a0a962e124f2-0000000000001fc8-000551f5d8945a9e.journal\r\nPASS: \/run\/log\/journal\/2a5d5f96ef9147c0b35535562b32d0ff\/system@816533ecd00843c4a877a0a962e124f2-0000000000000d4f-000551f1becab02f.journal\r\nPASS: \/run\/log\/journal\/2a5d5f96ef9147c0b35535562b32d0ff\/system@816533ecd00843c4a877a0a962e124f2-0000000000000001-000551f01cfcedff.journal\r\n<\/pre>\n
Deleting Old Journal Files<\/h4>\n
You can also display the current disk usage of all journal files with the\u00a0--disk-usage<\/code>\u00a0options. It shows the sum of the disk usage of all archived and active journal files:<\/p>\n
$ journalctl --disk-usage\r\n<\/pre>\n
To delete old (archived) journal files run the commands below:<\/p>\n
$ sudo journalctl --vacuum-size=50M  #delete files until the disk space they use falls below the specified size\r\n$ sudo journalctl --vacuum-time=1years\t#delete files so that all journal files contain no data older than the specified timespan\r\n$ sudo journalctl --vacuum-files=4     #delete files so that no more than the specified number of separate journal files remain in storage location\r\n<\/pre>\n
Rotating Journal Files<\/h4>\n
Last but not least, you can instruct journald to rotate journal files with the\u00a0--rotate<\/code>\u00a0option. Note that this directive does not return until the rotation operation is finished:<\/p>\n
$ sudo journalctl --rotate\r\n<\/pre>\n
For an in-depth usage guide and options, view the journalctl man page as follows.<\/p>\n
$ man journalctl\r\n<\/pre>\n
Do check out some useful articles.<\/p>\n
    \n
  1. Managing System Startup Process and Services (SysVinit, Systemd and Upstart)<\/a><\/li>\n
  2. Petiti \u2013 An Open Source Log Analysis Tool for Linux SysAdmins<\/a><\/li>\n
  3. How to Setup and Manage Log Rotation Using Logrotate in Linux<\/a><\/li>\n
  4. lnav \u2013 Watch and Analyze Apache Logs from a Linux Terminal<\/a><\/li>\n<\/ol>\n
    That\u2019s it for now. Use the feedback from below to ask any questions or add you thoughts on this topic.<\/p>\n
    Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    Systemd\u00a0is a cutting-edge system and service manager for Linux systems: an init daemon replacement intended to start processes in parallel at system boot. It is now supported in a number of current mainstream distribution including Fedora, Debian, Ubuntu, OpenSuSE, Arch, RHEL, CentOS, etc. Earlier on, we explained the\u00a0story behind \u2018init\u2019 and \u2018systemd\u2019; where we discussed … <\/p>\n
    Continue reading “Manage Log Messages Under Systemd Using Journalctl [Comprehensive Guide]”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-12357","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12357","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=12357"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12357\/revisions"}],"predecessor-version":[{"id":12358,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/12357\/revisions\/12358"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=12357"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=12357"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=12357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}