{"id":11528,"date":"2019-03-14T11:03:58","date_gmt":"2019-03-14T11:03:58","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=11528"},"modified":"2019-03-14T11:03:58","modified_gmt":"2019-03-14T11:03:58","slug":"bcc-dynamic-tracing-tools-for-linux-performance-monitoring-networking-and-more","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/03\/14\/bcc-dynamic-tracing-tools-for-linux-performance-monitoring-networking-and-more\/","title":{"rendered":"BCC \u2013 Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More"},"content":{"rendered":"

BCC<\/strong>\u00a0(BPF Compiler Collection<\/strong>) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs. It utilizes extended\u00a0BPF<\/strong>\u00a0(Berkeley Packet Filters<\/strong>), initially known as\u00a0eBPF<\/strong>\u00a0which was one of the new features in\u00a0Linux 3.15<\/strong>.<\/p>\n

\n

\"BCC\/BPF<\/a><\/p>\n

BCC\/BPF \u2013 Dynamic Tracing Tools for Linux Performance Monitoring<\/p>\n<\/div>\n

Practically, most of the components used by\u00a0BCC<\/strong>\u00a0require\u00a0Linux 4.1<\/strong>\u00a0or above, and its noteworthy features include:<\/p>\n

    \n
  1. Requires no 3rd party kernel module, since all the tools work based on BPF which is built into the kernel and BCC uses features added in Linux 4.x series.<\/li>\n
  2. Enables observation of software execution.<\/li>\n
  3. Comprises of several performance analysis tools with example files and man pages.<\/li>\n<\/ol>\n

    Suggested Read:<\/b>\u00a020 Command Line Tools to Monitor Linux Performance<\/a><\/p>\n

    Best suited for advanced Linux users,\u00a0BCC<\/strong>\u00a0makes it easy to write\u00a0BPF<\/strong>\u00a0programs using kernel instrumentation in\u00a0C<\/strong>, and front-ends in\u00a0Python<\/strong>\u00a0and\u00a0lua<\/strong>. Additionally, it supports multiple tasks such as performance analysis, monitoring, network traffic control plus lots more.<\/p>\n

    How To Install BCC in Linux Systems<\/h3>\n

    Remember that\u00a0BCC<\/strong>\u00a0uses features added in Linux kernel version\u00a04.1<\/strong>\u00a0or above, and as a requirement, the kernel should have been compiled with the flags set below:<\/p>\n

    CONFIG_BPF<\/strong>=y<\/strong>\r\nCONFIG_BPF_SYSCALL<\/strong>=y<\/strong>\r\n# [optional, for tc filters]\r\nCONFIG_NET_CLS_BPF<\/strong>=m<\/strong>\r\n# [optional, for tc actions]\r\nCONFIG_NET_ACT_BPF<\/strong>=m<\/strong>\r\nCONFIG_BPF_JIT<\/strong>=y<\/strong>\r\nCONFIG_HAVE_BPF_JIT<\/strong>=y<\/strong>\r\n# [optional, for kprobes]\r\nCONFIG_BPF_EVENTS<\/strong>=y<\/strong><\/pre>\n
    To check your kernel flags, view the file\u00a0\/proc\/config.gz<\/strong>\u00a0or run the commands as in the examples below:<\/p>\n
    tecmint@TecMint ~ $<\/strong> grep CONFIG_BPF= \/boot\/config-`uname -r`\r\nCONFIG_BPF<\/strong>=y<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_BPF_SYSCALL= \/boot\/config-`uname -r`\r\nCONFIG_BPF_SYSCALL<\/strong>=y<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_NET_CLS_BPF= \/boot\/config-`uname -r`\r\nCONFIG_NET_CLS_BPF<\/strong>=m<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_NET_ACT_BPF= \/boot\/config-`uname -r`\r\nCONFIG_NET_ACT_BPF<\/strong>=m<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_BPF_JIT= \/boot\/config-`uname -r`\r\nCONFIG_BPF_JIT<\/strong>=y<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_HAVE_BPF_JIT= \/boot\/config-`uname -r`\r\nCONFIG_HAVE_BPF_JIT<\/strong>=y<\/strong>\r\ntecmint@TecMint ~ $<\/strong> grep CONFIG_BPF_EVENTS= \/boot\/config-`uname -r`\r\nCONFIG_BPF_EVENTS<\/strong>=y<\/strong>\r\n<\/pre>\n
    After verifying kernel flags, it\u2019s time to install\u00a0BCC<\/strong>\u00a0tools in Linux systems.<\/p>\n
    On Ubuntu 16.04<\/h4>\n
    Only the nightly packages are created for\u00a0Ubuntu 16.04<\/strong>, but the installation instructions are very straightforward. No need of kernel upgrade or compile it from source.<\/p>\n
    $ echo \"deb [trusted=yes] https:\/\/repo.iovisor.org\/apt\/xenial xenial-nightly main\" | sudo tee \/etc\/apt\/sources.list.d\/iovisor.list\r\n$ sudo apt-get update\r\n$ sudo apt-get install bcc-tools\r\n<\/pre>\n
    On Ubuntu 14.04<\/h4>\n
    Begin by installing a\u00a04.3+<\/strong>\u00a0Linux kernel, from\u00a0http:\/\/kernel.ubuntu.com\/~kernel-ppa\/mainline<\/a>.<\/p>\n
    As an example, write a small shell script \u201cbcc-install.sh<\/strong>\u201d with the content below.<\/p>\n
    Note<\/strong>: update\u00a0PREFIX<\/strong>\u00a0value to the latest date, and also browse the files in the\u00a0PREFIX url<\/strong>\u00a0provided to get the actual\u00a0REL<\/strong>\u00a0value, substitute them in the shell script.<\/p>\n
    #!\/bin\/bash\r\nVER=4.5.1-040501\r\nPREFIX=http:\/\/kernel.ubuntu.com\/~kernel-ppa\/mainline\/v4.5.1-wily\/\r\nREL=201604121331\r\nwget ${PREFIX}\/linux-headers-${VER}-generic_${VER}.${REL}_amd64.deb\r\nwget ${PREFIX}\/linux-headers-${VER}_${VER}.${REL}_all.deb\r\nwget ${PREFIX}\/linux-image-${VER}-generic_${VER}.${REL}_amd64.deb\r\nsudo dpkg -i linux-*${VER}.${REL}*.deb\r\n<\/pre>\n
    Save the file and exit. Make it executable, then run it as shown:<\/p>\n
    $ chmod +x bcc-install.sh\r\n$ sh bcc-install.sh\r\n<\/pre>\n
    Afterwards, reboot your system.<\/p>\n
    $ reboot\r\n<\/pre>\n
    Next, run the commands below to install signed BCC packages:<\/p>\n
    $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD\r\n$ echo \"deb https:\/\/repo.iovisor.org\/apt trusty main\" | sudo tee \/etc\/apt\/sources.list.d\/iovisor.list\r\n$ sudo apt-get update\r\n$ sudo apt-get install binutils bcc bcc-tools libbcc-examples python-bcc\r\n<\/pre>\n
    On Fedora 24-23<\/h4>\n
    Install a\u00a04.2+<\/strong>\u00a0kernel from\u00a0http:\/\/alt.fedoraproject.org\/pub\/alt\/rawhide-kernel-nodebug<\/a>, if your system has a version lower than what is required. Below is an example of how to do that:<\/p>\n
    $ sudo dnf config-manager --add-repo=http:\/\/alt.fedoraproject.org\/pub\/alt\/rawhide-kernel-nodebug\/fedora-rawhide-kernel-nodebug.repo\r\n$ sudo dnf update\r\n$ reboot\r\n<\/pre>\n
    After that, add the\u00a0BBC<\/strong>\u00a0tools repository, update your system and install the tools by executing the next series of commands:<\/p>\n
    $ echo -e '[iovisor]\\nbaseurl=https:\/\/repo.iovisor.org\/yum\/nightly\/f23\/$basearch\\nenabled=1\\ngpgcheck=0' | sudo tee \/etc\/yum.repos.d\/iovisor.repo\r\n$ sudo dnf update\r\n$ sudo dnf install bcc-tools\r\n<\/pre>\n
    On Arch Linux \u2013 AUR<\/h4>\n
    You should start by upgrading your kernel to at least version\u00a04.3.1-1<\/strong>, subsequently install the packages below using any Arch package managers such as\u00a0pacaur<\/strong>,\u00a0yaourt<\/strong>,\u00a0cower<\/strong>, etc.<\/p>\n
    bcc bcc-tools python-bcc python2-bcc\r\n<\/pre>\n
    How To Use BCC Tools in Linux Systems<\/h3>\n
    All the\u00a0BCC<\/strong>\u00a0tools are installed under\u00a0\/usr\/share\/bcc\/tools<\/code>\u00a0directory. However, you can alternatively run them from the\u00a0BCC Github<\/strong>\u00a0repository under\u00a0\/tools<\/code>\u00a0where they end with a\u00a0.py<\/code>\u00a0extension.<\/p>\n
    $ ls \/usr\/share\/bcc\/tools <\/strong>\r\n\r\nargdist       capable     filetop         offwaketime  stackcount  vfscount\r\nbashreadline  cpudist     funccount       old          stacksnoop  vfsstat\r\nbiolatency    dcsnoop     funclatency     oomkill      statsnoop   wakeuptime\r\nbiosnoop      dcstat      gethostlatency  opensnoop    syncsnoop   xfsdist\r\nbiotop        doc         hardirqs        pidpersec    tcpaccept   xfsslower\r\nbitesize      execsnoop   killsnoop       profile      tcpconnect  zfsdist\r\nbtrfsdist     ext4dist    mdflush         runqlat      tcpconnlat  zfsslower\r\nbtrfsslower   ext4slower  memleak         softirqs     tcpretrans\r\ncachestat     filelife    mysqld_qslower  solisten     tplist\r\ncachetop      fileslower  offcputime      sslsniff     trace\r\n<\/pre>\n
    We shall cover a few examples under \u2013 monitoring general\u00a0Linux system performance<\/a>\u00a0and networking.<\/p>\n
    Trace open() syscalls<\/h4>\n
    Let\u2019s start by tracing all\u00a0open()<\/code>\u00a0syscalls using\u00a0opensnoop<\/strong>. This enable us tell us how various applications work by identifying their data files, config files and many more:<\/p>\n
    $ cd \/usr\/share\/bcc\/tools <\/strong>\r\n$ sudo .\/opensnoop<\/strong>\r\n\r\nPID    COMM               FD ERR PATH\r\n1      systemd            35   0 \/proc\/self\/mountinfo\r\n2797   udisksd            13   0 \/proc\/self\/mountinfo\r\n1      systemd            35   0 \/sys\/devices\/pci0000:00\/0000:00:0d.0\/ata3\/host2\/target2:0:0\/2:0:0:0\/block\/sda\/sda1\/uevent\r\n1      systemd            35   0 \/run\/udev\/data\/b8:1\r\n1      systemd            -1   2 \/etc\/systemd\/system\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/run\/systemd\/system\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/run\/systemd\/generator\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/usr\/local\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount\r\n2247   systemd            15   0 \/proc\/self\/mountinfo\r\n1      systemd            -1   2 \/lib\/systemd\/system\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/usr\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/run\/systemd\/generator.late\/sys-kernel-debug-tracing.mount\r\n1      systemd            -1   2 \/etc\/systemd\/system\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/etc\/systemd\/system\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/run\/systemd\/system\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/run\/systemd\/system\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/run\/systemd\/generator\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/run\/systemd\/generator\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/usr\/local\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/usr\/local\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/usr\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/usr\/lib\/systemd\/system\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/run\/systemd\/generator.late\/sys-kernel-debug-tracing.mount.wants\r\n1      systemd            -1   2 \/run\/systemd\/generator.late\/sys-kernel-debug-tracing.mount.requires\r\n1      systemd            -1   2 \/etc\/systemd\/system\/sys-kernel-debug-tracing.mount.d\r\n1      systemd            -1   2 \/run\/systemd\/system\/sys-kernel-debug-tracing.mount.d\r\n1      systemd            -1   2 \/run\/systemd\/generator\/sys-kernel-debug-tracing.mount.d\r\n....\r\n<\/pre>\n
    Summarize Block Device I\/O Latency<\/h4>\n
    In this example, it shows a summarized distribution of disk I\/O latency using biolatecncy. After executing the command, wait for a few minutes and hit\u00a0Ctrl-C<\/strong>\u00a0to end it and view the output.<\/p>\n
    $ sudo .\/biolatecncy<\/strong>\r\n\r\nTracing block device I\/O... Hit Ctrl-C to end.\r\n^C\r\n     usecs               : count     distribution\r\n         0 -> 1          : 0        |                                        |\r\n         2 -> 3          : 0        |                                        |\r\n         4 -> 7          : 0        |                                        |\r\n         8 -> 15         : 0        |                                        |\r\n        16 -> 31         : 0        |                                        |\r\n        32 -> 63         : 0        |                                        |\r\n        64 -> 127        : 0        |                                        |\r\n       128 -> 255        : 3        |****************************************|\r\n       256 -> 511        : 3        |****************************************|\r\n       512 -> 1023       : 1        |*************                           |\r\n<\/pre>\n
    Trace New Processes via exec() Syscalls<\/h4>\n
    In this section, we shall move to tracing new processes in execution using\u00a0execsnoop<\/strong>\u00a0tool. Each time a process is forked by\u00a0fork()<\/code>\u00a0and\u00a0exec()<\/code>\u00a0syscalls, it is shown in the output. However, not all processes are captured.<\/p>\n
    $ sudo .\/execsnoop<\/strong>\r\n\r\nPCOMM            PID    PPID   RET ARGS\r\ngnome-screensho  14882  14881    0 \/usr\/bin\/gnome-screenshot --gapplication-service\r\nsystemd-hostnam  14892  1        0 \/lib\/systemd\/systemd-hostnamed\r\nnautilus         14897  2767    -2 \/home\/tecmint\/bin\/net usershare info\r\nnautilus         14897  2767    -2 \/home\/tecmint\/.local\/bin\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/local\/sbin\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/local\/bin\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/sbin\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/bin\/net usershare info\r\nnautilus         14897  2767    -2 \/sbin\/net usershare info\r\nnautilus         14897  2767    -2 \/bin\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/games\/net usershare info\r\nnautilus         14897  2767    -2 \/usr\/local\/games\/net usershare info\r\nnautilus         14897  2767    -2 \/snap\/bin\/net usershare info\r\ncompiz           14899  14898   -2 \/home\/tecmint\/bin\/libreoffice --calc\r\ncompiz           14899  14898   -2 \/home\/tecmint\/.local\/bin\/libreoffice --calc\r\ncompiz           14899  14898   -2 \/usr\/local\/sbin\/libreoffice --calc\r\ncompiz           14899  14898   -2 \/usr\/local\/bin\/libreoffice --calc\r\ncompiz           14899  14898   -2 \/usr\/sbin\/libreoffice --calc\r\nlibreoffice      14899  2252     0 \/usr\/bin\/libreoffice --calc\r\ndirname          14902  14899    0 \/usr\/bin\/dirname \/usr\/bin\/libreoffice\r\nbasename         14903  14899    0 \/usr\/bin\/basename \/usr\/bin\/libreoffice\r\n...\r\n<\/pre>\n
    Trace Slow ext4 Operations<\/h4>\n
    Using\u00a0ext4slower<\/strong>\u00a0to trace the\u00a0ext4<\/strong>\u00a0file system common operations that are slower than\u00a010ms<\/strong>, to help us identify independently slow disk I\/O via the file system.<\/p>\n
    Suggested Read:<\/b>\u00a013 Linux Performance Monitoring Tools<\/a><\/p>\n
    It only outputs those operations that exceed a threshold:<\/p>\n
    $ sudo .\/execslower<\/strong>\r\n\r\nTracing ext4 operations slower than 10 ms\r\nTIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME\r\n11:59:13 upstart        2252   W 48      1          10.76 dbus.log\r\n11:59:13 gnome-screensh 14993  R 144     0          10.96 settings.ini\r\n11:59:13 gnome-screensh 14993  R 28      0          16.02 gtk.css\r\n11:59:13 gnome-screensh 14993  R 3389    0          18.32 gtk-main.css\r\n11:59:25 rs:main Q:Reg  1826   W 156     60         31.85 syslog\r\n11:59:25 pool           15002  R 208     0          14.98 .xsession-errors\r\n11:59:25 pool           15002  R 644     0          12.28 .ICEauthority\r\n11:59:25 pool           15002  R 220     0          13.38 .bash_logout\r\n11:59:27 dconf-service  2599   S 0       0          22.75 user.BHDKOY\r\n11:59:33 compiz         2548   R 4096    0          19.03 firefox.desktop\r\n11:59:34 compiz         15008  R 128     0          27.52 firefox.sh\r\n11:59:34 firefox        15008  R 128     0          36.48 firefox\r\n11:59:34 zeitgeist-daem 2988   S 0       0          62.23 activity.sqlite-wal\r\n11:59:34 zeitgeist-fts  2996   R 8192    40         15.67 postlist.DB\r\n11:59:34 firefox        15008  R 140     0          18.05 dependentlibs.list\r\n11:59:34 zeitgeist-fts  2996   S 0       0          25.96 position.tmp\r\n11:59:34 firefox        15008  R 4096    0          10.67 libplc4.so\r\n11:59:34 zeitgeist-fts  2996   S 0       0          11.29 termlist.tmp\r\n...\r\n<\/pre>\n
    Trace Block Device I\/O with PID and Latency<\/h4>\n
    Next off, let\u2019s dive into printing a line per disk I\/O each second, with details such as process ID, sector, bytes, latency among others using biosnoop:<\/p>\n
    $ sudo .\/biosnoop<\/strong>\r\n\r\nTIME(s)        COMM           PID    DISK    T  SECTOR    BYTES   LAT(ms)\r\n0.000000000    ?              0              R  -1        8          0.26\r\n2.047897000    ?              0              R  -1        8          0.21\r\n3.280028000    kworker\/u4:0   14871  sda     W  30552896  4096       0.24\r\n3.280271000    jbd2\/sda1-8    545    sda     W  29757720  12288      0.40\r\n3.298318000    jbd2\/sda1-8    545    sda     W  29757744  4096       0.14\r\n4.096084000    ?              0              R  -1        8          0.27\r\n6.143977000    ?              0              R  -1        8          0.27\r\n8.192006000    ?              0              R  -1        8          0.26\r\n8.303938000    kworker\/u4:2   15084  sda     W  12586584  4096       0.14\r\n8.303965000    kworker\/u4:2   15084  sda     W  25174736  4096       0.14\r\n10.239961000   ?              0              R  -1        8          0.26\r\n12.292057000   ?              0              R  -1        8          0.20\r\n14.335990000   ?              0              R  -1        8          0.26\r\n16.383798000   ?              0              R  -1        8          0.17\r\n...\r\n<\/pre>\n
    Trace Page Cache hit\/miss Ratio<\/h4>\n
    Thereafter, we proceed to using\u00a0cachestat<\/strong>\u00a0to displays one line of summarized statistics from the system cache every second. This enables for system tuning operations by pointing out low cache hit ratio and high rate of misses:<\/p>\n
    $ sudo .\/cachestat<\/strong>\r\n\r\n HITS   MISSES  DIRTIES  READ_HIT% WRITE_HIT%   BUFFERS_MB  CACHED_MB\r\n       0        0        0       0.0%       0.0%           19        544\r\n       4        4        2      25.0%      25.0%           19        544\r\n    1321       33        4      97.3%       2.3%           19        545\r\n    7476        0        2     100.0%       0.0%           19        545\r\n    6228       15        2      99.7%       0.2%           19        545\r\n       0        0        0       0.0%       0.0%           19        545\r\n    7391      253      108      95.3%       2.7%           19        545\r\n   33608     5382       28      86.1%      13.8%           19        567\r\n   25098       37       36      99.7%       0.0%           19        566\r\n   17624      239      416      96.3%       0.5%           19        520\r\n...\r\n<\/pre>\n
    Trace TCP Active Connections<\/h4>\n
    Monitoring TCP connections every second using\u00a0tcpconnect<\/strong>. Its output includes source and destination address, and port number. This tool is useful for tracing unexpected TCP connections, thereby helping us to identify inefficiencies in application configurations or an attacker.<\/p>\n
    $ sudo .\/tcpconnect<\/strong>\r\n\r\nPID    COMM         IP SADDR            DADDR            DPORT\r\n15272  Socket Threa 4  10.0.2.15        91.189.89.240    80  \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.142   443 \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.142   80  \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.174   443 \r\n15272  Socket Threa 4  10.0.2.15        54.200.62.216    443 \r\n15272  Socket Threa 4  10.0.2.15        54.200.62.216    443 \r\n15272  Socket Threa 4  10.0.2.15        117.18.237.29    80  \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.142   80  \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.131   80  \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.131   443 \r\n15272  Socket Threa 4  10.0.2.15        52.222.135.52    443 \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.131   443 \r\n15272  Socket Threa 4  10.0.2.15        54.200.62.216    443 \r\n15272  Socket Threa 4  10.0.2.15        54.200.62.216    443 \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.132   443 \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.131   443 \r\n15272  Socket Threa 4  10.0.2.15        216.58.199.142   443 \r\n15272  Socket Threa 4  10.0.2.15        54.69.17.198     443 \r\n15272  Socket Threa 4  10.0.2.15        54.69.17.198     443 \r\n...\r\n<\/pre>\n
    All the tools above can also be used with various options, to enable the help page for a given tool, make use of the\u00a0-h<\/code>\u00a0option, for example:<\/p>\n
    $ sudo .\/tcpconnect -h<\/strong>\r\n\r\nusage: tcpconnect [-h] [-t] [-p PID] [-P PORT]\r\n\r\nTrace TCP connects\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  -t, --timestamp       include timestamp on output\r\n  -p PID, --pid PID     trace this PID only\r\n  -P PORT, --port PORT  comma-separated list of destination ports to trace.\r\n\r\nexamples:\r\n    .\/tcpconnect           # trace all TCP connect()s\r\n    .\/tcpconnect -t        # include timestamps\r\n    .\/tcpconnect -p 181    # only trace PID 181\r\n    .\/tcpconnect -P 80     # only trace port 80\r\n    .\/tcpconnect -P 80,81  # only trace port 80 and 81\r\n<\/pre>\n
    Trace Failed exec()s Syscalls<\/h4>\n
    To trace failed exec()s syscalls, employ the\u00a0-x<\/code>\u00a0option with opensnoop as below:<\/p>\n
    $ sudo .\/opensnoop -x<\/strong>\r\n\r\nPID    COMM               FD ERR PATH\r\n15414  pool               -1   2 \/home\/.hidden\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/cpu\/system.slice\/systemd-hostnamed.service\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/cpu\/system.slice\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/cpuacct\/system.slice\/systemd-hostnamed.service\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/cpuacct\/system.slice\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/blkio\/system.slice\/systemd-hostnamed.service\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/blkio\/system.slice\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/memory\/system.slice\/systemd-hostnamed.service\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/memory\/system.slice\/cgroup.procs\r\n15415  (ostnamed)         -1   2 \/sys\/fs\/cgroup\/pids\/system.slice\/systemd-hostnamed.service\/cgroup.procs\r\n2548   compiz             -1   2 \r\n15416  systemd-cgroups    -1   2 \/run\/systemd\/container\r\n15416  systemd-cgroups    -1   2 \/sys\/fs\/kdbus\/0-system\/bus\r\n15415  systemd-hostnam    -1   2 \/run\/systemd\/container\r\n15415  systemd-hostnam    -1  13 \/proc\/1\/environ\r\n15415  systemd-hostnam    -1   2 \/sys\/fs\/kdbus\/0-system\/bus\r\n1695   dbus-daemon        -1   2 \/run\/systemd\/users\/0\r\n15415  systemd-hostnam    -1   2 \/etc\/machine-info\r\n15414  pool               -1   2 \/home\/tecmint\/.hidden\r\n15414  pool               -1   2 \/home\/tecmint\/Binary\/.hidden\r\n2599   dconf-service      -1   2 \/run\/user\/1000\/dconf\/user\r\n...\r\n<\/pre>\n
    Trace Particular Process Functions<\/h4>\n
    The last example below demonstrates how to execute a custom trace operation. We are tracing a particular process using its PID.<\/p>\n
    Suggested Read:<\/b>\u00a0Netdata \u2013 A Real-Time Performance Monitoring Tool for Linux<\/a><\/p>\n
    First determine the process ID:<\/p>\n
    $ pidof firefox<\/strong>\r\n\r\n15437\r\n<\/pre>\n
    Later on, run the custom\u00a0trace<\/strong>\u00a0command. In the command below:\u00a0-p<\/code>\u00a0specifies the process ID,\u00a0do_sys_open()<\/code>\u00a0is a kernel function that is traced dynamically including its second argument as a string.<\/p>\n
    $ sudo .\/trace -p 4095 'do_sys_open \"%s\", arg2'<\/strong>\r\n\r\nTIME     PID    COMM         FUNC             -\r\n12:17:14 15437  firefox      do_sys_open      \/run\/user\/1000\/dconf\/user\r\n12:17:14 15437  firefox      do_sys_open      \/home\/tecmint\/.config\/dconf\/user\r\n12:18:07 15437  firefox      do_sys_open      \/run\/user\/1000\/dconf\/user\r\n12:18:07 15437  firefox      do_sys_open      \/home\/tecmint\/.config\/dconf\/user\r\n12:18:13 15437  firefox      do_sys_open      \/sys\/devices\/system\/cpu\/present\r\n12:18:13 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:13 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:14 15437  firefox      do_sys_open      \/usr\/share\/fonts\/truetype\/liberation\/LiberationSans-Italic.ttf\r\n12:18:14 15437  firefox      do_sys_open      \/usr\/share\/fonts\/truetype\/liberation\/LiberationSans-Italic.ttf\r\n12:18:14 15437  firefox      do_sys_open      \/usr\/share\/fonts\/truetype\/liberation\/LiberationSans-Italic.ttf\r\n12:18:14 15437  firefox      do_sys_open      \/sys\/devices\/system\/cpu\/present\r\n12:18:14 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:14 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:14 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:14 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:15 15437  firefox      do_sys_open      \/sys\/devices\/system\/cpu\/present\r\n12:18:15 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:15 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:15 15437  firefox      do_sys_open      \/sys\/devices\/system\/cpu\/present\r\n12:18:15 15437  firefox      do_sys_open      \/dev\/urandom\r\n12:18:15 15437  firefox      do_sys_open      \/dev\/urandom\r\n....\r\n<\/pre>\n
    Summary<\/h3>\n
    BCC<\/strong>\u00a0is a powerful and easy-to-use toolkit for various System administration tasks such as tracing system performance monitoring, tracing block device I\/O, TCP functions, file system operations, syscalls, Node.js probes, plus lots more. Importantly, it ships in with several example files and man pages for the tools to guide you, making it user friendly and reliable.<\/p>\n
    Last but not least, you can get back to us by sharing your thoughts about the subject, ask questions, make useful suggestions or any constructive feedback via the comment section below.<\/p>\n
    Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
    BCC\u00a0(BPF Compiler Collection) is a powerful set of appropriate tools and example files for creating resourceful kernel tracing and manipulation programs. It utilizes extended\u00a0BPF\u00a0(Berkeley Packet Filters), initially known as\u00a0eBPF\u00a0which was one of the new features in\u00a0Linux 3.15. BCC\/BPF \u2013 Dynamic Tracing Tools for Linux Performance Monitoring Practically, most of the components used by\u00a0BCC\u00a0require\u00a0Linux 4.1\u00a0or above, … <\/p>\n
    Continue reading “BCC \u2013 Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11528","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/11528","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=11528"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/11528\/revisions"}],"predecessor-version":[{"id":11529,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/11528\/revisions\/11529"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=11528"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=11528"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=11528"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}