{"id":13219,"date":"2019-04-01T08:15:28","date_gmt":"2019-04-01T08:15:28","guid":{"rendered":"http:\/\/www.appservgrid.com\/paw92\/?p=13219"},"modified":"2019-04-01T08:15:28","modified_gmt":"2019-04-01T08:15:28","slug":"zstd-a-fast-data-compression-algorithm-used-by-facebook","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2019\/04\/01\/zstd-a-fast-data-compression-algorithm-used-by-facebook\/","title":{"rendered":"zstd \u2013 A Fast Data Compression Algorithm Used By Facebook"},"content":{"rendered":"

Zstandard<\/strong>\u00a0(also known as\u00a0zstd<\/strong>) is a free open source, fast real-time data compression program with better compression ratios, developed by\u00a0Facebook<\/strong>. It is a\u00a0lossless compression algorithm<\/a>\u00a0written in\u00a0C<\/strong>\u00a0(there is a re-implementation in\u00a0Java<\/strong>) \u2013 its thus a native Linux program.<\/p>\n

Read Also<\/strong>:\u00a010 7zip (Data Comperssion) Command Examples in Linux<\/a><\/p>\n

When required, it can trade compression speed for stronger compression ratios (compression speed vs compression ratio trade-off can be configured by small increments), vice versa. It has a special mode for small data compression, known as dictionary compression, and can build dictionaries from any sample set provided. It comes with a command line utility for creating and decoding\u00a0.zst<\/strong>,\u00a0.gz<\/strong>,\u00a0.xz<\/strong>\u00a0and\u00a0.lz4<\/strong>\u00a0files.<\/p>\n

Importantly,\u00a0Zstandard<\/strong>\u00a0has a rich collection of APIs, supports almost all popular programming languages including Python, Java, JavaScript, Nodejs, Perl, Ruby, C#, Go, Rust, PHP, Switft, and lots more.<\/p>\n

It is actively used to compress large volumes of data in multiple formats and use cases in\u00a0Facebook<\/strong>; services such as\u00a0Amazon Redshift<\/strong>\u00a0data warehousing; databases such as Hadoop and Redis; the Tor network and many other applications including games.<\/p>\n

The following results are obtained by doing several fast compression algorithms tests on a server running Linux Debian using\u00a0lzbench<\/a>, an open-source in-memory benchmark tool.<\/p>\n

\n

\"Zstandard<\/a><\/p>\n

Zstandard Compression Testing<\/p>\n<\/div>\n

How to Install Zstandard Compression Tool in Linux<\/h3>\n

To install\u00a0Zstandard<\/strong>\u00a0on a Linux distribution, you need to compile it from sources, but before that first you need to install the necessary development tools on your system using your distribution package manager as shown.<\/p>\n

$ sudo apt update && sudo apt install build-essential\t\t#Ubuntu\/Debian\r\n# yum group install \"Development Tools\" \t\t\t#CentOS\/REHL\r\n# dnf groupinstall \"C Development Tools and Libraries\"\t\t#Fedora 22+\r\n<\/pre>\n
Once all the needed development tools installed, now you can download the source package, move into the local repo directory, build the binary and install it as shown.<\/p>\n
$ cd ~\/Downloads\r\n$ git clone https:\/\/github.com\/facebook\/zstd.git\r\n$ cd zstd\r\n$ make\r\n$ sudo make install \r\n<\/pre>\n
Once\u00a0Zstandard<\/strong>\u00a0installed, now we can move further to learn some basic usage of\u00a0Zstd<\/strong>\u00a0command examples in the following section.<\/p>\n
Learn 10 Zstd Command Usage Examples in Linux<\/h3>\n
Zstd\u2019s command line syntax is generally similar to that of\u00a0gzip<\/strong>\u00a0and\u00a0xz<\/strong>\u00a0tools, with a few differences.<\/p>\n
1.<\/strong>\u00a0To create a\u00a0.zst<\/code>\u00a0compression file, simply provide a filename to compress it or use the\u00a0-z<\/code>\u00a0flag also means compress, which is the default action.<\/p>\n
$ zstd etcher-1.3.1-x86_64.AppImage \r\nOR\r\n$ zstd -z etcher-1.3.1-x86_64.AppImage \r\n<\/pre>\n
2.<\/strong>\u00a0To decompress a\u00a0.zst<\/code>\u00a0compression file, use the\u00a0-d<\/code>\u00a0flag or the\u00a0unzstd<\/strong>\u00a0utility as shown.<\/p>\n
$ zstd -d etcher-1.3.1-x86_64.AppImage.zst \r\nOR\r\n$ unzstd etcher-1.3.1-x86_64.AppImage.zst \r\n<\/pre>\n
3.<\/strong>\u00a0To remove source file after an operation, by default, the source file is not deleted after successful compression or decompression, to delete it, use the\u00a0--rm<\/code>\u00a0option.<\/p>\n
$ ls etcher-1.3.1-x86_64.AppImage\r\n$ zstd --rm  etcher-1.3.1-x86_64.AppImage\r\n$ ls etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
4.<\/strong>\u00a0To set a compression level, zstd has a number of operation modifiers, for instance you can specify a compression level as\u00a0-6<\/code>(a number 1-19, default is 3) as shown.<\/p>\n
$ zstd -6 --rm etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
5.<\/strong>\u00a0To set a compression speed, zstd has a compression speed ratio\u00a01-10<\/strong>, the default compression speed is\u00a01<\/strong>. You can trade compression ratio for compression speed with the\u00a0--fast<\/code>\u00a0option, the higher the number the faster the compression speed.<\/p>\n
$ zstd --fast=10 etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
6.<\/strong>\u00a0To display information about a compressed file, use the\u00a0-l<\/code>\u00a0flag, which is used to display information about a compressed file, for example.<\/p>\n
$ zstd -l etcher-1.3.1-x86_64.AppImage.zst\r\n<\/pre>\n
7.<\/strong>\u00a0To test the integrity of a compressed files, use the\u00a0-t<\/code>\u00a0flag as shown.<\/p>\n
$ zstd -t etcher-1.3.1-x86_64.AppImage.zst\r\n<\/pre>\n
8.<\/strong>\u00a0To enable verbose mode, use the\u00a0-v<\/code>\u00a0option.<\/p>\n
$ zstd -v -5 etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
9.<\/strong>\u00a0To use other file compression or decompression formats such as gzip, xz, lzma, and lz4, using the\u00a0--format=FORMAT<\/code>\u00a0as shown.<\/p>\n
$ zstd -v --format=gzip etcher-1.3.1-x86_64.AppImage\r\n$ zstd -v --format=xz  etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
10.<\/strong>\u00a0To set a zstd process priority to real-time, you can use the option\u00a0\u2013priority=rt<\/strong>\u00a0as shown.<\/p>\n
$zstd --priority=rt etcher-1.3.1-x86_64.AppImage\r\n<\/pre>\n
The\u00a0-r<\/code>\u00a0flag instructs zstd to operate recursively on dictionaries. You can find lots of useful and advanced options, how to read or create dictionaries by consulting the zstd man page.<\/p>\n
$ man zstd\r\n<\/pre>\n
Zstandard Github Repository<\/strong>:\u00a0https:\/\/github.com\/facebook\/zstd<\/a><\/p>\n
Zstandard<\/strong>\u00a0is a fast real-time, lossless data compression algorithm and compression tool which offers high compression ratios. Try it out and share your thoughts about it or ask questions via the feedback form below.<\/p>\n
Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Zstandard\u00a0(also known as\u00a0zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by\u00a0Facebook. It is a\u00a0lossless compression algorithm\u00a0written in\u00a0C\u00a0(there is a re-implementation in\u00a0Java) \u2013 its thus a native Linux program. Read Also:\u00a010 7zip (Data Comperssion) Command Examples in Linux When required, it can trade compression speed for stronger compression … <\/p>\n
Continue reading “zstd \u2013 A Fast Data Compression Algorithm Used By Facebook”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-13219","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13219","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=13219"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13219\/revisions"}],"predecessor-version":[{"id":13220,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/13219\/revisions\/13220"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=13219"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=13219"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=13219"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}