zstd
Original author(s) | Yann Collet |
---|---|
Developer(s) | Yann Collet, Nick Terrell, Przemysław Skibiński[1] |
Initial release | 23 January 2015 |
Stable release | 1.5.6[2]
/ 27 March 2024 |
Repository | |
Written in | Cross-platform |
Platform | Portable |
Type | Data compression |
License | BSD-3-Clause or GPL-2.0-or-later (dual-licensed) |
Website | facebook |
Zstandard is a lossless data compression algorithm developed by Yann Collet at Facebook. Zstd is the corresponding reference implementation in C, released as open-source software on 31 August 2016.[3][4]
Features
Zstandard was designed to give a
Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication (--long
, 128 MiB window) similar to
Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.[7] Zstandard command-line has an "adaptive" (--adapt
) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.
Zstd at its maximum compression level gives a compression ratio close to lzma,
Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.
Design
Zstandard combines a dictionary-matching stage (
Usage
Filename extension |
.zst[15] |
---|---|
Internet media type |
application/zstd RFC 8878 |
Website | github |
RFC 8878 | |
---|---|
Website | github |
The
In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel,[19] and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept OpenZFS compression method[7] which was integrated in 2020.[20]
The
In March 2018,
In 2018 the algorithm was published as
zstd -c -T0 --ultra -20 -
, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used.[27][28][29] Arch Linux later also switched to zstd as default compression algorithm for mkinitcpio initial ramdisk generator.[30]Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ[34] file formats developed by the homebrew community for the Nintendo Switch hybrid game console.[35] Similarly, it is also one of many supported compression algorithms in the .RVZ Wii and GameCube disc image file format.
On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]
In March 2024,
License
The reference implementation is licensed under the BSD license, published at GitHub.[40] Since version 1.0, it had an additional Grant of Patent Rights.[41]
From version 1.3.1,[42] this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.[43]
See also
- LZ4 (compression algorithm) – a fast member of the LZ77 family
- LZFSE – a similar algorithm by Apple used since iOS 9 and OS X 10.11 and made open source on 1 June 2016
- Zlib
References
- ^ "Contributors to facebook/zstd". github.com. Archived from the original on 27 January 2021. Retrieved 26 January 2021.
- ^ "Release Zstandard v1.5.6 - Chrome Edition · facebook/zstd". Retrieved 27 March 2024.
- ^ Sergio De Simone (2 September 2016). "Facebook Open-Sources New Compression Algorithm Outperforming Zlib". InfoQ. Retrieved 20 April 2019.
- ^ "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 31 August 2016. Retrieved 6 September 2016.
- ^ "Release Zstandard v1.3.4 - faster everything · facebook/zstd". GitHub. Retrieved 27 March 2024.
- ^ "Command Line Interface for Zstandard library". GitHub. 28 October 2021.
- ^ a b "ZStandard in ZFS" (PDF). open-zfs.org. 2017. Retrieved 20 April 2019.
- ^ Matt Mahoney. "Silesia Open Source Compression Benchmark". Retrieved 10 May 2019.
- ^ Matt Mahoney (29 August 2016). "Large Text Compression Benchmark, .2157 zstd". Retrieved 1 September 2016.
- ^ TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
- ^ Matt Mahoney, Silesia Open Source Compression Benchmark
- ^ "Facebook developers report massive speedups and compression ratio improvements when using dictionaries" (PDF). Fermilab. 11 October 2017. Retrieved 27 March 2024.
- ^ "Smaller and faster data compression with Zstandard". Facebook. 31 August 2016.
- ^ "facebook/zstd". GitHub. 28 October 2021.
- ^ a b c d Collet, Yann (February 2021). . Retrieved 26 February 2023.
- lwn.net. Retrieved 27 March 2024.
- ^ "Linux_4.14 - Linux Kernel Newbies". Kernelnewbies.org. 30 December 2017. Retrieved 16 August 2018.
- ^ Larabel, Michael (8 September 2017). "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix". www.phoronix.com.
- ^ "Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165". GitHub.
- ^ "Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f". GitHub.
- ^ "Zstandard Encoding - Amazon Redshift". 20 April 2019.
- ^ a b Larabel, Michael (12 March 2018). "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix.com. Phoronix Media. Retrieved 29 October 2019.
The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.
- ^ "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. 12 March 2018. Retrieved 13 August 2018.
- ^ "Debian Changelog for apt". Debian. 19 April 2021. Retrieved 7 November 2022.
- ^ Collet, Yann (October 2018). . Retrieved 7 October 2020.
- ^ Larabel, Michael (16 October 2019). "Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs". Phoronix.
- ^ Broda, Mara (4 January 2020). "Now using Zstandard instead of xz for package compression". Arch Linux. Retrieved 5 January 2020.
- ^ Broda, Mara (25 March 2019). "RFC: (devtools) Changing default compression method to zstd". arch-dev-public (Mailing list).
- ^ Broda, Mara; Polyak, Levente (27 December 2019). "makepkg.conf: change default compression method to zstd". GitHub.
- ^ Razzolini, Giancarlo (19 February 2021). "News: Moving to Zstandard images by default on mkinitcpio". Arch Linux. Retrieved 28 December 2021.
- ^ "Changes/Switch RPMS to ZSTD compression". Fedora Project Wiki.
- ^ "Fedora Workstation 34 feature focus: Btrfs transparent compression". Fedora Magazine. 14 April 2021. Retrieved 12 May 2022.
- ^ "Changes/BtrfsTransparentCompression". Fedora Project Wiki. Retrieved 12 May 2022.
- ^ "RELEASE - nsZip - NSP compressor/decompressor to reduce storage". GBAtemp.net - The Independent Video Game Community. 20 October 2019. Retrieved 3 November 2019.
- ^ Bosshard, Nico (31 October 2019), nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip, retrieved 3 November 2019
- ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8, 15 June 2020, retrieved 7 July 2020
- ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7, 1 June 2020, retrieved 6 June 2020
- ^ "New in Chrome 123 | Chrome Blog". Chrome for Developers. 19 March 2024. Retrieved 16 April 2024.
- ^ "1301878 - (zstd) Implement support for Zstandard (zstd)". bugzilla.mozilla.org. 10 September 2016. Retrieved 16 April 2024.
- ^ "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. 31 August 2016. Retrieved 1 September 2016.
- ^ "zstd/PATENTS at v1.3.0 · facebook/zstd". GitHub. Retrieved 27 March 2024.
- ^ "Release Zstandard v1.3.1 · facebook/zstd". GitHub. Retrieved 27 March 2024.
- ^ "New license by Cyan4973 · Pull Request #801 · facebook/zstd". GitHub. Retrieved 27 March 2024.
External links
- Official website
- zstd on GitHub
- 7zip with Zstandard on GitHub
- "Smaller and faster data compression with Zstandard", Yann Collet and Chip Turner, 31 August 2016, Facebook Announcement
- The Guardian is using ZStandard instead of zlib