源码安装zstd、p7zip-zstd及一些简单的性能对比
2018-03-26 18:27:06 
阿炯
-------------------------------
源码安装zstd
在源码安装时没有了'configure'来指定安装路径，只有'make & make install'，默认只会安装到'/usr/local'目录下，不想安装到默认目录该如何处理，搜索了一下：
DESTDIR=/path_to_install
有的修改Makefile 找到里面的安装路径并修改它，Makefile 其实就是把一个个安装过程批处理了。但并不是每款软件的makefile都有指定路径的定义，zstd就是这种，最终在安装时使用了如下指令：
make install PREFIX=/usr
以下对比7z与zstd软件指令在linux下的性能(目录下有各种类型的文件)
7z
compression
real 1m31.700s
user 4m24.027s
sys 0m1.817s
decompression
real 0m20.340s
user 0m19.917s
sys 0m0.340s
zst
zstd -19 -v -z -r --rm mydocs
compression
real 1m42.989s
user 1m42.283s
sys 0m0.473s
decompression
time zstd -d -v -r --rm mydocs
real 0m0.833s
user 0m0.510s
sys 0m0.283s
纯文本文件：cp.txt，大小2.0GB(2015 MiB)。默认的压缩级别下压缩后的大小
356M：cp.7z
699M：cp.zst
time 7za a -t7z cp.7z cp.txt
real 5m44.650s
user 15m58.237s
sys 0m4.180s
time perl comp.zstd.pl
real 0m11.486s
user 0m10.967s
sys 0m0.503s
解压缩用时：
time zstd -d -v -o /opt/cp.st cp.zst
real 0m4.630s
user 0m3.907s
sys 0m0.717s
time 7za x cp.7z -o/opt/
real 0m25.609s
user 0m24.207s
sys 0m0.703s
-------------------------------
源码安装p7zip-zstd
Linux上的支持zstd的7z软件可以参看p7zip with Zstandard(不过版本稍旧此)。
p7zip-zstd/p7zip_16.02
编辑 makefile.common 来定义安装的路径，默认为'/usr/local'，执行编译安装：
make all3 all4
make install
./install.sh /usr/bin /usr/lib/p7zip /usr/man /usr/share/doc/p7zip
- installing /usr/bin/7za
- installing /usr/bin/7zr
- installing /usr/bin/7z
- installing /usr/lib/p7zip/7zCon.sfx
- installing /usr/lib/p7zip/7z.so
- installing /usr/lib/p7zip/Codecs
- installing /usr/man/man1/7z.1
- installing /usr/man/man1/7za.1
- installing /usr/man/man1/7zr.1
- installing /usr/share/doc/p7zip/README
- installing /usr/share/doc/p7zip/ChangeLog
- installing HTML help in /usr/share/doc/p7zip/DOC
在'install.sh'文件中，修改这里：
# global install
DEST_HOME=/usr/local
并不能起到作用！
time /usr/local/bin/7za a -mx=9 freeoa.sdoc.7z freeoa.sdoc/
Files read from disk: 644
Archive size: 765223508 bytes (730 MiB)
real 10m13.706s
user 14m58.563s
sys 1m23.643s
time /usr/local/bin/7za a -m0=zstd -mx=12 freeoa.sys.doc.7z freeoa.sys.doc/
Scanning the drive:
38 folders, 644 files, 1087816074 bytes (1038 MiB)
Files read from disk: 644
Archive size: 813051714 bytes (776 MiB)
real 1m10.442s
user 2m16.237s
sys 0m1.797s
time /usr/local/bin/7za a -m0=zstd -mx=19 freeoa.sys.doc.7z freeoa.sys.doc/
Files read from disk: 644
Archive size: 797668903 bytes (761 MiB)
real 5m34.741s
user 11m1.027s
sys 0m3.027s
time /usr/local/bin/7za x freeoa.sys.doc.7z
Folders: 38
Files: 644
Size: 1087816074
Compressed: 813051714
real 0m10.116s
user 0m3.600s
sys 0m2.150s
time /usr/local/bin/7za x freeoa.sdoc.7z
Size: 1087816074
Compressed: 765223508
real 0m48.399s
user 0m45.970s
sys 0m2.303s
官方文档中的一些调用方法：
7z a archiv.7z -m0=zstd -mx0 Zstandard Fastest Mode, without BCJ preprocessor
7z a archiv.7z -m0=zstd -mx1 Zstandard Fast mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=zstd -mx.. ...
7z a archiv.7z -m0=zstd -mx21 Zstandard 2nd Slowest Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=zstd -mx22 Zstandard Ultra Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=lz4 -mx0 LZ4 Fastest Mode, without BCJ preprocessor
7z a archiv.7z -m0=lz4 -mx1 LZ4 Fast mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=lz4 -mx.. ...
7z a archiv.7z -m0=lz4 -mx12 LZ4 Ultra Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=lz5 -mx0 LZ5 Version 1.5 Fastest Mode, without BCJ preprocessor
7z a archiv.7z -m0=lz5 -mx1 LZ5 Version 1.5 Fast mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=lz5 -mx.. ...
7z a archiv.7z -m0=lz5 -mx16 LZ5 Version 1.5 Ultra Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=bcj -m1=zstd -mx1 Fast mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=bcj -m1=zstd -mx.. ...
7z a archiv.7z -m0=bcj -m1=zstd -mx21 2nd Slowest Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=bcj -m1=zstd -mx22 Ultra Mode, with BCJ preprocessor on executables
7z a archiv.7z -m0=bcj -m1=brotli -mxN ...
7z a archiv.7z -m0=bcj -m1=lizard -mxN ...
7z a archiv.7z -m0=bcj -m1=lz4 -mxN ...
7z a archiv.7z -m0=bcj -m1=lz5 -mxN ...
在gcct第10版下编译不通过的情况
2022年1月上旬消息，在gcc-v10编译安装p7zip-zstd.16.02_zstd_1.1.4时报不能通过，而在gcc-v8.3没有这个错误问题，该错误在sf.net上有所提及。
通过检索发现这个链接能解决问题：
A quick & dirty workaround (but with no downside afaik) is to set the environment variable CC_EXE to -w:
export CC_EXE=-w
看到debian11的源中有gcc9，不知道使用第9版的gcc会不会有问题，ubuntu下切换的方式可参考这里。
-------------------------------
perl mod gz(IO::Compress::Gzip) vs zstd(Compress::Zstd)
1016M:cp.txt
大小：
306M:cp.gz
340M:cp.zst
用时：
gz:
real 1m29.788s
user 1m29.287s
sys 0m0.340
zst:
real 0m5.169s
user 0m4.927s
sys 0m0.233s
同目录下用指令解压
zst:
real 0m2.231s
user 0m1.893s
sys 0m0.337s
gz:
real 0m7.287s
user 0m6.943s
sys 0m0.330s
-------------------------------
与gz对比
文件压缩
$ time gzip -7 -v -c > /home/hto/yqt.sql.gz yqt.sql
real 3m40.463s
user 3m37.940s
sys 0m2.060s
$ time /usr/bin/zstd -12 -v -o /home/hto/yqt.sql.zst yqt.sql
real 4m26.597s
user 4m23.953s
sys 0m2.113s
1.4G:yqt.sql.gz
747M:yqt.sql.zst
文件解压缩
$ time /usr/bin/zstd -d -v yqt.sql.zst -c > /opt/yqt.sql
real 0m15.468s
user 0m7.830s
sys 0m1.927s
$ time /bin/gzip -v -d -c yqt.sql.gz > /opt/yqt.sql
real 0m40.582s
user 0m38.157s
sys 0m2.150s
再来看看对较大文件的处理
35G:/opt/20160520
$ time /usr/bin/zstd -19 -v -o /home/hto/20160520.zst /opt/20160520
/opt/20160520 : 14.36% (37241429093 => 5348277554 bytes, /home/hto/20160520.zst)
real 226m11.672s
user 225m37.249s
sys 0m21.990s
5.0G:/home/hto/20160520.zst
$ time /usr/bin/zstd -d -v -o /mnt/pd/20160520 20160520.zst
20160520.zst : 37241429093 bytes
real 3m12.522s
user 1m1.777s
sys 0m12.060s
解压缩真的很赞。
与7z相比，在压缩比上还有一些差距，下面来看看压缩与解压缩的情况：
25G:web.data.csv
$ time /usr/bin/zstd -19 -o /home/hto/web.data.csv.zst /opt/web.data.csv
/opt/web.data.csv : 19.51% (26546774686 => 5179359656 bytes, /home/hto/web.data.csv.zst)
real 156m0.925s
user 155m36.759s
sys 0m15.720s
$ time /usr/bin/7za a -mx=9 /home/hto/web.data.csv.7z /opt/web.data.csv
Scanning the drive:
1 file, 26546774686 bytes (25 GiB)
Creating archive: /home/hto/web.data.csv.7z
Archive size: 4297100331 bytes (4099 MiB)
Everything is Ok
real 110m25.149s
user 333m1.635s
sys 1m23.907s
4.1G:web.data.csv.7z
4.9G:web.data.csv.zst
解压缩
$ time /usr/bin/zstd -d web.data.csv.zst -d -v -o /opt/web.data.csv web.data.csv.zst
web.data.csv.zst : 26546774686 bytes
real 2m45.849s
user 1m46.997s
sys 0m18.487s
$ time /usr/bin/7za x -o/opt web.data.csv.7z
Scanning the drive for archives:
1 file, 4297100331 bytes (4099 MiB)
Extracting archive: web.data.csv.7z
--
Path = web.data.csv.7z
Type = 7z
Physical Size = 4297100331
Headers Size = 130
Method = LZMA2:26
Solid = -
Blocks = 1
Everything is Ok
Size: 26546774686
Compressed: 4297100331
real 6m3.239s
user 5m13.220s
sys 0m10.290s