FSArchiver 0.4.5 compression problem

shantanu_gadgil · Post by **shantanu_gadgil** » Sat Mar 21, 2009 4:50 am

Hi,
I am using 1.1.7 beta 6 (six) and FSArchiver 0.4.5 (zero four five) ...

When I run the following command ...

[email protected] % fsarchiver -z 9 savefs f10server.fsa  /dev/sda2

...it shows the following error continuously ...

Code: Select all

comp_lzma.c#0042,compress_block_lzma(): lzma_easy_encoder(6, LZMA_CHECK_CRC32) failed with res=5
comp_lzma.c#0042,compress_block_lzma(): lzma_easy_encoder(6, LZMA_CHECK_CRC32) failed with res=5
comp_lzma.c#0042,compress_block_lzma(): lzma_easy_encoder(6, LZMA_CHECK_CRC32) failed with res=5
comp_lzma.c#0042,compress_block_lzma(): lzma_easy_encoder(6, LZMA_CHECK_CRC32) failed with res=5

and the file size ends up nearly equal to the occupied size of the partition ....
1.6 GB data size and compressed size 1.3 GB

Code: Select all

[email protected] % fsarchiver archinfo f10server.fsa 
====================== archive information ======================
Archive type: 			filesystems
Filesystems count: 		1
Archive id: 			49cee279
Archive file format: 		FsArCh_00Y
Archive created with: 		0.4.5
Archive creation date: 		20090319-21:57:17
Archive label: 			archive-label
Compression level: 		9 (lzma level 9)
Encryption algorithm: 		none

===================== filesystem information ====================
Filesystem id in archive: 	0
Filesystem format: 		ext3
Filesystem label: 		/
Filesystem uuid: 		7288a317-7196-4527-ba10-8ab9fca78d59
Original filesystem size: 	30.55 GB (32805474304 bytes)
Space used in filesystem: 	1.60 GB (1715044352 bytes)

[email protected] % ls -ldh f10server.fsa
-rw-r--r-- 1 root root 1.3G 2009-03-19 16:37 f10server.fsa

but ... when I omit any compression ...

$> fsarchiver savefs f10serverb.fsa /dev/sda2

... I end up with a much smaller size ...

Code: Select all

[email protected] % fsarchiver archinfo f10serverb.fsa 
====================== archive information ======================
Archive type: 			filesystems
Filesystems count: 		1
Archive id: 			49c42acb
Archive file format: 		FsArCh_00Y
Archive created with: 		0.4.5
Archive creation date: 		20090320-12:18:33
Archive label: 			archive-label
Compression level: 		3 (gzip level 6)
Encryption algorithm: 		none

===================== filesystem information ====================
Filesystem id in archive: 	0
Filesystem format: 		ext3
Filesystem label: 		/
Filesystem uuid: 		7288a317-7196-4527-ba10-8ab9fca78d59
Original filesystem size: 	30.55 GB (32805474304 bytes)
Space used in filesystem: 	1.60 GB (1715044352 bytes)

[email protected] % ls -ldh f10serverb.fsa 
-rw-r--r-- 1 root root 453M 2009-03-20 06:54 f10serverb.fsa

Regards,
Shantanu

Post by **admin** » Sat Mar 21, 2009 9:09 am

Thanks for reporting that problem. It does not seems to be a bug in the program anyway:

When lzma_easy_encoder() returns 5 it means it's an "out of memory" error. The lzma compression requires a lot of resources (cpu and memory) and it just fails to allocate the memory. As a consequence, fsarchiver writes an uncompressed version to the archive. So it's not a bug in fsarchiver, but I will change the error message to an explicit "out of memory" error.

I will also try to optimize the memory management, but I don't know if I can really get it to work with less memory.

The default compression level is 3, which is similar to "gzip -6", and it requires less memory to compress, then it works, and the archive file is smaller.

shantanu_gadgil · Post by **shantanu_gadgil** » Sat Mar 21, 2009 2:54 pm

Could the problem be that I am booting with the "lowmem" option ?

The machine has 2 (two) GB RAM ... which I guess should not be less ?!?

Regards,
Shantanu

Post by **admin** » Sat Mar 21, 2009 3:10 pm

No, lowmem is only involved in the SystemRescueCd startup script, for instance it won't cache secondary files in memory when docache is used.

This is only a problem in how fsarchiver and the lzma library works. I think the lzma library (xz-utils) is good, but this algorithm is just resources intensive.

In fsarchiver, it keeps a queue of blocks to compress in memory (32 blocks of up to 900 KiB each by default), so that multi-processor machines can compress multiple blocks in the same time (one compression thread on each core of the cpu).

But it seems you don't use multi-threading, so only one lzma compression function (which requires extra memory) is running at a given time. The queue size is 32 blocks anyway.

You can try "-z 8" which has blocks of 512KiB rather than 900KiB. And it can be quicker, but less efficient.

I used valgrind (debugging and profiling tool for developers) to see where the memory is used, but I cannot see any obvious waste of memory unfortunately. Here is the command I used:

Code: Select all

valgrind --tool=massif fsarchiver <arguments>
ms_print <file-generated-by-valgrind>

You can also use htop (which is part of sysresccd) to see what is in the VIRT column "virtual memory" size of the process (all its threads), ie the whole size used by fsarchiver (data, code, and all the libs it uses). Type M in htop to see the biggest process first.

A normal fsarchiver instance has at least three threads even when no extra compression threads are created (option "-j" not used)

You can use pmap ("pmap -x <fsarchiver-pid>") to see what the breakdown of the VIRT memory is, so which amount of that memory is used to store the code of the library and which amount is really allocated for fsarchiver data.

Post by **admin** » Sat Mar 21, 2009 3:37 pm

You can see that the compression level you use has a huge impact on the memory requirements. What matters is the "writeable/private" memory:

Code: Select all

Tests with one compression thread (-j1)
* z1=20MB
* z2=20MB
* z3=23MB
* z4=27MB
* z5=50MB
* z6=76MB
* z7=27MB
* z8=172MB
* z9=754MB

Code: Select all

# pmap -d $(pgrep fsarchiver)
3799:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z1 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 118584K    writeable/private: 20556K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3772:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z2 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 118268K    writeable/private: 19992K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3755:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z3 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 183936K    writeable/private: 23024K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3737:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z4 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 185548K    writeable/private: 27396K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3676:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z5 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 130852K    writeable/private: 50056K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3411:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z6 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 142556K    writeable/private: 75728K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3552:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z7 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 184688K    writeable/private: 27664K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3523:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z8 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 227232K    writeable/private: 172100K    shared: 0K

------------------------------------------------------------------------------------------

# pmap -d $(pgrep fsarchiver)
3484:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z9 -vv
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 823764K    writeable/private: 754076K    shared: 0K

If you use multi-threading, there will be several compression-threads running in the same time, each one is using some memory. multi-threading will be faster or multi-core processors or systems with more than one cpu in general, but the compression ratio is the same.

Here is an example:

Code: Select all

# pmap -d $(pgrep fsarchiver)
3865:   ./fsarchiver savefs -o /mnt/archives/test02.fsa /dev/vgtest/test01reiserfs -z9 -vv -j2
Address           Kbytes Mode  Offset           Device    Mapping
.........
.........
mapped: 1521460K    writeable/private: 1437780K    shared: 0K

So we can see that the same command with two threads and compression level is z9 is using 1438MB of memory (the same command requires 754MB when it has only one compression thread)

So the biggest part of the memory requirement is the compression threads. The more compression threads you have, the more memory you need. Very high compression levels (especially -z9) requires a huge amount of memory. If you don't have enough memory, use -z8 rather than -z9 and disable the multi-threading if you have time.

More details about compression levels here:
http://www.fsarchiver.org/Compression

shantanu_gadgil · Post by **shantanu_gadgil** » Mon Mar 23, 2009 7:22 pm

Thanks for the detailed explanation .... "-z 8" works nicely!

* z9=754MB

But that would mean that the 2 GB that the machine has is not falling short, right ? I am not using the multi-threaded options either!

Once SystemRescueCD boots what exactly would help to determine beforehand if the '-z 9' option could fail ?!?

Regards,
Shantanu

Post by **admin** » Mon Mar 23, 2009 8:34 pm

I just made a single test with -z9 where the program used 754MB to run. My conclusion is that "-z9" requires four times more memory than "-z8", don't consider that "-z9" will always work with 754MB of memory. It's a very specific test. It depends on what sort of files I compress.

About the multi-threading: the memory requirement will be multiplied by the number of threads, so that's fine if you have enough memory, it's only a problem with high compression ratios if you don't have a large amount of memory.

I think the lzma library (xz-utils) provides functions to estimate the amount of memory required. But it's not my priority at the moment. It will never be possible to tell you exactly if it will work because it depends on the other processes running on your machine.

The best think you can do is to make your own estimation of how much memory you need per compression thread, using the same command (pmap) and then decide what level is the best suited for your own usage.

So I just recommend using "-z8" for people who want a very good compression ratio. I think "-z9" is an extreme level and should only be used on computers with lot of resources when the compression time does not matter.