Splitting in very big files

Please ask questions here if you are not familiar with fsarchiver
Post Reply
lucabu
Posts: 1
Joined: Tue Nov 17, 2009 12:33 pm

Splitting in very big files

Post by lucabu » Tue Nov 17, 2009 12:51 pm

Hello,

I need to split archive in very big sub-archives, since i have a backup device to store them that holds about 32 Gb.

I'm using a command-line like that:

fsarchiver -vv -o -z 8 -j 4 -s 31750 savedir /home/path-to-backup/backup.fsa /home/dir-to-save/

I want to overvrite old backup (-o), to give a good compression (-z 8), to use all my four cores (-j 4).

That command should also split backup in several archives which size should be 31750Mb, but It does not work: I get a single file of about 55Gb, which is useless for me, since I cannot copy it easily in my device.

Any clue about that?

P.S. Testing that command line with small split size (e.g. 30Mb) works as expected...

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: Splitting in very big files

Post by admin » Sat Nov 21, 2009 9:04 pm

Can you please run that again using a patched version of fsarchiver-0.6.1:
http://pastebin.com/f6ed089bf

You have to save that to a file (eg: fsarchiver.patch), and then:

Code: Select all

tar xf fsarchiver-0.6.1.tar.gz
cd fsarchiver-0.6.1
cat /path/to/fsarchiver.patch | patch -p1
./configure && make
And then you can run fsarchiver in debug mode (-dddd), it will produce logs in /var/log, it should tell us why the splitting fails.

Do you know what's the threshold for that size of the volume to start failing ?
You can try with "-s 15000" for instance and see what's the biggest size for which splitting works.

It works for me with that command:

Code: Select all

./fsarchiver -vvvv -o -j4 -s 31750 savedir /mnt/backup2/test.fsa /data

Code: Select all

thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=33290859373, g_options.splitsize=33292288000, cursize+wb->size=33291121230, cursize=33290859373, wb->size=261857
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=33291121230, g_options.splitsize=33292288000, cursize+wb->size=33291382769, cursize=33291121230, wb->size=261539
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=33291382769, g_options.splitsize=33292288000, cursize+wb->size=33291644374, cursize=33291382769, wb->size=261605
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=33291644374, g_options.splitsize=33292288000, cursize+wb->size=33291906190, cursize=33291644374, wb->size=261816
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=33291906190, g_options.splitsize=33292288000, cursize+wb->size=33292167846, cursize=33291906190, wb->size=261656
thread_archio.c#363,thread_writer_fct(): splitchk: YES --> cursize=33292167846, g_options.splitsize=33292288000, cursize+wb->size=33292429595, cursize=33292167846, wb->size=261749
Creating new volume: [/mnt/backup2/test.f01]
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=261816, g_options.splitsize=33292288000, cursize+wb->size=523614, cursize=261816, wb->size=261798
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=523614, g_options.splitsize=33292288000, cursize+wb->size=785484, cursize=523614, wb->size=261870
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=785484, g_options.splitsize=33292288000, cursize+wb->size=1047236, cursize=785484, wb->size=261752
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=1047236, g_options.splitsize=33292288000, cursize+wb->size=1309168, cursize=1047236, wb->size=261932
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=1309168, g_options.splitsize=33292288000, cursize+wb->size=1570626, cursize=1309168, wb->size=261458
thread_archio.c#383,thread_writer_fct(): splitchk: NO --> cursize=1570626, g_options.splitsize=33292288000, cursize+wb->size=1832431, cursize=1570626, wb->size=261805

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: Splitting in very big files

Post by admin » Mon Dec 21, 2009 4:02 pm

I think I understand what the problem is. It only affects fsachiver compiled in 32bit since "long" integers are only 32bit, and then there is an integer overflow with bigger numbers.

This bug has been fixed in fsarchiver-0.6.3_beta10 (and more recent), and I have patched fsarchiver-0.6.2 in SystemRescueCd-1.3.4-rc1 (which also contains fsarchiver-0.6.3_beta10).

Post Reply