Page 1 of 1

Restore filesystem with small block size requirements

Posted: Wed Dec 18, 2013 5:39 pm
by stephen.labar
I am using RHEL6.2. The version of e2fsprogs installed is 1.41.12-11.el6.i686. I created an ext3 filesystem using mkfs -t ext3 -b 2048 -i 2048 to support a large volume of small files. On my 50GB partition, this created 26.2 million inodes. I saved this filesystem using fsarchiver 0.6.12. When I restore the filesystem, I do not get the same inode count that is created from manual mkfs. The inode count after fsarchive restore is only 3.2 million. I compared a dumpe2fs after mkfs and after fsarchive restore. Besides the inode count, I noticed the "inode blocks per group" were different. The mkfs command created 2048 inode blocks per group. The fsarchive restore created 256 inode blocks per group.

I did the same test using fsarchiver 0.6.17 with the same results. I also updated e2fsprogs to 1.41.12-18.el6.i686 with the same results.

Any help would be appreciated.

Re: Restore filesystem with small block size requirements

Posted: Wed Dec 18, 2013 5:58 pm
by Lazy_Kent
I think, fsarchiver doesn't restore your filesystem state. It is a file archiver.

> When I restore the filesystem

Better say, you restore the files. The filesystem is created "from scratch" with default settings.

Re: Restore filesystem with small block size requirements

Posted: Wed Dec 18, 2013 9:43 pm
by stephen.labar
fsarchiver does format the filesystem. It is a filesystem archiver. I have been using this product for years and have been able to use fsarchiver to restore a raw partition. It looks like it calls mkfs internally. I don't see any options to control how mkfs is called. I always thought it collected the filesystem information during the savefs. That doesn't appear to be the case here.

Re: Restore filesystem with small block size requirements

Posted: Thu Dec 19, 2013 4:11 am
by Berix
Hi,

It does re-create the filesystem on restore. However, it doesn't seem to record certain information about the filesystem (this might be changeable with a patch?)

It seems that it tries to read parameters from the superblock information when it reads the ext2/3/4 filesystem information, but the 'bytes per inode' isn't being stored/retrieved. It does seem to be examining and storing the 'inode size' parameter (specified on mke2fs with -I), but it does nothing with the inodes count (this might be for good reason, I just don't know what it might be right now).

I could maybe try to make a quick patch and try this out -- not really sure if it'll work but it's worth a shot.

Re: Restore filesystem with small block size requirements

Posted: Thu Dec 19, 2013 5:11 am
by Berix
Well after a bit of digging -- it seems that the ext2 superblock doesn't contain the 'inode blocks per group' anywhere, and that it's a calculated field (the tune2fs program is calculating it before displaying it):

Code: Select all

inode_blocks_per_group = (((sb->s_inodes_per_group *
EXT2_INODE_SIZE(sb)) +
EXT2_BLOCK_SIZE(sb) - 1) /
EXT2_BLOCK_SIZE(sb));
Knowing this, the patch I made will do this calculation at the time of the archiving and store this value in the archive. Then, on restore, it will grab this value and append the -i flag to the mke2fs command when it executes.

Now some tests. For these tests, I'm operating on an 8GB filesystem.

Step 1. Create normal ext3 system for a baseline read. Upon creation (without specifying any information about inodes), it creates a filesystem with 524,288 inodes.
Step 2. Create ext3 system with -i 2048 flag specified. This created a filesystem with 4,194,304 inodes.
Step 3. Backup this new filesystem with the original un-modified fsarchiver. Then restore over the top, and the newly created filesystem is back at the original 524,288 inodes (as you saw in your trials).
Step 4. Re-create filesystem with -i 2048 bringing me back to 4,194,304 inodes.
Step 5. Back up this filesystem with the modified fsarchiver.
Step 6. Restore filesystem and the newly created filesystem is now at the correct 4,194,304 inodes. As seen in the verbose output of the fsarchiver restore, the mke2fs command is being executed with the new -i flag:

Code: Select all

# ./fsarchiver -vvv restfs test5.fsa id=0,dest=/dev/sdb1
Detected fileformat=2 in archive /root/tmp2/fsarchiver-0.6.17/src/test5.fsa
Current fsarchiver version: 0.6.17.0
Minimum fsarchiver version for that archive: 0.6.4.0
============= extracting filesystem 0 =============
Current fsarchiver version: 0.6.17.0
Minimum fsarchiver version for that filesystem: 0.6.4.0
strdico_get_string(dicocmdline, 'mkfsopt') doesn't exist
filesystem=[ext3]
filesystemoptions=[]
fsbytestotal=[7.00 GB]
fsbytesused=[171.85 MB]
getpathtoprog(mke2fs)=[/sbin/mke2fs]
executing [mke2fs -V]...
End of volume [/root/tmp2/fsarchiver-0.6.17/src/test5.fsa]
command [mke2fs -V] returned 0
getpathtoprog(mke2fs)=[/sbin/mke2fs]
executing [mke2fs -V]...
command [mke2fs -V] returned 0
the filesystem type determined by the original filesystem features is [ext3]
the filesystem type to create considering the command options is [ext3]
--> feature [has_journal]=YES
--> feature [ext_attr]=YES
--> feature [resize_inode]=YES
--> feature [dir_index]=YES
--> feature [filetype]=YES
--> feature [extent]=NO
--> feature [journal_dev]=NO
--> feature [flex_bg]=NO
--> feature [meta_bg]=NO
--> feature [large_file]=YES
--> feature [huge_file]=NO
--> feature [sparse_super]=YES
--> feature [uninit_bg]=NO
--> feature [dir_nlink]=NO
--> feature [extra_isize]=NO
features: mkfs_options+=[-O has_journal,ext_attr,resize_inode,dir_index,filetype,^extent,^journal_dev,^flex_bg,^meta_bg,large_file,^huge_file,sparse_super,^uninit_bg,^dir_nlink,^extra_isize]
mke2fs version detected: 1.41.12
mke2fs version required: 1.39.0
exec: mke2fs -V
getpathtoprog(mke2fs)=[/sbin/mke2fs]
executing [mke2fs /dev/sdb1  -q  -r 1    -b 4096  -I 256  -i 2048  -O has_journal,ext_attr,resize_inode,dir_index,filetype,^extent,^journal_dev,^flex_bg,^meta_bg,large_file,^huge_file,sparse_super,^uninit_bg,^dir_nlink,^extra_isize ]...
command [mke2fs /dev/sdb1  -q  -r 1    -b 4096  -I 256  -i 2048  -O has_journal,ext_attr,resize_inode,dir_index,filetype,^extent,^journal_dev,^flex_bg,^meta_bg,large_file,^huge_file,sparse_super,^uninit_bg,^dir_nlink,^extra_isize ] returned 0
getpathtoprog(tune2fs)=[/sbin/tune2fs]
executing [tune2fs /dev/sdb1  -U 808fed74-1365-4510-bc02-078724ff5b0d  -c 36  -i 180d ]...
command [tune2fs /dev/sdb1  -U 808fed74-1365-4510-bc02-078724ff5b0d  -c 36  -i 180d ] returned 0
Mount information: []
the filesystem of [/dev/sdb1] type determined by the features is [ext3]
partition [/dev/sdb1] was successfully mounted on [/tmp/fsa/20131218-225611-00] as [ext3] with options [user_xattr,acl]
-[00][ 50%][DIR     ] /
-[00][100%][DIR     ] /lost+found
Statistics for filesystem 0
* files successfully processed:....regfiles=0, directories=2, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
And finally, here's the patch. You can save this into your fsarchiver source directory and apply it with the patch command.

URL: http://berix.us/fsa-iblocks.patch

# cd fsarchiver-0.6.17
# wget http://berix.us/fsa-iblocks.patch
# patch -p1 < fsa-iblocks.patch
patching file src/fsarchiver.h
patching file src/fs_ext2.c[/code]

Then a recompile will put the new logic in place. I'd say use at your own risk and do some testing yourself before using in a production environment -- I'm not really sure what negative side effects this might have, if any. If anyone wants to expand on this or improve it, please do.

Re: Restore filesystem with small block size requirements

Posted: Tue Dec 24, 2013 4:45 pm
by stephen.labar
Thank you for your repsonse. I have finally been able to get back to this test.

Using the patch command gave the following result:
patching file src/fsarchiver.h
Hunk #1 FAILED at 78.
1 out of 1 hunk FAILED -- saving rejects to file src/fsarchiver.h.rej
patching file src/fs_ext2.c
Hunk #1 FAILED at 190.
Hunk #2 FAILED at 365.
2 out of 2 hunks FAILED -- saving rejects to file src/fs_ext2.c.rej

The fs_ext2.h.rej contains:
--- src/fsarchiver.h 2013-02-23 09:37:22.000000000 -0600
+++ src/fsarchiver.h 2013-12-18 23:05:30.946600701 -0600
@@ -78,7 +78,7 @@
FSYSHEADKEY_BTRFSFEATUREROCOMPAT, FSYSHEADKEY_NTFSUUID, FSYSHEADKEY_MINFSAVERSION,
FSYSHEADKEY_MOUNTINFO, FSYSHEADKEY_ORIGDEV, FSYSHEADKEY_TOTALCOST,
FSYSHEADKEY_FSEXTFSCKMAXMNTCOUNT, FSYSHEADKEY_FSEXTFSCKCHECKINTERVAL,
- FSYSHEADKEY_FSEXTEOPTRAIDSTRIPEWIDTH, FSYSHEADKEY_FSEXTEOPTRAIDSTRIDE};
+ FSYSHEADKEY_FSEXTEOPTRAIDSTRIPEWIDTH, FSYSHEADKEY_FSEXTEOPTRAIDSTRIDE, FSYSHEADKEY_FSINODEBLOCKSPERGROUP};

enum {DIRSINFOKEY_NULL=0, DIRSINFOKEY_TOTALCOST};

The fs_ext2.c.rej contains:
--- src/fs_ext2.c 2013-02-23 09:37:18.000000000 -0600
+++ src/fs_ext2.c 2013-12-18 23:06:48.632599737 -0600
@@ -190,6 +190,9 @@

if (dico_get_u64(d, 0, FSYSHEADKEY_FSINODESIZE, &temp64)==0)
strlcatf(options, sizeof(options), " -I %ld ", (long)temp64);
+
+ if (dico_get_u64(d, 0, FSYSHEADKEY_FSINODEBLOCKSPERGROUP, &temp64)==0)
+ strlcatf(options, sizeof(options), " -i %ld ", (long)temp64);

// ---- get original filesystem features (if the original filesystem was an ext{2,3,4})
if (dico_get_u64(d, 0, FSYSHEADKEY_FSEXTFEATURECOMPAT, &features_tab[E2P_FEATURE_COMPAT])!=0 ||
@@ -365,6 +368,13 @@
// ---- filesystem revision (good-old-rev or dynamic)
dico_add_u64(d, 0, FSYSHEADKEY_FSEXTREVISION, super->s_rev_level);

+ // ---- inode blocks per group
+ int inode_blocks_per_group = (((super->s_inodes_per_group *
+ super->s_inode_size) +
+ EXT2_BLOCK_SIZE(super) - 1) /
+ EXT2_BLOCK_SIZE(super));
+ dico_add_u64(d, 0, FSYSHEADKEY_FSINODEBLOCKSPERGROUP, inode_blocks_per_group);
+
// ---- inode size
if (super->s_rev_level >= EXT2_DYNAMIC_REV)
dico_add_u64(d, 0, FSYSHEADKEY_FSINODESIZE, super->s_inode_size);

Re: Restore filesystem with small block size requirements

Posted: Tue Dec 24, 2013 5:12 pm
by Berix
Odd...

Guess that's what I get for not testing it! (happens the same to me when I apply the patch).

I'm wondering if it has something to do with copying/pasting from the forum post. Here's the file as a download instead, that seems to work for me when I try it:

URL: http://berix.us/fsa-iblocks.patch

Code: Select all

# tar xzf fsarchiver-0.6.17.tar.gz
# cd fsarchiver-0.6.17
# wget http://berix.us/fsa-iblocks.patch
# patch -p1 < fsa-iblocks.patch
patching file src/fs_ext2.c
patching file src/fsarchiver.h
#
I'll update my previous post with just the URL

Re: Restore filesystem with small block size requirements

Posted: Wed Jan 08, 2014 6:51 pm
by stephen.labar
I have finished testing archive and restore using the patch provided. I am able to restore the filesystem with the block size and inode count I needed.

On a side note, my current requirement is 2k blocks. I did a quick test using 1k blocks, but the fsarchiver restore gave me similar numbers to what the 2k blocks provided.

Will this patch be rolled into the main tree soon?

Thanks for your help.

Re: Restore filesystem with small block size requirements

Posted: Thu Jan 09, 2014 3:05 pm
by Berix
So when you tested with 1k blocks using the patched version, it came back out at with numbers similar to as if you had it as 2k blocks?

Also -- I'm not affiliated with this project in any way, I'm just interested in it because I'm using it as a bare-metal backup/restore system for my work. It'd be up to the developers if they want this contribution included in the main tree.