fsarchiver Boot Loaders, partition labels and Fstab support?

Please ask questions here if you are not familiar with fsarchiver
Post Reply
oiaohm
Posts: 2
Joined: Mon Jan 19, 2009 11:34 pm

fsarchiver Boot Loaders, partition labels and Fstab support?

Post by oiaohm » Tue Jan 20, 2009 12:02 am

My first email to François Dupoux
Boot Loaders, partition labels and Fstab support?

They are the major things that block fsarchiver being used instead of partition image. The means to restore a Linux onto any partition type from back up would be good. Also nothing I have read so far says about a meta data block. Something to tell person before extracting what kind of filesystem information has been saved inside fsarchiver and if the current OS has all the features required to extract completely or not.

The tool is so close to perfect its not funny.

Major advantage of fsarchiver over partition image due to the way fsarchiver works it defrags the partition. Major disadvantage compared to parition image that its missing information about the partition itself.

Question I have are you open to have extra support added so fsachiver can be used as a complete OS packer and restore on another machine? Or will it have to be a independant project.
François Dupoux answer
fsarchiver has to be used by someone who knows what he is doing.
fsarchiver is supposed to do one thing (do one thing but to it well)
which is archiving and extracting a filesystem with no data or
attribute losses. So it is supposed to save all the files, its
attributes
and also the filesystem attributes such as the label, uuid, block
size, ... There are still things to do (support for ACL and advanced
filesystem attributes) but the most important stuff is already
implemented. fsarchiver cannot really do the bootloader installation,
of fstab modifications. Just because the program can't guess the
configuration. For instance, if an user has two harddisks, and the
bios is configured to boot from the second disk (/dev/sdb), how can
fsarchiver know that grub must be installed on the second disk
and not on the first one. It's the same thing with fstab: when you
have several partitions, with several linux flavors installed,
what should we do ?

We can develop advanced options to help the end user to do that by
displaying a list of options, but the program cannot make
assumptions about the disk / partition configuration, so we can't just
do everything instead of the user. I first want to improve
the quality of the code, continue to remove bugs, add missing features
such as the encryption, and then I will think about
working on a Qt GUI, on network support, or advanced installation options.

About the meta-data and what the archive contains ? You can already
use "archinfo" to display basic info about your archive,
but things like that have to be improved anyway.

About the abilities of the running OS to restore the partition: in
general, a recent system will work, but a lot of checks
have to be added to manage all the special cases. For instance,
mounting a filesystem with options such as "xattr" to make
sure we will be able to archive/extract the extended attributes, and
things like that. This will require a lot of tests,
a large amount of time, so I will do it gradually.

A lot of important features are implemented, but there is still a lot
of things to do, to manage the corner cases, debug, optimise the speed
and the memory, ...
Ok opps missed seeing the uuid and block sizes.

Ok good news that safe guards for extraction are under development. With xattr word of advice when packing record the largest size of the xattr in the compressed filesystem. Xfs suports 64 kb xattr yet ext2 and ext3 is based on block size so it can be as small as 4kb. Then there are some with unlimited size xattr.

Big thing with fstab and bootloader is being able to record in archive number 1 where the fstab is hidding what compress partition its on so after restore you can correct it. Same with bootloader.

Both could be covered by including a good attach notes. When you have many distributions inside one pack set having a good description of what is inside would be kinda useful at times. Recording like device location the partition had to start off with. Yes there are still people who don't use uiid or labels for mounting there drives.

Processing multiple fstabs is not a major headache if you know where they all are. Ok more generic would be clearly taging what partition contains the /etc directory on a per installed distribution base for Linux and bsd. Allowing when doing mass deployment extraction system to extract and auto alter like hostname and the like.

Big headache is really not knowing where the /etc is and how the boot loader should be configured. Night mares of jumping across systems is invalid settings.

Processing fstabs just come to mind as kinda required when you change the file-system under a OS. Same with the notes thing again recording what the file-system type was to start off with.

Really I should have asked for /etc not fstab. Not thinking big picture.

You have done a good job so far. Yes I do expect some sections to land in the GUI interface like reapplying boot loader and editing fstab if needed. Notes at the package stage make correcting problems at extraction simple.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Post by admin » Tue Jan 20, 2009 11:14 pm

I think I see what you mean. Options to help in changing the contents of the filesystem to deploy a os to different machines. Yes, it's possible to do that, and it will be useful. Anyway, I have a lot more to do first, like adding the ACL, encryption, archive testing. It should not take too long. And fixing all bugs, optimizing, adding extra minor options, so stabilizing.

I also have in mind the ability to create the partition using libparted, to allow extracting a filesystem to a new disk with no disk label or things like that. And then the Qt GUI. But the tool must first do the basics things well, else the advanced options will be useless if we can't trust the basics.

Anyway, I keep your ideas, and I will have to think about the detailed implementation. It won't be implemented in the short term.

oiaohm
Posts: 2
Joined: Mon Jan 19, 2009 11:34 pm

Post by oiaohm » Tue Jan 20, 2009 11:56 pm

Little thing on a QT front end. Remember clonezilla. I see a bigger use for fsacchiver in mass deployments. No fragmentation also simpler to upgrade network to faster filesystem.

So long term means to operate silent from PXE could be really handy. Yes don't forget a silent mode when you create the GUI. ie "program" runs and displays gui. program --silent=operationfile.xml. Runs without creating GUI or even attempting to connect to X11 server. Makes it fully operation for a console based restore disk if needed.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Post by admin » Wed Jan 21, 2009 7:33 am

Yes, a Qt GUI can only be an option, the command line will still be available anyway.

frindly
Posts: 25
Joined: Fri Nov 28, 2008 2:31 pm
Location: Germany

Post by frindly » Fri Jan 30, 2009 11:57 am

hello,
when i disable the autocheck from fsck with tune2fs (like tune2fs -c0)
will this be stored in the image file.
when i restore the image, is the file system new created,
like mkfs ?
has the new file-system the same options?

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Post by admin » Fri Jan 30, 2009 5:30 pm

The superblock contains many things and you can see that using "dumpe2fs -h". At the moment, only the most important settings are preserved: label, uuid, and ext3/ext4 features. At the moment, this sort of setting (fsck frequency) is not saved so it will just restore the default value and the counter is set to zero).

Anyway I really plan to support these things in the future when I consider all the more important things have been done.

tuipveus
Posts: 44
Joined: Thu May 14, 2009 7:02 pm

keep it simple

Post by tuipveus » Wed May 20, 2009 9:09 pm

I think fsarchiver should be kept simple.

Making things automatic (such as taking image of whole disk) should be another separate project, which could be included to system rescue cd and which would use existing programs (dd, partimage, fsachiver) to copy whole operating system including all partitions/disks asking from user as little as possible, but beeing customizable if user prefers that.

And about the booting-problem and grub... Would it be possible to find out which files are needed for booting and restore their blocks from image file to their original places, before restoring other files? This way it wouldn't be needed to reinstall grub.

This ofcourse causes another problems, not everyone is using grub, at least with Windows.

I think program or script with right kind of intelligence, could handle all the variations of different kind of operatingsystems, partitiontables, bootloaders.

I have made one script which works perfectly with Windows-computers with specific partitiontable, another which works with Linux-servers. I think it is not impossible to do the whole program with bash or python for example.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Post by admin » Wed May 20, 2009 9:34 pm

These things are good ideas but I don't want to use too many externals programs for these things. It's a simple C program because it's the best language for system tools. If you need python to restore your boot loader it's a real problem (I like that language anyway).

I want to try to provide all these services with only one main program and the minimum external dependencies. If we want to mix different programs such as sfdisk, partimage, ... you can be sure that you will never have the right version of something or a library will be missing. As a simple user of quite complex programs, it's a real pain to spend hours just to figure out how to combine these things together.

We can try to add some intelligent functions to install the right boot loader, or just options to let the user choose. Installing a basic MBR for Windows is very simple. I can try to see how grub-install works or call an external grub command if it's the only way to reinstall grub.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Post by admin » Wed May 20, 2009 9:46 pm

I found this program called grubinstall:
http://giraffe-data.com/software/grubinstall.html

It seems to be a simple grub installer perl script that does the same sort of things as grub-install (the official command that comes with grub) but another way.

It's a quite small perl script, I can try to reproduce it in fsarchiver. It would allow to reinstall grub properly, even on partitions with a different size (it would stay flexible), and it would not require the grub programs to be installed to work (just the grub files which are part of the archive).

tuipveus
Posts: 44
Joined: Thu May 14, 2009 7:02 pm

Re: fsarchiver Boot Loaders, partition labels and Fstab support?

Post by tuipveus » Sun Sep 27, 2009 12:13 pm

If you need python to restore your boot loader it's a real problem (I like that language anyway).
It's a quite small perl script, I can try to reproduce it in fsarchiver.
You want to reproduce, because you afraid that perl would work different way in different versions? Well well, I think it would be easier to maintain scripts which work with standard tools (like fdisk/sfdisk) than maintain C program. But that depends of you, I can maintain my scripts beside. I have tested that they work and I know their limitations. Backup-program is clearly nothing that I want to test. Maturity reliability are the most important things for me and I think also for other people.

I just hope that in future (too) you can give commandline-parameters to fsarchiver and do everything manually. I think that there are so many different kind of variations of partitiontables, mbrs etc, that it is almost impossible to make program to handle all those exceptions. Most of the people want to make exact copy of their harddisk, they don't want to know anything about bootloaders and they don't want to install those.

I really highly appreciate your work with srcd, partimage and fsarchiver. But still, I am critical. I think your time is going to re-inventing wheel. Redoing things what others have already done.

I like unix-philosophy:
http://en.wikipedia.org/wiki/Unix_philosophy wrote:This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
For example, think how nice it would be
a) to make an archive go to stream, so you can use any compression program what you want.
b) stream to another computers over the network
c) encrypt the stream with any program what you want

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: fsarchiver Boot Loaders, partition labels and Fstab support?

Post by admin » Sun Oct 04, 2009 6:20 pm

I also like the unix philosophy but I don't like complexity. It's one of my most important rules: Keep It Simple.
I have seen many projects where things don't work just because they are too complicated: I often have to install software, and many program just never work (especially java applications) just because they depend on many external things which are not very standard, and there is always something missing. Or all these things don't work together with the versions you have. I prefer an integrated c program that just need standard system libraries to work.

fsarchiver is not using streams. The problem with streams is you have to process bytes in the natural order. You can't go back. And seeks have been necessary to implement correct error handling, ie the ability to restore corrupt archive files. Also when you have a corruption in a gzip file, because byte(n) depends on byte(n-1) you can loose everything. It's a real problem when compression is done on the top or archiving: tar.gz / tar.bz2. There is a compression at byte(n) of your tar.gz then you can't read byte(n+1) so you cannot read the next file. In fsarchiver the compression is done by block: if you loose a file, you will be able to restore other files or the archive.

I see no reason to use another external program for compression: you already have to four best compression levels: lzo for speed, gzip, bzip2, and lzma for performances. They are all implemented in very robust libraries, and the management is optimal (multi-threading compression). So fsarchiver won't depend on external processes to compress, that's more robust. Also a program has a bad control on external processes compared to internal functions of the code with return values, and buffers passed in parameters.

Post Reply