[errno=95, Operation not supported]:

Please ask questions here if you are not familiar with fsarchiver
Post Reply
rhubarbpie
Posts: 2
Joined: Mon May 10, 2010 1:07 pm

[errno=95, Operation not supported]:

Post by rhubarbpie » Mon May 10, 2010 1:19 pm

I'm receiving endless [errno=95, Operation not supported]: messages when attempting to restore a partition backup.

I imaged (apparently successfully) using "fsarchiver savefs Backup.fsa /dev/sda2." I received no errors and the recap showed files processed.

However, when restoring using "fsarchiver restfs Backup.fsa id=0,dest=/dev/sda2" I receive wall-to-wall errno=95's. I've used Partition Image successfully for years but assume I'm doing something wrong with fsarchiver. ?

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: [errno=95, Operation not supported]:

Post by admin » Mon May 10, 2010 2:43 pm

Can you run the command again with -vvvv and paste few line of such error messages ?

rhubarbpie
Posts: 2
Joined: Mon May 10, 2010 1:07 pm

Re: [errno=95, Operation not supported]:

Post by rhubarbpie » Mon May 10, 2010 9:17 pm

Thank you for replying and the suggestion. However, it seems to be storing properly now with my original command.

Although I still suspect I did something wrong, I have no idea what. I attempted to restore numerous times over several days prior to posting. I also backed up and attempted to restore numerous times.

It's possible I attempted to restore while booted to a different distribution and kernel. However, I believe it was my currently-booted distribution using 2.6.26.

So the good news is I no longer have an error. The bad news is I don't know why. If I again encounter the error I'll re-post.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: [errno=95, Operation not supported]:

Post by admin » Tue May 11, 2010 11:56 am

The running system may have an impact on that. For instance if you run savefs / restfs from a system where selinux is enabled the result will be different. On SystemRescueCd selinux is disabled so there won't be any permission denied for that reason, but it can still read/restore selinux attributes which are just xattr on the filesystem.

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Sat Jul 31, 2010 2:23 pm

I'm trying to build a bunch of CentOS Virtual Machine clones (VMWare) and am having the same problem -- consistently. Here's what I'm doing:

Install CentOS 5.4 w/ SELinux enabled on an ext2 partition.
Reboot host via netboot to a small Gentoo kernel (2.6.34 r1) -- no SELinux
Archive the install image: fsarchiver -j2 -z6 savefs /root/images/centos5-4.fsa /dev/hda1

Boot a second host via netboot to same small Gentoo kernel (2.6.34 r1) -- no SELinux
Create an ext2 partition as /dev/hda1
Restore the image: fsarchiver -v restfs /root/images/centos5-4.fsa id=0,dest=/dev/hda1

I get a huge wall of errno=95's. Here's the tail of the output:

Code: Select all

[errno=95, Operation not supported]: oper_restore.c#213,extractar_restore_attr_xattr(): xattr:lsetxattr(/sbin/pccardctl,security.selinux) failed
-[00][ 99%][REGFILEM] /sbin/lsusb
[errno=95, Operation not supported]: oper_restore.c#213,extractar_restore_attr_xattr(): xattr:lsetxattr(/sbin/lsusb,security.selinux) failed
-[00][ 99%][REGFILEM] /sbin/pcmcia-check-broken-cis
[errno=95, Operation not supported]: oper_restore.c#213,extractar_restore_attr_xattr(): xattr:lsetxattr(/sbin/pcmcia-check-broken-cis,security.selinux) failed
-[00][100%][REGFILEM] /sbin/pcmcia-socket-startup
[errno=95, Operation not supported]: oper_restore.c#213,extractar_restore_attr_xattr(): xattr:lsetxattr(/sbin/pcmcia-socket-startup,security.selinux) failed
Statistics for filesystem 0
* files successfully processed:....regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=81377, directories=7813, symlinks=11575, hardlinks=3946, specials=7
I have tried this with fsarchiver version 0.5.8 and 0.6.10 and get the same error.

The CentOS clones appear to boot and run, however the SELinux attribute labels were not restored, so I don't know what might be broken.

Any help would be greatly appreciated.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: [errno=95, Operation not supported]:

Post by admin » Sat Jul 31, 2010 6:46 pm

selinux labels are implemented as simple "extended attributes". I suspect the kernel you are running does not support extended attributes on that filesystem. You should have something like that in your kernel options:

Code: Select all

$ zcat /proc/config.gz | grep XATTR
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_REISERFS_FS_XATTR=y
You can run fsarchiver from SystemRescueCd, it has the appropriate options.
Please forget old fsarchiver versions such as 0.5.8, only use recent 0.6.10 with have many bug fixes.

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Sat Jul 31, 2010 7:05 pm

Thank you for your response. The /proc/config.gz file does not exist in the gentoo kernel, so the command you posed will not work. I will double check the extended file system attributes in our gentoo kernel, but it should be noted that fsarchiver is CLEARLY saving the extended attributes in the archive it creates while running in gentoo, otherwise it would not be trying to restore them. I am strongly suspicious this is a bug in the fsarchiver software.

We cannot run with a rescue CD. This will not work in our environment.

We just upgraded to 0.6.10 from 0.5.8, but had to roll back because we encountered other problems with 0.6.10 and Windows systems. We need a reliable solution that works for both Windows and Linux.

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: [errno=95, Operation not supported]:

Post by admin » Sat Jul 31, 2010 7:40 pm

1) /proc/config.gz is a very easy way to check which options have been enabled in the kernel, but many distribution don't provide that and put the configuration in /boot instead
2) A lot of bugs have been fixed since 0.5.8 so such old version should really not be used. If there are problems with 0.6.10 then it has to be reported and fixed in the new version
3) What fsarchiver does about extended attributes is quite simple on linux filesystems. It simply reads and writes attributes using system functions. The error we can see (Operation not supported) is just what the system returns when fsarchiver tries to restore the attribute. That sort of problem is likely due to a problem in the environment (kernel support for extended attributes / selinux, permissions problems, ...). Can you run fsarchiver from strace to see exactly what happens:

Code: Select all

strace -o /var/log/fsarchiver-strace.log fsarchiver restfs <your-args>
4) Running fsarchiver from SystemRescueCd would help to debug the problem even if you cannot rely on that for your production environment

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Sat Jul 31, 2010 8:43 pm

Thanks for your fast response!

1) I checked out /boot/config. It contains the following:

Code: Select all

CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_REISERFS_FS_XATTR=y
It looks like it has the required support. Am I reading this correctly?

2) The issues we are having with Windows may well be in the environment -- we have a very old gentoo kernel which we are also trying to update. We get errors complaining about the ntfs drivers and fsarchiver hangs in other instances. Before we bring the issue(s) here, we need to do more local investigation to try and rule out the environment.

3) I ran strace. Here's a snapshot of part of the output:

Code: Select all

lstat64("/tmp/fsa/20100731-111046-00/sbin", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
open("/tmp/fsa/20100731-111046-00/sbin/pcmcia-socket-startup", O_RDWR|O_CREAT|O_TRUNC|O_LARGEFILE, 0644) = 3
write(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\2\0\3\0\1\0\0\0\20\214\4\0104\0\0\0"..., 26348) = 26348
close(3)                                = 0
lchown32("/tmp/fsa/20100731-111046-00/sbin/pcmcia-socket-startup", 0, 0) = 0
chmod("/tmp/fsa/20100731-111046-00/sbin/pcmcia-socket-startup", 0100755) = 0
utimes("/tmp/fsa/20100731-111046-00/sbin/pcmcia-socket-startup", {{1168126906, 0}, {1168126906, 0}}) = 0
lsetxattr("/tmp/fsa/20100731-111046-00/sbin/pcmcia-socket-startup", "security.selinux"..., "unlabeled", 10, 0) = -1 EOPNOTSUPP (Operation not supported)
write(2, "[errno=95, Operation not support"..., 152) = 152
What I am still confused about it why fsarchiver would be able to save the attributes but not restore them, if the kernel had no support for the attributes. Note #1 above. I also checked the system.map file. It contains "sys_lsetxattr".

I spoke to the guy who built the kernel. He says SELinux is not compiled in, which I thought was correct based on the information here: http://www.fsarchiver.org/Attributes#SE ... d_Linux.29. We can try compiling SELinux in but disabling it, but I'm not sure that will make any difference.

4) We may try that if recompiling the kernel with SELinux doesn't fix the problem.

Thanks so much for your help!!!

admin
Site Admin
Posts: 550
Joined: Sat Feb 21, 2004 12:12 pm

Re: [errno=95, Operation not supported]:

Post by admin » Sat Jul 31, 2010 10:24 pm

Yes it sounds you have support for extended attributes, so that's fine. Can you run the restfs command with "-v" and see what's the mkfs command and how the partition is mounted ? It may be necessary to enable support for xattr explicitly at mount time, but fsarchiver should do that automatically. Are you really using ext2, not ext3 ?

What happens when you restore to an ext3 or reiserfs filesystem ? (use "id=0,mkfs=reiserfs" for instance to restore as a reiserfs filesystem). I have never tested selinux attribute with an old ext2 filesystem. My tests just worked when I saved from a CentOS5 live system and restored it from SystemRescueCd.

Can you please attempt a restfs from SystemRescueCd on a test box so that we know whether or not the environment has an impact on that problem ?

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Sat Jul 31, 2010 10:58 pm

Here's the output from restfs with -v

Code: Select all

============= extracting filesystem 0 =============
executing [mke2fs -V]...
command [mke2fs -V] returned 0
executing [which mke2fs]...
command [which mke2fs] returned 0
executing [mke2fs -V]...
command [mke2fs -V] returned 0
executing [mke2fs /dev/hda1  -q  -r 1  -L '/'  -b 4096  -I 128  -O ^has_journal,resize_inode,dir_index,filetype,^journal_dev,sparse_super ]...
Yes, we are using ext2. We're using an automated lab control system that was inherited from another project and is old. We're slowly trying to update it while using it. I will try a manual restore to an ext3 and see if that works. It may take a while. Also, I am not familiar with reiserfs, but will look at it.

I can try a SystemRescueCd on Monday. I am remote at present and don't have physical access.

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Sat Jul 31, 2010 11:24 pm

I manually recreated /dev/hda1 as an ext3 file system, manually ran fsarchiver (outside our system to ensure it wasn't doing something to revert back to ext2), and got the same results as the ext2.

[Update 1: After fsarchiver restored to the ext3 file system, "fsarchiver probe" revealed that the file system had reverted back to ext2. I am assuming this is because the original file system was an ext2 and fsarchiver restored it to be exactly like the original file system... ]

[Update 2: I reran the fsarchiver using "mkfs=ext3". I got the same errno=95 problem. I verified using "fsarchiver probe" that the file system was in fact ext3. ]

Next I will try using the "mkfs=reiserfs" flag.

[Update 3: I ran the fsarchiver using "mkfs=reiserfs". It failed because it's looking for "mkreiserfs", which does not exist on the system. ]

ltinnel
Posts: 6
Joined: Sat Jul 31, 2010 2:02 pm

Re: [errno=95, Operation not supported]:

Post by ltinnel » Tue Aug 03, 2010 8:26 pm

Problem solved. Sorta. Actually, we worked around it.

An upgrade to the latest gentoo kernel and fsarchiver did NOT solve the problem. We did some investigating, and here is what we discovered:

1. If you install CentOS with SELinux enabled and apply a bunch of software packages (i.e., using a kickstart file), everything works fine. You can create a backup and then restore and no errors are generated.

2. If you install CentOS with SELinux disabled and apply a bunch of software packages (i.e., using a kickstart file), you can create an archive using fsarchiver but the restore will spew out errno=95 like crazy. It *appears* (this is the working theory) that when software packages are installed, they include security labels on the embedded files. fsarchiver merrily copies these labels when it creates an archive. When fsarchiver attempts to restore the files (with labels) from an archive, it cannot and the error messages are generated. Not quite sure where the problem really lies, but...

Here is how we worked around the problem. In our kickstart file:
1. Install CentOS with SELinux enabled.
2. Install the needed packages (with SELinux enabled).
3. Run a post processing script that disables SELinux.

After the install has completed, create an archive. Subsequent installs succeed with no error messages. It appears that by disabling SELinux after the installs, that the system does something to disable the labels so they don't wind up in the archive. At least, that is the working theory...

Post Reply