Page 1 of 2

Cannot allocate memory error when calling lgetxattr

Posted: Thu Oct 08, 2009 10:10 am
by robertchen
My /dev/sdb2 is a NTFS partition (usage 1.2G / total 4G) in my flash storage device.
I use fsarchiver (0.5.8) to backup /dev/sdb2.
Sometimes the following errors occurs. It seems fsarchiver needs a lot of memory.
However, before running fsarchiver I still have more than 1 G free memory, and running fsarchiver will comsume free memory very soon (until less than 100m).
My cpu is Intel Core2 Quad 2.33GHz. (total ram 3.5G).
My OS is Ubuntu 9.04 (kernel 2.6.28-15-generic)

=======my log=======

sudo fsarchiver -o savefs -z2 usbntfs.fsa /dev/sdb2

[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/IME/IMJP8_1/Help,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/INF/printupg.inf,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/Installer/{350C97B0-3D7C-4EE8-BAA9-00BCB3D54227},system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/PCHEALTH/HelpCtr/Indices/scoped_2.hhk,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/srchasst/msgr3en.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/srchasst/srchctls.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/chtbrkr.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/ieencode.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/iepeers.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/oembios.cat,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/rdsaddin.exe,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/redir.exe,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/scredir.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wbem/Repository/FS/OBJECTS.MAP,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/system.drv,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMASF.DLL,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMVSDECD.DLL,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WPDShextAutoplay.exe,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/xpsp3res.dll,system.ntfs_acl)=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=6260, directories=681, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=17, directories=2, symlinks=0, hardlinks=0, specials=0

======dmesg======
[29749.479790] fsarchiver: page allocation failure. order:4, mode:0xc0d0
[29749.479795] Pid: 14720, comm: fsarchiver Not tainted 2.6.28-15-generic #52-Ubuntu
[29749.479797] Call Trace:
[29749.479804] [<c04fe166>] ? printk+0x18/0x1a
[29749.479808] [<c0194507>] __alloc_pages_internal+0x387/0x490
[29749.479811] [<c0194661>] __get_free_pages+0x21/0x40
[29749.479815] [<c01b902e>] __kmalloc+0xee/0xf0
[29749.479819] [<c02ce267>] ? strncpy_from_user+0x37/0x60
[29749.479823] [<c01d7396>] getxattr+0xf6/0x120
[29749.479827] [<c01c75d9>] ? __link_path_walk+0x559/0xb30
[29749.479830] [<c0500398>] ? _spin_lock+0x8/0x10
[29749.479833] [<c01d3b7d>] ? mntput_no_expire+0x1d/0x120
[29749.479835] [<c01c6230>] ? path_put+0x20/0x30
[29749.479838] [<c01c8052>] ? path_walk+0xa2/0xb0
[29749.479841] [<c01c6075>] ? putname+0x25/0x40
[29749.479844] [<c01c6075>] ? putname+0x25/0x40
[29749.479846] [<c01c8de3>] ? user_path_at+0x53/0x80
[29749.479850] [<c01d3b7d>] ? mntput_no_expire+0x1d/0x120
[29749.479852] [<c0500398>] ? _spin_lock+0x8/0x10
[29749.479855] [<c01d748a>] sys_lgetxattr+0x4a/0x60
[29749.479858] [<c0104062>] syscall_call+0x7/0xb
[29749.479860] Mem-Info:
[29749.479862] DMA per-cpu:
[29749.479864] CPU 0: hi: 0, btch: 1 usd: 0
[29749.479865] CPU 1: hi: 0, btch: 1 usd: 0
[29749.479867] CPU 2: hi: 0, btch: 1 usd: 0
[29749.479869] CPU 3: hi: 0, btch: 1 usd: 0
[29749.479870] Normal per-cpu:
[29749.479872] CPU 0: hi: 186, btch: 31 usd: 0
[29749.479874] CPU 1: hi: 186, btch: 31 usd: 0
[29749.479876] CPU 2: hi: 186, btch: 31 usd: 115
[29749.479877] CPU 3: hi: 186, btch: 31 usd: 0
[29749.479879] HighMem per-cpu:
[29749.479880] CPU 0: hi: 186, btch: 31 usd: 0
[29749.479882] CPU 1: hi: 186, btch: 31 usd: 0
[29749.479884] CPU 2: hi: 186, btch: 31 usd: 0
[29749.479885] CPU 3: hi: 186, btch: 31 usd: 0
[29749.479888] Active_anon:135627 active_file:39060 inactive_anon:41380
[29749.479890] inactive_file:559369 unevictable:0 dirty:9109 writeback:0 unstable:0
[29749.479891] free:90955 slab:14041 mapped:26170 pagetables:1045 bounce:0
[29749.479894] DMA free:3512kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:128kB inactive_file:4956kB unevictable:0kB present:15804kB pages_scanned:0 all_unreclaimable? no
[29749.479897] lowmem_reserve[]: 0 861 3505 3505
[29749.479902] Normal free:6268kB min:3720kB low:4648kB high:5580kB active_anon:4kB inactive_anon:8092kB active_file:14864kB inactive_file:706664kB unevictable:0kB present:881880kB pages_scanned:174 all_unreclaimable? no
[29749.479905] lowmem_reserve[]: 0 0 21157 21157
[29749.479911] HighMem free:354040kB min:512kB low:3368kB high:6224kB active_anon:542504kB inactive_anon:157428kB active_file:141248kB inactive_file:1525672kB unevictable:0kB present:2708216kB pages_scanned:0 all_unreclaimable? no
[29749.479914] lowmem_reserve[]: 0 0 0 0
[29749.479917] DMA: 12*4kB 5*8kB 4*16kB 5*32kB 8*64kB 19*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3512kB
[29749.479926] Normal: 1126*4kB 4*8kB 5*16kB 4*32kB 6*64kB 4*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 6408kB
[29749.479935] HighMem: 184*4kB 109*8kB 35*16kB 11*32kB 1477*64kB 407*128kB 71*256kB 17*512kB 12*1024kB 17*2048kB 32*4096kB = 354200kB
[29749.479944] 625575 total pagecache pages
[29749.479946] 7730 pages in swap cache
[29749.479948] Swap cache stats: add 9121, delete 1391, find 4932/5049
[29749.479949] Free swap = 482548kB
[29749.479951] Total swap = 514040kB
[29749.489457] 908672 pages RAM
[29749.489459] 682386 pages HighMem
[29749.489460] 13688 pages reserved
[29749.489462] 501343 pages shared
[29749.489463] 370193 pages non-shared
[31396.159658] fsarchiver: page allocation failure. order:4, mode:0xc0d0
[31396.159663] Pid: 15295, comm: fsarchiver Not tainted 2.6.28-15-generic #52-Ubuntu
[31396.159665] Call Trace:
[31396.159673] [<c04fe166>] ? printk+0x18/0x1a
[31396.159677] [<c0194507>] __alloc_pages_internal+0x387/0x490
[31396.159680] [<c0194661>] __get_free_pages+0x21/0x40
[31396.159684] [<c01b902e>] __kmalloc+0xee/0xf0
[31396.159688] [<c02ce267>] ? strncpy_from_user+0x37/0x60
[31396.159692] [<c01d7396>] getxattr+0xf6/0x120
[31396.159696] [<c01c75d9>] ? __link_path_walk+0x559/0xb30
[31396.159699] [<c0500398>] ? _spin_lock+0x8/0x10
[31396.159702] [<c01d3b7d>] ? mntput_no_expire+0x1d/0x120
[31396.159705] [<c01c6230>] ? path_put+0x20/0x30
[31396.159708] [<c01c8052>] ? path_walk+0xa2/0xb0
[31396.159710] [<c01c6075>] ? putname+0x25/0x40
[31396.159713] [<c01c6075>] ? putname+0x25/0x40
[31396.159716] [<c01c8de3>] ? user_path_at+0x53/0x80
[31396.159719] [<c01d3b7d>] ? mntput_no_expire+0x1d/0x120
[31396.159721] [<c0500398>] ? _spin_lock+0x8/0x10
[31396.159724] [<c01d748a>] sys_lgetxattr+0x4a/0x60
[31396.159728] [<c0104062>] syscall_call+0x7/0xb
[31396.159731] [<c0500000>] ? rwsem_down_failed_common+0xa0/0x1b0
[31396.159733] Mem-Info:
[31396.159734] DMA per-cpu:
[31396.159736] CPU 0: hi: 0, btch: 1 usd: 0
[31396.159738] CPU 1: hi: 0, btch: 1 usd: 0
[31396.159740] CPU 2: hi: 0, btch: 1 usd: 0
[31396.159741] CPU 3: hi: 0, btch: 1 usd: 0
[31396.159743] Normal per-cpu:
[31396.159745] CPU 0: hi: 186, btch: 31 usd: 0
[31396.159746] CPU 1: hi: 186, btch: 31 usd: 0
[31396.159748] CPU 2: hi: 186, btch: 31 usd: 127
[31396.159750] CPU 3: hi: 186, btch: 31 usd: 0
[31396.159751] HighMem per-cpu:
[31396.159753] CPU 0: hi: 186, btch: 31 usd: 0
[31396.159755] CPU 1: hi: 186, btch: 31 usd: 0
[31396.159756] CPU 2: hi: 186, btch: 31 usd: 0
[31396.159758] CPU 3: hi: 186, btch: 31 usd: 0
[31396.159762] Active_anon:134711 active_file:39010 inactive_anon:41383
[31396.159763] inactive_file:523104 unevictable:0 dirty:10217 writeback:0 unstable:0
[31396.159764] free:128509 slab:13708 mapped:26155 pagetables:1046 bounce:0
[31396.159767] DMA free:3560kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:40kB inactive_file:4800kB unevictable:0kB present:15804kB pages_scanned:0 all_unreclaimable? no
[31396.159770] lowmem_reserve[]: 0 861 3505 3505
[31396.159775] Normal free:6232kB min:3720kB low:4648kB high:5580kB active_anon:4kB inactive_anon:8092kB active_file:13752kB inactive_file:709476kB unevictable:0kB present:881880kB pages_scanned:128 all_unreclaimable? no
[31396.159778] lowmem_reserve[]: 0 0 21157 21157
[31396.159784] HighMem free:504244kB min:512kB low:3368kB high:6224kB active_anon:538840kB inactive_anon:157440kB active_file:142248kB inactive_file:1378140kB unevictable:0kB present:2708216kB pages_scanned:0 all_unreclaimable? no
[31396.159787] lowmem_reserve[]: 0 0 0 0
[31396.159790] DMA: 3*4kB 5*8kB 3*16kB 3*32kB 2*64kB 1*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3524kB
[31396.159800] Normal: 1304*4kB 34*8kB 12*16kB 5*32kB 10*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6480kB
[31396.159808] HighMem: 162*4kB 106*8kB 33*16kB 12*32kB 4*64kB 0*128kB 2*256kB 75*512kB 32*1024kB 22*2048kB 94*4096kB = 504424kB
[31396.159818] 589294 total pagecache pages
[31396.159819] 7733 pages in swap cache
[31396.159821] Swap cache stats: add 9126, delete 1393, find 5262/5380
[31396.159823] Free swap = 482556kB
[31396.159824] Total swap = 514040kB
[31396.169102] 908672 pages RAM
[31396.169105] 682386 pages HighMem
[31396.169106] 13688 pages reserved
[31396.169107] 466911 pages shared
[31396.169109] 367243 pages non-shared


Please help to fix this problem.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Sun Oct 11, 2009 10:07 am
by admin
Have a look at these two pages about memory requirement:
- http://www.fsarchiver.org/Compression
- http://www.fsarchiver.org/forums/viewtopic.php?p=2259

So fsarchiver may use a huge amount of memory if you use a very high compression level (lzma: -z7, -z8, -z9), and the memory requirement will be multiplied if you have more than one compression job (-j2, -j3, ...)

It shouldn't use that much memory with a low compression level (-z2) and a single compression job (option -j not use: 1 job by default)

Can you show /proc/meminfo before you run fsarchiver ?

You can run "pmap -d <pid>" to see how much memory it's using when it runs. Also please update fsarchiver to the latest version first, we don't want to investigate for a bug that may be fixed already.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Tue Oct 13, 2009 4:14 am
by robertchen
I update to fsarchiver-0.6.1 and the same problem still often occurs.
I use -z2 compression leve and only one (default) compression thread.

My build option:
./configure --prefix=/home/robert/soft_output/fsarchiver061 --disable-lzma --disable-lzo --disable-crypto

Just before running, I show /proc/meminfo:

[email protected] ~ $ cat /proc/meminfo
MemTotal: 3579940 kB
MemFree: 1994188 kB
Buffers: 21076 kB
Cached: 1179076 kB
SwapCached: 3616 kB
Active: 250080 kB
Inactive: 1233976 kB
Active(anon): 179556 kB
Inactive(anon): 181048 kB
Active(file): 70524 kB
Inactive(file): 1052928 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 2729544 kB
HighFree: 1315004 kB
LowTotal: 850396 kB
LowFree: 679184 kB
SwapTotal: 514040 kB
SwapFree: 507396 kB
Dirty: 1740 kB
Writeback: 0 kB
AnonPages: 280520 kB
Mapped: 74012 kB
Slab: 43584 kB
SReclaimable: 28988 kB
SUnreclaim: 14596 kB
PageTables: 3592 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2304008 kB
Committed_AS: 1322208 kB
VmallocTotal: 122880 kB
VmallocUsed: 13016 kB
VmallocChunk: 102304 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 61432 kB
DirectMap4M: 843776 kB

[email protected] ~/soft_output/fsarchiver-0.6.1/sbin $ sudo ./fsarchiver savefs -z2 -o ~/Test/ntfs.fsa /dev/sdb2
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/config/SYSTEM.LOG,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/iepeers.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/msutb.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/oembios.cat,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/shellstyle.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/win32spl.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/winchat.exe,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmpasf.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#339,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMVDECOD.DLL,system.ntfs_acl)=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=6268, directories=683, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=9, directories=0, symlinks=0, hardlinks=0, specials=0

I wonder if the system call lgetxattr() needs a lot of memory sometimes?

(In the running, I use free -m to see my free memory size.
It does not always decrease free memory to less than 100mb.
In this test, my free memory are more than 600 mb when the error text are printed.)

[email protected] ~/SoftInst/fsarchiver-0.6.1 $ free -m
total used free shared buffers cached
Mem: 3496 2276 1220 0 479 1384
-/+ buffers/cache: 412 3083
Swap: 501 14 487
[email protected] ~/SoftInst/fsarchiver-0.6.1 $ free -m
total used free shared buffers cached
Mem: 3496 2635 860 0 621 1598
-/+ buffers/cache: 415 3080
Swap: 501 14 487
[email protected] ~/SoftInst/fsarchiver-0.6.1 $ free -m <====== error log appears around this time
total used free shared buffers cached
Mem: 3496 2828 667 0 712 1699
-/+ buffers/cache: 415 3080
Swap: 501 15 486
[email protected] ~/SoftInst/fsarchiver-0.6.1 $ free -m
total used free shared buffers cached
Mem: 3496 2798 697 0 714 1667
-/+ buffers/cache: 417 3078
Swap: 501 15 486

Re: Cannot allocate memory error when calling lgetxattr

Posted: Sat Oct 17, 2009 8:08 pm
by admin
It's really bizarre: the lgetxattr function returns ENOMEM even when you have plenty of memory.

Anyway, I replaced the static memory allocation with dynamic memory in a new test version.

1) Can you please compile and use this test fsarchiver version ?
http://beta.fsarchiver.org/fsarchiver-0 ... st1.tar.gz

2) Do you always have the problem on the same files ?

3) Which ntfs-3g version are you using, on which linux distribution ?

Re: Cannot allocate memory error when calling lgetxattr

Posted: Fri Oct 23, 2009 11:00 am
by robertchen
The problem is the same in fsarchiver 0.6.2-test1.

I've tried ntfs-3g 2009.4.4AR.19 and ntfs-3g 2009.4.4AR.21.
My OS is Ubuntu 9.04 (kernel 2.6.28-15-generic).

The problem doesn't occur on the same files.

The bug may be inside the system call lgetxattr(). "Cannot allocate memory" doesn't necessarily mean the free memory is not enough.

Here are the log of running 0.6.2-test1 twice in the same environmemnt:

[email protected] ~/Test $ sudo ~/soft_output/fsarchiver-0.6.2-test1/sbin/fsarchiver savefs -z2 -o wes.fsa /dev/sdb2
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/PCHEALTH/HelpCtr/DataColl,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/imm32.dll,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/netui0.dll,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/oembios.cat,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/s3iset32.dll,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/win32spl.dll,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmploc.DLL,system.ntfs_reparse_data)=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=6271, directories=682, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=6, directories=1, symlinks=0, hardlinks=0, specials=0
[email protected] ~/Test $ sudo ~/soft_output/fsarchiver-0.6.2-test1/sbin/fsarchiver savefs -z2 -o wes.fsa /dev/sdb2
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/Resources/Themes/Embedded/Shell,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/Resources/Themes/Luna/MUI,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/asferror.dll,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/config/SYSTEM.LOG,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/mpnotify.exe,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/oembios.cat,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wbem/AutoRecover/42355E8E232EF8CADD187D531DEC55DD.mof,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmploc.DLL,system.ntfs_reparse_data)=-1
[errno=12, Cannot allocate memory]: create.c#345,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMVDECOD.DLL,system.ntfs_reparse_data)=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=6270, directories=681, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=7, directories=2, symlinks=0, hardlinks=0, specials=0

Re: Cannot allocate memory error when calling lgetxattr

Posted: Sat Oct 24, 2009 11:10 am
by admin
It would be interesting to reproduce this problem outside of fsarchiver. I agree with you, the problem could be in the lgetxattr() implementation, so in ntfs-3g.

Can you run this sort of test to see if you can see memory errors:

Code: Select all

# apt-get install attr
# ntfs-3g /dev/xxx /mnt/windows -o streams_interface=xattr
# for attr in system.ntfs_reparse_data system.ntfs_acl ; do find /mnt/windows -exec getfattr -h -n ${attr}  {} \; 2>&1 ; done
# umount /mnt/windows
You will see "No such attribute" errors, it's normal since all the files/dirs are not using all the attributes.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Oct 26, 2009 6:24 am
by robertchen
I want to reproduce the problem outside of fsarchiver, but no memory error in my log was found yet as expected. (Maybe I should try more times).

Go back to see create.c code.

In createar_item_winattr (), the author makes a first call of lgetxattr() to get the attrsize, allocates that size +1 of memory, and makes a second call of lgetxattr() to get the item's xattr.

I output the variable "attrsize". When the error occur, attrsize is -1 in all problem cases, (See the following log).

Code: Select all

sysprintf("            winattr:lgetxattr(%s,%s)=%d, attrsize=%lld\n", relpath, winattr[i], valsize, attrsize);//robert fix
[email protected] ~/SoftInst/ntfs-3g-2009.4.4AR.21 $ sudo ~/soft_output/fsarchiver-0.6.2-test1/sbin/fsarchiver savefs -z2 -o test.fsa /dev/sdb2
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/usmt/sysmod.dll,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/winchat.exe,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmpasf.dll,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmpmde.dll,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMVDECOD.DLL,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WPDShextAutoplay.exe,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/Web/Wallpaper/Windows XP.jpg,system.ntfs_reparse_data)=-1, attrsize=-1
[errno=12, Cannot allocate memory]: create.c#346,createar_item_winattr(): winattr:lgetxattr(/Windows/winsxs/x86_microsoft.windows.networking.dxmrtp_6595b64144ccf1df_5.2.2.3_x-ww_468466a7,system.ntfs_reparse_data)=-1, attrsize=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=6270, directories=682, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=7, directories=1, symlinks=0, hardlinks=0, specials=0


That means the first call of lgetxattr(fullpath, winattr, NULL, 0) return -1 in create.c #318
When attrsize is -1, should we go on making a second call of lgetxattr () ? or we sould skip that item ?

Furthermore, I output more to know that when the first lgetxattr() calls returns -1, the errno is 61 [errno=61, No data available].
(Actually I found that many many first lgetxattr() calls returns -1.)

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Oct 26, 2009 7:35 am
by admin
You are right, we should not make the second call when the first one returns -1. The point of the first call is to know how much memory is required to store the attribute. If the first call returns -1 and the errno is ENODATA / ENOATTR then we should just do a continue to ignore that attribute, because it's not an error.

$ grep 61 /usr/include/asm-generic/errno.h
#define ENODATA 61 /* No data available */

$ cat /usr/include/attr/xattr.h | grep ENOATTR
# define ENOATTR ENODATA /* No such attribute */

We can add this test after the first call to lgetxattr(), and we add errno=0 to make sure the value is accurate.

Code: Select all

	errno=0;
	attrsize=lgetxattr(fullpath, winattr[i], NULL, 0);
	if (attrsize<0 && errno==ENOATTR)
		continue;
So does that means that lgetxattr() returns ENOMEM when we call it a second time when there is no attribute ? Let's see what it does with this modification. I can prepare an fsarchiver-0.6.2-test2.tar.gz version is you need it.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Oct 26, 2009 12:38 pm
by robertchen
Then I add the following code, between the two calls of lgetxattr(), to fsarchiver 0.6.2-test1.

Code: Select all

if( attrsize < 0 )
{
    if( errno!=ENOATTR ) // ENOATTR=ENODATA=61
    {
        sysprintf("  Failed to get attrsize in winattr:lgetxattr(%s,%s)\n", relpath, winattr[i]); // output if there are any other error
    }
    continue;
}
The problem seems to be fixed. Several tries with no error message.

The dynamic allocation version (i.e. 0.6.2-test1) does make sense. Because I go back to try 0.6.1 and find that when 0.6.1 outpus "Cannot allocate memory", the attrsize is positive (but not large, about 100~164); while 0.6.2-test1 outputs "Cannot allocate memory", with attrsize=-1, because ntfs-3g's lgetxattr() cannot malloc(-1).

Anyway fsarchiver 0.6.2-test1 + the code above fixes my problem.
Thanks a lot.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Oct 26, 2009 10:03 pm
by admin
Thanks for your test. I prepared a new version based on 0.6.1 with this fix:
http://beta.fsarchiver.org/fsarchiver-0 ... ta1.tar.gz
I dropped the dynamic memory allocation since it make the code more complex.

Can you please confirm that this version works ? It just to confirm it's ok before the final 0.6.2.

Thanks

Re: Cannot allocate memory error when calling lgetxattr

Posted: Thu Oct 29, 2009 4:32 am
by robertchen
We should add ret=-1 .

Code: Select all

if ((attrsize=lgetxattr(fullpath, winattr[i], NULL, 0)) < 0) // get the size of the attribute
{
	if (errno!=ENOATTR)
	{
		sysprintf("           winattr:lgetxattr(%s,%s): returned negative attribute size\n", relpath, winattr[i]); // output if there are any other error
		ret=-1;
	}
	continue; // ignore the current xattr
}
No, this version does not work fine.
Some system.ntfs_acl attributes cannot be created.
We should use dynamic memory allocation instead.

[email protected] ~$ sudo ~/soft_output/fsarchiver062beta/sbin/fsarchiver savefs -z2 -s100 -o /var/www/firmware/wes/test3/test.fsa /dev/sdb2
[errno=12, Cannot allocate memory]: create.c#350,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmpasf.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#350,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/wmpmde.dll,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#350,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/WMVDECOD.DLL,system.ntfs_acl)=-1
[errno=12, Cannot allocate memory]: create.c#350,createar_item_winattr(): winattr:lgetxattr(/Windows/System32/xpsp3res.dll,system.ntfs_acl)=-1
Statistics for filesystem 0
* files successfully processed:....regfiles=7452, directories=857, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=4, directories=0, symlinks=0, hardlinks=0, specials=0

Re: Cannot allocate memory error when calling lgetxattr

Posted: Thu Oct 29, 2009 10:19 pm
by admin
Thanks for your new test. I prepared a beta2 version based on the dynamic memory and with ret=-1:
http://beta.fsarchiver.org/fsarchiver-0 ... ta2.tar.gz

Can you please try that one ?

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Nov 09, 2009 12:02 pm
by robertchen
This version is ok.
No error message ever appears in my test.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Mon Nov 09, 2009 7:15 pm
by admin
Thanks for coming back with your test. These changes will be in the final 0.6.2 then.

Re: Cannot allocate memory error when calling lgetxattr

Posted: Tue Nov 10, 2009 10:30 am
by Michiel
Hi,

Just wanted to say I had the same errors, using fsarchiver on systemrescuecd version 1.3.1. Today I used the new systemrescuecd version 1.3.2 and I'm not getting the errors anymore. But according to the systemrescuecd changelog (http://www.sysresccd.org/Changes-x86) there hasn't been any update on fsarchiver between those two versions. The fsarchiver version on both 1.3.1 and 1.3.2 is 0.6.1.

So it could be that the cause of the bug is not with fsarchiver...

Good luck and thanks for this very useful tool!