Page 1 of 1

Performance

Posted: Wed Dec 16, 2009 1:35 pm
by mydadisalive
Hi,

I'm using fsarchiver to backup a system during load and it takes 50 minutes to finish:
I tried to use the -j8 (it's a IBM blade with 8 cores) and didn't help much.

The system sizes:
[[email protected] ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 18G 12G 5.2G 70% /
/dev/sda7 4.8G 139M 4.4G 4% /tmp
/dev/sda6 9.5G 151M 8.9G 2% /dd
/dev/sda3 18G 6.7G 11G 39% /var
/dev/sda1 190M 14M 167M 8% /boot
tmpfs 2.0G 0 2.0G 0% /dev/shm
10.45.97.55:/vol/EDR/
51G 8.1G 43G 17% /usr/cti/store_edr
10.45.97.55:/vol/CPS 31G 9.8G 21G 33% /usr/cti/CPS
10.45.97.55:/vol/EDR 51G 8.1G 43G 17% /usr/cti/EDR
10.45.97.55:/vol/logs
11G 447M 10G 5% /var/log/bkup
10.45.97.55:/vol/backup
31G 7.0G 24G 23% /var/cti/swp/backup
[[email protected] ~]#

System top process information:
top - 14:22:20 up 8 days, 1:53, 5 users, load average: 5.22, 3.58, 2.43
Tasks: 153 total, 2 running, 151 sleeping, 0 stopped, 0 zombie
Cpu0 : 24.5%us, 2.0%sy, 0.0%ni, 61.8%id, 9.8%wa, 1.0%hi, 1.0%si, 0.0%st
Cpu1 : 27.7%us, 4.0%sy, 0.0%ni, 58.4%id, 8.9%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu2 : 21.6%us, 2.9%sy, 0.0%ni, 69.6%id, 5.9%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 24.5%us, 3.9%sy, 0.0%ni, 60.8%id, 9.8%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu4 : 24.8%us, 3.0%sy, 0.0%ni, 62.4%id, 8.9%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu5 : 25.5%us, 4.9%sy, 0.0%ni, 60.8%id, 8.8%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 22.3%us, 1.9%sy, 0.0%ni, 68.9%id, 5.8%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu7 : 28.7%us, 4.0%sy, 0.0%ni, 47.5%id, 18.8%wa, 0.0%hi, 1.0%si, 0.0%st
Mem: 4149848k total, 3994628k used, 155220k free, 45360k buffers
Swap: 8193140k total, 76k used, 8193064k free, 2776664k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29317 root 18 0 99.6m 5164 464 D 100 0.1 4:08.96 fsarchiver -A savef
19391 adsuser 15 0 866m 281m 14m S 65 7.0 400:31.67 /opt/MVAP/bin/SMART
5031 adsuser 17 0 1870m 740m 13m S 47 18.3 206:59.90 /usr/java/jdk1.5.0_
273 root 10 -5 0 0 0 S 1 0.0 0:02.48 [kswapd0]
3022 root 16 0 7824 4420 1476 S 1 0.1 0:12.32 /opt/mount_WD/mount
18040 root 16 0 2292 1056 796 S 1 0.0 0:05.11 top
1 root 15 0 2044 636 548 S 0 0.0 0:09.26 init [3]
2 root RT 0 0 0 0 S 0 0.0 0:01.58 [migration/0]
3 root 34 19 0 0 0 S 0 0.0 0:00.06 [ksoftirqd/0]
4 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/0]
5 root RT 0 0 0 0 S 0 0.0 0:12.41 [migration/1]
6 root 34 19 0 0 0 S 0 0.0 0:00.58 [ksoftirqd/1]
7 root RT 0 0 0 0 S 0 0.0 0:00.00 [watchdog/1]

Running:
time ./ImageBackup.sh --batch --name ADS1 --image (without using -j flag)
*** Saving partition table to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda-partition-info.sfdisk
*** Saving the MBR to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda.mbr
*** Saving partition labels of /dev/sda to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda.labels
*** Backing up /dev/sda1 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda1 ...
*** Backing up /dev/sda3 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda3 ...
*** Backing up /dev/sda5 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda5 ...
*** Backing up /dev/sda6 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda6 ...
*** Backing up /dev/sda7 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:31-ADS1/sda7 ...
Congratulations! The System Disk Backup process has finished successfully

real 29m52.416s
user 20m56.196s
sys 1m10.341s

total time= 50 minutes

Running:
time ./ImageBackup.sh --batch --name ADS1 --image (-j8 , j=number of threads)

*** Saving partition table to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda-partition-info.sfdisk
*** Saving the MBR to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda.mbr
*** Saving partition labels of /dev/sda to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda.labels
*** Backing up /dev/sda1 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda1 ...
*** Backing up /dev/sda3 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda3 ...
*** Backing up /dev/sda5 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda5 ...
*** Backing up /dev/sda6 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda6 ...
*** Backing up /dev/sda7 to /var/cti/swp/backup/ImageBackup/data/ADS1/image/Dec-16-2009-14:12-ADS1/sda7 ...
Congratulations! The System Disk Backup process has finished successfully

real 16m44.435s
user 22m35.852s
sys 1m38.702s


Total time = 49minutes.

Any ideas?

Re: Performance

Posted: Wed Dec 16, 2009 1:43 pm
by mydadisalive
oh and I'm using -z3

Re: Performance

Posted: Wed Dec 16, 2009 10:47 pm
by admin
If you have a very powerful CPU, the disk is likely to be the bottleneck (the CPU is waiting for the disk). You should run dstat in the same time (dstat 10) and observe both the CPU usage and disk throughput.

You should also use a better compression level such as -z7 or -z8 (be careful with very high levels they require a huge amount of ram multiplied by the number of compression threads). A better compression means a smaller archive file and then less things to write to the disk, which makes sense if the disk is the bottleneck.

Re: Performance

Posted: Thu Dec 17, 2009 9:01 am
by mydadisalive
I ran the backup on a customer who claims that with Acronis it took them to do the same backup in 15 minutes on the exact same conditions.

If this is the case indeed, I find it hard to believe that the disks are the bottleneck.

By the way, with -z2 the backup took 40 minutes (instead of 50 with -z3).
With -z5 it took 240 minutes (!!!)

Thanks again,
Avichay.

Re: Performance

Posted: Thu Dec 17, 2009 10:10 pm
by admin
On computers with 8 CPUs, there are reasons to think the disk may be the bottleneck. You should really run "dstat 30" during the test to see really which resources are doing what. It's very simple to use.

Re: Performance

Posted: Sun Dec 20, 2009 10:07 am
by mydadisalive
i'm using redhat, where do i get this dstat?

Avichay.

Re: Performance

Posted: Sun Dec 20, 2009 11:00 am
by admin
On CentOS-5 I can install dstat just with "yum install dstat" with the default repositories. So I guess it will be the same thing with RHEL5.
If it does not work, you can just download an rpm for dstat. It's a simple Python script, that's why it's a noarch rpm package.