[tools] dstat shows incoherent disk transfers

[Carlos Carvalho]

Dag Wieers (dag at wieers.com) wrote on 21 March 2008 19:50:
 >On Fri, 21 Mar 2008, Carlos Carvalho wrote:
 >
 >> I run dstat continuously and often find incoherent numbers. In the
 >> example below there are 2 problems. The first line shows that the sum
 >> of writes from md0,1,2,4 are significantly smaller than the total,
 >> which doesn't make sense (there are no other disks). The second line
 >> shows that reading from md0 is bigger than the total...
 >>
 >> These two particular lines are consecutive but the problems show up in
 >> non-contiguous ones as well.
 >>
 >> --dsk/md4-----dsk/md0----dsk/total----dsk/md2-----dsk/md1--
 >> _read _writ:_read _writ:_read _writ:_read _writ:_read _writ
 >> 134k   21k:  23M 7373B:  23M  291k:   0  1536B:   0     0
 >>  38k   74k:  30M 7373B:  28M  514k:   0    23k:   0     0
 >>              ***         ***
 >>
 >> The invocation is
 >>
 >> dstat -cdny --noupdate -D md4,md0,total,md2,md1 5
 >>
 >> Above I've removed the other output columns. All disks are raid arrays.
 >
 >The total only includes the numbers of the physical disks, not the numbers 
 >of the arrays. The easiest to verify that what you are monitoring is 
 >effectively going to disk is to look at the numbers of the physical disks 
 >as well.

md4 and md0 are 15-disk raid6, so the total for each should closely
match the value for the md device (parity is 2/15 only). md1 and 3 are
2-disk raid1, so the total should be twice the md device. However
their traffic is seen to be very small, so the numbers don't match.

 >If for some reason the Linux kernel is quicker in registering the software 
 >RAID throughput than physical disks, than what you are seeing makes a lot 
 >of sense. It all depends on the different layers and how the kernel is 
 >providing that information.

 >In a sense dstat cannot be wrong, but the kernel could be. Or because of 
 >the timing you see the same throughput first in de md? devices and only a 
 >second later in the physical disk layer. If you are considering it on a 
 >per second basis, you are probably not considering what goes on inside of 
 >the kernel.

That's more likely. The invocation shows a 5s delay, with --noupdate.
I'd expect these differences to be small for a 5s sampling. There
could be bursts though. A significantly longer sampling is needed to
check this.