On 07/06/2014 07:58 PM, Pranith Kumar Karampuri wrote:
On 07/06/2014 02:53 AM, Benjamin Turner wrote:
Hi all. I have been running FS sanity on daily builds(glusterfs
mounts only at this point) for a few days for a few days and I have
been hitting a couple of problems:
================ final pass/fail report =================
Test Date: Sat Jul 5 01:53:00 EDT 2014
Total : [44]
Passed: [41]
Failed: [3]
Abort : [0]
Crash : [0]
---------------------------------------------------------
[ PASS ] FS Sanity Setup
[ PASS ] Running tests.
[ PASS ] FS SANITY TEST - arequal
[ PASS ] FS SANITY LOG SCAN - arequal
[ PASS ] FS SANITY LOG SCAN - bonnie
[ PASS ] FS SANITY TEST - glusterfs_build
[ PASS ] FS SANITY LOG SCAN - glusterfs_build
[ PASS ] FS SANITY TEST - compile_kernel
[ PASS ] FS SANITY LOG SCAN - compile_kernel
[ PASS ] FS SANITY TEST - dbench
[ PASS ] FS SANITY LOG SCAN - dbench
[ PASS ] FS SANITY TEST - dd
[ PASS ] FS SANITY LOG SCAN - dd
[ PASS ] FS SANITY TEST - ffsb
[ PASS ] FS SANITY LOG SCAN - ffsb
[ PASS ] FS SANITY TEST - fileop
[ PASS ] FS SANITY LOG SCAN - fileop
[ PASS ] FS SANITY TEST - fsx
[ PASS ] FS SANITY LOG SCAN - fsx
[ PASS ] FS SANITY LOG SCAN - fs_mark
[ PASS ] FS SANITY TEST - iozone
[ PASS ] FS SANITY LOG SCAN - iozone
[ PASS ] FS SANITY TEST - locks
[ PASS ] FS SANITY LOG SCAN - locks
[ PASS ] FS SANITY TEST - ltp
[ PASS ] FS SANITY LOG SCAN - ltp
[ PASS ] FS SANITY TEST - multiple_files
[ PASS ] FS SANITY LOG SCAN - multiple_files
[ PASS ] FS SANITY TEST - posix_compliance
[ PASS ] FS SANITY LOG SCAN - posix_compliance
[ PASS ] FS SANITY TEST - postmark
[ PASS ] FS SANITY LOG SCAN - postmark
[ PASS ] FS SANITY TEST - read_large
[ PASS ] FS SANITY LOG SCAN - read_large
[ PASS ] FS SANITY TEST - rpc
[ PASS ] FS SANITY LOG SCAN - rpc
[ PASS ] FS SANITY TEST - syscallbench
[ PASS ] FS SANITY LOG SCAN - syscallbench
[ PASS ] FS SANITY TEST - tiobench
[ PASS ] FS SANITY LOG SCAN - tiobench
[ PASS ] FS Sanity Cleanup
[ FAIL ] FS SANITY TEST - bonnie
[ FAIL ] FS SANITY TEST - fs_mark
[ FAIL ]
/rhs-tests/beaker/rhs/auto-tests/components/sanity/fs-sanity-tests-v2
Bonnie++ is just very slow(running for 10+ hours on 1 16 GB file) and FS mark
has been failing. The bonnie slowness is in re read, here is the best
explanation I can find on it:
https://blogs.oracle.com/roch/entry/decoding_bonnie
*Rewriting...done*
This gets a little interesting. It actually reads 8K, lseek back to the start
of the block, overwrites the 8K with new data and loops. (see article for
more.).
On FS mark I am seeing:
# fs_mark -d . -D 4 -t 4 -S 5
# Version 3.3, 4 thread(s) starting at Sat Jul 5 00:54:00 2014
# Sync method: POST: Reopen and fsync() each file in order after main
write loop.
# Directories: Time based hash between directories across 4
subdirectories with 180 seconds per subdirectory.
# File names: 40 bytes long, (16 initial bytes of time stamp with 24
random bytes at end of name)
# Files info: size 51200 bytes, written with an IO size of 16384 bytes
per write
# App overhead is time in microseconds spent in the test not doing file
writing related system calls.
FSUse% Count Size Files/sec App Overhead
Error in unlink of ./00/53b784e8~~~~~~~~SKZ0QS9BO7O2EG1DIFQLRDYY : No such file
or directory
fopen failed to open: fs_log.txt.26676
fs-mark pass # 5 failed
I am working on reporting so look for a daily status report email from my
jenkins server soon. How do we want to handle failures like this moving
forward? Should I just open a BZ after I triage? Do you guys do a new BZ for
every failure in the normal regressions tests?
Yes bz would be great with all the logs. For spurious regressions at
least I just opened one bz and fixed all the bugs reported by Justin
against that one.
Ben,
Did you get a chance to raise the bug?
Pranith
Pranith
-b
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-devel
