Re: [lustre-discuss] Lustre 2.10.0 mmap() Issues

[Christopher Johnston]

Sure can Peter, will do that later this morning.

On Thu, Aug 10, 2017 at 8:58 AM, Jones, Peter A 
wrote:

> Christopher
>
> Could you please open a JIRA ticket about this?
>
> Thanks
>
> Peter
>
> On 8/8/17, 8:58 AM, "lustre-discuss on behalf of Christopher Johnston" <
> lustre-discuss-boun...@lists.lustre.org on behalf of chjoh...@gmail.com>
> wrote:
>
> At my company we use mmap() exclusively for accessing our data on Lustre.
> For starters we are seeing some very weird (or maybe expected) poor random
> read/write performance for these types of access patterns.  I decided to
> give Lustre 2.10.0 a try with ZFS 0.7.0 as the backend instead of ldisk and
> after compiling and building the RPMs, the filesystem mounted up just
> fine.  I then started doing some iozone runs to test the stability of the
> filesystem and although izone does complete uts benchmark I am seeing a lot
> of stack traces coming out of various kernel threads.  Note I am only
> seeing this when using mmap().  We also ran our application as well just to
> verify.  I am also going to try with an ldiskfs format as well to see if
> this changes anything.
>
> My ZFS settings are modest, with 50% of memory allocated to the ARC:
>
> options zfs zfs_arc_max=3921674240 zfs_prefetch_disable=1
> recordsize=1M
> compression=on
> dedupe=off
> xattr=sa
> dnodesize=auto
>
>
> Below is the output from the stack trace:
>
> Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state: 87 messages
> suppressed
> Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state in process
> socknal_sd00_01  pfn:1cbac1
> Aug  8 09:38:04 dev-gc01-oss001 kernel: page:ea00072eb040 count:0
> mapcount:-1 mapping:  (null) index:0x0
> Aug  8 09:38:04 dev-gc01-oss001 kernel: page flags: 0x2f8000(tail)
> Aug  8 09:38:04 dev-gc01-oss001 kernel: page dumped because: nonzero
> mapcount
> Aug  8 09:38:04 dev-gc01-oss001 kernel: Modules linked in: 8021q garp mrp
> stp llc osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE)
> fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE)
> iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
> glue_helper ablk_helper cryptd ppdev sg i2c_piix4 parport_pc i2c_core
> parport pcspkr nfsd nfs_acl lockd grace binfmt_misc auth_rpcgss sunrpc
> ip_tables xfs libcrc32c zfs(POE) zunicode(POE) zavl(POE) icp(POE)
> zcommon(POE) znvpair(POE) spl(OE) zlib_deflate sd_mod crc_t10dif
> crct10dif_generic virtio_net virtio_scsi crct10dif_pclmul crct10dif_common
> crc32c_intel serio_raw virtio_pci virtio_ring virtio
> Aug  8 09:38:04 dev-gc01-oss001 kernel: CPU: 0 PID: 2558 Comm:
> socknal_sd00_01 Tainted: PB  OE  
> 3.10.0-514.26.2.el7.x86_64 #1
> Aug  8 09:38:04 dev-gc01-oss001 kernel: Hardware name: Google Google
> Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Aug  8 09:38:04 dev-gc01-oss001 kernel: ea00072eb040 005e265e
> 8800b879f5a8 81687133
> Aug  8 09:38:04 dev-gc01-oss001 kernel: 8800b879f5d0 81682368
> ea00072eb040 
> Aug  8 09:38:04 dev-gc01-oss001 kernel: 000f 8800b879f618
> 8118946d fff0fe00
> Aug  8 09:38:04 dev-gc01-oss001 kernel: Call Trace:
> Aug  8 09:38:04 dev-gc01-oss001 kernel: []
> dump_stack+0x19/0x1b
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> bad_page.part.75+0xdf/0xfc
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> free_pages_prepare+0x16d/0x190
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> __free_pages_ok+0x19/0xd0
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> free_compound_page+0x1b/0x20
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> __put_compound_page+0x1f/0x22
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> put_compound_page+0x16f/0x17d
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> put_page+0x4c/0x60
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> skb_release_data+0x8f/0x140
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> skb_release_all+0x24/0x30
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> consume_skb+0x2c/0x80
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> __dev_kfree_skb_any+0x3d/0x50
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> free_old_xmit_skbs.isra.32+0x6b/0xc0 [virtio_net]
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> start_xmit+0x5f/0x4f0 [virtio_net]
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> dev_hard_start_xmit+0x171/0x3b0
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> sch_direct_xmit+0x104/0x200
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> __dev_queue_xmit+0x23c/0x570
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> dev_queue_xmit+0x10/0x20
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> ip_finish_output+0x466/0x750
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> ip_output+0x73/0xe0
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> ip_local_out_sk+0x31/0x40
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> ip_queue_xmit+0x143/0x3a0
> Aug  8 09:38:05 dev-gc01-oss001 kernel: []
> 

Re: [lustre-discuss] Lustre 2.10.0 mmap() Issues

[Jones, Peter A]

Christopher

Could you please open a JIRA ticket about this?

Thanks

Peter

On 8/8/17, 8:58 AM, "lustre-discuss on behalf of Christopher Johnston" 

 on behalf of chjoh...@gmail.com> wrote:

At my company we use mmap() exclusively for accessing our data on Lustre.  For 
starters we are seeing some very weird (or maybe expected) poor random 
read/write performance for these types of access patterns.  I decided to give 
Lustre 2.10.0 a try with ZFS 0.7.0 as the backend instead of ldisk and after 
compiling and building the RPMs, the filesystem mounted up just fine.  I then 
started doing some iozone runs to test the stability of the filesystem and 
although izone does complete uts benchmark I am seeing a lot of stack traces 
coming out of various kernel threads.  Note I am only seeing this when using 
mmap().  We also ran our application as well just to verify.  I am also going 
to try with an ldiskfs format as well to see if this changes anything.

My ZFS settings are modest, with 50% of memory allocated to the ARC:

options zfs zfs_arc_max=3921674240 zfs_prefetch_disable=1
recordsize=1M
compression=on
dedupe=off
xattr=sa
dnodesize=auto


Below is the output from the stack trace:

Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state: 87 messages 
suppressed
Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state in process 
socknal_sd00_01  pfn:1cbac1
Aug  8 09:38:04 dev-gc01-oss001 kernel: page:ea00072eb040 count:0 
mapcount:-1 mapping:  (null) index:0x0
Aug  8 09:38:04 dev-gc01-oss001 kernel: page flags: 0x2f8000(tail)
Aug  8 09:38:04 dev-gc01-oss001 kernel: page dumped because: nonzero mapcount
Aug  8 09:38:04 dev-gc01-oss001 kernel: Modules linked in: 8021q garp mrp stp 
llc osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) 
fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) iosf_mbi 
crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper 
ablk_helper cryptd ppdev sg i2c_piix4 parport_pc i2c_core parport pcspkr nfsd 
nfs_acl lockd grace binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c 
zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) 
zlib_deflate sd_mod crc_t10dif crct10dif_generic virtio_net virtio_scsi 
crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring 
virtio
Aug  8 09:38:04 dev-gc01-oss001 kernel: CPU: 0 PID: 2558 Comm: socknal_sd00_01 
Tainted: PB  OE     3.10.0-514.26.2.el7.x86_64 #1
Aug  8 09:38:04 dev-gc01-oss001 kernel: Hardware name: Google Google Compute 
Engine/Google Compute Engine, BIOS Google 01/01/2011
Aug  8 09:38:04 dev-gc01-oss001 kernel: ea00072eb040 005e265e 
8800b879f5a8 81687133
Aug  8 09:38:04 dev-gc01-oss001 kernel: 8800b879f5d0 81682368 
ea00072eb040 
Aug  8 09:38:04 dev-gc01-oss001 kernel: 000f 8800b879f618 
8118946d fff0fe00
Aug  8 09:38:04 dev-gc01-oss001 kernel: Call Trace:
Aug  8 09:38:04 dev-gc01-oss001 kernel: [] 
dump_stack+0x19/0x1b
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
bad_page.part.75+0xdf/0xfc
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
free_pages_prepare+0x16d/0x190
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
__free_pages_ok+0x19/0xd0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
free_compound_page+0x1b/0x20
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
__put_compound_page+0x1f/0x22
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
put_compound_page+0x16f/0x17d
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] put_page+0x4c/0x60
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
skb_release_data+0x8f/0x140
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
skb_release_all+0x24/0x30
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
consume_skb+0x2c/0x80
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
__dev_kfree_skb_any+0x3d/0x50
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
free_old_xmit_skbs.isra.32+0x6b/0xc0 [virtio_net]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
start_xmit+0x5f/0x4f0 [virtio_net]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
dev_hard_start_xmit+0x171/0x3b0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
sch_direct_xmit+0x104/0x200
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
__dev_queue_xmit+0x23c/0x570
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
dev_queue_xmit+0x10/0x20
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
ip_finish_output+0x466/0x750
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] ip_output+0x73/0xe0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
ip_local_out_sk+0x31/0x40
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
ip_queue_xmit+0x143/0x3a0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
tcp_transmit_skb+0x4af/0x990
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
tcp_write_xmit+0x15a/0xce0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] 
__tcp_push_pending_frames+0x2e/0xc0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [] tcp_push+0xec/0x120