How about this as a band-aid, since non-chunked readdir is going away:
https://review.gerrithub.io/396194
Daniel
On 01/24/2018 05:29 AM, Kinglong Mee wrote:
With the latest codes (nfs-ganesha-2.6.0-0.2rc3.el7.centos.x86_64),
set ganesha.conf includes,
CACHEINODE {
Dir_Max = 50;
Dir_Chunk = 0;
}
that forces mdcache_readdir enter mdcache_dirent_populate, but mdc_add_cache
return ERR_FSAL_OVERFLOW when a directory contains larger than 50 files
(I tests with 100 files).
After return ERR_FSAL_OVERFLOW, a handle is left in gluster fsal,
but no entry added to mdcache.
Right now, restarting nfs-ganesha will crash as,
#0 0x00007f75bb9ea6e7 in __inode_unref () from /lib64/libglusterfs.so.0
#1 0x00007f75bb9eaf41 in inode_unref () from /lib64/libglusterfs.so.0
#2 0x00007f75bbccdad6 in glfs_h_close () from /lib64/libgfapi.so.0
#3 0x00007f75bc0eb93f in handle_release ()
from /usr/lib64/ganesha/libfsalgluster.so
#4 0x00007f75c12c6faf in destroy_fsals ()
#5 0x00007f75c12e158f in admin_thread ()
#6 0x00007f75bf849dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f75bef1b73d in clone () from /lib64/libc.so.6
For gluster, those sources (eg, handles, inode) belong to a glfs,
the glfs is freed at remove_all_exports() before destory_fsal(),
so that, glfs_h_close() use after freed memory.
I have no idea how to fix the problem,
1. Adds the gluster handle (meets the ERR_FSAL_OVERFLOW) to mdcache entry ?
2. Binds gluster handles to a glfs export, and release before freeing glfs?
3. Only move the shutdown_handles() before remove_all_exports()?
valgrind shows many messages as,
==13796== Invalid read of size 8
==13796== at 0xA5E39C2: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA5E5D58: inode_forget (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA3A0ACD: glfs_h_close (in /usr/lib64/libgfapi.so.0.0.0)
==13796== by 0x9F6B93E: ??? (in /usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796== by 0x147FAE: destroy_fsals (in /usr/bin/ganesha.nfsd)
==13796== by 0x16258E: admin_thread (in /usr/bin/ganesha.nfsd)
==13796== by 0x6441DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==13796== by 0x6DAA73C: clone (in /usr/lib64/libc-2.17.so)
==13796== Address 0x1d3b6d58 is 104 bytes inside a block of size 256 free'd
==13796== at 0x4C28CDD: free (vg_replace_malloc.c:530)
==13796== by 0xA5FD9BB: free_obj_list (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA5FDDA7: mem_pools_fini (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA38A2D6: glfs_fini (in /usr/lib64/libgfapi.so.0.0.0)
==13796== by 0x9F6687A: glusterfs_free_fs (in
/usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796== by 0x9F66C19: ??? (in /usr/lib64/ganesha/libfsalgluster.so.4.2.0)
==13796== by 0x223E1E: ??? (in /usr/bin/ganesha.nfsd)
==13796== by 0x200FFA: free_export_resources (in /usr/bin/ganesha.nfsd)
==13796== by 0x212288: free_export (in /usr/bin/ganesha.nfsd)
==13796== by 0x215329: remove_all_exports (in /usr/bin/ganesha.nfsd)
==13796== by 0x1624B8: admin_thread (in /usr/bin/ganesha.nfsd)
==13796== by 0x6441DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==13796== Block was alloc'd at
==13796== at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==13796== by 0xA5FE3C4: mem_get (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA5FE4D2: mem_get0 (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA5E3BD3: ??? (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0xA5E510A: inode_new (in /usr/lib64/libglusterfs.so.0.0.1)
==13796== by 0x178B5779: ???
==13796== by 0x178EA503: ???
==13796== by 0x178C3E46: ???
==13796== by 0xA8B8E7F: rpc_clnt_handle_reply (in
/usr/lib64/libgfrpc.so.0.0.1)
==13796== by 0xA8B9202: rpc_clnt_notify (in /usr/lib64/libgfrpc.so.0.0.1)
==13796== by 0xA8B4F82: rpc_transport_notify (in
/usr/lib64/libgfrpc.so.0.0.1)
==13796== by 0x1741D595: ??? (in
/usr/lib64/glusterfs/3.13.1/rpc-transport/socket.so)
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
