On April 7, 2020 4:59:03 PM GMT+03:00, Gianluca Cecchi
<gianluca.cec...@gmail.com> wrote:
>OK. So I set log at least at INFO level on all subsystems and tried a
>redeploy of Openshift with 3 mater nodes and 7 worker nodes.
>One worker got the error and VM in paused mode
>
>Apr 7, 2020, 3:27:28 PM VM worker-6 has been paused due to unknown
>storage
>error.
>
>The vm has only one 100Gb virtual disk on gluster volume named vmstore
>
>
>Here below all the logs around time at the different layers.
>Let me know if you need another log file not yet considered.
>
>From what I see, the matching error is found in
>
>- rhev-data-center-mnt-glusterSD-ovirtst.mydomain.storage:_vmstore.log
>
>[2020-04-07 13:27:28.721262] E [MSGID: 133010]
>[shard.c:2327:shard_common_lookup_shards_cbk] 0-vmstore-shard: Lookup
>on
>shard 523 failed. Base file gfid = d22530cf-2e50-4059-8924-0aafe38497b1
>[No
>such file or directory]
>[2020-04-07 13:27:28.721432] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4435189: WRITE => -1
>gfid=d22530cf-2e50-4059-8924-0aafe38497b1 fd=0x7f3c4c07ab38 (No such
>file
>or directory)
>
>and
>
>- gluster_bricks-vmstore-vmstore.log
>
>[2020-04-07 13:27:28.719391] W [MSGID: 113020]
>[posix-helpers.c:1051:posix_gfid_set] 0-vmstore-posix: setting GFID on
>/gluster_bricks/vmstore
>/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File
>exists]
>[2020-04-07 13:27:28.719978] E [MSGID: 113020]
>[posix-entry-ops.c:517:posix_mknod] 0-vmstore-posix: setting gfid on
>/gluster_bricks/vmstore/v
>mstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File
>exists]
>
>
>Here below all the files checked.
>Any hint?
>
>Gianluca
>
>- qemu.log of the vm
>
>2020-04-07T12:30:29.954084Z qemu-kvm: -drive
>file=/rhev/data-center/mnt/glusterSD/ovirtst.mydomain.storage:_vmstore/81b97244-4b69-4d49-84c4-c822387adc6a/images/04ec5a8d-4ee6-4661-a832-0c094968aa5b/716fdc98-2d0e-44c3-b5fe-c0187cdad751,format=raw,if=none,id=drive-ua-04ec5a8d-4ee6-4661-a832-0c094968aa5b,serial=04ec5a8d-4ee6-4661-a832-0c094968aa5b,werror=stop,rerror=stop,cache=none,aio=threads:
>'serial' is deprecated, please use the corresponding option of
>'-device'
>instead
>Spice-Message: 14:30:29.963: setting TLS option 'CipherString' to
>'kECDHE+FIPS:kDHE+FIPS:kRSA+FIPS:!eNULL:!aNULL' from
>/etc/pki/tls/spice.cnf
>configuration file
>2020-04-07T12:30:29.976216Z qemu-kvm: warning: CPU(s) not present in
>any
>NUMA nodes: CPU 8 [socket-id: 2, core-id: 0, thread-id: 0], CPU 9
>[socket-id: 2, core-id: 1, thread-id: 0], CPU 10 [socket-id: 2,
>core-id: 2,
>thread-id: 0], CPU 11 [socket-id: 2, core-id: 3, thread-id: 0], CPU 12
>[socket-id: 3, core-id: 0, thread-id: 0], CPU 13 [socket-id: 3,
>core-id: 1,
>thread-id: 0], CPU 14 [socket-id: 3, core-id: 2, thread-id: 0], CPU 15
>[socket-id: 3, core-id: 3, thread-id: 0], CPU 16 [socket-id: 4,
>core-id: 0,
>thread-id: 0], CPU 17 [socket-id: 4, core-id: 1, thread-id: 0], CPU 18
>[socket-id: 4, core-id: 2, thread-id: 0], CPU 19 [socket-id: 4,
>core-id: 3,
>thread-id: 0], CPU 20 [socket-id: 5, core-id: 0, thread-id: 0], CPU 21
>[socket-id: 5, core-id: 1, thread-id: 0], CPU 22 [socket-id: 5,
>core-id: 2,
>thread-id: 0], CPU 23 [socket-id: 5, core-id: 3, thread-id: 0], CPU 24
>[socket-id: 6, core-id: 0, thread-id: 0], CPU 25 [socket-id: 6,
>core-id: 1,
>thread-id: 0], CPU 26 [socket-id: 6, core-id: 2, thread-id: 0], CPU 27
>[socket-id: 6, core-id: 3, thread-id: 0], CPU 28 [socket-id: 7,
>core-id: 0,
>thread-id: 0], CPU 29 [socket-id: 7, core-id: 1, thread-id: 0], CPU 30
>[socket-id: 7, core-id: 2, thread-id: 0], CPU 31 [socket-id: 7,
>core-id: 3,
>thread-id: 0], CPU 32 [socket-id: 8, core-id: 0, thread-id: 0], CPU 33
>[socket-id: 8, core-id: 1, thread-id: 0], CPU 34 [socket-id: 8,
>core-id: 2,
>thread-id: 0], CPU 35 [socket-id: 8, core-id: 3, thread-id: 0], CPU 36
>[socket-id: 9, core-id: 0, thread-id: 0], CPU 37 [socket-id: 9,
>core-id: 1,
>thread-id: 0], CPU 38 [socket-id: 9, core-id: 2, thread-id: 0], CPU 39
>[socket-id: 9, core-id: 3, thread-id: 0], CPU 40 [socket-id: 10,
>core-id:
>0, thread-id: 0], CPU 41 [socket-id: 10, core-id: 1, thread-id: 0], CPU
>42
>[socket-id: 10, core-id: 2, thread-id: 0], CPU 43 [socket-id: 10,
>core-id:
>3, thread-id: 0], CPU 44 [socket-id: 11, core-id: 0, thread-id: 0], CPU
>45
>[socket-id: 11, core-id: 1, thread-id: 0], CPU 46 [socket-id: 11,
>core-id:
>2, thread-id: 0], CPU 47 [socket-id: 11, core-id: 3, thread-id: 0], CPU
>48
>[socket-id: 12, core-id: 0, thread-id: 0], CPU 49 [socket-id: 12,
>core-id:
>1, thread-id: 0], CPU 50 [socket-id: 12, core-id: 2, thread-id: 0], CPU
>51
>[socket-id: 12, core-id: 3, thread-id: 0], CPU 52 [socket-id: 13,
>core-id:
>0, thread-id: 0], CPU 53 [socket-id: 13, core-id: 1, thread-id: 0], CPU
>54
>[socket-id: 13, core-id: 2, thread-id: 0], CPU 55 [socket-id: 13,
>core-id:
>3, thread-id: 0], CPU 56 [socket-id: 14, core-id: 0, thread-id: 0], CPU
>57
>[socket-id: 14, core-id: 1, thread-id: 0], CPU 58 [socket-id: 14,
>core-id:
>2, thread-id: 0], CPU 59 [socket-id: 14, core-id: 3, thread-id: 0], CPU
>60
>[socket-id: 15, core-id: 0, thread-id: 0], CPU 61 [socket-id: 15,
>core-id:
>1, thread-id: 0], CPU 62 [socket-id: 15, core-id: 2, thread-id: 0], CPU
>63
>[socket-id: 15, core-id: 3, thread-id: 0]
>2020-04-07T12:30:29.976247Z qemu-kvm: warning: All CPU(s) up to maxcpus
>should be described in NUMA config, ability to start up with partial
>NUMA
>mappings is obsoleted and will be removed in future
>main_channel_link: add main channel client
>main_channel_client_handle_pong: net test: latency 52.540000 ms,
>bitrate
>6113597 bps (5.830380 Mbps) LOW BANDWIDTH
>inputs_connect: inputs channel client create
>red_qxl_set_cursor_peer:
>inputs_channel_detach_tablet:
>red_channel_client_disconnect: rcc=0x5626c717a9c0
>(channel=0x5626c616a880
>type=3 id=0)
>red_channel_client_disconnect: rcc=0x5626c8351dd0
>(channel=0x5626c7bd38d0
>type=4 id=0)
>red_channel_client_disconnect: rcc=0x5626c834c5b0
>(channel=0x5626c616a230
>type=2 id=0)
>red_channel_client_disconnect: rcc=0x5626c719e9f0
>(channel=0x5626c616a160
>type=1 id=0)
>main_channel_client_on_disconnect: rcc=0x5626c719e9f0
>red_client_destroy: destroy client 0x5626c6195570 with #channels=4
>red_qxl_disconnect_cursor_peer:
>red_qxl_disconnect_display_peer:
>inputs_channel_detach_tablet:
>inputs_channel_detach_tablet:
>main_channel_link: add main channel client
>main_channel_client_handle_pong: net test: latency 126.308000 ms,
>bitrate
>9060022 bps (8.640310 Mbps) LOW BANDWIDTH
>red_qxl_set_cursor_peer:
>inputs_connect: inputs channel client create
>red_channel_client_disconnect: rcc=0x5626c717a9f0
>(channel=0x5626c616a160
>type=1 id=0)
>main_channel_client_on_disconnect: rcc=0x5626c717a9f0
>red_client_destroy: destroy client 0x5626c7ab6870 with #channels=4
>red_channel_client_disconnect: rcc=0x5626c71529c0
>(channel=0x5626c616a880
>type=3 id=0)
>red_qxl_disconnect_cursor_peer:
>red_channel_client_disconnect: rcc=0x5626c7e59dd0
>(channel=0x5626c7bd38d0
>type=4 id=0)
>red_qxl_disconnect_display_peer:
>red_channel_client_disconnect: rcc=0x5626c7e5e5b0
>(channel=0x5626c616a230
>type=2 id=0)
>[root@ovirt qemu]#
>
>
>- /var/log/glusterfs/cli.log
>
>[2020-04-07 13:25:39.305818] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:39.306091] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:39.306531] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:39.306758] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:39.307081] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:25:54.365843] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:25:54.448475] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:25:54.448600] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:25:54.449348] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:25:54.485288] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:25:54.566723] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:25:54.566792] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:25:54.567275] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:25:54.567305] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:25:54.603618] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:25:54.685718] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:25:54.685821] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:25:54.686537] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:54.686994] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:54.687289] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:54.687774] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:54.688061] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:25:54.688368] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:09.748782] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:09.831541] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:09.831610] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:09.832596] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:09.871269] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:09.955626] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:09.955726] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:09.956239] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:26:09.956284] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:09.993929] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:10.073634] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:10.073680] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:10.074454] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:10.074932] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:10.075241] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:10.075677] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:10.075920] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:10.076233] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:22.933923] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:23.016382] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:23.016486] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:23.020067] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:25.149347] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:25.229667] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:25.229716] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:25.230491] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:25.266853] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:25.348774] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:25.348815] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:25.349253] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:26:25.349284] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:25.385951] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:25.468521] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:25.468571] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:25.469286] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:25.469703] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:25.469965] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:25.470381] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:25.470595] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:25.470877] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:40.546233] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:40.629567] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:40.629667] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:40.630529] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:40.666769] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:40.746861] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:40.746934] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:40.747436] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:26:40.747474] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:40.789486] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:40.881750] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:40.881871] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:40.882615] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:40.883142] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:40.883532] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:40.884020] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:40.884283] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:40.884615] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:55.966643] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:56.049639] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:56.049735] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:56.050543] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:56.096892] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:56.182275] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:56.182335] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:56.182883] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:26:56.182918] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:26:56.221093] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:26:56.303302] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:26:56.303337] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:26:56.303996] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:56.304432] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:56.304706] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:56.305135] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:56.305359] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:26:56.305645] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:11.378699] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:11.457630] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:11.457697] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:11.458355] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:11.493643] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:11.571358] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:11.571360] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:11.572137] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:27:11.572167] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:11.608432] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:11.689353] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:11.689437] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:11.690117] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:11.690569] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:11.690910] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:11.691343] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:11.691577] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:11.691853] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:23.058796] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:23.141280] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:23.141326] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:23.144418] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:26.767415] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:26.846945] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:26.847030] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:26.847818] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:26.883554] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:26.962804] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:26.962905] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:26.963380] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:27:26.963415] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:27.000197] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:27.077981] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:27.078029] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:27.078729] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:27.079174] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:27.079448] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:27.079873] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:27.080105] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:27.080429] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:42.152150] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:42.233109] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:42.233149] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:42.233818] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:42.269022] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:42.347310] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:42.347347] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:42.347771] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:27:42.347798] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:42.383260] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:42.460650] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:42.460741] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:42.461348] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:42.461780] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:42.462085] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:42.462493] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:42.462711] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:42.462990] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:57.534379] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:57.614068] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:57.614115] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:57.614835] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:57.650436] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:57.729556] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:57.729652] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:57.730164] I
>[cli-rpc-ops.c:1297:gf_cli3_1_uuid_get_cbk]
>0-cli: Received resp to uuid get
>[2020-04-07 13:27:57.730204] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:27:57.768743] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:27:57.848062] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:27:57.848153] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:27:57.848830] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:57.849277] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:57.849554] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:57.849983] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:57.850217] I
>[cli-rpc-ops.c:867:gf_cli_get_volume_cbk]
>0-cli: Received resp to get vol: 0
>[2020-04-07 13:27:57.850522] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:28:12.934596] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>[2020-04-07 13:28:13.015126] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 0
>[2020-04-07 13:28:13.015167] I [MSGID: 101190]
>[event-epoll.c:688:event_dispatch_epoll_worker] 0-epoll: Started thread
>with index 1
>[2020-04-07 13:28:13.015948] I [input.c:31:cli_batch] 0-: Exiting with:
>0
>[2020-04-07 13:28:13.054894] I [cli.c:844:main] 0-cli: Started running
>/usr/sbin/gluster with version 6.8
>
>
>- glusterd.log
>
>[2020-04-07 13:25:22.884457] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume vmstore
>The message "I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req" repeated 7 times between [2020-04-07
>13:24:06.750320] and [2020-04-07 13:25:54.449004]
>The message "I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req" repeated 39 times between [2020-04-07
>13:24:06.989378] and [2020-04-07 13:25:54.688025]
>[2020-04-07 13:26:09.832058] I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req
>[2020-04-07 13:26:10.074287] I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req
>[2020-04-07 13:26:23.017801] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume big
>[2020-04-07 13:26:23.018753] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume data
>[2020-04-07 13:26:23.019190] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume engine
>[2020-04-07 13:26:23.019628] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume vmstore
>[2020-04-07 13:27:23.143376] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume data
>[2020-04-07 13:27:23.143718] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume engine
>[2020-04-07 13:27:23.142636] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume big
>[2020-04-07 13:27:23.144070] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume vmstore
>The message "I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req" repeated 7 times between [2020-04-07
>13:26:09.832058] and [2020-04-07 13:27:57.614518]
>The message "I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req" repeated 39 times between [2020-04-07
>13:26:10.074287] and [2020-04-07 13:27:57.850194]
>[2020-04-07 13:28:13.015558] I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req
>[2020-04-07 13:28:13.256713] I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req
>[2020-04-07 13:28:23.264138] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume big
>[2020-04-07 13:28:23.264884] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume data
>[2020-04-07 13:28:23.265316] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume engine
>[2020-04-07 13:28:23.265644] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume vmstore
>[2020-04-07 13:29:23.389341] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume data
>[2020-04-07 13:29:23.389702] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume engine
>[2020-04-07 13:29:23.388761] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume big
>[2020-04-07 13:29:23.390058] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume vmstore
>The message "I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req" repeated 6 times between [2020-04-07
>13:28:13.015558] and [2020-04-07 13:29:45.291552]
>The message "I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req" repeated 34 times between [2020-04-07
>13:28:13.256713] and [2020-04-07 13:29:45.524677]
>[2020-04-07 13:30:00.664512] I [MSGID: 106487]
>[glusterd-handler.c:1516:__glusterd_handle_cli_list_friends]
>0-glusterd:
>Received cli list req
>[2020-04-07 13:30:00.899102] I [MSGID: 106488]
>[glusterd-handler.c:1577:__glusterd_handle_cli_get_volume]
>0-management:
>Received get vol req
>[2020-04-07 13:30:23.505881] I [MSGID: 106499]
>[glusterd-handler.c:4429:__glusterd_handle_status_volume] 0-management:
>Received status volume req for volume big
>
>- events.log
>
>[2020-04-07 15:21:30,978] INFO [utils - 66:log_event] - {'event':
>'CLIENT_CONNECT', 'message': {'client_identifier': '10.50.50.11:49135',
>'server_identifier': '10.50.50.11:49153', 'client_uid':
>'CTX_ID:815a9519-d88e-402d-9473-19015cca1db8-GRAPH_ID:0-PID:131714-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0',
>'brick_path': '/gluster_bricks/engine/engine', 'subdir_mount':
>'(null)'},
>'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586265690}
>[2020-04-07 15:21:30,980] INFO [utils - 66:log_event] - {'event':
>'CLIENT_DISCONNECT', 'message': {'client_identifier':
>'10.50.50.11:49135',
>'server_identifier': '10.50.50.11:49153', 'client_uid':
>'CTX_ID:815a9519-d88e-402d-9473-19015cca1db8-GRAPH_ID:0-PID:131714-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0',
>'brick_path': '/gluster_bricks/engine/engine'}, 'nodeid':
>'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586265690}
>[2020-04-07 15:36:33,598] INFO [utils - 66:log_event] - {'event':
>'CLIENT_CONNECT', 'message': {'client_identifier': '10.50.50.11:49144',
>'server_identifier': '10.50.50.11:49155', 'client_uid':
>'CTX_ID:e1d09577-50d6-4679-b0e8-0fe554f9fd49-GRAPH_ID:0-PID:139160-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0',
>'brick_path': '/gluster_bricks/big/big', 'subdir_mount': '(null)'},
>'nodeid': 'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586266593}
>[2020-04-07 15:36:33,599] INFO [utils - 66:log_event] - {'event':
>'CLIENT_DISCONNECT', 'message': {'client_identifier':
>'10.50.50.11:49144',
>'server_identifier': '10.50.50.11:49155', 'client_uid':
>'CTX_ID:e1d09577-50d6-4679-b0e8-0fe554f9fd49-GRAPH_ID:0-PID:139160-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0',
>'brick_path': '/gluster_bricks/big/big'}, 'nodeid':
>'ea83cff6-4cc6-4399-a096-5892476721f0', 'ts': 1586266593}
>
>
>- rhev-data-center-mnt-glusterSD-ovirtst.mydomain.storage:_vmstore.log
>
>[2020-04-07 13:24:16.070237] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4197133: WRITE => -1
>gfid=aa68cd63-f3d3-47bd-a177-c7f48bef3c59 fd=0x7f3c5019f348 (Invalid
>argument)
>[2020-04-07 13:24:16.074521] I [MSGID: 133022]
>[shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of
>gfid=aa68cd63-f3d3-47bd-a177-c7f48bef3c59 from backend
>[2020-04-07 13:27:28.721262] E [MSGID: 133010]
>[shard.c:2327:shard_common_lookup_shards_cbk] 0-vmstore-shard: Lookup
>on
>shard 523 failed. Base file gfid = d22530cf-2e50-4059-8924-0aafe38497b1
>[No
>such file or directory]
>[2020-04-07 13:27:28.721432] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4435189: WRITE => -1
>gfid=d22530cf-2e50-4059-8924-0aafe38497b1 fd=0x7f3c4c07ab38 (No such
>file
>or directory)
>[2020-04-07 13:29:16.232630] W [MSGID: 114031]
>[client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0:
>remote
>operation failed [Invalid argument]
>[2020-04-07 13:29:16.232675] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4496167: WRITE => -1
>gfid=54bafbb5-edd6-42a0-aaba-a5295031be17 fd=0x7f3c48125838 (Invalid
>argument)
>[2020-04-07 13:29:16.236352] I [MSGID: 133022]
>[shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of
>gfid=54bafbb5-edd6-42a0-aaba-a5295031be17 from backend
>[2020-04-07 13:34:16.378326] W [MSGID: 114031]
>[client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0:
>remote
>operation failed [Invalid argument]
>[2020-04-07 13:34:16.378461] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4698415: WRITE => -1
>gfid=6beaa642-e73e-40c7-8adc-64275528b44f fd=0x7f3c48138cc8 (Invalid
>argument)
>[2020-04-07 13:34:16.382824] I [MSGID: 133022]
>[shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of
>gfid=6beaa642-e73e-40c7-8adc-64275528b44f from backend
>[2020-04-07 13:39:16.546919] W [MSGID: 114031]
>[client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-vmstore-client-0:
>remote
>operation failed [Invalid argument]
>[2020-04-07 13:39:16.547022] W [fuse-bridge.c:2918:fuse_writev_cbk]
>0-glusterfs-fuse: 4884227: WRITE => -1
>gfid=f32843b7-4131-4536-abe2-f6969aa455ae fd=0x7f3c4813fb58 (Invalid
>argument)
>[2020-04-07 13:39:16.550630] I [MSGID: 133022]
>[shard.c:3674:shard_delete_shards] 0-vmstore-shard: Deleted shards of
>gfid=f32843b7-4131-4536-abe2-f6969aa455ae from backend
>
>
>- gluster_bricks-vmstore-vmstore.log
>
>[2020-04-07 13:21:28.128381] I [MSGID: 115036]
>[server.c:499:server_rpc_notify] 0-vmstore-server: disconnecting
>connection
>from CTX_ID:12a5f3
>7d-94d3-4b22-aaf6-4ac5bb7567d6-GRAPH_ID:0-PID:131656-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0
>[2020-04-07 13:21:28.128565] I [MSGID: 101055]
>[client_t.c:436:gf_client_unref] 0-vmstore-server: Shutting down
>connection
>CTX_ID:12a5f37d-94
>d3-4b22-aaf6-4ac5bb7567d6-GRAPH_ID:0-PID:131656-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0
>[2020-04-07 13:24:16.069408] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed:
>offset 0, [Invalid arg
>ument]
>[2020-04-07 13:24:16.070018] E [MSGID: 115067]
>[server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-vmstore-server:
>3528796:
>WRITEV 18 (aa68cd63-
>f3d3-47bd-a177-c7f48bef3c59), client:
>CTX_ID:20665547-59fd-4b4a-94fb-7fdfdca9f5a9-GRAPH_ID:0-PID:17102-HOST:ovirt.mydomain.local-PC_NAME:vmsto
>re-client-0-RECON_NO:-0, error-xlator: vmstore-posix [Invalid argument]
>[2020-04-07 13:27:28.719391] W [MSGID: 113020]
>[posix-helpers.c:1051:posix_gfid_set] 0-vmstore-posix: setting GFID on
>/gluster_bricks/vmstore
>/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File
>exists]
>[2020-04-07 13:27:28.719978] E [MSGID: 113020]
>[posix-entry-ops.c:517:posix_mknod] 0-vmstore-posix: setting gfid on
>/gluster_bricks/vmstore/v
>mstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523 failed [File
>exists]
>[2020-04-07 13:29:16.232263] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed:
>offset 0, [Invalid argument]
>[2020-04-07 13:29:16.232578] E [MSGID: 115067]
>[server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-vmstore-server:
>3777485:
>WRITEV 12 (54bafbb5-edd6-42a0-aaba-a5295031be17), client:
>CTX_ID:20665547-59fd-4b4a-94fb-7fdfdca9f5a9-GRAPH_ID:0-PID:17102-HOST:ovirt.mydomain.local-PC_NAME:vmstore-client-0-RECON_NO:-0,
>error-xlator: vmstore-posix [Invalid argument]
>[2020-04-07 13:34:16.377622] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1886:posix_writev] 0-vmstore-posix: write failed:
>offset 0, [Invalid argument]
>
>- messages
>
>Apr 7 15:25:26 ovirt gluster_bricks-big-big[39632]: [2020-04-07
>13:25:26.789943] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1886:posix_writev]
>0-big-posix: write failed: offset 0, [Invalid argument]
>Apr 7 15:25:26 ovirt gluster_bricks-big-big[39632]: [2020-04-07
>13:25:26.790284] E [MSGID: 115067]
>[server-rpc-fops_v2.c:1373:server4_writev
>_cbk] 0-big-server: 20535: WRITEV 1
>(e965b2e9-500b-4520-a17e-d00b93a2e1bf),
>client: CTX_ID:2530ad70-1cec-43a2-bdf8-8c5a085fecb4-GRAPH_ID:0-PI
>D:41137-HOST:ovirt.mydomain.local-PC_NAME:big-client-0-RECON_NO:-0,
>error-xlator: big-posix [Invalid argument]
>Apr 7 15:26:16 ovirt gluster_bricks-engine-engine[10652]: [2020-04-07
>13:26:16.183229] E [MSGID: 113072] [posix-inode-fd-ops.c:1886:posix_wr
>itev] 0-engine-posix: write failed: offset 0, [Invalid argument]
>Apr 7 15:26:16 ovirt gluster_bricks-engine-engine[10652]: [2020-04-07
>13:26:16.183318] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_
>writev_cbk] 0-engine-server: 359871: WRITEV 5
>(cb0ea391-ed2f-4318-91d3-b8e18f6bdfb7), client:
>CTX_ID:08f296e3-beeb-4658-8e79-d9add812c3d7-GRA
>PH_ID:0-PID:11400-HOST:ovirt.mydomain.local-PC_NAME:engine-client-0-RECON_NO:-0,
>error-xlator: engine-posix [Invalid argument]
>Apr 7 15:27:28 ovirt gluster_bricks-vmstore-vmstore[10678]:
>[2020-04-07
>13:27:28.719391] W [MSGID: 113020]
>[posix-helpers.c:1051:posix_gfid_set]
>0-vmstore-posix: setting GFID on
>/gluster_bricks/vmstore/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523
>failed [File exists]
>Apr 7 15:27:28 ovirt gluster_bricks-vmstore-vmstore[10678]:
>[2020-04-07
>13:27:28.719978] E [MSGID: 113020] [posix-entry-ops.c:517:posix_mknod]
>0-vmstore-posix: setting gfid on
>/gluster_bricks/vmstore/vmstore/.shard/d22530cf-2e50-4059-8924-0aafe38497b1.523
>failed [File exists]
>Apr 7 15:29:16 ovirt gluster_bricks-data-data[10609]: [2020-04-07
>13:29:16.232149] E [MSGID: 113072]
>[posix-inode-fd-ops.c:1886:posix_writev]
>0-data-posix: write failed: offset 0, [Invalid argument]
Hi Gianluca,
The positive thing is that you can reproduce the issue.
I would ask you to check your gluster version and if there are any updates -
update the cluster.
Also check the gluster's op-version, as this limits some of the features.
If there are none - enable trace logs in gluster (
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/configuring_the_log_level
), start volume peofiling, reproduce the issue and then reduce the log level
(it's generating a lot of logs) and stop the profiling.
Once that info is collected, some of the Gluster members can check the
situation.
Best Regards,
Strahil Nikolov
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UZUKK7TUTQTBDD3MZNEG7XSUTP6GEX2L/
