Since last few days I was trying to understand the nightly failures we were seeing even after addressing the port already in use issue. So here's the analysis:
>From console output of https://build.gluster.org/job/regression-test-burn-in/4710/consoleFull <snip> *19:51:56* Started by upstream project "nightly-master <https://build.gluster.org/job/nightly-master/>" build number 843 <https://build.gluster.org/job/nightly-master/843>*19:51:56* originally caused by:*19:51:56* Started by timer*19:51:56* Running as SYSTEM*19:51:57* Building remotely on builder209.aws.gluster.org <https://build.gluster.org/computer/builder209.aws.gluster.org> (centos7) in workspace /home/jenkins/root/workspace/regression-test-burn-in*19:51:58* No credentials specified*19:51:58* > git rev-parse --is-inside-work-tree # timeout=10*19:51:58* Fetching changes from the remote Git repository*19:51:58* > git config remote.origin.url git://review.gluster.org/glusterfs.git # timeout=10*19:51:58* Fetching upstream changes from git://review.gluster.org/glusterfs.git*19:51:58* > git --version # timeout=10*19:51:58* > git fetch --tags --progress git://review.gluster.org/glusterfs.git refs/heads/master # timeout=10*19:52:01* > git rev-parse origin/master^{commit} # timeout=10*19:52:01* Checking out Revision a31fad885c30cbc1bea652349c7d52bac1414c08 (origin/master)*19:52:01* > git config core.sparsecheckout # timeout=10*19:52:01 > git checkout -f a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10 19:52:02 Commit message: "tests: heal-info add --xml option for more coverage"**19:52:02* > git rev-list --no-walk a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10*19:52:02* [regression-test-burn-in] $ /bin/bash /tmp/jenkins7274529097702336737.sh*19:52:02* Start time Mon Aug 26 14:22:02 UTC 2019 </snip> The latest commit which it picked up as part of git checkout is quite old and hence we continue to see the similar failures in the latest nightly runs which has been already addressed by commit c370c70 commit c370c70f77079339e2cfb7f284f3a2fb13fd2f97 Author: Mohit Agrawal <moagra...@redhat.com> Date: Tue Aug 13 18:45:43 2019 +0530 rpc: glusterd start is failed and throwing an error Address already in use Problem: Some of the .t are failed due to bind is throwing an error EADDRINUSE Solution: After killing all gluster processes .t is trying to start glusterd but somehow if kernel has not cleaned up resources(socket) then glusterd startup is failed due to bind system call failure.To avoid the issue retries to call bind 10 times to execute system call succesfully Change-Id: Ia5fd6b788f7b211c1508c1b7304fc08a32266629 Fixes: bz#1743020 Signed-off-by: Mohit Agrawal <moagra...@redhat.com> So the (puzzling) question is - why are we picking up old commit? In my local setup when I run the following command I do see the latest commit id being picked up: atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git rev-parse origin/master^{commit} # timeout=10 7926992e65d0a07fdc784a6e45740306d9b4a9f2 atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git show 7926992e65d0a07fdc784a6e45740306d9b4a9f2 commit 7926992e65d0a07fdc784a6e45740306d9b4a9f2 (origin/master, origin/HEAD, master) Author: Sanju Rakonde <srako...@redhat.com> Date: Mon Aug 26 12:38:40 2019 +0530 glusterd: Unused value coverity fix CID: 1288765 updates: bz#789278 Change-Id: Ie6b01f81339769f44d82fd7c32ad0ed1a697c69c Signed-off-by: Sanju Rakonde <srako...@redhat.com> ---------- Forwarded message --------- From: <jenk...@build.gluster.org> Date: Mon, Aug 26, 2019 at 11:32 PM Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-burn-in #4710 To: <maintain...@gluster.org> See < https://build.gluster.org/job/regression-test-burn-in/4710/display/redirect> ------------------------------------------ [...truncated 4.18 MB...] ./tests/features/lock-migration/lkmigration-set-option.t - 7 second ./tests/bugs/upcall/bug-1458127.t - 7 second ./tests/bugs/transport/bug-873367.t - 7 second ./tests/bugs/snapshot/bug-1260848.t - 7 second ./tests/bugs/shard/shard-inode-refcount-test.t - 7 second ./tests/bugs/replicate/bug-986905.t - 7 second ./tests/bugs/replicate/bug-921231.t - 7 second ./tests/bugs/replicate/bug-1132102.t - 7 second ./tests/bugs/replicate/bug-1037501.t - 7 second ./tests/bugs/posix/bug-1175711.t - 7 second ./tests/bugs/posix/bug-1122028.t - 7 second ./tests/bugs/glusterfs/bug-861015-log.t - 7 second ./tests/bugs/fuse/bug-983477.t - 7 second ./tests/bugs/ec/bug-1227869.t - 7 second ./tests/bugs/distribute/bug-1086228.t - 7 second ./tests/bugs/cli/bug-1087487.t - 7 second ./tests/bitrot/br-stub.t - 7 second ./tests/basic/ctime/ctime-noatime.t - 7 second ./tests/basic/afr/ta-write-on-bad-brick.t - 7 second ./tests/basic/afr/ta.t - 7 second ./tests/basic/afr/ta-shd.t - 7 second ./tests/basic/afr/root-squash-self-heal.t - 7 second ./tests/basic/afr/granular-esh/add-brick.t - 7 second ./tests/bugs/upcall/bug-1369430.t - 6 second ./tests/bugs/snapshot/bug-1064768.t - 6 second ./tests/bugs/shard/bug-1258334.t - 6 second ./tests/bugs/replicate/bug-1250170-fsync.t - 6 second ./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t - 6 second ./tests/bugs/quota/bug-1243798.t - 6 second ./tests/bugs/quota/bug-1104692.t - 6 second ./tests/bugs/protocol/bug-1321578.t - 6 second ./tests/bugs/nfs/bug-915280.t - 6 second ./tests/bugs/io-cache/bug-858242.t - 6 second ./tests/bugs/glusterfs-server/bug-877992.t - 6 second ./tests/bugs/glusterfs/bug-902610.t - 6 second ./tests/bugs/distribute/bug-884597.t - 6 second ./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t - 6 second ./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t - 6 second ./tests/bugs/bug-1702299.t - 6 second ./tests/bugs/bug-1371806_2.t - 6 second ./tests/bugs/bug-1258069.t - 6 second ./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t - 6 second ./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t - 6 second ./tests/basic/glusterd/thin-arbiter-volume-probe.t - 6 second ./tests/basic/gfapi/libgfapi-fini-hang.t - 6 second ./tests/basic/fencing/fencing-crash-conistency.t - 6 second ./tests/basic/ec/statedump.t - 6 second ./tests/basic/distribute/file-create.t - 6 second ./tests/basic/afr/tarissue.t - 6 second ./tests/basic/afr/gfid-heal.t - 6 second ./tests/basic/afr/afr-read-hash-mode.t - 6 second ./tests/basic/afr/add-brick-self-heal.t - 6 second ./tests/gfid2path/gfid2path_fuse.t - 5 second ./tests/bugs/shard/bug-1259651.t - 5 second ./tests/bugs/replicate/bug-767585-gfid.t - 5 second ./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t - 5 second ./tests/bugs/replicate/bug-1626994-info-split-brain.t - 5 second ./tests/bugs/replicate/bug-1365455.t - 5 second ./tests/bugs/replicate/bug-1101647.t - 5 second ./tests/bugs/nfs/bug-877885.t - 5 second ./tests/bugs/nfs/bug-847622.t - 5 second ./tests/bugs/nfs/bug-1116503.t - 5 second ./tests/bugs/md-cache/setxattr-prepoststat.t - 5 second ./tests/bugs/md-cache/bug-1211863_unlink.t - 5 second ./tests/bugs/md-cache/afr-stale-read.t - 5 second ./tests/bugs/io-stats/bug-1598548.t - 5 second ./tests/bugs/glusterfs/bug-895235.t - 5 second ./tests/bugs/glusterfs/bug-856455.t - 5 second ./tests/bugs/glusterfs/bug-848251.t - 5 second ./tests/bugs/glusterd/quorum-value-check.t - 5 second ./tests/bugs/glusterd/bug-1242875-do-not-pass-volinfo-quota.t - 5 second ./tests/bugs/ec/bug-1179050.t - 5 second ./tests/bugs/distribute/bug-912564.t - 5 second ./tests/bugs/distribute/bug-1368012.t - 5 second ./tests/bugs/core/bug-986429.t - 5 second ./tests/bugs/core/bug-908146.t - 5 second ./tests/bugs/bug-1371806_1.t - 5 second ./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t - 5 second ./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t - 5 second ./tests/basic/playground/template-xlator-sanity.t - 5 second ./tests/basic/hardlink-limit.t - 5 second ./tests/basic/glusterd/arbiter-volume-probe.t - 5 second ./tests/basic/ec/nfs.t - 5 second ./tests/basic/ec/ec-read-policy.t - 5 second ./tests/basic/ec/ec-anonymous-fd.t - 5 second ./tests/basic/afr/arbiter-remove-brick.t - 5 second ./tests/gfid2path/gfid2path_nfs.t - 4 second ./tests/gfid2path/get-gfid-to-path.t - 4 second ./tests/gfid2path/block-mount-access.t - 4 second ./tests/bugs/upcall/bug-upcall-stat.t - 4 second ./tests/bugs/trace/bug-797171.t - 4 second ./tests/bugs/snapshot/bug-1178079.t - 4 second ./tests/bugs/shard/bug-1342298.t - 4 second ./tests/bugs/shard/bug-1272986.t - 4 second ./tests/bugs/rpc/bug-954057.t - 4 second ./tests/bugs/replicate/bug-886998.t - 4 second ./tests/bugs/replicate/bug-1480525.t - 4 second ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t - 4 second ./tests/bugs/replicate/bug-1325792.t - 4 second ./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t - 4 second ./tests/bugs/posix/bug-gfid-path.t - 4 second ./tests/bugs/posix/bug-765380.t - 4 second ./tests/bugs/posix/bug-1619720.t - 4 second ./tests/bugs/nfs/zero-atime.t - 4 second ./tests/bugs/nfs/subdir-trailing-slash.t - 4 second ./tests/bugs/nfs/socket-as-fifo.t - 4 second ./tests/bugs/nfs/showmount-many-clients.t - 4 second ./tests/bugs/nfs/bug-1161092-nfs-acls.t - 4 second ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t - 4 second ./tests/bugs/glusterfs-server/bug-873549.t - 4 second ./tests/bugs/glusterfs-server/bug-864222.t - 4 second ./tests/bugs/glusterfs/bug-893378.t - 4 second ./tests/bugs/glusterd/bug-948729/bug-948729-force.t - 4 second ./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t - 4 second ./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t - 4 second ./tests/bugs/geo-replication/bug-1296496.t - 4 second ./tests/bugs/ec/bug-1161621.t - 4 second ./tests/bugs/distribute/bug-1088231.t - 4 second ./tests/bugs/cli/bug-977246.t - 4 second ./tests/bugs/cli/bug-1004218.t - 4 second ./tests/bugs/bug-1138841.t - 4 second ./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t - 4 second ./tests/bugs/access-control/bug-1051896.t - 4 second ./tests/bitrot/bug-1221914.t - 4 second ./tests/basic/ec/ec-internal-xattrs.t - 4 second ./tests/basic/distribute/non-root-unlink-stale-linkto.t - 4 second ./tests/basic/distribute/bug-1265677-use-readdirp.t - 4 second ./tests/basic/changelog/changelog-rename.t - 4 second ./tests/basic/afr/ta-check-locks.t - 4 second ./tests/basic/afr/heal-info.t - 4 second ./tests/performance/quick-read.t - 3 second ./tests/line-coverage/meta-max-coverage.t - 3 second ./tests/bugs/upcall/bug-1422776.t - 3 second ./tests/bugs/upcall/bug-1394131.t - 3 second ./tests/bugs/unclassified/bug-1034085.t - 3 second ./tests/bugs/snapshot/bug-1111041.t - 3 second ./tests/bugs/shard/bug-1256580.t - 3 second ./tests/bugs/shard/bug-1250855.t - 3 second ./tests/bugs/replicate/bug-976800.t - 3 second ./tests/bugs/replicate/bug-880898.t - 3 second ./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t - 3 second ./tests/bugs/readdir-ahead/bug-1446516.t - 3 second ./tests/bugs/readdir-ahead/bug-1439640.t - 3 second ./tests/bugs/readdir-ahead/bug-1390050.t - 3 second ./tests/bugs/quota/bug-1287996.t - 3 second ./tests/bugs/quick-read/bug-846240.t - 3 second ./tests/bugs/nl-cache/bug-1451588.t - 3 second ./tests/bugs/nfs/bug-1210338.t - 3 second ./tests/bugs/nfs/bug-1166862.t - 3 second ./tests/bugs/md-cache/bug-1632503.t - 3 second ./tests/bugs/md-cache/bug-1476324.t - 3 second ./tests/bugs/glusterfs-server/bug-861542.t - 3 second ./tests/bugs/glusterfs/bug-869724.t - 3 second ./tests/bugs/glusterfs/bug-844688.t - 3 second ./tests/bugs/glusterfs/bug-1482528.t - 3 second ./tests/bugs/glusterd/bug-948729/bug-948729.t - 3 second ./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t - 3 second ./tests/bugs/fuse/bug-1336818.t - 3 second ./tests/bugs/fuse/bug-1126048.t - 3 second ./tests/bugs/distribute/bug-907072.t - 3 second ./tests/bugs/core/log-bug-1362520.t - 3 second ./tests/bugs/core/io-stats-1322825.t - 3 second ./tests/bugs/core/bug-913544.t - 3 second ./tests/bugs/core/bug-845213.t - 3 second ./tests/bugs/core/bug-834465.t - 3 second ./tests/bugs/core/bug-1421721-mpx-toggle.t - 3 second ./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t - 3 second ./tests/bugs/core/bug-1117951.t - 3 second ./tests/bugs/core/949327.t - 3 second ./tests/bugs/cli/bug-983317-volume-get.t - 3 second ./tests/bugs/cli/bug-961307.t - 3 second ./tests/bugs/access-control/bug-1387241.t - 3 second ./tests/bitrot/bug-internal-xattrs-check-1243391.t - 3 second ./tests/basic/quota-rename.t - 3 second ./tests/basic/glusterd/check-cloudsync-ancestry.t - 3 second ./tests/basic/fops-sanity.t - 3 second ./tests/basic/fencing/test-fence-option.t - 3 second ./tests/basic/ec/ec-fallocate.t - 3 second ./tests/basic/ec/dht-rename.t - 3 second ./tests/basic/distribute/lookup.t - 3 second ./tests/basic/distribute/debug-xattrs.t - 3 second ./tests/line-coverage/some-features-in-libglusterfs.t - 2 second ./tests/bugs/unclassified/bug-991622.t - 2 second ./tests/bugs/shard/bug-1245547.t - 2 second ./tests/bugs/replicate/bug-884328.t - 2 second ./tests/bugs/readdir-ahead/bug-1512437.t - 2 second ./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t - 2 second ./tests/bugs/nfs/bug-970070.t - 2 second ./tests/bugs/nfs/bug-1302948.t - 2 second ./tests/bugs/logging/bug-823081.t - 2 second ./tests/bugs/glusterfs-server/bug-889996.t - 2 second ./tests/bugs/glusterfs/bug-860297.t - 2 second ./tests/bugs/glusterfs/bug-811493.t - 2 second ./tests/bugs/glusterd/bug-1085330-and-bug-916549.t - 2 second ./tests/bugs/fuse/bug-1283103.t - 2 second ./tests/bugs/distribute/bug-924265.t - 2 second ./tests/bugs/distribute/bug-1204140.t - 2 second ./tests/bugs/core/bug-924075.t - 2 second ./tests/bugs/core/bug-903336.t - 2 second ./tests/bugs/core/bug-1119582.t - 2 second ./tests/bugs/core/bug-1111557.t - 2 second ./tests/bugs/cli/bug-969193.t - 2 second ./tests/bugs/cli/bug-949298.t - 2 second ./tests/bugs/cli/bug-1378842-volume-get-all.t - 2 second ./tests/basic/md-cache/bug-1418249.t - 2 second ./tests/basic/afr/arbiter-cli.t - 2 second ./tests/line-coverage/volfile-with-all-graph-syntax.t - 1 second ./tests/bugs/shard/bug-1261773.t - 1 second ./tests/bugs/replicate/ta-inode-refresh-read.t - 1 second ./tests/bugs/glusterfs/bug-892730.t - 1 second ./tests/bugs/glusterfs/bug-853690.t - 1 second ./tests/bugs/cli/bug-921215.t - 1 second ./tests/bugs/cli/bug-867252.t - 1 second ./tests/bugs/cli/bug-764638.t - 1 second ./tests/bugs/cli/bug-1047378.t - 1 second ./tests/basic/posixonly.t - 1 second ./tests/basic/peer-parsing.t - 1 second ./tests/basic/netgroup_parsing.t - 1 second ./tests/basic/gfapi/sink.t - 1 second ./tests/basic/exports_parsing.t - 1 second ./tests/basic/glusterfsd-args.t - 0 second 4 test(s) failed ./tests/bugs/core/multiplex-limit-issue-151.t ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t ./tests/bugs/glusterd/brick-mux-validation.t ./tests/bugs/glusterd/bug-1595320.t 0 test(s) generated core 10 test(s) needed retry ./tests/bugs/core/bug-1119582.t ./tests/bugs/core/multiplex-limit-issue-151.t ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t ./tests/bugs/glusterd/brick-mux-validation.t ./tests/bugs/glusterd/bug-1595320.t ./tests/bugs/glusterd/bug-1696046.t ./tests/bugs/glusterd/optimized-basic-testcases.t ./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t ./tests/bugs/replicate/bug-976800.t ./tests/bugs/snapshot/bug-1111041.t Result is 1 tar: Removing leading `/' from member names kernel.core_pattern = /%e-%p.core Build step 'Execute shell' marked build as failure _______________________________________________ maintainers mailing list maintain...@gluster.org https://lists.gluster.org/mailman/listinfo/maintainers
_______________________________________________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/836554017 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/486278655 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
