Re: [Gluster-devel] Spurious regression failure? tests/basic/ec/ec-background-heals.t
Pranith,
In this test tests/basic/ec/ec-background-heals.t, I think the line number 86
actually creating a heal entry instead of
helping data heal quickly. What if all the data was already healed at that
moment, truncate came and in preop set the dirty flag and at the
end, as part of the heal, dirty flag was unset on previous good bricks only and
the brick which acted as heal-sink still has dirty marked by truncate.
That is why we are only seeing "1" as get_pending_heal_count. If a file was
actually not healed it should be "2".
If heal on this file completes and unset of dirty flag happens before truncate
everything will be fine.
I think we can wait for file to be heal without truncate?
71 #Test that disabling background-heals still drains the queue
72 TEST $CLI volume set $V0 disperse.background-heals 1
73 TEST touch $M0/{a,b,c,d}
74 TEST kill_brick $V0 $H0 $B0/${V0}2
75 EXPECT_WITHIN $CONFIG_UPDATE_TIMEOUT "1" mount_get_option_value $M0
$V0-disperse-0 background-heals
76 EXPECT_WITHIN $CONFIG_UPDATE_TIMEOUT "200" mount_get_option_value $M0
$V0-disperse-0 heal-wait-qlength
77 TEST truncate -s 1GB $M0/a
78 echo abc > $M0/b
79 echo abc > $M0/c
80 echo abc > $M0/d
81 TEST $CLI volume start $V0 force
82 EXPECT_WITHIN $CHILD_UP_TIMEOUT "3" ec_child_up_count $V0 0
83 TEST chown root:root $M0/{a,b,c,d}
84 TEST $CLI volume set $V0 disperse.background-heals 0
85 EXPECT_NOT "0" mount_get_option_value $M0 $V0-disperse-0 heal-waiters
86 TEST truncate -s 0 $M0/a # This completes the heal fast ;-) <<<
87 EXPECT_WITHIN $HEAL_TIMEOUT "^0$" get_pending_heal_count $V0
Ashish
- Original Message -
From: "Raghavendra Gowdappa"
To: "Nithya Balachandran"
Cc: "Gluster Devel" , "Pranith Kumar Karampuri"
, "Ashish Pandey"
Sent: Wednesday, January 25, 2017 9:41:38 AM
Subject: Re: [Gluster-devel] Spurious regression failure?
tests/basic/ec/ec-background-heals.t
Found another failure on same test:
https://build.gluster.org/job/centos6-regression/2874/consoleFull
- Original Message -
> From: "Nithya Balachandran"
> To: "Gluster Devel" , "Pranith Kumar Karampuri"
> , "Ashish Pandey"
>
> Sent: Tuesday, January 24, 2017 9:16:31 AM
> Subject: [Gluster-devel] Spurious regression failure?
> tests/basic/ec/ec-background-heals.t
>
> Hi,
>
>
> Can you please take a look at
> https://build.gluster.org/job/centos6-regression/2859/console ?
>
> tests/basic/ec/ec-background-heals.t has failed.
>
> Thanks,
> Nithya
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Priority based ping packet for 3.10
Yes, the earlier a fault is detected the better. On January 24, 2017 9:21:27 PM PST, Jeff Darcy wrote: >> If there are no responses to be received and no requests being >> sent to a brick, why would be a client be interested in the health of >> server/brick? > >The client (code) might not, but the user might want to find out and >fix >the fault before the brick gets busy again. >___ >Gluster-devel mailing list >Gluster-devel@gluster.org >http://lists.gluster.org/mailman/listinfo/gluster-devel -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Priority based ping packet for 3.10
> If there are no responses to be received and no requests being > sent to a brick, why would be a client be interested in the health of > server/brick? The client (code) might not, but the user might want to find out and fix the fault before the brick gets busy again. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Priority based ping packet for 3.10
On Tue, Jan 24, 2017 at 10:39 AM, Vijay Bellur wrote: > > > On Thu, Jan 19, 2017 at 8:06 AM, Jeff Darcy wrote: > >> > The more relevant question would be with TCP_KEEPALIVE and >> TCP_USER_TIMEOUT >> > on sockets, do we really need ping-pong framework in Clients? We might >> need >> > that in transport/rdma setups, but my question is concentrating on >> > transport/rdma. In other words would like to hear why do we need >> heart-beat >> > mechanism in the first place. One scenario might be a healthy socket >> level >> > connection but an unhealthy brick/client (like a deadlocked one). >> >> This is an important case to consider. On the one hand, I think it >> answers >> your question about TCP_KEEPALIVE. What we really care about is whether a >> brick's request queue is moving. In other words, what's the time since >> the >> last reply from that brick, and does that time exceed some threshold? > > I agree with this. > On a >> busy system, we don't even need ping packets to know that. We can just >> use >> responses on other requests to set/reset that timer. We only need to send >> ping packets when our *outbound* queue has remained empty for some >> fraction >> of our timeout. >> > Do we need ping packets sent even when client is not waiting for any replies? I assume no. If there are no responses to be received and no requests being sent to a brick, why would be a client be interested in the health of server/brick? > >> However, it's important that our measurements be *end to end* and not just >> at the transport level. This is particularly true with multiplexing, >> where multiple bricks will share and contend on various resources. We >> should ping *through* client and server, with separate translators above >> and below each. This would give us a true end-to-end ping *for that >> brick*, and also keep the code nicely modular. >> > Agree with this. My understanding of ping framework is a tool to identify unhealthy bricks (we are interested in bricks as they are the ones going to serve fops). With that understanding ping-pong should be end to end (to whatever logical entity that constitutes brick). However, where in the brick xlator stack ping packets should be responded? Should they go all the way down to storage/posix? > > +1 to this. Having ping, pong xlators immediately above and below protocol > translators would also address the problem of epoll threads getting blocked > in gluster's xlator stacks in busy systems. > > Having said that, I do see value in Rafi's patch that prompted this > thread. Would it not help to prioritize ping - pong traffic in all parts of > the gluster stack including the send queue on the client? > I've two concerns here: 1. Responsiveness of brick to client invariably involves latency of network and our own transport's io-queue. Wouldn't prioritizing ping packets over normal data give us a skewed view of brick's responsiveness? For eg., On a network with heavy traffic ping-pong might be happening, but fops might be moving very slowely. What is that we achieve with a successful ping-pong in this scenario? Also, Is our response to the opposite scenario of ping-timeout happening and disconnecting the transport achieves anything substantially good? May be it helps to bring the latency of syscalls down (as experienced by application), as our HA translators like afr, EC add the latency of identifying disconnect (or a successful fop) to latency of syscalls. As developers many of us keep wondering what is that we are trying to achieve with an heart beat mechanism. 2. Assuming that we want to prioritize ping traffic over normal traffic (which we do logically now as ping packets doesn't traverse the entire brick xlator stack all the way down to posix, instead short circuit at protocol/server), the fix in discussion here is partial (as we can't prioritize ping traffic ON the WIRE and through tcp/ip stack). While I don't have strong objections to it, I feel that its partial solution and might be inconsequential (just an hunch, no data). However, I can accept the patch, if we feel it helps. > Regards, > Vijay > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel > -- Raghavendra G ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Spurious regression failure? tests/basic/ec/ec-background-heals.t
Found another failure on same test: https://build.gluster.org/job/centos6-regression/2874/consoleFull - Original Message - > From: "Nithya Balachandran" > To: "Gluster Devel" , "Pranith Kumar Karampuri" > , "Ashish Pandey" > > Sent: Tuesday, January 24, 2017 9:16:31 AM > Subject: [Gluster-devel] Spurious regression failure? > tests/basic/ec/ec-background-heals.t > > Hi, > > > Can you please take a look at > https://build.gluster.org/job/centos6-regression/2859/console ? > > tests/basic/ec/ec-background-heals.t has failed. > > Thanks, > Nithya > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] gluster tiering maintainer
Hi Dan, Thank you for your contributions to Gluster as a maintainer! Look forward to your continued participation in the community. I am working on a plan for tiering going forward and will share details in the few days. Good luck for your future endeavors! Best, Vijay On Tue, Jan 24, 2017 at 9:14 AM, Dan Lambright wrote: > Dear Gluster Community, > > 2017 has arrived, and I have taken an opportunity which will require a new > maintainer for gluster tiering to replace me. I will continue to be > available to help with the feature as a contributor. > > As seen in this year's CES conference [1], new storage types are coming > fast. Customers will likely use a mix of them, and tiering is a logical > consideration. Going forward the feature should support multiple tiers, > better performance while migration is underway, and migration according to > attributes other than hit rate. Under the hood, tiering should evolve with > the rest of gluster (e.g. DHT2), the database vs other algorithms should be > analyzed critically, and somehow the codebase should be detached from DHT. > All significant and interesting challenges. > > Gluster has come a long way since I joined- and I think it will only get > better. > > Dan > > [1] > http://www.theregister.co.uk/2017/01/04/optane_arrives_at_ces/ > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Multiplexing status, January 24
Coming down to the wire here. Here's the latest. https://review.gluster.org/#/c/14763/ With the latest patchset (61!) I've addressed most of people's review comments. I've also put a dent in the last functional area, which is full support for snapshots. Previously, snapshot bricks were exempt from multiplexing even if it was otherwise enabled. Now, basic snapshot operations seem to work, but clones and USS still need some work. The biggest obstacle is actually *shutting down* snapshot bricks. Some as-yet-unidentified translator is keeping the brick mountpoint alive, even when we try to unmount it. This causes failures near the end of some snapshot tests if multiplexing is enabled. I'll investigate that, along with two other lesser issues: * A better way to enable/disable multiplexing (Vijay and Samikshan both have patches) * Some way to set multiplexing levels/policies for performance testing. IMO this code is very close to being finished, except for one thing: our regression tests. Dozens upon dozens of times, regression tests have failed in our test infrastructure even though they have worked and continue to work reliably on a regular test machine. I've addressed a lot of these - the patch fixes or disables 33 tests - but more pop up all the time. The simple fact is that many of our tests make timing or other assumptions that were never really valid, but those assumptions were violated rarely enough that the tests would pass anyway most of the time. With multiplexing, many of those assumptions are even less often true, so tests fail on a more regular basis. Going forward, it's going to be more important than ever that people flag spurious regression-test failures as such. I'll certainly be doing so, and "encouraging" test owners to participate in fixing their tests. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] gluster tiering maintainer
Dear Gluster Community, 2017 has arrived, and I have taken an opportunity which will require a new maintainer for gluster tiering to replace me. I will continue to be available to help with the feature as a contributor. As seen in this year's CES conference [1], new storage types are coming fast. Customers will likely use a mix of them, and tiering is a logical consideration. Going forward the feature should support multiple tiers, better performance while migration is underway, and migration according to attributes other than hit rate. Under the hood, tiering should evolve with the rest of gluster (e.g. DHT2), the database vs other algorithms should be analyzed critically, and somehow the codebase should be detached from DHT. All significant and interesting challenges. Gluster has come a long way since I joined- and I think it will only get better. Dan [1] http://www.theregister.co.uk/2017/01/04/optane_arrives_at_ces/ ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
