I observed that when the ITtests hange, the ZK connection exhaustion is caused
by ViewUtil.dropChildViews , and I opened a JIRA PHOENIX-5970 to solve it,
seems that it works.
At 2020-06-19 14:58:27, "Istvan Toth" <[email protected]> wrote:
>The hanging test suite issue has been fixed by Richard Antal's PHOENIX-5962
><https://issues.apache.org/jira/browse/PHOENIX-5962> .
>Tests should be back to normal (that is, they're still flakey as fresh snow)
>
>Rajeshbabu, is there an easy way (specific message in the logs) for
>identifying the ZK connection exhaustion situation?
>
>Istvan
>
>On Thu, Jun 18, 2020 at 2:49 AM [email protected] <
>[email protected]> wrote:
>
>> Just my observation related tests hang or flaky ness is that when there are
>> connections created which involves zookeeper connection then at some point
>> of time zookeeper sees too many connections and not allow to create further
>> connections then the cluster won't be terminated immediately and tests
>> hangs. We can identify such cases and close the connections properly if
>> any.
>>
>> Thanks,
>> Rajeshbabu.
>>
>> On Tue, Jun 16, 2020, 12:12 PM Istvan Toth <[email protected]> wrote:
>>
>> > What we really need is getting our test suite (and infra) sorted out.
>> > Your suggestions are already part of the process, it's just that lately
>> > everyone ignores them, because the precommit tests have been next to
>> > useless for a good while.
>> >
>> > I think that if developers can be reasonably sure that the precommit
>> > failure that they see is not some kind of random/known flake, they (we)
>> > will take them seriously, and not commit until they(we) get it right.
>> >
>> > Blocking further commits until the tests are fixed is one drastic, but
>> > possibly necessary measure to achieve this.
>> > And once we get them fixed, we will indeed need to be more vigilant in
>> not
>> > letting the situation deteriorate.
>> >
>> > Istvan
>> >
>> > On Mon, Jun 15, 2020 at 7:30 PM Andrew Purtell <[email protected]>
>> > wrote:
>> >
>> > > So it's time to enforce an automatic revert if build fails policy?
>> > > Ideally, no commit before a precommit passes.
>> > > If that fails, if someone discovers failing tests and can git bisect to
>> > the
>> > > cause, automatic revert of the offending commit.
>> > >
>> > > YDYT?
>> > >
>> > >
>> > > On Sun, Jun 14, 2020 at 10:20 PM Istvan Toth
>> <[email protected]
>> > >
>> > > wrote:
>> > >
>> > > > Unfortunately you are not missing anything, Lars :(
>> > > >
>> > > > What's worse, the tests are not only failing, they even hang since
>> Jun
>> > 2.
>> > > > Master is in a similar state.
>> > > >
>> > > > regards
>> > > > Istvan
>> > > >
>> > > > On Sun, Jun 14, 2020 at 12:03 AM [email protected] <[email protected]>
>> > > > wrote:
>> > > >
>> > > > > Thanks Istvan.
>> > > > >
>> > > > > Just checked the builds. Looks like the 4.x builds have not passed
>> a
>> > > > > single time since May 20th.
>> > > > >
>> > > > > I hope I am missing something... Otherwise this would be pretty
>> > > > > frustrating. :)
>> > > > > (Since pleading doesn't appear to help, maybe we should
>> automatically
>> > > > > block all commits until all tests pass...?)
>> > > > >
>> > > > > -- Lars
>> > > > >
>> > > > >
>> > > > > On Tuesday, April 21, 2020, 5:33:52 AM PDT, Istvan Toth <
>> > > > [email protected]>
>> > > > > wrote:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > I've deleted all but the four Phoenix jobs listed in the JIRA.
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > *István Tóth* | Staff Software Engineer
>> > > > [email protected] <https://www.cloudera.com>
>> > > > [image: Cloudera] <https://www.cloudera.com/>
>> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
>> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
>> > > Cloudera
>> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
>> > > > <https://www.cloudera.com/>
>> > > > ------------------------------
>> > > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > > Andrew
>> > >
>> > > Words like orphans lost among the crosstalk, meaning torn from truth's
>> > > decrepit hands
>> > > - A23, Crosstalk
>> > >
>> >
>>
