Re: Unit tests are hanging?

2019-08-12 Thread Kirk Lund 
Thanks Owen!

On Sat, Aug 10, 2019 at 1:53 PM Owen Nichols  wrote:

> Done (increased from 10 minutes to 20 minutes for PR pipeline, and changed
> from none to 20 minutes for develop pipeline).
>
> -Owen
>
> > On Aug 10, 2019, at 11:01 AM, Kirk Lund  wrote:
> >
> > I just saw the Build job exceed timeout during pmdMain so I think we need
> > to increase its timeout as well.
> > https://concourse.apachegeode-ci.info/builds/85560
> >
> >
> > On Thu, Aug 8, 2019 at 3:53 PM Ryan McMahon 
> wrote:
> >
> >> I have a PR up for this now.
> >> https://github.com/apache/geode/pull/3900
> >>
> >> It bumps the timeout to 20 minutes but also changes the
> CALL_STACK_TIMEOUT
> >> to be 1140 seconds (19 minutes).  The latter configuration parameter
> >> controls when we declare the task "hung" and dump stacks.  We were not
> >> dumping stacks at all before these changes because the value was 1800
> >> seconds which is well above the timeout of 10 minutes.  With my proposed
> >> change we will dump stacks before the task times out so we will have
> >> something to look at in the test artifacts to identify the hang.
> >>
> >> Thanks,
> >> Ryan
> >>
> >> On Thu, Aug 8, 2019 at 11:28 AM Ryan McMahon 
> wrote:
> >>
> >>> Looks like we have a general consensus from the community.  I'll go
> ahead
> >>> and make a PR for the changes.
> >>>
> >>> Thanks,
> >>> Ryan
> >>>
> >>> On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos 
> >> wrote:
> >>>
>  +1
> 
>  On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:
> 
> > +1
> >
> > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
> >
> >>> With all that, I propose we permanently bump the timeouts on
>  UnitTestX
> >> jobs
> >>> across the board (main pipeline, PR pipeline, etc) from 10 to 20
> > minutes
> >> to
> >>> be more tolerant of these types of degradations.
> >>>
> >>
> >> +1
> >>
> >> -Dan
> >>
> >
> 
> 
>  --
>  Juan José Ramos Cassella
>  Senior Technical Support Engineer
>  Email: jra...@pivotal.io
>  Office#: +353 21 4238611
>  Mobile#: +353 87 2074066
>  After Hours Contact#: +1 877 477 2269
>  Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
>  How to upload artifacts:
>  https://support.pivotal.io/hc/en-us/articles/204369073
>  How to escalate a ticket:
>  https://support.pivotal.io/hc/en-us/articles/203809556
> 
>  [image: support]  [image: twitter]
>   [image: linkedin]
>   [image: facebook]
>   [image: google plus]
>   [image: youtube]
>  <
> >>
> https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl
> >
> 
> >>>
> >>
>
>

Re: Unit tests are hanging?

2019-08-10 Thread Owen Nichols 
Done (increased from 10 minutes to 20 minutes for PR pipeline, and changed from 
none to 20 minutes for develop pipeline).

-Owen

> On Aug 10, 2019, at 11:01 AM, Kirk Lund  wrote:
> 
> I just saw the Build job exceed timeout during pmdMain so I think we need
> to increase its timeout as well.
> https://concourse.apachegeode-ci.info/builds/85560
> 
> 
> On Thu, Aug 8, 2019 at 3:53 PM Ryan McMahon  wrote:
> 
>> I have a PR up for this now.
>> https://github.com/apache/geode/pull/3900
>> 
>> It bumps the timeout to 20 minutes but also changes the CALL_STACK_TIMEOUT
>> to be 1140 seconds (19 minutes).  The latter configuration parameter
>> controls when we declare the task "hung" and dump stacks.  We were not
>> dumping stacks at all before these changes because the value was 1800
>> seconds which is well above the timeout of 10 minutes.  With my proposed
>> change we will dump stacks before the task times out so we will have
>> something to look at in the test artifacts to identify the hang.
>> 
>> Thanks,
>> Ryan
>> 
>> On Thu, Aug 8, 2019 at 11:28 AM Ryan McMahon  wrote:
>> 
>>> Looks like we have a general consensus from the community.  I'll go ahead
>>> and make a PR for the changes.
>>> 
>>> Thanks,
>>> Ryan
>>> 
>>> On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos 
>> wrote:
>>> 
 +1
 
 On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:
 
> +1
> 
> On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
> 
>>> With all that, I propose we permanently bump the timeouts on
 UnitTestX
>> jobs
>>> across the board (main pipeline, PR pipeline, etc) from 10 to 20
> minutes
>> to
>>> be more tolerant of these types of degradations.
>>> 
>> 
>> +1
>> 
>> -Dan
>> 
> 
 
 
 --
 Juan José Ramos Cassella
 Senior Technical Support Engineer
 Email: jra...@pivotal.io
 Office#: +353 21 4238611
 Mobile#: +353 87 2074066
 After Hours Contact#: +1 877 477 2269
 Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
 How to upload artifacts:
 https://support.pivotal.io/hc/en-us/articles/204369073
 How to escalate a ticket:
 https://support.pivotal.io/hc/en-us/articles/203809556
 
 [image: support]  [image: twitter]
  [image: linkedin]
  [image: facebook]
  [image: google plus]
  [image: youtube]
 <
>> https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl
> 
 
>>> 
>>

Re: Unit tests are hanging?

2019-08-10 Thread Kirk Lund 
I just saw the Build job exceed timeout during pmdMain so I think we need
to increase its timeout as well.
https://concourse.apachegeode-ci.info/builds/85560


On Thu, Aug 8, 2019 at 3:53 PM Ryan McMahon  wrote:

> I have a PR up for this now.
> https://github.com/apache/geode/pull/3900
>
> It bumps the timeout to 20 minutes but also changes the CALL_STACK_TIMEOUT
> to be 1140 seconds (19 minutes).  The latter configuration parameter
> controls when we declare the task "hung" and dump stacks.  We were not
> dumping stacks at all before these changes because the value was 1800
> seconds which is well above the timeout of 10 minutes.  With my proposed
> change we will dump stacks before the task times out so we will have
> something to look at in the test artifacts to identify the hang.
>
> Thanks,
> Ryan
>
> On Thu, Aug 8, 2019 at 11:28 AM Ryan McMahon  wrote:
>
> > Looks like we have a general consensus from the community.  I'll go ahead
> > and make a PR for the changes.
> >
> > Thanks,
> > Ryan
> >
> > On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos 
> wrote:
> >
> >> +1
> >>
> >> On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:
> >>
> >> > +1
> >> >
> >> > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
> >> >
> >> > > > With all that, I propose we permanently bump the timeouts on
> >> UnitTestX
> >> > > jobs
> >> > > > across the board (main pipeline, PR pipeline, etc) from 10 to 20
> >> > minutes
> >> > > to
> >> > > > be more tolerant of these types of degradations.
> >> > > >
> >> > >
> >> > > +1
> >> > >
> >> > > -Dan
> >> > >
> >> >
> >>
> >>
> >> --
> >> Juan José Ramos Cassella
> >> Senior Technical Support Engineer
> >> Email: jra...@pivotal.io
> >> Office#: +353 21 4238611
> >> Mobile#: +353 87 2074066
> >> After Hours Contact#: +1 877 477 2269
> >> Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
> >> How to upload artifacts:
> >> https://support.pivotal.io/hc/en-us/articles/204369073
> >> How to escalate a ticket:
> >> https://support.pivotal.io/hc/en-us/articles/203809556
> >>
> >> [image: support]  [image: twitter]
> >>  [image: linkedin]
> >>  [image: facebook]
> >>  [image: google plus]
> >>  [image: youtube]
> >> <
> https://www.youtube.com/playlist?list=PLAdzTan_eSPScpj2J50ErtzR9ANSzv3kl
> >> >
> >>
> >
>

Re: Unit tests are hanging?

2019-08-08 Thread Ryan McMahon 
I have a PR up for this now.
https://github.com/apache/geode/pull/3900

It bumps the timeout to 20 minutes but also changes the CALL_STACK_TIMEOUT
to be 1140 seconds (19 minutes).  The latter configuration parameter
controls when we declare the task "hung" and dump stacks.  We were not
dumping stacks at all before these changes because the value was 1800
seconds which is well above the timeout of 10 minutes.  With my proposed
change we will dump stacks before the task times out so we will have
something to look at in the test artifacts to identify the hang.

Thanks,
Ryan

On Thu, Aug 8, 2019 at 11:28 AM Ryan McMahon  wrote:

> Looks like we have a general consensus from the community.  I'll go ahead
> and make a PR for the changes.
>
> Thanks,
> Ryan
>
> On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos  wrote:
>
>> +1
>>
>> On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:
>>
>> > +1
>> >
>> > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
>> >
>> > > > With all that, I propose we permanently bump the timeouts on
>> UnitTestX
>> > > jobs
>> > > > across the board (main pipeline, PR pipeline, etc) from 10 to 20
>> > minutes
>> > > to
>> > > > be more tolerant of these types of degradations.
>> > > >
>> > >
>> > > +1
>> > >
>> > > -Dan
>> > >
>> >
>>
>>
>> --
>> Juan José Ramos Cassella
>> Senior Technical Support Engineer
>> Email: jra...@pivotal.io
>> Office#: +353 21 4238611
>> Mobile#: +353 87 2074066
>> After Hours Contact#: +1 877 477 2269
>> Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
>> How to upload artifacts:
>> https://support.pivotal.io/hc/en-us/articles/204369073
>> How to escalate a ticket:
>> https://support.pivotal.io/hc/en-us/articles/203809556
>>
>> [image: support]  [image: twitter]
>>  [image: linkedin]
>>  [image: facebook]
>>  [image: google plus]
>>  [image: youtube]
>> > >
>>
>

Re: Unit tests are hanging?

2019-08-08 Thread Ryan McMahon 
Looks like we have a general consensus from the community.  I'll go ahead
and make a PR for the changes.

Thanks,
Ryan

On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos  wrote:

> +1
>
> On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:
>
> > +1
> >
> > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
> >
> > > > With all that, I propose we permanently bump the timeouts on
> UnitTestX
> > > jobs
> > > > across the board (main pipeline, PR pipeline, etc) from 10 to 20
> > minutes
> > > to
> > > > be more tolerant of these types of degradations.
> > > >
> > >
> > > +1
> > >
> > > -Dan
> > >
> >
>
>
> --
> Juan José Ramos Cassella
> Senior Technical Support Engineer
> Email: jra...@pivotal.io
> Office#: +353 21 4238611
> Mobile#: +353 87 2074066
> After Hours Contact#: +1 877 477 2269
> Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
> How to upload artifacts:
> https://support.pivotal.io/hc/en-us/articles/204369073
> How to escalate a ticket:
> https://support.pivotal.io/hc/en-us/articles/203809556
>
> [image: support]  [image: twitter]
>  [image: linkedin]
>  [image: facebook]
>  [image: google plus]
>  [image: youtube]
> 
>

Re: Unit tests are hanging?

2019-08-08 Thread Juan José Ramos 
+1

On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund  wrote:

> +1
>
> On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:
>
> > > With all that, I propose we permanently bump the timeouts on UnitTestX
> > jobs
> > > across the board (main pipeline, PR pipeline, etc) from 10 to 20
> minutes
> > to
> > > be more tolerant of these types of degradations.
> > >
> >
> > +1
> >
> > -Dan
> >
>


-- 
Juan José Ramos Cassella
Senior Technical Support Engineer
Email: jra...@pivotal.io
Office#: +353 21 4238611
Mobile#: +353 87 2074066
After Hours Contact#: +1 877 477 2269
Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
How to upload artifacts:
https://support.pivotal.io/hc/en-us/articles/204369073
How to escalate a ticket:
https://support.pivotal.io/hc/en-us/articles/203809556

[image: support]  [image: twitter]
 [image: linkedin]
 [image: facebook]
 [image: google plus]
 [image: youtube]

Re: Unit tests are hanging?

2019-08-08 Thread Kirk Lund 
+1

On Thu, Aug 8, 2019 at 10:14 AM Dan Smith  wrote:

> > With all that, I propose we permanently bump the timeouts on UnitTestX
> jobs
> > across the board (main pipeline, PR pipeline, etc) from 10 to 20 minutes
> to
> > be more tolerant of these types of degradations.
> >
>
> +1
>
> -Dan
>

Re: Unit tests are hanging?

2019-08-08 Thread Dan Smith 
> With all that, I propose we permanently bump the timeouts on UnitTestX jobs
> across the board (main pipeline, PR pipeline, etc) from 10 to 20 minutes to
> be more tolerant of these types of degradations.
>

+1

-Dan

Re: Unit tests are hanging?

2019-08-08 Thread Ryan McMahon 
So we temporarily bumped the timeout from 10 minutes to 2 hours on the
UnitTestOpenJDK11 execute_tests Concourse task, originally with the
intention of logging into the heavy lifter to debug further.  However,
after doing that we see that the jobs are all succeeding in roughly the
same amount of time as they did before we started seeing these timeouts
(total job time 25-30 minutes, total execute_tests task time of 7.5ish
minutes).  This makes me think that we were seeing some sort of performance
degradation at the infrastructure level, as opposed to some recent change
to Geode which is causing the timeouts.

With all that, I propose we permanently bump the timeouts on UnitTestX jobs
across the board (main pipeline, PR pipeline, etc) from 10 to 20 minutes to
be more tolerant of these types of degradations.  Please let me know if
anyone is opposed to doing this.

Thanks,
Ryan

On Wed, Aug 7, 2019 at 11:39 AM Ryan McMahon  wrote:

> Still trying to identify the cause of the unit test hangs.stay tuned.
> For other people who might not know this, the reason the entire CI job
> fails rather than the hanging test is because we don't have any global
> test-level timeouts, so a hanging test will run up until the Concourse
> timeout is reached.  I wrote a story to explore adding test level timeouts
> so we can get actual feedback in the CI about which test is hanging without
> waiting for the whole job to timeout, downloading artifacts, looking at the
> stack dumps, etc etc.  I think we could have some reasonable global upper
> limit (30 minutes?) for any test.  We could always tune that as needed.
>
> https://issues.apache.org/jira/projects/GEODE/issues/GEODE-7057?filter=allopenissues
>
> Ryan
>
> On Wed, Aug 7, 2019 at 11:21 AM Bruce Schuchardt 
> wrote:
>
>> Yeah, that test passed on my branch in unit tests and stress tests but
>> for some reason is hanging after the merge to develop. I've pushed an
>> @Ignore for the test that you should pick up.
>>
>> On 8/7/19 10:38 AM, Kirk Lund wrote:
>> > Yep, that's the same test I'm seeing in the callstacks of my dunit tgz
>> from
>> > concourse...
>> >
>> > Started @ 2019-08-07 07:25:18.494 +
>> > 2019-08-07 07:51:25.252 +
>> >   org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
>> > testMulticastAfterReconnect
>> > Ended @ 2019-08-07 08:28:16.591 +
>> >
>> > Thanks for digging into it!
>> >
>> > On Wed, Aug 7, 2019 at 10:16 AM Ryan McMahon 
>> wrote:
>> >
>> >> I think the reflection and PowerMock warnings here are probably a red
>> >> herring.  We pulled down the artifacts and found that the DUnit job is
>> >> hanging due to stuck threads in a newer DUnit test.  I am not sure why
>> it
>> >> isn't failing the test but rather failing the entire job.  I believe
>> Bruce
>> >> Schuchart is going to dig into this a bit.  Analysis from Lynn
>> >> Hughes-Godfrey:
>> >>
>> >> So, I downloaded
>> >>
>> >>
>> >>
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/961
>> >>
>> >> to
>> >>
>> >>
>> >>
>> /Users/lhughesgodfrey/Downloads/distributedtestfiles-OpenJDK8-1.11.0-SNAPSHOT.0010/geode-core/build/distributedTest
>> >>
>> >> In that callstacks directory, you'll see dunit-hangs.txt ... and this
>> >> test has been running for over an hour ...
>> >>
>> >> ```
>> >>
>> >> Started @ 2019-08-07 07:54:10.272 +
>> >>
>> >> 2019-08-07 08:22:25.440 +
>> >> org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
>> >> testMulticastAfterReconnect
>> >>
>> >> Ended @ 2019-08-07 08:58:50.243 +
>> >>
>> >> ```
>> >>
>> >> When I look at the callstacks (at the top level), I see this:
>> >>
>> >> ```
>> >>
>> >> "Pooled Waiting Message Processor 1" #213 daemon prio=5 os_prio=0
>> >> tid=0x7fc738024000 nid=0x488 waiting for monitor entry
>> >> [0x7fc7ec5f7000]
>> >>
>> >> java.lang.Thread.State: BLOCKED (on object monitor)
>> >>
>> >>  at
>> >>
>> org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.startEventProcessing(GMSMembershipManager.java:1179)
>> >>
>> >>  - waiting to lock <0xe03a75e0> (a java.lang.Object)
>> >>
>> >>  at
>> >>
>> org.apache.geode.distributed.internal.ClusterDistributionManager.readyForMessages(ClusterDistributionManager.java:1152)
>> >>
>> >>  at
>> >>
>> org.apache.geode.distributed.internal.ClusterDistributionManager.lambda$startThreads$10(ClusterDistributionManager.java:1129)
>> >>
>> >>  at
>> >>
>> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$87/1408912936.run(Unknown
>> >> Source)
>> >>
>> >>  at
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> >>
>> >>  at
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> >>
>> >>  at
>> >>
>> org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManage

Re: Unit tests are hanging?

2019-08-07 Thread Ryan McMahon 
Still trying to identify the cause of the unit test hangs.stay tuned.
For other people who might not know this, the reason the entire CI job
fails rather than the hanging test is because we don't have any global
test-level timeouts, so a hanging test will run up until the Concourse
timeout is reached.  I wrote a story to explore adding test level timeouts
so we can get actual feedback in the CI about which test is hanging without
waiting for the whole job to timeout, downloading artifacts, looking at the
stack dumps, etc etc.  I think we could have some reasonable global upper
limit (30 minutes?) for any test.  We could always tune that as needed.
https://issues.apache.org/jira/projects/GEODE/issues/GEODE-7057?filter=allopenissues

Ryan

On Wed, Aug 7, 2019 at 11:21 AM Bruce Schuchardt 
wrote:

> Yeah, that test passed on my branch in unit tests and stress tests but
> for some reason is hanging after the merge to develop. I've pushed an
> @Ignore for the test that you should pick up.
>
> On 8/7/19 10:38 AM, Kirk Lund wrote:
> > Yep, that's the same test I'm seeing in the callstacks of my dunit tgz
> from
> > concourse...
> >
> > Started @ 2019-08-07 07:25:18.494 +
> > 2019-08-07 07:51:25.252 +
> >   org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
> > testMulticastAfterReconnect
> > Ended @ 2019-08-07 08:28:16.591 +
> >
> > Thanks for digging into it!
> >
> > On Wed, Aug 7, 2019 at 10:16 AM Ryan McMahon 
> wrote:
> >
> >> I think the reflection and PowerMock warnings here are probably a red
> >> herring.  We pulled down the artifacts and found that the DUnit job is
> >> hanging due to stuck threads in a newer DUnit test.  I am not sure why
> it
> >> isn't failing the test but rather failing the entire job.  I believe
> Bruce
> >> Schuchart is going to dig into this a bit.  Analysis from Lynn
> >> Hughes-Godfrey:
> >>
> >> So, I downloaded
> >>
> >>
> >>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/961
> >>
> >> to
> >>
> >>
> >>
> /Users/lhughesgodfrey/Downloads/distributedtestfiles-OpenJDK8-1.11.0-SNAPSHOT.0010/geode-core/build/distributedTest
> >>
> >> In that callstacks directory, you'll see dunit-hangs.txt ... and this
> >> test has been running for over an hour ...
> >>
> >> ```
> >>
> >> Started @ 2019-08-07 07:54:10.272 +
> >>
> >> 2019-08-07 08:22:25.440 +
> >> org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
> >> testMulticastAfterReconnect
> >>
> >> Ended @ 2019-08-07 08:58:50.243 +
> >>
> >> ```
> >>
> >> When I look at the callstacks (at the top level), I see this:
> >>
> >> ```
> >>
> >> "Pooled Waiting Message Processor 1" #213 daemon prio=5 os_prio=0
> >> tid=0x7fc738024000 nid=0x488 waiting for monitor entry
> >> [0x7fc7ec5f7000]
> >>
> >> java.lang.Thread.State: BLOCKED (on object monitor)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.startEventProcessing(GMSMembershipManager.java:1179)
> >>
> >>  - waiting to lock <0xe03a75e0> (a java.lang.Object)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager.readyForMessages(ClusterDistributionManager.java:1152)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager.lambda$startThreads$10(ClusterDistributionManager.java:1129)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$87/1408912936.run(Unknown
> >> Source)
> >>
> >>  at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>
> >>  at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager.doWaitingThread(ClusterDistributionManager.java:851)
> >>
> >>  at
> >>
> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$53/274751999.invoke(Unknown
> >> Source)
> >>
> >>  at
> >>
> org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)
> >>
> >>  at
> >>
> org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$49/1824093464.run(Unknown
> >> Source)
> >>
> >>  at java.lang.Thread.run(Thread.java:748)
> >>
> >> Locked ownable synchronizers:
> >>
> >>  - <0xe03a7660> (a
> >> java.util.concurrent.ThreadPoolExecutor$Worker)
> >>
> >> ```
> >>
> >> This thread holds the lock:
> >>
> >> ```
> >>
> >> "ReconnectThread" #193 prio=5 os_prio=0 tid=0x7fc7cc219000
> >> nid=0x46a waiting on condition [0x7fc78bdfb000]
> >>
> >> java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>
> >>  at java.lang.Thread.sl

Re: Unit tests are hanging?

2019-08-07 Thread Bruce Schuchardt 
Yeah, that test passed on my branch in unit tests and stress tests but 
for some reason is hanging after the merge to develop. I've pushed an 
@Ignore for the test that you should pick up.


On 8/7/19 10:38 AM, Kirk Lund wrote:

Yep, that's the same test I'm seeing in the callstacks of my dunit tgz from
concourse...

Started @ 2019-08-07 07:25:18.494 +
2019-08-07 07:51:25.252 +
  org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
testMulticastAfterReconnect
Ended @ 2019-08-07 08:28:16.591 +

Thanks for digging into it!

On Wed, Aug 7, 2019 at 10:16 AM Ryan McMahon  wrote:


I think the reflection and PowerMock warnings here are probably a red
herring.  We pulled down the artifacts and found that the DUnit job is
hanging due to stuck threads in a newer DUnit test.  I am not sure why it
isn't failing the test but rather failing the entire job.  I believe Bruce
Schuchart is going to dig into this a bit.  Analysis from Lynn
Hughes-Godfrey:

So, I downloaded


https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/961

to


/Users/lhughesgodfrey/Downloads/distributedtestfiles-OpenJDK8-1.11.0-SNAPSHOT.0010/geode-core/build/distributedTest

In that callstacks directory, you'll see dunit-hangs.txt ... and this
test has been running for over an hour ...

```

Started @ 2019-08-07 07:54:10.272 +

2019-08-07 08:22:25.440 +
org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
testMulticastAfterReconnect

Ended @ 2019-08-07 08:58:50.243 +

```

When I look at the callstacks (at the top level), I see this:

```

"Pooled Waiting Message Processor 1" #213 daemon prio=5 os_prio=0
tid=0x7fc738024000 nid=0x488 waiting for monitor entry
[0x7fc7ec5f7000]

java.lang.Thread.State: BLOCKED (on object monitor)

 at
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.startEventProcessing(GMSMembershipManager.java:1179)

 - waiting to lock <0xe03a75e0> (a java.lang.Object)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager.readyForMessages(ClusterDistributionManager.java:1152)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager.lambda$startThreads$10(ClusterDistributionManager.java:1129)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$87/1408912936.run(Unknown
Source)

 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager.doWaitingThread(ClusterDistributionManager.java:851)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$53/274751999.invoke(Unknown
Source)

 at
org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)

 at
org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$49/1824093464.run(Unknown
Source)

 at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:

 - <0xe03a7660> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

```

This thread holds the lock:

```

"ReconnectThread" #193 prio=5 os_prio=0 tid=0x7fc7cc219000
nid=0x46a waiting on condition [0x7fc78bdfb000]

java.lang.Thread.State: TIMED_WAITING (sleeping)

 at java.lang.Thread.sleep(Native Method)

 at
org.apache.geode.internal.tcp.Connection.createSender(Connection.java:959)

 at
org.apache.geode.internal.tcp.ConnectionTable.handleNewPendingConnection(ConnectionTable.java:297)

 at
org.apache.geode.internal.tcp.ConnectionTable.getSharedConnection(ConnectionTable.java:408)

 at
org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:593)

 at
org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:825)

 at
org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:537)

 at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:326)

 at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:241)

 at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:596)

 at
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1558)

 at
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.send(GMSMembershipManager.java:1739)

 at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2849)

 at
org.

Re: Unit tests are hanging?

2019-08-07 Thread Kirk Lund 
Yep, that's the same test I'm seeing in the callstacks of my dunit tgz from
concourse...

Started @ 2019-08-07 07:25:18.494 +
2019-08-07 07:51:25.252 +
 org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
testMulticastAfterReconnect
Ended @ 2019-08-07 08:28:16.591 +

Thanks for digging into it!

On Wed, Aug 7, 2019 at 10:16 AM Ryan McMahon  wrote:

> I think the reflection and PowerMock warnings here are probably a red
> herring.  We pulled down the artifacts and found that the DUnit job is
> hanging due to stuck threads in a newer DUnit test.  I am not sure why it
> isn't failing the test but rather failing the entire job.  I believe Bruce
> Schuchart is going to dig into this a bit.  Analysis from Lynn
> Hughes-Godfrey:
>
> So, I downloaded
>
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/961
>
> to
>
>
> /Users/lhughesgodfrey/Downloads/distributedtestfiles-OpenJDK8-1.11.0-SNAPSHOT.0010/geode-core/build/distributedTest
>
> In that callstacks directory, you'll see dunit-hangs.txt ... and this
> test has been running for over an hour ...
>
> ```
>
> Started @ 2019-08-07 07:54:10.272 +
>
> 2019-08-07 08:22:25.440 +
> org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
> testMulticastAfterReconnect
>
> Ended @ 2019-08-07 08:58:50.243 +
>
> ```
>
> When I look at the callstacks (at the top level), I see this:
>
> ```
>
> "Pooled Waiting Message Processor 1" #213 daemon prio=5 os_prio=0
> tid=0x7fc738024000 nid=0x488 waiting for monitor entry
> [0x7fc7ec5f7000]
>
>java.lang.Thread.State: BLOCKED (on object monitor)
>
> at
> org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.startEventProcessing(GMSMembershipManager.java:1179)
>
> - waiting to lock <0xe03a75e0> (a java.lang.Object)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.readyForMessages(ClusterDistributionManager.java:1152)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.lambda$startThreads$10(ClusterDistributionManager.java:1129)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$87/1408912936.run(Unknown
> Source)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.doWaitingThread(ClusterDistributionManager.java:851)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$53/274751999.invoke(Unknown
> Source)
>
> at
> org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)
>
> at
> org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$49/1824093464.run(Unknown
> Source)
>
> at java.lang.Thread.run(Thread.java:748)
>
>Locked ownable synchronizers:
>
> - <0xe03a7660> (a
> java.util.concurrent.ThreadPoolExecutor$Worker)
>
> ```
>
> This thread holds the lock:
>
> ```
>
> "ReconnectThread" #193 prio=5 os_prio=0 tid=0x7fc7cc219000
> nid=0x46a waiting on condition [0x7fc78bdfb000]
>
>java.lang.Thread.State: TIMED_WAITING (sleeping)
>
> at java.lang.Thread.sleep(Native Method)
>
> at
> org.apache.geode.internal.tcp.Connection.createSender(Connection.java:959)
>
> at
> org.apache.geode.internal.tcp.ConnectionTable.handleNewPendingConnection(ConnectionTable.java:297)
>
> at
> org.apache.geode.internal.tcp.ConnectionTable.getSharedConnection(ConnectionTable.java:408)
>
> at
> org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:593)
>
> at
> org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:825)
>
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:537)
>
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:326)
>
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:241)
>
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:596)
>
> at
> org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1558)
>
> at
> org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.send(GMSMembershipManager.java:1739)
>
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2849)
>
> at
> org.apache.geode.distributed.internal.Cl

Re: Unit tests are hanging?

2019-08-07 Thread Ryan McMahon 
I think the reflection and PowerMock warnings here are probably a red
herring.  We pulled down the artifacts and found that the DUnit job is
hanging due to stuck threads in a newer DUnit test.  I am not sure why it
isn't failing the test but rather failing the entire job.  I believe Bruce
Schuchart is going to dig into this a bit.  Analysis from Lynn
Hughes-Godfrey:

So, I downloaded

https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/961

to

/Users/lhughesgodfrey/Downloads/distributedtestfiles-OpenJDK8-1.11.0-SNAPSHOT.0010/geode-core/build/distributedTest

In that callstacks directory, you'll see dunit-hangs.txt ... and this
test has been running for over an hour ...

```

Started @ 2019-08-07 07:54:10.272 +

2019-08-07 08:22:25.440 +
org.apache.geode.cache30.DistributedMulticastRegionDUnitTest
testMulticastAfterReconnect

Ended @ 2019-08-07 08:58:50.243 +

```

When I look at the callstacks (at the top level), I see this:

```

"Pooled Waiting Message Processor 1" #213 daemon prio=5 os_prio=0
tid=0x7fc738024000 nid=0x488 waiting for monitor entry
[0x7fc7ec5f7000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at 
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.startEventProcessing(GMSMembershipManager.java:1179)

- waiting to lock <0xe03a75e0> (a java.lang.Object)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.readyForMessages(ClusterDistributionManager.java:1152)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.lambda$startThreads$10(ClusterDistributionManager.java:1129)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$87/1408912936.run(Unknown
Source)

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.runUntilShutdown(ClusterDistributionManager.java:961)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.doWaitingThread(ClusterDistributionManager.java:851)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager$$Lambda$53/274751999.invoke(Unknown
Source)

at 
org.apache.geode.internal.logging.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:121)

at 
org.apache.geode.internal.logging.LoggingThreadFactory$$Lambda$49/1824093464.run(Unknown
Source)

at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:

- <0xe03a7660> (a
java.util.concurrent.ThreadPoolExecutor$Worker)

```

This thread holds the lock:

```

"ReconnectThread" #193 prio=5 os_prio=0 tid=0x7fc7cc219000
nid=0x46a waiting on condition [0x7fc78bdfb000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at 
org.apache.geode.internal.tcp.Connection.createSender(Connection.java:959)

at 
org.apache.geode.internal.tcp.ConnectionTable.handleNewPendingConnection(ConnectionTable.java:297)

at 
org.apache.geode.internal.tcp.ConnectionTable.getSharedConnection(ConnectionTable.java:408)

at 
org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:593)

at 
org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:825)

at 
org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:537)

at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:326)

at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:241)

at 
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:596)

at 
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1558)

at 
org.apache.geode.distributed.internal.membership.adapter.GMSMembershipManager.send(GMSMembershipManager.java:1739)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2849)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2776)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2813)

at 
org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523)

at 
org.apache.geode.distributed.internal.ReplyMessage.send(ReplyMessage.java:174)

at 
org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.sendReply(DistributedCacheOperation.java:1269)

at

Re: Unit tests are hanging?

2019-08-07 Thread Murtuza Boxwala 
That makes sense…I’m running 11 and I see them

> On Aug 7, 2019, at 1:11 PM, Helena Bales  wrote:
> 
> Kirk, as Lynn said in slack, these warnings are showing up with JDK11. So
> you'd probably see them locally if you used 11 instead of 8.
> 
> On Wed, Aug 7, 2019, 10:05 AM Kirk Lund  wrote:
> 
>> We previously decided to stop using PowerMock in any Geode tests and remove
>> it. Ryan and I created https://issues.apache.org/jira/browse/GEODE-6143
>> but
>> it's languishing because it takes a lot of time to strip out PowerMock from
>> a test.
>> 
>> I don't get any PowerMock warnings locally when I run unit tests.
>> 
>> On Wed, Aug 7, 2019 at 9:50 AM Murtuza Boxwala 
>> wrote:
>> 
>>> Yea.  I think it might be a red herring, because I am seeing those errors
>>> in every run, passing ones two…just double checked on
>>> 
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
>>> <
>>> 
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
 
>>> 
>>> If we don’t have any other ideas, though, it might be worth removing
>> them.
>>> 
>>> It is odd that you don’t see them locally. Let me see if I can get them
>> to
>>> show up…it might have something to do with the logging level.
>>> 
 On Aug 7, 2019, at 12:45 PM, Kirk Lund  wrote:
 
 I don't know if the PowerMock warnings are related but that's the only
 thing interesting in the output before "timeout exceeded".
 
 On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala 
>>> wrote:
 
> How down know these warnings are related to the builds hanging.  When
>> I
> was on CIO duty a couple days ago, I remember seeing this warning in a
> failing build, but then I looked back at passing builds on saw this as
>>> well.
> 
>> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
>> 
>> The build is broken in CI right now (for main CI and PRs). The
>> UnitTest
>> jobs are timing out so I assume there's a hang of some sort.
>> 
>> The WARNINGs appear to be related to PowerMock and begins with "An
> illegal
>> reflective access operation" in geode-assembly:test.
>> 
>> I'm running unit tests locally and haven't hit this yet, but maybe we
>> should consider disabling all PowerMock tests and increase priority
>> on
>> re-enabling them without PowerMock?
>> 
>>> Task :geode-assembly:test
>> WARNING: An illegal reflective access operation has occurred
>> WARNING: Illegal reflective access by
>> org.mockito.internal.util.reflection.AccessibilityChanger
>> 
> 
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>> to field java.io.File.path
>> WARNING: Please consider reporting this to the maintainers of
>> org.mockito.internal.util.reflection.AccessibilityChanger
>> WARNING: Use --illegal-access=warn to enable warnings of further
>>> illegal
>> reflective access operations
>> WARNING: All illegal access operations will be denied in a future
>>> release
>> WARNING: An illegal reflective access operation has occurred
>> WARNING: Illegal reflective access by
>> org.mockito.internal.util.reflection.AccessibilityChanger
>> 
> 
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>> to field java.io.File.path
>> WARNING: Please consider reporting this to the maintainers of
>> org.mockito.internal.util.reflection.AccessibilityChanger
>> WARNING: Use --illegal-access=warn to enable warnings of further
>>> illegal
>> reflective access operations
>> WARNING: All illegal access operations will be denied in a future
>>> release
>> 
>>> Task :geode-core:test
>> WARNING: An illegal reflective access operation has occurred
>> WARNING: Illegal reflective access by
>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
>> (file:/home/geode/geode/geode-core/build/classes/java/main/) to
>> method
>> java.lang.ThreadLocal.getMap(java.lang.Thread)
>> WARNING: Please consider reporting this to the maintainers of
>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
>> WARNING: Use --illegal-access=warn to enable warnings of further
>>> illegal
>> reflective access operations
>> WARNING: All illegal access operations will be denied in a future
>>> release
>> WARNING: An illegal reflective access operation has occurred
>> WARNING: Illegal reflective access by
>> org.mockito.internal.util.reflection.AccessibilityChanger
>> 
> 
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>> to field java.ut

Re: Unit tests are hanging?

2019-08-07 Thread Helena Bales 
Kirk, as Lynn said in slack, these warnings are showing up with JDK11. So
you'd probably see them locally if you used 11 instead of 8.

On Wed, Aug 7, 2019, 10:05 AM Kirk Lund  wrote:

> We previously decided to stop using PowerMock in any Geode tests and remove
> it. Ryan and I created https://issues.apache.org/jira/browse/GEODE-6143
> but
> it's languishing because it takes a lot of time to strip out PowerMock from
> a test.
>
> I don't get any PowerMock warnings locally when I run unit tests.
>
> On Wed, Aug 7, 2019 at 9:50 AM Murtuza Boxwala 
> wrote:
>
> > Yea.  I think it might be a red herring, because I am seeing those errors
> > in every run, passing ones two…just double checked on
> >
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> > <
> >
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> > >
> >
> > If we don’t have any other ideas, though, it might be worth removing
> them.
> >
> > It is odd that you don’t see them locally. Let me see if I can get them
> to
> > show up…it might have something to do with the logging level.
> >
> > > On Aug 7, 2019, at 12:45 PM, Kirk Lund  wrote:
> > >
> > > I don't know if the PowerMock warnings are related but that's the only
> > > thing interesting in the output before "timeout exceeded".
> > >
> > > On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala 
> > wrote:
> > >
> > >> How down know these warnings are related to the builds hanging.  When
> I
> > >> was on CIO duty a couple days ago, I remember seeing this warning in a
> > >> failing build, but then I looked back at passing builds on saw this as
> > well.
> > >>
> > >>> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
> > >>>
> > >>> The build is broken in CI right now (for main CI and PRs). The
> UnitTest
> > >>> jobs are timing out so I assume there's a hang of some sort.
> > >>>
> > >>> The WARNINGs appear to be related to PowerMock and begins with "An
> > >> illegal
> > >>> reflective access operation" in geode-assembly:test.
> > >>>
> > >>> I'm running unit tests locally and haven't hit this yet, but maybe we
> > >>> should consider disabling all PowerMock tests and increase priority
> on
> > >>> re-enabling them without PowerMock?
> > >>>
> >  Task :geode-assembly:test
> > >>> WARNING: An illegal reflective access operation has occurred
> > >>> WARNING: Illegal reflective access by
> > >>> org.mockito.internal.util.reflection.AccessibilityChanger
> > >>>
> > >>
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > >>> to field java.io.File.path
> > >>> WARNING: Please consider reporting this to the maintainers of
> > >>> org.mockito.internal.util.reflection.AccessibilityChanger
> > >>> WARNING: Use --illegal-access=warn to enable warnings of further
> > illegal
> > >>> reflective access operations
> > >>> WARNING: All illegal access operations will be denied in a future
> > release
> > >>> WARNING: An illegal reflective access operation has occurred
> > >>> WARNING: Illegal reflective access by
> > >>> org.mockito.internal.util.reflection.AccessibilityChanger
> > >>>
> > >>
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > >>> to field java.io.File.path
> > >>> WARNING: Please consider reporting this to the maintainers of
> > >>> org.mockito.internal.util.reflection.AccessibilityChanger
> > >>> WARNING: Use --illegal-access=warn to enable warnings of further
> > illegal
> > >>> reflective access operations
> > >>> WARNING: All illegal access operations will be denied in a future
> > release
> > >>>
> >  Task :geode-core:test
> > >>> WARNING: An illegal reflective access operation has occurred
> > >>> WARNING: Illegal reflective access by
> > >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> > >>> (file:/home/geode/geode/geode-core/build/classes/java/main/) to
> method
> > >>> java.lang.ThreadLocal.getMap(java.lang.Thread)
> > >>> WARNING: Please consider reporting this to the maintainers of
> > >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> > >>> WARNING: Use --illegal-access=warn to enable warnings of further
> > illegal
> > >>> reflective access operations
> > >>> WARNING: All illegal access operations will be denied in a future
> > release
> > >>> WARNING: An illegal reflective access operation has occurred
> > >>> WARNING: Illegal reflective access by
> > >>> org.mockito.internal.util.reflection.AccessibilityChanger
> > >>>
> > >>
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > >>> to field java.util.Date.fastTime
> > >>> WARNING: Please consider reporting this to the maintainers of
> > >>>

Re: Unit tests are hanging?

2019-08-07 Thread Kirk Lund 
We previously decided to stop using PowerMock in any Geode tests and remove
it. Ryan and I created https://issues.apache.org/jira/browse/GEODE-6143 but
it's languishing because it takes a lot of time to strip out PowerMock from
a test.

I don't get any PowerMock warnings locally when I run unit tests.

On Wed, Aug 7, 2019 at 9:50 AM Murtuza Boxwala  wrote:

> Yea.  I think it might be a red herring, because I am seeing those errors
> in every run, passing ones two…just double checked on
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> >
>
> If we don’t have any other ideas, though, it might be worth removing them.
>
> It is odd that you don’t see them locally. Let me see if I can get them to
> show up…it might have something to do with the logging level.
>
> > On Aug 7, 2019, at 12:45 PM, Kirk Lund  wrote:
> >
> > I don't know if the PowerMock warnings are related but that's the only
> > thing interesting in the output before "timeout exceeded".
> >
> > On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala 
> wrote:
> >
> >> How down know these warnings are related to the builds hanging.  When I
> >> was on CIO duty a couple days ago, I remember seeing this warning in a
> >> failing build, but then I looked back at passing builds on saw this as
> well.
> >>
> >>> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
> >>>
> >>> The build is broken in CI right now (for main CI and PRs). The UnitTest
> >>> jobs are timing out so I assume there's a hang of some sort.
> >>>
> >>> The WARNINGs appear to be related to PowerMock and begins with "An
> >> illegal
> >>> reflective access operation" in geode-assembly:test.
> >>>
> >>> I'm running unit tests locally and haven't hit this yet, but maybe we
> >>> should consider disabling all PowerMock tests and increase priority on
> >>> re-enabling them without PowerMock?
> >>>
>  Task :geode-assembly:test
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.io.File.path
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.io.File.path
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>>
>  Task :geode-core:test
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> >>> (file:/home/geode/geode/geode-core/build/classes/java/main/) to method
> >>> java.lang.ThreadLocal.getMap(java.lang.Thread)
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.util.Date.fastTime
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.powermock.reflect.internal.Wh

Re: Unit tests are hanging?

2019-08-07 Thread Helena Bales 
I'm on CIO duty today. There have been some network issues over the last
24h and all of the hangs in the CIO dashboard corresponded with network
failures. The jobs have been kicked off again, so hopefully we won't see
this issues again for a while.

Are you consistently seeing these hangs in the PR pipeline?

On Wed, Aug 7, 2019 at 9:50 AM Murtuza Boxwala  wrote:

> Yea.  I think it might be a red herring, because I am seeing those errors
> in every run, passing ones two…just double checked on
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> <
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
> >
>
> If we don’t have any other ideas, though, it might be worth removing them.
>
> It is odd that you don’t see them locally. Let me see if I can get them to
> show up…it might have something to do with the logging level.
>
> > On Aug 7, 2019, at 12:45 PM, Kirk Lund  wrote:
> >
> > I don't know if the PowerMock warnings are related but that's the only
> > thing interesting in the output before "timeout exceeded".
> >
> > On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala 
> wrote:
> >
> >> How down know these warnings are related to the builds hanging.  When I
> >> was on CIO duty a couple days ago, I remember seeing this warning in a
> >> failing build, but then I looked back at passing builds on saw this as
> well.
> >>
> >>> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
> >>>
> >>> The build is broken in CI right now (for main CI and PRs). The UnitTest
> >>> jobs are timing out so I assume there's a hang of some sort.
> >>>
> >>> The WARNINGs appear to be related to PowerMock and begins with "An
> >> illegal
> >>> reflective access operation" in geode-assembly:test.
> >>>
> >>> I'm running unit tests locally and haven't hit this yet, but maybe we
> >>> should consider disabling all PowerMock tests and increase priority on
> >>> re-enabling them without PowerMock?
> >>>
>  Task :geode-assembly:test
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.io.File.path
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.io.File.path
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>>
>  Task :geode-core:test
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> >>> (file:/home/geode/geode/geode-core/build/classes/java/main/) to method
> >>> java.lang.ThreadLocal.getMap(java.lang.Thread)
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>>
> >>
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> >>> to field java.util.Date.fastTime
> >>> WARNING: Please consider reporting this to the maintainers of
> >>> org.mockito.internal.util.reflection.AccessibilityChanger
> >>> WARNING: Use --illegal-access=warn to enable warnings of further
> illegal
> >>> reflective access operations
> >>> WARNING: All illegal access operations will be denied in a future
> release
> >>> WARNING: An illegal reflective access operation has occurred
> >>> WARNING: Illegal reflective access by
> >>> org.powermock.reflect.inte

Re: Unit tests are hanging?

2019-08-07 Thread Murtuza Boxwala 
Yea.  I think it might be a red herring, because I am seeing those errors in 
every run, passing ones two…just double checked on 
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948
 


If we don’t have any other ideas, though, it might be worth removing them.

It is odd that you don’t see them locally. Let me see if I can get them to show 
up…it might have something to do with the logging level.

> On Aug 7, 2019, at 12:45 PM, Kirk Lund  wrote:
> 
> I don't know if the PowerMock warnings are related but that's the only
> thing interesting in the output before "timeout exceeded".
> 
> On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala  wrote:
> 
>> How down know these warnings are related to the builds hanging.  When I
>> was on CIO duty a couple days ago, I remember seeing this warning in a
>> failing build, but then I looked back at passing builds on saw this as well.
>> 
>>> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
>>> 
>>> The build is broken in CI right now (for main CI and PRs). The UnitTest
>>> jobs are timing out so I assume there's a hang of some sort.
>>> 
>>> The WARNINGs appear to be related to PowerMock and begins with "An
>> illegal
>>> reflective access operation" in geode-assembly:test.
>>> 
>>> I'm running unit tests locally and haven't hit this yet, but maybe we
>>> should consider disabling all PowerMock tests and increase priority on
>>> re-enabling them without PowerMock?
>>> 
 Task :geode-assembly:test
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>>> to field java.io.File.path
>>> WARNING: Please consider reporting this to the maintainers of
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>>> to field java.io.File.path
>>> WARNING: Please consider reporting this to the maintainers of
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release
>>> 
 Task :geode-core:test
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by
>>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
>>> (file:/home/geode/geode/geode-core/build/classes/java/main/) to method
>>> java.lang.ThreadLocal.getMap(java.lang.Thread)
>>> WARNING: Please consider reporting this to the maintainers of
>>> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
>>> to field java.util.Date.fastTime
>>> WARNING: Please consider reporting this to the maintainers of
>>> org.mockito.internal.util.reflection.AccessibilityChanger
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release
>>> WARNING: An illegal reflective access operation has occurred
>>> WARNING: Illegal reflective access by
>>> org.powermock.reflect.internal.WhiteboxImpl
>>> 
>> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.powermock/powermock-reflect/2.0.0-beta.5/4ea415348f15620783a1f26343d6732adfa86bc8/powermock-reflect-2.0.0-beta.5.jar)
>>> to method java.lang.Object.finalize()
>>> WARNING: Please consider reporting this to the maintainers of
>>> org.powermock.reflect.internal.WhiteboxImpl
>>> WARNING: Use --illegal-access=warn to enable warnings of further illegal
>>> reflective access operations
>>> WARNING: All illegal access operations will be denied in a future release

Re: Unit tests are hanging?

2019-08-07 Thread Kirk Lund 
I don't know if the PowerMock warnings are related but that's the only
thing interesting in the output before "timeout exceeded".

On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala  wrote:

> How down know these warnings are related to the builds hanging.  When I
> was on CIO duty a couple days ago, I remember seeing this warning in a
> failing build, but then I looked back at passing builds on saw this as well.
>
> > On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
> >
> > The build is broken in CI right now (for main CI and PRs). The UnitTest
> > jobs are timing out so I assume there's a hang of some sort.
> >
> > The WARNINGs appear to be related to PowerMock and begins with "An
> illegal
> > reflective access operation" in geode-assembly:test.
> >
> > I'm running unit tests locally and haven't hit this yet, but maybe we
> > should consider disabling all PowerMock tests and increase priority on
> > re-enabling them without PowerMock?
> >
> >> Task :geode-assembly:test
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.mockito.internal.util.reflection.AccessibilityChanger
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > to field java.io.File.path
> > WARNING: Please consider reporting this to the maintainers of
> > org.mockito.internal.util.reflection.AccessibilityChanger
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access operations will be denied in a future release
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.mockito.internal.util.reflection.AccessibilityChanger
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > to field java.io.File.path
> > WARNING: Please consider reporting this to the maintainers of
> > org.mockito.internal.util.reflection.AccessibilityChanger
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access operations will be denied in a future release
> >
> >> Task :geode-core:test
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> > (file:/home/geode/geode/geode-core/build/classes/java/main/) to method
> > java.lang.ThreadLocal.getMap(java.lang.Thread)
> > WARNING: Please consider reporting this to the maintainers of
> > org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access operations will be denied in a future release
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.mockito.internal.util.reflection.AccessibilityChanger
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> > to field java.util.Date.fastTime
> > WARNING: Please consider reporting this to the maintainers of
> > org.mockito.internal.util.reflection.AccessibilityChanger
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access operations will be denied in a future release
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.powermock.reflect.internal.WhiteboxImpl
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.powermock/powermock-reflect/2.0.0-beta.5/4ea415348f15620783a1f26343d6732adfa86bc8/powermock-reflect-2.0.0-beta.5.jar)
> > to method java.lang.Object.finalize()
> > WARNING: Please consider reporting this to the maintainers of
> > org.powermock.reflect.internal.WhiteboxImpl
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access operations will be denied in a future release
> > WARNING: An illegal reflective access operation has occurred
> > WARNING: Illegal reflective access by
> > org.junit.contrib.java.lang.system.EnvironmentVariables
> >
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/com.github.stefanbirkner/system-rules/1.19.0/d541c9a1cff0dda32e2436c74562e2e4aa6c88cd/system-rules-1.19.0.jar)
> > to field java.util.Collections$UnmodifiableMap.m
> > WARNING: Please consider reporting this to the maintainers of
> > org.junit.contrib.java.lang.system.EnvironmentVariables
> > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > reflective access operations
> > WARNING: All illegal access

Re: Unit tests are hanging?

2019-08-07 Thread Murtuza Boxwala 
How down know these warnings are related to the builds hanging.  When I was on 
CIO duty a couple days ago, I remember seeing this warning in a failing build, 
but then I looked back at passing builds on saw this as well.

> On Aug 7, 2019, at 12:40 PM, Kirk Lund  wrote:
> 
> The build is broken in CI right now (for main CI and PRs). The UnitTest
> jobs are timing out so I assume there's a hang of some sort.
> 
> The WARNINGs appear to be related to PowerMock and begins with "An illegal
> reflective access operation" in geode-assembly:test.
> 
> I'm running unit tests locally and haven't hit this yet, but maybe we
> should consider disabling all PowerMock tests and increase priority on
> re-enabling them without PowerMock?
> 
>> Task :geode-assembly:test
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.mockito.internal.util.reflection.AccessibilityChanger
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> to field java.io.File.path
> WARNING: Please consider reporting this to the maintainers of
> org.mockito.internal.util.reflection.AccessibilityChanger
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.mockito.internal.util.reflection.AccessibilityChanger
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> to field java.io.File.path
> WARNING: Please consider reporting this to the maintainers of
> org.mockito.internal.util.reflection.AccessibilityChanger
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 
>> Task :geode-core:test
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> (file:/home/geode/geode/geode-core/build/classes/java/main/) to method
> java.lang.ThreadLocal.getMap(java.lang.Thread)
> WARNING: Please consider reporting this to the maintainers of
> org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.mockito.internal.util.reflection.AccessibilityChanger
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
> to field java.util.Date.fastTime
> WARNING: Please consider reporting this to the maintainers of
> org.mockito.internal.util.reflection.AccessibilityChanger
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.powermock.reflect.internal.WhiteboxImpl
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/org.powermock/powermock-reflect/2.0.0-beta.5/4ea415348f15620783a1f26343d6732adfa86bc8/powermock-reflect-2.0.0-beta.5.jar)
> to method java.lang.Object.finalize()
> WARNING: Please consider reporting this to the maintainers of
> org.powermock.reflect.internal.WhiteboxImpl
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.junit.contrib.java.lang.system.EnvironmentVariables
> (file:/home/geode/.gradle/caches/modules-2/files-2.1/com.github.stefanbirkner/system-rules/1.19.0/d541c9a1cff0dda32e2436c74562e2e4aa6c88cd/system-rules-1.19.0.jar)
> to field java.util.Collections$UnmodifiableMap.m
> WARNING: Please consider reporting this to the maintainers of
> org.junit.contrib.java.lang.system.EnvironmentVariables
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 
> timeout exceeded

Unit tests are hanging?

2019-08-07 Thread Kirk Lund 
The build is broken in CI right now (for main CI and PRs). The UnitTest
jobs are timing out so I assume there's a hang of some sort.

The WARNINGs appear to be related to PowerMock and begins with "An illegal
reflective access operation" in geode-assembly:test.

I'm running unit tests locally and haven't hit this yet, but maybe we
should consider disabling all PowerMock tests and increase priority on
re-enabling them without PowerMock?

> Task :geode-assembly:test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.mockito.internal.util.reflection.AccessibilityChanger
(file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
to field java.io.File.path
WARNING: Please consider reporting this to the maintainers of
org.mockito.internal.util.reflection.AccessibilityChanger
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.mockito.internal.util.reflection.AccessibilityChanger
(file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
to field java.io.File.path
WARNING: Please consider reporting this to the maintainers of
org.mockito.internal.util.reflection.AccessibilityChanger
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release

> Task :geode-core:test
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
(file:/home/geode/geode/geode-core/build/classes/java/main/) to method
java.lang.ThreadLocal.getMap(java.lang.Thread)
WARNING: Please consider reporting this to the maintainers of
org.apache.geode.distributed.internal.deadlock.UnsafeThreadLocal
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.mockito.internal.util.reflection.AccessibilityChanger
(file:/home/geode/.gradle/caches/modules-2/files-2.1/org.mockito/mockito-core/2.23.0/497ddb32fd5d01f9dbe99a2ec790aeb931dff1b1/mockito-core-2.23.0.jar)
to field java.util.Date.fastTime
WARNING: Please consider reporting this to the maintainers of
org.mockito.internal.util.reflection.AccessibilityChanger
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.powermock.reflect.internal.WhiteboxImpl
(file:/home/geode/.gradle/caches/modules-2/files-2.1/org.powermock/powermock-reflect/2.0.0-beta.5/4ea415348f15620783a1f26343d6732adfa86bc8/powermock-reflect-2.0.0-beta.5.jar)
to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of
org.powermock.reflect.internal.WhiteboxImpl
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.junit.contrib.java.lang.system.EnvironmentVariables
(file:/home/geode/.gradle/caches/modules-2/files-2.1/com.github.stefanbirkner/system-rules/1.19.0/d541c9a1cff0dda32e2436c74562e2e4aa6c88cd/system-rules-1.19.0.jar)
to field java.util.Collections$UnmodifiableMap.m
WARNING: Please consider reporting this to the maintainers of
org.junit.contrib.java.lang.system.EnvironmentVariables
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release

timeout exceeded