[GitHub] storm issue #2933: STORM-3309: Fix flaky tick tuple test

2019-01-05 Thread srdo 
Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2933
  
The test passed 100 iterations using `@RepeatedTest`


---

[GitHub] storm pull request #2933: STORM-3309: Fix flaky tick tuple test

2019-01-05 Thread srdo 
GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/2933

STORM-3309: Fix flaky tick tuple test

https://issues.apache.org/jira/browse/STORM-3309

I've made the following changes:
* When message timeout is disabled, the acker shouldn't time out tuples. 
Disable ticks for the acker if message timeouts are disabled
* The spout and bolt executors don't integrate with time simulation, in the 
sense that they don't require simulated time to increment in order to run. This 
is fine, but if they aren't going to pause for simulated time to increase, they 
also shouldn't potentially pause during initialization, waiting for Nimbus to 
activate the topology.
* InProcMessaging (used by the FeederSpout) will wait for the receiver to 
show up when sending the initial message. It waits at most 20 seconds, but if 
time simulation is enabled, it only waits 2. This is not enough for the 
topology/spout to start most of the time. I set the simulated time increment to 
match the real time spent waiting.
* The Zookeeper log drowns out any useful logging, set its level to WARN in 
storm-server

The TickTupleTest has been amended a bit. The problem with the current code 
is that LocalCluster.waitForIdle doesn't cover spout and bolt executor async 
loops, so we can end up in a situations where the test fails spuriously.

Example:
The test starts by incrementing cluster time until the bolt receives a tick 
tuple. Starting from t=0, it is possible that the test sets cluster time to 10 
and waits until the tick thread has added some tuples. The bolt thread runs 
independently of time simulation, and will consume the first tick at some 
arbitrary time. If we are unlucky, we can get the following sequence:

* 10 ticks are added by tick thread
* Bolt consumes first tick
* All threads covered by LocalCluster.waitForIdle (but not the bolt thread) 
are now idle, so the test exits the loop waiting for ticks
* The received ticks list is cleared
* The test stores what time the list was cleared at, advances cluster time 
by 1 and checks that a tick is received
* The bolt may just now be processing some of the previously queued ticks. 
This will cause the test to fail, because the bolt may receive multiple ticks 
at the same simulated time.

The replacement test instead uses a bootstrap tuple to verify that the 
executor (and tick thread) have started, and then increments the full tick 
interval. The tick interval is chosen so the tick thread will not produce any 
ticks until the test advances time enough to trigger one. This allows the test 
to verify that exactly one tick is received per second.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-3309

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2933


commit 6ca657d13d7f0ec50be2baed7fd8c70df5c9deca
Author: Stig Rohde Døssing 
Date:   2019-01-05T13:38:04Z

STORM-3309: Fix flaky tick tuple test




---

[GitHub] storm pull request #2930: STORM-3274: Migrates storm CLI to using argparse m...

2019-01-05 Thread srdo 
Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2930#discussion_r245480567
  
--- Diff: storm-client/pom.xml ---
@@ -240,6 +240,29 @@
 
 
 
+
+org.codehaus.mojo
+exec-maven-plugin
+
+
+
+python2.7
--- End diff --

I'd like the storm-client build to still work on Windows. One option would 
be to only run this execution when the OS is UNIX-y, e.g.
```


unix



```


---

[GitHub] storm pull request #2927: STORM-1307: Port testing4j_test.clj to Java

2019-01-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2927


---

[GitHub] storm pull request #2924: STORM-1289: Port integration-test.clj to Java

2019-01-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2924


---

[GitHub] storm issue #2924: STORM-1289: Port integration-test.clj to Java

2019-01-05 Thread srdo 
Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2924
  
Approved as part of https://github.com/apache/storm/pull/2927


---

[GitHub] storm pull request #2929: MINOR: Correct comment about overflow limiting in ...

2019-01-05 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2929


---

Re: Storm 2.0.0 release?

2019-01-05 Thread Kishorkumar Patil 
+1


On Sat, Jan 5, 2019 at 5:04 AM Stig Rohde Døssing 
wrote:

> +1
>
> Den lør. 5. jan. 2019 kl. 07.42 skrev Govind Menon <
> govindappume...@gmail.com>:
>
> > +1
> >
> > On Jan 4, 2019 23:46, "Roshan Naik" 
> wrote:
> >
> > +1
> >
> >
> > Sent from Yahoo Mail for iPhone
> >
> >
> >
> > On Friday, January 4, 2019, 9:02 PM, Ethan Li  >
> > wrote:
> >
> > +1 on this
> >
> > Ethan Li
> >
> > > On Jan 4, 2019, at 20:19, P. Taylor Goetz  wrote:
> > >
> > > If no one objects, I’ll kick off a release candidate.
> > >
> > > -Taylor
> > >
> > >> On Jan 4, 2019, at 9:06 PM, saurabh mimani 
> > wrote:
> > >>
> > >> Hey, Any approximate date for 2.0 release given there are no blockers?
> > >>
> > >>
> > >>
> > >> Best Regards
> > >>
> > >> Saurabh Kumar Mimani
> > >>
> > >>
> > >>
> > >>
> > >> On Sat, Dec 22, 2018 at 1:13 AM Stig Rohde Døssing <
> > stigdoess...@gmail.com>
> > >> wrote:
> > >>
> > >>> Looks good to me, the blockers list in JIRA is empty for 2.0.0.
> > >>>
> >  Den fre. 21. dec. 2018 kl. 19.52 skrev Bobby Evans <
> bo...@apache.org
> > >:
> > 
> >  I think all of the blockers are in now.  Please take a look and
> > >>> hopefully,
> >  we can get a release out soon.
> > 
> >  Thanks,
> > 
> >  Bobby
> > 
> > 
> > > On Fri, Dec 14, 2018 at 9:02 AM Bobby Evans 
> > wrote:
> > >
> > > Sorry I was out at a conference for the past week, and have been
> > heads
> >  down
> > > on a different project for a while before that.  I'll respond to
> the
> >  JIRA.
> > > I am happy to let it go in.
> > >
> > > Thanks,
> > >
> > > Bobby
> > >
> > > On Thu, Dec 13, 2018 at 1:07 PM Stig Rohde Døssing <
> >  stigdoess...@gmail.com
> > >>
> > > wrote:
> > >
> > >> I think STORM-2990/3279 is ready. Bobby had a question (
> > >> https://github.com/apache/storm/pull/2907#discussion_r234329136)
> > > regarding
> > >> whether Kafka offsets loop, but I wasn't sure where he was going
> > with
> >  it,
> > >> so I didn't want to merge prematurely.
> > >>
> > >> I agree that we can postpone STORM-2720. As far as I know it's
> > >>> waiting
> > > for
> > >> STORM-2990 to go in, since it's going to be touching the same
> code.
> > >>
> > >> Den tor. 13. dec. 2018 kl. 19.54 skrev Roshan Naik
> > >> :
> > >>
> > >>> Sounds like -  https://github.com/apache/storm/pull/2913
> >  (STORM-3290)
> > >> is
> > >>> merged -  https://github.com/apache/storm/pull/2908 (STORM-3276)
> > >>> is
> > >>> nearly complete and may need some small tweaks. -
> > >>> https://github.com/apache/storm/pull/2907  (STORM-2990,
> > >>> STORM-3279)
> > >>> appears ready to be committed ?
> > >>>
> > >>> and- https://github.com/apache/storm/pull/2911   - (STORM-2720)
> > > seems a
> > >>> bit inactive and may not be critical enough to wait on.
> > >>>
> > >>> -roshan
> > >>>  On Monday, November 26, 2018, 9:49:30 AM PST, Stig Rohde
> > >>> Døssing
> >  <
> > >>> stigdoess...@gmail.com> wrote:
> > >>>
> > >>> I would like to get at least
> > > https://github.com/apache/storm/pull/2913
> > >>> (breaking changes) and https://github.com/apache/storm/pull/2908
> > >>> (regression) in.
> > >>>
> > >>> I think it would be nice to also get
> > >>> https://github.com/apache/storm/pull/2907 and
> > >>> https://github.com/apache/storm/pull/2911 in, but if we're in a
> >  hurry
> > >> they
> > >>> could go in the next release.
> > >>>
> > >>> Den man. 26. nov. 2018 kl. 17.37 skrev Julien Nioche <
> > >>> lists.digitalpeb...@gmail.com>:
> > >>>
> >  Hi devs,
> > 
> >  Is there anything blocking the release of Storm 2.0? Any idea of
> >  when
> > >> it
> >  could happen?
> > 
> >  Thanks
> > 
> >  Julien
> > 
> >  --
> > 
> >  *Open Source Solutions for Text Engineering*
> > 
> >  http://www.digitalpebble.com
> >  http://digitalpebble.blogspot.com/
> >  #digitalpebble 
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> >
>
-- 
-Kishor

Re: Storm 2.0.0 release?

2019-01-05 Thread Stig Rohde Døssing 
+1

Den lør. 5. jan. 2019 kl. 07.42 skrev Govind Menon <
govindappume...@gmail.com>:

> +1
>
> On Jan 4, 2019 23:46, "Roshan Naik"  wrote:
>
> +1
>
>
> Sent from Yahoo Mail for iPhone
>
>
>
> On Friday, January 4, 2019, 9:02 PM, Ethan Li 
> wrote:
>
> +1 on this
>
> Ethan Li
>
> > On Jan 4, 2019, at 20:19, P. Taylor Goetz  wrote:
> >
> > If no one objects, I’ll kick off a release candidate.
> >
> > -Taylor
> >
> >> On Jan 4, 2019, at 9:06 PM, saurabh mimani 
> wrote:
> >>
> >> Hey, Any approximate date for 2.0 release given there are no blockers?
> >>
> >>
> >>
> >> Best Regards
> >>
> >> Saurabh Kumar Mimani
> >>
> >>
> >>
> >>
> >> On Sat, Dec 22, 2018 at 1:13 AM Stig Rohde Døssing <
> stigdoess...@gmail.com>
> >> wrote:
> >>
> >>> Looks good to me, the blockers list in JIRA is empty for 2.0.0.
> >>>
>  Den fre. 21. dec. 2018 kl. 19.52 skrev Bobby Evans  >:
> 
>  I think all of the blockers are in now.  Please take a look and
> >>> hopefully,
>  we can get a release out soon.
> 
>  Thanks,
> 
>  Bobby
> 
> 
> > On Fri, Dec 14, 2018 at 9:02 AM Bobby Evans 
> wrote:
> >
> > Sorry I was out at a conference for the past week, and have been
> heads
>  down
> > on a different project for a while before that.  I'll respond to the
>  JIRA.
> > I am happy to let it go in.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Thu, Dec 13, 2018 at 1:07 PM Stig Rohde Døssing <
>  stigdoess...@gmail.com
> >>
> > wrote:
> >
> >> I think STORM-2990/3279 is ready. Bobby had a question (
> >> https://github.com/apache/storm/pull/2907#discussion_r234329136)
> > regarding
> >> whether Kafka offsets loop, but I wasn't sure where he was going
> with
>  it,
> >> so I didn't want to merge prematurely.
> >>
> >> I agree that we can postpone STORM-2720. As far as I know it's
> >>> waiting
> > for
> >> STORM-2990 to go in, since it's going to be touching the same code.
> >>
> >> Den tor. 13. dec. 2018 kl. 19.54 skrev Roshan Naik
> >> :
> >>
> >>> Sounds like -  https://github.com/apache/storm/pull/2913
>  (STORM-3290)
> >> is
> >>> merged -  https://github.com/apache/storm/pull/2908 (STORM-3276)
> >>> is
> >>> nearly complete and may need some small tweaks. -
> >>> https://github.com/apache/storm/pull/2907  (STORM-2990,
> >>> STORM-3279)
> >>> appears ready to be committed ?
> >>>
> >>> and- https://github.com/apache/storm/pull/2911   - (STORM-2720)
> > seems a
> >>> bit inactive and may not be critical enough to wait on.
> >>>
> >>> -roshan
> >>>  On Monday, November 26, 2018, 9:49:30 AM PST, Stig Rohde
> >>> Døssing
>  <
> >>> stigdoess...@gmail.com> wrote:
> >>>
> >>> I would like to get at least
> > https://github.com/apache/storm/pull/2913
> >>> (breaking changes) and https://github.com/apache/storm/pull/2908
> >>> (regression) in.
> >>>
> >>> I think it would be nice to also get
> >>> https://github.com/apache/storm/pull/2907 and
> >>> https://github.com/apache/storm/pull/2911 in, but if we're in a
>  hurry
> >> they
> >>> could go in the next release.
> >>>
> >>> Den man. 26. nov. 2018 kl. 17.37 skrev Julien Nioche <
> >>> lists.digitalpeb...@gmail.com>:
> >>>
>  Hi devs,
> 
>  Is there anything blocking the release of Storm 2.0? Any idea of
>  when
> >> it
>  could happen?
> 
>  Thanks
> 
>  Julien
> 
>  --
> 
>  *Open Source Solutions for Text Engineering*
> 
>  http://www.digitalpebble.com
>  http://digitalpebble.blogspot.com/
>  #digitalpebble 
> 
> >>>
> >>
> >
> 
> >>>
>