[GitHub] storm issue #1082: STORM-1492 With nimbus.seeds set to default, a nimbus for...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1082
  
@revans2 Thanks for reviewing. Just got rebased against current master.


---

[GitHub] storm issue #2309: (1.x) STORM-2724 Shutdown ExecutorService in WaterMarkEve...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2309
  
@srdo Addressed review comments.


---

[GitHub] storm pull request #2304: STORM-2445 Topology log search refers supervisor I...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2304


---

[GitHub] storm issue #2310: STORM-2724 Shutdown ExecutorService in WaterMarkEventGene...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2310
  
@srdo Addressed review comments.


---

[GitHub] storm pull request #2309: (1.x) STORM-2724 Shutdown ExecutorService in Water...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2309#discussion_r137663649
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/windowing/WaterMarkEventGeneratorTest.java 
---
@@ -56,6 +57,13 @@ public void add(Event event) {
 waterMarkEventGenerator.start();
 }
 
+@After
+public void tearDown() {
+if (waterMarkEventGenerator != null) {
--- End diff --

Yes I just add it as defensive progressing, but can be removed. I'll 
address.


---

[GitHub] storm pull request #2309: (1.x) STORM-2724 Shutdown ExecutorService in Water...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2309#discussion_r137663770
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/windowing/WaterMarkEventGenerator.java ---
@@ -126,4 +135,20 @@ private void checkFailures() {
 public void start() {
 this.executorFuture = executorService.scheduleAtFixedRate(this, 
interval, interval, TimeUnit.MILLISECONDS);
 }
+
+public void shutdown() {
+LOG.debug("Shutting down WaterMarkEventGenerator");
+if (this.executorService != null) {
--- End diff --

Same here. Will address.


---

[GitHub] storm issue #2309: (1.x) STORM-2724 Shutdown ExecutorService in WaterMarkEve...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2309
  
@srdo 
Unfortunately we should always have two PRs for fixing storm-client or 
storm-server or storm-core because master and 1.x version lines are diverged.


---

[GitHub] storm issue #2289: STORM-2702: storm-loadgen

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2289
  
sure thanks.


---

[GitHub] storm issue #2289: STORM-2702: storm-loadgen

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2289
  
@roshannaik I merged this into master, but if you want any changes to it 
please just ask and I'll see what I can do.


---

[GitHub] storm pull request #2289: STORM-2702: storm-loadgen

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2289


---

[GitHub] storm pull request #2312: YSTORM-2725: Generic Resource Scheduling - initial...

[govind-menon]

GitHub user govind-menon opened a pull request:

https://github.com/apache/storm/pull/2312

YSTORM-2725: Generic Resource Scheduling - initial config changes and…

… TopologyBuilder API

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/govind-menon/storm YSTORM-2725

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2312.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2312


commit ab2fb4663e8548f37707bf423668bfc9e5607710
Author: Govind Menon 
Date:   2017-09-07T18:50:05Z

YSTORM-2725: Generic Resource Scheduling - initial config changes and 
TopologyBuilder API




---

Re: [VOTE] Release Apache Storm 1.0.5 (rc1)

[Stig Rohde Døssing]

* Extracted binary zip and built storm-starter using the included example
code and the Nexus staging repo.
* After working around https://issues.apache.org/jira/browse/STORM-2451 (I
forgot to pull it into 1.0.x after the 1.0.4 release, and don't want to
delay this release for that issue), started Nimbus, one supervisor and UI
daemons.
* Verified storm-local is no longer relative to the directory where storm
nimbus is executed.
* Deployed WordCountTopology and verified no errors in the worker logs, and
that the topology produces word counts in the log.
* Clicked around a bit in UI for the topology and components
* Killed the topology.

+1 (binding)

2017-09-07 10:56 GMT+02:00 Alexandre Vermeerbergen :

> Hello,
>
> (non binding)
>
> [x]  0 No opinion
>
> => we are currently running with 1.1.0, waiting for include
> https://issues.apache.org/jira/browse/STORM-2648 fix was unfortunately
> shifted in 1.2.0 (although lack of acks for KafkaSpout when it's running in
> autocommit mode sounds more like a bug than a feature-thing), so we skip
> testing 1.0.5
>
> Best regards,
> Alexandre Vermeerbergen
>
>
> 2017-09-07 2:38 GMT+02:00 Jungtaek Lim :
>
> > +1 (binding)
> >
> > > source
> >
> > - verify file (signature, MD5, SHA)
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - extract file
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - diff-ing extracted files between tar.gz and zip : OK
> >
> > - build source with JDK 7
> > -- source, tar.gz : OK
> >
> > - build source dist
> > -- source, tar.gz : OK
> >
> > - build binary dist
> > -- source, tar.gz : OK
> >
> > > binary
> >
> > - verify file (signature, MD5, SHA)
> > -- binary, tar.gz : OK
> > -- binary, zip : OK
> >
> > - extract file
> > -- source, tar.gz : OK
> > -- source, zip : OK
> >
> > - diff-ing extracted files between tar.gz and zip : OK
> >
> > - launch daemons : OK
> >
> > - run RollingTopWords (local) : OK
> >
> > - run RollingTopWords (remote) : OK
> >   - activate / deactivate / rebalance / kill : OK
> >   - logviewer (worker dir, daemon dir) : OK
> >   - change log level : OK
> >   - thread dump, heap dump, restart worker :
> >   - log search : OK
> >
> > Most critical issue in 1.0.5 has been verified: upgrade from Storm 1.0.4
> to
> > Storm 1.0.5 RC1 succeeds, though I've not tried RU. I tested this via
> > shutting down 1.0.4 cluster and starting 1.0.5 cluster which two clusters
> > are sharing storm-local directory and ZK path.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 9월 7일 (목) 오전 3:29, P. Taylor Goetz 님이 작성:
> >
> > > This is a call to vote on releasing Apache Storm 1.0.5 (rc1)
> > >
> > > Full list of changes in this release:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.5-rc1/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v1.0.5:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > 239fd1522114c4928ed99b4ab5bcbf027540d5e7;hb=
> a445446e46e2228fabb83368cc25cf
> > 5422a2c400
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.5-rc1/apache-storm-1.0.5-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.5-rc1/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1051
> > >
> > > Please vote on releasing this package as Apache Storm 1.0.5.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.0.5
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor
> >
>

[GitHub] storm pull request #2302: STORM-2635 Deep log search doesn’t work when the...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2302


---

[GitHub] storm pull request #2261: STORM-2678 Improve performance of LoadAwareShuffle...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2261


---

[GitHub] storm issue #2289: STORM-2702: storm-loadgen

[knusbaum]

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/2289
  
+1


---

[GitHub] storm issue #2310: STORM-2724 Shutdown ExecutorService in WaterMarkEventGene...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2310
  
+1, same nits as on the 1.x version


---

[GitHub] storm pull request #2311: [STORM-2729] Fix NPE in WorkerState runWorkerStart...

[Ethanlm]

GitHub user Ethanlm opened a pull request:

https://github.com/apache/storm/pull/2311

[STORM-2729] Fix NPE in WorkerState runWorkerStartHooks and 
runWorkerShutdownHooks methods

worker_hooks is an optional field in StormTopology struct (storm.thrift). 
So when we call get_worker_hooks(), we need to check is_set_worker_hooks().

We probably want to pay attention to optional fields when we write the code 
because this can be a common issue for all thrift structs. We should always 
check is_set_xxx before we call get_xxx if xxx is an optional field.

For details, see https://issues.apache.org/jira/browse/STORM-2729.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ethanlm/storm STORM-2729

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2311.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2311


commit e7e1177992ba3a151d3162b8c6de041bf6708005
Author: Ethan Li 
Date:   2017-09-07T17:15:50Z

[STORM-2729] Fix NPE in WorkerState runWorkerStartHooks and 
runWorkerShutdownHooks methods




---

Re: Running topologies in LocalCluster mode leaves junk in ${user.dir}/logs

[Alexandre Vermeerbergen]

Hello Bobby,

Thanks a lot for your rapid answer.
At least there's hope for improvement in LocalCluster when 2.x will be
available as a stable release.
For 1.x, you mention that I could use Clojure APIs.
But unless I'm mistaking, Clojure is just "yet another JVM compliant"
language, so isn't there a way to use this API with Java (maybe at the
expense of using Java reflection API?)

Best regards,
Alexandre


2017-09-06 22:29 GMT+02:00 Bobby Evans :

> Under 2.x LocalCluster is much more fully featured and you can set daemon
> configs using the LocalCluster.Builder.withDaemonConf.  But for all other
> versions of storm the only way to change things like the daemon conf are
> through the clojure API, which I assume you don't want to use.
>
> So sorry but no there is no better way right now unless you want to move to
> an unreleased version of Storm.
>
> If you do want to clean it up and make support for something like this more
> official I would love to see it happen, but I don't really have a lot of
> time to make the changes myself right now.
>
> -
> Bobby
>
>
> On Wed, Sep 6, 2017 at 3:05 PM Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
>
> > Hello,
> >
> > I have noticed that when I run topologies in LocalCluster mode (for
> > automated test purposes), there are residual files left in a directory
> > corresponding to System.getProperty("user.dir")+"/logs".
> >
> > Inside this directory, a subdirectory called "workers-artifacts" is
> created
> > when the topology is submitted, and it contains again a subdir then some
> > files ; nothing is cleaned up when the topology shuts down.
> >
> > This causes our tests to fail in our automated test replay system,
> because
> > tests are ran with a special user id which has no write permission in
> > System.getProperty("user.dir")+"/logs".
> >
> > I have tried change this behavior by using these setters on the Config
> > object passed to LocalCluster.submitTopology:
> >
> > Config conf = new Config();
> > conf.setDebug(true);
> >
> > conf.put(Config.STORM_LOCAL_DIR, "/tmp/localdir");
> > conf.put(Config.STORM_WORKERS_ARTIFACTS_DIR, "/tmp/localdir");
> >
> > but it changed nothing.
> >
> > I finally found a by-pass by putting this line at the beginning of our
> test
> > code:
> >
> > System.setProperty("user.dir", "/tmp/localdir");
> >
> > But I feel that it's a dirty hack.
> >
> > What is the official way to control LocalCluster's behavior which regards
> > to these directories & files ?
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
>

[GitHub] storm issue #2289: STORM-2702: storm-loadgen

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2289
  
@knusbaum do you have any other comments?


---

[GitHub] storm pull request #2309: (1.x) STORM-2724 Shutdown ExecutorService in Water...

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2309#discussion_r137563632
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/windowing/WaterMarkEventGenerator.java ---
@@ -126,4 +135,20 @@ private void checkFailures() {
 public void start() {
 this.executorFuture = executorService.scheduleAtFixedRate(this, 
interval, interval, TimeUnit.MILLISECONDS);
 }
+
+public void shutdown() {
+LOG.debug("Shutting down WaterMarkEventGenerator");
+if (this.executorService != null) {
--- End diff --

Nit: This doesn't seem like it can be null?


---

[GitHub] storm pull request #2309: (1.x) STORM-2724 Shutdown ExecutorService in Water...

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2309#discussion_r137560854
  
--- Diff: 
storm-core/test/jvm/org/apache/storm/windowing/WaterMarkEventGeneratorTest.java 
---
@@ -56,6 +57,13 @@ public void add(Event event) {
 waterMarkEventGenerator.start();
 }
 
+@After
+public void tearDown() {
+if (waterMarkEventGenerator != null) {
--- End diff --

Nit: I don't think this can be null


---

[GitHub] storm pull request #2310: STORM-2724 Shutdown ExecutorService in WaterMarkEv...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/2310

STORM-2724 Shutdown ExecutorService in WaterMarkEventGenerator in shu…

…tdown phase

* also names unnamed executor services in windowing package

For 1.x version: #2309 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2724

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2310


commit f3a7be1a16c9fe9208c2af94505ea29b2b327a8b
Author: Jungtaek Lim 
Date:   2017-09-07T14:36:10Z

STORM-2724 Shutdown ExecutorService in WaterMarkEventGenerator in shutdown 
phase

* also names unnamed executor services in windowing package




---

[GitHub] storm pull request #2309: (1.x) STORM-2724 Shutdown ExecutorService in Water...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/2309

(1.x) STORM-2724 Shutdown ExecutorService in WaterMarkEventGenerator in 
shu…

…tdown phase

* also names unnamed executor services in windowing package

This issue affects all 1.x version lines. (starting at 1.0.0)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2724-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2309.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2309


commit 4db94248f711f19409bdb4f6739bfaa5feabcc03
Author: Jungtaek Lim 
Date:   2017-09-07T14:36:10Z

STORM-2724 Shutdown ExecutorService in WaterMarkEventGenerator in shutdown 
phase

* also names unnamed executor services in windowing package




---

Re: [VOTE] Release Apache Storm 1.0.5 (rc1)

[Alexandre Vermeerbergen]

Hello,

(non binding)

[x]  0 No opinion

=> we are currently running with 1.1.0, waiting for include
https://issues.apache.org/jira/browse/STORM-2648 fix was unfortunately
shifted in 1.2.0 (although lack of acks for KafkaSpout when it's running in
autocommit mode sounds more like a bug than a feature-thing), so we skip
testing 1.0.5

Best regards,
Alexandre Vermeerbergen


2017-09-07 2:38 GMT+02:00 Jungtaek Lim :

> +1 (binding)
>
> > source
>
> - verify file (signature, MD5, SHA)
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - extract file
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - build source with JDK 7
> -- source, tar.gz : OK
>
> - build source dist
> -- source, tar.gz : OK
>
> - build binary dist
> -- source, tar.gz : OK
>
> > binary
>
> - verify file (signature, MD5, SHA)
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - extract file
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - launch daemons : OK
>
> - run RollingTopWords (local) : OK
>
> - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker dir, daemon dir) : OK
>   - change log level : OK
>   - thread dump, heap dump, restart worker :
>   - log search : OK
>
> Most critical issue in 1.0.5 has been verified: upgrade from Storm 1.0.4 to
> Storm 1.0.5 RC1 succeeds, though I've not tried RU. I tested this via
> shutting down 1.0.4 cluster and starting 1.0.5 cluster which two clusters
> are sharing storm-local directory and ZK path.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 9월 7일 (목) 오전 3:29, P. Taylor Goetz 님이 작성:
>
> > This is a call to vote on releasing Apache Storm 1.0.5 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 0.5-rc1/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.0.5:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> 239fd1522114c4928ed99b4ab5bcbf027540d5e7;hb=a445446e46e2228fabb83368cc25cf
> 5422a2c400
> >
> > The source archive being voted upon can be found here:
> >
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 0.5-rc1/apache-storm-1.0.5-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.5-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1051
> >
> > Please vote on releasing this package as Apache Storm 1.0.5.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.0.5
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>