[GitHub] storm pull request: [STORM-1420] solr CountBasedCommit should rese...

[vesense]

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/978#issuecomment-171244000
  
Thanks @revans2 @hmcl for your explaining. These changes are not necessary. 
I will close this issue as 'Not A Problem'.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1420) solr CountBasedCommit should reset count=0 after commited

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095951#comment-15095951
 ] 

ASF GitHub Bot commented on STORM-1420:
---

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/978#issuecomment-171244000
  
Thanks @revans2 @hmcl for your explaining. These changes are not necessary. 
I will close this issue as 'Not A Problem'.


> solr CountBasedCommit should reset count=0 after commited
> -
>
> Key: STORM-1420
> URL: https://issues.apache.org/jira/browse/STORM-1420
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> The variable count should reset to 0, if not so it will increase infinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: 1.x storm and jstorm merger

[刘键(Basti Liu)]

Hi Bobby,

For dependencies, we have not done much work on these modules except the ones 
related to zookeeper (cluster.clj and zookeeper.clj).
When doing the migration of Nimbus and Supervisor, we just updated the name of 
relative functions called in Nimbus and Supervisor by the 
guidelines you gave below, and left empty implementation for these in relative 
modules.
So, please feel free to start on utils.clj and config.clj.

Following is the detailed status for Nimbus, Supervisor and zookeeper modules 
currently. We will also update the status in JIRAs.
- Nimbus: 30% The structure of state transition and service handler are almost 
done. Blob store, replication between Nimbus and security have not been started 
yet.
- Supervisor: 90% Only health check was left.
- Zookeeper: 40% Most interfaces of reading operation have been done.

Regards
Basti

-Original Message-
From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
Sent: Tuesday, January 12, 2016 10:35 PM
To: 刘键(Basti Liu); dev@storm.apache.org
Subject: Re: 1.x storm and jstorm merger

For the migration of fixes, we have the policy that fixes need to go into 
master before they can be merged into previous branches, so with that the fix 
would go into master and anyone who wants the fix on an older branch would be 
responsible for porting it to clojure.
It is great to hear that there is progress being made on Nimbus and the 
Supervisor, but those are very large pieces of code with lots of dependencies.  
 I would really like to sync up with you and what you are doing so we don't get 
too much duplicated efforts.  Specifically my team is starting on utils.clj and 
config.clj.  But if you have already done some/all of the work for them I would 
rather use that work instead.  I also want to check changes in frequently with 
smaller differences.  Less differences means we should be able to find bug 
sooner and adjust accordingly.  Is there any way you could take some of what 
you have done, even if it is not complete and put up pull requests for the 
portions that do work and can be swapped out? Particularly in underlying files. 
 Or at least put it in a place that we can look at it? 
 - Bobby 

On Monday, January 11, 2016 9:31 PM, 刘键(Basti Liu) 
 wrote:
 

 Hi Bobby,

It is great to see that we are going to finalize Storm 1.x and start the 
migration.
Just to synchronize current status of migration in Alibaba, the migration of 
Nimbus and Supervisor part is in progress(around 50% is completed).
Besides it, we'd like to confirm the handling of following bug fixes in 
branch-1.x. Who is responsible to migrate the fix from branch-1.x to mater 
branch(2.0.0-snapshot)? the owner of the JIRA in branch-1.x or the owner of 
corresponding migration JIRA in master branch?

Regards
Basti

-Original Message-
From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
Sent: Tuesday, January 12, 2016 5:39 AM
To: Dev
Subject: 1.x storm and jstorm merger

I think we are finally at the point were this is going to start happening.
I have merged in as many JIRA/PULL requests that looked like they were ready.  
The most disruptive of these is probably STORM-1202, which translated 
backtype.storm to org.apache.storm and storm.trident to 
org.apache.storm.trident  There is some hack code that we will remove in the 
future that when submitting a topology using the new client it will translate 
your jar for you to the new namespaces.  You can enable it by setting 
`client.jartransformer.class` to `org.apache.storm.hack.StormShadeTransformer`

With that I changed the 0.11.0 tag to 1.0.0.  And I created a branch-1.x branch 
in the main repo.  I have not started creating a release candidate just yet as 
there still are a few outstanding bugs that I would like to see resolved before 
hand.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20and%20priority%20%3E%3D%20Blocker

Of them only STORM-1452 really feels like a blocker, but I am open to others 
opinions as we work towards a release.  I would encourage everyone to kick the 
tires on it and file JIRA if you do find any issues.



I also changed the version on master to 2.0.0-SNAPSHOT in preparation for the 
migration to java and the JStorm merger.  This means that we are now open to 
start pulling in java migration pull requests as outlined on the wiki
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
I know that my team is going to start working on this process now to try and 
shorten the time of the transition as much as possible.  From this point on 
until the transition to java is complete there will be a moratorium on new 
features going into storm.  This is not a hard moratorium.  If your changes 
only touch java code, like in the external projects, etc.  I don't think anyone 
will complain if new features go in, but this is mostly to reduce churn in the 
code so the transition 

[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1010#issuecomment-171383682
  
+1 pending travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096708#comment-15096708
 ] 

ASF GitHub Bot commented on STORM-1470:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1010#issuecomment-171383682
  
+1 pending travis


> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

[Garrick Dasbach (JIRA)]

Garrick Dasbach created STORM-1471:
--

 Summary: Make FetchRequestBuilder.minBytes a configurable 
parameter in SpoutConfig
 Key: STORM-1471
 URL: https://issues.apache.org/jira/browse/STORM-1471
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-kafka
Reporter: Garrick Dasbach


We currently have an issue in our storm cluster where our Kafka brokers are 
under heavy load due to too many fetch requests from storm.  We've narrowed the 
problem to the way Fetch Requests are build in KafkaUtils.  When using the 
FetchRequestBuilder, storm provides overrides for all the properties except 
minBytes.  The default for that field is 0 (even though the Kafka default for 
the high-level consumer is 1).  When paired with a maxWait > 0, this creates a 
situation where the broker can immediately return a response without waiting 
(due to minBytes 0).  This puts a heavy load on the brokers and defeats the 
purpose of any long polling.

By making this a SpoutConfig option, it will allow the user to set that as 
appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 1.x storm and jstorm merger

[Haohui Mai]

I wonder, is it possible to separate the ZooKeeper part of the code to
make things easier to review?

~Haohui

On Wed, Jan 13, 2016 at 6:39 AM, Bobby Evans
 wrote:
> That is great to hear.  We will work on getting your dependencies put in 
> place ASAP.
>  - Bobby
>
> On Wednesday, January 13, 2016 6:03 AM, 刘键(Basti Liu) 
>  wrote:
>
>
>  Hi Bobby,
>
> For dependencies, we have not done much work on these modules except the ones 
> related to zookeeper (cluster.clj and zookeeper.clj).
> When doing the migration of Nimbus and Supervisor, we just updated the name 
> of relative functions called in Nimbus and Supervisor by the
> guidelines you gave below, and left empty implementation for these in 
> relative modules.
> So, please feel free to start on utils.clj and config.clj.
>
> Following is the detailed status for Nimbus, Supervisor and zookeeper modules 
> currently. We will also update the status in JIRAs.
> - Nimbus: 30% The structure of state transition and service handler are 
> almost done. Blob store, replication between Nimbus and security have not 
> been started yet.
> - Supervisor: 90% Only health check was left.
> - Zookeeper: 40% Most interfaces of reading operation have been done.
>
> Regards
> Basti
>
> -Original Message-
> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> Sent: Tuesday, January 12, 2016 10:35 PM
> To: 刘键(Basti Liu); dev@storm.apache.org
> Subject: Re: 1.x storm and jstorm merger
>
> For the migration of fixes, we have the policy that fixes need to go into 
> master before they can be merged into previous branches, so with that the fix 
> would go into master and anyone who wants the fix on an older branch would be 
> responsible for porting it to clojure.
> It is great to hear that there is progress being made on Nimbus and the 
> Supervisor, but those are very large pieces of code with lots of 
> dependencies.  I would really like to sync up with you and what you are doing 
> so we don't get too much duplicated efforts.  Specifically my team is 
> starting on utils.clj and config.clj.  But if you have already done some/all 
> of the work for them I would rather use that work instead.  I also want to 
> check changes in frequently with smaller differences.  Less differences means 
> we should be able to find bug sooner and adjust accordingly.  Is there any 
> way you could take some of what you have done, even if it is not complete and 
> put up pull requests for the portions that do work and can be swapped out? 
> Particularly in underlying files.  Or at least put it in a place that we can 
> look at it?
>  - Bobby
>
> On Monday, January 11, 2016 9:31 PM, 刘键(Basti Liu) 
>  wrote:
>
>
>  Hi Bobby,
>
> It is great to see that we are going to finalize Storm 1.x and start the 
> migration.
> Just to synchronize current status of migration in Alibaba, the migration of 
> Nimbus and Supervisor part is in progress(around 50% is completed).
> Besides it, we'd like to confirm the handling of following bug fixes in 
> branch-1.x. Who is responsible to migrate the fix from branch-1.x to mater 
> branch(2.0.0-snapshot)? the owner of the JIRA in branch-1.x or the owner of 
> corresponding migration JIRA in master branch?
>
> Regards
> Basti
>
> -Original Message-
> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> Sent: Tuesday, January 12, 2016 5:39 AM
> To: Dev
> Subject: 1.x storm and jstorm merger
>
> I think we are finally at the point were this is going to start happening.
> I have merged in as many JIRA/PULL requests that looked like they were ready. 
>  The most disruptive of these is probably STORM-1202, which translated 
> backtype.storm to org.apache.storm and storm.trident to 
> org.apache.storm.trident  There is some hack code that we will remove in the 
> future that when submitting a topology using the new client it will translate 
> your jar for you to the new namespaces.  You can enable it by setting 
> `client.jartransformer.class` to `org.apache.storm.hack.StormShadeTransformer`
>
> With that I changed the 0.11.0 tag to 1.0.0.  And I created a branch-1.x 
> branch in the main repo.  I have not started creating a release candidate 
> just yet as there still are a few outstanding bugs that I would like to see 
> resolved before hand.
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20and%20priority%20%3E%3D%20Blocker
>
> Of them only STORM-1452 really feels like a blocker, but I am open to others 
> opinions as we work towards a release.  I would encourage everyone to kick 
> the tires on it and file JIRA if you do find any issues.
>
>
>
> I also changed the version on master to 2.0.0-SNAPSHOT in preparation for the 
> migration to java and the JStorm merger.  This means that we are now open to 
> start pulling in java migration pull requests as outlined on the wiki
> 

HdfsSpout

[Matthias J. Sax]

Hi,

if I am not wrong, Storm comes only with an HdfsBolt but not with a
HdfsSpout. Would it be worth to add a HdfsSpout? Not sure how common
reading data from HDFS in Strom is.

-Matthias



signature.asc
Description: OpenPGP digital signature

[jira] [Commented] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096943#comment-15096943
 ] 

ASF GitHub Bot commented on STORM-1466:
---

Github user redsanket commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171424617
  
+1 NB


> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 1.x storm and jstorm merger

[Bobby Evans]

Of course.  the JIRA I filed are just suggestions.  If we want to rearrange 
things that is fine, we just need to update JIRA accordingly.
 - Bobby 

On Wednesday, January 13, 2016 1:34 PM, Haohui Mai  
wrote:
 

 I wonder, is it possible to separate the ZooKeeper part of the code to
make things easier to review?

~Haohui

On Wed, Jan 13, 2016 at 6:39 AM, Bobby Evans
 wrote:
> That is great to hear.  We will work on getting your dependencies put in 
> place ASAP.
>  - Bobby
>
>    On Wednesday, January 13, 2016 6:03 AM, 刘键(Basti Liu) 
> wrote:
>
>
>  Hi Bobby,
>
> For dependencies, we have not done much work on these modules except the ones 
> related to zookeeper (cluster.clj and zookeeper.clj).
> When doing the migration of Nimbus and Supervisor, we just updated the name 
> of relative functions called in Nimbus and Supervisor by the
> guidelines you gave below, and left empty implementation for these in 
> relative modules.
> So, please feel free to start on utils.clj and config.clj.
>
> Following is the detailed status for Nimbus, Supervisor and zookeeper modules 
> currently. We will also update the status in JIRAs.
> - Nimbus: 30% The structure of state transition and service handler are 
> almost done. Blob store, replication between Nimbus and security have not 
> been started yet.
> - Supervisor: 90% Only health check was left.
> - Zookeeper: 40% Most interfaces of reading operation have been done.
>
> Regards
> Basti
>
> -Original Message-
> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> Sent: Tuesday, January 12, 2016 10:35 PM
> To: 刘键(Basti Liu); dev@storm.apache.org
> Subject: Re: 1.x storm and jstorm merger
>
> For the migration of fixes, we have the policy that fixes need to go into 
> master before they can be merged into previous branches, so with that the fix 
> would go into master and anyone who wants the fix on an older branch would be 
> responsible for porting it to clojure.
> It is great to hear that there is progress being made on Nimbus and the 
> Supervisor, but those are very large pieces of code with lots of 
> dependencies.  I would really like to sync up with you and what you are doing 
> so we don't get too much duplicated efforts.  Specifically my team is 
> starting on utils.clj and config.clj.  But if you have already done some/all 
> of the work for them I would rather use that work instead.  I also want to 
> check changes in frequently with smaller differences.  Less differences means 
> we should be able to find bug sooner and adjust accordingly.  Is there any 
> way you could take some of what you have done, even if it is not complete and 
> put up pull requests for the portions that do work and can be swapped out? 
> Particularly in underlying files.  Or at least put it in a place that we can 
> look at it?
>  - Bobby
>
>    On Monday, January 11, 2016 9:31 PM, 刘键(Basti Liu) 
> wrote:
>
>
>  Hi Bobby,
>
> It is great to see that we are going to finalize Storm 1.x and start the 
> migration.
> Just to synchronize current status of migration in Alibaba, the migration of 
> Nimbus and Supervisor part is in progress(around 50% is completed).
> Besides it, we'd like to confirm the handling of following bug fixes in 
> branch-1.x. Who is responsible to migrate the fix from branch-1.x to mater 
> branch(2.0.0-snapshot)? the owner of the JIRA in branch-1.x or the owner of 
> corresponding migration JIRA in master branch?
>
> Regards
> Basti
>
> -Original Message-
> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> Sent: Tuesday, January 12, 2016 5:39 AM
> To: Dev
> Subject: 1.x storm and jstorm merger
>
> I think we are finally at the point were this is going to start happening.
> I have merged in as many JIRA/PULL requests that looked like they were ready. 
>  The most disruptive of these is probably STORM-1202, which translated 
> backtype.storm to org.apache.storm and storm.trident to 
> org.apache.storm.trident  There is some hack code that we will remove in the 
> future that when submitting a topology using the new client it will translate 
> your jar for you to the new namespaces.  You can enable it by setting 
> `client.jartransformer.class` to `org.apache.storm.hack.StormShadeTransformer`
>
> With that I changed the 0.11.0 tag to 1.0.0.  And I created a branch-1.x 
> branch in the main repo.  I have not started creating a release candidate 
> just yet as there still are a few outstanding bugs that I would like to see 
> resolved before hand.
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20and%20priority%20%3E%3D%20Blocker
>
> Of them only STORM-1452 really feels like a blocker, but I am open to others 
> opinions as we work towards a release.  I would encourage everyone to kick 
> the tires on it and file JIRA if you do find any issues.
>
>
>
> I 

[GitHub] storm pull request: STORM-1466: Move the org.apache.thrift7 namesp...

[redsanket]

Github user redsanket commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171424617
  
+1 NB


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

[Boyang Jerry Peng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Jerry Peng updated STORM-1450:
-
Description: 
Bug fixes:

Minor bug in displaying the assigned resources for a supervisor on the UI
this line:
_ (reset! (:node-id->resources nimbus) (.getSupervisorsResourcesMap cluster))

needs to be

_ (reset! (:node-id->resources nimbus) (merge (deref (:node-id->resources 
nimbus)) (.getSupervisorsResourcesMap cluster)))




> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>
> Bug fixes:
> Minor bug in displaying the assigned resources for a supervisor on the UI
> this line:
> _ (reset! (:node-id->resources nimbus) (.getSupervisorsResourcesMap cluster))
> needs to be
> _ (reset! (:node-id->resources nimbus) (merge (deref (:node-id->resources 
> nimbus)) (.getSupervisorsResourcesMap cluster)))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

[d2r]

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1012

[STORM-1452] Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm 
storm-1452-profiler-broken-by-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1012.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1012


commit ae647a2a93af203778f273316ec6bb3f149ed56d
Author: Derek Dagit 
Date:   2016-01-13T22:34:14Z

Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1450) Fix bugs and refactor code in ResourceAwareScheduler

[Boyang Jerry Peng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Jerry Peng updated STORM-1450:
-
Summary: Fix bugs and refactor code in ResourceAwareScheduler  (was: 
Refactor some code in ResourceAwareScheduler)

> Fix bugs and refactor code in ResourceAwareScheduler
> 
>
> Key: STORM-1450
> URL: https://issues.apache.org/jira/browse/STORM-1450
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JStorm CGroup

[Erik Weathers]

Perhaps rather than just bolting on "cgroup support", we could instead open
a dialogue about having Mesos support be a core feature of Storm.

The current integration is a bit unwieldy & hackish at the moment, arising
from the conflicting natures of Mesos and Storm w.r.t. scheduling of
resources.  i.e., Storm assumes you have existing "slots" for running
workers on, whereas Mesos is more dynamic, requiring frameworks that run on
top of it to tell Mesos just how many resources (CPUs, Memory, etc.) are
needed by the framework's tasks.

One example of an issue with Storm-on-Mesos:  the Storm logviewer is
completely busted when you are using Mesos, I filed a ticket with a
description of the issue and proposed modifications to allow it to function:

   - https://issues.apache.org/jira/browse/STORM-1342

Furthermore, there are fundamental behaviors in Storm that don't mesh well
with Mesos:

   - the interfaces of INimbus (allSlotsAvailableForScheduling(),
   assignSlots(), getForcedScheduler(), etc.) make it difficult to create an
   ideal Mesos integration framework, since they don't allow the Mesos
   integration code to *really* know what's going on from the Nimbus's
   perspective. e.g.,
  - knowing which topologies & how many workers need to be scheduled at
  any given moment.
  - since the integration code cannot know what is actually needed to
  be run when it receives offers from Mesos, it just hoards those offers,
  leading to resource starvation in the Mesos cluster.
   - the "fallback" behavior of allowing the topology to settle for having
   less worker processes than requested should be disable-able.  For carefully
   tuned topologies it is quite bad to run on less than the expected number of
   worker processes.
  - also, this behavior endangers the idea of having the Mesos
  integration code *only* hoard Mesos offers after a successful round-trip
  through the allSlotsAvailableForScheduling() polling calls (i.e., only
  hoard when we know there are pending topologies).  It's dangerous because
  while we wait for another call to allSlotsAvailableForScheduling(), the
  Nimbus may have decided that it's okie dokie to use less than
the requested
  number of worker processes.

I'm sure there are other issues that I can conjure up, but those are the
major ones that came to mind instantly.  I'm happy to explain more about
this, since I realize the above bulleted info may lack context.

I wish I knew something about how Twitter's new Heron project addresses the
concerns above since it comes with Mesos support out-of-the-box, but it's
unclear at this point what they're doing until they open source it.

Thanks!

- Erik

On Wed, Jan 13, 2016 at 6:27 PM, 刘键(Basti Liu) 
wrote:

> Hi Bobby & Jerry,
>
> Yes, JStorm implements generic cgroup support. But just only cpu control
> is enable when starting worker.
>
> Regards
> Basti
>
> -Original Message-
> From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID]
> Sent: Wednesday, January 13, 2016 11:14 PM
> To: dev@storm.apache.org
> Subject: Re: JStorm CGroup
>
> Jerry,
> I think most of the code you are going to want to look at is here
> https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor/CgroupManager.java
> The back end for most of it seems to come from
>
>
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/container
>
> Which looks like it implements a somewhat generic cgroup support.
>  - Bobby
>
> On Wednesday, January 13, 2016 1:34 AM, 刘键(Basti Liu) <
> basti...@alibaba-inc.com> wrote:
>
>
>  Hi Jerry,
>
> Currently, JStorm supports to control the upper limit of cpu time for a
> worker by cpu.cfs_period_us & cpu.cfs_quota_us in cgroup.
> e.g. cpu.cfs_period_us= 10, cpu.cfs_quota_us=3*10. Cgroup will
> limit the corresponding process to occupy at most 300% cpu (3 cores).
>
> Regards
> Basti
>
> -Original Message-
> From: Jerry Peng [mailto:jerry.boyang.p...@gmail.com]
> Sent: Wednesday, January 13, 2016 1:57 PM
> To: dev@storm.apache.org
> Subject: JStorm CGroup
>
> Hello everyone,
>
> This question is directed more towards the people that worked on JStorm.
> If I recall correctly JStorm offers some sort of resource isolation through
> CGroups.  What kind of support does JStorm offer for resource isolation?
> Can someone elaborate on this feature in JStorm.
>
> Best,
>
> Jerry
>
>
>
>
>

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097742#comment-15097742
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693819
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693819
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+return this;
+  }
+
+  public Path getLockDirPath() {
+return lockDirPath;
+  }
+
+  public SpoutOutputCollector getCollector() {
+return collector;
+  }
+
+  public void nextTuple() {
+  

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693847
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+return this;
+  }
+
+  public Path getLockDirPath() {
+return lockDirPath;
+  }
+
+  public SpoutOutputCollector getCollector() {
+return collector;
+  }
+
+  public void nextTuple() {
+  

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097739#comment-15097739
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693731
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693731
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+return this;
+  }
+
+  public Path getLockDirPath() {
+return lockDirPath;
+  }
+
+  public SpoutOutputCollector getCollector() {
+return collector;
+  }
+
+  public void nextTuple() {
+  

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693776
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+return this;
+  }
+
+  public Path getLockDirPath() {
+return lockDirPath;
+  }
+
+  public SpoutOutputCollector getCollector() {
+return collector;
+  }
+
+  public void nextTuple() {
+  

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097741#comment-15097741
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693776
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+

[jira] [Updated] (STORM-1257) port backtype.storm.zookeeper to java

[Satish Duggana (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana updated STORM-1257:
--
Assignee: John Fang  (was: Satish Duggana)

> port backtype.storm.zookeeper to java
> -
>
> Key: STORM-1257
> URL: https://issues.apache.org/jira/browse/STORM-1257
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: java-migration, jstorm-merger
>
> A wrapper around zookeeper/curator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097734#comment-15097734
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693592
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693592
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+return this;
+  }
+
+  public Path getLockDirPath() {
+return lockDirPath;
+  }
+
+  public SpoutOutputCollector getCollector() {
+return collector;
+  }
+
+  public void nextTuple() {
+  

[GitHub] storm pull request: STORM-1199 : HDFS Spout Functionally complete....

[erikdw]

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693978
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/common/ModifTimeComparator.java
 ---
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.common;
+
+import org.apache.hadoop.fs.FileStatus;
+
+import java.util.Comparator;
+
+
+public class ModifTimeComparator
+implements Comparator {
+   @Override
+public int compare(FileStatus o1, FileStatus o2) {
--- End diff --

indentation is messed up.  3 and 4 spaces.  Should be 2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097747#comment-15097747
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693978
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/common/ModifTimeComparator.java
 ---
@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.common;
+
+import org.apache.hadoop.fs.FileStatus;
+
+import java.util.Comparator;
+
+
+public class ModifTimeComparator
+implements Comparator {
+   @Override
+public int compare(FileStatus o1, FileStatus o2) {
--- End diff --

indentation is messed up.  3 and 4 spaces.  Should be 2.


> Create HDFS Spout
> -
>
> Key: STORM-1199
> URL: https://issues.apache.org/jira/browse/STORM-1199
> Project: Apache Storm
>  Issue Type: New Feature
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: HDFSSpoutforStorm v2.pdf, HDFSSpoutforStorm.pdf, 
> hdfs-spout.1.patch
>
>
> Create an HDFS spout so that Storm can suck in data from files in a HDFS 
> directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097743#comment-15097743
 ] 

ASF GitHub Bot commented on STORM-1199:
---

Github user erikdw commented on a diff in the pull request:

https://github.com/apache/storm/pull/936#discussion_r49693847
  
--- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
@@ -0,0 +1,739 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.hdfs.spout;
+
+import java.io.IOException;
+import java.lang.reflect.Constructor;
+import java.net.URI;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import backtype.storm.Config;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.storm.hdfs.common.HdfsUtils;
+import org.apache.storm.hdfs.common.security.HdfsSecurityUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import backtype.storm.spout.SpoutOutputCollector;
+import backtype.storm.task.TopologyContext;
+import backtype.storm.topology.OutputFieldsDeclarer;
+import backtype.storm.topology.base.BaseRichSpout;
+import backtype.storm.tuple.Fields;
+
+public class HdfsSpout extends BaseRichSpout {
+
+  // user configurable
+  private String hdfsUri;// required
+  private String readerType; // required
+  private Fields outputFields;   // required
+  private Path sourceDirPath;// required
+  private Path archiveDirPath;   // required
+  private Path badFilesDirPath;  // required
+  private Path lockDirPath;
+
+  private int commitFrequencyCount = Configs.DEFAULT_COMMIT_FREQ_COUNT;
+  private int commitFrequencySec = Configs.DEFAULT_COMMIT_FREQ_SEC;
+  private int maxOutstanding = Configs.DEFAULT_MAX_OUTSTANDING;
+  private int lockTimeoutSec = Configs.DEFAULT_LOCK_TIMEOUT;
+  private boolean clocksInSync = true;
+
+  private String inprogress_suffix = ".inprogress";
+  private String ignoreSuffix = ".ignore";
+
+  // other members
+  private static final Logger LOG = 
LoggerFactory.getLogger(HdfsSpout.class);
+
+  private ProgressTracker tracker = null;
+
+  private FileSystem hdfs;
+  private FileReader reader;
+
+  private SpoutOutputCollector collector;
+  HashMap inflight = new HashMap<>();
+  LinkedBlockingQueue> retryList = 
new LinkedBlockingQueue<>();
+
+  private Configuration hdfsConfig;
+
+  private Map conf = null;
+  private FileLock lock;
+  private String spoutId = null;
+
+  HdfsUtils.Pair lastExpiredLock = null;
+  private long lastExpiredLockTime = 0;
+
+  private long tupleCounter = 0;
+  private boolean ackEnabled = false;
+  private int acksSinceLastCommit = 0 ;
+  private final AtomicBoolean commitTimeElapsed = new AtomicBoolean(false);
+  private Timer commitTimer;
+  private boolean fileReadCompletely = true;
+
+  private String configKey = Configs.DEFAULT_HDFS_CONFIG_KEY; // key for 
hdfs Kerberos configs
+
+  public HdfsSpout() {
+  }
+  /** Name of the output field names. Number of fields depends upon the 
reader type */
+  public HdfsSpout withOutputFields(String... fields) {
+outputFields = new Fields(fields);
+return this;
+  }
+
+  /** set key name under which HDFS options are placed. (similar to HDFS 
bolt).
+   * default key name is 'hdfs.config' */
+  public HdfsSpout withConfigKey(String configKey) {
+this.configKey = configKey;
+

[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097216#comment-15097216
 ] 

ASF GitHub Bot commented on STORM-1452:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1013

[STORM-1452] Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm 
storm-1452-profiler-broken-by-default-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1013


commit 0c3a78accf0cae37b93640b7082578ca406db8f5
Author: Derek Dagit 
Date:   2016-01-13T22:34:14Z

Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes




> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1472) Change long to readble date/time format for error log link

[Zhuo Liu (JIRA)]

Zhuo Liu created STORM-1472:
---

 Summary: Change long to readble date/time format for error log link
 Key: STORM-1472
 URL: https://issues.apache.org/jira/browse/STORM-1472
 Project: Apache Storm
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Zhuo Liu
Assignee: Zhuo Liu
Priority: Minor
 Fix For: 1.0.0


In the error log link of UI, long of millisecond dates makes little sense to 
user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1445) storm-cassandra uses cassandra-unit which license is LGPL v3

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1445.
-
   Resolution: Fixed
 Assignee: Robert Joseph Evans
Fix Version/s: 1.0.0

Addressed via 
https://github.com/apache/storm/commit/3db968092077363bd766db516b9e1135273672bf

> storm-cassandra uses cassandra-unit which license is LGPL v3
> 
>
> Key: STORM-1445
> URL: https://issues.apache.org/jira/browse/STORM-1445
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-cassandra
>Reporter: Jungtaek Lim
>Assignee: Robert Joseph Evans
>Priority: Critical
> Fix For: 1.0.0
>
>
> Digging into the test failures on storm-cassandra, I saw license of 
> cassandra-unit is LGPL v3 by chance.
> https://github.com/jsevellec/cassandra-unit
> From http://www.apache.org/legal/resolved.html, the page describes that 
> {quote} LGPL-licensed works must not be included in Apache Products {quote}
> We should get rid of cassandra-unit for now. Maybe we will make great lack of 
> tests but license issue is more critical.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim closed STORM-1471.
---
Resolution: Not A Problem

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -
>
> Key: STORM-1471
> URL: https://issues.apache.org/jira/browse/STORM-1471
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are 
> under heavy load due to too many fetch requests from storm.  We've narrowed 
> the problem to the way Fetch Requests are build in KafkaUtils.  When using 
> the FetchRequestBuilder, storm provides overrides for all the properties 
> except minBytes.  The default for that field is 0 (even though the Kafka 
> default for the high-level consumer is 1).  When paired with a maxWait > 0, 
> this creates a situation where the broker can immediately return a response 
> without waiting (due to minBytes 0).  This puts a heavy load on the brokers 
> and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as 
> appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097329#comment-15097329
 ] 

Jungtaek Lim commented on STORM-1471:
-

AFAIK, It is intended behavior since nextTuple() in Spout shouldn't be blocked.
You can refer http://storm.apache.org/documentation/Concepts.html - Spouts 
section.

JStorm provides multi-thread mode of Spout, and it could be applied to Storm 
when merging Storm and JStorm.
https://issues.apache.org/jira/browse/STORM-1358

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -
>
> Key: STORM-1471
> URL: https://issues.apache.org/jira/browse/STORM-1471
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are 
> under heavy load due to too many fetch requests from storm.  We've narrowed 
> the problem to the way Fetch Requests are build in KafkaUtils.  When using 
> the FetchRequestBuilder, storm provides overrides for all the properties 
> except minBytes.  The default for that field is 0 (even though the Kafka 
> default for the high-level consumer is 1).  When paired with a maxWait > 0, 
> this creates a situation where the broker can immediately return a response 
> without waiting (due to minBytes 0).  This puts a heavy load on the brokers 
> and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as 
> appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1452) Worker "profiler" actions broken by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097175#comment-15097175
 ] 

ASF GitHub Bot commented on STORM-1452:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1012

[STORM-1452] Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm 
storm-1452-profiler-broken-by-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1012.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1012


commit ae647a2a93af203778f273316ec6bb3f149ed56d
Author: Derek Dagit 
Date:   2016-01-13T22:34:14Z

Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes




> Worker "profiler" actions broken by default
> ---
>
> Key: STORM-1452
> URL: https://issues.apache.org/jira/browse/STORM-1452
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> * The profiler script flight.bash is not packaged by default.
> * The default options enable the Oracle specific flight-recorder that 
> requires a support subscription.
> The option to enable the profiler should not be enabled by default.  Other 
> actions such as worker restart, debugging, and heap can remain enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1452] Fixes profiling/debugging out of ...

[d2r]

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1013

[STORM-1452] Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm 
storm-1452-profiler-broken-by-default-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1013


commit 0c3a78accf0cae37b93640b7082578ca406db8f5
Author: Derek Dagit 
Date:   2016-01-13T22:34:14Z

Fixes profiling/debugging out of the box

Ships the flight.bash script
Disables UI display of Profiling and Debugging if on windows
Changes worker.profiler.enabled to control only profiling, not other 
debugging actions
Changes default of worker.profiler.enabled to false
Adds missing UI REST API data.
Populates user context on profiler UI routes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1472) Change long to readble date/time format for error log link

[Zhuo Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhuo Liu updated STORM-1472:

Issue Type: Bug  (was: Improvement)

> Change long to readble date/time format for error log link
> --
>
> Key: STORM-1472
> URL: https://issues.apache.org/jira/browse/STORM-1472
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> In the error log link of UI, long of millisecond dates makes little sense to 
> user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1262) port backtype.storm.command.dev-zookeeper to java

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1262:


Assignee: John Fang

> port backtype.storm.command.dev-zookeeper to java
> -
>
> Key: STORM-1262
> URL: https://issues.apache.org/jira/browse/STORM-1262
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: java-migration, jstorm-merger
>
> Launch a development zookeeper instance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1273) port backtype.storm.cluster to java

[John Fang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fang reassigned STORM-1273:


Assignee: John Fang

> port backtype.storm.cluster to java
> ---
>
> Key: STORM-1273
> URL: https://issues.apache.org/jira/browse/STORM-1273
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: John Fang
>  Labels: java-migration, jstorm-merger
>
> current state of the cluster (Some of this moves to java as a part of 
> heartbeat server)
> https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/cluster
>  as an example



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1472) Change long to readble date/time format for error log link

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097496#comment-15097496
 ] 

ASF GitHub Bot commented on STORM-1472:
---

GitHub user zhuoliu opened a pull request:

https://github.com/apache/storm/pull/1014

[STORM-1472] Fix the errorTime bug and show the time to be readable

1. Checking the code, find a bug in 
storm-core/src/clj/backtype/storm/ui/core.clj
"time" changed to "errorTime", however the REST info is still "time".
Error log link can not show any info for time.

2.In the error log link of UI, long of millisecond dates makes little sense 
to user. It would be nice to change it to be readable date.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuoliu/storm 1472

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1014.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1014


commit b48f02e39dcbec62a29da6989c98e004b6b1c940
Author: zhuol 
Date:   2016-01-14T02:52:59Z

[STORM-1472] Fix the errorTime bug and show the time to be readable




> Change long to readble date/time format for error log link
> --
>
> Key: STORM-1472
> URL: https://issues.apache.org/jira/browse/STORM-1472
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> In the error log link of UI, long of millisecond dates makes little sense to 
> user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1472] Fix the errorTime bug and show th...

[zhuoliu]

GitHub user zhuoliu opened a pull request:

https://github.com/apache/storm/pull/1014

[STORM-1472] Fix the errorTime bug and show the time to be readable

1. Checking the code, find a bug in 
storm-core/src/clj/backtype/storm/ui/core.clj
"time" changed to "errorTime", however the REST info is still "time".
Error log link can not show any info for time.

2.In the error log link of UI, long of millisecond dates makes little sense 
to user. It would be nice to change it to be readable date.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuoliu/storm 1472

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1014.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1014


commit b48f02e39dcbec62a29da6989c98e004b6b1c940
Author: zhuol 
Date:   2016-01-14T02:52:59Z

[STORM-1472] Fix the errorTime bug and show the time to be readable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1472] Fix the errorTime bug and show th...

[zhuoliu]

Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1014#discussion_r49682555
  
--- Diff: storm-core/src/clj/org/apache/storm/ui/core.clj ---
@@ -646,7 +646,7 @@
 reverse)]
 {"componentErrors"
  (for [^ErrorInfo e errors]
-   {"time" (* 1000 (long (.get_error_time_secs e)))
+   {"errorTime" (* 1000 (long (.get_error_time_secs e)))
--- End diff --

Here fix a bug.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1471) Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig

[Garrick Dasbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097475#comment-15097475
 ] 

Garrick Dasbach commented on STORM-1471:


That's a fair point, however it's somewhat confusing then that the KafkaConfig 
allows you to set fetchMaxWait, which defaults to 10s.

> Make FetchRequestBuilder.minBytes a configurable parameter in SpoutConfig
> -
>
> Key: STORM-1471
> URL: https://issues.apache.org/jira/browse/STORM-1471
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Reporter: Garrick Dasbach
>
> We currently have an issue in our storm cluster where our Kafka brokers are 
> under heavy load due to too many fetch requests from storm.  We've narrowed 
> the problem to the way Fetch Requests are build in KafkaUtils.  When using 
> the FetchRequestBuilder, storm provides overrides for all the properties 
> except minBytes.  The default for that field is 0 (even though the Kafka 
> default for the high-level consumer is 1).  When paired with a maxWait > 0, 
> this creates a situation where the broker can immediately return a response 
> without waiting (due to minBytes 0).  This puts a heavy load on the brokers 
> and defeats the purpose of any long polling.
> By making this a SpoutConfig option, it will allow the user to set that as 
> appropriate for their situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1472) Change long to readble date/time format for error log link

[Zhuo Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097493#comment-15097493
 ] 

Zhuo Liu commented on STORM-1472:
-

Checking the code, find a bug in storm-core/src/clj/backtype/storm/ui/core.clj
"time" changed to "errorTime", however the REST info is still "time".
Line 649:
  [errors-list topology-id secure?]
  (let [errors (->> errors-list
(sort-by #(.get_error_time_secs ^ErrorInfo %))
reverse)]
{"componentErrors"
 (for [^ErrorInfo e errors]
   {"time" (* 1000 (long (.get_error_time_secs e)))
"errorHost" (.get_host e)
"errorPort"  (.get_port e)

"componentErrors.errorTime| Long | Timestamp when the exception occurred (Prior 
to 0.11.0, this field was named 'time'.)"

> Change long to readble date/time format for error log link
> --
>
> Key: STORM-1472
> URL: https://issues.apache.org/jira/browse/STORM-1472
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> In the error log link of UI, long of millisecond dates makes little sense to 
> user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1257) port backtype.storm.zookeeper to java

[John Fang (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097590#comment-15097590
 ] 

John Fang commented on STORM-1257:
--

I am merging zookeeper now,  cluster is  linked with  zookeeper.  If 
you don' start worker at this issue, can this issue be assigned to me in order 
to avoid to  duplicate worker?

> port backtype.storm.zookeeper to java
> -
>
> Key: STORM-1257
> URL: https://issues.apache.org/jira/browse/STORM-1257
> Project: Apache Storm
>  Issue Type: New Feature
>  Components: storm-core
>Reporter: Robert Joseph Evans
>Assignee: Satish Duggana
>  Labels: java-migration, jstorm-merger
>
> A wrapper around zookeeper/curator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: HdfsSpout

[Jungtaek Lim]

Hi Matthias,

AFAIK, Roshan is working on providing HdfsSpout -
https://issues.apache.org/jira/browse/STORM-1199.
It's in reviewing so you can see the pull request and start reviewing, too.

Best,
Jungtaek Lim (HeartSaVioR)


On Thu, Jan 14, 2016 at 6:22 AM, Matthias J. Sax  wrote:

> Hi,
>
> if I am not wrong, Storm comes only with an HdfsBolt but not with a
> HdfsSpout. Would it be worth to add a HdfsSpout? Not sure how common
> reading data from HDFS in Strom is.
>
> -Matthias
>
>


-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

RE: JStorm CGroup

[刘键(Basti Liu)]

Hi Bobby & Jerry,

Yes, JStorm implements generic cgroup support. But just only cpu control is 
enable when starting worker.

Regards
Basti

-Original Message-
From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
Sent: Wednesday, January 13, 2016 11:14 PM
To: dev@storm.apache.org
Subject: Re: JStorm CGroup

Jerry,
I think most of the code you are going to want to look at is here 
https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor/CgroupManager.java
The back end for most of it seems to come from

https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/container

Which looks like it implements a somewhat generic cgroup support.
 - Bobby 

On Wednesday, January 13, 2016 1:34 AM, 刘键(Basti Liu) 
 wrote:
 

 Hi Jerry,

Currently, JStorm supports to control the upper limit of cpu time for a worker 
by cpu.cfs_period_us & cpu.cfs_quota_us in cgroup.
e.g. cpu.cfs_period_us= 10, cpu.cfs_quota_us=3*10. Cgroup will limit 
the corresponding process to occupy at most 300% cpu (3 cores).

Regards
Basti

-Original Message-
From: Jerry Peng [mailto:jerry.boyang.p...@gmail.com]
Sent: Wednesday, January 13, 2016 1:57 PM
To: dev@storm.apache.org
Subject: JStorm CGroup

Hello everyone,

This question is directed more towards the people that worked on JStorm.  If I 
recall correctly JStorm offers some sort of resource isolation through CGroups. 
 What kind of support does JStorm offer for resource isolation? Can someone 
elaborate on this feature in JStorm.

Best,

Jerry


  

[jira] [Commented] (STORM-1472) Change long to readble date/time format for error log link

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097500#comment-15097500
 ] 

ASF GitHub Bot commented on STORM-1472:
---

Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1014#discussion_r49682600
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -32,11 +32,13 @@
 public class Cluster {
 
 /**
- * key: supervisor id, value: supervisor details
+ * key: supervisor id,
+ * value: supervisor's total and used resources, i.e. {totalMem, 
totalCpu, usedMem, usedCpu}
  */
 private Map supervisors;
 /**
  * key: supervisor id, value: supervisor's total and used resources
+ * value: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
--- End diff --

Adding some comment to code (irrelevant).


> Change long to readble date/time format for error log link
> --
>
> Key: STORM-1472
> URL: https://issues.apache.org/jira/browse/STORM-1472
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> In the error log link of UI, long of millisecond dates makes little sense to 
> user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1472) Change long to readble date/time format for error log link

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097499#comment-15097499
 ] 

ASF GitHub Bot commented on STORM-1472:
---

Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1014#discussion_r49682555
  
--- Diff: storm-core/src/clj/org/apache/storm/ui/core.clj ---
@@ -646,7 +646,7 @@
 reverse)]
 {"componentErrors"
  (for [^ErrorInfo e errors]
-   {"time" (* 1000 (long (.get_error_time_secs e)))
+   {"errorTime" (* 1000 (long (.get_error_time_secs e)))
--- End diff --

Here fix a bug.


> Change long to readble date/time format for error log link
> --
>
> Key: STORM-1472
> URL: https://issues.apache.org/jira/browse/STORM-1472
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Fix For: 1.0.0
>
>
> In the error log link of UI, long of millisecond dates makes little sense to 
> user. It would be nice to change it to be readable date. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1472] Fix the errorTime bug and show th...

[zhuoliu]

Github user zhuoliu commented on a diff in the pull request:

https://github.com/apache/storm/pull/1014#discussion_r49682600
  
--- Diff: storm-core/src/jvm/org/apache/storm/scheduler/Cluster.java ---
@@ -32,11 +32,13 @@
 public class Cluster {
 
 /**
- * key: supervisor id, value: supervisor details
+ * key: supervisor id,
+ * value: supervisor's total and used resources, i.e. {totalMem, 
totalCpu, usedMem, usedCpu}
  */
 private Map supervisors;
 /**
  * key: supervisor id, value: supervisor's total and used resources
+ * value: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, 
assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
--- End diff --

Adding some comment to code (irrelevant).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: How can i monitor topology in storm ui ?

[Bobby Evans]

There is both a REST API that you can use to get anything off of the UI.
https://github.com/apache/storm/blob/master/docs/documentation/ui-rest-api.md
And you can also use the Thrift API to talk directly to nimbus and pull out all 
of that same information, although a little more processing may be needed in 
some cases.
https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift
 - Bobby 

On Wednesday, January 13, 2016 12:38 AM, master researcher 
 wrote:
 

 Thanks for the link I'm asking about monitoring topology I need to know how
can I know the performance of my topology and latency if it has and so
on... as I want to compare between two topologies with different coding
which is better than another . if there is possible another tool or can u
made what I want with storm ui and how ?
Thanks

On Wednesday, January 13, 2016, Cody Innowhere  wrote:

> I'm not sure whether I catch you correctly, if you mean a storm metrics
> monitor which enables you to see the historic metrics, you may refer to
> this article:
>
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> On Wed, Jan 13, 2016 at 11:21 AM, master researcher  >
> wrote:
>
> > is there another tool i can monitor topology ?
> >
> > On Tue, Jan 12, 2016 at 2:40 AM, master researcher  >
> > wrote:
> >
> > > Thanks in advance for any help , i have topology submitted it but need
> to
> > > know if i made change int the code an need to submitted new one how
> can i
> > > compare between old and new topology
> > >
> > > should i compare between it through Executed latency only or what ?
> > >
> > > if someone has links or vidoes for someone illustrate monitioring well
> ,
> > > i'll apperciate it
> > >
> > > Waiting Your Reply ...
> > >
> >
>


  

[jira] [Created] (STORM-1469) Unable to deploy large topologies on apache storm

[Rudra Sharma (JIRA)]

Rudra Sharma created STORM-1469:
---

 Summary: Unable to deploy large topologies on apache storm
 Key: STORM-1469
 URL: https://issues.apache.org/jira/browse/STORM-1469
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 1.0.0
Reporter: Rudra Sharma
 Fix For: 1.0.0


When deploying to a nimbus a topology which is larger in size >17MB, we get an 
exception. In storm 0.9.3 this could be mitigated by using the following config 
on the storm.yaml to increse the buffer size to handle the topology size. i.e. 
50MB would be

nimbus.thrift.max_buffer_size: 5000

This configuration does not resolve the issue in the master branch of storm and 
we cannot deploy topologies which are large in size.

Here is the log on the client side when attempting to deploy to the nimbus node:
java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
at 
backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149)
 [siem-ng-storm-deployer-cloud.jar:]
at 
com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87)
 [siem-ng-storm-deployer-cloud.jar:]
Caused by: org.apache.thrift7.transport.TTransportException
at 
org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
 ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
 ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
 ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
at 
backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) 
~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT]
... 4 more

Here is the log on the server side (nimbus.log):

2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: 
stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar
2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to 
/var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from 
client: 
/var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar
2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] 
Exception while invoking!
org.apache.thrift7.transport.TTransportException: Frame size (17435758) larger 
than max length (16384000)!
at 
org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
at 
org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at 
org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at 
org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:27)
at 
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158)
at 

Re: Metrics,access and custom Log files

[researcher cs]

sorry i mean after submitted topology i found those files but empty ! how
can i use it  ?

On Wed, Jan 13, 2016 at 4:45 PM, researcher cs 
wrote:

> There are files in storm/logs  like metrics , access and custom but
> there are empty how can i use it ?
>

[jira] [Commented] (STORM-1468) Merge changes from {master}/docs to {asf-site} and remove {master}/docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096346#comment-15096346
 ] 

ASF GitHub Bot commented on STORM-1468:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1008#issuecomment-171331905
  
I am +1 on this.  Not having 2 copies is good and a lot better then what we 
have now, but I really would prefer to see a hybrid approach.  Some 
documentation is really tied to a release.  The REST API for example, where as 
other documentation is not, the list of contributors and users for example.  I 
really would prefer to see what I see on other sites, which is some common 
things but they point into release specific sections that are generated from 
the release branch, and are obviously tied to a release.

We could also use symlinks to denote stable releases vs releases that are 
no longer supported.


> Merge changes from {master}/docs to {asf-site} and remove {master}/docs
> ---
>
> Key: STORM-1468
> URL: https://issues.apache.org/jira/browse/STORM-1468
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> This will consist of two pull requests:
> 1. Merge changes from {master}/docs to {asf-site}/
> 2. Remove the {master}/docs directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1468: Merge changes from {master}/docs t...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1008#issuecomment-171331905
  
I am +1 on this.  Not having 2 copies is good and a lot better then what we 
have now, but I really would prefer to see a hybrid approach.  Some 
documentation is really tied to a release.  The REST API for example, where as 
other documentation is not, the list of contributors and users for example.  I 
really would prefer to see what I see on other sites, which is some common 
things but they point into release specific sections that are generated from 
the release branch, and are obviously tied to a release.

We could also use symlinks to denote stable releases vs releases that are 
no longer supported.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...

[hmcl]

Github user hmcl commented on the pull request:

https://github.com/apache/storm/pull/977#issuecomment-171332402
  
+1 LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096352#comment-15096352
 ] 

ASF GitHub Bot commented on STORM-1419:
---

Github user hmcl commented on the pull request:

https://github.com/apache/storm/pull/977#issuecomment-171332402
  
+1 LGTM


> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096322#comment-15096322
 ] 

ASF GitHub Bot commented on STORM-1419:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/977#issuecomment-171325985
  
+1


> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1468) Merge changes from {master}/docs to {asf-site} and remove {master}/docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096348#comment-15096348
 ] 

ASF GitHub Bot commented on STORM-1468:
---

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1009#issuecomment-171332062
  
+1 see https://github.com/apache/storm/pull/1008#issuecomment-171331905


> Merge changes from {master}/docs to {asf-site} and remove {master}/docs
> ---
>
> Key: STORM-1468
> URL: https://issues.apache.org/jira/browse/STORM-1468
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> This will consist of two pull requests:
> 1. Merge changes from {master}/docs to {asf-site}/
> 2. Remove the {master}/docs directory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1468: remove {master}/docs

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/1009#issuecomment-171332062
  
+1 see https://github.com/apache/storm/pull/1008#issuecomment-171331905


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

topology.stats.sample.rate

[researcher cs]

what is this properties used for topology.stats.sample.rate ?

Re: JStorm CGroup

[Bobby Evans]

Jerry,
I think most of the code you are going to want to look at is here
https://github.com/apache/storm/blob/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor/CgroupManager.java
The back end for most of it seems to come from

https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/container

Which looks like it implements a somewhat generic cgroup support.
 - Bobby 

On Wednesday, January 13, 2016 1:34 AM, 刘键(Basti Liu) 
 wrote:
 

 Hi Jerry,

Currently, JStorm supports to control the upper limit of cpu time for a worker 
by cpu.cfs_period_us & cpu.cfs_quota_us in cgroup.
e.g. cpu.cfs_period_us= 10, cpu.cfs_quota_us=3*10. Cgroup will limit 
the corresponding process to occupy at most 300% cpu (3 cores).

Regards
Basti

-Original Message-
From: Jerry Peng [mailto:jerry.boyang.p...@gmail.com] 
Sent: Wednesday, January 13, 2016 1:57 PM
To: dev@storm.apache.org
Subject: JStorm CGroup

Hello everyone,

This question is directed more towards the people that worked on JStorm.  If I 
recall correctly JStorm offers some sort of resource isolation through CGroups. 
 What kind of support does JStorm offer for resource isolation? Can someone 
elaborate on this feature in JStorm.

Best,

Jerry


  

Re: How can i monitor topology in storm ui ?

[Bobby Evans]

I forgot to mention that any registered IMetricsConsumer will also get latency 
and throughput metrics that you could use for monitoring.  IMetricsConsumer is 
the preferable way because it scales much better than the other APIs do.
 - Bobby 

On Wednesday, January 13, 2016 9:19 AM, Bobby Evans  
wrote:
 

 There is both a REST API that you can use to get anything off of the UI.
https://github.com/apache/storm/blob/master/docs/documentation/ui-rest-api.md
And you can also use the Thrift API to talk directly to nimbus and pull out all 
of that same information, although a little more processing may be needed in 
some cases.
https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift
 - Bobby 

On Wednesday, January 13, 2016 12:38 AM, master researcher 
 wrote:
 

 Thanks for the link I'm asking about monitoring topology I need to know how
can I know the performance of my topology and latency if it has and so
on... as I want to compare between two topologies with different coding
which is better than another . if there is possible another tool or can u
made what I want with storm ui and how ?
Thanks

On Wednesday, January 13, 2016, Cody Innowhere  wrote:

> I'm not sure whether I catch you correctly, if you mean a storm metrics
> monitor which enables you to see the historic metrics, you may refer to
> this article:
>
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> On Wed, Jan 13, 2016 at 11:21 AM, master researcher  >
> wrote:
>
> > is there another tool i can monitor topology ?
> >
> > On Tue, Jan 12, 2016 at 2:40 AM, master researcher  >
> > wrote:
> >
> > > Thanks in advance for any help , i have topology submitted it but need
> to
> > > know if i made change int the code an need to submitted new one how
> can i
> > > compare between old and new topology
> > >
> > > should i compare between it through Executed latency only or what ?
> > >
> > > if someone has links or vidoes for someone illustrate monitioring well
> ,
> > > i'll apperciate it
> > >
> > > Waiting Your Reply ...
> > >
> >
>


   

  

Re: 1.x storm and jstorm merger

[Bobby Evans]

That is great to hear.  We will work on getting your dependencies put in place 
ASAP.
 - Bobby 

On Wednesday, January 13, 2016 6:03 AM, 刘键(Basti Liu) 
 wrote:
 

 Hi Bobby,

For dependencies, we have not done much work on these modules except the ones 
related to zookeeper (cluster.clj and zookeeper.clj).
When doing the migration of Nimbus and Supervisor, we just updated the name of 
relative functions called in Nimbus and Supervisor by the 
guidelines you gave below, and left empty implementation for these in relative 
modules.
So, please feel free to start on utils.clj and config.clj.

Following is the detailed status for Nimbus, Supervisor and zookeeper modules 
currently. We will also update the status in JIRAs.
- Nimbus: 30% The structure of state transition and service handler are almost 
done. Blob store, replication between Nimbus and security have not been started 
yet.
- Supervisor: 90% Only health check was left.
- Zookeeper: 40% Most interfaces of reading operation have been done.

Regards
Basti

-Original Message-
From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
Sent: Tuesday, January 12, 2016 10:35 PM
To: 刘键(Basti Liu); dev@storm.apache.org
Subject: Re: 1.x storm and jstorm merger

For the migration of fixes, we have the policy that fixes need to go into 
master before they can be merged into previous branches, so with that the fix 
would go into master and anyone who wants the fix on an older branch would be 
responsible for porting it to clojure.
It is great to hear that there is progress being made on Nimbus and the 
Supervisor, but those are very large pieces of code with lots of dependencies.  
I would really like to sync up with you and what you are doing so we don't get 
too much duplicated efforts.  Specifically my team is starting on utils.clj and 
config.clj.  But if you have already done some/all of the work for them I would 
rather use that work instead.  I also want to check changes in frequently with 
smaller differences.  Less differences means we should be able to find bug 
sooner and adjust accordingly.  Is there any way you could take some of what 
you have done, even if it is not complete and put up pull requests for the 
portions that do work and can be swapped out? Particularly in underlying files. 
 Or at least put it in a place that we can look at it? 
 - Bobby 

    On Monday, January 11, 2016 9:31 PM, 刘键(Basti Liu) 
 wrote:
 

 Hi Bobby,

It is great to see that we are going to finalize Storm 1.x and start the 
migration.
Just to synchronize current status of migration in Alibaba, the migration of 
Nimbus and Supervisor part is in progress(around 50% is completed).
Besides it, we'd like to confirm the handling of following bug fixes in 
branch-1.x. Who is responsible to migrate the fix from branch-1.x to mater 
branch(2.0.0-snapshot)? the owner of the JIRA in branch-1.x or the owner of 
corresponding migration JIRA in master branch?

Regards
Basti

-Original Message-
From: Bobby Evans [mailto:ev...@yahoo-inc.com.INVALID] 
Sent: Tuesday, January 12, 2016 5:39 AM
To: Dev
Subject: 1.x storm and jstorm merger

I think we are finally at the point were this is going to start happening.
I have merged in as many JIRA/PULL requests that looked like they were ready.  
The most disruptive of these is probably STORM-1202, which translated 
backtype.storm to org.apache.storm and storm.trident to 
org.apache.storm.trident  There is some hack code that we will remove in the 
future that when submitting a topology using the new client it will translate 
your jar for you to the new namespaces.  You can enable it by setting 
`client.jartransformer.class` to `org.apache.storm.hack.StormShadeTransformer`

With that I changed the 0.11.0 tag to 1.0.0.  And I created a branch-1.x branch 
in the main repo.  I have not started creating a release candidate just yet as 
there still are a few outstanding bugs that I would like to see resolved before 
hand.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20type%20%3D%20Bug%20and%20resolution%20%3D%20Unresolved%20and%20priority%20%3E%3D%20Blocker

Of them only STORM-1452 really feels like a blocker, but I am open to others 
opinions as we work towards a release.  I would encourage everyone to kick the 
tires on it and file JIRA if you do find any issues.



I also changed the version on master to 2.0.0-SNAPSHOT in preparation for the 
migration to java and the JStorm merger.  This means that we are now open to 
start pulling in java migration pull requests as outlined on the wiki
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
I know that my team is going to start working on this process now to try and 
shorten the time of the transition as much as possible.  From this point on 
until the transition to java is complete there will be a moratorium on new 
features going into storm.  This is not a hard moratorium.  If your 

Metrics,access and custom Log files

[researcher cs]

There are files in storm/logs  like metrics , access and custom but
there are empty how can i use it ?

[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...

[revans2]

Github user revans2 commented on the pull request:

https://github.com/apache/storm/pull/977#issuecomment-171325985
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1466: Move the org.apache.thrift7 namesp...

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171365717
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096555#comment-15096555
 ] 

ASF GitHub Bot commented on STORM-1466:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171365717
  
+1


> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1467) Switch apache-rat plugin off by default, but enable for Travis-CI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096561#comment-15096561
 ] 

ASF GitHub Bot commented on STORM-1467:
---

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1006#issuecomment-171366113
  
+1


> Switch apache-rat plugin off by default, but enable for Travis-CI
> -
>
> Key: STORM-1467
> URL: https://issues.apache.org/jira/browse/STORM-1467
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Kyle Nusbaum
>
> apache-rat is super annoying when developing. We should change it so that it 
> doesn't run all the time, but can be manually triggered and runs in CI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1467: Switch apache-rat plugin off by de...

[ptgoetz]

Github user ptgoetz commented on the pull request:

https://github.com/apache/storm/pull/1006#issuecomment-171366113
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1466: Move the org.apache.thrift7 namesp...

[knusbaum]

Github user knusbaum commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171368349
  
I'm also going to cherry-pick this back into the 1.0 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1466) Move the org.apache.thrift7 namespace to something correct/sensible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096589#comment-15096589
 ] 

ASF GitHub Bot commented on STORM-1466:
---

Github user knusbaum commented on the pull request:

https://github.com/apache/storm/pull/1007#issuecomment-171368349
  
I'm also going to cherry-pick this back into the 1.0 branch.


> Move the org.apache.thrift7 namespace to something correct/sensible
> ---
>
> Key: STORM-1466
> URL: https://issues.apache.org/jira/browse/STORM-1466
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>
> Thrift is still shaded to org.apache.thrift7 even though we are not using 
> Thrift 7.
> We should also add something for temporary backwards compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

[Derek Dagit (JIRA)]

Derek Dagit created STORM-1470:
--

 Summary: org.apache.hadoop:hadoop-auth not shaded, breaks spnego 
authentication
 Key: STORM-1470
 URL: https://issues.apache.org/jira/browse/STORM-1470
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 2.0.0, 1.0.0
Reporter: Derek Dagit
Assignee: Derek Dagit
Priority: Blocker


{noformat}
2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
/favicon.ico
java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at 
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
at 
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
at 
org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
at 
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
at 
org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
at 
org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at 
org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at 
org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
at 
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
at 
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
at 
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
at 
org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at 
org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at 
org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at 
org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at 
org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at 
org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at 
org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
{noformat}

We already shade commons-codec:commons-codec, but we don't apply that shading 
to org.apache.hadoop:hadoop-auth.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1419) Solr bolt should handle tick tuples

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1419.

   Resolution: Fixed
Fix Version/s: 1.0.0

Merged to master/1.x-branch.

> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
> Fix For: 1.0.0
>
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1453) nimbus.clj/wait-for-desired-code-replication print wrong log message

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096621#comment-15096621
 ] 

ASF GitHub Bot commented on STORM-1453:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1003


> nimbus.clj/wait-for-desired-code-replication print wrong log message
> 
>
> Key: STORM-1453
> URL: https://issues.apache.org/jira/browse/STORM-1453
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: caofangkun
>Assignee: caofangkun
>Priority: Trivial
>
> https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L516



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1420] solr CountBasedCommit should rese...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/978


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096636#comment-15096636
 ] 

ASF GitHub Bot commented on STORM-1470:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1011

[STORM-1470] Applies shading to hadoop-auth, cleaner exclusions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1470-hadoop-auth-shade-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit 57021116d1399108acd17d22ab94962d82516177
Author: Derek Dagit 
Date:   2016-01-13T17:26:10Z

Applies shading to hadoop-auth, cleaner exclusions




> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1470) org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096633#comment-15096633
 ] 

ASF GitHub Bot commented on STORM-1470:
---

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1010

[STORM-1470] Applies shading to hadoop-auth, cleaner exclusions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1470-hadoop-auth-shade-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1010


commit 70197332bba46ffd2b38da183e835bd370a68e70
Author: Derek Dagit 
Date:   2016-01-13T17:26:10Z

Applies shading to hadoop-auth, cleaner exclusions




> org.apache.hadoop:hadoop-auth not shaded, breaks spnego authentication
> --
>
> Key: STORM-1470
> URL: https://issues.apache.org/jira/browse/STORM-1470
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Derek Dagit
>Assignee: Derek Dagit
>Priority: Blocker
>
> {noformat}
> 2016-01-12 20:07:45.642 o.a.s.s.o.e.j.s.ServletHandler [WARN] Error for 
> /favicon.ico
> java.lang.NoClassDefFoundError: org/apache/commons/codec/binary/Base64
> at 
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:343)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at 
> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
> at 
> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at 
> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
> at 
> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at 
> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We already shade commons-codec:commons-codec, but we don't apply that shading 
> to org.apache.hadoop:hadoop-auth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1420) solr CountBasedCommit should reset count=0 after commited

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096637#comment-15096637
 ] 

ASF GitHub Bot commented on STORM-1420:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/978


> solr CountBasedCommit should reset count=0 after commited
> -
>
> Key: STORM-1420
> URL: https://issues.apache.org/jira/browse/STORM-1420
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> The variable count should reset to 0, if not so it will increase infinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

[d2r]

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1011

[STORM-1470] Applies shading to hadoop-auth, cleaner exclusions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1470-hadoop-auth-shade-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1011


commit 57021116d1399108acd17d22ab94962d82516177
Author: Derek Dagit 
Date:   2016-01-13T17:26:10Z

Applies shading to hadoop-auth, cleaner exclusions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: [STORM-1419] Solr bolt should handle tick tupl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/977


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1419) Solr bolt should handle tick tuples

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096608#comment-15096608
 ] 

ASF GitHub Bot commented on STORM-1419:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/977


> Solr bolt should handle tick tuples
> ---
>
> Key: STORM-1419
> URL: https://issues.apache.org/jira/browse/STORM-1419
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> Solr bolt should handle tick tuples.
> Forcing solr client commit when bolt received tick tuple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1453] nimbus.clj/wait-for-desired-code-...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1453) nimbus.clj/wait-for-desired-code-replication print wrong log message

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1453.

   Resolution: Fixed
Fix Version/s: 1.0.0

Merged to master/1.x-branch.

> nimbus.clj/wait-for-desired-code-replication print wrong log message
> 
>
> Key: STORM-1453
> URL: https://issues.apache.org/jira/browse/STORM-1453
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: caofangkun
>Assignee: caofangkun
>Priority: Trivial
> Fix For: 1.0.0
>
>
> https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L516



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1470] Applies shading to hadoop-auth, c...

[d2r]

GitHub user d2r opened a pull request:

https://github.com/apache/storm/pull/1010

[STORM-1470] Applies shading to hadoop-auth, cleaner exclusions



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/d2r/storm storm-1470-hadoop-auth-shade-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1010


commit 70197332bba46ffd2b38da183e835bd370a68e70
Author: Derek Dagit 
Date:   2016-01-13T17:26:10Z

Applies shading to hadoop-auth, cleaner exclusions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1420) solr CountBasedCommit should reset count=0 after commited

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1420.

Resolution: Not A Problem

> solr CountBasedCommit should reset count=0 after commited
> -
>
> Key: STORM-1420
> URL: https://issues.apache.org/jira/browse/STORM-1420
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-solr
>Reporter: Xin Wang
>Assignee: Xin Wang
>
> The variable count should reset to 0, if not so it will increase infinitely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)