[GitHub] storm pull request: STORM-1679: add storm Scheduler documents

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1305#issuecomment-212284839
  
@vesense 
+1
Could you rebase and squash the commits into one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1679) add storm Scheduler documents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249389#comment-15249389
 ] 

ASF GitHub Bot commented on STORM-1679:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1305#issuecomment-212284839
  
@vesense 
+1
Could you rebase and squash the commits into one?


> add storm Scheduler documents
> -
>
> Key: STORM-1679
> URL: https://issues.apache.org/jira/browse/STORM-1679
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1679 (1.x): add storm Scheduler document...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1306#issuecomment-212284727
  
+1
Could you rebase and squash the commits into one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1679) add storm Scheduler documents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249387#comment-15249387
 ] 

ASF GitHub Bot commented on STORM-1679:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1306#issuecomment-212284727
  
+1
Could you rebase and squash the commits into one?


> add storm Scheduler documents
> -
>
> Key: STORM-1679
> URL: https://issues.apache.org/jira/browse/STORM-1679
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212283858
  
@satishd 
+1 Retested with local build.
Thanks for the quick fix! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1715) Jedis Default Host

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249368#comment-15249368
 ] 

ASF GitHub Bot commented on STORM-1715:
---

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/1348#issuecomment-212280349
  
@HeartSaVioR  could you review this one ?


> Jedis Default Host
> --
>
> Key: STORM-1715
> URL: https://issues.apache.org/jira/browse/STORM-1715
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-redis
>Affects Versions: 1.0.0, 2.0.0
>Reporter: darion yaphet
>Assignee: darion yaphet
>Priority: Minor
>
> Using Jedis Protocol.DEFAULT_HOST to replace DEFAULT_HOST



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1715] Using Jedis Protocol.DEFAULT_HOST...

[darionyaphet]

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/1348#issuecomment-212280349
  
@HeartSaVioR  could you review this one ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[satishd]

Github user satishd commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212279085
  
@HeartSaVioR done with the suggested changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249318#comment-15249318
 ] 

ASF GitHub Bot commented on STORM-1649:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212272758
  
At least commit message should be "STORM-1649 kryo serialization in 
windowing" to match to PR against master. I guess squashing commits would be 
good since this is not huge patch.


> Optimize Kryo instaces creation in HBaseWindowsStore
> 
>
> Key: STORM-1649
> URL: https://issues.apache.org/jira/browse/STORM-1649
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: trident
>Reporter: Satish Duggana
>Assignee: Satish Duggana
> Fix For: 2.0.0, 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212272758
  
At least commit message should be "STORM-1649 kryo serialization in 
windowing" to match to PR against master. I guess squashing commits would be 
good since this is not huge patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212271938
  
@satishd 
I'd like to see the commits having meaningful messages. Could you reword 
the commit messages?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212271463
  
+1 Confirmed local build passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1704) When logviewer_search.html opens daemon file, next search always show no result

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249307#comment-15249307
 ] 

ASF GitHub Bot commented on STORM-1704:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1330#issuecomment-212269533
  
+1


> When logviewer_search.html opens daemon file, next search always show no 
> result
> ---
>
> Key: STORM-1704
> URL: https://issues.apache.org/jira/browse/STORM-1704
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Minor
>
> When searching keyword in /logviewer_search.html with daemon log file, 
> is-daemon=yes parameter is gone so search shows no result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1704 (1.x) When logviewer_search.html op...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1330#issuecomment-212269533
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1676) NullPointerException while serializing ClusterWorkerHearbeat

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249305#comment-15249305
 ] 

ASF GitHub Bot commented on STORM-1676:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1298#issuecomment-212268905
  
+1


> NullPointerException while serializing ClusterWorkerHearbeat
> 
>
> Key: STORM-1676
> URL: https://issues.apache.org/jira/browse/STORM-1676
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>
> `Map executor_stats` had null value in the key 
> which was causing NPE during serialization. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1676] Filter null executor stats from w...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1298#issuecomment-212268905
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: Storm 1649 Optimize Kryo instaces creation in ...

[harshach]

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1302#issuecomment-212268744
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15249166#comment-15249166
 ] 

ASF GitHub Bot commented on STORM-1649:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1295#issuecomment-212225678
  
+1
Btw, build passed for my local builds.


> Optimize Kryo instaces creation in HBaseWindowsStore
> 
>
> Key: STORM-1649
> URL: https://issues.apache.org/jira/browse/STORM-1649
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: trident
>Reporter: Satish Duggana
>Assignee: Satish Duggana
> Fix For: 2.0.0, 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1649 Optimize Kryo instaces creation in ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1295#issuecomment-212225678
  
+1
Btw, build passed for my local builds.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Publishing cluster stats on Apache Storm

[Jungtaek Lim]

Harsha,

Yes I'm thinking about pushing so Nimbus is a best place to add.

It would be different than MetricsConsumer due to,
- Only leader of Nimbus should publish cluster metrics to consumer.
- Nimbus shouldn't be affected by crashing or heavy latency on consumer.
- Consumer should have resilient when crashing or Nimbus should take care
of.

I'll sketch the design and come up with JIRA issue or new thread on dev@.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 4월 20일 (수) 오전 1:23, Harsha 님이 작성:

> Jungtaek,
>   Ideally we should avoid using NimbusClient as much as
>   possible for a polling reporter like this. Can we have
>   plug gable interface that can be run inside the nimbus at
>   configured interval . Since it will be plugged into nimbus
>   we would have all cluster details including topologies as
>   well.
>
> Thanks,
> Harsha
>
>
> On Mon, Apr 18, 2016, at 10:18 PM, Jungtaek Lim wrote:
> > Cody,
> >
> > Great idea! Let's try it out.
> > I'll send the mail to user@ regarding use cases or concerns of metrics
> on
> > Apache Storm. We may have a luck to find brilliant ideas or use cases we
> > didn't know.
> >
> > 2016년 4월 19일 (화) 오후 2:07, Cody Innowhere 님이 작성:
> >
> > > Jungtaek,
> > > Thanks for your explanation, still +1 for your proposal.
> > > Moreover, I'd like to hear storm users about this, especially use
> cases for
> > > metrics so that we can see what's best to do this.
> > >
> > > On Tue, Apr 19, 2016 at 11:19 AM, Jungtaek Lim 
> wrote:
> > >
> > > > Cody,
> > > > Thanks for giving an opinion.
> > > >
> > > > I guess there was a miscommunication between you and me.
> > > > This shouldn't be related to topology. Plugin should be set up via
> > > cluster
> > > > configuration, and singular daemon like Nimbus (master) or UI should
> > > > publish cluster metrics to the plugin.
> > > > I agree Nimbus should have this feature, but only master should
> publish
> > > the
> > > > cluster metrics so we should handle it.
> > > >
> > > > Btw, since we don't have built-in metrics storage, we can't provide
> query
> > > > API with time-series fashion.
> > > > I guess publishing (pushing) seems the better way what we can address
> > > > without huge modification.
> > > > (Did you mean Nimbus has query API which queries the external
> storage?)
> > > >
> > > > Please note that it's an idea for improvement against 1.x, not 2.0.
> > > > In phase 2 we should evaluate and replace metrics feature with
> JStorm or
> > > > another after porting to Java.
> > > >
> > > > Thanks,
> > > > Jungtaek Lim (HeartSaVioR)
> > > >
> > > > 2016년 4월 19일 (화) 오전 11:59, Cody Innowhere 님이
> 작성:
> > > >
> > > > > I'm +1 for publishing cluster stats, but don't quite understand
> your
> > > > > implementation.
> > > > > Do you mean that a topology which wants cluster stats can get
> cluster
> > > > stats
> > > > > by implementing the pluggable consumer/reporter?
> > > > > Personally I would prefer to put this role into nimbus (can be
> > > pluggable
> > > > > too) since it's the responsibility of nimbus to take care of all
> > > > > topologies, naturally all topology stats. Then we expose API's from
> > > > nimbus
> > > > > for external queries.
> > > > >
> > > > > On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim 
> > > wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > While Storm publishes topology metrics by metrics consumers,
> Storm
> > > > > doesn't
> > > > > > publish cluster metrics any way.
> > > > > >
> > > > > > Therefore, I'd like to introduce the feature of publishing
> cluster
> > > > stats
> > > > > to
> > > > > > pluggable consumers so that users can also push to external
> storages
> > > > and
> > > > > > query on it or even configure dashboard.
> > > > > > (For topology stats it's supported via MetricsConsumer.)
> > > > > >
> > > > > > I'm seeing some workarounds like having their own reporter
> process
> > > > > polling
> > > > > > cluster information from Nimbus or REST API.
> > > > > > (For example, Ambari has cluster metrics reporter for Storm
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java
> > > > > > >
> > > > > > but it relies on custom build of Storm.)
> > > > > > Yes it works anyway but if Storm provides the feature naturally,
> with
> > > > > > pluggable way like MetricsConsumer it would be great for users
> who
> > > want
> > > > > to
> > > > > > configure dashboard regarding this.
> > > > > >
> > > > > > I also saw that STORM-1158 <
> > > > > > http://issues.apache.org/jira/browse/STORM-1158>
> > > > > > adds metrics reporter for internal actions, but it has some
> > > limitation
> > > > on
> > > > > > it.
> > > > > >
> > > > > > - It focuses how many requests the daemon receives, not cluster
> > > stats.
> > > > > > - It requires reporter to use codaha

[DISCUSS] Release Apache Storm 1.0.1

[P. Taylor Goetz]

Following up on the discussion last week of releasing 1.0.1 quickly to address 
high-priority issues that have been identified, I’ve created the following JIRA 
epic for tracking the release:

https://issues.apache.org/jira/browse/STORM-1721 


Any critical issues that should be included in the 1.0.1 release should be 
added to that epic so they can be tracked easily. Alternatively reply to this 
thread and I will add.

I hope to cut a release candidate by the end of the week.

-Taylor


signature.asc
Description: Message signed with OpenPGP using GPGMail

[jira] [Updated] (STORM-1649) Optimize Kryo instaces creation in HBaseWindowsStore

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1649:
---
Fix Version/s: (was: 1.0.0)
   1.0.1

> Optimize Kryo instaces creation in HBaseWindowsStore
> 
>
> Key: STORM-1649
> URL: https://issues.apache.org/jira/browse/STORM-1649
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: trident
>Reporter: Satish Duggana
>Assignee: Satish Duggana
> Fix For: 2.0.0, 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1658) documents improvements and links fixes

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1658:
---
Fix Version/s: 1.0.1

> documents improvements and links fixes
> --
>
> Key: STORM-1658
> URL: https://issues.apache.org/jira/browse/STORM-1658
> Project: Apache Storm
>  Issue Type: Bug
>  Components: documentation
>Reporter: Xin Wang
>Assignee: Xin Wang
> Fix For: 2.0.0, 1.0.1
>
>
> Direct groupings(_Direct-groupings.html_) in releases 
> 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT,0.10.0 is not found(404).
> Deamon Metrics/Monitoring(_storm-metrics-profiling-internal-actions.html_) in 
> releases 2.0.0-SNAPSHOT,1.0.0-SNAPSHOT is not found(404).
> For the former, I cannot find _Direct-groupings.md_ in docs. For the latter, 
> I find _storm-metrics-profiling-internal-actions.md_ in docs dir.
> According to Brian‘s report in storm-user mailing list:
> (1) http://storm.apache.org/
> at the bottom has a link to "Tutorial":
> http://storm.apache.org/releases/current/tutorial.html
> but this gives a 404
> Either the link needs to change to
> http://storm.apache.org/releases/current/Tutorial.html
> or the page itself needs to be renamed to "tutorial.html"
> (2) http://storm.apache.org/releases/0.10.0/index.html
> Under "Intermediate"
> * The "Direct Groupings" link points to
> http://storm.apache.org/releases/0.10.0/Direct-groupings.html
> which is non-existent.
> * The "Lifecycle of a trident tuple" link just points to the documentation 
> index,
> http://storm.apache.org/releases/0.10.0/index.html
> (3) There are many pages with broken images. For example
> * 
> http://storm.apache.org/releases/0.10.0/Understanding-the-parallelism-of-a-Storm-topology.html
> has broken image pointing to
> http://storm.apache.org/releases/0.10.0/images/relationships-worker-processes-executors-tasks.png
> http://storm.apache.org/releases/0.10.0/images/example-of-a-running-topology.png
> (these give 404 errors)
> * http://storm.apache.org/releases/0.10.0/Trident-tutorial.html
> has a broken image pointing to
> http://storm.apache.org/releases/0.10.0/images/batched-stream.png



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1681) Bug in scheduling cyclic topologies when scheduling with RAS

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1681:
---
Fix Version/s: 1.0.1
   2.0.0

> Bug in scheduling cyclic topologies when scheduling with RAS
> 
>
> Key: STORM-1681
> URL: https://issues.apache.org/jira/browse/STORM-1681
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Critical
> Fix For: 2.0.0, 1.0.1
>
>
> There is a bug in the bfs algorithm in RAS that does not correctly account 
> for components already visited during the breadth first traveral



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1681) Bug in scheduling cyclic topologies when scheduling with RAS

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1681.

Resolution: Fixed

> Bug in scheduling cyclic topologies when scheduling with RAS
> 
>
> Key: STORM-1681
> URL: https://issues.apache.org/jira/browse/STORM-1681
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Boyang Jerry Peng
>Assignee: Boyang Jerry Peng
>Priority: Critical
> Fix For: 2.0.0, 1.0.1
>
>
> There is a bug in the bfs algorithm in RAS that does not correctly account 
> for components already visited during the breadth first traveral



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1585) Add DDL support for UDFs in Storm-sql

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1585:
---
Affects Version/s: 2.0.0
   1.0.0
Fix Version/s: 1.0.1
   2.0.0

> Add DDL support for UDFs in Storm-sql
> -
>
> Key: STORM-1585
> URL: https://issues.apache.org/jira/browse/STORM-1585
> Project: Apache Storm
>  Issue Type: Sub-task
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1585) Add DDL support for UDFs in Storm-sql

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1585.

Resolution: Fixed

> Add DDL support for UDFs in Storm-sql
> -
>
> Key: STORM-1585
> URL: https://issues.apache.org/jira/browse/STORM-1585
> Project: Apache Storm
>  Issue Type: Sub-task
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1706) Add storm-env.ini and RELEASE to storm-dist assembly

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1706.

Resolution: Fixed

> Add storm-env.ini and RELEASE to storm-dist assembly
> 
>
> Key: STORM-1706
> URL: https://issues.apache.org/jira/browse/STORM-1706
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
> Fix For: 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1693) Negative counts in the UI for __metrics stream

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1693.

Resolution: Fixed

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Fix For: 1.0.1
>
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1693) Negative counts in the UI for __metrics stream

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1693:
---
Fix Version/s: 1.0.1

> Negative counts in the UI for __metrics stream
> --
>
> Key: STORM-1693
> URL: https://issues.apache.org/jira/browse/STORM-1693
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Abhishek Agarwal
>Assignee: Abhishek Agarwal
>Priority: Critical
> Fix For: 1.0.1
>
> Attachments: Screen Shot 2016-04-07 at 7.05.01 PM.png
>
>
> Metrics reported by UI are not correct. I am seeing negative counts for 
> output stats in the bolt. The same application code works fine on 0.9.6 
> version of storm. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1696) Backpressure flag not sync if zookeeper connection errors

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1696:
---
Fix Version/s: (was: 1.0.0)
   1.0.1

> Backpressure flag not sync if zookeeper connection errors
> -
>
> Key: STORM-1696
> URL: https://issues.apache.org/jira/browse/STORM-1696
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Blocker
> Fix For: 2.0.0, 1.0.1
>
>
> When there is a zk exception happens during worker-backpressure!,
> there is a bad state which can block the topology from running normally any 
> more.
> The root cause: in worker/mk-backpressure-handler
> if the worker-backpressure! fails once due to zk connection exception,
> next time when this method gets called by WordBackpressureThread, because 
> (when (not= prev-backpressure-flag curr-backpressure-flag) will never be 
> true, the remote zk node can not be synced with local state.
> This also explains why we will not see any problem when testing in a stable 
> (zk never fail) environment.
> Solution is quite straightforward: first change the zk status, if succeeds, 
> change local status.
> This fixes the hidden bug and removes redundant flags in executor-data and 
> worker-data (since we can get the executor status directly from the 
> "_throttleOn" boolean in the DisruptorQueue)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1696) Backpressure flag not sync if zookeeper connection errors

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1696.

Resolution: Fixed

> Backpressure flag not sync if zookeeper connection errors
> -
>
> Key: STORM-1696
> URL: https://issues.apache.org/jira/browse/STORM-1696
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Blocker
> Fix For: 2.0.0, 1.0.1
>
>
> When there is a zk exception happens during worker-backpressure!,
> there is a bad state which can block the topology from running normally any 
> more.
> The root cause: in worker/mk-backpressure-handler
> if the worker-backpressure! fails once due to zk connection exception,
> next time when this method gets called by WordBackpressureThread, because 
> (when (not= prev-backpressure-flag curr-backpressure-flag) will never be 
> true, the remote zk node can not be synced with local state.
> This also explains why we will not see any problem when testing in a stable 
> (zk never fail) environment.
> Solution is quite straightforward: first change the zk status, if succeeds, 
> change local status.
> This fixes the hidden bug and removes redundant flags in executor-data and 
> worker-data (since we can get the executor status directly from the 
> "_throttleOn" boolean in the DisruptorQueue)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1706) Add storm-env.ini and RELEASE to storm-dist assembly

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1706:
---
Affects Version/s: 1.0.0
Fix Version/s: (was: 1.0.0)
   1.0.1

> Add storm-env.ini and RELEASE to storm-dist assembly
> 
>
> Key: STORM-1706
> URL: https://issues.apache.org/jira/browse/STORM-1706
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Sriharsha Chintalapani
>Assignee: Priyank Shah
> Fix For: 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1687) Divide by zero exception in stats

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1687:
---
Fix Version/s: (was: 1.0.0)
   1.0.1

> Divide by zero exception in stats
> -
>
> Key: STORM-1687
> URL: https://issues.apache.org/jira/browse/STORM-1687
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
> Fix For: 2.0.0, 1.0.1
>
>
> Since uptime can be 0, this will cause ArithmeticException: Divide by zero in 
> compute-agg-capacity.
> This will happen for both stats.clj in 1.x and StatsUtil.java in master (2.0).
> {noformat}
> java.lang.ArithmeticException: Divide by zero
> at clojure.lang.Numbers.divide(Numbers.java:156)
> at clojure.core$SLASH.invoke(core.clj:986)
> at clojure.lang.AFn.applyToHelper(AFn.java:156)
> at clojure.lang.RestFn.applyTo(RestFn.java:132)
> at clojure.core$apply.invoke(core.clj:626)
> at backtype.storm.util$div.doInvoke(util.clj:355)
> at clojure.lang.RestFn.invoke(RestFn.java:423)
> at backtype.storm.stats$compute_agg_capacity$fn__2249.invoke(stats.clj:409)
> at backtype.storm.stats$compute_agg_capacity.invoke(stats.clj:404)
> at backtype.storm.stats$agg_pre_merge_topo_page_bolt.invoke(stats.clj:555)
> at backtype.storm.stats$agg_topo_exec_stats_STAR_.invoke(stats.clj:724)
> at backtype.storm.stats$fn__2319.invoke(stats.clj:772)
> at clojure.lang.MultiFn.invoke(MultiFn.java:241)
> at clojure.lang.AFn.applyToHelper(AFn.java:165)
> at clojure.lang.AFn.applyTo(AFn.java:144)
> at clojure.core$apply.invoke(core.clj:628)
> at clojure.core$partial$fn__4230.doInvoke(core.clj:2470)
> at clojure.lang.RestFn.invoke(RestFn.java:421)
> at clojure.core.protocols$fn__6086.invoke(protocols.clj:143)
> at clojure.core.protocols$fn_6057$G6052_6066.invoke(protocols.clj:19)
> at clojure.core.protocols$seq_reduce.invoke(protocols.clj:31)
> at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
> at clojure.core.protocols$fn_6031$G6026_6044.invoke(protocols.clj:13)
> at clojure.core$reduce.invoke(core.clj:6289)
> at backtype.storm.stats$aggregate_topo_stats.invoke(stats.clj:854)
> at backtype.storm.stats$agg_topo_execs_stats.invoke(stats.clj:1008)
> at 
> backtype.storm.daemon.nimbus$fn_5838$exec_fn1478auto$reify_5862.getTopologyPageInfo(nimbus.clj:1729)
> at 
> backtype.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3651)
> at 
> backtype.storm.generated.Nimbus$Processor$getTopologyPageInfo.getResult(Nimbus.java:3635)
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> backtype.storm.security.auth.SaslTransportPlugin$TUGIWrapProcessor.process(SaslTransportPlugin.java:143)
> at 
> org.apache.thrift7.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1613) upgrade hbase version from 0.98.4-hadoop2 to 1.1

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1613:
---
Fix Version/s: 1.0.1

> upgrade hbase version from 0.98.4-hadoop2 to 1.1
> 
>
> Key: STORM-1613
> URL: https://issues.apache.org/jira/browse/STORM-1613
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Affects Versions: 1.0.0
>Reporter: Satish Duggana
>Assignee: Satish Duggana
> Fix For: 1.0.1
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1721) Epic for 1.0.1 Release

[P. Taylor Goetz (JIRA)]

P. Taylor Goetz created STORM-1721:
--

 Summary: Epic for 1.0.1 Release
 Key: STORM-1721
 URL: https://issues.apache.org/jira/browse/STORM-1721
 Project: Apache Storm
  Issue Type: Epic
Reporter: P. Taylor Goetz
Assignee: P. Taylor Goetz






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1721) Release Apache Storm 1.0.1

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz updated STORM-1721:
---
Summary: Release Apache Storm 1.0.1  (was: Epic for 1.0.1 Release)

> Release Apache Storm 1.0.1
> --
>
> Key: STORM-1721
> URL: https://issues.apache.org/jira/browse/STORM-1721
> Project: Apache Storm
>  Issue Type: Epic
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (STORM-1491) Release Apache Storm 1.0.0

[P. Taylor Goetz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

P. Taylor Goetz resolved STORM-1491.

Resolution: Fixed

> Release Apache Storm 1.0.0
> --
>
> Key: STORM-1491
> URL: https://issues.apache.org/jira/browse/STORM-1491
> Project: Apache Storm
>  Issue Type: Epic
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>
> This is an umbrella task for tracking all new/unresolved issues that should 
> be fixed prior to releasing 1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Publishing cluster stats on Apache Storm

[Harsha]

Jungtaek,
  Ideally we should avoid using NimbusClient as much as
  possible for a polling reporter like this. Can we have
  plug gable interface that can be run inside the nimbus at
  configured interval . Since it will be plugged into nimbus
  we would have all cluster details including topologies as
  well.

Thanks,
Harsha


On Mon, Apr 18, 2016, at 10:18 PM, Jungtaek Lim wrote:
> Cody,
> 
> Great idea! Let's try it out.
> I'll send the mail to user@ regarding use cases or concerns of metrics on
> Apache Storm. We may have a luck to find brilliant ideas or use cases we
> didn't know.
> 
> 2016년 4월 19일 (화) 오후 2:07, Cody Innowhere 님이 작성:
> 
> > Jungtaek,
> > Thanks for your explanation, still +1 for your proposal.
> > Moreover, I'd like to hear storm users about this, especially use cases for
> > metrics so that we can see what's best to do this.
> >
> > On Tue, Apr 19, 2016 at 11:19 AM, Jungtaek Lim  wrote:
> >
> > > Cody,
> > > Thanks for giving an opinion.
> > >
> > > I guess there was a miscommunication between you and me.
> > > This shouldn't be related to topology. Plugin should be set up via
> > cluster
> > > configuration, and singular daemon like Nimbus (master) or UI should
> > > publish cluster metrics to the plugin.
> > > I agree Nimbus should have this feature, but only master should publish
> > the
> > > cluster metrics so we should handle it.
> > >
> > > Btw, since we don't have built-in metrics storage, we can't provide query
> > > API with time-series fashion.
> > > I guess publishing (pushing) seems the better way what we can address
> > > without huge modification.
> > > (Did you mean Nimbus has query API which queries the external storage?)
> > >
> > > Please note that it's an idea for improvement against 1.x, not 2.0.
> > > In phase 2 we should evaluate and replace metrics feature with JStorm or
> > > another after porting to Java.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2016년 4월 19일 (화) 오전 11:59, Cody Innowhere 님이 작성:
> > >
> > > > I'm +1 for publishing cluster stats, but don't quite understand your
> > > > implementation.
> > > > Do you mean that a topology which wants cluster stats can get cluster
> > > stats
> > > > by implementing the pluggable consumer/reporter?
> > > > Personally I would prefer to put this role into nimbus (can be
> > pluggable
> > > > too) since it's the responsibility of nimbus to take care of all
> > > > topologies, naturally all topology stats. Then we expose API's from
> > > nimbus
> > > > for external queries.
> > > >
> > > > On Tue, Apr 19, 2016 at 9:47 AM, Jungtaek Lim 
> > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > While Storm publishes topology metrics by metrics consumers, Storm
> > > > doesn't
> > > > > publish cluster metrics any way.
> > > > >
> > > > > Therefore, I'd like to introduce the feature of publishing cluster
> > > stats
> > > > to
> > > > > pluggable consumers so that users can also push to external storages
> > > and
> > > > > query on it or even configure dashboard.
> > > > > (For topology stats it's supported via MetricsConsumer.)
> > > > >
> > > > > I'm seeing some workarounds like having their own reporter process
> > > > polling
> > > > > cluster information from Nimbus or REST API.
> > > > > (For example, Ambari has cluster metrics reporter for Storm
> > > > > <
> > > > >
> > > >
> > >
> > https://github.com/apache/ambari/blob/trunk/ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java
> > > > > >
> > > > > but it relies on custom build of Storm.)
> > > > > Yes it works anyway but if Storm provides the feature naturally, with
> > > > > pluggable way like MetricsConsumer it would be great for users who
> > want
> > > > to
> > > > > configure dashboard regarding this.
> > > > >
> > > > > I also saw that STORM-1158 <
> > > > > http://issues.apache.org/jira/browse/STORM-1158>
> > > > > adds metrics reporter for internal actions, but it has some
> > limitation
> > > on
> > > > > it.
> > > > >
> > > > > - It focuses how many requests the daemon receives, not cluster
> > stats.
> > > > > - It requires reporter to use codahale metrics. Moreover it shades
> > > > > codahale-metrics so other implementations of reporter may not work
> > > > properly
> > > > > if it uses other codahale-metrics plugin like ganglia.
> > > > >
> > > > > So I'm planning to add metrics feature of cluster stat by not relying
> > > on
> > > > > daemon metrics reporter, but introduce new pluggable
> > consumer/reporter.
> > > > >
> > > > > What do you think? Do you have opinion that we need to add cluster
> > > stats
> > > > to
> > > > > daemon metrics reporter?
> > > > >
> > > > > Thanks,
> > > > > Jungtaek Lim (HeartSaVioR)
> > > > >
> > > >
> > >
> >