Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)
+1 -- Derek - Original Message - From: P. Taylor Goetz To: dev@storm.apache.org Cc: Sent: Wednesday, November 11, 2015 4:21 PM Subject: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release) Changing subject in order to consolidate discussion of a 1.0 release in one thread (there was some additional discussion in the thread regarding the JStorm code merge). I just want to make sure I’m accurately capturing the sentiment of the community with regard to a 1.0 release. Please correct me if my summary seems off-base or jump in with an opinion. In summary: 1. What we have been calling “0.11.0” will become the Storm 1.0 release. 2. We will NOT be migrating package names for this release (i.e. “backtype.storm” —> “org.apache.storm”). 3. Post 1.0 release we will go into feature freeze for core functionality to facilitate the JStorm merge. 4. During the feature freeze only fixes for high priority bugs in core functionality will be accepted (no new features). 5. During the feature freeze, enhancements to “external” modules can be accepted. 6. We will stop using the “beta” flag in favor of purely numeric version numbers. Stable vs. non-stable (development) releases can be indicated on the download page. Do we all agree? -Taylor > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz wrote: > > >> On Nov 11, 2015, at 9:28 AM, Bobby Evans wrote: >> >> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, >> STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the list. >> >> Some of them are more important then others but they are all things I would >> like to see in a 0.11.0 release. > > > Cool. Thanks for listing them out. I will add them so they get tracked. > > >> On a side note I don't think the beta releases have been helpful. I would >> much rather just have a 0.11.0 go to 0.11.1 ... instead of 0.11.0-beta1, >> 0.11.0-beta2. To me the beta label is not that helpful, but it is not that >> big of a deal for me. > > In my mind, having releases tagged as “beta” were a way for us to say to > users “here’s a preview release that will allow you to kick the tires on > upcoming features, but be aware that there might be bugs. Let us know if you > find any so we can fix them.” > > I think the intent was sound, but what I didn’t really see was user feedback > on the beta releases, which is what I hoped would happen. So I have no > problem with dropping the use of “beta” flags. > > Another approach I’ve seen other Apache projects use is to the numbering > scheme you describe, and just have different labels/descriptions on the > download page — i.e. “Latest stable release” and “Latest development > release.” The nice part of that convention is that it does not have any > impact on the release process. For example if we’re confident that a > particular “development” release is actually quite stable, all we would have > to do is update the downloads page, rather than go through the whole > release/vote process just to remove the “beta” tag. > >> I also would like to see the 0.11 release tied to the plan for the JStorm >> merger. If we don't tie them together there can be code that does not make >> it into 0.11, but could make it into a 0.12 that will immediately be caught >> up in the merger, that could take a long time to complete. I mostly want us >> to be very transparent about what is likely to happen after 0.11 is >> released. So if someone has a feature that is close to getting something in >> to 0.11 that they speak up here instead of just deciding to wait for a 0.12 >> release. > > > I agree that the 0.11 release needs to be tied to the JStorm merger. Once > that release goes out, we’ll be going into lockdown mode for the merge > effort, which is likely to take a while. > > During that time it’s highly unlikely that any changes/additions to Storm’s > core functionality will be accepted beyond high-priority bug fixes. Adding > new features to the “external” components during that time is probably okay, > since those components are sufficiently decoupled from Storm’s core > functionality. > > So to reiterate Bobby’s statement: > > Please speak up now if there are any specific features or changes to Storm’s > core functionality that you’d like to see in the next release. Otherwise you > will have to wait. > >> - Bobby > > -Taylor > >> >> >>On Wednesday, November 11, 2015 6:32 AM, John Fang >> wrote: >> >> >> Totally agree， it's great to accelerate the release speed. it is better >> release one official version within 3 months. The first month is for >> developing new features , .If some features hasn't been finished in this >> month, it will be put in the next release ticket. In the last two month, we >> just do test. >> Storm may face the greatest challenge in a big cluster, so I am more >> concerned about ZK Optimization as jstorm did. At last, it's great for the >> next release to
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[
https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001244#comment-15001244
]
ASF GitHub Bot commented on STORM-1198:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44596285
--- Diff: storm-core/src/ui/public/templates/index-page-template.html ---
@@ -229,6 +249,10 @@
{{uptime}}
{{slotsTotal}}
{{slotsUsed}}
+ {{totalMem}}
+ {{usedMem}}
+ {{totalCpu}}
+ {{usedCpu}}
--- End diff --
Could you update the REST API docs to indicate the new fields
> Web UI to show resource usages and Total Resources on all supervisors
> -
>
> Key: STORM-1198
> URL: https://issues.apache.org/jira/browse/STORM-1198
> Project: Apache Storm
> Issue Type: Story
> Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Attachments: supervisor-resources.png
>
>
> As we have resource aware scheduler (STORM-894), we want to be able to
> display resource capacity (CPU, memory; and network in future) and scheduled
> resource usage on each supervisor node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (STORM-1200) Support collations of primary keys
Haohui Mai created STORM-1200: - Summary: Support collations of primary keys Key: STORM-1200 URL: https://issues.apache.org/jira/browse/STORM-1200 Project: Apache Storm Issue Type: New Feature Components: storm-sql Reporter: Haohui Mai Assignee: Haohui Mai This jira proposes to add support of specifying collations of primary keys. Collations provide information on uniqueness and monotonicity of columns. The information is essential to implement aggregation functions over streaming data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001256#comment-15001256 ] ASF GitHub Bot commented on STORM-1198: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155931351 I just tested what @redsanket asked and topologies not using the resource aware scheduler show no resources used at all. It really would be good if we could parse out the heap from the command line of a worker, and do a quick SWAG on the CPU when not using RAS. That way we could at least see if memory/CPU is being over committed on a node. > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)
+1. On 11/11/15, 2:27 PM, "Derek Dagit" wrote: >+1 > > -- >Derek > > >- Original Message - >From: P. Taylor Goetz >To: dev@storm.apache.org >Cc: >Sent: Wednesday, November 11, 2015 4:21 PM >Subject: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release) > >Changing subject in order to consolidate discussion of a 1.0 release in >one thread (there was some additional discussion in the thread regarding >the JStorm code merge). > >I just want to make sure I’m accurately capturing the sentiment of the >community with regard to a 1.0 release. Please correct me if my summary >seems off-base or jump in with an opinion. > >In summary: > >1. What we have been calling “0.11.0” will become the Storm 1.0 release. >2. We will NOT be migrating package names for this release (i.e. >“backtype.storm” ―> “org.apache.storm”). >3. Post 1.0 release we will go into feature freeze for core functionality >to facilitate the JStorm merge. >4. During the feature freeze only fixes for high priority bugs in core >functionality will be accepted (no new features). >5. During the feature freeze, enhancements to “external” modules can be >accepted. >6. We will stop using the “beta” flag in favor of purely numeric version >numbers. Stable vs. non-stable (development) releases can be indicated on >the download page. > >Do we all agree? > >-Taylor > > >> On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz wrote: >> >> >>> On Nov 11, 2015, at 9:28 AM, Bobby Evans >>>wrote: >>> >>> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, >>>STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the >>>list. >>> >>> Some of them are more important then others but they are all things I >>>would like to see in a 0.11.0 release. >> >> >> Cool. Thanks for listing them out. I will add them so they get tracked. >> >> >>> On a side note I don't think the beta releases have been helpful. I >>>would much rather just have a 0.11.0 go to 0.11.1 ... instead of >>>0.11.0-beta1, 0.11.0-beta2. To me the beta label is not that helpful, >>>but it is not that big of a deal for me. >> >> In my mind, having releases tagged as “beta” were a way for us to say >>to users “here’s a preview release that will allow you to kick the tires >>on upcoming features, but be aware that there might be bugs. Let us know >>if you find any so we can fix them.” >> >> I think the intent was sound, but what I didn’t really see was user >>feedback on the beta releases, which is what I hoped would happen. So I >>have no problem with dropping the use of “beta” flags. >> >> Another approach I’ve seen other Apache projects use is to the >>numbering scheme you describe, and just have different >>labels/descriptions on the download page ― i.e. “Latest stable release” >>and “Latest development release.” The nice part of that convention is >>that it does not have any impact on the release process. For example if >>we’re confident that a particular “development” release is actually >>quite stable, all we would have to do is update the downloads page, >>rather than go through the whole release/vote process just to remove the >>“beta” tag. >> >>> I also would like to see the 0.11 release tied to the plan for the >>>JStorm merger. If we don't tie them together there can be code that >>>does not make it into 0.11, but could make it into a 0.12 that will >>>immediately be caught up in the merger, that could take a long time to >>>complete. I mostly want us to be very transparent about what is likely >>>to happen after 0.11 is released. So if someone has a feature that is >>>close to getting something in to 0.11 that they speak up here instead >>>of just deciding to wait for a 0.12 release. >> >> >> I agree that the 0.11 release needs to be tied to the JStorm merger. >>Once that release goes out, we’ll be going into lockdown mode for the >>merge effort, which is likely to take a while. >> >> During that time it’s highly unlikely that any changes/additions to >>Storm’s core functionality will be accepted beyond high-priority bug >>fixes. Adding new features to the “external” components during that time >>is probably okay, since those components are sufficiently decoupled from >>Storm’s core functionality. >> >> So to reiterate Bobby’s statement: >> >> Please speak up now if there are any specific features or changes to >>Storm’s core functionality that you’d like to see in the next release. >>Otherwise you will have to wait. >> >>> - Bobby >> >> -Taylor >> >>> >>> >>>On Wednesday, November 11, 2015 6:32 AM, John Fang >>> wrote: >>> >>> >>> Totally agree， it's great to accelerate the release speed. it is >>>better release one official version within 3 months. The first month is >>>for developing new features , .If some features hasn't been finished in >>>this month, it will be put in the next release ticket. In the last two >>>month, we just do test. >>> Storm may face the greatest challenge in a big cluster, so I am more >>>
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155931351 I just tested what @redsanket asked and topologies not using the resource aware scheduler show no resources used at all. It really would be good if we could parse out the heap from the command line of a worker, and do a quick SWAG on the CPU when not using RAS. That way we could at least see if memory/CPU is being over committed on a node. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-1201) Support distributed deployment in StormSQL
Haohui Mai created STORM-1201: - Summary: Support distributed deployment in StormSQL Key: STORM-1201 URL: https://issues.apache.org/jira/browse/STORM-1201 Project: Apache Storm Issue Type: New Feature Components: storm-sql Reporter: Haohui Mai StormSQL compiles the SQL statements into Java classes. The Trident topology executes these classes in order to execute the SQL statements. These classes need to be properly distributed through Nimbus so that the topology can be run in distributed mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44597539
--- Diff: storm-core/src/ui/public/templates/index-page-template.html ---
@@ -229,6 +249,10 @@
{{uptime}}
{{slotsTotal}}
{{slotsUsed}}
+ {{totalMem}}
+ {{usedMem}}
+ {{totalCpu}}
+ {{usedCpu}}
--- End diff --
Sure, adding it.
For non-RAS scheduler, the totalMem and totalCpu would be default(3000 and
400), and the usedMem and usedCpu will be always 0 and 0, no matter whether
there is topology or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[
https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001258#comment-15001258
]
ASF GitHub Bot commented on STORM-1198:
---
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44597539
--- Diff: storm-core/src/ui/public/templates/index-page-template.html ---
@@ -229,6 +249,10 @@
{{uptime}}
{{slotsTotal}}
{{slotsUsed}}
+ {{totalMem}}
+ {{usedMem}}
+ {{totalCpu}}
+ {{usedCpu}}
--- End diff --
Sure, adding it.
For non-RAS scheduler, the totalMem and totalCpu would be default(3000 and
400), and the usedMem and usedCpu will be always 0 and 0, no matter whether
there is topology or not.
> Web UI to show resource usages and Total Resources on all supervisors
> -
>
> Key: STORM-1198
> URL: https://issues.apache.org/jira/browse/STORM-1198
> Project: Apache Storm
> Issue Type: Story
> Components: storm-core
>Reporter: Zhuo Liu
>Assignee: Zhuo Liu
>Priority: Minor
> Attachments: supervisor-resources.png
>
>
> As we have resource aware scheduler (STORM-894), we want to be able to
> display resource capacity (CPU, memory; and network in future) and scheduled
> resource usage on each supervisor node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/871#discussion_r44597850 --- Diff: external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java --- @@ -56,9 +56,9 @@ private ExecutorService callTimeoutPool; private transient Timer heartBeatTimer; private Boolean kerberosEnabled = false; -private AtomicBoolean timeToSendHeartBeat = new AtomicBoolean(false); +private Boolean sendHeartBeat = true; --- End diff -- I think this needs to be volatile or atomic. It is written in one thread and read in another with no locks around any of it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[ https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001261#comment-15001261 ] ASF GitHub Bot commented on STORM-1030: --- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/871#discussion_r44597850 --- Diff: external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java --- @@ -56,9 +56,9 @@ private ExecutorService callTimeoutPool; private transient Timer heartBeatTimer; private Boolean kerberosEnabled = false; -private AtomicBoolean timeToSendHeartBeat = new AtomicBoolean(false); +private Boolean sendHeartBeat = true; --- End diff -- I think this needs to be volatile or atomic. It is written in one thread and read in another with no locks around any of it. > Hive Connector Fixes > > > Key: STORM-1030 > URL: https://issues.apache.org/jira/browse/STORM-1030 > Project: Apache Storm > Issue Type: Bug > Components: storm-hive >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 0.11.0 > > > 1. Schedule Hive transaction heartbeats outside of execute method. > 2. Fix retiring idleWriters > 3. Do not call flush if there is no data added to a txnbatch > 4. Catch any exception and abort transaction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155932528 @redsanket good point! the ui should still be compatible if RAS is not used --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001262#comment-15001262 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155932528 @redsanket good point! the ui should still be compatible if RAS is not used > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155933049 Can we just not show any of the columns that deal with resources if the scheduler used is not RAS --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001268#comment-15001268 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155933049 Can we just not show any of the columns that deal with resources if the scheduler used is not RAS > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)
+1 Jungtaek Lim (HeartSaVioR) 2015-11-12 7:21 GMT+09:00 P. Taylor Goetz : > Changing subject in order to consolidate discussion of a 1.0 release in > one thread (there was some additional discussion in the thread regarding > the JStorm code merge). > > I just want to make sure I’m accurately capturing the sentiment of the > community with regard to a 1.0 release. Please correct me if my summary > seems off-base or jump in with an opinion. > > In summary: > > 1. What we have been calling “0.11.0” will become the Storm 1.0 release. > 2. We will NOT be migrating package names for this release (i.e. > “backtype.storm” —> “org.apache.storm”). > 3. Post 1.0 release we will go into feature freeze for core functionality > to facilitate the JStorm merge. > 4. During the feature freeze only fixes for high priority bugs in core > functionality will be accepted (no new features). > 5. During the feature freeze, enhancements to “external” modules can be > accepted. > 6. We will stop using the “beta” flag in favor of purely numeric version > numbers. Stable vs. non-stable (development) releases can be indicated on > the download page. > > Do we all agree? > > -Taylor > > > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz wrote: > > > > > >> On Nov 11, 2015, at 9:28 AM, Bobby Evans > wrote: > >> > >> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, > STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the > list. > >> > >> Some of them are more important then others but they are all things I > would like to see in a 0.11.0 release. > > > > > > Cool. Thanks for listing them out. I will add them so they get tracked. > > > > > >> On a side note I don't think the beta releases have been helpful. I > would much rather just have a 0.11.0 go to 0.11.1 ... instead of > 0.11.0-beta1, 0.11.0-beta2. To me the beta label is not that helpful, but > it is not that big of a deal for me. > > > > In my mind, having releases tagged as “beta” were a way for us to say to > users “here’s a preview release that will allow you to kick the tires on > upcoming features, but be aware that there might be bugs. Let us know if > you find any so we can fix them.” > > > > I think the intent was sound, but what I didn’t really see was user > feedback on the beta releases, which is what I hoped would happen. So I > have no problem with dropping the use of “beta” flags. > > > > Another approach I’ve seen other Apache projects use is to the numbering > scheme you describe, and just have different labels/descriptions on the > download page — i.e. “Latest stable release” and “Latest development > release.” The nice part of that convention is that it does not have any > impact on the release process. For example if we’re confident that a > particular “development” release is actually quite stable, all we would > have to do is update the downloads page, rather than go through the whole > release/vote process just to remove the “beta” tag. > > > >> I also would like to see the 0.11 release tied to the plan for the > JStorm merger. If we don't tie them together there can be code that does > not make it into 0.11, but could make it into a 0.12 that will immediately > be caught up in the merger, that could take a long time to complete. I > mostly want us to be very transparent about what is likely to happen after > 0.11 is released. So if someone has a feature that is close to getting > something in to 0.11 that they speak up here instead of just deciding to > wait for a 0.12 release. > > > > > > I agree that the 0.11 release needs to be tied to the JStorm merger. > Once that release goes out, we’ll be going into lockdown mode for the merge > effort, which is likely to take a while. > > > > During that time it’s highly unlikely that any changes/additions to > Storm’s core functionality will be accepted beyond high-priority bug fixes. > Adding new features to the “external” components during that time is > probably okay, since those components are sufficiently decoupled from > Storm’s core functionality. > > > > So to reiterate Bobby’s statement: > > > > Please speak up now if there are any specific features or changes to > Storm’s core functionality that you’d like to see in the next release. > Otherwise you will have to wait. > > > >> - Bobby > > > > -Taylor > > > >> > >> > >>On Wednesday, November 11, 2015 6:32 AM, John Fang < > xiaojian@alibaba-inc.com> wrote: > >> > >> > >> Totally agree， it's great to accelerate the release speed. it is > better release one official version within 3 months. The first month is for > developing new features , .If some features hasn't been finished in this > month, it will be put in the next release ticket. In the last two month, we > just do test. > >> Storm may face the greatest challenge in a big cluster, so I am more > concerned about ZK Optimization as jstorm did. At last, it's great for the > next release to has a test report about the stability and performance , due > t
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598471
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Why ack this, when before we would fail it? I realize replaying this is
not going to "fix" the problem, but hiding a failure seems worse.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user zhuoliu commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155933406 @revans2 If we want to show allocated cpu/mem for non-RAS schedulers, we will also need to do that for topology view except for the supervisor view. Also, it would be also possible that the usedMem might be larger than totalMem (since totalMem may not be correctly configured for non-RAS schedulers, e.g., using default 3GB). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001274#comment-15001274 ] ASF GitHub Bot commented on STORM-1198: --- Github user zhuoliu commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155933406 @revans2 If we want to show allocated cpu/mem for non-RAS schedulers, we will also need to do that for topology view except for the supervisor view. Also, it would be also possible that the usedMem might be larger than totalMem (since totalMem may not be correctly configured for non-RAS schedulers, e.g., using default 3GB). > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[
https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001273#comment-15001273
]
ASF GitHub Bot commented on STORM-1030:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598471
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Why ack this, when before we would fail it? I realize replaying this is
not going to "fix" the problem, but hiding a failure seems worse.
> Hive Connector Fixes
>
>
> Key: STORM-1030
> URL: https://issues.apache.org/jira/browse/STORM-1030
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>
> 1. Schedule Hive transaction heartbeats outside of execute method.
> 2. Fix retiring idleWriters
> 3. Do not call flush if there is no data added to a txnbatch
> 4. Catch any exception and abort transaction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598712
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Also the other error handling code now deals with a tuple batch and
fails/cleans it up. Does this code need to handle that too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[
https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001279#comment-15001279
]
ASF GitHub Bot commented on STORM-1030:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598712
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Also the other error handling code now deals with a tuple batch and
fails/cleans it up. Does this code need to handle that too?
> Hive Connector Fixes
>
>
> Key: STORM-1030
> URL: https://issues.apache.org/jira/browse/STORM-1030
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>
> 1. Schedule Hive transaction heartbeats outside of execute method.
> 2. Fix retiring idleWriters
> 3. Do not call flush if there is no data added to a txnbatch
> 4. Catch any exception and abort transaction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598767
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
} catch(Exception e) {
this.collector.reportError(e);
collector.fail(tuple);
-try {
-flushAndCloseWriters();
-LOG.info("acknowledging tuples after writers flushed and
closed");
-for (Tuple t : tupleBatch)
-collector.ack(t);
-tupleBatch.clear();
-} catch (Exception e1) {
-//If flushAndClose fails assume tuples are lost, do not ack
-LOG.warn("Error while flushing and closing writers, tuples
will NOT be acknowledged");
-for (Tuple t : tupleBatch)
-collector.fail(t);
-tupleBatch.clear();
-}
+for (Tuple t : tupleBatch)
--- End diff --
Could you wrap the body of this in '{}' I know it is just one line, but for
coding conventions sake.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[
https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001280#comment-15001280
]
ASF GitHub Bot commented on STORM-1030:
---
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44598767
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
} catch(Exception e) {
this.collector.reportError(e);
collector.fail(tuple);
-try {
-flushAndCloseWriters();
-LOG.info("acknowledging tuples after writers flushed and
closed");
-for (Tuple t : tupleBatch)
-collector.ack(t);
-tupleBatch.clear();
-} catch (Exception e1) {
-//If flushAndClose fails assume tuples are lost, do not ack
-LOG.warn("Error while flushing and closing writers, tuples
will NOT be acknowledged");
-for (Tuple t : tupleBatch)
-collector.fail(t);
-tupleBatch.clear();
-}
+for (Tuple t : tupleBatch)
--- End diff --
Could you wrap the body of this in '{}' I know it is just one line, but for
coding conventions sake.
> Hive Connector Fixes
>
>
> Key: STORM-1030
> URL: https://issues.apache.org/jira/browse/STORM-1030
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>
> 1. Schedule Hive transaction heartbeats outside of execute method.
> 2. Fix retiring idleWriters
> 3. Do not call flush if there is no data added to a txnbatch
> 4. Catch any exception and abort transaction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/871#issuecomment-155934542 Done with a quick pass through the code, for the most part it looks good, just a few minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[ https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001282#comment-15001282 ] ASF GitHub Bot commented on STORM-1030: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/871#issuecomment-155934542 Done with a quick pass through the code, for the most part it looks good, just a few minor comments. > Hive Connector Fixes > > > Key: STORM-1030 > URL: https://issues.apache.org/jira/browse/STORM-1030 > Project: Apache Storm > Issue Type: Bug > Components: storm-hive >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 0.11.0 > > > 1. Schedule Hive transaction heartbeats outside of execute method. > 2. Fix retiring idleWriters > 3. Do not call flush if there is no data added to a txnbatch > 4. Catch any exception and abort transaction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[
https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001296#comment-15001296
]
ASF GitHub Bot commented on STORM-1030:
---
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44600086
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
@revans2 Hive SerializationError means the tuple fields to table column
mapping is nto right. For example if table has 6 rows and this tuple has only 5
fields than it can throw SerializationError. In this case instead of failing
the tuple and keep re-running into this error. Its better we ack and log an
error.
hiveWriter.write will throw an error on the current tuple and it wil be not
be added to the tupleBatch. Since this is only happening for current the other
tuples in the batch are good and we can still proceed writing them to hive.
> Hive Connector Fixes
>
>
> Key: STORM-1030
> URL: https://issues.apache.org/jira/browse/STORM-1030
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>
> 1. Schedule Hive transaction heartbeats outside of execute method.
> 2. Fix retiring idleWriters
> 3. Do not call flush if there is no data added to a txnbatch
> 4. Catch any exception and abort transaction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44600086
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
@revans2 Hive SerializationError means the tuple fields to table column
mapping is nto right. For example if table has 6 rows and this tuple has only 5
fields than it can throw SerializationError. In this case instead of failing
the tuple and keep re-running into this error. Its better we ack and log an
error.
hiveWriter.write will throw an error on the current tuple and it wil be not
be added to the tupleBatch. Since this is only happening for current the other
tuples in the batch are good and we can still proceed writing them to hive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-1030) Hive Connector Fixes
[
https://issues.apache.org/jira/browse/STORM-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001300#comment-15001300
]
ASF GitHub Bot commented on STORM-1030:
---
Github user dossett commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44600468
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Should the try {...} catch (SerializationError) block be moved to cover
just the writer.write() call then?
> Hive Connector Fixes
>
>
> Key: STORM-1030
> URL: https://issues.apache.org/jira/browse/STORM-1030
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-hive
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.11.0
>
>
> 1. Schedule Hive transaction heartbeats outside of execute method.
> 2. Fix retiring idleWriters
> 3. Do not call flush if there is no data added to a txnbatch
> 4. Catch any exception and abort transaction.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1030. Hive Connector Fixes.
Github user dossett commented on a diff in the pull request:
https://github.com/apache/storm/pull/871#discussion_r44600468
--- Diff:
external/storm-hive/src/main/java/org/apache/storm/hive/bolt/HiveBolt.java ---
@@ -134,22 +131,16 @@ public void execute(Tuple tuple) {
collector.ack(t);
tupleBatch.clear();
}
+} catch(SerializationError se) {
+LOG.info("Serialization exception occurred, tuple is
acknowledged but not written to Hive.", tuple);
+collector.ack(tuple);
--- End diff --
Should the try {...} catch (SerializationError) block be moved to cover
just the writer.write() call then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-794 Modify Spout async loop to treat act...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/542#issuecomment-155947307 @revans2 Upmerged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-794) Trident Topology with some situation seems not handle deactivate during graceful shutdown
[
https://issues.apache.org/jira/browse/STORM-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001353#comment-15001353
]
ASF GitHub Bot commented on STORM-794:
--
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/542#issuecomment-155947307
@revans2 Upmerged.
> Trident Topology with some situation seems not handle deactivate during
> graceful shutdown
> -
>
> Key: STORM-794
> URL: https://issues.apache.org/jira/browse/STORM-794
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.9.3
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> I met an issue from Trident Topology in production env.
> Normally, when we kill a topology via UI, Nimbus changes Topology status to
> "killed", and when Spout determines new status, it becomes deactivated so
> bolts can handle remain tuples within wait-time.
> AFAIK that's how Storm guarantees graceful shutdown.
> But, Trident Topology seems not handle "deactivate" while we try shutdown
> topology gracefully.
> MasterBatchCoordinator never stops making next transaction, so Trident Spout
> never stops emitting, bolts (function) always take care of tuples.
> Topology setting
> - 1 worker, 1 acker
> - max spout pending: 1
> - TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS : 5
> -- It may be weird but MasterBatchCoordinator's default value is 1
> * Nimbus log
> {code}
> 2015-04-20 09:59:07.954 INFO [pool-5-thread-41][nimbus] Delaying event
> :remove for 120 secs for BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 09:59:07.955 INFO [pool-5-thread-41][nimbus] Updated
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015 with status {:type
> :killed, :kill-time-secs 120}
> ...
> 2015-04-20 10:01:07.956 INFO [timer][nimbus] Killing topology:
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 10:01:14.448 INFO [timer][nimbus] Cleaning up
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> {code}
> * Supervisor log
> {code}
> 2015-04-20 10:01:07.960 INFO [Thread-1][supervisor] Removing code for storm
> id BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down and
> clearing state for id 9719259e-528c-4336-abf9-592c1bb9a00b. Current
> supervisor time: 1429491667. State: :disallowed, Heartbeat:
> #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1429491667,
> :storm-id "BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015", :executors
> #{[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [11 11] [12 12] [13
> 13] [14 14] [-1 -1] [1 1]}, :port 6706}
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> 2015-04-20 10:01:08.974 INFO [Thread-2][supervisor] Shut down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> {code}
> * Worker log
> {code}
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down worker
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 5bc084a2-b668-4610-86f6-9b93304d40a8 6706
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down receive thread
> 2015-04-20 10:01:07.988 WARN [Thread-33][ExponentialBackoffRetry] maxRetries
> too large (300). Pinning to 29
> 2015-04-20 10:01:07.988 INFO
> [Thread-33][StormBoundedExponentialBackoffRetry] The baseSleepTimeMs [100]
> the maxSleepTimeMs [1000] the maxRetries [300]
> 2015-04-20 10:01:07.988 INFO [Thread-33][Client] New Netty Client, connect
> to localhost, 6706, config: , buffer_size: 5242880
> 2015-04-20 10:01:07.991 INFO [client-schedule-service-1][Client] Reconnect
> started for Netty-Client-localhost/127.0.0.1:6706... [0]
> 2015-04-20 10:01:07.996 INFO [Thread-33][loader] Shutting down
> receiving-thread: [BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015, 6706]
> ...
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Closing Netty Client
> Netty-Client-localhost/127.0.0.1:6706
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Waiting for pending batchs
> to be sent with Netty-Client-localhost/127.0.0.1:6706..., timeout: 60ms,
> pendings: 1
> {code}
> I found activating log, but cannot find deactivating log.
> {code}
> 2015-04-20 09:50:24.556 INFO [Thread-30-$mastercoord-bg0][executor]
> Activating spout $mastercoord-bg0:(1)
> {code}
> Please note that it doesn't work when I just push button to "deactivate"
> topology via UI.
> We're changing our Topology to normal Spout-Bolt, but personally I'd like to
> see it resolved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Attachment: HDFSSpoutforStorm.pdf Uploading requirements and design notes document > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik > Attachments: HDFSSpoutforStorm.pdf > > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1155: Supervisor recurring health checks
Github user longdafeng commented on the pull request: https://github.com/apache/storm/pull/849#issuecomment-155701669 +1 sorry for late response --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1155) Supervisor recurring health checks
[ https://issues.apache.org/jira/browse/STORM-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000109#comment-15000109 ] ASF GitHub Bot commented on STORM-1155: --- Github user longdafeng commented on the pull request: https://github.com/apache/storm/pull/849#issuecomment-155701669 +1 sorry for late response > Supervisor recurring health checks > -- > > Key: STORM-1155 > URL: https://issues.apache.org/jira/browse/STORM-1155 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Thomas Graves >Assignee: Thomas Graves > > Add the ability for the supervisor to call out to health check scripts to > allow some validation of the health of the node the supervisor is running on. > It could regularly run scripts in a directory provided by the cluster admin. > If any scripts fail, it should kill the workers and stop itself. > This could work very much like the Hadoop scripts and if ERROR is returned on > stdout it means the node has some issue and we should shut down. > If a non-zero exit code is returned it indicates that the scripts failed to > execute properly so you don't want to mark the node as unhealthy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name alignment, the "jstorm-import" branch will
largely be read-only throughout the process.
During migration, it's probably easiest to operate with two local clones of the
Apache Storm repo: one for working (i.e. checked out to working branch) and one
for reference/copying (i.e. checked out to "jstorm-import").
Feature-freeze probably only needs to be enforced against core functionality.
Components under "external" can likely be exempt, but we should figure out a
process for accepting and releasing new features during the migration.
Performance testing should be continuous throughout the process. Since we don't
really have ASF infrastructure for performance testing, we will need a
volunteer(s) to host and run the performance tests. Performance test results
can be posted to the wiki [2]. It would probably be a good idea to establish a
baseline with the 0.10.0 release.
I’ve attached an analysis document Sean Zhong put together a while back to the
JStorm merge JIRA [4]. The analysis was against the 0.9.3 release but is still
relevant and has a lot of good information.
[1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
[2] https://cwiki.apache.org/confluence/display/STORM/Storm+Home
[3] http://semver.org
[4] https://issues.apache.org/jira/browse/STORM-717
-Taylor
答复: [DISCUSS] Initial 0.11.0 Release
Totally agree， it's great to accelerate the release speed. it is better release one official version within 3 months. The first month is for developing new features , .If some features hasn't been finished in this month, it will be put in the next release ticket. In the last two month, we just do test. Storm may face the greatest challenge in a big cluster, so I am more concerned about ZK Optimization as jstorm did. At last, it's great for the next release to has a test report about the stability and performance , due to lots of new features in the latest versions. Regards John Fang -邮件原件- 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] 发送时间: 2015年11月11日 6:16 收件人: dev@storm.apache.org 主题: [DISCUSS] Initial 0.11.0 Release I’d like to start discussing our next release (0.11.0). First off, I’d like to create a list of issues/features that are in-progress and not yet merged to master, so we can start tracking them for inclusion in the release. If there are specific JIRAs you would like included, please reply, and I will add them to the wiki page for the 0.11.0 release [1]. Second, I’d like to propose modifying the release cycle a bit. I’d like to continue the beta release cycle we started with 0.10.0, but this time I’d like to consider adding some sort of temporal constraint so we release new betas more often — something like a new beta release every 3 weeks or so until we feel confident in dropping the beta tag. IMHO, there was too long a gap between 0.10.0-beta1 and 0.10.0 and we should have had more interim releases during that time. Any thoughts? -Taylor [1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
[jira] [Commented] (STORM-1190) System load spikes in recent snapshot
[ https://issues.apache.org/jira/browse/STORM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000410#comment-15000410 ] ASF GitHub Bot commented on STORM-1190: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155793697 Looks like we have a test that keeps getting stuck. I am going to do some debugging to see if I can understand what is happening. > System load spikes in recent snapshot > - > > Key: STORM-1190 > URL: https://issues.apache.org/jira/browse/STORM-1190 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 > Environment: 10x (CoreOS stable (766.4.0) / k8s 1.0.1 / docker > running on Azure VMs) >Reporter: Michael Schonfeld >Priority: Critical > Attachments: Screenshot 2015-11-08 22.17.57.png, Screenshot > 2015-11-08 22.18.06.png > > > We've been running Storm's snapshots on our production cluster for a little > while now (that back pressure support really helped us), and we've noticed a > sudden spike in system load when going from > commit@ba1250993d10ffc523c9f5464371fbeb406d216f to the current latest > commit@c12e28c829fcfabc0a3a775fb9714968b7e3e349. Both versions were running > the exact same topologies, and there was no significant change in workload. > Not exactly sure how to even begin to debug this, so we ended up just rolling > back. Thoughts? > Stats screenshots attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1190: System Load too high after recent ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155793697 Looks like we have a test that keeps getting stuck. I am going to do some debugging to see if I can understand what is happening. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: 答复: [DISCUSS] Initial 0.11.0 Release
I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the list. Some of them are more important then others but they are all things I would like to see in a 0.11.0 release. On a side note I don't think the beta releases have been helpful. I would much rather just have a 0.11.0 go to 0.11.1 ... instead of 0.11.0-beta1, 0.11.0-beta2. To me the beta label is not that helpful, but it is not that big of a deal for me. I also would like to see the 0.11 release tied to the plan for the JStorm merger. If we don't tie them together there can be code that does not make it into 0.11, but could make it into a 0.12 that will immediately be caught up in the merger, that could take a long time to complete. I mostly want us to be very transparent about what is likely to happen after 0.11 is released. So if someone has a feature that is close to getting something in to 0.11 that they speak up here instead of just deciding to wait for a 0.12 release. - Bobby On Wednesday, November 11, 2015 6:32 AM, John Fang wrote: Totally agree， it's great to accelerate the release speed. it is better release one official version within 3 months. The first month is for developing new features , .If some features hasn't been finished in this month, it will be put in the next release ticket. In the last two month, we just do test. Storm may face the greatest challenge in a big cluster, so I am more concerned about ZK Optimization as jstorm did. At last, it's great for the next release to has a test report about the stability and performance , due to lots of new features in the latest versions. Regards John Fang -邮件原件- 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] 发送时间: 2015年11月11日 6:16 收件人: dev@storm.apache.org 主题: [DISCUSS] Initial 0.11.0 Release I’d like to start discussing our next release (0.11.0). First off, I’d like to create a list of issues/features that are in-progress and not yet merged to master, so we can start tracking them for inclusion in the release. If there are specific JIRAs you would like included, please reply, and I will add them to the wiki page for the 0.11.0 release [1]. Second, I’d like to propose modifying the release cycle a bit. I’d like to continue the beta release cycle we started with 0.10.0, but this time I’d like to consider adding some sort of temporal constraint so we release new betas more often — something like a new beta release every 3 weeks or so until we feel confident in dropping the beta tag. IMHO, there was too long a gap between 0.10.0-beta1 and 0.10.0 and we should have had more interim releases during that time. Any thoughts? -Taylor [1] https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set
Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct to JStorm migrated separately).
2. Port JStorm-specific features.
3. Begin releasing preview/beta versions.
4. Code cleanup (across the board) and refactoring using established coding
conventions, and leveraging PMD/Checkstyle reports for reference. (Note: good
oportunity for new contributors.)
5. Release 0.12.0 (or whatever version # we want to use) and lift feature
freeze.
Notes:
We should consider bumping up to version 1.0 sometime soon and then switching
to semantic versioning [3] from then on.
With the exception of package name alignment, the "jstorm-import" branch will
Re: [DISCUSS] Plan for Merging JStorm Code
Hi Basti,
Seems like Phase 3.1 covers Phase 2.2 since Phase 3.1 is actual work for
Phase 2.2.
So in Phase 3.1, if JStorm code is fully compatible, we can just pick it.
If JStorm code is not compatible, we can port Clojure code, or implement it
based on JStorm code.
If you were talking about baseline, please read my opinion on phase 3.
All,
For phase 1, I'm also +1 for releasing 0.11.0 as 1.0.0.
Apache Storm already has various use cases, which feels me no longer beta.
For phase 3, I'm +1 for what Taylor suggests.
Precondition of my opinion is that we agree that our merged product will be
named as 'Apache Storm'.
Based on this decision, we may want to set baseline to Apache Storm, cause
users expect features based backward compatibility to newer releases.
(Brand and community is most important thing for open source product, and
our moves should avoid hurting them. Losing users lose all.)
Many features are included via 0.10.x and 0.11.0 (master branch), which
means Apache Storm and JStorm is diverged. 'Core' is no except.
Not fast but safest is, keeping Apache Storm as it is (or port Clojure code
to Java, as we start to discuss), and applying JStorm's amazing features
via pull requests, like normal contributions. I love this approach since it
is more natural.
Thanks,
Jungtaek Lim (HeartSaVioR)
2015-11-11 20:41 GMT+09:00 刘键(Basti Liu) :
> Hi Taylor,
>
>
>
> Thanks for the merge plan. I have a question about “Phase 2.2”.
>
> Do you mean community plan to create a fresh new “java core” based on
> current “clojure core” firstly, and then migrate the features from JStorm?
>
> If so, it confused me. It is really a huge job which might require a long
> developing time to make it stable, while JStorm is already a stable version.
>
> The release planned to be release after Nov 11th has already run online
> stably several month in Alibaba.
>
> Besides this, there are many valuable internal requirements in Alibaba,
> the fast evolution of JStorm is forseeable in next few months.
>
> If the “java core” is totally fresh new, it might bring many problems for
> the coming merge.
>
> So, from the point of this view, I think it is much better and easier to
> migrate the features of “clojure core” basing on JStorm for the “java core”.
>
> Please correct me, if any misunderstanding.
>
>
>
> Regards
>
> Basti
>
>
>
> 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
> 发送时间: 2015年11月11日 5:32
> 收件人: dev@storm.apache.org
> 主题: [DISCUSS] Plan for Merging JStorm Code
>
>
>
> Based on a number of discussions regarding merging the JStorm code, I’ve
> tried to distill the ideas presented and inserted some of my own. The
> result is below.
>
>
>
> I’ve divided the plan into three phases, though they are not necessarily
> sequential — obviously some tasks can take place in parallel.
>
>
>
> None of this is set in stone, just presented for discussion. Any and all
> comments are welcome.
>
>
>
> ---
>
>
>
> Phase 1 - Plan for 0.11.x Release
>
> 1. Determine feature set for 0.11.x and publish to wiki [1].
>
> 2. Announce feature-freeze for 0.11.x
>
> 3. Create 0.11.x branch from master (Phase 2.4 can begin.)
>
> 4. Release 0.11.0 (or whatever version # we want to use)
>
> 5. Bug fixes and subsequent releases from 0.11.x-branch
>
>
>
>
>
> Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
>
> 1. Determine/document unique features in JStorm (e.g. classpath isolation,
> cgroups, etc.) and create JIRA for migrating the feature.
>
> 2. Create JIRA for migrating each clojure component (or logical group of
> components) to Java. Assumes tests will be ported as well.
>
> 3. Discuss/establish style guide for Java coding conventions. Consider
> using Oracle’s or Google’s Java conventions as a base — they are both
> pretty solid.
>
> 4. align package names (e.g backtype.storm --> org.apache.storm /
> com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
>
>
>
>
>
> Phase 3 - Migrate Clojure --> Java
>
> 1. Port code/tests to Java, leveraging existing JStorm code wherever
> possible (core functionality only, features distinct to JStorm migrated
> separately).
>
> 2. Port JStorm-specific features.
>
> 3. Begin releasing preview/beta versions.
>
> 4. Code cleanup (across the board) and refactoring using established
> coding conventions, and leveraging PMD/Checkstyle reports for reference.
> (Note: good oportunity for new contributors.)
>
> 5. Release 0.12.0 (or whatever version # we want to use) and lift feature
> freeze.
>
>
>
>
>
> Notes:
>
> We should consider bumping up to version 1.0 sometime soon and then
> switching to semantic versioning [3] from then on.
>
>
>
>
>
> With the exception of package name alignment, the "jstorm-import" branch
> will largely be read-only throughout the process.
>
>
>
> During migration, it's probably easiest to operate with two local clones
> of the Apache Storm repo: one for working (i.e. checked out to working
> branch) and one for reference/copying (i.e. checked out to "jstor
Re: [DISCUSS] Plan for Merging JStorm Code
>>> We should consider bumping up to version 1.0 sometime soon and then
>>> switching to semantic versioning [3] from then on.
> 1.0 release based off of the clojure 0.11.x code.
Will 0.11.x -> 1.0 also communicate a move from "beta" to "stable" or anything
like this, or will it be simply a version scheme change?
> It may take a little longer, but we will always have a working codebase
Agree 100% this is important.
--
Derek
- Original Message -
From: Bobby Evans
To: "dev@storm.apache.org"
Cc:
Sent: Wednesday, November 11, 2015 8:51 AM
Subject: Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phase 3 - Migrate Clojure --> Java
1. Port code/tests to Java, leveraging existing JStorm code wherever possible
(core functionality only, features distinct
[GitHub] storm pull request: STORM-1196: Upgrade to thrift 0.9.3
Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/873#issuecomment-155829362 +1. This would simplify a lot of pull requests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1196) Upgrade to thrift 0.9.3
[ https://issues.apache.org/jira/browse/STORM-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000577#comment-15000577 ] ASF GitHub Bot commented on STORM-1196: --- Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/873#issuecomment-155829362 +1. This would simplify a lot of pull requests > Upgrade to thrift 0.9.3 > --- > > Key: STORM-1196 > URL: https://issues.apache.org/jira/browse/STORM-1196 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > > Thrift 0.9.3 has been released. > Upgrade allows us to get rid of the pesky date changes in all the thrift > files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1197: Migrate Travis-CI to container-bas...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/874#issuecomment-155838704 @HeartSaVioR Yes, I remembered that the external parts I split off the storm-core build are dependent on the storm-core jar, so they must be done after storm-core. There isn't any benefit to doing the external, etc. in parallel with each other either, since they are so small that the startup cost overwhelms the parallelization benefit. The caching is still valid, but it only shaves a few seconds off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Plan for Merging JStorm Code
I personally would like to see the move to 1.0 be when we are stable.
- Bobby
On Wednesday, November 11, 2015 9:42 AM, Derek Dagit
wrote:
>>> We should consider bumping up to version 1.0 sometime soon and then
>>> switching to semantic versioning [3] from then on.
> 1.0 release based off of the clojure 0.11.x code.
Will 0.11.x -> 1.0 also communicate a move from "beta" to "stable" or anything
like this, or will it be simply a version scheme change?
> It may take a little longer, but we will always have a working codebase
Agree 100% this is important.
--
Derek
- Original Message -
From: Bobby Evans
To: "dev@storm.apache.org"
Cc:
Sent: Wednesday, November 11, 2015 8:51 AM
Subject: Re: [DISCUSS] Plan for Merging JStorm Code
+1 for doing a 1.0 release based off of the clojure 0.11.x code. Migrating the
APIs to org.apache.storm is a big non-backwards compatible move, and a major
version bump to 2.x seems like a good move there.
+1 for the release plan
I would like the move for user facing APIs to org.apache to be one of the last
things we do. Translating clojure code into java and moving it to org.apache I
am not too concerned about.
Basti,
We have two code bases that have diverged significantly from one another in
terms of functionality. The storm code now or soon will have A Heartbeat
Server, Nimbus HA (Different Implementation), Resource Aware Scheduling, a
distributed cache like API, log searching, security, massive performance
improvements, shaded almost all of our dependencies, a REST API for
programtically accessing everything on the UI, and I am sure I am missing a few
other things. JStorm also has many changes including cgroup isolation,
restructured zookeeper layout, classpath isolation, and more too.
No matter what we do it will be a large effort to port changes from one code
base to another, and from clojure to java. I proposed this initially because
it can be broken up into incremental changes. It may take a little longer, but
we will always have a working codebase that is testable and compatible with the
current storm release, at least until we move the user facing APIs to be under
org.apache. This lets the community continue to build and test the master
branch and report problems that they find, which is incredibly valuable. I
personally don't think it will be much easier, especially if we are intent on
always maintaining compatibility with storm. - Bobby
On Wednesday, November 11, 2015 5:42 AM, 刘键(Basti Liu)
wrote:
Hi Taylor,
Thanks for the merge plan. I have a question about “Phase 2.2”.
Do you mean community plan to create a fresh new “java core” based on current
“clojure core” firstly, and then migrate the features from JStorm?
If so, it confused me. It is really a huge job which might require a long
developing time to make it stable, while JStorm is already a stable version.
The release planned to be release after Nov 11th has already run online stably
several month in Alibaba.
Besides this, there are many valuable internal requirements in Alibaba, the
fast evolution of JStorm is forseeable in next few months.
If the “java core” is totally fresh new, it might bring many problems for the
coming merge.
So, from the point of this view, I think it is much better and easier to
migrate the features of “clojure core” basing on JStorm for the “java core”.
Please correct me, if any misunderstanding.
Regards
Basti
发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com]
发送时间: 2015年11月11日 5:32
收件人: dev@storm.apache.org
主题: [DISCUSS] Plan for Merging JStorm Code
Based on a number of discussions regarding merging the JStorm code, I’ve tried
to distill the ideas presented and inserted some of my own. The result is below.
I’ve divided the plan into three phases, though they are not necessarily
sequential — obviously some tasks can take place in parallel.
None of this is set in stone, just presented for discussion. Any and all
comments are welcome.
---
Phase 1 - Plan for 0.11.x Release
1. Determine feature set for 0.11.x and publish to wiki [1].
2. Announce feature-freeze for 0.11.x
3. Create 0.11.x branch from master (Phase 2.4 can begin.)
4. Release 0.11.0 (or whatever version # we want to use)
5. Bug fixes and subsequent releases from 0.11.x-branch
Phase 2 - Prepare for Merge ("master" and "jstorm-import" branches)
1. Determine/document unique features in JStorm (e.g. classpath isolation,
cgroups, etc.) and create JIRA for migrating the feature.
2. Create JIRA for migrating each clojure component (or logical group of
components) to Java. Assumes tests will be ported as well.
3. Discuss/establish style guide for Java coding conventions. Consider using
Oracle’s or Google’s Java conventions as a base — they are both pretty solid.
4. align package names (e.g backtype.storm --> org.apache.storm /
com.alibaba.jstorm --> org.apache.storm) (Phase 3 can begin)
Phas
[jira] [Commented] (STORM-1197) Migrate Travis-CI to container-based builds.
[ https://issues.apache.org/jira/browse/STORM-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000632#comment-15000632 ] ASF GitHub Bot commented on STORM-1197: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/874#issuecomment-155838704 @HeartSaVioR Yes, I remembered that the external parts I split off the storm-core build are dependent on the storm-core jar, so they must be done after storm-core. There isn't any benefit to doing the external, etc. in parallel with each other either, since they are so small that the startup cost overwhelms the parallelization benefit. The caching is still valid, but it only shaves a few seconds off. > Migrate Travis-CI to container-based builds. > > > Key: STORM-1197 > URL: https://issues.apache.org/jira/browse/STORM-1197 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kyle Nusbaum > > http://docs.travis-ci.com/user/migrating-from-legacy/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44554614 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- Doesn't seem like this should be a percentage --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000642#comment-15000642 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44554614 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- Doesn't seem like this should be a percentage > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44554722 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) + + + + + Used CPU (%) --- End diff -- Same here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000648#comment-15000648 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44554722 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) + + + + + Used CPU (%) --- End diff -- Same here > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user zhuoliu commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44555808 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- Thanks for the comment. This is a percentage to a vCore. So every 1 equals to 1% of a core. E.g., 400 here means 4 CPU vCore. I would appreciate if we can have a more descriptive unit sign here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000669#comment-15000669 ] ASF GitHub Bot commented on STORM-1198: --- Github user zhuoliu commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44555808 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- Thanks for the comment. This is a percentage to a vCore. So every 1 equals to 1% of a core. E.g., 400 here means 4 CPU vCore. I would appreciate if we can have a more descriptive unit sign here. > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1191: bump timeout by 50% due to intermi...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/869#issuecomment-155847070 Most all of the spurious test failures I've seen have been *real* failures and not timeouts. I'm okay with this going in, even though I don't like potentially making the build time longer and I'm not optimistic about this fixing any problems. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1191) Random test failures on backtype.storm.transactional-test
[ https://issues.apache.org/jira/browse/STORM-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000682#comment-15000682 ] ASF GitHub Bot commented on STORM-1191: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/869#issuecomment-155847070 Most all of the spurious test failures I've seen have been *real* failures and not timeouts. I'm okay with this going in, even though I don't like potentially making the build time longer and I'm not optimistic about this fixing any problems. > Random test failures on backtype.storm.transactional-test > - > > Key: STORM-1191 > URL: https://issues.apache.org/jira/browse/STORM-1191 > Project: Apache Storm > Issue Type: Bug >Reporter: Chuck Burgess >Assignee: Chuck Burgess >Priority: Minor > > Travis builds show occasional backtype.storm.transactional-test failures due > to the 100s timeout (https://travis-ci.org/apache/storm/jobs/90105000). > Assuming this is not a performance test, I'll submit PR to bump timeout by > 50% to cut back on these failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1181) Compile SQLs in Tridient topology and execute them in LocalCluster
[ https://issues.apache.org/jira/browse/STORM-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated STORM-1181: -- Summary: Compile SQLs in Tridient topology and execute them in LocalCluster (was: Execute StormSQL in Trident topology) > Compile SQLs in Tridient topology and execute them in LocalCluster > -- > > Key: STORM-1181 > URL: https://issues.apache.org/jira/browse/STORM-1181 > Project: Apache Storm > Issue Type: New Feature > Components: storm-sql >Reporter: Haohui Mai >Assignee: Haohui Mai > > This jira tracks the effort of compiling SQL to Trident topology to enable > users to run StormSQL at production scale. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1181. Compile SQLs into Tridient topolog...
GitHub user haohui opened a pull request: https://github.com/apache/storm/pull/876 STORM-1181. Compile SQLs into Tridient topology and execute them in LocalCluster This PR expands the capability of StormSQL to compile SQLs into Trident topology and execute them in LocalCluster. This PR has not implemented the support of running the compiled Trident topology on worker nodes yet, as it requires the generated classes to be presented in the class path of the worker's JVM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/haohui/storm STORM-1181 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/876.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #876 commit 8948a3344feb470d8a6e04312d8a847a6c808be5 Author: Haohui Mai Date: 2015-11-04T23:51:11Z STORM-1173. Support string operations in StormSQL. commit 5925da47293d5c273d983b1235da924742572e2a Author: Haohui Mai Date: 2015-11-11T18:03:07Z Allow deserializing Java class with custom classloaders in tests. commit 6cba5c3e819bbd2a390a0a51d2f7d261d507c267 Author: Haohui Mai Date: 2015-11-06T23:50:31Z Refactor to support compiling StormSQL to Trident topology. commit 5dc1f386bde57fe65c60e475a4cd9593d3c0bf1f Author: Haohui Mai Date: 2015-11-11T18:07:54Z STORM-1181. Compile SQLs into Tridient topology and execute them in LocalCluster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1181) Compile SQLs in Tridient topology and execute them in LocalCluster
[ https://issues.apache.org/jira/browse/STORM-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000822#comment-15000822 ] ASF GitHub Bot commented on STORM-1181: --- GitHub user haohui opened a pull request: https://github.com/apache/storm/pull/876 STORM-1181. Compile SQLs into Tridient topology and execute them in LocalCluster This PR expands the capability of StormSQL to compile SQLs into Trident topology and execute them in LocalCluster. This PR has not implemented the support of running the compiled Trident topology on worker nodes yet, as it requires the generated classes to be presented in the class path of the worker's JVM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/haohui/storm STORM-1181 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/876.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #876 commit 8948a3344feb470d8a6e04312d8a847a6c808be5 Author: Haohui Mai Date: 2015-11-04T23:51:11Z STORM-1173. Support string operations in StormSQL. commit 5925da47293d5c273d983b1235da924742572e2a Author: Haohui Mai Date: 2015-11-11T18:03:07Z Allow deserializing Java class with custom classloaders in tests. commit 6cba5c3e819bbd2a390a0a51d2f7d261d507c267 Author: Haohui Mai Date: 2015-11-06T23:50:31Z Refactor to support compiling StormSQL to Trident topology. commit 5dc1f386bde57fe65c60e475a4cd9593d3c0bf1f Author: Haohui Mai Date: 2015-11-11T18:07:54Z STORM-1181. Compile SQLs into Tridient topology and execute them in LocalCluster. > Compile SQLs in Tridient topology and execute them in LocalCluster > -- > > Key: STORM-1181 > URL: https://issues.apache.org/jira/browse/STORM-1181 > Project: Apache Storm > Issue Type: New Feature > Components: storm-sql >Reporter: Haohui Mai >Assignee: Haohui Mai > > This jira tracks the effort of compiling SQL to Trident topology to enable > users to run StormSQL at production scale. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-1181) Compile SQLs into Tridient topology and execute them in LocalCluster
[ https://issues.apache.org/jira/browse/STORM-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated STORM-1181: -- Summary: Compile SQLs into Tridient topology and execute them in LocalCluster (was: Compile SQLs in Tridient topology and execute them in LocalCluster) > Compile SQLs into Tridient topology and execute them in LocalCluster > > > Key: STORM-1181 > URL: https://issues.apache.org/jira/browse/STORM-1181 > Project: Apache Storm > Issue Type: New Feature > Components: storm-sql >Reporter: Haohui Mai >Assignee: Haohui Mai > > This jira tracks the effort of compiling SQL to Trident topology to enable > users to run StormSQL at production scale. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1190: System Load too high after recent ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155867024 Turns out that invokeAll waits for all of the processes to finish, and they could block so it was not what I wanted/expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1190) System load spikes in recent snapshot
[ https://issues.apache.org/jira/browse/STORM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000844#comment-15000844 ] ASF GitHub Bot commented on STORM-1190: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155867024 Turns out that invokeAll waits for all of the processes to finish, and they could block so it was not what I wanted/expected. > System load spikes in recent snapshot > - > > Key: STORM-1190 > URL: https://issues.apache.org/jira/browse/STORM-1190 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 > Environment: 10x (CoreOS stable (766.4.0) / k8s 1.0.1 / docker > running on Azure VMs) >Reporter: Michael Schonfeld >Priority: Critical > Attachments: Screenshot 2015-11-08 22.17.57.png, Screenshot > 2015-11-08 22.18.06.png > > > We've been running Storm's snapshots on our production cluster for a little > while now (that back pressure support really helped us), and we've noticed a > sudden spike in system load when going from > commit@ba1250993d10ffc523c9f5464371fbeb406d216f to the current latest > commit@c12e28c829fcfabc0a3a775fb9714968b7e3e349. Both versions were running > the exact same topologies, and there was no significant change in workload. > Not exactly sure how to even begin to debug this, so we ended up just rolling > back. Thoughts? > Stats screenshots attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1190: System Load too high after recent ...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155888418 This time the travis failure is unrelated. It got past the disruptor tests and failed to bind to a port for a DRPC test. The code should be good to go now. Reviews are welcome. On a side note this dropped the CPU utilization enough that I was now able to do 30,000 sentences/second with ThroughputVsLatency, where as before I was only able to do 27,000. So it improves the performance as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1190) System load spikes in recent snapshot
[ https://issues.apache.org/jira/browse/STORM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000951#comment-15000951 ] ASF GitHub Bot commented on STORM-1190: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/870#issuecomment-155888418 This time the travis failure is unrelated. It got past the disruptor tests and failed to bind to a port for a DRPC test. The code should be good to go now. Reviews are welcome. On a side note this dropped the CPU utilization enough that I was now able to do 30,000 sentences/second with ThroughputVsLatency, where as before I was only able to do 27,000. So it improves the performance as well. > System load spikes in recent snapshot > - > > Key: STORM-1190 > URL: https://issues.apache.org/jira/browse/STORM-1190 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.11.0 > Environment: 10x (CoreOS stable (766.4.0) / k8s 1.0.1 / docker > running on Azure VMs) >Reporter: Michael Schonfeld >Priority: Critical > Attachments: Screenshot 2015-11-08 22.17.57.png, Screenshot > 2015-11-08 22.18.06.png > > > We've been running Storm's snapshots on our production cluster for a little > while now (that back pressure support really helped us), and we've noticed a > sudden spike in system load when going from > commit@ba1250993d10ffc523c9f5464371fbeb406d216f to the current latest > commit@c12e28c829fcfabc0a3a775fb9714968b7e3e349. Both versions were running > the exact same topologies, and there was no significant change in workload. > Not exactly sure how to even begin to debug this, so we ended up just rolling > back. Thoughts? > Stats screenshots attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44579118 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- ok this make sense --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000993#comment-15000993 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on a diff in the pull request: https://github.com/apache/storm/pull/875#discussion_r44579118 --- Diff: storm-core/src/ui/public/templates/index-page-template.html --- @@ -215,6 +215,26 @@ + + Total Mem (MB) + + + + + Used Mem (MB) + + + + + Total CPU (%) --- End diff -- ok this make sense > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155897894 LGTM +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001003#comment-15001003 ] ASF GitHub Bot commented on STORM-1198: --- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155897894 LGTM +1 > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Plan for Merging JStorm Code
> On Nov 11, 2015, at 2:04 AM, Cody Innowhere wrote: > >> During migration, it's probably easiest to operate with two local clones > of the Apache Storm repo: one for working (i.e. checked out to working > branch) > and one for reference/copying (i.e. checked out to > "jstorm-import"). > Do you mean two storm branches(both clojure core) or one storm branch with > one JStorm-imported branch? That paragraph was mainly me thinking out loud, and probably could have been left out. I meant that as a developer working on merging two divergent branches branches, it might be easiest to have two local copies of the repo to do comparisons, diffs, etc., rather than working in one repo and having to constantly switch between branches. That’s just the way I would likely do it. Ultimately it’s up to each individual developer’s personal preference. -Taylor signature.asc Description: Message signed with OpenPGP using GPGMail
[jira] [Updated] (STORM-1199) Create HDFS Spout
[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Issue Type: New Feature (was: Bug) > Create HDFS Spout > - > > Key: STORM-1199 > URL: https://issues.apache.org/jira/browse/STORM-1199 > Project: Apache Storm > Issue Type: New Feature >Reporter: Roshan Naik > > Create an HDFS spout so that Storm can suck in data from files in a HDFS > directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1199) Create HDFS Spout
Roshan Naik created STORM-1199: -- Summary: Create HDFS Spout Key: STORM-1199 URL: https://issues.apache.org/jira/browse/STORM-1199 Project: Apache Storm Issue Type: Bug Reporter: Roshan Naik Create an HDFS spout so that Storm can suck in data from files in a HDFS directory -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1016: Generate trident bolt ids with sor...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-155911293 @victor-wong any update on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1016) Generate trident bolt ids with sorted group names
[ https://issues.apache.org/jira/browse/STORM-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001092#comment-15001092 ] ASF GitHub Bot commented on STORM-1016: --- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/706#issuecomment-155911293 @victor-wong any update on this? > Generate trident bolt ids with sorted group names > - > > Key: STORM-1016 > URL: https://issues.apache.org/jira/browse/STORM-1016 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Jiasheng Wang > > In genBoltId() method of TridentTopology, groups are stored in HashSet. > So everytime I submit my trident topology, I get different id for my > component. > For example, this time my bolt ids are (b-0-abc, b-1-def), next time they are > (b-1-abc, b-0-def). > It makes harder for users to track their metrics, and the bolt names in UI > are changing too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Initial 0.11.0 Release
> On Nov 11, 2015, at 9:28 AM, Bobby Evans wrote: > > I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, > STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the list. > > Some of them are more important then others but they are all things I would > like to see in a 0.11.0 release. Cool. Thanks for listing them out. I will add them so they get tracked. > On a side note I don't think the beta releases have been helpful. I would > much rather just have a 0.11.0 go to 0.11.1 ... instead of 0.11.0-beta1, > 0.11.0-beta2. To me the beta label is not that helpful, but it is not that > big of a deal for me. In my mind, having releases tagged as “beta” were a way for us to say to users “here’s a preview release that will allow you to kick the tires on upcoming features, but be aware that there might be bugs. Let us know if you find any so we can fix them.” I think the intent was sound, but what I didn’t really see was user feedback on the beta releases, which is what I hoped would happen. So I have no problem with dropping the use of “beta” flags. Another approach I’ve seen other Apache projects use is to the numbering scheme you describe, and just have different labels/descriptions on the download page — i.e. “Latest stable release” and “Latest development release.” The nice part of that convention is that it does not have any impact on the release process. For example if we’re confident that a particular “development” release is actually quite stable, all we would have to do is update the downloads page, rather than go through the whole release/vote process just to remove the “beta” tag. > I also would like to see the 0.11 release tied to the plan for the JStorm > merger. If we don't tie them together there can be code that does not make > it into 0.11, but could make it into a 0.12 that will immediately be caught > up in the merger, that could take a long time to complete. I mostly want us > to be very transparent about what is likely to happen after 0.11 is released. > So if someone has a feature that is close to getting something in to 0.11 > that they speak up here instead of just deciding to wait for a 0.12 release. I agree that the 0.11 release needs to be tied to the JStorm merger. Once that release goes out, we’ll be going into lockdown mode for the merge effort, which is likely to take a while. During that time it’s highly unlikely that any changes/additions to Storm’s core functionality will be accepted beyond high-priority bug fixes. Adding new features to the “external” components during that time is probably okay, since those components are sufficiently decoupled from Storm’s core functionality. So to reiterate Bobby’s statement: Please speak up now if there are any specific features or changes to Storm’s core functionality that you’d like to see in the next release. Otherwise you will have to wait. > - Bobby -Taylor > > > On Wednesday, November 11, 2015 6:32 AM, John Fang > wrote: > > > Totally agree， it's great to accelerate the release speed. it is better > release one official version within 3 months. The first month is for > developing new features , .If some features hasn't been finished in this > month, it will be put in the next release ticket. In the last two month, we > just do test. > Storm may face the greatest challenge in a big cluster, so I am more > concerned about ZK Optimization as jstorm did. At last, it's great for the > next release to has a test report about the stability and performance , due > to lots of new features in the latest versions. > > Regards > John Fang > > -邮件原件- > 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] > 发送时间: 2015年11月11日 6:16 > 收件人: dev@storm.apache.org > 主题: [DISCUSS] Initial 0.11.0 Release > > I’d like to start discussing our next release (0.11.0). > > First off, I’d like to create a list of issues/features that are in-progress > and not yet merged to master, so we can start tracking them for inclusion in > the release. If there are specific JIRAs you would like included, please > reply, and I will add them to the wiki page for the 0.11.0 release [1]. > > Second, I’d like to propose modifying the release cycle a bit. I’d like to > continue the beta release cycle we started with 0.10.0, but this time I’d > like to consider adding some sort of temporal constraint so we release new > betas more often — something like a new beta release every 3 weeks or so > until we feel confident in dropping the beta tag. IMHO, there was too long a > gap between 0.10.0-beta1 and 0.10.0 and we should have had more interim > releases during that time. > > Any thoughts? > > -Taylor > > [1] > https://cwiki.apache.org/confluence/display/STORM/Release+0.11.0+Feature+Set > > signature.asc Description: Message signed with OpenPGP using GPGMail
[jira] [Resolved] (STORM-1155) Supervisor recurring health checks
[ https://issues.apache.org/jira/browse/STORM-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved STORM-1155. Resolution: Fixed Fix Version/s: 0.11.0 Thanks [~tgraves], I merged this into master. > Supervisor recurring health checks > -- > > Key: STORM-1155 > URL: https://issues.apache.org/jira/browse/STORM-1155 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.11.0 > > > Add the ability for the supervisor to call out to health check scripts to > allow some validation of the health of the node the supervisor is running on. > It could regularly run scripts in a directory provided by the cluster admin. > If any scripts fail, it should kill the workers and stop itself. > This could work very much like the Hadoop scripts and if ERROR is returned on > stdout it means the node has some issue and we should shut down. > If a non-zero exit code is returned it indicates that the scripts failed to > execute properly so you don't want to mark the node as unhealthy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1155) Supervisor recurring health checks
[ https://issues.apache.org/jira/browse/STORM-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001125#comment-15001125 ] ASF GitHub Bot commented on STORM-1155: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/849 > Supervisor recurring health checks > -- > > Key: STORM-1155 > URL: https://issues.apache.org/jira/browse/STORM-1155 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.11.0 > > > Add the ability for the supervisor to call out to health check scripts to > allow some validation of the health of the node the supervisor is running on. > It could regularly run scripts in a directory provided by the cluster admin. > If any scripts fail, it should kill the workers and stop itself. > This could work very much like the Hadoop scripts and if ERROR is returned on > stdout it means the node has some issue and we should shut down. > If a non-zero exit code is returned it indicates that the scripts failed to > execute properly so you don't want to mark the node as unhealthy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1155: Supervisor recurring health checks
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/849 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Add codota search for apache/storm to Readme
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/781 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-831] Adding jira and central logging li...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-155916435 @kishorvpatil could you rebase? I really would like to get this into 0.11.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-831) Add Jira and Central Logging URL to UI
[ https://issues.apache.org/jira/browse/STORM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001144#comment-15001144 ] ASF GitHub Bot commented on STORM-831: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-155916435 @kishorvpatil could you rebase? I really would like to get this into 0.11.0. > Add Jira and Central Logging URL to UI > -- > > Key: STORM-831 > URL: https://issues.apache.org/jira/browse/STORM-831 > Project: Apache Storm > Issue Type: Documentation > Components: documentation >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Trivial > > As a user, I would like to see a link to take me to JIRA for reporting bug. > Also, optionally if link to splunk/logstash/kibana from UI would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-710] bin/storm command list out all the...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/553#issuecomment-155916895 @oleg03 I don't really like the idea of adding in a -v option that restricts all other commands, including storm jar from supporting it. I would much rather have a long parameter that is unlikely to collide with something someone else is already using, or us a config or environment variable to trigger it instead. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-710) bin/storm command list out all the classes in the output for a command
[ https://issues.apache.org/jira/browse/STORM-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001149#comment-15001149 ] ASF GitHub Bot commented on STORM-710: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/553#issuecomment-155916895 @oleg03 I don't really like the idea of adding in a -v option that restricts all other commands, including storm jar from supporting it. I would much rather have a long parameter that is unlikely to collide with something someone else is already using, or us a config or environment variable to trigger it instead. > bin/storm command list out all the classes in the output for a command > -- > > Key: STORM-710 > URL: https://issues.apache.org/jira/browse/STORM-710 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Sriharsha Chintalapani >Assignee: Oleg Ostashchuk >Priority: Minor > Labels: newbie > > when running bin/storm list command or any other command it outputs > Running: > /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/bin/java > -client -Dstorm.options= > -Dstorm.home=/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT > -Dstorm.log.dir=/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/logs > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= > -cp > /Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/asm-4.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/carbonite-1.4.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/chill-java-0.3.5.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/clj-stacktrace-0.2.7.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/clj-time-0.8.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/clojure-1.6.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/clout-1.0.1.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-codec-1.6.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-exec-1.1.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-fileupload-1.2.1.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-io-2.4.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-lang-2.5.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/commons-logging-1.1.3.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/compojure-1.1.3.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/core.incubator-0.1.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/crypto-equality-1.0.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/crypto-random-1.2.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/disruptor-2.10.1.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/hadoop-auth-2.4.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/hiccup-0.3.6.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/httpclient-4.3.3.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/httpcore-4.3.2.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/java.classpath-0.2.2.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/javax.servlet-2.5.0.v201103041518.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-client-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-continuation-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-http-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-io-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-security-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-server-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-servlet-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-servlets-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jetty-util-7.6.13.v20130916.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/jgrapht-core-0.9.0.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/joda-time-2.3.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/json-simple-1.1.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/kryo-2.21.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/log4j-over-slf4j-1.6.6.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/logback-classic-1.0.13.jar:/Users/schintalapani/build/apache-storm-0.10.0-SNAPSHOT/lib/logback-c
[GitHub] storm pull request: STORM-794 Modify Spout async loop to treat act...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/542#issuecomment-155917369 @HeartSaVioR is this something you still want? I like the idea of the change, and the code looks good, but we would need to upmerge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-436] Clean up ns declarations
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/534#issuecomment-155917634 @danielcompton any update on this? It would be a nice change to get in before 0.11, if you have time to do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155917632 what would happens to the UI information if it is not using RAS? Just curious if it is turned off then do we want to show this information? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-794) Trident Topology with some situation seems not handle deactivate during graceful shutdown
[
https://issues.apache.org/jira/browse/STORM-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001154#comment-15001154
]
ASF GitHub Bot commented on STORM-794:
--
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/542#issuecomment-155917369
@HeartSaVioR is this something you still want? I like the idea of the
change, and the code looks good, but we would need to upmerge.
> Trident Topology with some situation seems not handle deactivate during
> graceful shutdown
> -
>
> Key: STORM-794
> URL: https://issues.apache.org/jira/browse/STORM-794
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.9.3
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> I met an issue from Trident Topology in production env.
> Normally, when we kill a topology via UI, Nimbus changes Topology status to
> "killed", and when Spout determines new status, it becomes deactivated so
> bolts can handle remain tuples within wait-time.
> AFAIK that's how Storm guarantees graceful shutdown.
> But, Trident Topology seems not handle "deactivate" while we try shutdown
> topology gracefully.
> MasterBatchCoordinator never stops making next transaction, so Trident Spout
> never stops emitting, bolts (function) always take care of tuples.
> Topology setting
> - 1 worker, 1 acker
> - max spout pending: 1
> - TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS : 5
> -- It may be weird but MasterBatchCoordinator's default value is 1
> * Nimbus log
> {code}
> 2015-04-20 09:59:07.954 INFO [pool-5-thread-41][nimbus] Delaying event
> :remove for 120 secs for BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 09:59:07.955 INFO [pool-5-thread-41][nimbus] Updated
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015 with status {:type
> :killed, :kill-time-secs 120}
> ...
> 2015-04-20 10:01:07.956 INFO [timer][nimbus] Killing topology:
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 10:01:14.448 INFO [timer][nimbus] Cleaning up
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> {code}
> * Supervisor log
> {code}
> 2015-04-20 10:01:07.960 INFO [Thread-1][supervisor] Removing code for storm
> id BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down and
> clearing state for id 9719259e-528c-4336-abf9-592c1bb9a00b. Current
> supervisor time: 1429491667. State: :disallowed, Heartbeat:
> #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1429491667,
> :storm-id "BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015", :executors
> #{[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [11 11] [12 12] [13
> 13] [14 14] [-1 -1] [1 1]}, :port 6706}
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> 2015-04-20 10:01:08.974 INFO [Thread-2][supervisor] Shut down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> {code}
> * Worker log
> {code}
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down worker
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 5bc084a2-b668-4610-86f6-9b93304d40a8 6706
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down receive thread
> 2015-04-20 10:01:07.988 WARN [Thread-33][ExponentialBackoffRetry] maxRetries
> too large (300). Pinning to 29
> 2015-04-20 10:01:07.988 INFO
> [Thread-33][StormBoundedExponentialBackoffRetry] The baseSleepTimeMs [100]
> the maxSleepTimeMs [1000] the maxRetries [300]
> 2015-04-20 10:01:07.988 INFO [Thread-33][Client] New Netty Client, connect
> to localhost, 6706, config: , buffer_size: 5242880
> 2015-04-20 10:01:07.991 INFO [client-schedule-service-1][Client] Reconnect
> started for Netty-Client-localhost/127.0.0.1:6706... [0]
> 2015-04-20 10:01:07.996 INFO [Thread-33][loader] Shutting down
> receiving-thread: [BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015, 6706]
> ...
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Closing Netty Client
> Netty-Client-localhost/127.0.0.1:6706
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Waiting for pending batchs
> to be sent with Netty-Client-localhost/127.0.0.1:6706..., timeout: 60ms,
> pendings: 1
> {code}
> I found activating log, but cannot find deactivating log.
> {code}
> 2015-04-20 09:50:24.556 INFO [Thread-30-$mastercoord-bg0][executor]
> Activating spout $mastercoord-bg0:(1)
> {code}
> Please note that it doesn't work when I just push button to "deactivate"
> topology via UI.
> We're changing our Topology to normal Spout-Bolt, but personally I'd like to
> see it resolved.
--
This messag
[jira] [Commented] (STORM-436) Clean up Clojure namespace declarations
[ https://issues.apache.org/jira/browse/STORM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001159#comment-15001159 ] ASF GitHub Bot commented on STORM-436: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/534#issuecomment-155917634 @danielcompton any update on this? It would be a nice change to get in before 0.11, if you have time to do it. > Clean up Clojure namespace declarations > --- > > Key: STORM-436 > URL: https://issues.apache.org/jira/browse/STORM-436 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.2-incubating >Reporter: Daniel Compton >Priority: Minor > Labels: newbie > > Some of the Clojure namespace declarations in the storm project are messy and > non-idiomatic. > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/ui/core.clj#L18-L38 > is a good example of this. > There are a few things I'd like to improve: > 1. Coalesce multiple use/require/import's into a single use/require/import. > 2. Order imports use, require, import. > 3. Optionally, replacing use with some mix of > * [... :refer :all] > * Referring just the vars that are used > * Qualifying the namespace imports > Is there a reason why the namespaces were done in the way they have been? > What would be the preferred way to do 3? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1198) Web UI to show resource usages and Total Resources on all supervisors
[ https://issues.apache.org/jira/browse/STORM-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001158#comment-15001158 ] ASF GitHub Bot commented on STORM-1198: --- Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/875#issuecomment-155917632 what would happens to the UI information if it is not using RAS? Just curious if it is turned off then do we want to show this information? > Web UI to show resource usages and Total Resources on all supervisors > - > > Key: STORM-1198 > URL: https://issues.apache.org/jira/browse/STORM-1198 > Project: Apache Storm > Issue Type: Story > Components: storm-core >Reporter: Zhuo Liu >Assignee: Zhuo Liu >Priority: Minor > Attachments: supervisor-resources.png > > > As we have resource aware scheduler (STORM-894), we want to be able to > display resource capacity (CPU, memory; and network in future) and scheduled > resource usage on each supervisor node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-756] ShellBolt can delay sending tuples...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/532#issuecomment-155917971 @itaifrenkel any update on this? we now have automatic back-pressure like heron so this would be a nice addition to it too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-756) [multilang] Introduce overflow control mechanism
[ https://issues.apache.org/jira/browse/STORM-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001161#comment-15001161 ] ASF GitHub Bot commented on STORM-756: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/532#issuecomment-155917971 @itaifrenkel any update on this? we now have automatic back-pressure like heron so this would be a nice addition to it too. > [multilang] Introduce overflow control mechanism > > > Key: STORM-756 > URL: https://issues.apache.org/jira/browse/STORM-756 > Project: Apache Storm > Issue Type: Improvement > Components: storm-multilang >Affects Versions: 0.10.0, 0.9.4, 0.11.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > It's from STORM-738, > https://issues.apache.org/jira/browse/STORM-738?focusedCommentId=14394106&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14394106 > A. ShellBolt side control > We can modify ShellBolt to have sent tuple ids list, and stop sending tuples > when list exceeds configured max value. In order to achieve this, subprocess > should notify "tuple id is complete" to ShellBolt. > * It introduces new commands for multi-lang, "proceed" (or better name) > * ShellBolt stores in-progress-of-processing tuples list. > * Its overhead could be big, subprocess should always notify to ShellBolt > when any tuples are processed. > B. subprocess side control > We can modify subprocess to check pending queue after reading tuple. > If it exceeds configured max value, subprocess can request "delay" to > ShellBolt for slowing down. > When ShellBolt receives "delay", BoltWriterRunnable should stop polling > pending queue and continue polling later. > How long ShellBolt wait for resending? Its unit would be "delay time" or > "tuple count". I don't know which is better yet. > * It introduces new commands for multi-lang, "delay" (or better name) > * I don't think it would be introduced soon, but subprocess can request delay > based on own statistics. (ex. pending tuple count * average tuple processed > time for time unit, average pending tuple count for count unit) > ** We can leave when and how much to request "delay" to user. User can make > his/her own algorithm to control flooding. > In my opinion B seems to more natural cause current issue is by subprocess > side so it would be better to let subprocess overcome it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-436] Clean up ns declarations
Github user danielcompton commented on the pull request: https://github.com/apache/storm/pull/534#issuecomment-155923799 I've let things lapse a little bit on this. I'll try to get it up and running against master soon though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-436) Clean up Clojure namespace declarations
[ https://issues.apache.org/jira/browse/STORM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001210#comment-15001210 ] ASF GitHub Bot commented on STORM-436: -- Github user danielcompton commented on the pull request: https://github.com/apache/storm/pull/534#issuecomment-155923799 I've let things lapse a little bit on this. I'll try to get it up and running against master soon though. > Clean up Clojure namespace declarations > --- > > Key: STORM-436 > URL: https://issues.apache.org/jira/browse/STORM-436 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Affects Versions: 0.9.2-incubating >Reporter: Daniel Compton >Priority: Minor > Labels: newbie > > Some of the Clojure namespace declarations in the storm project are messy and > non-idiomatic. > https://github.com/apache/incubator-storm/blob/master/storm-core/src/clj/backtype/storm/ui/core.clj#L18-L38 > is a good example of this. > There are a few things I'd like to improve: > 1. Coalesce multiple use/require/import's into a single use/require/import. > 2. Order imports use, require, import. > 3. Optionally, replacing use with some mix of > * [... :refer :all] > * Referring just the vars that are used > * Qualifying the namespace imports > Is there a reason why the namespaces were done in the way they have been? > What would be the preferred way to do 3? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-794 Modify Spout async loop to treat act...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/542#issuecomment-155928236 @revans2 Yes, as I described to [JIRA issue](https://issues.apache.org/jira/browse/STORM-794?focusedCommentId=14526316&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14526316), MasterBatchCoordinator doesn't play well with async loop, since it handles emitting batch independently. MasterBatchCoordinator could make pending tuple count always same to max spout pending in point of async loop's view, which makes async loop to never trigger deactivate. I'll do upmerge and let you know. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)
Changing subject in order to consolidate discussion of a 1.0 release in one thread (there was some additional discussion in the thread regarding the JStorm code merge). I just want to make sure I’m accurately capturing the sentiment of the community with regard to a 1.0 release. Please correct me if my summary seems off-base or jump in with an opinion. In summary: 1. What we have been calling “0.11.0” will become the Storm 1.0 release. 2. We will NOT be migrating package names for this release (i.e. “backtype.storm” —> “org.apache.storm”). 3. Post 1.0 release we will go into feature freeze for core functionality to facilitate the JStorm merge. 4. During the feature freeze only fixes for high priority bugs in core functionality will be accepted (no new features). 5. During the feature freeze, enhancements to “external” modules can be accepted. 6. We will stop using the “beta” flag in favor of purely numeric version numbers. Stable vs. non-stable (development) releases can be indicated on the download page. Do we all agree? -Taylor > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz wrote: > > >> On Nov 11, 2015, at 9:28 AM, Bobby Evans wrote: >> >> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, >> STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the list. >> >> Some of them are more important then others but they are all things I would >> like to see in a 0.11.0 release. > > > Cool. Thanks for listing them out. I will add them so they get tracked. > > >> On a side note I don't think the beta releases have been helpful. I would >> much rather just have a 0.11.0 go to 0.11.1 ... instead of 0.11.0-beta1, >> 0.11.0-beta2. To me the beta label is not that helpful, but it is not that >> big of a deal for me. > > In my mind, having releases tagged as “beta” were a way for us to say to > users “here’s a preview release that will allow you to kick the tires on > upcoming features, but be aware that there might be bugs. Let us know if you > find any so we can fix them.” > > I think the intent was sound, but what I didn’t really see was user feedback > on the beta releases, which is what I hoped would happen. So I have no > problem with dropping the use of “beta” flags. > > Another approach I’ve seen other Apache projects use is to the numbering > scheme you describe, and just have different labels/descriptions on the > download page — i.e. “Latest stable release” and “Latest development > release.” The nice part of that convention is that it does not have any > impact on the release process. For example if we’re confident that a > particular “development” release is actually quite stable, all we would have > to do is update the downloads page, rather than go through the whole > release/vote process just to remove the “beta” tag. > >> I also would like to see the 0.11 release tied to the plan for the JStorm >> merger. If we don't tie them together there can be code that does not make >> it into 0.11, but could make it into a 0.12 that will immediately be caught >> up in the merger, that could take a long time to complete. I mostly want us >> to be very transparent about what is likely to happen after 0.11 is >> released. So if someone has a feature that is close to getting something in >> to 0.11 that they speak up here instead of just deciding to wait for a 0.12 >> release. > > > I agree that the 0.11 release needs to be tied to the JStorm merger. Once > that release goes out, we’ll be going into lockdown mode for the merge > effort, which is likely to take a while. > > During that time it’s highly unlikely that any changes/additions to Storm’s > core functionality will be accepted beyond high-priority bug fixes. Adding > new features to the “external” components during that time is probably okay, > since those components are sufficiently decoupled from Storm’s core > functionality. > > So to reiterate Bobby’s statement: > > Please speak up now if there are any specific features or changes to Storm’s > core functionality that you’d like to see in the next release. Otherwise you > will have to wait. > >> - Bobby > > -Taylor > >> >> >>On Wednesday, November 11, 2015 6:32 AM, John Fang >> wrote: >> >> >> Totally agree， it's great to accelerate the release speed. it is better >> release one official version within 3 months. The first month is for >> developing new features , .If some features hasn't been finished in this >> month, it will be put in the next release ticket. In the last two month, we >> just do test. >> Storm may face the greatest challenge in a big cluster, so I am more >> concerned about ZK Optimization as jstorm did. At last, it's great for the >> next release to has a test report about the stability and performance , due >> to lots of new features in the latest versions. >> >> Regards >> John Fang >> >> -邮件原件- >> 发件人: P. Taylor Goetz [mailto:ptgo...@gmail.com] >> 发
[jira] [Commented] (STORM-794) Trident Topology with some situation seems not handle deactivate during graceful shutdown
[
https://issues.apache.org/jira/browse/STORM-794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001237#comment-15001237
]
ASF GitHub Bot commented on STORM-794:
--
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/542#issuecomment-155928236
@revans2
Yes, as I described to [JIRA
issue](https://issues.apache.org/jira/browse/STORM-794?focusedCommentId=14526316&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14526316),
MasterBatchCoordinator doesn't play well with async loop, since it handles
emitting batch independently. MasterBatchCoordinator could make pending tuple
count always same to max spout pending in point of async loop's view, which
makes async loop to never trigger deactivate.
I'll do upmerge and let you know. Thanks!
> Trident Topology with some situation seems not handle deactivate during
> graceful shutdown
> -
>
> Key: STORM-794
> URL: https://issues.apache.org/jira/browse/STORM-794
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 0.9.3
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>
> I met an issue from Trident Topology in production env.
> Normally, when we kill a topology via UI, Nimbus changes Topology status to
> "killed", and when Spout determines new status, it becomes deactivated so
> bolts can handle remain tuples within wait-time.
> AFAIK that's how Storm guarantees graceful shutdown.
> But, Trident Topology seems not handle "deactivate" while we try shutdown
> topology gracefully.
> MasterBatchCoordinator never stops making next transaction, so Trident Spout
> never stops emitting, bolts (function) always take care of tuples.
> Topology setting
> - 1 worker, 1 acker
> - max spout pending: 1
> - TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS : 5
> -- It may be weird but MasterBatchCoordinator's default value is 1
> * Nimbus log
> {code}
> 2015-04-20 09:59:07.954 INFO [pool-5-thread-41][nimbus] Delaying event
> :remove for 120 secs for BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 09:59:07.955 INFO [pool-5-thread-41][nimbus] Updated
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015 with status {:type
> :killed, :kill-time-secs 120}
> ...
> 2015-04-20 10:01:07.956 INFO [timer][nimbus] Killing topology:
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> ...
> 2015-04-20 10:01:14.448 INFO [timer][nimbus] Cleaning up
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> {code}
> * Supervisor log
> {code}
> 2015-04-20 10:01:07.960 INFO [Thread-1][supervisor] Removing code for storm
> id BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down and
> clearing state for id 9719259e-528c-4336-abf9-592c1bb9a00b. Current
> supervisor time: 1429491667. State: :disallowed, Heartbeat:
> #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1429491667,
> :storm-id "BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015", :executors
> #{[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [11 11] [12 12] [13
> 13] [14 14] [-1 -1] [1 1]}, :port 6706}
> 2015-04-20 10:01:07.962 INFO [Thread-2][supervisor] Shutting down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> 2015-04-20 10:01:08.974 INFO [Thread-2][supervisor] Shut down
> 5bc084a2-b668-4610-86f6-9b93304d40a8:9719259e-528c-4336-abf9-592c1bb9a00b
> {code}
> * Worker log
> {code}
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down worker
> BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015
> 5bc084a2-b668-4610-86f6-9b93304d40a8 6706
> 2015-04-20 10:01:07.985 INFO [Thread-33][worker] Shutting down receive thread
> 2015-04-20 10:01:07.988 WARN [Thread-33][ExponentialBackoffRetry] maxRetries
> too large (300). Pinning to 29
> 2015-04-20 10:01:07.988 INFO
> [Thread-33][StormBoundedExponentialBackoffRetry] The baseSleepTimeMs [100]
> the maxSleepTimeMs [1000] the maxRetries [300]
> 2015-04-20 10:01:07.988 INFO [Thread-33][Client] New Netty Client, connect
> to localhost, 6706, config: , buffer_size: 5242880
> 2015-04-20 10:01:07.991 INFO [client-schedule-service-1][Client] Reconnect
> started for Netty-Client-localhost/127.0.0.1:6706... [0]
> 2015-04-20 10:01:07.996 INFO [Thread-33][loader] Shutting down
> receiving-thread: [BFDC-topology-DynamicCollect-68c9d7b4-72-1429491015, 6706]
> ...
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Closing Netty Client
> Netty-Client-localhost/127.0.0.1:6706
> 2015-04-20 10:01:08.044 INFO [Thread-33][Client] Waiting for pending batchs
> to be sent with Netty-Client-localhost/127.0.0.1:6706..., timeout: 60ms,
> pendings: 1
>
Re: [DISCUSS] 1.0 Release (was Re: [DISCUSS] Initial 0.11.0 Release)
+1 - Bobby On Wednesday, November 11, 2015 4:22 PM, P. Taylor Goetz wrote: Changing subject in order to consolidate discussion of a 1.0 release in one thread (there was some additional discussion in the thread regarding the JStorm code merge). I just want to make sure I’m accurately capturing the sentiment of the community with regard to a 1.0 release. Please correct me if my summary seems off-base or jump in with an opinion. In summary: 1. What we have been calling “0.11.0” will become the Storm 1.0 release. 2. We will NOT be migrating package names for this release (i.e. “backtype.storm” —> “org.apache.storm”). 3. Post 1.0 release we will go into feature freeze for core functionality to facilitate the JStorm merge. 4. During the feature freeze only fixes for high priority bugs in core functionality will be accepted (no new features). 5. During the feature freeze, enhancements to “external” modules can be accepted. 6. We will stop using the “beta” flag in favor of purely numeric version numbers. Stable vs. non-stable (development) releases can be indicated on the download page. Do we all agree? -Taylor > On Nov 11, 2015, at 4:10 PM, P. Taylor Goetz wrote: > > >> On Nov 11, 2015, at 9:28 AM, Bobby Evans wrote: >> >> I would like to see STORM-1190, STORM-1155, STORM-1198, STORM-1196, >> STORM-885, STORM-876, STORM-1145, STORM-831, and STORM-874 added to the list. >> >> Some of them are more important then others but they are all things I would >> like to see in a 0.11.0 release. > > > Cool. Thanks for listing them out. I will add them so they get tracked. > > >> On a side note I don't think the beta releases have been helpful. I would >> much rather just have a 0.11.0 go to 0.11.1 ... instead of 0.11.0-beta1, >> 0.11.0-beta2. To me the beta label is not that helpful, but it is not that >> big of a deal for me. > > In my mind, having releases tagged as “beta” were a way for us to say to > users “here’s a preview release that will allow you to kick the tires on > upcoming features, but be aware that there might be bugs. Let us know if you > find any so we can fix them.” > > I think the intent was sound, but what I didn’t really see was user feedback > on the beta releases, which is what I hoped would happen. So I have no > problem with dropping the use of “beta” flags. > > Another approach I’ve seen other Apache projects use is to the numbering > scheme you describe, and just have different labels/descriptions on the > download page — i.e. “Latest stable release” and “Latest development > release.” The nice part of that convention is that it does not have any > impact on the release process. For example if we’re confident that a > particular “development” release is actually quite stable, all we would have > to do is update the downloads page, rather than go through the whole > release/vote process just to remove the “beta” tag. > >> I also would like to see the 0.11 release tied to the plan for the JStorm >> merger. If we don't tie them together there can be code that does not make >> it into 0.11, but could make it into a 0.12 that will immediately be caught >> up in the merger, that could take a long time to complete. I mostly want us >> to be very transparent about what is likely to happen after 0.11 is >> released. So if someone has a feature that is close to getting something in >> to 0.11 that they speak up here instead of just deciding to wait for a 0.12 >> release. > > > I agree that the 0.11 release needs to be tied to the JStorm merger. Once > that release goes out, we’ll be going into lockdown mode for the merge > effort, which is likely to take a while. > > During that time it’s highly unlikely that any changes/additions to Storm’s > core functionality will be accepted beyond high-priority bug fixes. Adding > new features to the “external” components during that time is probably okay, > since those components are sufficiently decoupled from Storm’s core > functionality. > > So to reiterate Bobby’s statement: > > Please speak up now if there are any specific features or changes to Storm’s > core functionality that you’d like to see in the next release. Otherwise you > will have to wait. > >> - Bobby > > -Taylor > >> >> >> On Wednesday, November 11, 2015 6:32 AM, John Fang >> wrote: >> >> >> Totally agree， it's great to accelerate the release speed. it is better >>release one official version within 3 months. The first month is for >>developing new features , .If some features hasn't been finished in this >>month, it will be put in the next release ticket. In the last two month, we >>just do test. >> Storm may face the greatest challenge in a big cluster, so I am more >>concerned about ZK Optimization as jstorm did. At last, it's great for the >>next release to has a test report about the stability and performance , due >>to lots of new features in the latest versions. >> >> Regards >> John
[GitHub] storm pull request: [STORM-1198] Web UI to show resource usages an...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/875#discussion_r44596285
--- Diff: storm-core/src/ui/public/templates/index-page-template.html ---
@@ -229,6 +249,10 @@
{{uptime}}
{{slotsTotal}}
{{slotsUsed}}
+ {{totalMem}}
+ {{usedMem}}
+ {{totalCpu}}
+ {{usedCpu}}
--- End diff --
Could you update the REST API docs to indicate the new fields
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
