[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560125#comment-14560125 ] ASF GitHub Bot commented on STORM-821: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105697874 LGTM. :+1: Btw, we don't release storm-jdbc module to any Storm releases yet, so IMO it could be not mandatory to have a new issue regarding to Column when we promise #556 will be merged to 0.10.x before releasing. > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105697874 LGTM. :+1: Btw, we don't release storm-jdbc module to any Storm releases yet, so IMO it could be not mandatory to have a new issue regarding to Column when we promise #556 will be merged to 0.10.x before releasing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-817) Kafka Wildcard Topic Support
[ https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560107#comment-14560107 ] Sumit Chawla commented on STORM-817: Thanks [~kabhwan] .. Any idea what is typical time for this to get merged and who will be doing it? > Kafka Wildcard Topic Support > > > Key: STORM-817 > URL: https://issues.apache.org/jira/browse/STORM-817 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka >Reporter: Sumit Chawla >Assignee: Sumit Chawla > > Creating a feature request for supporting Wildcard Topic's for Kafka Spout. > We want to be able to run a aggregation stream for data coming from all > tenants. Tenants get added dynamically. So new kafka topics get created. All > the topics will be matching a regex pattern. > example: > clickstream:tenant1:log > clickstream:tenant2:log > clickstream:tenant3:log > Storm code should be able to perform auto-discovery, and should be able to to > fetch from newly created topics in run time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (STORM-817) Kafka Wildcard Topic Support
[ https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened STORM-817: Reopening this. Issue will be closed by committer when PR is merged to master branch, so you don't need to close this issue yourself. Same things applied to "Fix Version". > Kafka Wildcard Topic Support > > > Key: STORM-817 > URL: https://issues.apache.org/jira/browse/STORM-817 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka >Reporter: Sumit Chawla >Assignee: Sumit Chawla > > Creating a feature request for supporting Wildcard Topic's for Kafka Spout. > We want to be able to run a aggregation stream for data coming from all > tenants. Tenants get added dynamically. So new kafka topics get created. All > the topics will be matching a regex pattern. > example: > clickstream:tenant1:log > clickstream:tenant2:log > clickstream:tenant3:log > Storm code should be able to perform auto-discovery, and should be able to to > fetch from newly created topics in run time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (STORM-817) Kafka Wildcard Topic Support
[ https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-817: --- Fix Version/s: (was: 0.11.0) > Kafka Wildcard Topic Support > > > Key: STORM-817 > URL: https://issues.apache.org/jira/browse/STORM-817 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka >Reporter: Sumit Chawla >Assignee: Sumit Chawla > > Creating a feature request for supporting Wildcard Topic's for Kafka Spout. > We want to be able to run a aggregation stream for data coming from all > tenants. Tenants get added dynamically. So new kafka topics get created. All > the topics will be matching a regex pattern. > example: > clickstream:tenant1:log > clickstream:tenant2:log > clickstream:tenant3:log > Storm code should be able to perform auto-discovery, and should be able to to > fetch from newly created topics in run time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[
https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560076#comment-14560076
]
ASF GitHub Bot commented on STORM-821:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31089108
--- Diff: external/storm-jdbc/README.md ---
@@ -60,6 +96,7 @@
hikariConfigMap.put("dataSourceClassName","com.mysql.jdbc.jdbc2.optional.MysqlDa
hikariConfigMap.put("dataSource.url", "jdbc:mysql://localhost/test");
hikariConfigMap.put("dataSource.user","root");
hikariConfigMap.put("dataSource.password","password");
+ConnectionProvider connectionProvider = new
HikariCPConnectionProvider(map);
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
--- End diff --
@Parth-Brahmbhatt This line is not changed yet, so you may want to check it
again.
> storm-jdbc create a connection provider interface to decouple from hikariCP
> being the only connection pool implementation that can be used.
> ---
>
> Key: STORM-821
> URL: https://issues.apache.org/jira/browse/STORM-821
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
> Fix For: 0.10.0
>
>
> The current implementation of storm-jdbc is couple with HikariCP
> configuration. We propose to remove this coupling by introducing a
> connectionProvider interface with a default HikariCP implementation. This
> will allow users to do their own connection pool management or chose a
> different connection pooling library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31089108
--- Diff: external/storm-jdbc/README.md ---
@@ -60,6 +96,7 @@
hikariConfigMap.put("dataSourceClassName","com.mysql.jdbc.jdbc2.optional.MysqlDa
hikariConfigMap.put("dataSource.url", "jdbc:mysql://localhost/test");
hikariConfigMap.put("dataSource.user","root");
hikariConfigMap.put("dataSource.password","password");
+ConnectionProvider connectionProvider = new
HikariCPConnectionProvider(map);
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
--- End diff --
@Parth-Brahmbhatt This line is not changed yet, so you may want to check it
again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105693031 @Parth-Brahmbhatt I agree that Guava is widely-used library. I'm OK to keep dependency from example. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560063#comment-14560063 ] ASF GitHub Bot commented on STORM-821: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105693031 @Parth-Brahmbhatt I agree that Guava is widely-used library. I'm OK to keep dependency from example. :) > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-817) Kafka Wildcard Topic Support
[ https://issues.apache.org/jira/browse/STORM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Chawla resolved STORM-817. Resolution: Implemented Fix Version/s: 0.11.0 Changes have been implemented and a pull request is opened. https://github.com/apache/storm/pull/561 > Kafka Wildcard Topic Support > > > Key: STORM-817 > URL: https://issues.apache.org/jira/browse/STORM-817 > Project: Apache Storm > Issue Type: New Feature > Components: storm-kafka >Reporter: Sumit Chawla >Assignee: Sumit Chawla > Fix For: 0.11.0 > > > Creating a feature request for supporting Wildcard Topic's for Kafka Spout. > We want to be able to run a aggregation stream for data coming from all > tenants. Tenants get added dynamically. So new kafka topics get created. All > the topics will be matching a regex pattern. > example: > clickstream:tenant1:log > clickstream:tenant2:log > clickstream:tenant3:log > Storm code should be able to perform auto-discovery, and should be able to to > fetch from newly created topics in run time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560058#comment-14560058 ] ASF GitHub Bot commented on STORM-821: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105692443 @harshach @ptgoetz @HeartSaVioR I forgot to mention that I also made a minor change in column.java so we can handle database columns with null values. It was a small change so filing another jira felt like an overkill. If anyone has objection to that let me know and I will open a different jira. > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105692443 @harshach @ptgoetz @HeartSaVioR I forgot to mention that I also made a minor change in column.java so we can handle database columns with null values. It was a small change so filing another jira felt like an overkill. If anyone has objection to that let me know and I will open a different jira. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560049#comment-14560049 ] ASF GitHub Bot commented on STORM-821: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105691986 @HeartSaVioR all comments addressed, thanks for finding the README issues and keeping the documentation honest. I did not remove the Guava dependency as there are multiple references to Lists and I think Guava is so widely used its unlikely to confuse anyone reading the README. > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105691986 @HeartSaVioR all comments addressed, thanks for finding the README issues and keeping the documentation honest. I did not remove the Guava dependency as there are multiple references to Lists and I think Guava is so widely used its unlikely to confuse anyone reading the README. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[
https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560038#comment-14560038
]
ASF GitHub Bot commented on STORM-821:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087950
--- Diff: external/storm-jdbc/README.md ---
@@ -60,6 +96,7 @@
hikariConfigMap.put("dataSourceClassName","com.mysql.jdbc.jdbc2.optional.MysqlDa
hikariConfigMap.put("dataSource.url", "jdbc:mysql://localhost/test");
hikariConfigMap.put("dataSource.user","root");
hikariConfigMap.put("dataSource.password","password");
+ConnectionProvider connectionProvider = new
HikariCPConnectionProvider(map);
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
--- End diff --
After modifying https://github.com/apache/storm/pull/556/files#r31087884,
it should be changed to
```
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName,
connectionProvider);
```
> storm-jdbc create a connection provider interface to decouple from hikariCP
> being the only connection pool implementation that can be used.
> ---
>
> Key: STORM-821
> URL: https://issues.apache.org/jira/browse/STORM-821
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
> Fix For: 0.10.0
>
>
> The current implementation of storm-jdbc is couple with HikariCP
> configuration. We propose to remove this coupling by introducing a
> connectionProvider interface with a default HikariCP implementation. This
> will allow users to do their own connection pool management or chose a
> different connection pooling library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087950
--- Diff: external/storm-jdbc/README.md ---
@@ -60,6 +96,7 @@
hikariConfigMap.put("dataSourceClassName","com.mysql.jdbc.jdbc2.optional.MysqlDa
hikariConfigMap.put("dataSource.url", "jdbc:mysql://localhost/test");
hikariConfigMap.put("dataSource.user","root");
hikariConfigMap.put("dataSource.password","password");
+ConnectionProvider connectionProvider = new
HikariCPConnectionProvider(map);
String tableName = "user_details";
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName, map);
--- End diff --
After modifying https://github.com/apache/storm/pull/556/files#r31087884,
it should be changed to
```
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(tableName,
connectionProvider);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[
https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560034#comment-14560034
]
ASF GitHub Bot commented on STORM-821:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087884
--- Diff:
external/storm-jdbc/src/main/java/org/apache/storm/jdbc/mapper/SimpleJdbcMapper.java
---
@@ -33,9 +34,10 @@
private List schemaColumns;
-public SimpleJdbcMapper(String tableName, Map hikariConfigurationMap) {
+public SimpleJdbcMapper(String tableName, Map
connectionProvideConfig, ConnectionProvider connectionProvider) {
--- End diff --
Seems like we don't need to connectionProvideConfig here for now.
> storm-jdbc create a connection provider interface to decouple from hikariCP
> being the only connection pool implementation that can be used.
> ---
>
> Key: STORM-821
> URL: https://issues.apache.org/jira/browse/STORM-821
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
> Fix For: 0.10.0
>
>
> The current implementation of storm-jdbc is couple with HikariCP
> configuration. We propose to remove this coupling by introducing a
> connectionProvider interface with a default HikariCP implementation. This
> will allow users to do their own connection pool management or chose a
> different connection pooling library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087884
--- Diff:
external/storm-jdbc/src/main/java/org/apache/storm/jdbc/mapper/SimpleJdbcMapper.java
---
@@ -33,9 +34,10 @@
private List schemaColumns;
-public SimpleJdbcMapper(String tableName, Map hikariConfigurationMap) {
+public SimpleJdbcMapper(String tableName, Map
connectionProvideConfig, ConnectionProvider connectionProvider) {
--- End diff --
Seems like we don't need to connectionProvideConfig here for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[
https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560031#comment-14560031
]
ASF GitHub Bot commented on STORM-821:
--
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087567
--- Diff: external/storm-jdbc/README.md ---
@@ -29,13 +57,21 @@ The default is set to value of
topology.message.timeout.secs and a value of -1 w
You should set the query timeout value to be <=
topology.message.timeout.secs.
```java
-Config config = new Config();
-config.put("jdbc.conf", hikariConfigMap);
-JdbcInsertBolt userPersistanceBolt = new
JdbcInsertBolt("jdbc.conf",simpleJdbcMapper)
+Map hikariConfigMap = Maps.newHashMap();
--- End diff --
minor thing:
How about changing this line to
```
Map hikariConfigMap = new HashMap();
```
to remove dependency about Guava from example?
> storm-jdbc create a connection provider interface to decouple from hikariCP
> being the only connection pool implementation that can be used.
> ---
>
> Key: STORM-821
> URL: https://issues.apache.org/jira/browse/STORM-821
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Minor
> Fix For: 0.10.0
>
>
> The current implementation of storm-jdbc is couple with HikariCP
> configuration. We propose to remove this coupling by introducing a
> connectionProvider interface with a default HikariCP implementation. This
> will allow users to do their own connection pool management or chose a
> different connection pooling library.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/556#discussion_r31087567
--- Diff: external/storm-jdbc/README.md ---
@@ -29,13 +57,21 @@ The default is set to value of
topology.message.timeout.secs and a value of -1 w
You should set the query timeout value to be <=
topology.message.timeout.secs.
```java
-Config config = new Config();
-config.put("jdbc.conf", hikariConfigMap);
-JdbcInsertBolt userPersistanceBolt = new
JdbcInsertBolt("jdbc.conf",simpleJdbcMapper)
+Map hikariConfigMap = Maps.newHashMap();
--- End diff --
minor thing:
How about changing this line to
```
Map hikariConfigMap = new HashMap();
```
to remove dependency about Guava from example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560025#comment-14560025 ] ASF GitHub Bot commented on STORM-821: -- Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/556#discussion_r31087238 --- Diff: external/storm-jdbc/README.md --- @@ -17,11 +44,12 @@ The `getColumns()` method defines how a storm tuple maps to a list of columns re **The order of the returned list is important. The place holders in the supplied queries are resolved in the same order as returned list.** For example if the user supplied insert query is `insert into user(user_id, user_name, create_date) values (?,?, now())` the 1st item of the returned list of `getColumns` method will map to the 1st place holder and the 2nd to the 2nd and so on. We do not parse -the supplied queries to try and resolve place holder by column names. +the supplied queries to try and resolve place holder by column names. Not making any assumptions about the query syntax allows this connector +to be used by some non-standard sql frameworks like Pheonix which only supports upsert into. ### JdbcInsertBolt -To use the `JdbcInsertBolt`, you construct an instance of it and specify a configuration key in your storm config that holds the -hikari configuration map and a `JdbcMapper` implementation that coverts storm tuple to DB row. In addition, you must either supply +To use the `JdbcInsertBolt`, you construct an instance of it by specifying a `ConnectionProvider` implementation +and a `JdbcMapper` implementation that coverts storm tuple to DB row. In addition, you must either supply --- End diff -- coverts -> converts > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-834) Class cast exception Object to Iterable
[ https://issues.apache.org/jira/browse/STORM-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560023#comment-14560023 ] Jason Kania commented on STORM-834: --- The error is coming from a bolt as best I can tell from the log file in which the error is found, but the lack of detail in the exception means i can tell little else. > Class cast exception Object to Iterable > --- > > Key: STORM-834 > URL: https://issues.apache.org/jira/browse/STORM-834 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 > Environment: Debian linux >Reporter: Jason Kania > > This exception is happening after some hours of running. Unfortunately, > nothing in the trace suggests the source of the error for further > investigation. > 2015-05-23T13:24:22.343-0400 b.s.util [ERROR] Async loop died! > java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Object > cannot be cast to java.lang.Iterable > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$fn__3441$fn__3453$fn__3500.invoke(executor.clj:748) > ~[storm-core-0.9.3.jar:0.9.3] > at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) > ~[storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.ClassCastException: java.lang.Object cannot be cast to > java.lang.Iterable > at backtype.storm.util$get_iterator.invoke(util.clj:867) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$mk_task_receiver$fn__3364.invoke(executor.clj:397) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) > ~[storm-core-0.9.3.jar:0.9.3] > ... 6 common frames omitted -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/556#discussion_r31087238 --- Diff: external/storm-jdbc/README.md --- @@ -17,11 +44,12 @@ The `getColumns()` method defines how a storm tuple maps to a list of columns re **The order of the returned list is important. The place holders in the supplied queries are resolved in the same order as returned list.** For example if the user supplied insert query is `insert into user(user_id, user_name, create_date) values (?,?, now())` the 1st item of the returned list of `getColumns` method will map to the 1st place holder and the 2nd to the 2nd and so on. We do not parse -the supplied queries to try and resolve place holder by column names. +the supplied queries to try and resolve place holder by column names. Not making any assumptions about the query syntax allows this connector +to be used by some non-standard sql frameworks like Pheonix which only supports upsert into. ### JdbcInsertBolt -To use the `JdbcInsertBolt`, you construct an instance of it and specify a configuration key in your storm config that holds the -hikari configuration map and a `JdbcMapper` implementation that coverts storm tuple to DB row. In addition, you must either supply +To use the `JdbcInsertBolt`, you construct an instance of it by specifying a `ConnectionProvider` implementation +and a `JdbcMapper` implementation that coverts storm tuple to DB row. In addition, you must either supply --- End diff -- coverts -> converts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105689247 Class name has a typo: ConnectionPrvoider -> ConnectionProvider. It's not just documentation typo, but also real classname typo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560017#comment-14560017 ] ASF GitHub Bot commented on STORM-821: -- Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105689247 Class name has a typo: ConnectionPrvoider -> ConnectionProvider. It's not just documentation typo, but also real classname typo. > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560013#comment-14560013 ] ASF GitHub Bot commented on STORM-821: -- Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/556#discussion_r31086911 --- Diff: external/storm-jdbc/README.md --- @@ -1,10 +1,37 @@ #Storm JDBC Storm/Trident integration for JDBC. This package includes the core bolts and trident states that allows a storm topology to either insert storm tuples in a database table or to execute select queries against a database and enrich tuples -in a storm topology. This code uses HikariCP for connection pooling. See http://brettwooldridge.github.io/HikariCP. +in a storm topology. ## Inserting into a database. The bolt and trident state included in this package for inserting data into a database tables are tied to a single table. + +### ConnectionProvider +An interface that should be implemented by different connection pooling mechanism `org.apache.storm.jdbc.common.ConnectionPrvoider` --- End diff -- There's a typo, ConnectionProvioder -> ConnectionProvider. > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/556#discussion_r31086911 --- Diff: external/storm-jdbc/README.md --- @@ -1,10 +1,37 @@ #Storm JDBC Storm/Trident integration for JDBC. This package includes the core bolts and trident states that allows a storm topology to either insert storm tuples in a database table or to execute select queries against a database and enrich tuples -in a storm topology. This code uses HikariCP for connection pooling. See http://brettwooldridge.github.io/HikariCP. +in a storm topology. ## Inserting into a database. The bolt and trident state included in this package for inserting data into a database tables are tied to a single table. + +### ConnectionProvider +An interface that should be implemented by different connection pooling mechanism `org.apache.storm.jdbc.common.ConnectionPrvoider` --- End diff -- There's a typo, ConnectionProvioder -> ConnectionProvider. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-834) Class cast exception Object to Iterable
[ https://issues.apache.org/jira/browse/STORM-834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560003#comment-14560003 ] Jungtaek Lim commented on STORM-834: [~longtimer] Which component throws ClastCastException? (Spout / Bolt / Acker / etc) > Class cast exception Object to Iterable > --- > > Key: STORM-834 > URL: https://issues.apache.org/jira/browse/STORM-834 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.9.3 > Environment: Debian linux >Reporter: Jason Kania > > This exception is happening after some hours of running. Unfortunately, > nothing in the trace suggests the source of the error for further > investigation. > 2015-05-23T13:24:22.343-0400 b.s.util [ERROR] Async loop died! > java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Object > cannot be cast to java.lang.Iterable > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$fn__3441$fn__3453$fn__3500.invoke(executor.clj:748) > ~[storm-core-0.9.3.jar:0.9.3] > at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) > ~[storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75] > Caused by: java.lang.ClassCastException: java.lang.Object cannot be cast to > java.lang.Iterable > at backtype.storm.util$get_iterator.invoke(util.clj:867) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$mk_task_receiver$fn__3364.invoke(executor.clj:397) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$clojure_handler$reify__1447.onEvent(disruptor.clj:58) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) > ~[storm-core-0.9.3.jar:0.9.3] > ... 6 common frames omitted -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-805) Supervisor crashes with 'org.apache.commons.io.FileExistsException: Destination already exists' error
[
https://issues.apache.org/jira/browse/STORM-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559691#comment-14559691
]
Sanket Reddy commented on STORM-805:
Is this issue not resolved by https://github.com/apache/storm/pull/400 ?
> Supervisor crashes with 'org.apache.commons.io.FileExistsException:
> Destination already exists' error
> -
>
> Key: STORM-805
> URL: https://issues.apache.org/jira/browse/STORM-805
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Adrian Seungjin Lee
>
> Version : 0.9.4
> All of a sudden supervisor crashed with error log below.
> 2015-04-29 16:48:24 b.s.d.supervisor [INFO]
> 6d035728-bb86-486e-847e-94cd193a51db still hasn't started
> 2015-04-29 16:48:24 b.s.event [ERROR] Error when processing event
> org.apache.commons.io.FileExistsException: Destination
> '/somepath/storm-6-1430100813' already exists
> at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:2748)
> ~[commons-io-2.4.jar:2.4]
> at
> backtype.storm.daemon.supervisor$fn__5554.invoke(supervisor.clj:489)
> ~[storm-core-0.9.4.jar:0.9.4]
> at clojure.lang.MultiFn.invoke(MultiFn.java:241)
> ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__5474.invoke(supervisor.clj:374)
> ~[storm-core-0.9.4.jar:0.9.4]
> at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:40)
> ~[storm-core-0.9.4.jar:0.9.4]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
> 2015-04-29 16:48:24 b.s.util [ERROR] Halting process: ("Error when processing
> an event")
> java.lang.RuntimeException: ("Error when processing an event")
> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325)
> [storm-core-0.9.4.jar:0.9.4]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
> at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:48)
> [storm-core-0.9.4.jar:0.9.4]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
> 2015-04-29 16:48:24 b.s.d.supervisor [INFO] Shutting down supervisor
> 4f00962b-2af0-4922-97f8-142db00fbddf
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-813) Storm-starter cannot run multilang topology with exec:java with new multilang packages
[ https://issues.apache.org/jira/browse/STORM-813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559575#comment-14559575 ] ASF GitHub Bot commented on STORM-813: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/548#issuecomment-105627437 +1 > Storm-starter cannot run multilang topology with exec:java with new multilang > packages > -- > > Key: STORM-813 > URL: https://issues.apache.org/jira/browse/STORM-813 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.11.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim > > After STORM-748, storm-starter doesn't have multilang files now, and just > having multilang artifact as dependency. > With supervisor it works like a charm, but now mvn exec:java with multilang > topology doesn't work. > It would be better to remove broken functionality from README.markdown or > make it work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-813 Change storm-starter's README so tha...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/548#issuecomment-105627437 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-810) PartitionManager in storm-kafka should commit latest offset before close
[ https://issues.apache.org/jira/browse/STORM-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559559#comment-14559559 ] ASF GitHub Bot commented on STORM-810: -- Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/545#issuecomment-105624641 +1 > PartitionManager in storm-kafka should commit latest offset before close > - > > Key: STORM-810 > URL: https://issues.apache.org/jira/browse/STORM-810 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Affects Versions: 0.10.0, 0.11.0 >Reporter: xiajun >Assignee: xiajun > > When KafkaSpout refresh with kafka partitions, it will remove patitions that > no longer served by this spout and close the connection. But before close the > connection, PartitionManager don't commit the latest offset. As a result some > other spout that serve the removed partition will replay some tuple that > between last commit and close. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-810: PartitionManager should commit late...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/545#issuecomment-105624641 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-821) storm-jdbc create a connection provider interface to decouple from hikariCP being the only connection pool implementation that can be used.
[ https://issues.apache.org/jira/browse/STORM-821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559514#comment-14559514 ] ASF GitHub Bot commented on STORM-821: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105621487 +1 > storm-jdbc create a connection provider interface to decouple from hikariCP > being the only connection pool implementation that can be used. > --- > > Key: STORM-821 > URL: https://issues.apache.org/jira/browse/STORM-821 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Minor > Fix For: 0.10.0 > > > The current implementation of storm-jdbc is couple with HikariCP > configuration. We propose to remove this coupling by introducing a > connectionProvider interface with a default HikariCP implementation. This > will allow users to do their own connection pool management or chose a > different connection pooling library. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-821: Adding connection provider interfac...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/556#issuecomment-105621487 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-810) PartitionManager in storm-kafka should commit latest offset before close
[ https://issues.apache.org/jira/browse/STORM-810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559513#comment-14559513 ] ASF GitHub Bot commented on STORM-810: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/545#issuecomment-105621284 +1 > PartitionManager in storm-kafka should commit latest offset before close > - > > Key: STORM-810 > URL: https://issues.apache.org/jira/browse/STORM-810 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Affects Versions: 0.10.0, 0.11.0 >Reporter: xiajun >Assignee: xiajun > > When KafkaSpout refresh with kafka partitions, it will remove patitions that > no longer served by this spout and close the connection. But before close the > connection, PartitionManager don't commit the latest offset. As a result some > other spout that serve the removed partition will replay some tuple that > between last commit and close. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-810: PartitionManager should commit late...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/545#issuecomment-105621284 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-831) Add Jira and Central Logging URL to UI
[ https://issues.apache.org/jira/browse/STORM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559486#comment-14559486 ] ASF GitHub Bot commented on STORM-831: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-105618678 @kishorvpatil Are you sure the two binary images are in the public domain and that we would be allowed to include them in an Apache release? Is there an associated license somewhere? Also, I'd rather have the references to JIRA be more generic, for example "UI_PROJECT_BUGTRACKER_URL" instead of "UI_PROJECT_JIRA_URL". It would also help to add some documentation on how to set the links up. It's obvious to me, but it may not be to a new Storm user. > Add Jira and Central Logging URL to UI > -- > > Key: STORM-831 > URL: https://issues.apache.org/jira/browse/STORM-831 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Trivial > > As a user, I would like to see a link to take me to JIRA for reporting bug. > Also, optionally if link to splunk/logstash/kibana from UI would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-831] Adding jira and central logging li...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-105618678 @kishorvpatil Are you sure the two binary images are in the public domain and that we would be allowed to include them in an Apache release? Is there an associated license somewhere? Also, I'd rather have the references to JIRA be more generic, for example "UI_PROJECT_BUGTRACKER_URL" instead of "UI_PROJECT_JIRA_URL". It would also help to add some documentation on how to set the links up. It's obvious to me, but it may not be to a new Storm user. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: 1234
Github user Z-TAO closed the pull request at: https://github.com/apache/storm/pull/555 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: 1234
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/555#issuecomment-105606200 @Z-TAO will you close this pull request? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-831) Add Jira and Central Logging URL to UI
[ https://issues.apache.org/jira/browse/STORM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559414#comment-14559414 ] ASF GitHub Bot commented on STORM-831: -- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-105606041 +1 > Add Jira and Central Logging URL to UI > -- > > Key: STORM-831 > URL: https://issues.apache.org/jira/browse/STORM-831 > Project: Apache Storm > Issue Type: Improvement >Reporter: Kishor Patil >Assignee: Kishor Patil >Priority: Trivial > > As a user, I would like to see a link to take me to JIRA for reporting bug. > Also, optionally if link to splunk/logstash/kibana from UI would be helpful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-831] Adding jira and central logging li...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/559#issuecomment-105606041 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-835) Netty Client hold batch object until io operation complete
[
https://issues.apache.org/jira/browse/STORM-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559409#comment-14559409
]
ASF GitHub Bot commented on STORM-835:
--
Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/563#issuecomment-105605777
+1 Looks fine.
> Netty Client hold batch object until io operation complete
> --
>
> Key: STORM-835
> URL: https://issues.apache.org/jira/browse/STORM-835
> Project: Apache Storm
> Issue Type: Improvement
>Affects Versions: 0.10.0
>Reporter: zhanghailei
>Assignee: zhanghailei
>Priority: Minor
>
> Netty Client hold batch object until IO operation complete.
> Following code:
> {code:title=Client.java|borderStyle=solid}
> final int numMessages = batch.size();
> pendingMessages.getAndAdd(numMessages);
> LOG.debug("writing {} messages to channel {}", batch.size(),
> channel.toString());
> ChannelFuture future = channel.write(batch);
> future.addListener(new ChannelFutureListener() {
> public void operationComplete(ChannelFuture future) throws
> Exception {
> pendingMessages.getAndAdd(0 - numMessages);
> if (future.isSuccess()) {
> LOG.debug("sent {} messages to {}", numMessages,
> dstAddressPrefixedName);
> messagesSent.getAndAdd(batch.size());
> }
> {code}
> batch will be a field of anonymous inner classes.don't release after
> channel.write() invoked,until IO operation complete.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-835 Netty Client hold batch object until...
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/563#issuecomment-105605777 +1 Looks fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Storm-616 : Storm-jdbc connector.
Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/374#issuecomment-105598508 @Crystark You are right I will fix it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (STORM-837) HdfsState ignores commits
Robert Joseph Evans created STORM-837: - Summary: HdfsState ignores commits Key: STORM-837 URL: https://issues.apache.org/jira/browse/STORM-837 Project: Apache Storm Issue Type: Bug Reporter: Robert Joseph Evans Priority: Critical HdfsState works with trident which is supposed to provide exactly once processing. It does this two ways, first by informing the state about commits so it can be sure the data is written out, and second by having a commit id, so that double commits can be handled. HdfsState ignores the beginCommit and commit calls, and with that ignores the ids. This means that if you use HdfsState and your worker crashes you may both lose data and get some data twice. At a minimum the flush and file rotation should be tied to the commit in some way. The commit ID should at a minimum be written out with the data so someone reading the data can have a hope of deduping it themselves. Also with the rotationActions it is possible for a file that was partially written is leaked, and never moved to the final location, because it is not rotated. I personally think the actions are too generic for this case and need to be deprecated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Aeolus 0.1 available
Dear Storm community, we would like to share our project Aeolus with you. While the project is not finished, our first component --- a transparent batching layer --- is available now. Aeolus' batching component, is a transparent layer that can increase Storm's throughput by an order of magnitude while keeping tuple-by-tuple processing semantics. Batching happens transparent to the system and the user code. Thus, it can be used without changing existing code. Aeolus is available using Apache License 2.0 and would be happy to any feedback. If you like to try it out, you can download Aeolus from our git repository: https://github.com/mjsax/aeolus Happy hacking, Matthias signature.asc Description: OpenPGP digital signature
Re: how to get instant system loading of worker node in storm cluster?
Most of the people who have been doing resource feedback to storm have built an external feedback system, and written a scheduler that knows how to access that external data. Typically this information would be send as part of the worker heartbeat. You might want to look at doing that, but you are going to have to extend how it does the heartbeat, because most of the metrics are on a per task basis, and CPU is very difficult to capture per task. Capacity is a very rough approximation of this, but it only works if the bolt is truly single threaded. You could use it and the actual CPU measurement for the worker to approximate the CPU utilization for a single task, but it is not going to be simple. Especially if you use a Shell bolt or Shell spout, as they are not even part of the same process. - Bobby On Monday, May 25, 2015 3:25 AM, 陳怡仁 wrote: thanks for your tips again, These days I just find out a interesting paper on implementing a storm plug-gable scheduler that can run-time measure CPU utilization of each worker node. For Storm 0.9.4, it can not do that, right? where i can try to add extra measuring code into Storm to support measurement mechanism? Best regards, -YR
flux
Hi All, I am trying to test flux module for writing template driven topologies. I setting the topologies in localcluster mode. 1) using template simple_wordcount.yaml 2) using kafka_spout.yaml (using TestBolt) With (1) I don't get any output form TestBolt and the topology exits. With (2), I get the following error: java.lang.IllegalArgumentException: Couldn't find a suitable constructor for class 'storm.kafka.KafkaConfig' with arguments '[storm.kafka.ZkHosts@17d1e01f, TOPIC, /storm, myKafkaClientId]\ Any idea what I could be doing wrong? thanks Clay
[GitHub] storm pull request: Storm-616 : Storm-jdbc connector.
Github user Crystark commented on the pull request: https://github.com/apache/storm/pull/374#issuecomment-105472684 On more thing i noticed: [Column.hashCode](https://github.com/apache/storm/blob/master/external/storm-jdbc/src/main/java/org/apache/storm/jdbc/common/Column.java#L99) doesn't support the value beeing null. I've tried using those columns as keys of a HashMap and got a NPE. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
