[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426102#comment-15426102 ] ASF GitHub Bot commented on STORM-1881: --- Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1480 STORM-2016 is on reviewing, and it should help this get resolved. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388696#comment-15388696 ] ASF GitHub Bot commented on STORM-1881: --- Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1480 Sorry for long delay, say, 1 month. I've met similar issue on Storm SQL, so I've initiate discussion to dev@ list. Along with discussion, I'm trying to see how Spark handles dependencies and adding them to executor. I guess I understand how it works, but I didn't play with how to resolve transitive dependencies and download them. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343994#comment-15343994 ] ASF GitHub Bot commented on STORM-1881: --- Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1480 Getting rid of Guava seems easy since storm-redis uses Precondition, ImmutableMap, Maps, Lists. Getting rid of Jackson seems not easy since we take StateConfig to JSON format so we need JSON parser anyway. It might need to explain current status and discuss how to resolve it from dev@. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343973#comment-15343973 ] ASF GitHub Bot commented on STORM-1881: --- Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1480 Let me have a look. I guess storm-redis doesn't need to depend on other than storm-core and Jedis. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1555#comment-1555 ] ASF GitHub Bot commented on STORM-1881: --- Github user abhishekagarwal87 commented on the issue: https://github.com/apache/storm/pull/1480 If you use already relocated classes, you make storm-redis module dependent on a particular version of storm-core. relocation pattern is subject to changes in storm-core (since it is internal) and if that happens, your module stops being compatible with storm-core. So you should avoid using already relocated classes in storm-core. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333252#comment-15333252 ] ASF GitHub Bot commented on STORM-1881: --- Github user sakanaou commented on the issue: https://github.com/apache/storm/pull/1480 @abhishekagarwal87: Ok, didn't spot that. Was only looking for "guava" relocations. So, the dependency could be additionally included relocated into storm-redis. Or is it possible to use the already relocated package which is provided via storm-core? This would save some resources. I'm not very experienced with the `maven-shade-plugin` yet, unfortunately. What is preferred? > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329383#comment-15329383 ] ASF GitHub Bot commented on STORM-1881: --- Github user abhishekagarwal87 commented on the issue: https://github.com/apache/storm/pull/1480 guava dependency is relocated in storm-core. https://github.com/apache/storm/blob/2a435b0289d43494eb2f834e94ed61c175f9eea6/storm-core/pom.xml#L593 > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329336#comment-15329336 ] ASF GitHub Bot commented on STORM-1881: --- Github user sakanaou commented on the issue: https://github.com/apache/storm/pull/1480 @HeartSaVioR: `storm-core/pom.xml` includes a `maven-shade-plugin` step that lists both guava and jackson as artifactSet-includes without relocations. Why should they then not be provided via storm-core? I have successfully used storm-redis without placing guava or jackson explicitly in the classpath so they must be present already. From my understanding they are provided by the "fat" storm-core jar. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329229#comment-15329229 ] ASF GitHub Bot commented on STORM-1881: --- Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1480 We're shading the libraries which you're changing scope to `provided` in storm-core, so it will not be provided at runtime. Since storm-redis depends on Guava, placing uber storm-redis jar to extlib incurs dependency issue in whole topologies in cluster. If we still really want to place this to extlib, we may need to consider shading guava and others for storm-redis module. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324286#comment-15324286 ] ASF GitHub Bot commented on STORM-1881: --- Github user abhishekagarwal87 commented on the issue: https://github.com/apache/storm/pull/1480 Thanks @sakanaou > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324282#comment-15324282 ] ASF GitHub Bot commented on STORM-1881: --- Github user sakanaou commented on the issue: https://github.com/apache/storm/pull/1480 @abhishekagarwal87: Original jar is 65K, shaded jar is 709K. guava and jackson are already contained in storm-core so those can be ignored, luckily. > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324276#comment-15324276 ] ASF GitHub Bot commented on STORM-1881: --- Github user abhishekagarwal87 commented on the issue: https://github.com/apache/storm/pull/1480 just curious, how much is the size of fat jar? > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1881) storm-redis is missing dependant libraries in distribution
[ https://issues.apache.org/jira/browse/STORM-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324269#comment-15324269 ] ASF GitHub Bot commented on STORM-1881: --- GitHub user sakanaou opened a pull request: https://github.com/apache/storm/pull/1480 STORM-1881: Create fat-jar of storm-redis which contains needed dependencies `storm-redis` was packaged in a way that it was not immediately usable when put into `extlib` folder. `ClassNotFoundException`s would occur. Via `maven-shade-plugin` a "fat"-jar is created so all depending code is packaged in. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sakanaou/storm redis-shadedjar Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1480.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1480 commit b6f89e7a5f63a59d942ecbc63b7ea6711824946e Author: Daniel Klessing Date: 2016-06-10T10:36:53Z STORM-1881: Create fat-jar of storm-redis which contains needed dependencies > storm-redis is missing dependant libraries in distribution > -- > > Key: STORM-1881 > URL: https://issues.apache.org/jira/browse/STORM-1881 > Project: Apache Storm > Issue Type: Bug > Components: storm-redis >Affects Versions: 1.0.1 >Reporter: Daniel Klessing > > Despite the documentation on > http://storm.apache.org/releases/1.0.1/State-checkpointing.html it is not > enough to simply copy {{storm-redis-*.jar}} to {{extlib}} to get the > {{RedisKeyValueStateProvider}} working. Depending jedis and > apache-commons-pool2 jars are missing and must be copied by hand to get it > working. Else one is greeted with exception stack traces like: > {code} > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.pool2.impl.GenericObjectPoolConfig > {code} > or > {code} > Caused by: java.lang.ClassNotFoundException: > redis.clients.jedis.JedisPoolConfig > {code} > Copying {{commons-pool2-2.4.2.jar}} and {{jedis-2.8.1.jar}} from hand to > {{extlib}} solves the issue. > It might be better to create a "fat" jar of {{storm-redis-*.jar}} or provide > documentation, which libraries have to be made available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)