[jira] [Commented] (SPARK-13975) Cannot specify extra libs for executor from /extra-lib
[ https://issues.apache.org/jira/browse/SPARK-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199675#comment-15199675 ] Leonid Poliakov commented on SPARK-13975: - I am trying to avoid adding extra steps for users to perform when they deal with bundle. It would be great if libs were loaded out-of-box. And that's what user would actually expect, since there is no good excuse for an extra step for user to go in config and add absolute path to something that is already within bundle. > Cannot specify extra libs for executor from /extra-lib > -- > > Key: SPARK-13975 > URL: https://issues.apache.org/jira/browse/SPARK-13975 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 1.6.1 >Reporter: Leonid Poliakov > > If you build a framework on top of spark and want to bundle it with the > spark, there is no easy way to add your framework libs to executor classpath. > Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new > bundle (with my libs in it) to nodes, run the bundle. I want executors on > node to always automatically load my libs from {{/extra-lib}}, because that's > how future developers would use framework out-of-the-box. > The config doc says you can specify extraClasspath for the executor in > {{spark-defaults.conf}}, which is good because custom config may be put in > the bundle for the framework, but the syntax of the property is unclear. > You can basically specify the value that will be appended to {{-cp}} for a > executor Java process, so it follows the Java how-to-set-classpath rules, so > basically you have two options here: > 1. specify absolute path > bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/* > 2. specify relative path > bq. spark.executor.extraClassPath ../../../extra-lib/* > But none of these ways look good: absolute path won't work at all since you > cannot know where users will put the bundle, relative path looks weird > because executor will have it's work directory set to something like > {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker > folder is configured. > So, it's required to have a proper way to bundle custom libs and set executor > classpath to load them up. > *Expected*: you can specify {{spark.executor.extraClassPath}} relative to > {{$SPARK_HOME}} using placeholders, e.g. with next syntax: > bq. spark.executor.extraClassPath ${home}/extra-lib/* > Code will resolve placeholders in properties with a proper path > Executor will get absolute path in {{-cp}} this way > *Actual*: you cannot specify extra libs for executor relative to > {{$SPARK_HOME}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13975) Cannot specify extra libs for executor from /extra-lib
[ https://issues.apache.org/jira/browse/SPARK-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199705#comment-15199705 ] Leonid Poliakov commented on SPARK-13975: - Thanks for you quick response, Sean The problem is, I'm bundling extra files with my framework, specifically, a memory cache that the spark will interact with to store/read RDDs, etc. The libs for the memcache to work are quite huge, and I can't really reduce their size. So user apps have to do either: a. include libs and be used as fat jar This slows down the apps transfer, which I really want to avoid b. put a wrapper around spark-submit to append libs to {{--jars}} This is how I do it right now, but executors fetch the jars each time, which causes fetched jars to stack up in {{/work}} and manual clean-up is required c. load jars in each executor That's what I want, but can't figure out how to do I know it's a rare use case for spark, but I believe shipping custom libs with the spark to nodes in one bundle must be a very portable and easy approach. > Cannot specify extra libs for executor from /extra-lib > -- > > Key: SPARK-13975 > URL: https://issues.apache.org/jira/browse/SPARK-13975 > Project: Spark > Issue Type: Bug > Components: Spark Submit >Affects Versions: 1.6.1 >Reporter: Leonid Poliakov > > If you build a framework on top of spark and want to bundle it with the > spark, there is no easy way to add your framework libs to executor classpath. > Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new > bundle (with my libs in it) to nodes, run the bundle. I want executors on > node to always automatically load my libs from {{/extra-lib}}, because that's > how future developers would use framework out-of-the-box. > The config doc says you can specify extraClasspath for the executor in > {{spark-defaults.conf}}, which is good because custom config may be put in > the bundle for the framework, but the syntax of the property is unclear. > You can basically specify the value that will be appended to {{-cp}} for a > executor Java process, so it follows the Java how-to-set-classpath rules, so > basically you have two options here: > 1. specify absolute path > bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/* > 2. specify relative path > bq. spark.executor.extraClassPath ../../../extra-lib/* > But none of these ways look good: absolute path won't work at all since you > cannot know where users will put the bundle, relative path looks weird > because executor will have it's work directory set to something like > {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker > folder is configured. > So, it's required to have a proper way to bundle custom libs and set executor > classpath to load them up. > *Expected*: you can specify {{spark.executor.extraClassPath}} relative to > {{$SPARK_HOME}} using placeholders, e.g. with next syntax: > bq. spark.executor.extraClassPath ${home}/extra-lib/* > Code will resolve placeholders in properties with a proper path > Executor will get absolute path in {{-cp}} this way > *Actual*: you cannot specify extra libs for executor relative to > {{$SPARK_HOME}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13975) Cannot specify extra libs for executor from /extra-lib
[ https://issues.apache.org/jira/browse/SPARK-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199723#comment-15199723 ] Leonid Poliakov commented on SPARK-13975: - Do you know any other clean way to load bundled libs transparent to user? As for implementation, I will implement it and create a pull request, if it is a cool feature to have in your opinion. Might be useful for other properties. > Cannot specify extra libs for executor from /extra-lib > -- > > Key: SPARK-13975 > URL: https://issues.apache.org/jira/browse/SPARK-13975 > Project: Spark > Issue Type: Improvement > Components: Spark Submit >Affects Versions: 1.6.1 >Reporter: Leonid Poliakov >Priority: Minor > > If you build a framework on top of spark and want to bundle it with the > spark, there is no easy way to add your framework libs to executor classpath. > Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new > bundle (with my libs in it) to nodes, run the bundle. I want executors on > node to always automatically load my libs from {{/extra-lib}}, because that's > how future developers would use framework out-of-the-box. > The config doc says you can specify extraClasspath for the executor in > {{spark-defaults.conf}}, which is good because custom config may be put in > the bundle for the framework, but the syntax of the property is unclear. > You can basically specify the value that will be appended to {{-cp}} for a > executor Java process, so it follows the Java how-to-set-classpath rules, so > basically you have two options here: > 1. specify absolute path > bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/* > 2. specify relative path > bq. spark.executor.extraClassPath ../../../extra-lib/* > But none of these ways look good: absolute path won't work at all since you > cannot know where users will put the bundle, relative path looks weird > because executor will have it's work directory set to something like > {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker > folder is configured. > So, it's required to have a proper way to bundle custom libs and set executor > classpath to load them up. > *Expected*: you can specify {{spark.executor.extraClassPath}} relative to > {{$SPARK_HOME}} using placeholders, e.g. with next syntax: > bq. spark.executor.extraClassPath ${home}/extra-lib/* > Code will resolve placeholders in properties with a proper path > Executor will get absolute path in {{-cp}} this way > *Actual*: you cannot specify extra libs for executor relative to > {{$SPARK_HOME}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13975) Cannot specify extra libs for executor from /extra-lib
Leonid Poliakov created SPARK-13975: --- Summary: Cannot specify extra libs for executor from /extra-lib Key: SPARK-13975 URL: https://issues.apache.org/jira/browse/SPARK-13975 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 1.6.1 Reporter: Leonid Poliakov If you build a framework on top of spark and want to bundle it with the spark, there is no easy way to add your framework libs to executor classpath. Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new bundle (with my libs in it) to nodes, run the bundle. I want executors on node to always automatically load my libs from {{/extra-lib}}, because that's how future developers would use framework out-of-the-box. The config doc says you can specify extraClasspath for the executor in {{spark-defaults.conf}}, which is good because custom config may be put in the bundle for the framework, but the syntax of the property is unclear. You can basically specify the value that will be appended to {{-cp}} for a executor Java process, so it follows the Java how-to-set-classpath rules, so basically you have two options here: 1. specify absolute path bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/* 2. specify relative path bq. spark.executor.extraClassPath ../../../extra-lib/* But none of these ways look good: absolute path won't work at all since you cannot know where users will put the bundle, relative path looks weird because executor will have it's work directory set to something like {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker folder is configured. So, it's required to have a proper way to bundle custom libs and set executor classpath to load them up. *Expected*: you can specify {{spark.executor.extraClassPath}} relative to {{$SPARK_HOME}} using placeholders, e.g. with next syntax: bq. spark.executor.extraClassPath ${home}/extra-lib/* Code will resolve placeholders in properties with a proper path Executor will get absolute path in {{-cp}} this way *Actual*: you cannot specify extra libs for executor relative to {{$SPARK_HOME}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org