[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Github user olegz closed the pull request at: https://github.com/apache/spark/pull/2422 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Github user olegz commented on the pull request: https://github.com/apache/spark/pull/2422#issuecomment-56437906 @mridulm I do agree with Experimental so I'll change As for the rest I am working on it so stay tuned --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2422#issuecomment-56373277 Is there an example of how this is going to be leveraged ? The default case is the simple version delegating to existing spark - would be good to see how this is used against tez - so that there is a clearer picture of how the design changes fit into overall changes proposed : and what further changes (if any) might be required in case we need to generalize the interface. As @pwendell mentioned, currently this aspect is not exposed - and if we want to expose it via an spi, we need to have better understanding of how it will be leveraged. Also, would Experimental make more sense than DeveloperApi (until it stabilizes) ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2422#issuecomment-55848794 I commented on the JIRA - but this would benefit from a more complete design proposal on JIRA. Traditionally the SparkContext API has not been a point of pluggabilty. Instead we have a pluggable resource management subsystem and also pluggability in other areas such as the shuffle. So it would be helpful to understand the end-to-end proposal here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2422#issuecomment-55843559 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3561 - Pluggable strategy to facilitate ...
GitHub user olegz opened a pull request: https://github.com/apache/spark/pull/2422 SPARK-3561 - Pluggable strategy to facilitate access to native Hadoop/YARN resources Added HadoopExecutionContext trait and its default implementation DefaultHadoopExecutionContext Modified SparkContext to instantiate and delegate to the instance of HadoopExecutionContext where appropriate Changed HadoopExecutionContext to JobExecutionContext Changed DefaultHadoopExecutionContext to DefaultExecutionContext Name changes are due to the fact that when Spark executes outside of Hadoop having Hadoop in the name woudl be confusing Added initial documentation and tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/spark-1 SPARK-HADOOP Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2422 commit 15cd9b9110026b82a191ceb4d32f38230eec3b47 Author: Oleg Zhurakousky Date: 2014-09-16T15:26:48Z Initial commit to provide pluggable strategy to facilitate access to native Hadoop resources Added HadoopExecutionContext trait and its default implementation DefaultHadoopExecutionContext Modified SparkContext to instantiate and delegate to the instance of HadoopExecutionContext where appropriate Changed HadoopExecutionContext to JobExecutionContext Changed DefaultHadoopExecutionContext to DefaultExecutionContext Name changes are due to the fact that when Spark executes outside of Hadoop having Hadoop in the name woudl be confusing Added initial documentation and tests polished scaladoc annotated JobExecutionContext with @DeveloperAPI --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
