[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-11-06 Thread Oliver Bye (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200046#comment-14200046
 ] 

Oliver Bye edited comment on SPARK-2815 at 11/6/14 10:11 AM:
-

This is still and issue for CDH4.6

I can confirm https://github.com/apache/spark/pull/151/files does fix this 
issue. Specifically the patch to 
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala

I've applied 2 line change change based on v1.1.0 and made it available here.

{noformat}
git clone https://github.com/olibye/spark.git --branch SPARK-2815 
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install
{noformat}
I've not created a pull request, as this appears to be an unsupported branch, 
and would just be a repeat.


was (Author: olibye):
This is still and issue for CDH4.6

I can confirm https://github.com/apache/spark/pull/151/files does fix this 
issue. Specifically the patch to 
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala

I've applied 2 line change change based on v1.1.0 and made it available here.

git clone https://github.com/olibye/spark.git --branch SPARK-2815 
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install

I've not created a pull request, as this appears to be an unsupported branch, 
and would just be a repeat.

> Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
> -
>
> Key: SPARK-2815
> URL: https://issues.apache.org/jira/browse/SPARK-2815
> Project: Spark
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.1.0
>Reporter: pengyanhong
>Assignee: Guoqiang Li
>
> compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
> SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
> (yarn-stable/compile:compile) Compilation failed, the following is the detail 
> error on console:
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>  not found: value YarnClient
> [error]   val yarnClient = YarnClient.createYarnClient
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>  object util is not a member of package org.apache.hadoop.yarn.webapp
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
>  value RM_AM_MAX_ATTEMPTS is not a member of object 
> org.apache.hadoop.yarn.conf.YarnConfiguration
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> [error]   ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
>  not found: type AMRMClient
> [error]   private var amClient: AMRMClient[ContainerRequest] = _
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
>  not found: value AMRMClient
> [error] amClient = AMRMClient.createAMRMClient()
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
>  not found: value WebAppUtils
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMR

[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-08-04 Thread Patrick Wendell (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084407#comment-14084407
 ] 

Patrick Wendell edited comment on SPARK-2815 at 8/4/14 7:50 AM:


This may just be a "Won't Fix". I think it's the same issue as here:

https://github.com/apache/spark/pull/151/files


was (Author: pwendell):
This may just be a "Won't Fix". I thin it's the same issue as here:

https://github.com/apache/spark/pull/151/files

> Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
> -
>
> Key: SPARK-2815
> URL: https://issues.apache.org/jira/browse/SPARK-2815
> Project: Spark
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.1.0
>Reporter: pengyanhong
>Assignee: Guoqiang Li
>
> compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
> SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
> (yarn-stable/compile:compile) Compilation failed, the following is the detail 
> error on console:
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>  not found: value YarnClient
> [error]   val yarnClient = YarnClient.createYarnClient
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>  object util is not a member of package org.apache.hadoop.yarn.webapp
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
>  value RM_AM_MAX_ATTEMPTS is not a member of object 
> org.apache.hadoop.yarn.conf.YarnConfiguration
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> [error]   ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
>  not found: type AMRMClient
> [error]   private var amClient: AMRMClient[ContainerRequest] = _
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
>  not found: value AMRMClient
> [error] amClient = AMRMClient.createAMRMClient()
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
>  not found: value WebAppUtils
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:410:
>  value CO

[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-08-03 Thread Guoqiang Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084006#comment-14084006
 ] 

Guoqiang Li edited comment on SPARK-2815 at 8/3/14 3:10 PM:


[~pengyanhong] You can try this {{./sbt/sbt clean assembly -Pyarn-alpha -Phive 
-Dhadoop.version=2.0.0-cdh4.5.0}}



was (Author: gq):
[~pengyanhong] You can try this first {{./sbt/sbt clean assembly -Pyarn-alpha 
-Phive -Dhadoop.version=2.0.0-cdh4.5.0}}


> Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
> -
>
> Key: SPARK-2815
> URL: https://issues.apache.org/jira/browse/SPARK-2815
> Project: Spark
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.1.0
>Reporter: pengyanhong
>Assignee: Guoqiang Li
>Priority: Blocker
>
> compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
> SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
> (yarn-stable/compile:compile) Compilation failed, the following is the detail 
> error on console:
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>  not found: value YarnClient
> [error]   val yarnClient = YarnClient.createYarnClient
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>  object util is not a member of package org.apache.hadoop.yarn.webapp
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
>  value RM_AM_MAX_ATTEMPTS is not a member of object 
> org.apache.hadoop.yarn.conf.YarnConfiguration
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> [error]   ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
>  not found: type AMRMClient
> [error]   private var amClient: AMRMClient[ContainerRequest] = _
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
>  not found: value AMRMClient
> [error] amClient = AMRMClient.createAMRMClient()
> [error]^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
>  not found: value WebAppUtils
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>  object api is not a member of package org.apache.hadoop.yarn.client
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> [error]  ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>  not found: type AMRMClient
> [error]   amClient: AMRMClient[ContainerRequest],
> [error] ^
> [error] 
> /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.sc