[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-11-06 Thread Oliver Bye (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200046#comment-14200046
 ] 

Oliver Bye edited comment on SPARK-2815 at 11/6/14 10:11 AM:
-

This is still and issue for CDH4.6

I can confirm https://github.com/apache/spark/pull/151/files does fix this 
issue. Specifically the patch to 
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala

I've applied 2 line change change based on v1.1.0 and made it available here.

{noformat}
git clone https://github.com/olibye/spark.git --branch SPARK-2815 
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install
{noformat}
I've not created a pull request, as this appears to be an unsupported branch, 
and would just be a repeat.


was (Author: olibye):
This is still and issue for CDH4.6

I can confirm https://github.com/apache/spark/pull/151/files does fix this 
issue. Specifically the patch to 
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala

I've applied 2 line change change based on v1.1.0 and made it available here.

git clone https://github.com/olibye/spark.git --branch SPARK-2815 
--single-branch
mvn -T 1C -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.6.0 -DskipTests install

I've not created a pull request, as this appears to be an unsupported branch, 
and would just be a repeat.

 Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
 -

 Key: SPARK-2815
 URL: https://issues.apache.org/jira/browse/SPARK-2815
 Project: Spark
  Issue Type: Bug
  Components: Build
Affects Versions: 1.1.0
Reporter: pengyanhong
Assignee: Guoqiang Li

 compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
 SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
 (yarn-stable/compile:compile) Compilation failed, the following is the detail 
 error on console:
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.YarnClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
  not found: value YarnClient
 [error]   val yarnClient = YarnClient.createYarnClient
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
  object util is not a member of package org.apache.hadoop.yarn.webapp
 [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
  value RM_AM_MAX_ATTEMPTS is not a member of object 
 org.apache.hadoop.yarn.conf.YarnConfiguration
 [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
 YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
 [error]   ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
  not found: type AMRMClient
 [error]   private var amClient: AMRMClient[ContainerRequest] = _
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
  not found: value AMRMClient
 [error] amClient = AMRMClient.createAMRMClient()
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
  not found: value WebAppUtils
 [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error]

[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-08-04 Thread Patrick Wendell (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084407#comment-14084407
 ] 

Patrick Wendell edited comment on SPARK-2815 at 8/4/14 7:50 AM:


This may just be a Won't Fix. I think it's the same issue as here:

https://github.com/apache/spark/pull/151/files


was (Author: pwendell):
This may just be a Won't Fix. I thin it's the same issue as here:

https://github.com/apache/spark/pull/151/files

 Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
 -

 Key: SPARK-2815
 URL: https://issues.apache.org/jira/browse/SPARK-2815
 Project: Spark
  Issue Type: Bug
  Components: Build
Affects Versions: 1.1.0
Reporter: pengyanhong
Assignee: Guoqiang Li

 compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
 SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
 (yarn-stable/compile:compile) Compilation failed, the following is the detail 
 error on console:
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.YarnClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
  not found: value YarnClient
 [error]   val yarnClient = YarnClient.createYarnClient
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
  object util is not a member of package org.apache.hadoop.yarn.webapp
 [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
  value RM_AM_MAX_ATTEMPTS is not a member of object 
 org.apache.hadoop.yarn.conf.YarnConfiguration
 [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
 YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
 [error]   ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
  not found: type AMRMClient
 [error]   private var amClient: AMRMClient[ContainerRequest] = _
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
  not found: value AMRMClient
 [error] amClient = AMRMClient.createAMRMClient()
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
  not found: value WebAppUtils
 [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:410:
  value CONTAINER_ID is not a member of object

[jira] [Comment Edited] (SPARK-2815) Compilation failed upon the hadoop version 2.0.0-cdh4.5.0

2014-08-03 Thread Guoqiang Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14084006#comment-14084006
 ] 

Guoqiang Li edited comment on SPARK-2815 at 8/3/14 3:10 PM:


[~pengyanhong] You can try this {{./sbt/sbt clean assembly -Pyarn-alpha -Phive 
-Dhadoop.version=2.0.0-cdh4.5.0}}



was (Author: gq):
[~pengyanhong] You can try this first {{./sbt/sbt clean assembly -Pyarn-alpha 
-Phive -Dhadoop.version=2.0.0-cdh4.5.0}}


 Compilation failed upon the hadoop version 2.0.0-cdh4.5.0
 -

 Key: SPARK-2815
 URL: https://issues.apache.org/jira/browse/SPARK-2815
 Project: Spark
  Issue Type: Bug
  Components: Build
Affects Versions: 1.1.0
Reporter: pengyanhong
Assignee: Guoqiang Li
Priority: Blocker

 compile fail via SPARK_HADOOP_VERSION=2.0.0-cdh4.5.0 SPARK_YARN=true 
 SPARK_HIVE=true sbt/sbt assembly,  finally get error message : [error] 
 (yarn-stable/compile:compile) Compilation failed, the following is the detail 
 error on console:
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.YarnClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
  not found: value YarnClient
 [error]   val yarnClient = YarnClient.createYarnClient
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
  object util is not a member of package org.apache.hadoop.yarn.webapp
 [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:64:
  value RM_AM_MAX_ATTEMPTS is not a member of object 
 org.apache.hadoop.yarn.conf.YarnConfiguration
 [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS, 
 YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
 [error]   ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:66:
  not found: type AMRMClient
 [error]   private var amClient: AMRMClient[ContainerRequest] = _
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:92:
  not found: value AMRMClient
 [error] amClient = AMRMClient.createAMRMClient()
 [error]^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:137:
  not found: value WebAppUtils
 [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
  object api is not a member of package org.apache.hadoop.yarn.client
 [error] import org.apache.hadoop.yarn.client.api.AMRMClient
 [error]  ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
  not found: type AMRMClient
 [error]   amClient: AMRMClient[ContainerRequest],
 [error] ^
 [error] 
 /Users/pengyanhong/git/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:410:
  value CONTAINER_ID is not a member of object