Thanks All - perhaps I misread the earlier posts as dependencies with
Hadoop version, but the key is also the CDH 5.3.2 (not just Hadoop 2.5 v/s
2.4) etc.
After adding the classPath as Marcelo/Harsh suggested (loading CDH libs
front), I am able to get spark-shell started without invalid container etc
so that issue is solved.
When I run any query, it gives java.lang.NoSuchMethodError:
com.google.common.hash.HashFunction.hashInt(I)Lcom/google/common/hash/HashCode;
This seems to be Guava lib version issue that has been known ... I will
look into it.
Thanks again !
On Tue, Mar 24, 2015 at 12:50 PM, Harsh J <ha...@cloudera.com> wrote:
> My comment's still the same: Runtime-link-via-classpath Spark to use CDH
> 5.3.2 libraries, just like your cluster does, not Apache Hadoop 2.5.0
> (which CDH is merely based on, but carries several backports on top that
> aren't in Apache Hadoop 2.5.0, one of which addresses this parsing trouble).
>
> You do not require to recompile Spark, just alter its hadoop libraries in
> its classpath to be that of CDH server version (overwrite from parcels,
> etc.).
>
> On Wed, Mar 25, 2015 at 1:06 AM, Manoj Samel <manojsamelt...@gmail.com>
> wrote:
>
>> I recompiled Spark 1.3 with Hadoop 2.5; it still gives same stack trace.
>>
>> A quick browse into stacktrace with Hadoop 2.5.0
>> org.apache.hadoop.yarn.util.ConverterUtils ...
>>
>> 1. toContainerId gets parameter containerId which I assume is container_
>> *e*06_1427223073530_0001_01_000001
>> 2. It splits it using public static final Splitter _SPLITTER =
>> Splitter.on('_').trimResults();
>> 3. Line 172 checks container prefix with CONTAINER_PREFIX which is valid
>> ("container")
>> 4. It calls toApplicationAttemptId
>> 5. toApplicationAttemptId tries Long.parseLong(it.next()) on "e06" and
>> dies
>>
>> Seems like it is not expecting a non-numeric character. Is this a Yarn
>> issue ?
>>
>> Thanks,
>>
>> On Tue, Mar 24, 2015 at 8:25 AM, Manoj Samel <manoj.sa...@gmail.com>
>> wrote:
>>
>>> I'll compile Spark with Hadoop libraries and try again ...
>>>
>>> Thanks,
>>>
>>> Manoj
>>>
>>> On Mar 23, 2015, at 10:34 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>> This may happen if you are using different versions of CDH5 jars between
>>> Spark and the cluster. Can you ensure your Spark's Hadoop CDH jars match
>>> the cluster version exactly, since you seem to be using a custom version of
>>> Spark (out of CDH) here?
>>>
>>> On Tue, Mar 24, 2015 at 7:32 AM, Manoj Samel <manojsamelt...@gmail.com>
>>> wrote:
>>>
>>>> x-post to CDH list for any insight ...
>>>>
>>>> Thanks,
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Manoj Samel <manojsamelt...@gmail.com>
>>>> Date: Mon, Mar 23, 2015 at 6:32 PM
>>>> Subject: Invalid ContainerId ... Caused by:
>>>> java.lang.NumberFormatException: For input string: "e04"
>>>> To: "user@spark.apache.org" <user@spark.apache.org>
>>>>
>>>>
>>>> Spark 1.3, CDH 5.3.2, Kerberos
>>>>
>>>> Setup works fine with base configuration, spark-shell can be used in
>>>> yarn client mode etc.
>>>>
>>>> When work recovery feature is enabled via
>>>> http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_ha_yarn_work_preserving_recovery.html,
>>>> the spark-shell fails with following log
>>>>
>>>> 15/03/24 01:20:16 ERROR yarn.ApplicationMaster: Uncaught exception:
>>>> java.lang.IllegalArgumentException: Invalid ContainerId:
>>>> container_e04_1427159778706_0002_01_000001
>>>> at
>>>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>>>> at
>>>> org.apache.spark.deploy.yarn.YarnRMClient.getAttemptId(YarnRMClient.scala:93)
>>>> at
>>>> org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:83)
>>>> at
>>>> org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:576)
>>>> at
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:60)
>>>> at
>>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:59)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>>>> at
>>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:59)
>>>> at
>>>> org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:574)
>>>> at
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:597)
>>>> at
>>>> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>>>> Caused by: java.lang.NumberFormatException: For input string: "e04"
>>>> at
>>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>>> at java.lang.Long.parseLong(Long.java:589)
>>>> at java.lang.Long.parseLong(Long.java:631)
>>>> at
>>>> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>>>> at
>>>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>>>> ... 12 more
>>>> 15/03/24 01:20:16 INFO yarn.ApplicationMaster: Final app status:
>>>> FAILED, exitCode: 10, (reason: Uncaught exception: Invalid ContainerId:
>>>> container_e04_1427159778706_0002_01_000001)
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to cdh-user+unsubscr...@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to cdh-user+unsubscr...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to cdh-user+unsubscr...@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to cdh-user+unsubscr...@cloudera.org.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>
>
>
> --
> Harsh J
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscr...@cloudera.org.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
>
