Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Ted Yu
ment and used the >>> easy-to-use HBaseContext for HBase operations. Now, I want to use the >>> latest in Dataframes. Since the new functionality is only in the >>> hbase-spark module, I want to know how to get it and package it for CDH >>> 5.5, which still uses HBase 1.0

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Benjamin Kim
and package it for CDH 5.5, which still >> uses HBase 1.0.0. Can you tell me what version of hbase master is still >> backwards compatible? >> >> By the way, we are using Spark 1.6 if it matters. >> >> Thanks, >> Ben >> >>> On Feb 10, 2

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Ted Yu
; By the way, we are using Spark 1.6 if it matters. >> >> Thanks, >> Ben >> >> On Feb 10, 2016, at 2:34 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >> Have you tried adding hbase client jars to spark.executor.extraClassPath >> ? >> >>

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Benjamin Kim
t; >> Have you tried adding hbase client jars to spark.executor.extraClassPath ? >> >> Cheers >> >> On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph <prabhujose.ga...@gmail.com >> <mailto:prabhujose.ga...@gmail.com>> wrote: >> + Spark-Dev >

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Benjamin Kim
;> >> Cheers >> >> On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph <prabhujose.ga...@gmail.com >> <mailto:prabhujose.ga...@gmail.com>> wrote: >> + Spark-Dev >> >> For a Spark job on YARN accessing hbase table, added all hbase client jars >> into

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Ted Yu
M, Prabhu Joseph < > prabhujose.ga...@gmail.com> wrote: > >> + Spark-Dev >> >> For a Spark job on YARN accessing hbase table, added all hbase client >> jars into spark.yarn.dist.files, NodeManager when launching container i.e >> executor, does localization and

Re: Spark Job on YARN accessing Hbase Table

2016-03-13 Thread Benjamin Kim
Have you tried adding hbase client jars to spark.executor.extraClassPath ? > > Cheers > > On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph <prabhujose.ga...@gmail.com > <mailto:prabhujose.ga...@gmail.com>> wrote: > + Spark-Dev > > For a Spark job on YARN

Re: Spark Job on YARN accessing Hbase Table

2016-02-10 Thread Ted Yu
Have you tried adding hbase client jars to spark.executor.extraClassPath ? Cheers On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph <prabhujose.ga...@gmail.com> wrote: > + Spark-Dev > > For a Spark job on YARN accessing hbase table, added all hbase client jars > into spa

Re: Spark Job on YARN accessing Hbase Table

2016-02-10 Thread Prabhu Joseph
hih...@gmail.com> wrote: > Have you tried adding hbase client jars to spark.executor.extraClassPath ? > > Cheers > > On Wed, Feb 10, 2016 at 12:17 AM, Prabhu Joseph < > prabhujose.ga...@gmail.com> wrote: > >> + Spark-Dev >> >> For a Spark job on YARN ac

Re: Spark Job on YARN accessing Hbase Table

2016-02-10 Thread Prabhu Joseph
+ Spark-Dev For a Spark job on YARN accessing hbase table, added all hbase client jars into spark.yarn.dist.files, NodeManager when launching container i.e executor, does localization and brings all hbase-client jars into executor CWD, but still the executor tasks fail with ClassNotFoundException