Asher, I found a profile for Spark 2.11 and removed it. Now, it brings in 2.10. I ran some code and got further. Now, I get this error below when I do a “df.show”.
java.lang.AbstractMethodError at org.apache.spark.Logging$class.log(Logging.scala:50) at org.apache.spark.sql.execution.datasources.hbase.HBaseFilter$.log(HBaseFilter.scala:122) at org.apache.spark.sql.execution.datasources.hbase.HBaseFilter$.buildFilters(HBaseFilter.scala:125) at org.apache.spark.sql.execution.datasources.hbase.HBaseTableScanRDD.getPartitions(HBaseTableScan.scala:59) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) Thanks for all your help. Cheers, Ben > On Feb 3, 2017, at 8:16 AM, Asher Krim <ak...@hubspot.com> wrote: > > Did you check the actual maven dep tree? Something might be pulling in a > different version. Also, if you're seeing this locally, you might want to > check which version of the scala sdk your IDE is using > > Asher Krim > Senior Software Engineer > > > On Thu, Feb 2, 2017 at 5:43 PM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > Hi Asher, > > I modified the pom to be the same Spark (1.6.0), HBase (1.2.0), and Java > (1.8) version as our installation. The Scala (2.10.5) version is already the > same as ours. But I’m still getting the same error. Can you think of anything > else? > > Cheers, > Ben > > >> On Feb 2, 2017, at 11:06 AM, Asher Krim <ak...@hubspot.com >> <mailto:ak...@hubspot.com>> wrote: >> >> Ben, >> >> That looks like a scala version mismatch. Have you checked your dep tree? >> >> Asher Krim >> Senior Software Engineer >> >> >> On Thu, Feb 2, 2017 at 1:28 PM, Benjamin Kim <bbuil...@gmail.com >> <mailto:bbuil...@gmail.com>> wrote: >> Elek, >> >> Can you give me some sample code? I can’t get mine to work. >> >> import org.apache.spark.sql.{SQLContext, _} >> import org.apache.spark.sql.execution.datasources.hbase._ >> import org.apache.spark.{SparkConf, SparkContext} >> >> def cat = s"""{ >> |"table":{"namespace":"ben", "name":"dmp_test", >> "tableCoder":"PrimitiveType"}, >> |"rowkey":"key", >> |"columns":{ >> |"col0":{"cf":"rowkey", "col":"key", "type":"string"}, >> |"col1":{"cf":"d", "col":"google_gid", "type":"string"} >> |} >> |}""".stripMargin >> >> import sqlContext.implicits._ >> >> def withCatalog(cat: String): DataFrame = { >> sqlContext >> .read >> .options(Map(HBaseTableCatalog.tableCatalog->cat)) >> .format("org.apache.spark.sql.execution.datasources.hbase") >> .load() >> } >> >> val df = withCatalog(cat) >> df.show >> >> It gives me this error. >> >> java.lang.NoSuchMethodError: >> scala.runtime.ObjectRef.create(Ljava/lang/Object;)Lscala/runtime/ObjectRef; >> at >> org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:232) >> at >> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:77) >> at >> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) >> at >> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158) >> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119) >> >> If you can please help, I would be grateful. >> >> Cheers, >> Ben >> >> >>> On Jan 31, 2017, at 1:02 PM, Marton, Elek <h...@anzix.net >>> <mailto:h...@anzix.net>> wrote: >>> >>> >>> I tested this one with hbase 1.2.4: >>> >>> https://github.com/hortonworks-spark/shc >>> <https://github.com/hortonworks-spark/shc> >>> >>> Marton >>> >>> On 01/31/2017 09:17 PM, Benjamin Kim wrote: >>>> Does anyone know how to backport the HBase Spark module to HBase 1.2.0? I >>>> tried to build it from source, but I cannot get it to work. >>>> >>>> Thanks, >>>> Ben >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> <mailto:user-unsubscr...@spark.apache.org> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> <mailto:user-unsubscr...@spark.apache.org> >>> >> >> > >