Yes I fear you have to shade and create an uberjar
> Am 17.02.2020 um 23:27 schrieb Mich Talebzadeh <mich.talebza...@gmail.com>: > > > I stripped everything from the jar list. This is all I have > > sspark-shell --jars shc-core-1.1.1-2.1-s_2.11.jar, \ > json4s-native_2.11-3.5.3.jar, \ > json4s-jackson_2.11-3.5.3.jar, \ > hbase-client-1.2.3.jar, \ > hbase-common-1.2.3.jar > > Now I still get the same error! > > scala> val df = withCatalog(catalog) > java.lang.NoSuchMethodError: > org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; > at > org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257) > at > org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80) > at > org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) > at > org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) > at > org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) > at withCatalog(<console>:54) > > Thanks > > Disclaimer: Use it at your own risk. Any and all responsibility for any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > >> On Mon, 17 Feb 2020 at 21:37, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Dr Mich Talebzadeh >> >> LinkedIn >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >> http://talebzadehmich.wordpress.com >> >> Disclaimer: Use it at your own risk. Any and all responsibility for any >> loss, damage or destruction of data or any other property which may arise >> from relying on this email's technical content is explicitly disclaimed. The >> author will in no case be liable for any monetary damages arising from such >> loss, damage or destruction. >> >> Many thanks both. >> >> Let me check and confirm. >> >> regards, >> >> Mich >> >> >>> On Mon, 17 Feb 2020 at 21:33, Jörn Franke <jornfra...@gmail.com> wrote: >>> Is there a reason why different Scala (it seems at least 2.10/2.11) >>> versions are mixed? This never works. >>> Do you include by accident a dependency to with an old Scala version? Ie >>> the Hbase datasource maybe? >>> >>> >>>>> Am 17.02.2020 um 22:15 schrieb Mich Talebzadeh >>>>> <mich.talebza...@gmail.com>: >>>>> >>>> >>>> Thanks Muthu, >>>> >>>> >>>> I am using the following jar files for now in local mode i.e. >>>> spark-shell_local --jars ….. >>>> >>>> json4s-jackson_2.10-3.2.10.jar >>>> json4s_2.11-3.2.11.jar >>>> json4s-native_2.10-3.4.0.jar >>>> >>>> Which one is the incorrect one please/ >>>> >>>> Regards, >>>> >>>> Mich >>>> >>>> >>>> >>>> Disclaimer: Use it at your own risk. Any and all responsibility for any >>>> loss, damage or destruction of data or any other property which may arise >>>> from relying on this email's technical content is explicitly disclaimed. >>>> The author will in no case be liable for any monetary damages arising from >>>> such loss, damage or destruction. >>>> >>>> >>>> >>>>> On Mon, 17 Feb 2020 at 20:28, Muthu Jayakumar <bablo...@gmail.com> wrote: >>>>> I suspect the spark job is somehow having an incorrect (newer) version of >>>>> json4s in the classpath. json4s 3.5.3 is the utmost version that can be >>>>> used. >>>>> >>>>> Thanks, >>>>> Muthu >>>>> >>>>>> On Mon, Feb 17, 2020, 06:43 Mich Talebzadeh <mich.talebza...@gmail.com> >>>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> Spark version 2.4.3 >>>>>> Hbase 1.2.7 >>>>>> >>>>>> Data is stored in Hbase as Json. example of a row shown below >>>>>> <image.png> >>>>>> >>>>>> I am trying to read this table in Spark Scala >>>>>> >>>>>> import org.apache.spark.sql.{SQLContext, _} >>>>>> import org.apache.spark.sql.execution.datasources.hbase._ >>>>>> import org.apache.spark.{SparkConf, SparkContext} >>>>>> import spark.sqlContext.implicits._ >>>>>> import org.json4s._ >>>>>> import org.json4s.jackson.JsonMethods._ >>>>>> import org.json4s.jackson.Serialization.{read => JsonRead} >>>>>> import org.json4s.jackson.Serialization.{read, write} >>>>>> def catalog = s"""{ >>>>>> | "table":{"namespace":"trading", "name":"MARKETDATAHBASEBATCH", >>>>>> | "rowkey":"key", >>>>>> | "columns":{ >>>>>> | "rowkey":{"cf":"rowkey", "col":"key", "type":"string"}, >>>>>> | |"ticker":{"cf":"PRICE_INFO", "col":"ticker", >>>>>> "type":"string"}, >>>>>> | |"timeissued":{"cf":"PRICE_INFO", "col":"timeissued", >>>>>> "type":"string"}, >>>>>> | |"price":{"cf":"PRICE_INFO", "col":"price", "type":"string"} >>>>>> | |} >>>>>> | |}""".stripMargin >>>>>> def withCatalog(cat: String): DataFrame = { >>>>>> spark.sqlContext >>>>>> .read >>>>>> .options(Map(HBaseTableCatalog.tableCatalog->cat)) >>>>>> .format("org.apache.spark.sql.execution.datasources.hbase") >>>>>> .load() >>>>>> } >>>>>> val df = withCatalog(catalog) >>>>>> >>>>>> >>>>>> However, I am getting this error >>>>>> >>>>>> Spark session available as 'spark'. >>>>>> java.lang.NoSuchMethodError: >>>>>> org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.<init>(HBaseRelation.scala:80) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51) >>>>>> at >>>>>> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318) >>>>>> at >>>>>> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) >>>>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) >>>>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167) >>>>>> at withCatalog(testme.scala:49) >>>>>> ... 65 elided >>>>>> >>>>>> I have Googled it but with little luck! >>>>>> >>>>>> Thanks, >>>>>> Mich >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> Disclaimer: Use it at your own risk. Any and all responsibility for any >>>>>> loss, damage or destruction of data or any other property which may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>> damages arising from such loss, damage or destruction. >>>>>>
