the line of code that resulted in that error was : ch.cern.sparkmeasure.StageMetrics.saveData
it also was started while trying to save a dataframe in hbase using hbase spark connector : df1.write.options( Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog. newTable -> "4")) .format("org.apache.hadoop.hbase.spark") .save() the full stack trace is as follows: Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry; at org.apache.spark.sql.DataFrameWriter.saveToV1Source( DataFrameWriter.scala:277) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229) at ch.cern.sparkmeasure.StageMetrics.saveData(stagemetrics.scala:297) ........... ........... at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach( TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) Le mer. 13 nov. 2019 à 16:05, Femi Anthony <femib...@gmail.com> a écrit : > Can you post the line of code that’s resulting in that error along with > the stack trace ? > > Sent from my iPhone > > On Nov 13, 2019, at 9:53 AM, asma zgolli <zgollia...@gmail.com> wrote: > > > > Hello , > > > I'm using spark 2.4.4 and i keep receiving this error message. Can you > please help me identify the problem? > > > thank you , > > yours sincerely > Asma ZGOLLI > > PhD student in data engineering - computer science > > PJ: > > > > "main" java.lang.NoSuchMethodError: > org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry; > > at org.apache.spark.sql.DataFrameWriter.saveToV1Source( > DataFrameWriter.scala:277) > > > > > > -- Asma ZGOLLI PhD student in data engineering - computer science