the line of code that resulted in that error was :
ch.cern.sparkmeasure.StageMetrics.saveData
it also was started while trying to save a dataframe in hbase using hbase
spark connector :
df1.write.options(
Map(HBaseTableCatalog.tableCatalog -> catalog, HBaseTableCatalog.
newTable -> "4"))
.format("org.apache.hadoop.hbase.spark")
.save()
the full stack trace is as follows:
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(
DataFrameWriter.scala:277)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at ch.cern.sparkmeasure.StageMetrics.saveData(stagemetrics.scala:297)
...........
...........
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(
TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
Le mer. 13 nov. 2019 à 16:05, Femi Anthony <[email protected]> a écrit :
> Can you post the line of code that’s resulting in that error along with
> the stack trace ?
>
> Sent from my iPhone
>
> On Nov 13, 2019, at 9:53 AM, asma zgolli <[email protected]> wrote:
>
>
>
> Hello ,
>
>
> I'm using spark 2.4.4 and i keep receiving this error message. Can you
> please help me identify the problem?
>
>
> thank you ,
>
> yours sincerely
> Asma ZGOLLI
>
> PhD student in data engineering - computer science
>
> PJ:
>
>
>
> "main" java.lang.NoSuchMethodError:
> org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;
>
> at org.apache.spark.sql.DataFrameWriter.saveToV1Source(
> DataFrameWriter.scala:277)
>
>
>
>
>
>
--
Asma ZGOLLI
PhD student in data engineering - computer science
