My guess would be this is a Spark Version mismatch. The option is added https://github.com/apache/spark/commit/df9a50637e2622a15e9af7d837986a0e868878b1
https://issues.apache.org/jira/browse/SPARK-27453 In 2.4.2 I would make sure your Spark installs are all 2.4.4 and that your code is compiled against 2.4.4 spark libs. For example if one node had 2.4.1 or 2.4.0 you would end up with an error. On Wed, Nov 13, 2019 at 9:05 AM Femi Anthony <femib...@gmail.com> wrote: > Can you post the line of code that’s resulting in that error along with > the stack trace ? > > Sent from my iPhone > > On Nov 13, 2019, at 9:53 AM, asma zgolli <zgollia...@gmail.com> wrote: > > > > Hello , > > > I'm using spark 2.4.4 and i keep receiving this error message. Can you > please help me identify the problem? > > > thank you , > > yours sincerely > Asma ZGOLLI > > PhD student in data engineering - computer science > > PJ: > > > > "main" java.lang.NoSuchMethodError: > org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry; > > at org.apache.spark.sql.DataFrameWriter.saveToV1Source( > DataFrameWriter.scala:277) > > > > > >
