Hello all, we have a hadoop cluster (using yarn) using s3 as filesystem with s3guard is enabled. We are using hadoop 3.2.1 with spark 2.4.5.
When I try to save a dataframe in parquet format, I get the following exception: java.lang.ClassNotFoundException: com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol My relevant spark configurations are as following: "hadoop.mapreduce.outputcommitter.factory.scheme.s3a":"org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory", "fs.s3a.committer.name": "magic", "fs.s3a.committer.magic.enabled": true, "fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem", While spark streaming fails with the exception above, apache beam succeeds writing parquet files. What might be the problem? Thanks in advance -- "Talkers aren’t good doers. Rest assured that we’re going there to use our hands, not our tongues." W. Shakespeare
