Hi,
We have observed similar behavior in older versions of spark. But we
were are currently using 3.3.0 where we have not seen such issues.
Which version of Spark and Hadoop are you using?
On 18/10/2022 19:48, Sandeep Vinayak wrote:
Hello Everyone,
We are recently observing an intermittent data loss in the spark with
output to GCS (google cloud storage). When there are missing rows, they
are accompanied by duplicate rows. The re-run of the job doesn't have
any duplicate or missing rows. Since it's hard to debug, we are first
trying to understand the potential theoretical root cause of this issue,
can this be a GCS specific issue where GCS might not be handling the
consistencies well? Any tips will be super helpful.
Thanks,
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
