Hi Vaquar,
Thanks a lot! Accepted as the answer (yet there was the other answer that
was very helpful too). Tons of reading ahead to understand it more.
That once again makes me feel that Hadoop MapReduce experience would help a
great deal (and I've got none).
Pozdrawiam,
Jacek Laskowski
ht
Hi Jecek ,
I have answered , hope you find it useful.
Regards,
Viquar khan
On Sat, Apr 3, 2021 at 11:19 AM Jacek Laskowski wrote:
> Hi,
>
> I've just posted a question on StackOverflow [1] about the safety of the
> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing
> s
Hi,
From
https://spark.apache.org/docs/3.1.1/cloud-integration.html#recommended-settings-for-writing-to-object-stores
:
> For object stores whose consistency model means that rename-based commits
are safe use the FileOutputCommitter v2 algorithm for performance; v1 for
safety.
These are "safe" a
Hi Jacek,
Can you please clarify your question?
with regard to your point:
"... I think I'm missing some fundamentals on how cloud object stores work
(GCS in particular) and hence the question."
The end result is the safe storage of data in object storage in GCP right?
HTH
view my Linke
Hi,
I've just posted a question on StackOverflow [1] about the safety of the v2
algorithm while writing out to Google Cloud Storage. I think I'm missing
some fundamentals on how cloud object stores work (GCS in particular) and
hence the question.
Is this all about File.rename and how many HTTP ca
