Re: Writing to Google Cloud Storage with v2 algorithm safe?

[Jacek Laskowski]

Hi Vaquar,

Thanks a lot! Accepted as the answer (yet there was the other answer that
was very helpful too). Tons of reading ahead to understand it more.

That once again makes me feel that Hadoop MapReduce experience would help a
great deal (and I've got none).

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Sun, Apr 4, 2021 at 7:28 AM vaquar khan  wrote:

> Hi Jecek ,
>
> I have answered , hope you find it useful.
>
> Regards,
> Viquar khan
>
> On Sat, Apr 3, 2021 at 11:19 AM Jacek Laskowski  wrote:
>
>> Hi,
>>
>> I've just posted a question on StackOverflow [1] about the safety of the
>> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing
>> some fundamentals on how cloud object stores work (GCS in particular) and
>> hence the question.
>>
>> Is this all about File.rename and how many HTTP calls are there under the
>> covers? How to know it for GCS?
>>
>> Thank you for any help you can provide. Merci beaucoup mes amis :)
>>
>> [1] https://stackoverflow.com/q/66933229/1305344
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>
>
> --
> Regards,
> Vaquar Khan
> +1 -224-436-0783
> Greater Chicago
>

Re: Writing to Google Cloud Storage with v2 algorithm safe?

[vaquar khan]

Hi Jecek ,

I have answered , hope you find it useful.

Regards,
Viquar khan

On Sat, Apr 3, 2021 at 11:19 AM Jacek Laskowski  wrote:

> Hi,
>
> I've just posted a question on StackOverflow [1] about the safety of the
> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing
> some fundamentals on how cloud object stores work (GCS in particular) and
> hence the question.
>
> Is this all about File.rename and how many HTTP calls are there under the
> covers? How to know it for GCS?
>
> Thank you for any help you can provide. Merci beaucoup mes amis :)
>
> [1] https://stackoverflow.com/q/66933229/1305344
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>


-- 
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago

Re: Writing to Google Cloud Storage with v2 algorithm safe?

[Jacek Laskowski]

Hi,

From
https://spark.apache.org/docs/3.1.1/cloud-integration.html#recommended-settings-for-writing-to-object-stores
:

> For object stores whose consistency model means that rename-based commits
are safe use the FileOutputCommitter v2 algorithm for performance; v1 for
safety.

These are "safe" and "safety" meanings.

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski




On Sat, Apr 3, 2021 at 7:49 PM Mich Talebzadeh 
wrote:

> Hi Jacek,
>
> Can you please clarify your question?
>
> with regard to your point:
>
> "... I think I'm missing some fundamentals on how cloud object stores work
> (GCS in particular) and hence the question."
>
> The end result is the safe storage of data in object storage in GCP right?
>
> HTH
>
>
>
>
>view my Linkedin profile
> 
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 3 Apr 2021 at 17:13, Jacek Laskowski  wrote:
>
>> Hi,
>>
>> I've just posted a question on StackOverflow [1] about the safety of the
>> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing
>> some fundamentals on how cloud object stores work (GCS in particular) and
>> hence the question.
>>
>> Is this all about File.rename and how many HTTP calls are there under the
>> covers? How to know it for GCS?
>>
>> Thank you for any help you can provide. Merci beaucoup mes amis :)
>>
>> [1] https://stackoverflow.com/q/66933229/1305344
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books 
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> 
>>
>

Re: Writing to Google Cloud Storage with v2 algorithm safe?

[Mich Talebzadeh]

Hi Jacek,

Can you please clarify your question?

with regard to your point:

"... I think I'm missing some fundamentals on how cloud object stores work
(GCS in particular) and hence the question."

The end result is the safe storage of data in object storage in GCP right?

HTH




   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Sat, 3 Apr 2021 at 17:13, Jacek Laskowski  wrote:

> Hi,
>
> I've just posted a question on StackOverflow [1] about the safety of the
> v2 algorithm while writing out to Google Cloud Storage. I think I'm missing
> some fundamentals on how cloud object stores work (GCS in particular) and
> hence the question.
>
> Is this all about File.rename and how many HTTP calls are there under the
> covers? How to know it for GCS?
>
> Thank you for any help you can provide. Merci beaucoup mes amis :)
>
> [1] https://stackoverflow.com/q/66933229/1305344
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books 
> Follow me on https://twitter.com/jaceklaskowski
>
> 
>

Writing to Google Cloud Storage with v2 algorithm safe?

[Jacek Laskowski]

Hi,

I've just posted a question on StackOverflow [1] about the safety of the v2
algorithm while writing out to Google Cloud Storage. I think I'm missing
some fundamentals on how cloud object stores work (GCS in particular) and
hence the question.

Is this all about File.rename and how many HTTP calls are there under the
covers? How to know it for GCS?

Thank you for any help you can provide. Merci beaucoup mes amis :)

[1] https://stackoverflow.com/q/66933229/1305344

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski