D
Replied Message
| From | Tufan Rakshit |
| Date | 07/24/2022 18:59 |
| To | Shay Elbaz |
| Cc | kineret M,
user |
| Subject | Re: [EXTERNAL] Partial data with ADLS Gen2 |
Just use Delta
Best
Tufan
Sent from my iPhone
On 24 Jul 2022, at 12:20, Shay Elbaz wrote
ial"
> location, write it to some staging directory instead. Once the job is done,
> rename the staging dir to the official location.
> From: kineret M
> Sent: Sunday, July 24, 2022 1:06 PM
> To: user@spark.apache.org
> Subject: [EXTERNAL] Partial data with ADLS Gen2
>
l location.
From: kineret M
Sent: Sunday, July 24, 2022 1:06 PM
To: user@spark.apache.org
Subject: [EXTERNAL] Partial data with ADLS Gen2
ATTENTION: This email originated from outside of GM.
I have spark batch application writing to ADLS Gen2 (hierarchy).
When
I have spark batch application writing to ADLS Gen2 (hierarchy).
When designing the application I was sure the spark would perform global
commit once the job is committed, but what it really does it commits on
each task, meaning *once task completes writing it moves from temp to
target storage*.
