Hi all,
I am trying out Spark 3.2.1 on k8s using Hadoop 3.3.1
Running into issues with writing to s3 bucket using
TemporaryAWSCredentialsProvider
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Using_Session_Credentials_with_TemporaryAWSCredentialsProvider
While
if I am wrong.
>>
>>
>>
>> *From:* David Morin
>> *Sent:* Tuesday, 2 February 2021 22:26
>> *To:* user@spark.apache.org
>> *Subject:* S3a Committer
>>
>>
>>
>> Hi,
>>
>>
>>
>> I have some issues at the mom
/s3/consistency/, are
> you sure this is your problem?
>
> I think all these s3guard like wrappers are irrelevant right now. Please
> correct me if I am wrong.
>
>
>
> *From:* David Morin
> *Sent:* Tuesday, 2 February 2021 22:26
> *To:* user@spark.apache.org
> *Subject:*
: S3a Committer
Hi,
I have some issues at the moment with S3 API of Openstack Swift (S3a).
This one is eventually consistent and it causes lots of issues with my
distributed jobs in Spark.
Is the S3A committer able to fix that ? Or an "S3guard like" implementation is
the only way ?
David
e:
>
>> Hi,
>>
>> I have some issues at the moment with S3 API of Openstack Swift (S3a).
>> This one is eventually consistent and it causes lots of issues with my
>> distributed jobs in Spark.
>> Is the S3A committer able to fix that ? Or an "S3guard like"
>> implementation is the only way ?
>>
>> David
>>
>
sues with my
> distributed jobs in Spark.
> Is the S3A committer able to fix that ? Or an "S3guard like"
> implementation is the only way ?
>
> David
>
Hi,
I have some issues at the moment with S3 API of Openstack Swift (S3a).
This one is eventually consistent and it causes lots of issues with my
distributed jobs in Spark.
Is the S3A committer able to fix that ? Or an "S3guard like" implementation
is the only way ?
David
