| Yeah but can’t you use following? 1 . For data files: My/path/part- 2. For partitioned data: my/path/partition=
Best regards On 13 Apr 2023, at 12:58, Yuval Itzchakov <yuva...@gmail.com> wrote:
The problem is that specifying two lifecycle policies for the same path, the one with the shorter retention wins :(
"You might specify an S3 Lifecycle configuration in which you specify overlapping prefixes, or actions.
Generally, S3 Lifecycle optimizes for cost. For example, if two expiration policies overlap, the shorter expiration policy is honored so that data is not stored for longer than expected. Likewise, if two transition policies overlap, S3 Lifecycle transitions your objects to the lower-cost storage class."
On Thu, Apr 13, 2023, 12:29 "Yuri Oleynikov (יורי אולייניקוב)" < yur...@gmail.com> wrote: My naïve assumption that specifying lifecycle policy for _spark_metadata with longer retention will solve the issue
Best regards
> On 13 Apr 2023, at 11:52, Yuval Itzchakov <yuva...@gmail.com> wrote:
>
>
> Hi everyone,
>
> I am using Sparks FileStreamSink in order to write files to S3. On the S3 bucket, I have a lifecycle policy that deletes data older than X days back from the bucket in order for it to not infinitely grow. My problem starts with Spark jobs that don't have frequent data. What will happen in this case is that new batches will not be created, which in turn means no new checkpoints will be written to the output path and no overwrites to the _spark_metadata file will be performed, thus eventually causing the lifecycle policy to delete the file which causes the job to fail.
>
> As far as I can tell from reading the code and looking at StackOverflow answers, _spark_metadata file path is hardcoded to the base path of the output directory created by the DataStreamWriter, which means I cannot store this file in a separate prefix which is not under the lifecycle policy rule.
>
> Has anyone run into a similar problem?
>
>
>
> --
> Best Regards,
> Yuval Itzchakov.
|
