The problem is that specifying two lifecycle policies for the same path, the one with the shorter retention wins :(
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex4 "You might specify an S3 Lifecycle configuration in which you specify overlapping prefixes, or actions. Generally, S3 Lifecycle optimizes for cost. For example, if two expiration policies overlap, the shorter expiration policy is honored so that data is not stored for longer than expected. Likewise, if two transition policies overlap, S3 Lifecycle transitions your objects to the lower-cost storage class." On Thu, Apr 13, 2023, 12:29 "Yuri Oleynikov (יורי אולייניקוב)" < yur...@gmail.com> wrote: > My naïve assumption that specifying lifecycle policy for _spark_metadata > with longer retention will solve the issue > > Best regards > > > On 13 Apr 2023, at 11:52, Yuval Itzchakov <yuva...@gmail.com> wrote: > > > > > > Hi everyone, > > > > I am using Sparks FileStreamSink in order to write files to S3. On the > S3 bucket, I have a lifecycle policy that deletes data older than X days > back from the bucket in order for it to not infinitely grow. My problem > starts with Spark jobs that don't have frequent data. What will happen in > this case is that new batches will not be created, which in turn means no > new checkpoints will be written to the output path and no overwrites to the > _spark_metadata file will be performed, thus eventually causing the > lifecycle policy to delete the file which causes the job to fail. > > > > As far as I can tell from reading the code and looking at StackOverflow > answers, _spark_metadata file path is hardcoded to the base path of the > output directory created by the DataStreamWriter, which means I cannot > store this file in a separate prefix which is not under the lifecycle > policy rule. > > > > Has anyone run into a similar problem? > > > > > > > > -- > > Best Regards, > > Yuval Itzchakov. >
