Re: S3keysonsor

Great, I think we're in agreement on your definition of static. In my own experience, working with S3 keys can be painful if you can't anticipate the key name. I don't think the S3KeySensor will work as it's written. There's another operator that's not in the docs, but can be seen below the

Re: S3keysonsor

The sensor allows wild card (*) and there is also an S3PrefixSensor which might help in some cases. In one of my dags, I have a similar structure. wait_on_s3_source_data = S3KeySensor( task_id='wait_on_s3_source_data',

Re: S3keysonsor

+ Joe On Mon, May 21, 2018 at 2:56 PM purna pradeep wrote: > I do know only to some extent , I mean If you see my sample s3 locations > > s3a://mybucket/20180425_111447_data1/_SUCCESS > > s3a://mybucket/20180424_111241_data1/_SUCCESS > > > > The only values which are

Re: S3keysonsor

I do know only to some extent , I mean If you see my sample s3 locations s3a://mybucket/20180425_111447_data1/_SUCCESS s3a://mybucket/20180424_111241_data1/_SUCCESS The only values which are static in above location are s3a://mybucket/ data1/_SUCCESS Now I want to configure tolerance for

Re: S3keysonsor

Purna, with regards to "this path is not completely static," can you clarify what you mean? Do you mean that you don't know the actual key name beforehand? E.g. pertaining to "111447", "111241", and "111035" in your example? On Mon, May 21, 2018 at 2:23 PM, Brian Greene <

Re: S3keysonsor

I suggest it’ll work for your needs. Sent from a device with less than stellar autocorrect > On May 21, 2018, at 10:16 AM, purna pradeep wrote: > > Hi , > > I’m trying to evaluate airflow to see if it suits my needs. > > Basically i can have below steps in a DAG > >