I do know only to some extent , I mean If you see my sample s3 locations s3a://mybucket/20180425_111447_data1/_SUCCESS
s3a://mybucket/20180424_111241_data1/_SUCCESS The only values which are static in above location are s3a://mybucket/ data1/_SUCCESS Now I want to configure tolerance for _SUCCESS file as latest or 1 day older based on this configuration it should pick the right time stamp folder which has _SUCCESS file On Mon, May 21, 2018 at 2:35 PM Joe Napolitano <joe.napolit...@wework.com> wrote: > Purna, with regards to "this path is not completely static," can you > clarify what you mean? > > Do you mean that you don't know the actual key name beforehand? E.g. > pertaining to "111447", "111241", and "111035" in your example? > > On Mon, May 21, 2018 at 2:23 PM, Brian Greene < > br...@heisenbergwoodworking.com> wrote: > > > I suggest it’ll work for your needs. > > > > Sent from a device with less than stellar autocorrect > > > > > On May 21, 2018, at 10:16 AM, purna pradeep <purna2prad...@gmail.com> > > wrote: > > > > > > Hi , > > > > > > I’m trying to evaluate airflow to see if it suits my needs. > > > > > > Basically i can have below steps in a DAG > > > > > > > > > > > > 1)Look for a file arrival on given s3 location (this path is not > > completely > > > static) (i can use S3Keysensor in this step) > > > > > > i should be able to specify to look either for latest folder or 24hrs > or > > > n number of days older folder which has _SUCCESS file as mentioned > below > > > > > > sample file location(s): > > > > > > s3a://mybucket/20180425_111447_data1/_SUCCESS > > > > > > s3a://mybucket/20180424_111241_data1/_SUCCESS > > > > > > s3a://mybucket/20180424_111035_data1/_SUCCESS > > > > > > > > > > > > 2)invoke a simple restapi using HttpSimpleOperator once the above > > > dependency is met ,i can set upstream for step2 as step1 > > > > > > > > > > > > Does S3keysensor supports step1 out of the box? > > > > > > Also in some cases i may to have a DAG without start date & end date it > > > just needs to be triggered once file is available in a given s3 > location > > > > > > > > > > > > *Please suggest !* > > >
