Hi , I’m trying to evaluate airflow to see if it suits my needs.
Basically i can have below steps in a DAG 1)Look for a file arrival on given s3 location (this path is not completely static) (i can use S3Keysensor in this step) i should be able to specify to look either for latest folder or 24hrs or n number of days older folder which has _SUCCESS file as mentioned below sample file location(s): s3a://mybucket/20180425_111447_data1/_SUCCESS s3a://mybucket/20180424_111241_data1/_SUCCESS s3a://mybucket/20180424_111035_data1/_SUCCESS 2)invoke a simple restapi using HttpSimpleOperator once the above dependency is met ,i can set upstream for step2 as step1 Does S3keysensor supports step1 out of the box? Also in some cases i may to have a DAG without start date & end date it just needs to be triggered once file is available in a given s3 location *Please suggest !*
