FWIW, for our streaming jobs, we run a 5 minute schedule interval with max_active_runs=1
On Mon, Mar 13, 2017 at 2:00 PM, Maxime Beauchemin < [email protected]> wrote: > Airflow isn't designed to work well with short schedule intervals. The > guarantees that we give in terms of scheduling latency are limited as the > platform isn't optimized for that specifically. > > What is the type of operation that you are performing every 2 minutes? > > If you're doing data processing in microbatches you should look into data > streaming solutions like Spark Streaming, Flink, Sanza, or Storm. > > Max > > On Mon, Mar 13, 2017 at 10:17 AM, Edgardo Vega <[email protected]> > wrote: > > > Max, > > > > Sorry for so many ambiguous antecedents. > > > > I want to create a dag that does an operation waits 2 minutes and the > runs > > again over and over for all time. I don't know if that is possible to do > > with airflow or somehow trick airflow into doing this. > > > > I hope that clears things up. > > > > Cheers, > > > > Edgardo > > > > > > On Mon, Mar 13, 2017 at 12:39 PM, Maxime Beauchemin < > > [email protected]> wrote: > > > > > "Would this be possible in airflow?" > > > > > > What do you mean by "this"? > > > > > > Max > > > > > > On Mon, Mar 13, 2017 at 9:09 AM, Edgardo Vega <[email protected]> > > > wrote: > > > > > > > We are currently trying to port our current solution into airflow. > What > > > is > > > > currently stumping me is we have a few tasks we have running pretty > > much > > > > all the time. Once it is done we wait a few minutes and kick off > > another. > > > > > > > > Would this be possible in airflow? > > > > > > > > -- > > > > Cheers, > > > > > > > > Edgardo > > > > > > > > > > > > > > > -- > > Cheers, > > > > Edgardo > > >
