We have a very quick job that processes request for the entirety current hour in an idempotent way so we can get results to the user a bit quicker than every hour and then at the end of the hour run another job to make sure we didn't miss anything for the whole hour. Maybe there is a better way to accomplish the same thing. All recommendations welcomes ;-)
I can try the 5 minutes schedule interval with max_active_runs=1. Thanks for all the help. Cheers, Edgardo On Mon, Mar 13, 2017 at 5:12 PM, Alex Guziel <[email protected] > wrote: > FWIW, for our streaming jobs, we run a 5 minute schedule interval with > max_active_runs=1 > > On Mon, Mar 13, 2017 at 2:00 PM, Maxime Beauchemin < > [email protected]> wrote: > > > Airflow isn't designed to work well with short schedule intervals. The > > guarantees that we give in terms of scheduling latency are limited as the > > platform isn't optimized for that specifically. > > > > What is the type of operation that you are performing every 2 minutes? > > > > If you're doing data processing in microbatches you should look into data > > streaming solutions like Spark Streaming, Flink, Sanza, or Storm. > > > > Max > > > > On Mon, Mar 13, 2017 at 10:17 AM, Edgardo Vega <[email protected]> > > wrote: > > > > > Max, > > > > > > Sorry for so many ambiguous antecedents. > > > > > > I want to create a dag that does an operation waits 2 minutes and the > > runs > > > again over and over for all time. I don't know if that is possible to > do > > > with airflow or somehow trick airflow into doing this. > > > > > > I hope that clears things up. > > > > > > Cheers, > > > > > > Edgardo > > > > > > > > > On Mon, Mar 13, 2017 at 12:39 PM, Maxime Beauchemin < > > > [email protected]> wrote: > > > > > > > "Would this be possible in airflow?" > > > > > > > > What do you mean by "this"? > > > > > > > > Max > > > > > > > > On Mon, Mar 13, 2017 at 9:09 AM, Edgardo Vega < > [email protected]> > > > > wrote: > > > > > > > > > We are currently trying to port our current solution into airflow. > > What > > > > is > > > > > currently stumping me is we have a few tasks we have running pretty > > > much > > > > > all the time. Once it is done we wait a few minutes and kick off > > > another. > > > > > > > > > > Would this be possible in airflow? > > > > > > > > > > -- > > > > > Cheers, > > > > > > > > > > Edgardo > > > > > > > > > > > > > > > > > > > > > -- > > > Cheers, > > > > > > Edgardo > > > > > > -- Cheers, Edgardo
