I am trying to template a bash command and I need to have quotes in the
command so
bash_command= 'command opts="startTime={{ts}}
endTime={{ (execution_date + macros.timedelta(hours=1)).isoformat() }}"'
but jinja treats quotes as a literals.
Any workaround to this problem, that this noob hasn't fi
I'm going to deploy this to staging now. Fab work Bolke!
-s
On Mon, Mar 13, 2017 at 2:16 PM, Dan Davydov wrote:
> I'll test this on staging as soon as I get a chance (the testing is
> non-blocking on the rc5). Bolke very much in particular :).
>
> On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin
We have a very quick job that processes request for the entirety current
hour in an idempotent way so we can get results to the user a bit quicker
than every hour and then at the end of the hour run another job to make
sure we didn't miss anything for the whole hour. Maybe there is a better
way to
I'll test this on staging as soon as I get a chance (the testing is
non-blocking on the rc5). Bolke very much in particular :).
On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin wrote:
> +1 (binding) extremely impressed by the work and diligence all contributors
> have put in to getting these blo
FWIW, for our streaming jobs, we run a 5 minute schedule interval with
max_active_runs=1
On Mon, Mar 13, 2017 at 2:00 PM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:
> Airflow isn't designed to work well with short schedule intervals. The
> guarantees that we give in terms of schedulin
Airflow isn't designed to work well with short schedule intervals. The
guarantees that we give in terms of scheduling latency are limited as the
platform isn't optimized for that specifically.
What is the type of operation that you are performing every 2 minutes?
If you're doing data processing i
Hello!
I'm just getting started with airflow, and I'm trying to spin it up on a
fresh Ubuntu 14.04 VM. I install python and pip, set AIRFLOW_HOME, and then
run "pip install airflow" as stated here:
https://pythonhosted.org/airflow/start.html
When I do that, I get a long stream of messages that st
+1 (binding) extremely impressed by the work and diligence all contributors
have put in to getting these blockers fixed, Bolke in particular.
On Mon, Mar 13, 2017 at 1:07 AM Arthur Wiedmer wrote:
> +1 (binding)
>
> Thanks again for steering us through Bolke.
>
> Best,
> Arthur
>
> On Sun, Mar 12
Max,
Sorry for so many ambiguous antecedents.
I want to create a dag that does an operation waits 2 minutes and the runs
again over and over for all time. I don't know if that is possible to do
with airflow or somehow trick airflow into doing this.
I hope that clears things up.
Cheers,
Edgardo
"Would this be possible in airflow?"
What do you mean by "this"?
Max
On Mon, Mar 13, 2017 at 9:09 AM, Edgardo Vega
wrote:
> We are currently trying to port our current solution into airflow. What is
> currently stumping me is we have a few tasks we have running pretty much
> all the time. Once
We are currently trying to port our current solution into airflow. What is
currently stumping me is we have a few tasks we have running pretty much
all the time. Once it is done we wait a few minutes and kick off another.
Would this be possible in airflow?
--
Cheers,
Edgardo
11 matches
Mail list logo
