programatically configure aws_default, emr_default connections?

2017-10-16 Thread Frank Maritato
nd I didn’t see anything like this being done in there. Thanks! -- Frank Maritato

Re: programatically configure aws_default, emr_default connections?

2017-10-18 Thread Frank Maritato
Thanks, Andrew! This works nicely. On 10/16/17, 3:01 PM, "Andrew Chen" wrote: `airflow connections --add` might be what you're looking for. On Mon, Oct 16, 2017 at 2:58 PM, Frank Maritato wrote: > Hi, > > Is there any way to progr

Airflow cli to remote host

2018-05-23 Thread Frank Maritato
r not have to ssh into the production server to run these jobs. Thanks! -- Frank Maritato

Dynamic tasks in a dag?

2018-09-14 Thread Frank Maritato
tream(start) for file in glob('dags/snowsql/create/view/*.sql'): print("FILE {}".format(file)) task = create_snowflake_operator(file, dag, 'snowflake_default') task.set_upstream(start) print("done {}".format(start.downstream_task_ids)) Thanks in advance -- Frank Maritato

Re: [External] Re: Dynamic tasks in a dag?

2018-09-14 Thread Frank Maritato
l the operators I'm creating. I'm just not seeing them in the ui. On 9/14/18, 9:10 AM, "Sai Phanindhra" wrote: Hi frank, Can you try giving global paths? On Fri 14 Sep, 2018, 21:35 Frank Maritato, wrote: > Hi, > > I'm using

Re: [External] Re: Dynamic tasks in a dag?

2018-09-14 Thread Frank Maritato
where ever you are evoking 'airflow list_tasks', but that doesn't mean they work from wherever the webserver is parsing the dags from. Does running 'airflow list_tasks' from some other running directory work? On Fri, Sep 14, 2018 at 12:35 PM Frank

airflow ui not showing logs

2018-10-15 Thread Frank Maritato
;base_log_folder'. I don't see any errors or debug in the airflow-webserver.{out|err} logs. I tried restarting the process and it still doesn't work. Anyone know how I can track this down? -- Frank Maritato

Re: [External] RE: [IE] airflow ui not showing logs

2018-10-15 Thread Frank Maritato
s is the process that takes the log files written to disk > and serves them to the web UI. It should be running on port 8793 by default. > > Sunil > > -Original Message- > From: Frank Maritato [mailto:fmarit...@opentable.com.INVALID] > Sent: Monday, October 15, 2018

Re: [IE] Re: [External] RE: [IE] airflow ui not showing logs

2018-10-15 Thread Frank Maritato
; starts the serve_logs process. > > I've had a situation where the serve_logs process went down for some > reason and restarting the worker fixed it. It doesn't explain the cause > but could possibly resolve your issue. > > Sunil > > -Original Message- >

Re: [IE] Re: [External] RE: [IE] airflow ui not showing logs

2018-10-25 Thread Frank Maritato
;airflow >> worker' command. Try restarting your worker process and see if it also >> starts the serve_logs process. >> >> I've had a situation where the serve_logs process went down for some >> reason and restarting the worker fixed it. It doesn't e

backfilling before start_date?

2018-11-06 Thread Frank Maritato
there were no task failures and no errors in the logs. I also tried to change the start_date back to 8/1 in my dag, refreshing in the ui and then run the backfill for 8/1 and it started tasks this time, but they failed with a KeyError. Is there a proper way to do this? -- Frank Maritato