RE: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

2021-08-20 Thread Shaw, Damian P.
filepath) that would then go through a renderer (e.g. markdown). It could be a readme, or yaml content or anything the author wanted. Collin On Fri, Aug 20, 2021 at 3:27 PM Shaw, Damian P. mailto:damian.sha...@credit-suisse.com>> wrote: FYI this is what I did on one of my past projec

RE: [DISCUSS] Adding better support for parametrized DAGs and dynamic DAGs using JSON/YAML dataformats

2021-08-20 Thread Shaw, Damian P.
FYI this is what I did on one of my past projects for Airflow. The users wanted to write their DAGs as YAML files so my “DAG file” was a Python script that read the YAML files and converted them to DAGs. It was very easy to do and worked because of the flexibility of Airflow. The one thing that

RE: AIP 39: schedule_date renamed to logical_date

2021-08-03 Thread Shaw, Damian P.
As a user of Airflow and someone who has to keep teaching it to other users how Airflow works a big +1 from me. I am able to explain (usually with some diagrams) that an execution date isn’t when it’s scheduled, and get them over the hump of the confusing name. So if the date was renamed to “sc

RE: [DISCUSSION] Should we be more explicit about SQLite using for dev only (or kill it for non-dev entirely????)

2021-07-20 Thread Shaw, Damian P.
Some thought as a user of Airflow, I wouldn’t of adopted Airflow in the first place if I couldn’t test it with sqlite. And would be the same today, accessing docker isn’t always an easy in some companies. But having a warning that sqlite is development only and much slower than other solutions

RE: MSSQL compatbility for Airflow, YAY!

2021-05-26 Thread Shaw, Damian P.
Thanks Aneesh, and all who helped! Damian (he/him) From: Jarek Potiuk Sent: Wednesday, May 26, 2021 06:55 To: dev@airflow.apache.org Subject: Re: MSSQL compatbility for Airflow, YAY! I was just helping a bit :) and Kaxil, and Ash and others :) But Aneesh did all the hard work :) J. On Wed

RE: Apache Airflow 2.0.0 is released!

2020-12-17 Thread Shaw, Damian P.
Great news! Is there a single web page that highlights these major features as you’ve listed them? Damian From: Ash Berlin-Taylor Sent: Thursday, December 17, 2020 12:36 To: us...@airflow.apache.org Cc: annou...@apache.org; dev@airflow.apache.org Subject: Apache Airflow 2.0.0 is released! I am

RE: Default/supported Python versions for Airlfow 2.0

2020-11-20 Thread Shaw, Damian P.
Typo, Should of read: Python 3.9 will lose bugfix support approx. 6 months later on ~2022-04-05. Damian (he/him) From: Shaw, Damian P. (WTIZ 53) Sent: Friday, November 20, 2020 08:58 To: dev@airflow.apache.org Subject: RE: Default/supported Python versions for Airlfow 2.0 Hi Jarek, Took me

RE: Default/supported Python versions for Airlfow 2.0

2020-11-20 Thread Shaw, Damian P.
s have similar thoughts. Such agreement/policy would require formal voting though I think? WDYT everyone? J. On Thu, Nov 12, 2020 at 3:37 PM Shaw, Damian P. mailto:damian.sha...@credit-suisse.com>> wrote: I just wanted to add that if people are not aware PEP 0602<https://www.python.

RE: Default/supported Python versions for Airlfow 2.0

2020-11-12 Thread Shaw, Damian P.
I just wanted to add that if people are not aware PEP 0602 has been accepted and implemented for Python 3.9. This means 3 things for the Python release cycle: 1. A new version every 12 months 2. Each version receives 18 months of full suppor

RE: [VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

2020-06-19 Thread Shaw, Damian P.
Also +1 (non-binding) on the AIP but questions on the implementation. How would a user enable their own smart sensors? I don’t see any added documentation for this. It looks like they need to manually add the name of the class to the airflow configuration and do *something* to their sensor class

RE: Airflow Dev on Windows using WSL2

2020-05-28 Thread Shaw, Damian P.
onderful! Thank you. We already have some description of using dev > env in WSL > https://github.com/apache/airflow/blob/master/BREEZE.rst#docker-in-wsl > -> > So maybe that's a good chance to review it and make some PR with > clarifications there :)? > > J. > > On Th

Airflow Dev on Windows using WSL2

2020-05-28 Thread Shaw, Damian P.
Hi all, I posted this earlier on Slack but I thought I'd also include it to the mailing list. Windows 10 2004 Stable just got released and it supports WSL2 meaning no need to use a preview ring anymore, and the latest version of Docker Desktop now supports Windows Home with WSL2 so no need to

RE: Support for multiple cron expressions

2020-05-27 Thread Shaw, Damian P.
Big +1 to anything that extends the limitations of Airflow's current scheduling capability. For me the only drawback of this is it doesn't go far enough and further additions would needed to be added later, it would still be difficult to express things that require updatable calendars like "Ev

RE: Setting to add choice of schedule at end or schedule at start of interval

2020-05-11 Thread Shaw, Damian P.
I strongly agree with reducing Airflows complexity and the plan to serialize DAGs by default makes a lot of sense. So wouldn't it make sense to also serialize the schedule of a DAG? e.g. Provide a table of execution dates that the scheduler can keep upcoming ones cached and periodically check

RE: What's coming in Airflow 2.0 - NYC online meetup next week

2020-05-06 Thread Shaw, Damian P.
I think there's a small typo, it should be Wednesday 13th May? Damian -Original Message- From: Jarek Potiuk Sent: Wednesday, May 6, 2020 16:22 To: dev@airflow.apache.org Subject: What's coming in Airflow 2.0 - NYC online meetup next week Next week on Wednesday 23th of May, Noon EST, si

RE: API spec questions

2020-04-09 Thread Shaw, Damian P.
FYI if you look back at the thread "Re: [2.0 spring cleaning] Require unique conn_id" on 2019-04-14 you can see a message from Kevin Yang stating that this random choice of connections is a "feature" used to load balance connections in AirBnB. So users are relying on this behavior. -Origi

RE: [DISCUSS] Stop using Jira (since we aren't using it properly)

2020-03-16 Thread Shaw, Damian P.
I would just like to add some extra positive thoughts for this. Firstly as a newcomer JIRA is confusing, even coming from a word that does use JIRA internally it's not what you see for most open source projects so it's far more familiar to use GitHub issues. (actually as a side note one negati

RE: task failed without running

2020-02-26 Thread Shaw, Damian P.
s/lib/python3.6/subprocess.py", line 311, in check_call    raise CalledProcessError(retcode, cmd) Apparently Celery executor ran the worker in subprocess.py and returned non-zero retcode. Heng On Wednesday, February 26, 2020, 09:57:29 AM EST, Shaw, Damian P. wrote: FYI, I've

RE: task failed without running

2020-02-26 Thread Shaw, Damian P.
You should look in airflow-worker.out for an exception that's happening at about the same time. Damian -Original Message- From: heng gu Sent: Wednesday, February 26, 2020 11:55 To: dev@airflow.apache.org Subject: Re: task failed without running I got an advice from slack to check a

RE: task failed without running

2020-02-26 Thread Shaw, Damian P.
FYI, I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout

RE: How is DAG's default_arg parameter forwarded to operators ?

2020-02-19 Thread Shaw, Damian P.
ay also be interested in this PR: https://github.com/apache/airflow/pull/7450 Best regards, Kamil On Wed, Feb 19, 2020 at 4:57 PM Shaw, Damian P. wrote: > > I'm not an expert of the Airflow code, but in 1.10.2 I notice the decorator > on the __init__ of the BaseOperator named "apply_de

RE: How is DAG's default_arg parameter forwarded to operators ?

2020-02-19 Thread Shaw, Damian P.
I'm not an expert of the Airflow code, but in 1.10.2 I notice the decorator on the __init__ of the BaseOperator named "apply_defaults": https://github.com/apache/airflow/blob/1.10.2/airflow/models.py#L2472 Which is located here: https://github.com/apache/airflow/blob/1.10.2/airflow/utils/decorato

RE: Trigger based dag run

2020-02-14 Thread Shaw, Damian P.
A possible (hacky?) workaround would be: 1. Create a DAG that is scheduled to run every 1 minute with max_active_runs=2 and catchup=False 2. As the first task in the DAG set as the sensor which needs wait on when the DAG in Airflow should be triggered and set task_concurrency to 1 3. As the sec

RE: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-13 Thread Shaw, Damian P.
Here are a couple more examples of where as an Airflow user I am introducing state in to my tasks (so they technically breaks idempotency): * Authentication mechanisms that return errors based on wrong details, state is kept so retries don't cause multiple login attempts which would cause the ac

RE: Remove Dask Executor in Airflow 2.0 ?

2020-01-13 Thread Shaw, Damian P.
FYI I used Dash instead of Local Executor when first starting Airflow, it was a great way to make sure the Executor and Scheduler weren’t tied to each other with no difficulty in set-up. But once I actually started deploying to multiple boxes I needed queue names pretty quickly. So not going to

RE: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-10 Thread Shaw, Damian P.
hen only attempt to download the ones that are missing. Chris On Fri, Jan 10, 2020 at 12:52 PM Jarek Potiuk wrote: > I wonder what others think of it. > > On Fri, Jan 10, 2020 at 6:04 PM Shaw, Damian P. < > damian.sha...@credit-suisse.com> wrote: > > > I don't be

RE: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-10 Thread Shaw, Damian P.
uidelines should people follow when writing their operators ? And it's a totally different feature that should be discussed separately. J. On Fri, Jan 10, 2020 at 5:15 PM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > I just wanted to add a related use case is task retries

RE: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-10 Thread Shaw, Damian P.
I just wanted to add a related use case is task retries, there are lots of scenarios where keeping state between the retries as well as the reschedules would be extremely helpful, so as long as whatever the solution is isn't overly narrow I'd be extremely appreciative. Damian -Original Mes

RE: [DISCUSS] Using asserts in airflow code

2019-12-03 Thread Shaw, Damian P.
Hi all, Semantic arguments aside about the meaning of assert, or the visual nicety of raise exception vs. assert, then there is one practical difference between assert vs. any other exception raise mechanism: the assert statement is filtered out when generating bytecode with the "-O" flag. Giv

RE: Pros and cons of Oracle db as Airflow SqlAlchemy backend

2019-11-29 Thread Shaw, Damian P.
Hi Mehmet, I tried using Oracle as back-end when I first worked with Airflow and quickly realized it would require significant patching of the Airflow initdb script to ever get to work. Further there are no test cases or support for any issues encountered while using Oracle. Regards Damian

Airflow Scheduler PID file no longer generated in 1.10.6?

2019-11-26 Thread Shaw, Damian P.
Hi all, Wondering if this is feature, bug, or local environment issue. I'm migrating from Airflow 1.10.3 to 1.10.6 and my start-up/shutdown scripts rely on the PID files that Airflow generates. Worker, Flower, and Webserver all generate PID files when I use the "--pid {file_name}" but Scheduler

RE: [Final review] Airflow website + need your input to the web copy

2019-11-08 Thread Shaw, Damian P.
Hi Aizhamal, I really dislike the moving background elements on the lading page. I already found the previous example when they weren't moving hard to focus on other elements such as finding the header menu and going to the documentation, for me this is now easily twice as worse. If no one else

Can an xcom push and pull to the same task instance on the same execution date?

2019-10-25 Thread Shaw, Damian P.
Hi all, I am trying to pull the latest xcom value that could include the current task from a previous retry. In the UI I can see today's xcom value and yesterday's xcom value. I am retrying the task and unable to get today's value: Attempt 1: context['ti'].xcom_pull(key=key, include_prior_dat

RE: New PMC Member: Jarek Potiuk

2019-10-18 Thread Shaw, Damian P.
Congratulations Jarek! You've been extremely to even us novices learning the ins and outs of Airflow :). Damian -Original Message- From: Kaxil Naik [mailto:kaxiln...@gmail.com] Sent: Friday, October 18, 2019 5:06 PM To: dev@airflow.apache.org Cc: Jarek Potiuk Subject: New PMC Member: Ja

RE: [PLEASE PARTICIPATE][AIP-11] UI phase of Airflow website

2019-10-03 Thread Shaw, Damian P.
Oct 3, 2019 at 6:04 PM Shaw, Damian P. wrote: > > Hi, > > I don't have access to slack so I can't see this " CSS-animated elements > floating" but please do not add animated background elements on the page, at > least for accessibility. Looking at the UI yo

RE: [PLEASE PARTICIPATE][AIP-11] UI phase of Airflow website

2019-10-03 Thread Shaw, Damian P.
Hi, I don't have access to slack so I can't see this " CSS-animated elements floating" but please do not add animated background elements on the page, at least for accessibility. Looking at the UI you present my eyes struggle to focus on the "install" button, or any actual UI, for all the backg

RE: How to manage Airflow SIGTERM excetion catches?

2019-10-03 Thread Shaw, Damian P.
Hi Alex, Maybe I'm stating the obvious but if any executors catches Airflow exceptions to do cleanup before exiting they can still do that with SystemExit. E.g.: class AirflowSystemExit(SystemExit): pass try: raise AirflowSystemExit except AirflowSystemExit: print('cleanup for airfl

How to manage Airflow SIGTERM excetion catches?

2019-10-02 Thread Shaw, Damian P.
Today I had an issue where my operator caught the SIGTERM exception that Airflow throws. It seems to me that Airflow shouldn't be throwing a SIGTERM exception that subclasses Exception, particular as in arbitrary Python functions this might happen commonly. The code in my operator looked like t

RE: [PLEASE PARTICIPATE][AIP-11] UX phase of Airflow website

2019-09-18 Thread Shaw, Damian P.
This is a minor nitpick that I wouldn't mention being a prototype site except I do see it on live websites from time to time. On a Desktop when the vertical width is less than 1450px the you have to start to use the horizontal scroll bar to browse the whole website. I am commonly on a 1080p scr

RE: DAG "Schedule Filter Callback"?

2019-08-30 Thread Shaw, Damian P.
orted) but could optionally be something more flexible and dynamic? Instead of adding this parameter... On Fri, Aug 30, 2019 at 12:33 PM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > > You are correct the callback and dag would not use prev_ or next_ as these > a

RE: DAG "Schedule Filter Callback"?

2019-08-30 Thread Shaw, Damian P.
run. Anyway just offering it up in case this approach was not on your radar. On Fri, Aug 30, 2019 at 11:54 AM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > I believe TriggerDagRunOperator solves neither 1 or 2. > > For 1) The "depends_on_past" logic seems tenu

RE: DAG "Schedule Filter Callback"?

2019-08-30 Thread Shaw, Damian P.
conf that specifies which time range to handle. On Fri, Aug 30, 2019 at 11:16 AM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > My proposal is to have it at the DAG level rather than the operator level > as it means you don't have to deal with "skipped" beh

RE: DAG "Schedule Filter Callback"?

2019-08-30 Thread Shaw, Damian P.
mailto:kaxiln...@gmail.com] Sent: Friday, August 30, 2019 2:06 PM To: dev@airflow.apache.org Subject: Re: DAG "Schedule Filter Callback"? We can have a flag `depends_on_past_allow_skipped_state` or something similar that can take care of your 1st issue. On Fri, Aug 30, 2019 at 6:17

DAG "Schedule Filter Callback"?

2019-08-30 Thread Shaw, Damian P.
Hi all, After discussion at the NY Meetup this week I've been pondering how Airflow could support custom schedules with very little change to core Airflow logic and keeping backwards compatibility. As I understand the common way to support custom schedules is through a BranchOperator. You prov

RE: Setting to add choice of schedule at end or schedule at start of interval

2019-08-23 Thread Shaw, Damian P.
I can't understate what a conceptual improvement this would be for the end users of Airflow in our environment. I've written a lot of code so all our configuration works like this anyway. But the UI still shows the Airflow dates which still to this day sometimes confuse me. I'll be at the NY me

RE: Outage report

2019-08-16 Thread Shaw, Damian P.
Thanks, this is really useful to know! I often write my own Operators/Sensors/Hooks and was just looking at doing the same with the SFTPSensor and Operator. I've never formalized it but my current pattern is the follow: Hooks, Set self._conn to None on __init__, and have a property "self.conn"

RE: Manipulating the DAG Code View?

2019-08-14 Thread Shaw, Damian P.
eems to me that a better idea would be to add documentation to DAG. https://airflow.readthedocs.io/en/latest/concepts.html?highlight=doc_md#documentation-notes On Tue, Aug 13, 2019 at 8:55 PM Shaw, Damian P. wrote: > > As Ash says, I tried creating some test DAGs in 1.10.x and found that they

RE: Manipulating the DAG Code View?

2019-08-13 Thread Shaw, Damian P.
py#L229> >to the previous frame, but you could override it once you've created >the >DAG object. > >Chris > >On Tue, Aug 13, 2019 at 2:31 PM Shaw, Damian P. < >damian.sha...@credit-suisse.com> wrote: > >> Hi all, >> >> I'm in need of bei

Manipulating the DAG Code View?

2019-08-13 Thread Shaw, Damian P.
Hi all, I'm in need of being able to manipulate the code view for many of my DAGs which are generated by a single Python script that reads config files. My DAG creation script is getting increasingly less useful to the end users so I would like to instead present the config file that was used t

RE: Data Profiling / Graphs in RBAC UI?

2019-08-09 Thread Shaw, Damian P.
e feature are deprecated purposely in RBAC UI for security reasons. > To > > my understanding there is no plan to develop replacements for them, since > > it’s a deprecation rather than a “missing feature”. > > > > > > XD > > > > On Fri, Aug 9, 2019 at

RE: Data Profiling / Graphs in RBAC UI?

2019-08-09 Thread Shaw, Damian P.
re”. > > > XD > > On Fri, Aug 9, 2019 at 21:54 Shaw, Damian P. < > damian.sha...@credit-suisse.com> wrote: > > > Hi all, > > > > There are no "Data Profiling" features in the RBAC UI, is this or > > replacement planned to be developed? &g

Data Profiling / Graphs in RBAC UI?

2019-08-09 Thread Shaw, Damian P.
Hi all, There are no "Data Profiling" features in the RBAC UI, is this or replacement planned to be developed? If yes, is there any existing work done or what would need to be done to contribute to this? If no, what plans are there for users who use custom graphs from this section in the curre

RE: DummyOperator error in 1.10.4

2019-08-07 Thread Shaw, Damian P.
Hi, I'm not an Airflow expert and I've not had time to test this myself but poking around in the code it appears BaseOperator has a new parameter "do_xcom_push": https://github.com/apache/airflow/blob/master/airflow/models/baseoperator.py Whereas the old BaseOperator did not have this: https:

Best practises for debugging Macros or Plugins?

2019-07-16 Thread Shaw, Damian P.
Hi all, I've just been working on updating many of my macros to include logic to handle holiday calendars. There was a small mistake in one of the macros and I found it very difficult to debug, I'm not sure where a default logger in the plugins would log out to and there was no obvious way to r

RE: [ANNOUNCE] users@ list up and running

2019-05-28 Thread Shaw, Damian P.
Thanks Jakob! I've kicked it off with a question I've been unable to find an answer on searching the archives and StackOverflow but isn't really appropriate for Airflow Development list: https://lists.apache.org/thread.html/35a1af25202ee747998e3b0d7e684c87eee2edf6e974f348ceed0c79@%3Cusers.airfl

RE: Cron schedule with DST-aware timezone

2019-05-14 Thread Shaw, Damian P.
t; clear >>> expectations stated out as code. It makes it easier for people to get >>> consensus on expectations and for anyone to jump in and implement a fix. >>> >>> Max >>> >>> On Mon, May 13, 2019 at 12:48 PM David Klosowski >>> wro

RE: Cron schedule with DST-aware timezone

2019-05-13 Thread Shaw, Damian P.
Hi, I’m not part of the Airflow team but I came to the same conclusion, that the behavior is the opposite of what the documentation specifies. When using the cron notation then DST is properly honored, when using timedelta it is not. I played around with the DAG.following_schedule method to sat

RE: Manual raising of AirflowRescheduleException supported?

2019-05-09 Thread Shaw, Damian P.
idn't find time yet. Kind Regards, Stefan [1] https://issues.apache.org/jira/browse/AIRFLOW-2747 On 5/8/19 5:04 PM, Shaw, Damian P. wrote: > Hi all, > > Reading the reschedule code I notice that the execute method of the > BaseSensorOperator raises "AirflowRescheduleEx

Manual raising of AirflowRescheduleException supported?

2019-05-08 Thread Shaw, Damian P.
Hi all, Reading the reschedule code I notice that the execute method of the BaseSensorOperator raises "AirflowRescheduleException(reschedule_date)" I have a scenario where I would like to reschedule to a custom datetime, is it supported to manually raise this from my own custom Sensor? This is

RE: [DISCUSS] period_start/period_end instead of execution_date/next_execution_date

2019-04-09 Thread Shaw, Damian P.
Hi all, I'm new to this Airflow Dev mailing list so I wasn't expecting to reply to anything but I feel I am the target audience for this question. I am quite new to airflow and have been setting up an airflow environment for my business this last month. I find the current "execution_date" a sm