Re: Airflow Feature Design Feedback

Hi Ace, Any update on this one? I think there's a Github issue proposing similar functionality: https://github.com/apache/airflow/issues/12199 It would be good to coordinate the effort. Best, Tomek On 2020/10/18 13:09:01, Kaxil Naik wrote: > The Log table can be used at Dag Level too if we p

Re: Airflow Feature Design Feedback

The Log table can be used at Dag Level too if we pass task&instance=None For example: We log all the CLI actions in the Log table where we pass task_instance as None. Code: https://github.com/apache/airflow/blob/4e32546faf227a6497ce8b282fff7450cae6f665/airflow/utils/cli.py#L133 - Kaxil On Sun,

Re: Airflow Feature Design Feedback

I have the same concern as Yu Qian does, and a bit more elaborations follow below: - "log" table has fields: *id*, *dttm*, *dag_id*, *task_id*, *event*, *execution_date*, *owner*, *extra*. Seems to me only "*event*" or "*extra*"

Re: Airflow Feature Design Feedback

Did anyone consider having a more generic comments feature? The feature Ace described is an example of a comment being useful. E.g. when a user clears a task or marks it success/failed, he probably wants to include a comment for his action when he clicks OK. If a task failed and the user decided to

Re: Airflow Feature Design Feedback

Well thank you all for the feedback so quickly! Really appreciate it. We will get started in the coming week internally and hope within two weeks to have a pr for you all to review! I.e. our fabulous engineer Euccas Chen will be :) Thanks for the notes everyone - Ace > On Oct 16, 2020, at 10:

Re: Airflow Feature Design Feedback

+1 for feature. Strong preference for storing it in the log table, rather than in the task instance. On Fri, Oct 16, 2020 at 10:22 AM Tomasz Urbaszek wrote: > +1, I know a customer who was requesting it too > > On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik wrote: > >> +1 >> >> On Fri, Oct 16, 20

Re: Airflow Feature Design Feedback

+1, I know a customer who was requesting it too On Fri, Oct 16, 2020 at 6:52 PM Kaxil Naik wrote: > +1 > > On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk > wrote: > >> +1 >> >> On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari >> wrote: >> >>> +1 for the feature. Looking at the schema of the log

Re: Airflow Feature Design Feedback

+1 On Fri, Oct 16, 2020 at 5:48 PM Jarek Potiuk wrote: > +1 > > On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari > wrote: > >> +1 for the feature. Looking at the schema of the log table, I think it's >> perfect to store such kind of information. >> >> On Fri, Oct 16, 2020 at 9:55 PM Daniel Imbe

Re: Airflow Feature Design Feedback

+1 On Fri, Oct 16, 2020 at 6:42 PM Sumit Maheshwari wrote: > +1 for the feature. Looking at the schema of the log table, I think it's > perfect to store such kind of information. > > On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman > wrote: > >> This could be pretty valuable for future audits. I

Re: Airflow Feature Design Feedback

+1, Lyft uses the extra blob in the log table? to store UI auditing information. On Fri, Oct 16, 2020 at 9:42 AM Sumit Maheshwari wrote: > +1 for the feature. Looking at the schema of the log table, I think it's > perfect to store such kind of information. > > On Fri, Oct 16, 2020 at 9:55 PM Dan

Re: Airflow Feature Design Feedback

+1 for the feature. Looking at the schema of the log table, I think it's perfect to store such kind of information. On Fri, Oct 16, 2020 at 9:55 PM Daniel Imberman wrote: > This could be pretty valuable for future audits. I’d personally rather > avoid adding fields to the DB in general. Could we

Re: Airflow Feature Design Feedback

This could be pretty valuable for future audits. I’d personally rather avoid adding fields to the DB in general. Could we store it wherever we store normal failure messages? via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.6&source=email_footer_2] On Fri, Oct 16, 20

Airflow Feature Design Feedback

Hi team, My name is Ace, I’m a data engineer at Pinterest, and I wanted to get some feedback from the community on how they’d propose designing a feature that has been asked for us multiple times, that we’d like to implement in-house. The ask is the following: - Users want to be able to add an