Re: 1.10.4beta2 "snapshot" for available testing

2019-06-26 Thread airflowuser
thoughhttps://github.com/apache/airflow/blob/1.10.3/airflow/www/views.py#L113 > > > On 26 Jun 2019, at 11:34, airflowuser airflowu...@protonmail.com.INVALID > > wrote: > > http://localhost:8080/admin/dagmodel/ > > Sent with ProtonMail Secure Email. > > ‐‐‐ Origi

Re: 1.10.4beta2 "snapshot" for available testing

2019-06-26 Thread airflowuser
http://localhost:8080/admin/dagmodel/ Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Wednesday, June 26, 2019 1:32 PM, Ash Berlin-Taylor wrote: > What URL does that end up on? I can't reproduce that. > > -a > > > On 26 Jun 2019, at 10:5

Re: Field "owner" in DAG Model

2019-06-26 Thread airflowuser
If I may ask... in the Old UI how do you filter DAGs by owner? I'm running 1.10.3 and in the Search bar it searches only DAG it doesn't search by owner. Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Wednesday, June 26, 2019 12:27 PM, Deng Xiaodong wrote: > Hi folks, >

Re: 1.10.4beta2 "snapshot" for available testing

2019-06-26 Thread airflowuser
ck all 200+ items in the change log :) > > -a > > > On 25 Jun 2019, at 18:59, airflowuser airflowu...@protonmail.com.INVALID > > wrote: > > Why [AIRFLOW-1684] - Branching based on XCom variable (Docs) (#4365) listed > > under features? > > Sent with Proto

Re: 1.10.4beta2 "snapshot" for available testing

2019-06-25 Thread airflowuser
Why [AIRFLOW-1684] - Branching based on XCom variable (Docs) (#4365) listed under features? Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Tuesday, June 25, 2019 1:40 PM, Ash Berlin-Taylor wrote: > Hi Everyone, > > I'm excited to annoucne that 1.10.4 is almost ready f

Re: [DISCUSS] Pull Request Improvements

2019-06-19 Thread airflowuser
thing is updating all the PRs but leaving no trace other than an > change to the Last Updated column. > > -ash > > > On 19 Jun 2019, at 17:55, airflowuser airflowu...@protonmail.com.INVALID > > wrote: > > If commiters will simply use the Sort column and choose "rece

Re: [DISCUSS] Pull Request Improvements

2019-06-19 Thread airflowuser
If commiters will simply use the Sort column and choose "recently updated" it will sort all PRs by last activity time.. so old PRs that were recently updated/commented will bump up to the front. I wrote GitHub to set it as the default option. So, commiters if you will do that and pass on the 1-

MySQL mysql_postoperator parameter - To be deprecated in 2.0.0 ?

2019-05-30 Thread airflowuser
Question to MySQL users... I saw that HiveToMySQL have parameter called mysql_postoperator. This parameter runs 2nd query against the MySQL. This behavior doesn't exist in any of the other MySQL operators. While I'm all for having more features I'm not sure it's needed. The user can simply run MyS

Re: Proposal: Restructure slack

2019-05-26 Thread airflowuser
In most of the projects I see the tech/support questions are directed to stackoverflow. In general many consider popularity of project by how many answers are there in stackoverflow. users/dev list & Jira should be for something that needs discussions. Not for questions like: How to use X. I h

Re: Operator for syncing s3 buckets (and recursively copying)?

2019-05-22 Thread airflowuser
You can open a Jira for that and then submit your PR. There is a similar request for GCP: https://issues.apache.org/jira/browse/AIRFLOW-2842 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Thursday, May 23, 2019 6:32 AM, Austin Bennett wrote: > Hi All, > > I need s3 sy

Re: [Proposal] Component changes for Apache Airflow

2019-05-21 Thread airflowuser
>> >> Aizhamal >> >> On Fri, May 10, 2019 at 2:10 PM Aizhamal Nurmamat kyzy >> wrote: >> >>> Thank you for Jira powers, Fokko :) I am aiming to get it done by Monday. >>> >>> Regards, >>> Aizhamal >>> >>> *From: *Dr

Re: Call for fixes for 1.10.4

2019-05-19 Thread airflowuser
There are some PRs that were skipped from previous versions since no one reviewed them maybe now they can get in? Waiting for first review: https://github.com/apache/airflow/pull/4733 https://github.com/apache/airflow/pull/4603 https://github.com/apache/airflow/pull/4520 https://github.com/apac

Re: Deadlock at DAG SubDAG Execution at LocalExecutor

2019-05-03 Thread airflowuser
Subdags can be deadlocked when using local executor. This is why the default is sequential executor and there a warning about this in the code which is discussed in the stale PR about the issue: https://github.com/apache/airflow/pull/2367 Sent with [ProtonMail](https://protonmail.com) Secure Emai

Re: [Proposal] Component changes for Apache Airflow

2019-05-01 Thread airflowuser
is it possible to disable labels to begin with? I see no great benefit in having components with labels. We can do just fine with only components. I also think that the "version" field should be mandatory. It's important to know against which airflow version the ticket is reported. Sent with Pro

Re: Proposal: Automatically mark stale PRs in github

2019-04-23 Thread airflowuser
s will be > > > marked as stale by the stalebot around May 20th (providing that such > > > update > > > won't happen again). > > > Maybe then we can set it to 20 days + 7 for now to stale most issues up > > > in 3 days and delete them 10 days from

Re: Proposal: Automatically mark stale PRs in github

2019-04-17 Thread airflowuser
It's already on (or at least was on in December 2018). In any case here is a list of old PRs that are waiting for committers. [AIRFLOW-1956] Add parameter whether the navbar clock time is UTC https://github.com/apache/airflow/pull/2906 Status: ash commented but there are no further instructions.

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

2019-04-17 Thread airflowuser
To be honest on of the things that is very hard for a first time user in Airflow is to decide which executor to use. It would be extremely helpful if it would be possible to generate a table to compare properties between them. ‐‐‐ Original Message ‐‐‐ On Tuesday, April 9, 2019 10:52 PM,

Re: [DISCUSS] period_start/period_end instead of execution_date/next_execution_date

2019-04-15 Thread airflowuser
To quote my user-experience professor from ages ago: "If too many people misuse something you wrote it means that YOU are doing something wrong". Something can be well documented but if it's not intuitive it's likely that people will get it wrong. Say someone ask "When did you execute the code?

Re: [2.0 spring cleaning] Deprecate contrib folder?

2019-04-15 Thread airflowuser
Maybe a side issue but... Why must specify each operator in it's own import? Why can't we just something like do from airflow import operators,hook and all operators & hooks will become available. We don't have that many and even looking forward to the future it's unlikely that it will grow by hu

Re: [2.0 spring cleaning] Require unique conn_id

2019-04-13 Thread airflowuser
It can get more confusing because airflow allow to create two connection with same conn_id but different conn_type https://issues.apache.org/jira/browse/AIRFLOW-2784 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Sunday, April 14, 2019 12:22 AM, Maxime Beauchemin wrote

Re: Longer term Airflow planning

2019-04-10 Thread airflowuser
AIP's. The > > > question if there will be a true container-native, or cloud-native > > > version > > > of Airflow, is completely up to you and the community. I'm in favor of > > > jumping on the container train, but this requires to rework on the >

Re: Longer term Airflow planning

2019-04-10 Thread airflowuser
The Jira is a mess and it require committers time to organize it. Ideally users should report issues and committers should tag them with priority, milestone / fix version, labels (This is how for example it's done with https://github.com/pandas-dev/pandas ) When I have time I try to stack list

Re: Short Airflow user survey

2019-02-28 Thread airflowuser
Kamil the Jira contains many requests for new operators. To name a few: Zeppelin Operator https://issues.apache.org/jira/browse/AIRFLOW-1032 scrapyd operator https://issues.apache.org/jira/browse/AIRFLOW-827 PostgresTableSensor https://issues.apache.org/jira/browse/AIRFLOW-1759 S3 download fil

Re: Semantic versioning

2019-02-27 Thread airflowuser
There can be cases of small breaking changes If there are important notices it will be written here: https://github.com/apache/airflow/blob/master/UPDATING.md I asked before if Airflow will adopt calendar based versioning which also include semantic but no one replied. https://lists.apache.org/th

Re: [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0

2019-02-25 Thread airflowuser
This ticket also suggest function that doesn't support Python 3 https://issues.apache.org/jira/browse/AIRFLOW-2243 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Sunday, February 24, 2019 9:25 PM, Bas Harenslak wrote: > Following the comments to warn users about droppi

Switch to a Calendar based versioning scheme

2019-02-19 Thread airflowuser
May I suggest that the community consider to switch to a calendar based version release? see : https://calver.org/ Many projects have already adopted this numbering versions. This might be relevant now more than ever as pushing forward to Airflow 2.0.0 Then why not Airflow 19.0 ?

Re: Get a Head Start for Contributing to the Airflow Project

2019-02-04 Thread airflowuser
You can start with: Add shebang to docs/start_doc_server.sh https://issues.apache.org/jira/browse/AIRFLOW-3643 I'm also attaching some issues for anyone to stat with: Docs: https://issues.apache.org/jira/browse/AIRFLOW-2672 Tests (always a good start): Add tests for BigQueryToCloudStorageOpera

Re: AIP-12 Persist DAG into DB

2019-01-31 Thread airflowuser
I guess it's the same as: https://issues.apache.org/jira/browse/AIRFLOW-2619 However there is another related (?) issue: https://issues.apache.org/jira/browse/AIRFLOW-2637 When updating some of the DAG properties the DAG is flaky for a while until it stabilize. possibly this also related: https

Prevent using dot in DAG name (?)

2019-01-27 Thread airflowuser
Apparently dot - "." is a reserved char for sub-dags. See here: https://issues.apache.org/jira/browse/AIRFLOW-3292 I didn't know that, nor the author of this ticket (I assume others also don't know that as it isn't listed in the docs) Reserve words/chars is something that is pretty fundamental. S

Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?

2019-01-26 Thread airflowuser
The Mesos executor is being used (at least by some people) and there is an open PR to extend it: https://github.com/apache/airflow/pull/3739 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Friday, January 25, 2019 7:30 PM, Kaxil Naik wrote: > What about Dask Executor as

Re: AIP-8 Split Hooks/Operators into Separate Packages

2019-01-09 Thread airflowuser
1046752b1e/airflow/utils/log/gcs_task_handler.py#L45 > > > > > > We'd need to adjust the import path for the hook, but so long as the > > > > upload > > > > > > > / download method remains stable, it'll work the same. The sub-package &g

Re: AIP-8 Split Hooks/Operators into Separate Packages

2019-01-08 Thread airflowuser
I think the operator should be placed by the source. If it's MySQLToHiveOperator then it would be placed in MySQL package. The BIG question here is if this serve actual improvement like faster deployment of hook/operators bug-fix to Airflow users (faster than actual Airflow release) or this is