[jira] [Created] (AIRFLOW-7072) Task instance email_alert render template only if need
zhongjiajie created AIRFLOW-7072: Summary: Task instance email_alert render template only if need Key: AIRFLOW-7072 URL: https://issues.apache.org/jira/browse/AIRFLOW-7072 Project: Apache Airflow Issue Type: Improvement Components: models Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie Task instance email_alert render html_content_err template only if need -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7066) Use sphinx including syntax in concepts.rst
zhongjiajie created AIRFLOW-7066: Summary: Use sphinx including syntax in concepts.rst Key: AIRFLOW-7066 URL: https://issues.apache.org/jira/browse/AIRFLOW-7066 Project: Apache Airflow Issue Type: Improvement Components: documentation Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie Use sphinx syntax if possible instead of python code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7053) Fix make success/failed error in graph view when dag not run
zhongjiajie created AIRFLOW-7053: Summary: Fix make success/failed error in graph view when dag not run Key: AIRFLOW-7053 URL: https://issues.apache.org/jira/browse/AIRFLOW-7053 Project: Apache Airflow Issue Type: Bug Components: webserver Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie Attachments: Screen Shot 2020-03-13 at 11.22.25.png, Screen Shot 2020-03-13 at 11.27.36.png Graph view now will raise an error when click make task instance success/failed, with the init state and no dag run, we should make some hint instead of raise error in the webserver before !Screen Shot 2020-03-13 at 11.27.36.png! after !Screen Shot 2020-03-13 at 11.22.25.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7033) Change dag and task state meanwhile
zhongjiajie created AIRFLOW-7033: Summary: Change dag and task state meanwhile Key: AIRFLOW-7033 URL: https://issues.apache.org/jira/browse/AIRFLOW-7033 Project: Apache Airflow Issue Type: Improvement Components: api, webserver Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie In dag page, have make task instance state to success or failed bottom, when we click those bottom not only also change task instance state but also dag run state if possible. * When trigger make_failed bottom, the dag run state should change to failed * When trigger make_success bottom, then dag run state should change to success if all task instance state are success, otherwise should keep the original state -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6976) Correct cli dag command ignore-first-depends-on-past
[ https://issues.apache.org/jira/browse/AIRFLOW-6976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-6976. Resolution: Duplicate According to [https://github.com/apache/airflow/pull/7610#issuecomment-596494970] , this path should use the same JIAR ticket as https://issues.apache.org/jira/browse/AIRFLOW-6860, > Correct cli dag command ignore-first-depends-on-past > > > Key: AIRFLOW-6976 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6976 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 1.10.9 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > ref PR [https://github.com/apache/airflow/pull/7490] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6979) Fix breeze test-target specify individual tests not work
zhongjiajie created AIRFLOW-6979:
Summary: Fix breeze test-target specify individual tests not work
Key: AIRFLOW-6979
URL: https://issues.apache.org/jira/browse/AIRFLOW-6979
Project: Apache Airflow
Issue Type: Improvement
Components: breeze
Affects Versions: 1.10.9
Reporter: zhongjiajie
Assignee: zhongjiajie
We miss a test-target arg when running
{code:java}
./breeze test-target tests/test_core.py::TestCore{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6979) Fix breeze test-target specify individual tests not work
[
https://issues.apache.org/jira/browse/AIRFLOW-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-6979:
-
Issue Type: Bug (was: Improvement)
> Fix breeze test-target specify individual tests not work
>
>
> Key: AIRFLOW-6979
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6979
> Project: Apache Airflow
> Issue Type: Bug
> Components: breeze
>Affects Versions: 1.10.9
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
>
> We miss a test-target arg when running
> {code:java}
> ./breeze test-target tests/test_core.py::TestCore{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6976) Correct cli dag command ignore-first-depends-on-past
zhongjiajie created AIRFLOW-6976: Summary: Correct cli dag command ignore-first-depends-on-past Key: AIRFLOW-6976 URL: https://issues.apache.org/jira/browse/AIRFLOW-6976 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie ref PR [https://github.com/apache/airflow/pull/7490] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6959) Use NULL as dag.description default value and change related UI
zhongjiajie created AIRFLOW-6959: Summary: Use NULL as dag.description default value and change related UI Key: AIRFLOW-6959 URL: https://issues.apache.org/jira/browse/AIRFLOW-6959 Project: Apache Airflow Issue Type: Improvement Components: DAG, database, ui Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6952) Use property for dag default_view
[ https://issues.apache.org/jira/browse/AIRFLOW-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-6952: - Summary: Use property for dag default_view (was: Dag default_view should use property) > Use property for dag default_view > - > > Key: AIRFLOW-6952 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6952 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Affects Versions: 1.10.9 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > should use dag.default_view instead of dag._default_view -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6952) Dag default_view should use property
zhongjiajie created AIRFLOW-6952: Summary: Dag default_view should use property Key: AIRFLOW-6952 URL: https://issues.apache.org/jira/browse/AIRFLOW-6952 Project: Apache Airflow Issue Type: Improvement Components: DAG Affects Versions: 1.10.9 Reporter: zhongjiajie Assignee: zhongjiajie should use dag.default_view instead of dag._default_view -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6472) Some of the options we have are long options with single -
[
https://issues.apache.org/jira/browse/AIRFLOW-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047167#comment-17047167
]
zhongjiajie commented on AIRFLOW-6472:
--
Related dev mail list
[https://lists.apache.org/thread.html/r95c53953499a236466e0b762dfdadd5ca2ba9d6e2a3516c699a14380%40%3Cdev.airflow.apache.org%3E]
> Some of the options we have are long options with single -
> --
>
> Key: AIRFLOW-6472
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6472
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Jarek Potiuk
>Assignee: zhongjiajie
>Priority: Major
>
> We have some "short" options that are really "long" ones: namely -int and -sd
> in the run task. This is against the idea of short and long options in Unix
> (and argparse follows that). The main reason to have short options is that
> you can combine short options:
> {{airflow task run -iAlm}}
> When you have more than one letter "short" options this might become quickly
> ambiguous. We do not have -s yet but if we add few more options this might
> become a problem.
> Also in the argparse documentation
> [https://docs.python.org/2/library/argparse.html] it's mentioned that short
> options should be single character only.\
> This should be fixed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6570) Add dag tag for all example dag
zhongjiajie created AIRFLOW-6570: Summary: Add dag tag for all example dag Key: AIRFLOW-6570 URL: https://issues.apache.org/jira/browse/AIRFLOW-6570 Project: Apache Airflow Issue Type: Improvement Components: examples Affects Versions: 1.10.7 Reporter: zhongjiajie Assignee: zhongjiajie We miss add some dag tag in [https://github.com/apache/airflow/pull/6489] . this patch to add to all others -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4375) Replace airflow.utils.file.TemporaryDirectory by native tempfile.TemporaryDirectory
[ https://issues.apache.org/jira/browse/AIRFLOW-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015828#comment-17015828 ] zhongjiajie commented on AIRFLOW-4375: -- [~basph] [~kaxilnaik] This ticket is duplicates to https://issues.apache.org/jira/browse/AIRFLOW-6396 could you pls close this? > Replace airflow.utils.file.TemporaryDirectory by native > tempfile.TemporaryDirectory > --- > > Key: AIRFLOW-4375 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4375 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > Python 2 did not have a native temporarydirectory and I believe that's why > there is airflow.utils.file.TemporaryDirectory. > Since Python 3.2 this is supported by tempfile.TemporaryDirectory, so remove > the Airflow util and replace by the native Python tempfile.TemporaryDirectory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4209) Replace imp by importlib
[ https://issues.apache.org/jira/browse/AIRFLOW-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015822#comment-17015822 ] zhongjiajie commented on AIRFLOW-4209: -- [~fokko] I could not close this ticket, it is duplicated by https://issues.apache.org/jira/browse/AIRFLOW-6566 and 6566 was close, could you pls close this? > Replace imp by importlib > > > Key: AIRFLOW-4209 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4209 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > > Remove references to the imp module, this is replaced by importlib -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6472) Some of the options we have are long options with single -
[
https://issues.apache.org/jira/browse/AIRFLOW-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008993#comment-17008993
]
zhongjiajie commented on AIRFLOW-6472:
--
If you don't mind, I will create a PR for this issues [~potiuk]
> Some of the options we have are long options with single -
> --
>
> Key: AIRFLOW-6472
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6472
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Jarek Potiuk
>Assignee: zhongjiajie
>Priority: Major
>
> We have some "short" options that are really "long" ones: namely -int and -sd
> in the run task. This is against the idea of short and long options in Unix
> (and argparse follows that). The main reason to have short options is that
> you can combine short options:
> {{airflow task run -iAlm}}
> When you have more than one letter "short" options this might become quickly
> ambiguous. We do not have -s yet but if we add few more options this might
> become a problem.
> Also in the argparse documentation
> [https://docs.python.org/2/library/argparse.html] it's mentioned that short
> options should be single character only.\
> This should be fixed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6472) Some of the options we have are long options with single -
[
https://issues.apache.org/jira/browse/AIRFLOW-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie reassigned AIRFLOW-6472:
Assignee: zhongjiajie
> Some of the options we have are long options with single -
> --
>
> Key: AIRFLOW-6472
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6472
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Jarek Potiuk
>Assignee: zhongjiajie
>Priority: Major
>
> We have some "short" options that are really "long" ones: namely -int and -sd
> in the run task. This is against the idea of short and long options in Unix
> (and argparse follows that). The main reason to have short options is that
> you can combine short options:
> {{airflow task run -iAlm}}
> When you have more than one letter "short" options this might become quickly
> ambiguous. We do not have -s yet but if we add few more options this might
> become a problem.
> Also in the argparse documentation
> [https://docs.python.org/2/library/argparse.html] it's mentioned that short
> options should be single character only.\
> This should be fixed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6416) [AIRFLOW-6416] Sort default connection by conn_id
[ https://issues.apache.org/jira/browse/AIRFLOW-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-6416: - Description: Sort default connection by conn_id * Sort create_default_connections by connection id * Add sort connection id unittest was: Remove DB provide_session and sort by conn_id * Remove unnecessary annotation provide_session * Sort create_default_connections by connection id * Add sort connection id unittest > [AIRFLOW-6416] Sort default connection by conn_id > - > > Key: AIRFLOW-6416 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6416 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.7 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > Sort default connection by conn_id > * Sort create_default_connections by connection id > * Add sort connection id unittest -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6416) [AIRFLOW-6416] Sort default connection by conn_id
[ https://issues.apache.org/jira/browse/AIRFLOW-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-6416: - Summary: [AIRFLOW-6416] Sort default connection by conn_id (was: Remove DB provide_session and sort by conn_id) > [AIRFLOW-6416] Sort default connection by conn_id > - > > Key: AIRFLOW-6416 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6416 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Affects Versions: 1.10.7 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > Remove DB provide_session and sort by conn_id > * Remove unnecessary annotation provide_session > * Sort create_default_connections by connection id > * Add sort connection id unittest -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6416) Remove DB provide_session and sort by conn_id
zhongjiajie created AIRFLOW-6416: Summary: Remove DB provide_session and sort by conn_id Key: AIRFLOW-6416 URL: https://issues.apache.org/jira/browse/AIRFLOW-6416 Project: Apache Airflow Issue Type: Improvement Components: database Affects Versions: 1.10.7 Reporter: zhongjiajie Assignee: zhongjiajie Remove DB provide_session and sort by conn_id * Remove unnecessary annotation provide_session * Sort create_default_connections by connection id * Add sort connection id unittest -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6346) Enhance dag default_view and orientation
zhongjiajie created AIRFLOW-6346: Summary: Enhance dag default_view and orientation Key: AIRFLOW-6346 URL: https://issues.apache.org/jira/browse/AIRFLOW-6346 Project: Apache Airflow Issue Type: Improvement Components: DAG Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Enhance dag default_view * only DEFAULT_VIEW_PRESETS are acceptable * only ORIENTATION_PRESETS are acceptable * use webserver.dag_default_view in airflow.cfg as default value * remove DAG and DagModel function get_default_view * rename _default_value to default_value * -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6330) Show cli help when param blank or typo
zhongjiajie created AIRFLOW-6330:
Summary: Show cli help when param blank or typo
Key: AIRFLOW-6330
URL: https://issues.apache.org/jira/browse/AIRFLOW-6330
Project: Apache Airflow
Issue Type: Improvement
Components: cli
Affects Versions: 1.10.6
Reporter: zhongjiajie
Assignee: zhongjiajie
Show cli help when param blank or typo
When user enter Airflow cli with *blank parameter* or typo arg, or use wrong
parameter show Airflow cli help. Example like below
{code:java}
// when you enter airflow with blank parameter
$ airflow
usage: airflow [-h]
{config,connections,dags,db,flower,kerberos,pools,roles,rotate_fernet_key,scheduler,sync_perm,tasks,users,variables,version,webserver,worker}
sub-command help
config Show current application configuration
connections List/Add/Delete connections
dagsList and manage DAGs
db Database operations
flower Start a Celery Flower
kerberosStart a kerberos ticket renewer
pools CRUD operations on pools
roles Create/List roles
rotate_fernet_key Rotate all encrypted connection credentials and
variables; see
https://airflow.readthedocs.io/en/stable/howto/secure-
connections.html#rotating-encryption-keys
scheduler Start a scheduler instance
sync_perm Update permissions for existing roles and DAGs
tasks List and manage tasks
users CRUD operations on users
variables CRUD operations on variables
version Show the version
webserver Start a Airflow webserver instance
worker Start a Celery worker nodeoptional arguments:
-h, --helpshow this help message and exit
airflow command error: the following arguments are required: subcommand, see
help above.
// So as the subcommands
$ airflow connections
usage: airflow connections [-h] {list,add,delete} ...positional arguments:
{list,add,delete}
list List connections
add Add a connection
delete Delete a connectionoptional arguments:
-h, --help show this help message and exit
airflow connections command error: the following arguments are required:
subcommand, see help above.{code}
When you *typo* parameter
{code:java}
// when you typo or wrong parameter, if will show you cli help and the preset
parameter to use in error line
$ airflow connections docker
usage: airflow connections [-h] {list,add,delete} ...positional arguments:
{list,add,delete}
list List connections
add Add a connection
delete Delete a connectionoptional arguments:
-h, --help show this help message and exit
airflow connections command error: argument subcommand: invalid choice:
'docker' (choose from 'list', 'add', 'delete'), see help above.{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6326) Sort cli commands and arg
zhongjiajie created AIRFLOW-6326: Summary: Sort cli commands and arg Key: AIRFLOW-6326 URL: https://issues.apache.org/jira/browse/AIRFLOW-6326 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Sort cli commands and arg -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6325) Clear cli hint
zhongjiajie created AIRFLOW-6325: Summary: Clear cli hint Key: AIRFLOW-6325 URL: https://issues.apache.org/jira/browse/AIRFLOW-6325 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Clear cli hint * use complex to import objects * use all objects to export objects * use CURD to airflow users instead of List/Create/Delete/Update * make all hint without . in the end -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6320) Add quarterly to crontab presets
zhongjiajie created AIRFLOW-6320: Summary: Add quarterly to crontab presets Key: AIRFLOW-6320 URL: https://issues.apache.org/jira/browse/AIRFLOW-6320 Project: Apache Airflow Issue Type: Improvement Components: utils Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Add quarterly to crontab presets -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6316) Use sphinx render in tutorial.rst
zhongjiajie created AIRFLOW-6316: Summary: Use sphinx render in tutorial.rst Key: AIRFLOW-6316 URL: https://issues.apache.org/jira/browse/AIRFLOW-6316 Project: Apache Airflow Issue Type: Improvement Components: documentation Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Recently we hard code in tutorial.rst which is hard to maintain, such as `set_upstream` is change to shift in tutorial.py but still in tutorial.rst. Use sphinx is a better way -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6313) Unify example or doc dag owner
zhongjiajie created AIRFLOW-6313: Summary: Unify example or doc dag owner Key: AIRFLOW-6313 URL: https://issues.apache.org/jira/browse/AIRFLOW-6313 Project: Apache Airflow Issue Type: Improvement Components: examples Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie We already change Airflow DAG default owner to 'airflow' in [https://github.com/apache/airflow/pull/4151] but some of our example DAGs and docs are still using owner = 'Airflow', This patch to unify them -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6297) Add Airflow website link to UI docs
[ https://issues.apache.org/jira/browse/AIRFLOW-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-6297: - Attachment: Screen Shot 2019-12-19 at 15.44.31.png > Add Airflow website link to UI docs > --- > > Key: AIRFLOW-6297 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6297 > Project: Apache Airflow > Issue Type: New Feature > Components: ui >Affects Versions: 1.10.6 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Attachments: Screen Shot 2019-12-19 at 15.44.31.png > > > Add Airflow website link to UI docs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6297) Add Airflow website link to UI docs
zhongjiajie created AIRFLOW-6297: Summary: Add Airflow website link to UI docs Key: AIRFLOW-6297 URL: https://issues.apache.org/jira/browse/AIRFLOW-6297 Project: Apache Airflow Issue Type: New Feature Components: ui Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Add Airflow website link to UI docs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6089) Reorder setup.py dependencies and add ci
[ https://issues.apache.org/jira/browse/AIRFLOW-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-6089: - Summary: Reorder setup.py dependencies and add ci (was: Reorder setup package and add unittest for CI) > Reorder setup.py dependencies and add ci > > > Key: AIRFLOW-6089 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6089 > Project: Apache Airflow > Issue Type: New Feature > Components: ci >Affects Versions: 1.10.6 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > Reorder setup package in alphabetical and add unittest to ci -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6089) Reorder setup package and add unittest for CI
zhongjiajie created AIRFLOW-6089: Summary: Reorder setup package and add unittest for CI Key: AIRFLOW-6089 URL: https://issues.apache.org/jira/browse/AIRFLOW-6089 Project: Apache Airflow Issue Type: New Feature Components: ci Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie Reorder setup package in alphabetical and add unittest to ci -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5928) hive hooks load_file short circuit
zhongjiajie created AIRFLOW-5928: Summary: hive hooks load_file short circuit Key: AIRFLOW-5928 URL: https://issues.apache.org/jira/browse/AIRFLOW-5928 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.6 Reporter: zhongjiajie Assignee: zhongjiajie If function `load_file` with parameter `create` and `recreate` are `False`, `hql = ''` and should not call `HiveCliHook.run_cli` Due to `recreate` in two `if` statement, `HiveCliHook.run_cli` only need to in the last one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2697) Drop snakebite in favour of pyarrow
[ https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893291#comment-16893291 ] zhongjiajie commented on AIRFLOW-2697: -- Hi [~jrderuiter], seem you do not wok on this PR anymore. Do you mind I continue woking on you PR? > Drop snakebite in favour of pyarrow > --- > > Key: AIRFLOW-2697 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2697 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Julian de Ruiter >Assignee: Julian de Ruiter >Priority: Blocker > Fix For: 2.0.0 > > > The current HdfsHook relies on the snakebite library, which is unfortunately > not compatible with Python 3. To add Python 3 support for the HdfsHook > requires switching to a different library for interacting with HDFS. The > hdfs3 library is an attractive alternative, as it supports Python 3 and seems > to be stable and relatively well supported. > Update: hdfs3 doesn't get any updates anymore. The best library right now > seems to be pyarrow: https://arrow.apache.org/docs/python/filesystems.html > Therefore I would like to upgrade to pyarrow instead of hdfs3. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AIRFLOW-5000) Remove task instance duplicate end_date and reorder template context
[ https://issues.apache.org/jira/browse/AIRFLOW-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie resolved AIRFLOW-5000. -- Resolution: Fixed Fix Version/s: 1.10.5 > Remove task instance duplicate end_date and reorder template context > > > Key: AIRFLOW-5000 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5000 > Project: Apache Airflow > Issue Type: Improvement > Components: models >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Fix For: 1.10.5 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5000) Remove task instance duplicate end_date and reorder template context
zhongjiajie created AIRFLOW-5000: Summary: Remove task instance duplicate end_date and reorder template context Key: AIRFLOW-5000 URL: https://issues.apache.org/jira/browse/AIRFLOW-5000 Project: Apache Airflow Issue Type: Improvement Components: models Affects Versions: 1.10.3 Reporter: zhongjiajie Assignee: zhongjiajie -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (AIRFLOW-4828) Remove param python_version in PythonVirtualenvOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-4828. Resolution: Won't Fix According to Kurqq comment on Github > I don't think we should remove `python_version `parameter. > If I run Airflow on Python 3.5 and want to run a task with Python 3.7 I need > the `python_version `parameter. > Moreover in the future Python 4 will be released. Airflow will likely support > both for a while > > I think the `python_version `needs to stay it's not unique to Python 2.7 > > I also think that if someone wants to run something with Python 2.7 with the > `PythonVirtualenvOperator `he is more than welcome. There is no need to > prevent the user from doing so. Airflow can just remove the test for this so > if something is not working it's the user problem. > Remove param python_version in PythonVirtualenvOperator > --- > > Key: AIRFLOW-4828 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4828 > Project: Apache Airflow > Issue Type: Sub-task > Components: operators >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > > Remove param python_version in PythonVirtualenvOperator -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4828) Remove param python_version in PythonVirtualenvOperator
zhongjiajie created AIRFLOW-4828: Summary: Remove param python_version in PythonVirtualenvOperator Key: AIRFLOW-4828 URL: https://issues.apache.org/jira/browse/AIRFLOW-4828 Project: Apache Airflow Issue Type: Sub-task Components: operators Affects Versions: 1.10.3 Reporter: zhongjiajie Assignee: zhongjiajie Remove param python_version in PythonVirtualenvOperator -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4827) Remove compatible test for python 2
zhongjiajie created AIRFLOW-4827: Summary: Remove compatible test for python 2 Key: AIRFLOW-4827 URL: https://issues.apache.org/jira/browse/AIRFLOW-4827 Project: Apache Airflow Issue Type: Sub-task Components: tests Affects Versions: 1.10.3 Reporter: zhongjiajie Assignee: zhongjiajie Remove compatible test for python 2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4210) Update the CI image to Python3
[ https://issues.apache.org/jira/browse/AIRFLOW-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866138#comment-16866138 ] zhongjiajie commented on AIRFLOW-4210: -- Maybe we should close it [~Fokko] > Update the CI image to Python3 > -- > > Key: AIRFLOW-4210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4210 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > > Currently, we build from plain Debian, but we could now directly build from > Python3 image: https://github.com/apache/airflow-ci/blob/master/Dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4061) Remove incubator in ci docker-compose
[ https://issues.apache.org/jira/browse/AIRFLOW-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866134#comment-16866134 ] zhongjiajie commented on AIRFLOW-4061: -- Don't need to solve, And we will integrate CI image in Apache/Airflow repo, rather than Apache/Airflow-ci > Remove incubator in ci docker-compose > - > > Key: AIRFLOW-4061 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4061 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: CI > Fix For: 1.10.4 > > > We still use `airflowci/incubator-airflow-ci:latest` in our ci > docker-compose, should change it to `airflowci/airflow-ci:latest`, in this > case we have to change the setting for > [airflow-ci|https://github.com/apache/airflow-ci] docker-hub setting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4061) Remove incubator in ci docker-compose
[ https://issues.apache.org/jira/browse/AIRFLOW-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-4061. Resolution: Won't Fix Don't need to solve, And we will integrate CI image in Apache/Airflow repo, rather than Apache/Airflow-ci > Remove incubator in ci docker-compose > - > > Key: AIRFLOW-4061 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4061 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: CI > Fix For: 1.10.4 > > > We still use `airflowci/incubator-airflow-ci:latest` in our ci > docker-compose, should change it to `airflowci/airflow-ci:latest`, in this > case we have to change the setting for > [airflow-ci|https://github.com/apache/airflow-ci] docker-hub setting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4210) Update the CI image to Python3
[ https://issues.apache.org/jira/browse/AIRFLOW-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866132#comment-16866132 ] zhongjiajie commented on AIRFLOW-4210: -- Don't need to solve, And we will integrate CI image in Apache/Airflow repo, rather than Apache/Airflow-ci > Update the CI image to Python3 > -- > > Key: AIRFLOW-4210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4210 > Project: Apache Airflow > Issue Type: Sub-task > Components: core >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > > Currently, we build from plain Debian, but we could now directly build from > Python3 image: https://github.com/apache/airflow-ci/blob/master/Dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4460) Remove __future__ import in models
zhongjiajie created AIRFLOW-4460: Summary: Remove __future__ import in models Key: AIRFLOW-4460 URL: https://issues.apache.org/jira/browse/AIRFLOW-4460 Project: Apache Airflow Issue Type: Sub-task Components: models Reporter: zhongjiajie Assignee: zhongjiajie Remove __future__ should being done in https://issues.apache.org/jira/browse/AIRFLOW-4200, but in the same time we refactor models and move some code out of __init__.py. This Jira ticket to remove __future__ in models. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4381) Use get_direct_relative_ids to get task relatives
zhongjiajie created AIRFLOW-4381: Summary: Use get_direct_relative_ids to get task relatives Key: AIRFLOW-4381 URL: https://issues.apache.org/jira/browse/AIRFLOW-4381 Project: Apache Airflow Issue Type: Improvement Components: models Affects Versions: 1.10.3 Reporter: zhongjiajie Assignee: zhongjiajie Now in models.baseoperator we get relatives by attribute, but _downstream_task_ids and _upstream_task_ids is private member. This patch will get relatives from exists function `get_direct_relative_ids`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4296) Remove py2 in ci process
[
https://issues.apache.org/jira/browse/AIRFLOW-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie resolved AIRFLOW-4296.
--
Resolution: Fixed
Fix Version/s: 2.0.0
> Remove py2 in ci process
> -
>
> Key: AIRFLOW-4296
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4296
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: patch
> Fix For: 2.0.0
>
>
> We still have py2 in our CI process, and this will cause error when in AIP-3
> PR, we should remove that
>
> +Even+ though sub-task-1 merge, still error like below
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 21, in
> from airflow import configuration
> File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 36,
> in
> from airflow import settings, configuration as conf
> File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 32,
> in
> from airflow.utils.sqlalchemy import setup_event_handlers
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/sqlalchemy.py",
> line 22, in
> import json
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/json.py", line
> 27, in
> class AirflowJsonEncoder(json.JSONEncoder):
> AttributeError: 'module' object has no attribute 'JSONEncoder'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4314) List task Instance search for with state None
[ https://issues.apache.org/jira/browse/AIRFLOW-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4314: Assignee: (was: zhongjiajie) > List task Instance search for with state None > - > > Key: AIRFLOW-4314 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4314 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Priority: Major > Labels: patch > > In the *List Task Instance* UI view, we could not search task with state > `None`, both with conditions `null` or `None`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4314) List task Instance search for with state None
[ https://issues.apache.org/jira/browse/AIRFLOW-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821777#comment-16821777 ] zhongjiajie commented on AIRFLOW-4314: -- [~ash] I not use in RBAC mode. > List task Instance search for with state None > - > > Key: AIRFLOW-4314 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4314 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > > In the *List Task Instance* UI view, we could not search task with state > `None`, both with conditions `null` or `None`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4210) Update the CI image to Python3
[ https://issues.apache.org/jira/browse/AIRFLOW-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821762#comment-16821762 ] zhongjiajie commented on AIRFLOW-4210: -- I found out this Jira ticket not in Apache/Airflow repo, are we still want to tracking it in Airflow jira? > Update the CI image to Python3 > -- > > Key: AIRFLOW-4210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4210 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > > Currently, we build from plain Debian, but we could now directly build from > Python3 image: https://github.com/apache/airflow-ci/blob/master/Dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4210) Update the CI image to Python3
[ https://issues.apache.org/jira/browse/AIRFLOW-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4210: Assignee: zhongjiajie > Update the CI image to Python3 > -- > > Key: AIRFLOW-4210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4210 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > > Currently, we build from plain Debian, but we could now directly build from > Python3 image: https://github.com/apache/airflow-ci/blob/master/Dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4314) List task Instance search for with state None
[ https://issues.apache.org/jira/browse/AIRFLOW-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-4314: - Description: In the *List Task Instance* UI view, we could not search task with state `None`, both with conditions `null` or `None`. (was: In the *List Task Instance* UI view, we could not search task with state `None`, we can not search with `null` or `None`) > List task Instance search for with state None > - > > Key: AIRFLOW-4314 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4314 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.3 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > > In the *List Task Instance* UI view, we could not search task with state > `None`, both with conditions `null` or `None`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4314) List task Instance search for with state None
zhongjiajie created AIRFLOW-4314: Summary: List task Instance search for with state None Key: AIRFLOW-4314 URL: https://issues.apache.org/jira/browse/AIRFLOW-4314 Project: Apache Airflow Issue Type: Bug Components: ui Affects Versions: 1.10.3 Reporter: zhongjiajie Assignee: zhongjiajie In the *List Task Instance* UI view, we could not search task with state `None`, we can not search with `null` or `None` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4296) Remove py2 in ci process
[
https://issues.apache.org/jira/browse/AIRFLOW-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-4296:
-
Description:
We still have py2 in our CI process, and this will cause error when in AIP-3
PR, we should remove that
+Even+ though sub-task-1 merge, still error like below
{code:java}
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 21, in
from airflow import configuration
File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 36,
in
from airflow import settings, configuration as conf
File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 32,
in
from airflow.utils.sqlalchemy import setup_event_handlers
File "/usr/local/lib/python2.7/dist-packages/airflow/utils/sqlalchemy.py",
line 22, in
import json
File "/usr/local/lib/python2.7/dist-packages/airflow/utils/json.py", line 27,
in
class AirflowJsonEncoder(json.JSONEncoder):
AttributeError: 'module' object has no attribute 'JSONEncoder'
{code}
was:We still have py2 in our CI process, and this will cause error when in
AIP-3 PR, we should remove that
> Remove py2 in ci process
> -
>
> Key: AIRFLOW-4296
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4296
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: ci
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: patch
>
> We still have py2 in our CI process, and this will cause error when in AIP-3
> PR, we should remove that
>
> +Even+ though sub-task-1 merge, still error like below
> {code:java}
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 21, in
> from airflow import configuration
> File "/usr/local/lib/python2.7/dist-packages/airflow/__init__.py", line 36,
> in
> from airflow import settings, configuration as conf
> File "/usr/local/lib/python2.7/dist-packages/airflow/settings.py", line 32,
> in
> from airflow.utils.sqlalchemy import setup_event_handlers
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/sqlalchemy.py",
> line 22, in
> import json
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/json.py", line
> 27, in
> class AirflowJsonEncoder(json.JSONEncoder):
> AttributeError: 'module' object has no attribute 'JSONEncoder'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4296) Remove py2 in ci process
zhongjiajie created AIRFLOW-4296: Summary: Remove py2 in ci process Key: AIRFLOW-4296 URL: https://issues.apache.org/jira/browse/AIRFLOW-4296 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie We still have py2 in our CI process, and this will cause error when in AIP-3 PR, we should remove that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4258) UI add multi selection shortcut
zhongjiajie created AIRFLOW-4258: Summary: UI add multi selection shortcut Key: AIRFLOW-4258 URL: https://issues.apache.org/jira/browse/AIRFLOW-4258 Project: Apache Airflow Issue Type: New Feature Components: ui Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Recently our UI shortcut only support select single task selection, for more convenient, we should add feature to select multi task as shutcut in graph view. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4209) Replace imp by importlib
[ https://issues.apache.org/jira/browse/AIRFLOW-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4209: Assignee: zhongjiajie > Replace imp by importlib > > > Key: AIRFLOW-4209 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4209 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > > Remove references to the imp module, this is replaced by importlib -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4221) Make bitwise operation work between list[tasks]
zhongjiajie created AIRFLOW-4221: Summary: Make bitwise operation work between list[tasks] Key: AIRFLOW-4221 URL: https://issues.apache.org/jira/browse/AIRFLOW-4221 Project: Apache Airflow Issue Type: Improvement Components: dependencies Affects Versions: 1.10.2 Reporter: zhongjiajie We recently remove function chain in utils.helper module, and we still have cross_downstream in helper module. I think we should make bitwise operation wok between list[tasks] and then remove cross_downstream in utils.helper. This make our dependencies unity and easy to express -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4222) Add cli autocomplete
zhongjiajie created AIRFLOW-4222: Summary: Add cli autocomplete Key: AIRFLOW-4222 URL: https://issues.apache.org/jira/browse/AIRFLOW-4222 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Airflow cli can not autocomplete so far, this ticket will make it happen to let cli more user-friendliness -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4201) Replace unicode strings by "normal" strings
[ https://issues.apache.org/jira/browse/AIRFLOW-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4201: Assignee: zhongjiajie > Replace unicode strings by "normal" strings > --- > > Key: AIRFLOW-4201 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4201 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4200) Remove all __future__ imports
[ https://issues.apache.org/jira/browse/AIRFLOW-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4200: Assignee: zhongjiajie > Remove all __future__ imports > - > > Key: AIRFLOW-4200 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4200 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4199) Remove all sys.version_info[0] == 3 imports
[ https://issues.apache.org/jira/browse/AIRFLOW-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806673#comment-16806673 ] zhongjiajie commented on AIRFLOW-4199: -- [~Fokko] I have a question, should we remove all module `six` in py file. Or create a new Jira ticket to do it? If we do remove sys.version_info and six in one ticket seem too big to review. > Remove all sys.version_info[0] == 3 imports > --- > > Key: AIRFLOW-4199 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4199 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4202) Make setup.py compatible with Python 3 only
[ https://issues.apache.org/jira/browse/AIRFLOW-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4202: Assignee: zhongjiajie > Make setup.py compatible with Python 3 only > --- > > Key: AIRFLOW-4202 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4202 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4199) Remove all sys.version_info[0] == 3 imports
[ https://issues.apache.org/jira/browse/AIRFLOW-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-4199: Assignee: zhongjiajie > Remove all sys.version_info[0] == 3 imports > --- > > Key: AIRFLOW-4199 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4199 > Project: Apache Airflow > Issue Type: Sub-task >Reporter: Fokko Driesprong >Assignee: zhongjiajie >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3623) Support download log file from UI
[ https://issues.apache.org/jira/browse/AIRFLOW-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803866#comment-16803866 ] zhongjiajie commented on AIRFLOW-3623: -- Download Log only work in `tree` view but not in `graph` view. [~ash] > Support download log file from UI > - > > Key: AIRFLOW-3623 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3623 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Labels: newbie > Fix For: 1.10.3 > > > for some large log files, it is not a good idea to fetch and render in the > UI. Adding the ability to let users to download by try_number in the dag > modal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4111) Auto config airflow_db to database in initdb
[
https://issues.apache.org/jira/browse/AIRFLOW-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-4111.
Resolution: Won't Fix
Close by XD-DENG review. reason as below:
I think it's changing the behaviour, and may cause concern (in my person
opinion): Users have been used to the behaviour that all the default
connections are somehow "fake" and they know that they need to go there
configure explicitly.
Another example: I may be using a {{read+write}} DB account for
{{sql_alchemy_conn}} ({{write}} permission must be provided), while I will use
a {{read_only}} DB account for {{airflow_db}}. But your change will use the
{{read+write}} DB account info in {{sql_alchemy_conn}} to configure
{{airflow_db}}, which should not happen by default.
IMO, the concerns this PR brings > the pros it brings (also given that {{Data
profiling/Ad Hoc Query/Known Events}} are all deprecated). Personally I would
not merge it.
Hope this clarifies.
> Auto config airflow_db to database in initdb
>
>
> Key: AIRFLOW-4111
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4111
> Project: Apache Airflow
> Issue Type: Improvement
> Components: db
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Minor
> Labels: features
> Fix For: 1.10.3
>
>
> Make `airflow initdb` auto configure `airflow_db` by reading and parsing
> alchemy url in `airflow.cfg` section `[core]` param `sql_alchemy_conn`.
>
> `airflow_db` work on UI `Data profiling/Ad Hoc Query` and
> `Data profiling/Known Events` but using default constant values
> which not user friendly, specially `Data profiling/Known Events`
> use conn_id `airflow_db` as example.
>
> And as we know that `airflow_db`, airflow backend database, are
> already define in `airflow.cfg` section `[core]` paratemeter
> `sql_alchemy_conn` but alchemy url type, this Jira ticket help us
> make it real.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-4122: - Description: I had submit [PR|[https://github.com/apache/airflow/pull/4779]] and it related to Jira ticket . In PR comment, I was suggest close this PR and submit new one to remove function `chain` in `airflow.utils.heplers`, due to Airflower think use bit operation like `>>` or `<<` visual and easier to explain, and have multiple ways is confusion to users. BTW, this function was created 3 year ago and never get updated. was: I had submit [PR-4779|[https://github.com/apache/airflow/pull/4779]] and it related to [Jira ticket|https://issues.apache.org/jira/browse/AIRFLOW-3958] . In PR comment, I was suggest close this PR and submit new one to remove function `chain` in `airflow.utils.heplers`, due to Airflower think use bit operation like `>>` or `<<` visual and easier to explain, and have multiple ways is confusion to users. BTW, this function was created 3 year ago and never get updated. > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [PR|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4122) Remove chain function in airflow.utils
[ https://issues.apache.org/jira/browse/AIRFLOW-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-4122: - Description: I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it related to Jira ticket . In PR comment, I was suggest close this PR and submit new one to remove function `chain` in `airflow.utils.heplers`, due to Airflower think use bit operation like `>>` or `<<` visual and easier to explain, and have multiple ways is confusion to users. BTW, this function was created 3 year ago and never get updated. was: I had submit [PR|[https://github.com/apache/airflow/pull/4779]] and it related to Jira ticket . In PR comment, I was suggest close this PR and submit new one to remove function `chain` in `airflow.utils.heplers`, due to Airflower think use bit operation like `>>` or `<<` visual and easier to explain, and have multiple ways is confusion to users. BTW, this function was created 3 year ago and never get updated. > Remove chain function in airflow.utils > -- > > Key: AIRFLOW-4122 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: patch > Fix For: 1.10.3 > > > I had submit [pr|[https://github.com/apache/airflow/pull/4779]] and it > related to Jira ticket . > In PR comment, I was suggest close this PR and submit new one to remove > function `chain` in `airflow.utils.heplers`, due to Airflower think use bit > operation like `>>` or `<<` visual and easier to explain, and have multiple > ways is confusion to users. BTW, this function was created 3 year ago and > never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3958) Support list type as upstream in helper.chain
[
https://issues.apache.org/jira/browse/AIRFLOW-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3958.
Resolution: Won't Fix
Fix Version/s: (was: 1.10.3)
1.10.2
As suggest by committer and contributor, I close this PR and submit new one to
remove funciton `chain` in `airflow.utils.helpers`, see more information in
[this ticket|https://issues.apache.org/jira/browse/AIRFLOW-4122]
> Support list type as upstream in helper.chain
> -
>
> Key: AIRFLOW-3958
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3958
> Project: Apache Airflow
> Issue Type: Improvement
> Components: utils
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: patch
> Fix For: 1.10.2
>
>
> For now on, `airflow.utils.helpers.chain` only support list as downstream,
> but not upstream. This ticket is to support list as upstream.
> Only support
> {code:java}
> task_1 --> task_2 --> task_3 --> task_4
> or
> /--> task_3
> task_1 --> task_2
> \--> task_4{code}
> This improvement to be support
> {code:java}
> /--> task_3 \
> task_1 --> task_2 --> task_5
> \--> task_4 /
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4122) Remove chain function in airflow.utils
zhongjiajie created AIRFLOW-4122: Summary: Remove chain function in airflow.utils Key: AIRFLOW-4122 URL: https://issues.apache.org/jira/browse/AIRFLOW-4122 Project: Apache Airflow Issue Type: Improvement Components: utils Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 I had submit [PR-4779|[https://github.com/apache/airflow/pull/4779]] and it related to [Jira ticket|https://issues.apache.org/jira/browse/AIRFLOW-3958] . In PR comment, I was suggest close this PR and submit new one to remove function `chain` in `airflow.utils.heplers`, due to Airflower think use bit operation like `>>` or `<<` visual and easier to explain, and have multiple ways is confusion to users. BTW, this function was created 3 year ago and never get updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4112) Remove beeline_default as default connection
zhongjiajie created AIRFLOW-4112: Summary: Remove beeline_default as default connection Key: AIRFLOW-4112 URL: https://issues.apache.org/jira/browse/AIRFLOW-4112 Project: Apache Airflow Issue Type: Improvement Components: db Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 We use both `beeline_default` and `hive_cli_default` as our default when we run `airflow initdb`. But in airflow source part we only use `hive_cli_default` as conn_id in hive related hook/operator and only use `beeline_default` in airflow test part for test hive hook/operator. That why I think we should merge then as one default connection. And in [Hive doc|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli#LanguageManualCli-DeprecationinfavorofBeelineCLI] could know that hive cli will be deprecation in favor of Beeline, so in this situation I think we should reomve 'beeline_default' and change 'hive_cli_default' as same configure as 'beeline_default'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4111) Auto config airflow_db to database in initdb
zhongjiajie created AIRFLOW-4111: Summary: Auto config airflow_db to database in initdb Key: AIRFLOW-4111 URL: https://issues.apache.org/jira/browse/AIRFLOW-4111 Project: Apache Airflow Issue Type: Improvement Components: db Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 Make `airflow initdb` auto configure `airflow_db` by reading and parsing alchemy url in `airflow.cfg` section `[core]` param `sql_alchemy_conn`. `airflow_db` work on UI `Data profiling/Ad Hoc Query` and `Data profiling/Known Events` but using default constant values which not user friendly, specially `Data profiling/Known Events` use conn_id `airflow_db` as example. And as we know that `airflow_db`, airflow backend database, are already define in `airflow.cfg` section `[core]` paratemeter `sql_alchemy_conn` but alchemy url type, this Jira ticket help us make it real. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4076) Correct beeline_default port type in initdb function
zhongjiajie created AIRFLOW-4076:
Summary: Correct beeline_default port type in initdb function
Key: AIRFLOW-4076
URL: https://issues.apache.org/jira/browse/AIRFLOW-4076
Project: Apache Airflow
Issue Type: Improvement
Components: db
Affects Versions: 1.10.2
Reporter: zhongjiajie
Assignee: zhongjiajie
Fix For: 1.10.3
`beeline_default` conn_id create with command `airflow initdb` use String as
port type
{code:java}
merge_conn(
Connection(
conn_id='beeline_default', conn_type='beeline', port="1", <== HERE
host='localhost', extra="{\"use_beeline\": true, \"auth\": \"\"}",
schema='default'))
{code}
but `airflow.models.connection` use int type to store port
{code:java}
class Connection(Base, LoggingMixin):
__tablename__ = "connection"
id = Column(Integer(), primary_key=True)
conn_id = Column(String(ID_LEN))
conn_type = Column(String(500))
host = Column(String(500))
schema = Column(String(500))
login = Column(String(500))
_password = Column('password', String(5000))
port = Column(Integer()) <== HERE
is_encrypted = Column(Boolean, unique=False, default=False)
is_extra_encrypted = Column(Boolean, unique=False, default=False)
_extra = Column('extra', String(5000))
{code}
It's could be transfer str to int, but I think is better to pass int values
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4062) Clear install extra package command
[
https://issues.apache.org/jira/browse/AIRFLOW-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-4062:
-
Summary: Clear install extra package command (was: Clear install package
command)
> Clear install extra package command
> ---
>
> Key: AIRFLOW-4062
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4062
> Project: Apache Airflow
> Issue Type: Improvement
> Components: docs
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.3
>
>
> Some of our docs using
> {code:java}
> pip install apache-airflow[devl]{code}
> I think we should clear the install command, change to
> {code:java}
> pip install 'apache-airflow[devl]'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4062) Clear install package command
[
https://issues.apache.org/jira/browse/AIRFLOW-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-4062:
-
Flags: Patch
Labels: easyfix (was: )
Description:
Some of our docs using
{code:java}
pip install apache-airflow[devl]{code}
I think we should clear the install command, change to
{code:java}
pip install 'apache-airflow[devl]'{code}
was:
Some of our docs using
{code:java}
pip install apache-airflow[devl]{code}
> Clear install package command
> -
>
> Key: AIRFLOW-4062
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4062
> Project: Apache Airflow
> Issue Type: Improvement
> Components: docs
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Minor
> Labels: easyfix
> Fix For: 1.10.3
>
>
> Some of our docs using
> {code:java}
> pip install apache-airflow[devl]{code}
> I think we should clear the install command, change to
> {code:java}
> pip install 'apache-airflow[devl]'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4062) Clear install package command
zhongjiajie created AIRFLOW-4062:
Summary: Clear install package command
Key: AIRFLOW-4062
URL: https://issues.apache.org/jira/browse/AIRFLOW-4062
Project: Apache Airflow
Issue Type: Improvement
Components: docs
Affects Versions: 1.10.2
Reporter: zhongjiajie
Assignee: zhongjiajie
Fix For: 1.10.3
Some of our docs using
{code:java}
pip install apache-airflow[devl]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4061) Remove incubator in ci docker-compose
zhongjiajie created AIRFLOW-4061: Summary: Remove incubator in ci docker-compose Key: AIRFLOW-4061 URL: https://issues.apache.org/jira/browse/AIRFLOW-4061 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 We still use `airflowci/incubator-airflow-ci:latest` in our ci docker-compose, should change it to `airflowci/airflow-ci:latest`, in this case we have to change the setting for [airflow-ci|https://github.com/apache/airflow-ci] docker-hub setting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1526) Add dingding opeartor
[ https://issues.apache.org/jira/browse/AIRFLOW-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-1526: Assignee: zhongjiajie > Add dingding opeartor > - > > Key: AIRFLOW-1526 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1526 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.8.0 >Reporter: cocopc >Assignee: zhongjiajie >Priority: Major > Labels: features > > h4. This operator implements send message to > dingding(https://www.dingtalk.com/),like slack. > h5. usage scenario example: > * on task fail, send message to dingding group,rather than email. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4036) Add DingDing hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-4036. Resolution: Duplicate Duplicate with http://issues.apache.org/jira/browse/AIRFLOW-1526 > Add DingDing hook and operator > -- > > Key: AIRFLOW-4036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4036 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: patch > Fix For: 1.10.3 > > > *DingDing* is a popular business talk software in China, like *Slack*, It > support send message using webhook, This Jira ticket is to add DingDing hook > and operator to Airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4036) Add DingDing hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786714#comment-16786714 ] zhongjiajie commented on AIRFLOW-4036: -- get it > Add DingDing hook and operator > -- > > Key: AIRFLOW-4036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4036 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: patch > Fix For: 1.10.3 > > > *DingDing* is a popular business talk software in China, like *Slack*, It > support send message using webhook, This Jira ticket is to add DingDing hook > and operator to Airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4036) Add DingDing hook and operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786703#comment-16786703 ] zhongjiajie commented on AIRFLOW-4036: -- [~jackjack10] You are right, should I PR base on https://issues.apache.org/jira/browse/AIRFLOW-1526 or in https://issues.apache.org/jira/browse/AIRFLOW-4036 ? > Add DingDing hook and operator > -- > > Key: AIRFLOW-4036 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4036 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: patch > Fix For: 1.10.3 > > > *DingDing* is a popular business talk software in China, like *Slack*, It > support send message using webhook, This Jira ticket is to add DingDing hook > and operator to Airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4036) Add DingDing hook and operator
zhongjiajie created AIRFLOW-4036: Summary: Add DingDing hook and operator Key: AIRFLOW-4036 URL: https://issues.apache.org/jira/browse/AIRFLOW-4036 Project: Apache Airflow Issue Type: New Feature Components: hooks Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 *DingDing* is a popular business talk software in China, like *Slack*, It support send message using webhook, This Jira ticket is to add DingDing hook and operator to Airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3981) Make Airflow UI timezone aware
[ https://issues.apache.org/jira/browse/AIRFLOW-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781842#comment-16781842 ] zhongjiajie commented on AIRFLOW-3981: -- If no one assignee when I finish my other PR in hand. I will do it. > Make Airflow UI timezone aware > -- > > Key: AIRFLOW-3981 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3981 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3958) Support list type as upstream in helper.chain
zhongjiajie created AIRFLOW-3958:
Summary: Support list type as upstream in helper.chain
Key: AIRFLOW-3958
URL: https://issues.apache.org/jira/browse/AIRFLOW-3958
Project: Apache Airflow
Issue Type: Improvement
Components: utils
Affects Versions: 1.10.2
Reporter: zhongjiajie
Assignee: zhongjiajie
Fix For: 1.10.3
For now on, `airflow.utils.helpers.chain` only support list as downstream, but
not upstream. This ticket is to support list as upstream.
Only support
{code:java}
task_1 --> task_2 --> task_3 --> task_4
or
/--> task_3
task_1 --> task_2
\--> task_4{code}
This improvement to be support
{code:java}
/--> task_3 \
task_1 --> task_2 --> task_5
\--> task_4 /
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3733) raise NameError if hql result had no record
[ https://issues.apache.org/jira/browse/AIRFLOW-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie closed AIRFLOW-3733. Fix > raise NameError if hql result had no record > --- > > Key: AIRFLOW-3733 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3733 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Affects Versions: 1.10.1 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Labels: easyfix > Fix For: 1.10.3 > > > [to_csv|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/hive_hooks.py#L881-L903] > function will raise `NameError` while hql result had not record, we should > init variable i like > [oracle_hooks|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/oracle_hook.py#L92-L124] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3875) simplify SlackWebhookHook code and change docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie updated AIRFLOW-3875: - Fix Version/s: (was: 2.0.0) 1.10.3 > simplify SlackWebhookHook code and change docstring > --- > > Key: AIRFLOW-3875 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3875 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.10.2 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Minor > Labels: easyfix > Fix For: 1.10.3 > > > SlackWebhookHook __init__ function do not need self.http_conn_id, and > docstring missing some necessary str -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3741) oracle_hook add extra config
[
https://issues.apache.org/jira/browse/AIRFLOW-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3741.
> oracle_hook add extra config
>
>
> Key: AIRFLOW-3741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3741
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: features
> Fix For: 1.10.3
>
>
> oracle_hook get_conn block just like below
>
> {code:java}
> if dsn and sid and not service_name:
> dsn = cx_Oracle.makedsn(dsn, conn.port, sid)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> elif dsn and service_name and not sid:
> dsn = cx_Oracle.makedsn(dsn, conn.port, service_name=service_name)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> else:
> conn = cx_Oracle.connect(conn.login, conn.password, conn.host){code}
>
> oracle_hook only support default encoding, not UTF-8.
> In cx_oracle [official
> doc|https://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.connect]
> and [github issue|https://github.com/oracle/python-cx_Oracle/issues/157] we
> know how to set connection encoding. I think is necessary add this feature to
> oracle_hook
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3741) oracle_hook add extra config
[
https://issues.apache.org/jira/browse/AIRFLOW-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-3741:
-
Summary: oracle_hook add extra config (was: oracle_hook not support
specific encoding)
> oracle_hook add extra config
>
>
> Key: AIRFLOW-3741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3741
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: features
> Fix For: 1.10.3
>
>
> oracle_hook get_conn block just like below
>
> {code:java}
> if dsn and sid and not service_name:
> dsn = cx_Oracle.makedsn(dsn, conn.port, sid)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> elif dsn and service_name and not sid:
> dsn = cx_Oracle.makedsn(dsn, conn.port, service_name=service_name)
> conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
> else:
> conn = cx_Oracle.connect(conn.login, conn.password, conn.host){code}
>
> oracle_hook only support default encoding, not UTF-8.
> In cx_oracle [official
> doc|https://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.connect]
> and [github issue|https://github.com/oracle/python-cx_Oracle/issues/157] we
> know how to set connection encoding. I think is necessary add this feature to
> oracle_hook
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3734) function `load_file` will not run if param `partition` is None
[
https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie updated AIRFLOW-3734:
-
Fix Version/s: (was: 2.0.0)
1.10.3
> function `load_file` will not run if param `partition` is None
> --
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH OVERWRITE INTO TABLE \n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from
> `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3734) function `load_file` will not run if param `partition` is None
[
https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3734.
Fix
> function `load_file` will not run if param `partition` is None
> --
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
>Affects Versions: 1.10.1
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH OVERWRITE INTO TABLE \n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from
> `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3903) simplify function get_parser dict comprehension in cli.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhongjiajie closed AIRFLOW-3903.
Resolution: Won't Fix
Fix Version/s: (was: 1.10.3)
1.10.2
I finally find out the reason, Is not a bug
> simplify function get_parser dict comprehension in cli.py
> -
>
> Key: AIRFLOW-3903
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3903
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Minor
> Labels: patch
> Fix For: 1.10.2
>
>
> airflow/cli.py funciton *get_parser* dict comprehension have useless code
> {code:java}
> and getattr(arg, f){code}
> I think, this Jira ticket remove that part
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3903) simplify function get_parser dict comprehension in cli.py
[
https://issues.apache.org/jira/browse/AIRFLOW-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772624#comment-16772624
]
zhongjiajie commented on AIRFLOW-3903:
--
[~kamil.bregula] Done, sorry to close late.
> simplify function get_parser dict comprehension in cli.py
> -
>
> Key: AIRFLOW-3903
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3903
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.10.2
>Reporter: zhongjiajie
>Assignee: zhongjiajie
>Priority: Minor
> Labels: patch
> Fix For: 1.10.2
>
>
> airflow/cli.py funciton *get_parser* dict comprehension have useless code
> {code:java}
> and getattr(arg, f){code}
> I think, this Jira ticket remove that part
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3903) simplify function get_parser dict comprehension in cli.py
zhongjiajie created AIRFLOW-3903:
Summary: simplify function get_parser dict comprehension in cli.py
Key: AIRFLOW-3903
URL: https://issues.apache.org/jira/browse/AIRFLOW-3903
Project: Apache Airflow
Issue Type: Improvement
Components: cli
Affects Versions: 1.10.2
Reporter: zhongjiajie
Assignee: zhongjiajie
Fix For: 1.10.3
airflow/cli.py funciton *get_parser* dict comprehension have useless code
{code:java}
and getattr(arg, f){code}
I think, this Jira ticket remove that part
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3888) HA for metastore connection
[ https://issues.apache.org/jira/browse/AIRFLOW-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767886#comment-16767886 ] zhongjiajie commented on AIRFLOW-3888: -- [~jakub.guzik] you could submit a PR to Improving. > HA for metastore connection > --- > > Key: AIRFLOW-3888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3888 > Project: Apache Airflow > Issue Type: Improvement > Components: hive_hooks >Affects Versions: 1.10.1 > Environment: Linux Centos 7 >Reporter: Jakub Guzik >Priority: Major > Labels: high-availability, hive, metastore, patch > Fix For: 1.10.1 > > Attachments: hive_hook.py > > Original Estimate: 24h > Remaining Estimate: 24h > > Creating a connection to a metasotor with two hosts for high avitablity (eg > connection 1, connection 2) is not possible because the entire value entered > is taken. For our needs, it is necessary to go through subsequent hosts and > connect to the first working. > This change allows you to check and then connect to a working metastor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3881) Correct hive_hook to_csv row number
zhongjiajie created AIRFLOW-3881: Summary: Correct hive_hook to_csv row number Key: AIRFLOW-3881 URL: https://issues.apache.org/jira/browse/AIRFLOW-3881 Project: Apache Airflow Issue Type: Bug Components: hive_hooks Affects Versions: 1.10.2 Reporter: zhongjiajie Assignee: zhongjiajie Fix For: 1.10.3 hive_hooks to_csv function had wrong row number each path so far, this Jira ticket is correct row number -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3767) oracle hook bulk insert error while target_fields is DEFAULT values
zhongjiajie created AIRFLOW-3767:
Summary: oracle hook bulk insert error while target_fields is
DEFAULT values
Key: AIRFLOW-3767
URL: https://issues.apache.org/jira/browse/AIRFLOW-3767
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.2
Reporter: zhongjiajie
Assignee: zhongjiajie
{code:java}
def bulk_insert_rows(self, table, rows, target_fields=None, commit_every=5000):
conn = self.get_conn()
cursor = conn.cursor()
values = ', '.join(':%s' % i for i in range(1, len(target_fields) + 1))
prepared_stm = 'insert into {tablename} ({columns}) values
({values})'.format(
tablename=table,
columns=', '.join(target_fields),
values=values,
)
{code}
*target_fields* default *None* will leads
{code:java}
len(target_fields)
{code}
and
{code:java}
', '.join(target_fields)
{code}
fail
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2697) Drop snakebite in favour of hdfs3
[ https://issues.apache.org/jira/browse/AIRFLOW-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748370#comment-16748370 ] zhongjiajie commented on AIRFLOW-2697: -- Awesome I would like to see this moving forward. > Drop snakebite in favour of hdfs3 > - > > Key: AIRFLOW-2697 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2697 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: 1.9.0 >Reporter: Julian de Ruiter >Assignee: Julian de Ruiter >Priority: Major > > The current HdfsHook relies on the snakebite library, which is unfortunately > not compatible with Python 3. To add Python 3 support for the HdfsHook > requires switching to a different library for interacting with HDFS. The > hdfs3 library is an attractive alternative, as it supports Python 3 and seems > to be stable and relatively well supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3741) oracle_hook not support specific encoding
zhongjiajie created AIRFLOW-3741:
Summary: oracle_hook not support specific encoding
Key: AIRFLOW-3741
URL: https://issues.apache.org/jira/browse/AIRFLOW-3741
Project: Apache Airflow
Issue Type: New Feature
Components: hooks
Affects Versions: 1.10.1
Reporter: zhongjiajie
Assignee: zhongjiajie
oracle_hook get_conn block just like below
{code:java}
if dsn and sid and not service_name:
dsn = cx_Oracle.makedsn(dsn, conn.port, sid)
conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
elif dsn and service_name and not sid:
dsn = cx_Oracle.makedsn(dsn, conn.port, service_name=service_name)
conn = cx_Oracle.connect(conn.login, conn.password, dsn=dsn)
else:
conn = cx_Oracle.connect(conn.login, conn.password, conn.host){code}
oracle_hook only support default encoding, not UTF-8.
In cx_oracle [official
doc|https://cx-oracle.readthedocs.io/en/latest/module.html#cx_Oracle.connect]
and [github issue|https://github.com/oracle/python-cx_Oracle/issues/157] we
know how to set connection encoding. I think is necessary add this feature to
oracle_hook
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3733) raise NameError if hql result had no record
zhongjiajie created AIRFLOW-3733: Summary: raise NameError if hql result had no record Key: AIRFLOW-3733 URL: https://issues.apache.org/jira/browse/AIRFLOW-3733 Project: Apache Airflow Issue Type: Bug Components: hive_hooks Affects Versions: 1.10.1 Reporter: zhongjiajie Assignee: zhongjiajie [to_csv|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/hive_hooks.py#L881-L903] function will raise `NameError` while hql result had not record, we should init variable i like [oracle_hooks|https://github.com/apache/airflow/blob/233f6c750c543a669c5319c7167a68a1f5b33c16/airflow/hooks/oracle_hook.py#L92-L124] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3728) double comma in function `to_csv` in hive_hooks.py
zhongjiajie created AIRFLOW-3728: Summary: double comma in function `to_csv` in hive_hooks.py Key: AIRFLOW-3728 URL: https://issues.apache.org/jira/browse/AIRFLOW-3728 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.1 Reporter: zhongjiajie Assignee: zhongjiajie In master branch, hive_hooks function `to_csv` had [double comma|[https://github.com/apache/airflow/blob/0bd6d87e37f55a49768d30110d675e84a763103d/airflow/hooks/hive_hooks.py#L867],] I think we should fix it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3714) Quickstart references outdated DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhongjiajie reassigned AIRFLOW-3714: Assignee: zhongjiajie > Quickstart references outdated DAG > -- > > Key: AIRFLOW-3714 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3714 > Project: Apache Airflow > Issue Type: Bug > Components: Documentation >Reporter: Steve Greenberg >Assignee: zhongjiajie >Priority: Minor > > [https://airflow.apache.org/start.html] mentions > > "You should be able to see the status of the jobs change in the example1 DAG > as you run the commands below." > > There is no example1 DAG. This should probably say "example_bash_operator" > instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3643) Add shebang to docs/start_doc_server.sh
[
https://issues.apache.org/jira/browse/AIRFLOW-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743763#comment-16743763
]
zhongjiajie commented on AIRFLOW-3643:
--
Have anybody want to fix this issue?
> Add shebang to docs/start_doc_server.sh
> ---
>
> Key: AIRFLOW-3643
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3643
> Project: Apache Airflow
> Issue Type: Improvement
> Components: docs
>Reporter: Kengo Seki
>Priority: Minor
> Labels: easyfix, newbie
>
> Since this script uses bash specific syntax such as {{$()}}, {{#!/bin/bash}}
> should be added to the first line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
