[jira] [Resolved] (AIRFLOW-2734) Resolve setuptools normalized_version warning
[
https://issues.apache.org/jira/browse/AIRFLOW-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-2734.
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request #3588
[https://github.com/apache/incubator-airflow/pull/3588]
> Resolve setuptools normalized_version warning
> -
>
> Key: AIRFLOW-2734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2734
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Taylor Edmiston
>Assignee: Taylor Edmiston
>Priority: Minor
> Fix For: 2.0.0
>
>
> When running Airflow's setup.py, setuptools throws a warning about the
> version string not having a . between the version number and the dev suffix.
>
> {code}
> $ python setup.py
>
> /Users/taylor/.local/share/virtualenvs/airflow-no-hooks/lib/python3.6/site-packages/setuptools/dist.py:398:
> UserWarning: Normalizing '2.0.0dev0+incubating' to '2.0.0.dev0+incubating'
> normalized_version,
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> or: setup.py --help [cmd1 cmd2 ...]
> or: setup.py --help-commands
> or: setup.py cmd --help
> {code}
>
> The PR for this issue resolves this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2267) Implement DAG-level access for airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-2267. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3197 [https://github.com/apache/incubator-airflow/pull/3197] > Implement DAG-level access for airflow > -- > > Key: AIRFLOW-2267 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2267 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 2.0.0 > > > Now we have a new airflow UI based on RBAC, we would like to implement per > DAG-level access on top of it. I have started some initial discussions with > Max @mistercrunch, and will come up with a proposal shortly to gather more > feedback from the community. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2566) Change backfill to rerun failed tasks instead of throwing exception
[ https://issues.apache.org/jira/browse/AIRFLOW-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-2566. Resolution: Fixed Fix Version/s: 1.10.0 Issue resolved by pull request #3464 [https://github.com/apache/incubator-airflow/pull/3464] > Change backfill to rerun failed tasks instead of throwing exception > --- > > Key: AIRFLOW-2566 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2566 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2558) Clear TASK/DAG is clearing all executions
[
https://issues.apache.org/jira/browse/AIRFLOW-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-2558.
Resolution: Fixed
Fix Version/s: 1.10.0
Issue resolved by pull request #3465
[https://github.com/apache/incubator-airflow/pull/3465]
> Clear TASK/DAG is clearing all executions
> -
>
> Key: AIRFLOW-2558
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2558
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Marcos Bernardelli
>Assignee: Tao Feng
>Priority: Major
> Fix For: 1.10.0
>
>
> When I try to clear a DAG/TASK specific execution, the Airflow try to execute
> all the past executions:
> [Animeted
> GIF|https://gist.githubusercontent.com/bern4rdelli/34c1e57acd53c8c67417604202f3e0e6/raw/4bcb3d3c23f2a3bb7f7bfb3e977d935e5bb9f0ee/clear.gif]
> (I failed miserable trying to attache the animated GIF :()
>
> This behavior was changed here:
> [https://github.com/apache/incubator-airflow/pull/3444]
> The old version looks like this:
> {code:python}
> drs = session.query(DagModel).filter_by(dag_id=self.dag_id).all()
> {code}
> Then it's changed to:
> {code:python}
> drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
> {code}
> This new query (using DagRun) get all the past executions, even when the
> "Past" button is not checked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2526) params can be overridden through CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-2526.
Resolution: Fixed
> params can be overridden through CLI
>
>
> Key: AIRFLOW-2526
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2526
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> The idea is that the values in macros can be overridden through CLI.
> Given a dag like this:
> {code:java}
> dag = DAG('hello_world')
> templated_command = """
> echo "text = {{ params.text }}"
> """
> bash_operator = BashOperator(
> task_id='bash_task',
> bash_command=templated_command,
> dag=dag,
> params={"text": "this text should be overridden"}
> )
> {code}
> running
> {code:java}
> airflow trigger_dag hello_world -c "{"text": "overridden successfully"}"
> {code}
> should print
> {code}
> overridden successfully
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2517) backfill support passing key values through CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-2517.
Resolution: Fixed
> backfill support passing key values through CLI
> ---
>
> Key: AIRFLOW-2517
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2517
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Chao-Han Tsai
>Assignee: Chao-Han Tsai
>Priority: Major
>
> In backfill, we can provide key-value pairs through CLI and those pairs can
> be accessed through macros. This is just like the way `trigger_dag -c` works
> [1].
> Let's walk through an example.
> In the airflow CLI we specify a key-value pair.
> {code:java}
> airflow backfill hello_world -s 2018-02-01 -e 2018-02-08 -c ' {"text": "some
> text"}
> {code}
> In the DAG file, I have a `BashOperator` that contains a template command and
> I want
> \{{ dag_run.conf.text }} resolves to the text I passed in CLI.
> {code:java}
> templated_command = """
> echo "ds = {{ ds }}"
> echo "prev_ds = {{ macros.datetime.strftime(prev_execution_date, "%Y-%m-%d")
> }}"
> echo "next_ds = {{ macros.datetime.strftime(next_execution_date, "%Y-%m-%d")
> }}"
> echo "text_through_conf = {{ dag_run.conf.text }}"
> """
> bash_operator = BashOperator(
> task_id='bash_task',
> bash_command=templated_command,
> dag=dag
> )
> {code}
> [1] [https://airflow.apache.org/cli.html#trigger_dag]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2537) clearing tasks shouldn't set backfill DAG runs to `running`
Maxime Beauchemin created AIRFLOW-2537: -- Summary: clearing tasks shouldn't set backfill DAG runs to `running` Key: AIRFLOW-2537 URL: https://issues.apache.org/jira/browse/AIRFLOW-2537 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2521) backfill - make variable name and logging messages more accurate
Maxime Beauchemin created AIRFLOW-2521: -- Summary: backfill - make variable name and logging messages more accurate Key: AIRFLOW-2521 URL: https://issues.apache.org/jira/browse/AIRFLOW-2521 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin The term kicked_off in logging and the variable started are used to refer to `running` task instances. Let's clarify the variable names and messages here. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2520) [cli] backfill way too verbose
Maxime Beauchemin created AIRFLOW-2520: -- Summary: [cli] backfill way too verbose Key: AIRFLOW-2520 URL: https://issues.apache.org/jira/browse/AIRFLOW-2520 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin Used backfill recently and it would log a ton of logging messages telling me all the tasks that were not ready to run at every tick. These messages are not useful and should be muted by default. I understand that this may be helpful in the context of `airflow run` in the context where dependencies aren't met, so decided to manage a flag instead of simply going `logging.debug` on it. PR coming up -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1623) Clearing task in UI does not trigger on_kill method in operator
[ https://issues.apache.org/jira/browse/AIRFLOW-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1623: --- Fix Version/s: (was: Airflow 2.0) 1.10.0 > Clearing task in UI does not trigger on_kill method in operator > --- > > Key: AIRFLOW-1623 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1623 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 2.0 >Reporter: Richard Penman >Priority: Major > Fix For: 1.10.0 > > > When a task is cleared in the UI it doesn't call the [operators on_kill() > method|https://github.com/apache/incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/models.py#L2380] > to clean up the task. Apparently this is meant to be handled in the > [LocalTaskJob.on_kill()|https://github.com/apache/incubator-airflow/blob/b2e1753f5b74ad1b6e0889f7b784ce69623c95ce/airflow/jobs.py#L2512] > method, however it is not currently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-1460) Tasks get stuck in the "removed" state
[ https://issues.apache.org/jira/browse/AIRFLOW-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1460. Resolution: Fixed Assignee: Maxime Beauchemin (was: George Leslie-Waksman) > Tasks get stuck in the "removed" state > -- > > Key: AIRFLOW-1460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1460 > Project: Apache Airflow > Issue Type: Bug >Reporter: George Leslie-Waksman >Assignee: Maxime Beauchemin >Priority: Major > Fix For: 1.9.1 > > > The current handling of task instances that get assigned the state "removed" > is that they get ignored. > If the underlying task later gets re-added, the existing task instances will > prevent the task from running in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1460) Tasks get stuck in the "removed" state
[ https://issues.apache.org/jira/browse/AIRFLOW-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1460: --- Fix Version/s: 1.9.1 > Tasks get stuck in the "removed" state > -- > > Key: AIRFLOW-1460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1460 > Project: Apache Airflow > Issue Type: Bug >Reporter: George Leslie-Waksman >Assignee: George Leslie-Waksman >Priority: Major > Fix For: 1.9.1 > > > The current handling of task instances that get assigned the state "removed" > is that they get ignored. > If the underlying task later gets re-added, the existing task instances will > prevent the task from running in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts
Maxime Beauchemin created AIRFLOW-1698: -- Summary: Remove confusing SCHEDULER_RUNS env var from systemd scripts Key: AIRFLOW-1698 URL: https://issues.apache.org/jira/browse/AIRFLOW-1698 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin In the very early days, the Airflow scheduler needed to be restarted every so often to take new DAG_FOLDERS mutations into account properly. This is no longer needed, so I'm removing artifacts in the systemd scripts. I'm unclear whether daemon scripts belong here. I can see how it's useful to have this somewhere, but it sits on the edge of what should be in the repo and can get out-of-sync. As a result it ends up creating more confusion than help. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1525) Fix minor LICENSE & NOTICE issue
Maxime Beauchemin created AIRFLOW-1525: -- Summary: Fix minor LICENSE & NOTICE issue Key: AIRFLOW-1525 URL: https://issues.apache.org/jira/browse/AIRFLOW-1525 Project: Apache Airflow Issue Type: Improvement Reporter: Maxime Beauchemin Per Justin Mclean on the 1.8.2 [VOTE] thread: - year in NOTICE is incorrect - LICENSE is missing several things (see below) LICENSE is missing this BSD licensed file [1], this MIT licensed file [2] (and the MIT header) and license (and header) for normalize.css in [3]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1476) Add INSTALL file for source releases
Maxime Beauchemin created AIRFLOW-1476: -- Summary: Add INSTALL file for source releases Key: AIRFLOW-1476 URL: https://issues.apache.org/jira/browse/AIRFLOW-1476 Project: Apache Airflow Issue Type: Improvement Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1317) Fix minor issues in API reference
[
https://issues.apache.org/jira/browse/AIRFLOW-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-1317.
Resolution: Fixed
> Fix minor issues in API reference
> -
>
> Key: AIRFLOW-1317
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1317
> Project: Apache Airflow
> Issue Type: Bug
> Components: docs, Documentation
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>
> 1.
> https://airflow.incubator.apache.org/code.html#airflow.operators.HdfsSensor,
> https://airflow.incubator.apache.org/code.html#airflow.models.DAG and
> https://airflow.incubator.apache.org/code.html#airflow.models.TaskInstance
> Explanations for function parameters are not properly rendered.
> 2. https://airflow.incubator.apache.org/code.html#default-variables
> Some variables are not properly quoted by {{ }}.
> 3. https://airflow.incubator.apache.org/code.html#airflow.macros.random
> Standard library function random.random is exposed unnecessarily.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-836) The paused and queryview endpoints are vulnerable to CSRF
[ https://issues.apache.org/jira/browse/AIRFLOW-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-836: -- Fix Version/s: (was: 1.8.2) 1.9.0 > The paused and queryview endpoints are vulnerable to CSRF > - > > Key: AIRFLOW-836 > URL: https://issues.apache.org/jira/browse/AIRFLOW-836 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.9.0 > > > These endpoints use GET and are state-changing which is bad practice, and > allows CSRF -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1274) HttpSensor parameter params is overriding BaseOperator params and should be renamed
[
https://issues.apache.org/jira/browse/AIRFLOW-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1274:
---
Fix Version/s: (was: 1.8.2)
1.9.0
> HttpSensor parameter params is overriding BaseOperator params and should be
> renamed
> ---
>
> Key: AIRFLOW-1274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1274
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Alice Berard
>Assignee: Alice Berard
> Fix For: 1.9.0
>
>
> {{HttpSensor}} parameter {{params}} is overriding BaseOperator {{params}} and
> should be renamed.
> It is screwing DAG parsing:
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 264,
> in process_file
> m = imp.load_source(mod_name, filepath)
> File
> "/home/alice_berard/mydags/anomaly_detection/anomaly_detection/curiosity/curiosity_dags.py",
> line 14, in
> pool='curiosity',
> File
> "/home/alice_berard/mydags/anomaly_detection/anomaly_detection/curiosity/dag_builder.py",
> line 133, in build
> pool=self.pool
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/decorators.py",
> line 58, in wrapper
> dag_params.update(params)
> ValueError: dictionary update sequence element #0 has length 1; 2 is required
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1275) Fix `airflow pool` command exception
[
https://issues.apache.org/jira/browse/AIRFLOW-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1275:
---
Fix Version/s: (was: 1.8.2)
1.9.0
> Fix `airflow pool` command exception
>
>
> Key: AIRFLOW-1275
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1275
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.8.1
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.9.0
>
>
> {code}
> $ airflow pool
> [2017-06-04 19:07:49,805] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/Grammar.txt
> [2017-06-04 19:07:49,821] {driver.py:120} INFO - Generating grammar tables
> from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
> [2017-06-04 19:07:49,896] {__init__.py:45} INFO - Using executor
> SequentialExecutor
> Traceback (most recent call last):
> File "/home/stas/Work/incubator-airflow/.venv/bin/airflow", line 6, in
>
> exec(compile(open(__file__).read(), __file__, 'exec'))
> File "/home/stas/Work/incubator-airflow/airflow/bin/airflow", line 28, in
>
> args.func(args)
> File "/home/stas/Work/incubator-airflow/airflow/bin/cli.py", line 195, in
> pool
> .filter(Pool.pool == name)
> UnboundLocalError: local variable 'name' referenced before assignment
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1217) Enable logging in Sqoop hook
[ https://issues.apache.org/jira/browse/AIRFLOW-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1217: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Enable logging in Sqoop hook > > > Key: AIRFLOW-1217 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1217 > Project: Apache Airflow > Issue Type: Bug >Reporter: Fokko Driesprong > Fix For: 1.9.0 > > > Currently the sqoop hook does not give any output in Airflow which makes > debugging hard. We would like to change this and have proper logging from the > subprocess to airflow. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1221) Fix DatabricksSubmitRunOperator Templating
[ https://issues.apache.org/jira/browse/AIRFLOW-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1221: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Fix DatabricksSubmitRunOperator Templating > -- > > Key: AIRFLOW-1221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1221 > Project: Apache Airflow > Issue Type: Bug >Reporter: Andrew Chen > Fix For: 1.9.0 > > > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/databricks_operator.py#L207 > Bug where we mixed up index and content from enumerate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1213) Add hcatalog parameters to the sqoop operator/hook
[ https://issues.apache.org/jira/browse/AIRFLOW-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1213: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Add hcatalog parameters to the sqoop operator/hook > -- > > Key: AIRFLOW-1213 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1213 > Project: Apache Airflow > Issue Type: Bug >Reporter: Fokko Driesprong > Fix For: 1.9.0 > > > Hi, > For the Sqoop hook and operator we would like to add the hcatalog parameters > --hcatalog-database and --hcatalog-table. These are added in Sqoop 1.4.6: > https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html > This allows the user to import/export Hive tables instead of hdfs directories. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1191) Contrib Spark Submit hook should permit override of spark-submit cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1191: --- Fix Version/s: (was: 1.8.2) > Contrib Spark Submit hook should permit override of spark-submit cmd > > > Key: AIRFLOW-1191 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1191 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Affects Versions: 1.8.1 > Environment: Cloudera based Spark parcel >Reporter: Vianney FOUCAULT >Assignee: Vianney FOUCAULT > Fix For: 1.9.0 > > > Using Cloudera based cluster with spark 2 parcel that rename spark-submit to > spark2-submit > It should be possible to change the spark submit cmd without specifying a env > var -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1201) Update deprecated 'nose-parameterized' library to 'parameterized'
[
https://issues.apache.org/jira/browse/AIRFLOW-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1201:
---
Fix Version/s: (was: 1.8.2)
1.9.0
> Update deprecated 'nose-parameterized' library to 'parameterized'
> -
>
> Key: AIRFLOW-1201
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1201
> Project: Apache Airflow
> Issue Type: Bug
> Components: dependencies
>Affects Versions: Airflow 2.0
>Reporter: Stanislav Kudriashev
>Assignee: Stanislav Kudriashev
> Fix For: 1.9.0
>
>
> {code}
> /home/travis/build/apache/incubator-airflow/.tox/py27-cdh-airflow_backend_mysql/lib/python2.7/site-packages/nose_parameterized/__init__.py:7:
> UserWarning: The 'nose-parameterized' package has been renamed
> 'parameterized'. For the two step migration instructions, see:
> https://github.com/wolever/parameterized#migrating-from-nose-parameterized-to-parameterized
> (set NOSE_PARAMETERIZED_NO_WARN=1 to suppress this warning)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1197) Contrib Spark Submit Hook on_kill error
[ https://issues.apache.org/jira/browse/AIRFLOW-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1197: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Contrib Spark Submit Hook on_kill error > --- > > Key: AIRFLOW-1197 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1197 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.8.1 >Reporter: Vianney FOUCAULT > Fix For: 1.9.0 > > > the on_kill method is not ok, this needs to be fixed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1081) Task duration page is slow
[ https://issues.apache.org/jira/browse/AIRFLOW-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1081: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Task duration page is slow > -- > > Key: AIRFLOW-1081 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1081 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.9.0 > > > It makes a number of queries proportional to the data size, instead of just 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1047) Airflow logs vulnerable to XSS
[ https://issues.apache.org/jira/browse/AIRFLOW-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1047: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Airflow logs vulnerable to XSS > -- > > Key: AIRFLOW-1047 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1047 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.9.0 > > > Navigating to a page with dag_id param specified as a html tag leads to that > tag being rendered due to using Markup tag (which makes html be labeled as > safe) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1136) Invalid parameters are not captured for Sqoop operators
[ https://issues.apache.org/jira/browse/AIRFLOW-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1136: --- Fix Version/s: (was: 1.8.2) > Invalid parameters are not captured for Sqoop operators > --- > > Key: AIRFLOW-1136 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1136 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.8.1 >Reporter: Henk Griffioen >Assignee: Henk Griffioen >Priority: Minor > Fix For: 1.9.0 > > > 1. The Sqoophook exports data as text if argument file_type is anything other > than 'avro', 'sequence' or 'parquet'. Correct behaviour would be to error out > if file_type is not 'avro', 'sequence', 'parquet' or 'text'. > 2. SqoopOperator runs the import command if argument cmd_type is anything > other than 'export'. Correct behaviour would be to error out if cmd_type is > not 'import' or 'export'. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1007) Jinja sandbox is vulnerable to RCE
[ https://issues.apache.org/jira/browse/AIRFLOW-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1007: --- Fix Version/s: (was: 1.8.2) 1.9.0 > Jinja sandbox is vulnerable to RCE > -- > > Key: AIRFLOW-1007 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1007 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.9.0 > > > Right now, the jinja template functionality in chart_data takes arbitrary > strings and executes them. We should use the sandbox functionality to prevent > this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-945) Revert psycopg2 workaround when psycopg2 2.7.1 is released
[ https://issues.apache.org/jira/browse/AIRFLOW-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-945: -- Fix Version/s: (was: 1.8.2) 1.9.0 > Revert psycopg2 workaround when psycopg2 2.7.1 is released > -- > > Key: AIRFLOW-945 > URL: https://issues.apache.org/jira/browse/AIRFLOW-945 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Affects Versions: 1.8.0rc5 >Reporter: Bolke de Bruin > Fix For: 1.9.0 > > > A workaround around for https://github.com/psycopg/psycopg2/issues/517 is > present. We can revert this change when 2.7.1 is out. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1041) DockerOperator replaces its xcom_push method with a boolean
[ https://issues.apache.org/jira/browse/AIRFLOW-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1041: --- Fix Version/s: (was: 1.8.2) 1.9.0 > DockerOperator replaces its xcom_push method with a boolean > --- > > Key: AIRFLOW-1041 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1041 > Project: Apache Airflow > Issue Type: Bug > Components: docker >Affects Versions: 1.8.0 >Reporter: Ludovic Claude > Fix For: 1.9.0 > > > xcom_push method is not usable from DockerOperator as it is replaced by a > boolean value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-935) Impossible to use plugin executors
[ https://issues.apache.org/jira/browse/AIRFLOW-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-935: -- Fix Version/s: (was: 1.8.2) 1.9.0 > Impossible to use plugin executors > -- > > Key: AIRFLOW-935 > URL: https://issues.apache.org/jira/browse/AIRFLOW-935 > Project: Apache Airflow > Issue Type: Bug > Components: executor, plugins >Affects Versions: Airflow 1.8, 1.9.0 >Reporter: Stijn Verhaegen > Fix For: 1.9.0 > > > revision eb5982d (included in 1.8) breaks plugin executors altogether. It > makes a new module for every plugin, so import statements need to be adapted, > but the executor selection is left unchanged, so it ends up assigning the > plugin module as an executor. > To reproduce: take any plugin which defines a custom executor and try to get > it loaded by setting `executor` in the airflow.cfg -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-860) Circular module dependency prevents loading of custom executor plugin
[
https://issues.apache.org/jira/browse/AIRFLOW-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-860:
--
Fix Version/s: (was: 1.8.2)
1.9.0
> Circular module dependency prevents loading of custom executor plugin
> -
>
> Key: AIRFLOW-860
> URL: https://issues.apache.org/jira/browse/AIRFLOW-860
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor, plugins
>Affects Versions: Airflow 1.8
> Environment: Linux RHEL 7.6, Python 2.7.13
>Reporter: Nick Ivanov
> Fix For: 1.9.0
>
>
> Custom plugins cannot be loaded, which prevents airflow from running, due to
> apparent cyclic dependency in plugins_manager called in
> {{executors.\_\_init\_\_}} -- the top-level {{\_\_init\_\_}} attempts to load
> the default executor, which then goes back to plugins_manager etc.
> {noformat}
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> line 17, in
> from airflow import configuration
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor TestExecutor not supported.
> {noformat}
> While attempting {{from airflow.plugins_manager import executors_modules}}
> cycles right back where it came from:
> {noformat}
> >>> from airflow.plugins_manager import executor_modules
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
> {noformat}
> To reproduce:
> # Create a custom executor, e.g. by copying {{local_executor.py}} into
> {{$AIRFLOW_HOME/plugins/test_executor.py}} and replacing occurrences of
> "Local" with "Test" in the code, which will produce {{TestExecutor}}
> # Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}}
> section: {{executor = TestExecutor}}
> # Run any airflow command, e.g. "airflow version"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1121) airflow webserver --pid no longer write out pid file!
[ https://issues.apache.org/jira/browse/AIRFLOW-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1121: --- Fix Version/s: (was: 1.8.1) 1.9.0 > airflow webserver --pid no longer write out pid file! > - > > Key: AIRFLOW-1121 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1121 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.8.1rc0 >Reporter: Siddharth Anand >Assignee: Kengo Seki >Priority: Blocker > Fix For: 1.9.0 > > > I've run into a regression with the webserver. It looks like the --pid > argument is no longer honored in 1.8.1. The pid file is not being written > out! As a result, watchdog processes like monitd, which watch the processes > mentioned in the pid file, keep trying to spawn webservers because the pid > files are not being written. > HISTTIMEFORMAT="%d/%m/%y %T " > PYTHONPATH=/usr/local/agari/ep-pipeline/production/current/analysis/cluster/:/usr/local/agari/ep-pipeline/production/current/analysis/lookups/ > TMP=/data/tmp AIRFLOW_HOME=/data/airflow > PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin airflow webserver -p 8080 > --pid /data/airflow/pids/airflow-webserver.pid > The Patch that might have broken this is > https://github.com/apache/incubator-airflow/commit/a9b20a04 > If I include -D, things work: > airflow webserver -D --pid ~/this.pid > sid-as-mbp:~ siddharth$ ls -lrt *pid > -rw-r--r-- 1 siddharth staff 5 Apr 18 14:37 this-monitor.pid > -rw-r--r-- 1 siddharth staff 5 Apr 18 14:37 this.pid > sid-as-mbp:~ siddharth$ cat *pid > 8732 > 8735 > However, this doesn't help me as I don't want to run with -D. > cc [~sekikn] [~criccomini] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1165) airflow webservice crashes on ubuntu16 - python3
[
https://issues.apache.org/jira/browse/AIRFLOW-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1165:
---
Fix Version/s: (was: 1.8.1)
1.9.0
> airflow webservice crashes on ubuntu16 - python3
> -
>
> Key: AIRFLOW-1165
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1165
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Hamed
>Assignee: Arthur Wiedmer
> Fix For: 1.9.0
>
>
> I am trying to run airflow webserver on ubuntu16, python3 and ran to this
> issue. Any idea?
> {code}
> [2017-05-02 16:36:34,789] [24096] {_internal.py:87} WARNING - * Debugger is
> active!
> [2017-05-02 16:36:34,790] [24096] {_internal.py:87} INFO - * Debugger PIN:
> 294-518-137
> Exception in thread Thread-1:
> Traceback (most recent call last):
> File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
> self.run()
> File "/usr/lib/python3.5/threading.py", line 862, in run
> self._target(*self._args, **self._kwargs)
> File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line
> 696, in inner
> fd=fd)
> File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line
> 590, in make_server
> passthrough_errors, ssl_context, fd=fd)
> File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line
> 525, in __init__
> self.socket = ssl_context.wrap_socket(sock, server_side=True)
> File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line
> 447, in wrap_socket
> ssl_version=self._protocol, **kwargs)
> File "/usr/lib/python3.5/ssl.py", line 1069, in wrap_socket
> ciphers=ciphers)
> File "/usr/lib/python3.5/ssl.py", line 680, in __init__
> raise ValueError("certfile must be specified for server-side "
> ValueError: certfile must be specified for server-side operations
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1030) HttpHook error when creating HttpSensor
[
https://issues.apache.org/jira/browse/AIRFLOW-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1030:
---
Fix Version/s: (was: 1.8.1)
1.9.0
> HttpHook error when creating HttpSensor
> ---
>
> Key: AIRFLOW-1030
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1030
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: Airflow 1.8
> Environment: python3
>Reporter: Paulius Dambrauskas
>Assignee: Paulius Dambrauskas
> Fix For: 1.9.0
>
>
> Task:
> {code}
> sensor = HttpSensor(
> task_id='http_sensor_check',
> http_conn_id='http_default',
> endpoint='',
> params={},
> poke_interval=5,
> dag=dag
> )
> {code}
> Exception
> {code}
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.5/dist-packages/airflow-1.9.0.dev0+apache.incubating-py3.5.egg/airflow/models.py",
> line 268, in process_file
> m = imp.load_source(mod_name, filepath)
> File "/usr/lib/python3.5/imp.py", line 172, in load_source
> module = _load(spec)
> File "", line 693, in _load
> File "", line 673, in _load_unlocked
> File "", line 665, in exec_module
> File "", line 222, in _call_with_frames_removed
> File "/home/paulius/airflow/dags/cpg_4.py", line 43, in
> dag=dag)
> File
> "/usr/local/lib/python3.5/dist-packages/airflow-1.9.0.dev0+apache.incubating-py3.5.egg/airflow/utils/decorators.py",
> line 86, in wrapper
> result = func(*args, **kwargs)
> File
> "/usr/local/lib/python3.5/dist-packages/airflow-1.9.0.dev0+apache.incubating-py3.5.egg/airflow/operators/sensors.py",
> line 663, in __init__
> self.hook = hooks.http_hook.HttpHook(method='GET',
> http_conn_id=http_conn_id)
> File
> "/usr/local/lib/python3.5/dist-packages/airflow-1.9.0.dev0+apache.incubating-py3.5.egg/airflow/utils/helpers.py",
> line 436, in __getattr__
> raise AttributeError
> AttributeError
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1024) Handle CeleryExecutor errors gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1024: --- Fix Version/s: (was: 1.8.1) 1.9.0 > Handle CeleryExecutor errors gracefully > --- > > Key: AIRFLOW-1024 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1024 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dan Davydov >Priority: Critical > Fix For: 1.9.0 > > > If the Airflow celery executor receives a bad response from a worker (e.g. > unpickled response), then it will crash the scheduler and cause it to restart. > We should code defensively around the interactions with celery so that we > just log errors instead of crashing the scheduler. > It might makes sense to make the try catches one level higher (to catch > errors from all executors), but this needs some investigation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-991) Mark_success while a task is running leads to failure state
[ https://issues.apache.org/jira/browse/AIRFLOW-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-991: -- Fix Version/s: (was: 1.8.1) 1.9.0 > Mark_success while a task is running leads to failure state > --- > > Key: AIRFLOW-991 > URL: https://issues.apache.org/jira/browse/AIRFLOW-991 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Maxime Beauchemin > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-880) Fix remote log functionality inconsistencies for Webservers
[ https://issues.apache.org/jira/browse/AIRFLOW-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-880: -- Fix Version/s: (was: 1.8.2) 1.9.0 > Fix remote log functionality inconsistencies for Webservers > --- > > Key: AIRFLOW-880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-880 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov > Fix For: 1.9.0 > > > Right now log functionality is not consistent when it comes to remote logs. > 1. Lack of Complete Logs: Remote logs should be the default instead of the > log that is only loaded if the local log is not present, because the remote > log will have the logs for all of the tries of a task, whereas the local log > is only guaranteed to have the most recent one > 2. Lack of Consistency: The logs returned will always be the same from all > the webservers (right now different logs can be returned if a webserver has a > log vs doesn't, and there can be different logs between webservers that have > the log). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-880) Fix remote log functionality inconsistencies for Webservers
[ https://issues.apache.org/jira/browse/AIRFLOW-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-880: -- Fix Version/s: (was: 1.8.1) 1.9.0 > Fix remote log functionality inconsistencies for Webservers > --- > > Key: AIRFLOW-880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-880 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov > Fix For: 1.8.2 > > > Right now log functionality is not consistent when it comes to remote logs. > 1. Lack of Complete Logs: Remote logs should be the default instead of the > log that is only loaded if the local log is not present, because the remote > log will have the logs for all of the tries of a task, whereas the local log > is only guaranteed to have the most recent one > 2. Lack of Consistency: The logs returned will always be the same from all > the webservers (right now different logs can be returned if a webserver has a > log vs doesn't, and there can be different logs between webservers that have > the log). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-880) Fix remote log functionality inconsistencies for Webservers
[ https://issues.apache.org/jira/browse/AIRFLOW-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-880: -- Fix Version/s: (was: 1.9.0) 1.8.2 > Fix remote log functionality inconsistencies for Webservers > --- > > Key: AIRFLOW-880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-880 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov > Fix For: 1.8.2 > > > Right now log functionality is not consistent when it comes to remote logs. > 1. Lack of Complete Logs: Remote logs should be the default instead of the > log that is only loaded if the local log is not present, because the remote > log will have the logs for all of the tries of a task, whereas the local log > is only guaranteed to have the most recent one > 2. Lack of Consistency: The logs returned will always be the same from all > the webservers (right now different logs can be returned if a webserver has a > log vs doesn't, and there can be different logs between webservers that have > the log). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1198) HDFSOperator to operate HDFS
[ https://issues.apache.org/jira/browse/AIRFLOW-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1198: --- Fix Version/s: (was: 1.8.1) (was: Airflow 2.0) 1.9.0 > HDFSOperator to operate HDFS > > > Key: AIRFLOW-1198 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1198 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, operators >Affects Versions: Airflow 2.0, Airflow 1.8 >Reporter: Vianney FOUCAULT >Assignee: Vianney FOUCAULT > Fix For: 1.9.0 > > > There is no hdfs operator to move/create/delete file or folders withing hdfs -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1247) CLI: ignore all dependencies argument ignored
[ https://issues.apache.org/jira/browse/AIRFLOW-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1247: --- Fix Version/s: (was: 1.8.1) 1.9.0 > CLI: ignore all dependencies argument ignored > - > > Key: AIRFLOW-1247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1247 > Project: Apache Airflow > Issue Type: Bug > Components: cli >Affects Versions: 1.8.1 >Reporter: Matti Remes >Assignee: Matti Remes >Priority: Trivial > Fix For: 1.9.0 > > > Missing comma in Arg object creation -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1055) airflow/jobs.py:create_dag_run() exception for @once dag when catchup = False
[
https://issues.apache.org/jira/browse/AIRFLOW-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1055:
---
Fix Version/s: (was: 1.8.1)
1.9.0
> airflow/jobs.py:create_dag_run() exception for @once dag when catchup = False
> -
>
> Key: AIRFLOW-1055
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1055
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Siddharth Anand
>Assignee: Ruslan Dautkhanov
>Priority: Critical
> Labels: dagrun, once, scheduler, sla
> Fix For: 1.9.0
>
>
> Getting following exception
> {noformat}
> [2017-03-19 20:16:25,786] {jobs.py:354} DagFileProcessor2638 ERROR - Got an
> exception! Propagating...
> Traceback (most recent call last):
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 346, in helper
> pickle_dags)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1581, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1175, in _process_dags
> self.manage_slas(dag)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 595, in manage_slas
> while dttm < datetime.now():
> TypeError: can't compare datetime.datetime to NoneType
> {noformat}
> Exception is in airflow/jobs.py:manage_slas() :
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L595
> {code}
> ts = datetime.now()
> SlaMiss = models.SlaMiss
> for ti in max_tis:
> task = dag.get_task(ti.task_id)
> dttm = ti.execution_date
> if task.sla:
> dttm = dag.following_schedule(dttm)
> >>> while dttm < datetime.now(): <<< here
> following_schedule = dag.following_schedule(dttm)
> if following_schedule + task.sla < datetime.now():
> session.merge(models.SlaMiss(
> task_id=ti.task_id,
> {code}
> It seems that dag.following_schedule() returns None for @once dag?
> Here's how dag is defined:
> {code}
> import datetime as dt
> import airflow
> from airflow.models import DAG
> from airflow.operators.dummy_operator import DummyOperator
> from datetime import datetime, timedelta
> def sla_alert_func(dag, task_list, blocking_task_list, slas, blocking_tis):
> print('Executing SLA miss callback')
> now = datetime.now()
> now_to_the_hour = now.replace(hour=now.time().hour, minute=0, second=0,
> microsecond=0)
> START_DATE = now_to_the_hour + timedelta(hours=-3)
> DAG_NAME = 'manage_sla_once_dag'
> default_args = {
> 'owner': 'sanand',
> 'depends_on_past': False,
> 'start_date': START_DATE,
> 'sla': timedelta(hours=2)
> }
> dag = DAG(
> dag_id = 'manage_sla_once_dag',
> default_args = default_args,
> catchup = False,
> schedule_interval = '@once',
> sla_miss_callback = sla_alert_func
> )
> task1 = DummyOperator(task_id='task1', dag=dag)
> {code}
> This issue works if catchup = True.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1182) Contrib Spark Submit operator should template fields
[
https://issues.apache.org/jira/browse/AIRFLOW-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1182:
---
Fix Version/s: (was: 1.8.1)
(was: Airflow 2.0)
1.9.0
> Contrib Spark Submit operator should template fields
>
>
> Key: AIRFLOW-1182
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1182
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, operators
>Affects Versions: Airflow 2.0, Airflow 1.8
>Reporter: Vianney FOUCAULT
>Assignee: Vianney FOUCAULT
> Fix For: 1.9.0
>
>
> the spark submit operator is not templating any field making {{ ds }}
> unusable for spark apps.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1018) Scheduler DAG processes can not log to stdout
[ https://issues.apache.org/jira/browse/AIRFLOW-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1018: --- Fix Version/s: (was: 1.8.1) 1.9.0 > Scheduler DAG processes can not log to stdout > - > > Key: AIRFLOW-1018 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1018 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.8.0 > Environment: Airflow 1.8.0 >Reporter: Vincent Poulain >Assignee: Bolke de Bruin >Priority: Critical > Fix For: 1.9.0 > > > Each DAG has its own log file for the scheduler and we can specify the > directory with child_process_log_directory param. > Unfortunately we can not change device / by specifying /dev/stdout for > example. That is very useful when we execute Airflow in a container. > When we specify /dev/stdout it raises: > "OSError: [Errno 20] Not a directory: '/dev/stdout/2017-03-19'" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1019) active_dagruns shouldn't include paused DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1019: --- Fix Version/s: (was: 1.8.1) 1.9.0 > active_dagruns shouldn't include paused DAGs > > > Key: AIRFLOW-1019 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1019 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.8.0 >Reporter: Dan Davydov >Priority: Critical > Fix For: 1.9.0 > > > Since 1.8.0 Airflow resets orphaned tasks (tasks that are in the DB but not > in the executor's memory). The problem is that Airflow counts dagruns in > paused DAGs as running as long as the dagruns state is running. Instead we > should join against non-paused DAGs everywhere we calculate active dagruns > (e.g. in _process_task_instances in the Scheduler class in jobs.py). If there > are enough paused DAGs it brings the scheduler to a halt especially on > scheduler restarts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1013) airflow/jobs.py:manage_slas() exception for @once dag
[
https://issues.apache.org/jira/browse/AIRFLOW-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1013:
---
Fix Version/s: (was: 1.8.1)
1.9.0
> airflow/jobs.py:manage_slas() exception for @once dag
> -
>
> Key: AIRFLOW-1013
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1013
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 1.8, 1.8.1, 1.8.0, 1.8.2
>Reporter: Ruslan Dautkhanov
>Assignee: Siddharth Anand
>Priority: Blocker
> Labels: dagrun, once, scheduler, sla
> Fix For: 1.9.0
>
>
> Getting following exception
> {noformat}
> [2017-03-19 20:16:25,786] {jobs.py:354} DagFileProcessor2638 ERROR - Got an
> exception! Propagating...
> Traceback (most recent call last):
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 346, in helper
> pickle_dags)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1581, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1175, in _process_dags
> self.manage_slas(dag)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 595, in manage_slas
> while dttm < datetime.now():
> TypeError: can't compare datetime.datetime to NoneType
> {noformat}
> Exception is in airflow/jobs.py:manage_slas() :
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L595
> {code}
> ts = datetime.now()
> SlaMiss = models.SlaMiss
> for ti in max_tis:
> task = dag.get_task(ti.task_id)
> dttm = ti.execution_date
> if task.sla:
> dttm = dag.following_schedule(dttm)
> >>> while dttm < datetime.now(): <<< here
> following_schedule = dag.following_schedule(dttm)
> if following_schedule + task.sla < datetime.now():
> session.merge(models.SlaMiss(
> task_id=ti.task_id,
> {code}
> It seems that dag.following_schedule() returns None for @once dag?
> Here's how dag is defined:
> {code}
> main_dag = DAG(
> dag_id = 'DISCOVER-Oracle-Load',
> default_args = default_args,
> user_defined_macros= dag_macros,
> start_date = datetime.now(),
> catchup= False,
> schedule_interval = '@once',
> concurrency= 2,
> max_active_runs= 1,
> dagrun_timeout = timedelta(days=4),
> )
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1015) TreeView displayed over task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1015: --- Fix Version/s: (was: 1.8.1) 1.9.0 > TreeView displayed over task instances > -- > > Key: AIRFLOW-1015 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1015 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: Airflow 1.8 >Reporter: Ruslan Dautkhanov >Priority: Minor > Labels: ui > Fix For: 1.9.0 > > Attachments: TreeView-bug.png > > > See screnshot: > !TreeView-bug.png! > It would be nice if first TI horizontal offset would automatically shift to > the right, depending on how many tasks are in longest branch to the left. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-976) Mark success running task causes it to fail
[ https://issues.apache.org/jira/browse/AIRFLOW-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-976: -- Fix Version/s: (was: 1.8.1) 1.9.0 > Mark success running task causes it to fail > --- > > Key: AIRFLOW-976 > URL: https://issues.apache.org/jira/browse/AIRFLOW-976 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.8.0 >Reporter: Dan Davydov >Assignee: Alex Guziel > Fix For: 1.9.0 > > > Marking success on a running task in the UI causes it to fail. > Expected Behavior: > Task instance is killed and marked as successful > Actual Behavior: > Task instance is killed and marked as failed > [~saguziel] [~bolke] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-788) Context unexpectedly added to hive conf
[ https://issues.apache.org/jira/browse/AIRFLOW-788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-788: -- Fix Version/s: (was: 1.8.1) 1.9.0 > Context unexpectedly added to hive conf > --- > > Key: AIRFLOW-788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-788 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Bolke de Bruin > Fix For: 1.9.0 > > > If specifying hive_conf to run_cli extra variables are added from the > context, e.g. airflow.ctx.dag.dag_id . > In secured environments this can raise the need for a configuration change as > these variables might not be whitelisted. > Secondly one could regard it as information leakage, as its is added without > the user's consent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1294) Backfills can loose tasks to execute due to tasks setting themselves to NONE
[ https://issues.apache.org/jira/browse/AIRFLOW-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1294. Resolution: Fixed Assignee: Bolke de Bruin Bolke fixed it, PR got merged. > Backfills can loose tasks to execute due to tasks setting themselves to NONE > > > Key: AIRFLOW-1294 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1294 > Project: Apache Airflow > Issue Type: Bug > Components: backfill >Affects Versions: 1.8.1 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.2 > > > In the backfills we can loose tasks to execute due to a task setting its own > state to NONE if concurrency limits are reached, this makes them fall outside > of the scope the backfill is managing hence they will not be executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- Priority: Blocker (was: Major) > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Priority: Blocker > Fix For: 1.8.2 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1291: --- Priority: Blocker (was: Major) > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Priority: Blocker > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1294) Backfills can loose tasks to execute due to tasks setting themselves to NONE
[ https://issues.apache.org/jira/browse/AIRFLOW-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1294: --- Fix Version/s: 1.8.2 > Backfills can loose tasks to execute due to tasks setting themselves to NONE > > > Key: AIRFLOW-1294 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1294 > Project: Apache Airflow > Issue Type: Bug > Components: backfill >Affects Versions: 1.8.1 >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.2 > > > In the backfills we can loose tasks to execute due to a task setting its own > state to NONE if concurrency limits are reached, this makes them fall outside > of the scope the backfill is managing hence they will not be executed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1266) Long task names are truncated in gannt view
[ https://issues.apache.org/jira/browse/AIRFLOW-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1266: --- Fix Version/s: 1.8.2 > Long task names are truncated in gannt view > --- > > Key: AIRFLOW-1266 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1266 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov > Fix For: 1.8.2 > > > Long task names are truncated in gannt view, we should make the text smaller > instead. To reproduce create a task called > abczzzdef and note that the start/end characters > are not fully visible in the gant view. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1274) HttpSensor parameter params is overriding BaseOperator params and should be renamed
[
https://issues.apache.org/jira/browse/AIRFLOW-1274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin updated AIRFLOW-1274:
---
Fix Version/s: 1.8.2
> HttpSensor parameter params is overriding BaseOperator params and should be
> renamed
> ---
>
> Key: AIRFLOW-1274
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1274
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Alice Berard
>Assignee: Alice Berard
> Fix For: 1.8.2
>
>
> {{HttpSensor}} parameter {{params}} is overriding BaseOperator {{params}} and
> should be renamed.
> It is screwing DAG parsing:
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 264,
> in process_file
> m = imp.load_source(mod_name, filepath)
> File
> "/home/alice_berard/mydags/anomaly_detection/anomaly_detection/curiosity/curiosity_dags.py",
> line 14, in
> pool='curiosity',
> File
> "/home/alice_berard/mydags/anomaly_detection/anomaly_detection/curiosity/dag_builder.py",
> line 133, in build
> pool=self.pool
> File "/usr/local/lib/python2.7/dist-packages/airflow/utils/decorators.py",
> line 58, in wrapper
> dag_params.update(params)
> ValueError: dictionary update sequence element #0 has length 1; 2 is required
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-1263) Airflow Charts Pages Should Have Dynamic Heights
[ https://issues.apache.org/jira/browse/AIRFLOW-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1263. Resolution: Fixed > Airflow Charts Pages Should Have Dynamic Heights > > > Key: AIRFLOW-1263 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1263 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov > Fix For: 1.8.2 > > > Currently Airflow has a fixed height of 600px for the airflow task tries page > and the other chart pages (e.g. admin/airflow/tries) for DAGs. This becomes > unreadable for large dags where the legend squishes the chart itself to the > bottom since there are so many tasks. To fix this we might need to fix the > charting library dependency we have to allow for dynamic heights. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1263) Airflow Charts Pages Should Have Dynamic Heights
[ https://issues.apache.org/jira/browse/AIRFLOW-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1263: --- Fix Version/s: 1.8.2 > Airflow Charts Pages Should Have Dynamic Heights > > > Key: AIRFLOW-1263 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1263 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov > Fix For: 1.8.2 > > > Currently Airflow has a fixed height of 600px for the airflow task tries page > and the other chart pages (e.g. admin/airflow/tries) for DAGs. This becomes > unreadable for large dags where the legend squishes the chart itself to the > bottom since there are so many tasks. To fix this we might need to fix the > charting library dependency we have to allow for dynamic heights. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1074) Do not count queued tasks in scheduler concurrency check
[ https://issues.apache.org/jira/browse/AIRFLOW-1074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1074: --- Fix Version/s: 1.8.2 > Do not count queued tasks in scheduler concurrency check > > > Key: AIRFLOW-1074 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1074 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1078) Latest_runs endpoint broken in old flask versions
[ https://issues.apache.org/jira/browse/AIRFLOW-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1078: --- Fix Version/s: 1.8.2 > Latest_runs endpoint broken in old flask versions > - > > Key: AIRFLOW-1078 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1078 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-1078) Latest_runs endpoint broken in old flask versions
[ https://issues.apache.org/jira/browse/AIRFLOW-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1078. Resolution: Fixed > Latest_runs endpoint broken in old flask versions > - > > Key: AIRFLOW-1078 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1078 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1081) Task duration page is slow
[ https://issues.apache.org/jira/browse/AIRFLOW-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1081: --- Fix Version/s: (was: 1.9.0) 1.8.2 > Task duration page is slow > -- > > Key: AIRFLOW-1081 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1081 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > It makes a number of queries proportional to the data size, instead of just 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-1064) TaskInstanceModelView is slow
[ https://issues.apache.org/jira/browse/AIRFLOW-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1064. Resolution: Fixed merged before > TaskInstanceModelView is slow > - > > Key: AIRFLOW-1064 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1064 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > Due to a bad query (a full table scan), the TaskInstanceModelView is very > slow. Adding an index is costly, and job_id is a good approximation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1064) TaskInstanceModelView is slow
[ https://issues.apache.org/jira/browse/AIRFLOW-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1064: --- Fix Version/s: 1.8.2 > TaskInstanceModelView is slow > - > > Key: AIRFLOW-1064 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1064 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > Due to a bad query (a full table scan), the TaskInstanceModelView is very > slow. Adding an index is costly, and job_id is a good approximation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1007) Jinja sandbox is vulnerable to RCE
[ https://issues.apache.org/jira/browse/AIRFLOW-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1007: --- Fix Version/s: (was: 1.9.0) 1.8.2 > Jinja sandbox is vulnerable to RCE > -- > > Key: AIRFLOW-1007 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1007 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > Right now, the jinja template functionality in chart_data takes arbitrary > strings and executes them. We should use the sandbox functionality to prevent > this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1047) Airflow logs vulnerable to XSS
[ https://issues.apache.org/jira/browse/AIRFLOW-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1047: --- Fix Version/s: (was: 1.9.0) 1.8.2 > Airflow logs vulnerable to XSS > -- > > Key: AIRFLOW-1047 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1047 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > Navigating to a page with dag_id param specified as a html tag leads to that > tag being rendered due to using Markup tag (which makes html be labeled as > safe) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-836) The paused and queryview endpoints are vulnerable to CSRF
[ https://issues.apache.org/jira/browse/AIRFLOW-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-836: -- Fix Version/s: (was: 1.9.0) 1.8.2 > The paused and queryview endpoints are vulnerable to CSRF > - > > Key: AIRFLOW-836 > URL: https://issues.apache.org/jira/browse/AIRFLOW-836 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > These endpoints use GET and are state-changing which is bad practice, and > allows CSRF -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1038) Specify celery serializers explicitly and pin version
[ https://issues.apache.org/jira/browse/AIRFLOW-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1038: --- Fix Version/s: (was: 1.9.0) 1.8.2 > Specify celery serializers explicitly and pin version > - > > Key: AIRFLOW-1038 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1038 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > Fix For: 1.8.2 > > > Celery 3->4 upgrade changes the default task and result serializer from > pickle to json. Pickle is faster and supports more types > http://docs.celeryproject.org/en/latest/userguide/calling.html > This also causes issues when different versions of celery are running on > different hosts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- Fix Version/s: (was: 1.8.1) 1.8.2 > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.2 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1291: --- Fix Version/s: (was: 1.8.1) 1.8.2 > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1024) Handle CeleryExecutor errors gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1024: --- Fix Version/s: 1.8.1 > Handle CeleryExecutor errors gracefully > --- > > Key: AIRFLOW-1024 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1024 > Project: Apache Airflow > Issue Type: Bug >Reporter: Dan Davydov >Priority: Critical > Fix For: 1.8.1 > > > If the Airflow celery executor receives a bad response from a worker (e.g. > unpickled response), then it will crash the scheduler and cause it to restart. > We should code defensively around the interactions with celery so that we > just log errors instead of crashing the scheduler. > It might makes sense to make the try catches one level higher (to catch > errors from all executors), but this needs some investigation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-991) Mark_success while a task is running leads to failure state
[ https://issues.apache.org/jira/browse/AIRFLOW-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-991: -- Fix Version/s: 1.8.1 > Mark_success while a task is running leads to failure state > --- > > Key: AIRFLOW-991 > URL: https://issues.apache.org/jira/browse/AIRFLOW-991 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Maxime Beauchemin > Fix For: 1.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (AIRFLOW-991) Mark_success while a task is running leads to failure state
[ https://issues.apache.org/jira/browse/AIRFLOW-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin reopened AIRFLOW-991: --- Assignee: Maxime Beauchemin (was: Alex Guziel) > Mark_success while a task is running leads to failure state > --- > > Key: AIRFLOW-991 > URL: https://issues.apache.org/jira/browse/AIRFLOW-991 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Maxime Beauchemin > Fix For: 1.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-991) Mark_success while a task is running leads to failure state
[ https://issues.apache.org/jira/browse/AIRFLOW-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-991. --- Resolution: Fixed > Mark_success while a task is running leads to failure state > --- > > Key: AIRFLOW-991 > URL: https://issues.apache.org/jira/browse/AIRFLOW-991 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Maxime Beauchemin > Fix For: 1.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-880) Fix remote log functionality inconsistencies for Webservers
[ https://issues.apache.org/jira/browse/AIRFLOW-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-880: -- Fix Version/s: 1.8.1 > Fix remote log functionality inconsistencies for Webservers > --- > > Key: AIRFLOW-880 > URL: https://issues.apache.org/jira/browse/AIRFLOW-880 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Reporter: Dan Davydov >Assignee: Dan Davydov > Fix For: 1.8.1 > > > Right now log functionality is not consistent when it comes to remote logs. > 1. Lack of Complete Logs: Remote logs should be the default instead of the > log that is only loaded if the local log is not present, because the remote > log will have the logs for all of the tries of a task, whereas the local log > is only guaranteed to have the most recent one > 2. Lack of Consistency: The logs returned will always be the same from all > the webservers (right now different logs can be returned if a webserver has a > log vs doesn't, and there can be different logs between webservers that have > the log). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1291: --- Fix Version/s: (was: 1.8.2) 1.8.1 > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- Fix Version/s: (was: 1.8.2) 1.8.1 > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.1 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- Fix Version/s: (was: 1.8.1) 1.8.2 > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.2 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1291: --- Fix Version/s: (was: 1.8.1) 1.8.2 > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.2 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- External issue URL: https://github.com/apache/incubator-airflow/pull/2352 > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.1 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1290: --- Fix Version/s: 1.8.1 > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.1 > > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-1291: --- Fix Version/s: 1.8.1 > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.8.1 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
[ https://issues.apache.org/jira/browse/AIRFLOW-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041810#comment-16041810 ] Maxime Beauchemin commented on AIRFLOW-1291: John D. Ament said on the mailing list on May 7th 2017: - Would be good to list out the locations of each file (or path to a group of files) (some have this, and others do not so its hard to follow) - There's errant /* .. */ around each license declaration, which should be removed. - Missing license bodies for FooTable v2, jQuery Clock Plugin, Likewise, your NOTICE has copyright 2011-2017, however Airflow hasn't been incubating that long. If you like, you can give origination notices to the original creators here to specify the original copyright dates. I would challenge the podling to see if there's a way to simplify their LICENSE by instead using npm or some other javascript packaging tool to build a distribution, rather than shipping the dependencies in the source release, makes it much easier to use. As the podling matures, would be good to see information about the author switch from an individual to a community (in setup.cfg, its already in setup.py so may have been a miss) > Update NOTICE and LICENSE files to meet ASF specs > - > > Key: AIRFLOW-1291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1291) Update NOTICE and LICENSE files to meet ASF specs
Maxime Beauchemin created AIRFLOW-1291: -- Summary: Update NOTICE and LICENSE files to meet ASF specs Key: AIRFLOW-1291 URL: https://issues.apache.org/jira/browse/AIRFLOW-1291 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
[ https://issues.apache.org/jira/browse/AIRFLOW-1290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin resolved AIRFLOW-1290. Resolution: Fixed > Change docs author from "Maxime Beauchemin" to "Apache Airflow" > --- > > Key: AIRFLOW-1290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > > In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who > was the original author of the docs, but since then the whole community has > taken over authoring the Airflow documentation. > Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1290) Change docs author from "Maxime Beauchemin" to "Apache Airflow"
Maxime Beauchemin created AIRFLOW-1290: -- Summary: Change docs author from "Maxime Beauchemin" to "Apache Airflow" Key: AIRFLOW-1290 URL: https://issues.apache.org/jira/browse/AIRFLOW-1290 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin In `docs/conf.py`, the author of the docs is set to "Maxime Beauchemin" who was the original author of the docs, but since then the whole community has taken over authoring the Airflow documentation. Change all references from "Maxime Beauchemin" to "Apache Airflow" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1261) clarify the docs for the CLI's airflow run --local flag
Maxime Beauchemin created AIRFLOW-1261: -- Summary: clarify the docs for the CLI's airflow run --local flag Key: AIRFLOW-1261 URL: https://issues.apache.org/jira/browse/AIRFLOW-1261 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1228) Fix readthedocs timeout
Maxime Beauchemin created AIRFLOW-1228: -- Summary: Fix readthedocs timeout Key: AIRFLOW-1228 URL: https://issues.apache.org/jira/browse/AIRFLOW-1228 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin RTD is a service that build documentation automatically on the latest master and allow for serving specific versions of the docs. This will allow us to have each Apache release have its own documentation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-97) "airflow" "DAG" strings in file necessary to import dag
[ https://issues.apache.org/jira/browse/AIRFLOW-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966122#comment-15966122 ] Maxime Beauchemin commented on AIRFLOW-97: -- The rational for it was that people may and will dump python files that do significant work outside of some `if __name__ == '__main__':` block and that Airflow, as it crawls and imports these modules, will trigger them. We've also seen people dumping entire libs in our pipelines folder, and the DagBag parsing process will import the living hell out of it. This is a naive attempt at jumping over files that don't look like an Airflow pipeline by introspecting the code without evaluating it. It may have been introduced after the module parsing timeout rule was introduced. Note that the DagBag timeout logic may be preferable in some ways, but that it has limitations. First it won't work under LocalExecutor for some reason I won't get into here. Second it sucks to pay the timeout price at every scheduler cycle. Perhaps a better approach would be for the process to add timeout scripts to a blacklist and surface it in the UI. Then users would have to re-enable bad actors manually. > "airflow" "DAG" strings in file necessary to import dag > --- > > Key: AIRFLOW-97 > URL: https://issues.apache.org/jira/browse/AIRFLOW-97 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: Airflow 1.7.0 >Reporter: Etiene Dalcol >Priority: Minor > > Hello airflow team! Thanks for the awesome tool! > We made a small module to automate our DAG building process and we are using > this module on our DAG definition. Our airflow version is 1.7.0. > However, airflow will not import this file because it doesn't have the words > DAG and airflow on it. (The imports etc are done inside our little module). > Apparently there's a safe_mode that skips files without these strings. > (https://github.com/apache/incubator-airflow/blob/1.7.0/airflow/models.py#L197) > This safe_mode is default to True but is not passed to the process_file > function, so it is always True and there's no apparent way to disable it. > (https://github.com/apache/incubator-airflow/blob/1.7.0/airflow/models.py#L177) > (https://github.com/apache/incubator-airflow/blob/1.7.0/airflow/models.py#L313) > Putting this comment on the top of the file makes it work for the moment and > brought me a good laugh today 👯 > #DAG airflow —> DO NOT REMOVE. the world will explode -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-954) Installing future 0.16 breaks airflow initdb on Python 2.7
[
https://issues.apache.org/jira/browse/AIRFLOW-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-954.
---
Resolution: Fixed
> Installing future 0.16 breaks airflow initdb on Python 2.7
> --
>
> Key: AIRFLOW-954
> URL: https://issues.apache.org/jira/browse/AIRFLOW-954
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Python 2.7.3 on Ubuntu 12.04 with a clean virtualenv and
> HEAD of airflow master installed
>Reporter: Sean Cronin
>
> On master, if you run {{airflow initdb}} after installing HEAD of airflow
> master with Python 2.7, you get the following error:
> {code}
> Traceback (most recent call last):
> File "/home/sean/.virtualenvs/airflow-test/bin/airflow", line 17, in
>
> from airflow import configuration
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/__init__.py",
> line 29, in
> from airflow import configuration as conf
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/configuration.py",
> line 33, in
> from configparser import ConfigParser
> ImportError: No module named configparser
> {code}
> This seems to be due to https://github.com/apache/incubator-airflow/pull/2091
> which bumps {{future}} in {{setup.py}} so that future 0.16 can be installed.
> Python future [got rid
> of](http://python-future.org/whatsnew.html#what-s-new-in-version-0-16-0-2016-10-27)
> their {{configparser}} alias in 0.16
> The recommended way to fix this is to install {{future==0.16}} and install
> {{configparser}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AIRFLOW-636) Warn on DagBag skipping modules
[ https://issues.apache.org/jira/browse/AIRFLOW-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-636: -- External issue URL: https://github.com/apache/incubator-airflow/pull/1889 > Warn on DagBag skipping modules > --- > > Key: AIRFLOW-636 > URL: https://issues.apache.org/jira/browse/AIRFLOW-636 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-636) Warn on DagBag skipping modules
Maxime Beauchemin created AIRFLOW-636: - Summary: Warn on DagBag skipping modules Key: AIRFLOW-636 URL: https://issues.apache.org/jira/browse/AIRFLOW-636 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-612) Moving and linking to the ressources links in the to wiki
Maxime Beauchemin created AIRFLOW-612: - Summary: Moving and linking to the ressources links in the to wiki Key: AIRFLOW-612 URL: https://issues.apache.org/jira/browse/AIRFLOW-612 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-351) Failed to clear downstream tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536210#comment-15536210
]
Maxime Beauchemin commented on AIRFLOW-351:
---
There's something non-deepcopiable attached to your DAG object. Look at objects
in your default_args, params or callbacks.
> Failed to clear downstream tasks
>
>
> Key: AIRFLOW-351
> URL: https://issues.apache.org/jira/browse/AIRFLOW-351
> Project: Apache Airflow
> Issue Type: Bug
> Components: models, subdag, webserver
>Affects Versions: Airflow 1.7.1.3
>Reporter: Adinata
> Attachments: dag_error.py, error_on_clear_dag.txt
>
>
> {code}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: 9889a7c79e9b
> ---
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 68,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line
> 367, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/flask_login.py", line 755, in
> decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 118, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/utils.py", line
> 167, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line
> 1017, in clear
> include_upstream=upstream)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2870,
> in sub_dag
> dag = copy.deepcopy(self)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2856,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
> y = copier(memo)
> File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1974,
> in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
> y = _reconstruct(x, rv, 1, memo)
> File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
> state = deepcopy(state, memo)
> File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
> y = copier(x, memo)
> File "/usr/lib/python2.7/copy.py", line 257, in
[jira] [Updated] (AIRFLOW-526) Pin all dependencies versions in setup.py
[ https://issues.apache.org/jira/browse/AIRFLOW-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-526: -- External issue URL: https://github.com/apache/incubator-airflow/pull/1809 > Pin all dependencies versions in setup.py > - > > Key: AIRFLOW-526 > URL: https://issues.apache.org/jira/browse/AIRFLOW-526 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > > [AIRFLOW-526] pin all dependencies in setup.py > Pinning all dependencies in setup.py with help having more homogenous > installs across environments. The current approach of allowing version > ranges based on version notation is risky at best and results in bad > surprises. > The downside is that we won't get positive bug fixes and security fixes > for free, but these come at a high cost where a single bad version of a > package may take your Airflow environment down, without warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-526) Pin all dependencies versions in setup.py
[ https://issues.apache.org/jira/browse/AIRFLOW-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-526: -- Description: [AIRFLOW-526] pin all dependencies in setup.py Pinning all dependencies in setup.py with help having more homogenous installs across environments. The current approach of allowing version ranges based on version notation is risky at best and results in bad surprises. The downside is that we won't get positive bug fixes and security fixes for free, but these come at a high cost where a single bad version of a package may take your Airflow environment down, without warning. > Pin all dependencies versions in setup.py > - > > Key: AIRFLOW-526 > URL: https://issues.apache.org/jira/browse/AIRFLOW-526 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > > [AIRFLOW-526] pin all dependencies in setup.py > Pinning all dependencies in setup.py with help having more homogenous > installs across environments. The current approach of allowing version > ranges based on version notation is risky at best and results in bad > surprises. > The downside is that we won't get positive bug fixes and security fixes > for free, but these come at a high cost where a single bad version of a > package may take your Airflow environment down, without warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-526) Pin all dependencies versions in setup.py
Maxime Beauchemin created AIRFLOW-526: - Summary: Pin all dependencies versions in setup.py Key: AIRFLOW-526 URL: https://issues.apache.org/jira/browse/AIRFLOW-526 Project: Apache Airflow Issue Type: Bug Reporter: Maxime Beauchemin -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-518) Require DataProfilingMixin for the Variables CRUD access
Maxime Beauchemin created AIRFLOW-518: - Summary: Require DataProfilingMixin for the Variables CRUD access Key: AIRFLOW-518 URL: https://issues.apache.org/jira/browse/AIRFLOW-518 Project: Apache Airflow Issue Type: Improvement Reporter: Maxime Beauchemin Many of us use the "Variable" model CRUD (create/update/delete) as a k/v store to power frameworks that read these values to dynamically generate pipelines. With the basic "LoginMixin" role (lowest level of access to Airflow) having access to the Variable CRUD, people could easily alter a Variable to run arbitrary code on the platform, depending on how variables are use in that environment. It's a safer bet to elevate CRUD on Variable to DataProfilingMixin, and make sure that the lowest level of access cannot interfere with these Variables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-410) Improve the FAQ part of the docs
[ https://issues.apache.org/jira/browse/AIRFLOW-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Beauchemin updated AIRFLOW-410: -- External issue URL: https://github.com/apache/incubator-airflow/pull/1720 > Improve the FAQ part of the docs > > > Key: AIRFLOW-410 > URL: https://issues.apache.org/jira/browse/AIRFLOW-410 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Maxime Beauchemin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
