[jira] [Resolved] (AIRFLOW-1366) Add max_tries to task instance

[Allison Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allison Wang resolved AIRFLOW-1366.
---
Resolution: Done

> Add max_tries to task instance
> --
>
> Key: AIRFLOW-1366
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1366
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Allison Wang
>Assignee: Allison Wang
>
> Right now Airflow deletes the task instance when user clear it. We have no 
> way of keeping track of how many times a task instance gets run either via 
> user or itself. So instead of deleting the task instance record, we should 
> keep the task instance and make try_number monotonically increasing for every 
> task instance attempt. max_tries is introduced as an upper bound for retrying 
> tasks by task itself.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI

[user_airflow (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

user_airflow updated AIRFLOW-1397:
--
Description: 
Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the 
Last Run column in Airflow UI started showing as Blank for all the existing 
dags.

Created a pr for this bug: https://github.com/apache/incubator-airflow/pull/2430

  was:Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, 
the Last Run column in Airflow UI started showing as Blank for all the existing 
dags.


> Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
> -
>
> Key: AIRFLOW-1397
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1397
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, ui
>Affects Versions: 1.8.1
>Reporter: user_airflow
>Priority: Critical
>
> Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the 
> Last Run column in Airflow UI started showing as Blank for all the existing 
> dags.
> Created a pr for this bug: 
> https://github.com/apache/incubator-airflow/pull/2430



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI

[user_airflow (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

user_airflow updated AIRFLOW-1397:
--
Description: Recently upgraded Airflow version from 1.8.0 to 1.8.1. After 
upgrading, the Last Run column in Airflow UI started showing as Blank for all 
the existing dags.

> Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI
> -
>
> Key: AIRFLOW-1397
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1397
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, ui
>Affects Versions: 1.8.1
>Reporter: user_airflow
>Priority: Critical
>
> Recently upgraded Airflow version from 1.8.0 to 1.8.1. After upgrading, the 
> Last Run column in Airflow UI started showing as Blank for all the existing 
> dags.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1366] Add max_tries to task instance

[davydov]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master b532d8d77 -> 4f20f6077


[AIRFLOW-1366] Add max_tries to task instance

Right now Airflow deletes the task instance when
user clear it. We have no way of keeping track of
how many times a task instance gets run either via
user or itself. So instead of deleting the task
instance record, we should keep the task instance
and make try_number monotonically increasing for
every task instance attempt. max_tries is
introduced as an upper bound for retrying tasks by
task itself.

This new column will be used to update logic
behind clear_task_instances.

db migration is tested locally.

Closes #2409 from AllisonWang/allison--max-tries


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/4f20f607
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/4f20f607
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/4f20f607

Branch: refs/heads/master
Commit: 4f20f607764bb3477419321b5dfd0c53ba1db3c0
Parents: b532d8d
Author: AllisonWang 
Authored: Mon Jul 10 15:26:08 2017 -0700
Committer: Dan Davydov 
Committed: Mon Jul 10 15:26:12 2017 -0700

--
 ...dc7_add_max_tries_column_to_task_instance.py | 106 +++
 airflow/models.py   |   3 +
 2 files changed, 109 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4f20f607/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py
--
diff --git 
a/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py
 
b/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py
new file mode 100644
index 000..2d5ffc2
--- /dev/null
+++ 
b/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py
@@ -0,0 +1,106 @@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""add max tries column to task instance
+
+Revision ID: cc1e65623dc7
+Revises: 127d2bf2dfa7
+Create Date: 2017-06-19 16:53:12.851141
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'cc1e65623dc7'
+down_revision = '127d2bf2dfa7'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+from airflow import settings
+from airflow.models import DagBag, TaskInstance
+
+BATCH_SIZE = 5000
+
+def upgrade():
+op.add_column('task_instance', sa.Column('max_tries', sa.Integer,
+server_default="-1"))
+# Check if table task_instance exist before data migration. This check is
+# needed for database that does not create table until migration finishes.
+# Checking task_instance table exists prevent the error of querying
+# non-existing task_instance table.
+engine = settings.engine
+if engine.dialect.has_table(engine, 'task_instance'):
+# Get current session
+connection = op.get_bind()
+sessionmaker = sa.orm.sessionmaker()
+session = sessionmaker(bind=connection)
+dagbag = DagBag(settings.DAGS_FOLDER)
+query = session.query(sa.func.count(TaskInstance.max_tries)).filter(
+TaskInstance.max_tries == -1
+)
+# Separate db query in batch to prevent loading entire table
+# into memory and cause out of memory error.
+while query.scalar():
+tis = session.query(TaskInstance).filter(
+TaskInstance.max_tries == -1
+).limit(BATCH_SIZE).all()
+for ti in tis:
+dag = dagbag.get_dag(ti.dag_id)
+if not dag or not dag.has_task(ti.task_id):
+# task_instance table might not have the up-to-date
+# information, i.e dag or task might be modified or
+# deleted in dagbag but is reflected in task instance
+# table. In this case we do not retry the task that can't
+# be parsed.
+ti.max_tries = ti.try_number
+else:
+task = dag.get_task(ti.task_id)
+ti.max_tries = task.retries
+session.merge(ti)
+session.commit()
+# 

[jira] [Commented] (AIRFLOW-1366) Add max_tries to task instance

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081283#comment-16081283
 ] 

ASF subversion and git services commented on AIRFLOW-1366:
--

Commit 4f20f607764bb3477419321b5dfd0c53ba1db3c0 in incubator-airflow's branch 
refs/heads/master from [~allisonwang]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4f20f60 ]

[AIRFLOW-1366] Add max_tries to task instance

Right now Airflow deletes the task instance when
user clear it. We have no way of keeping track of
how many times a task instance gets run either via
user or itself. So instead of deleting the task
instance record, we should keep the task instance
and make try_number monotonically increasing for
every task instance attempt. max_tries is
introduced as an upper bound for retrying tasks by
task itself.

This new column will be used to update logic
behind clear_task_instances.

db migration is tested locally.

Closes #2409 from AllisonWang/allison--max-tries


> Add max_tries to task instance
> --
>
> Key: AIRFLOW-1366
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1366
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Allison Wang
>Assignee: Allison Wang
>
> Right now Airflow deletes the task instance when user clear it. We have no 
> way of keeping track of how many times a task instance gets run either via 
> user or itself. So instead of deleting the task instance record, we should 
> keep the task instance and make try_number monotonically increasing for every 
> task instance attempt. max_tries is introduced as an upper bound for retrying 
> tasks by task itself.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1397) Airflow 1.8.1 - No data displays in Last Run Column in Airflow UI

[user_airflow (JIRA)]

user_airflow created AIRFLOW-1397:
-

 Summary: Airflow 1.8.1 - No data displays in Last Run Column in 
Airflow UI
 Key: AIRFLOW-1397
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1397
 Project: Apache Airflow
  Issue Type: Bug
  Components: DAG, ui
Affects Versions: 1.8.1
Reporter: user_airflow
Priority: Critical






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-366) SchedulerJob gets locked up when when child processes attempt to log to single file

[Mike Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081168#comment-16081168
 ] 

Mike Perry commented on AIRFLOW-366:


[~bolke] we've been testing with this and seem to be in the clear now: 
https://github.com/astronomerio/incubator-airflow/commit/bdda7dc7e9ba48f981d38dbf3e55473a56bba610.
 Still unsure what the root cause is, but our best guess is possibly with the 
postgres connections that get opened when pickling the processed dags. Perhaps 
they just hang with no response, keeping the process alive. Not sure though.

 I'm thinking it just makes sense to always terminate here, because terminate 
joins with a timeout. Since the DagFileProcessor is presumably done here 
anyway, I don't think there's any harm in explicitly killing the child process 
every time.  Do you foresee any negative side-effects by doing this? 

> SchedulerJob gets locked up when when child processes attempt to log to 
> single file
> ---
>
> Key: AIRFLOW-366
> URL: https://issues.apache.org/jira/browse/AIRFLOW-366
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: scheduler
>Reporter: Greg Neiheisel
>Assignee: Bolke de Bruin
>
> After running the scheduler for a while (usually after 1 - 5 hours) it will 
> eventually lock up, and nothing will get scheduled.
> A `SchedulerJob` will end up getting stuck in the `while` loop around line 
> 730 of `airflow/jobs.py`.
> From what I can tell this is related to logging from within a forked process 
> using pythons multiprocessing module.
> The job will fork off some child processes to process the DAGs but one (or 
> more) will end up getting suck and not terminating, resulting in the while 
> loop getting hung up.  You can `kill -9 PID` the child process manually, and 
> the loop will end and the scheduler will go on it's way, until it happens 
> again.
> The issue is due to usage of the logging module from within the child 
> processes.  From what I can tell, logging to a file from multiple processes 
> is not supported by the multiprocessing module, but it is supported using 
> python multithreading, using some sort of locking mechanism.
> I think a child process will somehow inherit a logger that is locked, right 
> when it is forked, resulting it the process completely locking up.
> I went in and commented out all the logging statements that could possibly be 
> hit by the child process (jobs.py, models.py), and was able to keep the 
> scheduler alive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1300) Enable TBLPROPERTIES parameter in load_df and load_file methods of HiveCliHook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080910#comment-16080910
 ] 

ASF subversion and git services commented on AIRFLOW-1300:
--

Commit b532d8d7742e0a1141732654c2796bfc6dc6cabc in incubator-airflow's branch 
refs/heads/master from Krishna Bhupatiraju
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b532d8d ]

[AIRFLOW-1300] Enable table creation with TBLPROPERTIES

Enable TBLPROPERTIES parameter in load_df and
load_file methods of
HiveCliHook and TransferHive operators

Closes #2364 from
krishnabhupatiraju/tblproperties_hiveclihook


> Enable TBLPROPERTIES parameter in load_df and load_file methods of HiveCliHook
> --
>
> Key: AIRFLOW-1300
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1300
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: krishnabhupatiraju
>Assignee: krishnabhupatiraju
>
> It is currently not possible to set TBLPROPERTIES for tables created through 
> the load_file and load_df methods in HiveCliHook. Please enable this feature. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1300] Enable table creation with TBLPROPERTIES

[saguziel]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 0fc45045a -> b532d8d77


[AIRFLOW-1300] Enable table creation with TBLPROPERTIES

Enable TBLPROPERTIES parameter in load_df and
load_file methods of
HiveCliHook and TransferHive operators

Closes #2364 from
krishnabhupatiraju/tblproperties_hiveclihook


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b532d8d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b532d8d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b532d8d7

Branch: refs/heads/master
Commit: b532d8d7742e0a1141732654c2796bfc6dc6cabc
Parents: 0fc4504
Author: Krishna Bhupatiraju 
Authored: Mon Jul 10 12:14:19 2017 -0700
Committer: Alex Guziel 
Committed: Mon Jul 10 12:14:19 2017 -0700

--
 airflow/hooks/hive_hooks.py  | 29 ---
 airflow/operators/mssql_to_hive.py   |  7 ++-
 airflow/operators/mysql_to_hive.py   |  7 ++-
 airflow/operators/s3_to_hive_operator.py | 10 +++--
 tests/operators/operators.py | 16 +++
 5 files changed, 58 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b532d8d7/airflow/hooks/hive_hooks.py
--
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index 3e7d2db..d120769 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -343,7 +343,8 @@ class HiveCliHook(BaseHook):
 create=True,
 overwrite=True,
 partition=None,
-recreate=False):
+recreate=False,
+tblproperties=None):
 """
 Loads a local file into Hive
 
@@ -354,19 +355,28 @@ class HiveCliHook(BaseHook):
 stage the data into a temporary table before loading it into its
 final destination using a ``HiveOperator``.
 
+:param filepath: local filepath of the file to load
+:type filepath: str   
 :param table: target Hive table, use dot notation to target a
 specific database
 :type table: str
+:param delimiter: field delimiter in the file
+:type delimiter: str
+:param field_dict: A dictionary of the fields name in the file
+as keys and their Hive types as values
+:type field_dict: dict
 :param create: whether to create the table if it doesn't exist
 :type create: bool
-:param recreate: whether to drop and recreate the table at every
-execution
-:type recreate: bool
+:param overwrite: whether to overwrite the data in table or partition
+:type overwrite: bool
 :param partition: target partition as a dict of partition columns
 and values
 :type partition: dict
-:param delimiter: field delimiter in the file
-:type delimiter: str
+:param recreate: whether to drop and recreate the table at every
+execution
+:type recreate: bool
+:param tblproperties: TBLPROPERTIES of the hive table being created
+:type tblproperties: dict
 """
 hql = ''
 if recreate:
@@ -383,7 +393,12 @@ class HiveCliHook(BaseHook):
 hql += "PARTITIONED BY ({pfields})\n"
 hql += "ROW FORMAT DELIMITED\n"
 hql += "FIELDS TERMINATED BY '{delimiter}'\n"
-hql += "STORED AS textfile;"
+hql += "STORED AS textfile\n"
+if tblproperties is not None:
+tprops = ", ".join(
+["'{0}'='{1}'".format(k, v) for k, v in 
tblproperties.items()])
+hql += "TBLPROPERTIES({tprops})\n"
+hql += ";" 
 hql = hql.format(**locals())
 logging.info(hql)
 self.run_cli(hql)

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b532d8d7/airflow/operators/mssql_to_hive.py
--
diff --git a/airflow/operators/mssql_to_hive.py 
b/airflow/operators/mssql_to_hive.py
index 6d7521e..a0a2e10 100644
--- a/airflow/operators/mssql_to_hive.py
+++ b/airflow/operators/mssql_to_hive.py
@@ -57,6 +57,8 @@ class MsSqlToHiveTransfer(BaseOperator):
 :type mssql_conn_id: str
 :param hive_conn_id: destination hive connection
 :type hive_conn_id: str
+:param tblproperties: TBLPROPERTIES of the hive table being created
+:type tblproperties: dict
 """
 
 template_fields = ('sql', 'partition', 'hive_table')
@@ -74,6 +76,7 @@ class MsSqlToHiveTransfer(BaseOperator):
 delimiter=chr(1),
 

[jira] [Created] (AIRFLOW-1394) Add quote_character parameter to GoogleCloudStorageToBigQueryOperator

[David Clubb (JIRA)]

David Clubb created AIRFLOW-1394:


 Summary: Add quote_character parameter to 
GoogleCloudStorageToBigQueryOperator
 Key: AIRFLOW-1394
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1394
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp
Reporter: David Clubb
Assignee: David Clubb


Add quote_character parameter to the GoogleCloudStorageToBigQueryOperator and 
the BigQueryHook.run_load method. This parameter will make it possible to set 
the value that is used to quote data sections in a CSV file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1165) airflow webservice crashes on ubuntu16 - python3

[Arthur Wiedmer (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080698#comment-16080698
 ] 

Arthur Wiedmer commented on AIRFLOW-1165:
-

A short fix until the version is upgraded can be the following

At the prompt
# Generating an RSA public/private-key pair
openssl genrsa -out private.pem 2048
# Generating a self-signed certificate
openssl req -new -x509 -key private.pem -out cacert.pem -days 1095

# In your airflow.cfg under [webserver]
web_server_ssl_cert = path/to/cacert.pem
web_server_ssl_key = path/to/private.pem

> airflow webservice crashes on ubuntu16 - python3 
> -
>
> Key: AIRFLOW-1165
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1165
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Hamed
>Assignee: Arthur Wiedmer
> Fix For: 1.9.0
>
>
> I am trying to run airflow webserver on ubuntu16, python3 and ran to this 
> issue. Any idea?
> {code}
> [2017-05-02 16:36:34,789] [24096] {_internal.py:87} WARNING -  * Debugger is 
> active!
> [2017-05-02 16:36:34,790] [24096] {_internal.py:87} INFO -  * Debugger PIN: 
> 294-518-137
> Exception in thread Thread-1:
> Traceback (most recent call last):
>   File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
> self.run()
>   File "/usr/lib/python3.5/threading.py", line 862, in run
> self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 
> 696, in inner
> fd=fd)
>   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 
> 590, in make_server
> passthrough_errors, ssl_context, fd=fd)
>   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 
> 525, in __init__
> self.socket = ssl_context.wrap_socket(sock, server_side=True)
>   File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 
> 447, in wrap_socket
> ssl_version=self._protocol, **kwargs)
>   File "/usr/lib/python3.5/ssl.py", line 1069, in wrap_socket
> ciphers=ciphers)
>   File "/usr/lib/python3.5/ssl.py", line 680, in __init__
> raise ValueError("certfile must be specified for server-side "
> ValueError: certfile must be specified for server-side operations
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1393) Enable Python3 tests on contrib Spark Submit Hook

[Ash Berlin-Taylor (JIRA)]

Ash Berlin-Taylor created AIRFLOW-1393:
--

 Summary: Enable Python3 tests on contrib Spark Submit Hook
 Key: AIRFLOW-1393
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1393
 Project: Apache Airflow
  Issue Type: Bug
  Components: tests
Reporter: Ash Berlin-Taylor
Priority: Minor


The unit tests in {{tests/contrib/hooks/test_spark_submit_hook.py}} currently 
skip if run in Python3 because some test cases  loop forever due to a 
mismatch/misunderstanding about bytes vs string (i.e. the mocked data for 
{{subprocess.Popen}} doesn't behave the same as actually running Popen)

The fix is to use bytes and {{six.ByteIO}} so that the tests work on Py2 and 
Py3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1392) Documentation link broken

[Daniel Ortega Ufano (JIRA)]

Daniel Ortega Ufano created AIRFLOW-1392:


 Summary: Documentation link broken
 Key: AIRFLOW-1392
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1392
 Project: Apache Airflow
  Issue Type: Bug
  Components: webserver
Affects Versions: 1.8.1
Reporter: Daniel Ortega Ufano
Priority: Minor


The version link refers to https://pypi.python.org/pypi/airflow/1.8.1 which 
responses with HTTP 404 not found. 

I think that should refer to https://pypi.python.org/pypi/apache-airflow/1.8.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1392) Documentation link broken

[Daniel Ortega Ufano (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Ortega Ufano updated AIRFLOW-1392:
-
Attachment: Captura de pantalla 2017-07-10 a las 16.57.17.png

> Documentation link broken
> -
>
> Key: AIRFLOW-1392
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1392
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: webserver
>Affects Versions: 1.8.1
>Reporter: Daniel Ortega Ufano
>Priority: Minor
> Attachments: Captura de pantalla 2017-07-10 a las 16.57.17.png
>
>
> The version link refers to https://pypi.python.org/pypi/airflow/1.8.1 which 
> responses with HTTP 404 not found. 
> I think that should refer to https://pypi.python.org/pypi/apache-airflow/1.8.1



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1391) airflow trigger_dag cannot serialize exec_date when using the json client

[Ricardo Garcia Silva (JIRA)]

Ricardo Garcia Silva created AIRFLOW-1391:
-

 Summary: airflow trigger_dag cannot serialize exec_date when using 
the json client
 Key: AIRFLOW-1391
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1391
 Project: Apache Airflow
  Issue Type: Bug
  Components: api
Affects Versions: 1.8.0
Reporter: Ricardo Garcia Silva


The {{airflow trigger_dag}} command cannot serialize a {{datetime.datetime}} 
when the cli is configured to use the {{json_client}}.
The command:

{code}
airflow trigger_dag --run_id test1 --exec_date 2017-01-01 example_bash_operator
{code}

Throws the error:

{code}
Traceback (most recent call last):
  File "/home/geo2/.venvs/cglops-dissemination/bin/airflow", line 28, in 

args.func(args)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/airflow/bin/cli.py",
 line 180, in trigger_dag
execution_date=args.exec_date)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/airflow/api/client/json_client.py",
 line 32, in trigger_dag
"execution_date": execution_date,
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/api.py",
 line 112, in post
return request('post', url, data=data, json=json, **kwargs)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/api.py",
 line 58, in request
return session.request(method=method, url=url, **kwargs)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/sessions.py",
 line 488, in request
prep = self.prepare_request(req)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/sessions.py",
 line 431, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/models.py",
 line 308, in prepare
self.prepare_body(data, files, json)
  File 
"/home/geo2/.venvs/cglops-dissemination/local/lib/python2.7/site-packages/requests/models.py",
 line 458, in prepare_body
body = complexjson.dumps(json)
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2017, 1, 1, 0, 0) is not JSON serializable
{code}

The same command works fine if airflow is configured to use the 
{{local_client}} instead.
\\

A fix for this would need to encode the {{datetime}} as a string in the client 
then being able to deserialize back to a datetime in the server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1390) Airflow requiring alembic <0.9

[David De La Harpe Golden (JIRA)]

David De La Harpe Golden created AIRFLOW-1390:
-

 Summary: Airflow requiring alembic <0.9
 Key: AIRFLOW-1390
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1390
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.8.1
Reporter: David De La Harpe Golden
Priority: Minor


Airflow setup.py currently appears to require alembic <0.9.  However, alembic 
release is  up to 0.9.3.  Not sure whether this is an oversight or deliberate, 
but it is potentially inconvenient if you need newer alembic features in the 
same env as airflow. (In my case, alembic #412 / postgresql exclusion 
constraint support)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1355) Unable to launch jobs : DAGs not being executed.

[Massimo Menichinelli (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079911#comment-16079911
 ] 

Massimo Menichinelli commented on AIRFLOW-1355:
---

I solved the issue for me: the problem was a recent Docker update. So I had to 
delete all Docker files, install a previous version and reboot. Now it works 
with: Version 17.03.0-ce-mac2 (15654)

> Unable to launch jobs : DAGs not being executed.
> 
>
> Key: AIRFLOW-1355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1355
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG
>Affects Versions: Airflow 2.0
> Environment: Mac OS and Ubuntu
>Reporter: Pavan KN
>
> Steps to re-produce:
> 1. Create new installation
> 2. Launch Airflow
> 3. Enable a DAG and trigger it manually
> DAG/Job won't get executed. Will stay in Running status, but no execution 
> starts and continues to stay at same status.
> Same issues is there with Sequential, Local and Celeri executors.
> Happening in 2.0 version. Tried on multiple Mac machines and on Ubuntu.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)