[jira] [Created] (AIRFLOW-3340) Placeholder support in connections form
Sumit Maheshwari created AIRFLOW-3340: - Summary: Placeholder support in connections form Key: AIRFLOW-3340 URL: https://issues.apache.org/jira/browse/AIRFLOW-3340 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3302) Small CSS fixes
Sumit Maheshwari created AIRFLOW-3302: - Summary: Small CSS fixes Key: AIRFLOW-3302 URL: https://issues.apache.org/jira/browse/AIRFLOW-3302 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3062) Add Qubole in integration docs
[ https://issues.apache.org/jira/browse/AIRFLOW-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-3062. --- Resolution: Fixed Fix Version/s: 2.0.0 > Add Qubole in integration docs > -- > > Key: AIRFLOW-3062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3062 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3294) Add QuboleCheckOperator in integration docs
Sumit Maheshwari created AIRFLOW-3294: - Summary: Add QuboleCheckOperator in integration docs Key: AIRFLOW-3294 URL: https://issues.apache.org/jira/browse/AIRFLOW-3294 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (AIRFLOW-3062) Add Qubole in integration docs
[ https://issues.apache.org/jira/browse/AIRFLOW-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-3062 started by Sumit Maheshwari. - > Add Qubole in integration docs > -- > > Key: AIRFLOW-3062 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3062 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3062) Add Qubole in integration docs
Sumit Maheshwari created AIRFLOW-3062: - Summary: Add Qubole in integration docs Key: AIRFLOW-3062 URL: https://issues.apache.org/jira/browse/AIRFLOW-3062 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2565) Make cluster_label in QuboleOperator templated_field
[ https://issues.apache.org/jira/browse/AIRFLOW-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-2565. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3463 [https://github.com/apache/incubator-airflow/pull/3463] > Make cluster_label in QuboleOperator templated_field > > > Key: AIRFLOW-2565 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2565 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2213) Add Qubole Check Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-2213. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request #3300 [https://github.com/apache/incubator-airflow/pull/3300] > Add Qubole Check Operator > - > > Key: AIRFLOW-2213 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2213 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Sakshi Bansal >Assignee: Sakshi Bansal >Priority: Major > Fix For: 2.0.0 > > > It will be good to have a Qubole Check Operator extending CheckOperator to > handle checks on Qubole commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-1988) TIs with None state are hardly visible due to white background
Sumit Maheshwari created AIRFLOW-1988: - Summary: TIs with None state are hardly visible due to white background Key: AIRFLOW-1988 URL: https://issues.apache.org/jira/browse/AIRFLOW-1988 Project: Apache Airflow Issue Type: Bug Reporter: Sumit Maheshwari Attachments: Screen Shot 2018-01-09 at 6.21.15 PM.png See the attachment -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1315) Add Qubole File and Partition Sensors
[ https://issues.apache.org/jira/browse/AIRFLOW-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1315. --- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2401 [https://github.com/apache/incubator-airflow/pull/2401] > Add Qubole File and Partition Sensors > - > > Key: AIRFLOW-1315 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1315 > Project: Apache Airflow > Issue Type: Task >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > Labels: contrib, qds, sensors > Fix For: 1.9.0 > > > Support to call Qubole's File and Partitions sensors from Airflow. APIs are > listed here http://docs.qubole.com/en/latest/rest-api/sensor_api/index.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1657) Handle failure of Qubole Operator for s3distcp hadoop command
[ https://issues.apache.org/jira/browse/AIRFLOW-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1657. --- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2643 [https://github.com/apache/incubator-airflow/pull/2643] > Handle failure of Qubole Operator for s3distcp hadoop command > - > > Key: AIRFLOW-1657 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1657 > Project: Apache Airflow > Issue Type: Bug >Reporter: Rupesh Bansal >Assignee: Rupesh Bansal > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1677) Fix typo in example_qubole_operator
[ https://issues.apache.org/jira/browse/AIRFLOW-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1677. --- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2661 [https://github.com/apache/incubator-airflow/pull/2661] > Fix typo in example_qubole_operator > --- > > Key: AIRFLOW-1677 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1677 > Project: Apache Airflow > Issue Type: Bug >Reporter: Rupesh Bansal >Assignee: Rupesh Bansal > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-891) Webserver Clock Should Include Day
[ https://issues.apache.org/jira/browse/AIRFLOW-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-891. -- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2399 [https://github.com/apache/incubator-airflow/pull/2399] > Webserver Clock Should Include Day > -- > > Key: AIRFLOW-891 > URL: https://issues.apache.org/jira/browse/AIRFLOW-891 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Dan Davydov >Assignee: Kengo Seki >Priority: Minor > Labels: beginner, starter > Fix For: 1.9.0 > > > Currently the webserver clock shows "23:29 UTC" Ideally it would show the day > as well, like 21-23:29 UTC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1640) Add Qubole default connection in connection table
[ https://issues.apache.org/jira/browse/AIRFLOW-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1640. --- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2630 [https://github.com/apache/incubator-airflow/pull/2630] > Add Qubole default connection in connection table > - > > Key: AIRFLOW-1640 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1640 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Rupesh Bansal >Assignee: Rupesh Bansal > Fix For: 1.9.0 > > > This is to add a qubole default connection. This will create an entry in > `connection` table for qubole_default connection ID. This will help users at > Qubole to easily integrate with Airflow, and hence will reduce the hassle of > setting it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1561) Scheduler doesn't pick up example dags unless there is atleast 1 dag in dags folder
Sumit Maheshwari created AIRFLOW-1561: - Summary: Scheduler doesn't pick up example dags unless there is atleast 1 dag in dags folder Key: AIRFLOW-1561 URL: https://issues.apache.org/jira/browse/AIRFLOW-1561 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.8.2 Reporter: Sumit Maheshwari Attachments: airflow_scheduler_log.txt -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1498) Add Google Analytics to Airflow
Sumit Maheshwari created AIRFLOW-1498: - Summary: Add Google Analytics to Airflow Key: AIRFLOW-1498 URL: https://issues.apache.org/jira/browse/AIRFLOW-1498 Project: Apache Airflow Issue Type: New Feature Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari It would be very useful if we can configure google analytics into Airflow UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1478) Chart -> Owner column should be sortable
[ https://issues.apache.org/jira/browse/AIRFLOW-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1478. --- Resolution: Implemented Fix Version/s: (was: 1.8.1) 1.9.0 > Chart -> Owner column should be sortable > > > Key: AIRFLOW-1478 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1478 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.8.0 >Reporter: Stanislav Kudriashev >Assignee: Stanislav Kudriashev >Priority: Minor > Fix For: 1.9.0 > > Attachments: imageedit_5_9523291349.jpg, imageedit_8_3620789185.jpg > > > Currently owner column is not sortable on the Chart view. > Current: > !imageedit_5_9523291349.jpg! > Expected: > !imageedit_8_3620789185.jpg! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1321) 'Hide sensitive variable fields' feature doesn't works if key is uppercase.
[ https://issues.apache.org/jira/browse/AIRFLOW-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-1321. --- Resolution: Fixed > 'Hide sensitive variable fields' feature doesn't works if key is uppercase. > --- > > Key: AIRFLOW-1321 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1321 > Project: Apache Airflow > Issue Type: Improvement >Affects Versions: Airflow 1.8 >Reporter: Beomjun Kang >Assignee: Kengo Seki > Attachments: screenshot.png > > > I tried to put key as uppercase like 'GOOGLE_API_KEY'. but it doesn't work as > I expected. > So, I changed key like 'GOOGLE_api_key'. Finally, it works. > (Please see the attachment) > This doesn't make sense to me. > How about using case insensitive comparison for checking? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1315) Add Qubole File and Partition Sensors
Sumit Maheshwari created AIRFLOW-1315: - Summary: Add Qubole File and Partition Sensors Key: AIRFLOW-1315 URL: https://issues.apache.org/jira/browse/AIRFLOW-1315 Project: Apache Airflow Issue Type: Task Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari Support to call Qubole's File and Partitions sensors from Airflow. APIs are listed here http://docs.qubole.com/en/latest/rest-api/sensor_api/index.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1192) Add support to run Zeppelin Notebooks with Qubole Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-1192: -- Description: Qubole SDK (qds_sdk) supports running Zeppelin notebooks, but as of now Airflow can't do the same using Qubole's sdk. (was: Qubole SDK (qds_sdk) supports running Zeppelin notebooks, but as of now Airflow cant.) > Add support to run Zeppelin Notebooks with Qubole Operator > --- > > Key: AIRFLOW-1192 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1192 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, operators >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > > Qubole SDK (qds_sdk) supports running Zeppelin notebooks, but as of now > Airflow can't do the same using Qubole's sdk. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1192) Add support to run Zeppelin Notebooks with Qubole Operator
Sumit Maheshwari created AIRFLOW-1192: - Summary: Add support to run Zeppelin Notebooks with Qubole Operator Key: AIRFLOW-1192 URL: https://issues.apache.org/jira/browse/AIRFLOW-1192 Project: Apache Airflow Issue Type: New Feature Components: contrib, operators Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari Qubole SDK (qds_sdk) supports running Zeppelin notebooks, but as of now Airflow cant. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-958) Improve tooltip readability
[ https://issues.apache.org/jira/browse/AIRFLOW-958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari resolved AIRFLOW-958. -- Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2134 [https://github.com/apache/incubator-airflow/pull/2134] > Improve tooltip readability > --- > > Key: AIRFLOW-958 > URL: https://issues.apache.org/jira/browse/AIRFLOW-958 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Jesus Carpintero >Priority: Trivial > Fix For: 1.9.0 > > Attachments: 2017-03-09_1347.png > > > Task tooltips are difficult to read due to the wrapping lines. > See the attachment for my propposed solution -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-692) XCom page is visible to non-admin users
Sumit Maheshwari created AIRFLOW-692: Summary: XCom page is visible to non-admin users Key: AIRFLOW-692 URL: https://issues.apache.org/jira/browse/AIRFLOW-692 Project: Apache Airflow Issue Type: Bug Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari XCom page should not be visible to normal users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page
[ https://issues.apache.org/jira/browse/AIRFLOW-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-620: - External issue URL: https://github.com/apache/incubator-airflow/pull/1875/files > Ability to refresh logs of a TI without reloading whole page > > > Key: AIRFLOW-620 > URL: https://issues.apache.org/jira/browse/AIRFLOW-620 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > Labels: UI, usability > > As of now to see the latest logs of a task instance one has to refresh the > whole page, which is time consuming. It'll be great if just logs can be > re-fetched from backend. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-619) Gantt chart raises exception when there are running TaskInstances of the DagRun
[
https://issues.apache.org/jira/browse/AIRFLOW-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari resolved AIRFLOW-619.
--
Resolution: Fixed
Issue resolved by pull request #1874
[https://github.com/apache/incubator-airflow/pull/1874]
> Gantt chart raises exception when there are running TaskInstances of the
> DagRun
> ---
>
> Key: AIRFLOW-619
> URL: https://issues.apache.org/jira/browse/AIRFLOW-619
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 2.0
>Reporter: Daniel Gies
> Labels: gantt
>
> If you view the Gantt chart for a DagRun while there are TaskInstances still
> running, the Gantt chart throws an exception calling isoformat() on None
> because the task instance has no end_date
> {code}
> Traceback (most recent call last):
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask/app.py",
> line 1817, in wsgi_app
> response = self.full_dispatch_request()
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask/app.py",
> line 1477, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask/app.py",
> line 1381, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask/app.py",
> line 1475, in full_dispatch_request
> rv = self.dispatch_request()
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask/app.py",
> line 1461, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask_admin/base.py",
> line 68, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask_admin/base.py",
> line 367, in _run_view
> return fn(self, *args, **kwargs)
> File "/home/dgies/airflow/venv/lib/python2.7/site-packages/flask_login.py",
> line 755, in decorated_view
> return func(*args, **kwargs)
> File
> "/home/dgies/airflow/venv/lib/python2.7/site-packages/airflow/www/utils.py",
> line 125, in wrapper
> return f(*args, **kwargs)
> File
> "/home/dgies/airflow/venv/lib/python2.7/site-packages/airflow/www/views.py",
> line 1701, in gantt
> 'isoEnd': ti.end_date.isoformat()[:-4],
> AttributeError: 'NoneType' object has no attribute 'isoformat'
> {code}
> Proposed fix is to use `ti.end_date or datetime.now()` everywhere ti.end_date
> is used in the view. This problem doesn't seem to affect any of the other
> views, only Gantt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-620) Ability to refresh logs of a TI without reloading whole page
Sumit Maheshwari created AIRFLOW-620: Summary: Ability to refresh logs of a TI without reloading whole page Key: AIRFLOW-620 URL: https://issues.apache.org/jira/browse/AIRFLOW-620 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari As of now to see the latest logs of a task instance one has to refresh the whole page, which is time consuming. It'll be great if just logs can be re-fetched from backend. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-592) example_xcom import Error
[
https://issues.apache.org/jira/browse/AIRFLOW-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari resolved AIRFLOW-592.
--
Resolution: Fixed
Issue resolved by pull request #1853
[https://github.com/apache/incubator-airflow/pull/1853]
> example_xcom import Error
> -
>
> Key: AIRFLOW-592
> URL: https://issues.apache.org/jira/browse/AIRFLOW-592
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Centos
>Reporter: forevernull
>Assignee: forevernull
>
> python ./airflow/example_dags/example_xcom.py
> [2016-10-23 23:53:13,160] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> Traceback (most recent call last):
> File "./airflow/example_dags/example_xcom.py", line 62, in
> push1 = airflow.operators.python_operator.PythonOperator(
> AttributeError: 'module' object has no attribute 'python_operator'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-319) Optionally push HTTP response from HTTP_simple_operator to XCom
[
https://issues.apache.org/jira/browse/AIRFLOW-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari resolved AIRFLOW-319.
--
Resolution: Fixed
Fix Version/s: (was: Airflow 1.8)
Issue resolved by pull request #1658
[https://github.com/apache/incubator-airflow/pull/1658]
> Optionally push HTTP response from HTTP_simple_operator to XCom
> ---
>
> Key: AIRFLOW-319
> URL: https://issues.apache.org/jira/browse/AIRFLOW-319
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: Airflow 1.7.1.3
>Reporter: Jay Zelenkov
>Assignee: Jay Zelenkov
>Priority: Minor
>
> Documentation suggests that any operator's execute method that returns value
> should push it into XCom:
> http://airflow.incubator.apache.org/concepts.html#xcoms
> {quote}
> In addition, if a task returns a value (either from its Operator’s execute()
> method, or from a PythonOperator’s python_callable function), then an XCom
> containing that value is automatically pushed.
> {quote}
>
> This issue addresses missing xcom push behavior in {{http_operator}}
> Current implementation follows a pattern used in {{BashOperator}} and
> suggested by [mistercrunch|https://github.com/mistercrunch] in [GitHub PR
> 658|https://github.com/apache/incubator-airflow/pull/658]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-96) s3_conn_id of S3KeySensor and S3PrefixSensor cannot be defined using an environment variable
[
https://issues.apache.org/jira/browse/AIRFLOW-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari resolved AIRFLOW-96.
-
Resolution: Fixed
Issue resolved by pull request #1517
[https://github.com/apache/incubator-airflow/pull/1517]
> s3_conn_id of S3KeySensor and S3PrefixSensor cannot be defined using an
> environment variable
>
>
> Key: AIRFLOW-96
> URL: https://issues.apache.org/jira/browse/AIRFLOW-96
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.6.2
> Environment: Python Version: 2.7.11
> Operating System: OS X El Capitan 10.11.4
>Reporter: dmtran
>Assignee: dmtran
>Priority: Minor
>
> According to https://pythonhosted.org/airflow/concepts.html#connections,
> Airflow has the ability to reference connections via environment variables
> from the operating system. The environment variable needs to be prefixed with
> AIRFLOW_CONN_ to be considered a connection.
> This doesn't work with S3KeySensor (or S3PrefixSensor) , the following
> exception is raised:
> {noformat}
> [2016-05-10 17:01:37,101] {models.py:1041} ERROR - conn_id doesn't exist in
> the repository
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1000,
> in run
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py",
> line 65, in execute
> return_value = self.python_callable(*self.op_args, **self.op_kwargs)
> File "/Users/dmtran/git_repos/coordinator/airflow/dags/test-s3.py", line
> 24, in check_key_in_s3
> s3_conn_id='S3_CONNECTION')
> File "/usr/local/lib/python2.7/site-packages/airflow/utils.py", line 461,
> in wrapper
> result = func(*args, **kwargs)
> File "/usr/local/lib/python2.7/site-packages/airflow/operators/sensors.py",
> line 332, in __init__
> raise AirflowException("conn_id doesn't exist in the repository")
> AirflowException: conn_id doesn't exist in the repository
> {noformat}
> You can reproduce this issue by triggering a DAGrun of the following DAG:
> {code}
> from airflow.hooks.base_hook import CONN_ENV_PREFIX
> from airflow.operators import *
> from airflow.models import DAG
> from datetime import datetime
> import os
> args = {
> 'owner': 'airflow',
> 'start_date': datetime(2016, 5, 10, 7)
> }
> dag = DAG(dag_id='test-s3',
> default_args=args,
> schedule_interval=None)
> os.environ[CONN_ENV_PREFIX + 'S3_CONNECTION'] = 's3://:
> dummyAccessKey:dummySecretKey@S3'
> sensor = S3KeySensor(
> task_id='s3keysensor',
> bucket_name='dummy_bucket',
> bucket_key='dummy_key',
> dag=dag,
> s3_conn_id='S3_CONNECTION')
> {code}
> The exception is raised because of the following lines in method __init__ of
> class S3KeySensor:
> {code}
> db = session.query(DB).filter(DB.conn_id == s3_conn_id).first()
> if not db:
> raise AirflowException("conn_id doesn't exist in the repository")
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-566) Log view takes forever on tasks which not executed locally
Sumit Maheshwari created AIRFLOW-566: Summary: Log view takes forever on tasks which not executed locally Key: AIRFLOW-566 URL: https://issues.apache.org/jira/browse/AIRFLOW-566 Project: Apache Airflow Issue Type: Bug Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari When a task not ran on same machine on which webserver is running and that machine is down, Airflow takes a whole lote of time in trying out and finally gets time out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-436) Incorrect templating when used with .format
[
https://issues.apache.org/jira/browse/AIRFLOW-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571612#comment-15571612
]
Sumit Maheshwari commented on AIRFLOW-436:
--
Thanks [~lauralorenz] for detailed explanation.
> Incorrect templating when used with .format
> ---
>
> Key: AIRFLOW-436
> URL: https://issues.apache.org/jira/browse/AIRFLOW-436
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sumit Maheshwari
>
> Found that when {{.format}} is used in a templatable field, it produces weird
> results. For example:
> {code:none}
> t3 = BashOperator(
> task_id='wget',
> bash_command="wget {0}/{{ ds }}".format('google.com'),
> dag=dag)
> {code}
> produces following result:
> {code:none}
> {bash_operator.py:79} INFO - Running command: wget google.com/{ ds }
> {code}
> But if we change {{bash_command}} to following, it works as expected:
> {code:none}
> bash_command="wget %s/{{ ds }}"%('google.com')
> {code}
> {code:none}
> {bash_operator.py:79} INFO - Running command: wget google.com/2016-08-05
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-436) Incorrect templating when used with .format
[
https://issues.apache.org/jira/browse/AIRFLOW-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari resolved AIRFLOW-436.
--
Resolution: Workaround
> Incorrect templating when used with .format
> ---
>
> Key: AIRFLOW-436
> URL: https://issues.apache.org/jira/browse/AIRFLOW-436
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sumit Maheshwari
>
> Found that when {{.format}} is used in a templatable field, it produces weird
> results. For example:
> {code:none}
> t3 = BashOperator(
> task_id='wget',
> bash_command="wget {0}/{{ ds }}".format('google.com'),
> dag=dag)
> {code}
> produces following result:
> {code:none}
> {bash_operator.py:79} INFO - Running command: wget google.com/{ ds }
> {code}
> But if we change {{bash_command}} to following, it works as expected:
> {code:none}
> bash_command="wget %s/{{ ds }}"%('google.com')
> {code}
> {code:none}
> {bash_operator.py:79} INFO - Running command: wget google.com/2016-08-05
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-553) Wrong path used for filter.js
[
https://issues.apache.org/jira/browse/AIRFLOW-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari updated AIRFLOW-553:
-
External issue URL: https://github.com/apache/incubator-airflow/pull/1827
Component/s: ui
> Wrong path used for filter.js
> -
>
> Key: AIRFLOW-553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-553
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
> Attachments: Screen Shot 2016-10-06 at 6.49.26 PM.png
>
>
> At DAG model view page, filters doesn't work due to wrong path while loading
> {{filter.js}}. PFA image for the errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-553) Wrong path used for filter.js
[
https://issues.apache.org/jira/browse/AIRFLOW-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari updated AIRFLOW-553:
-
Attachment: Screen Shot 2016-10-06 at 6.49.26 PM.png
> Wrong path used for filter.js
> -
>
> Key: AIRFLOW-553
> URL: https://issues.apache.org/jira/browse/AIRFLOW-553
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
> Attachments: Screen Shot 2016-10-06 at 6.49.26 PM.png
>
>
> At DAG model view page, filters doesn't work due to wrong path while loading
> {{filter.js}}. PFA image for the errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-553) Wrong path used for filter.js
Sumit Maheshwari created AIRFLOW-553:
Summary: Wrong path used for filter.js
Key: AIRFLOW-553
URL: https://issues.apache.org/jira/browse/AIRFLOW-553
Project: Apache Airflow
Issue Type: Bug
Reporter: Sumit Maheshwari
Assignee: Sumit Maheshwari
At DAG model view page, filters doesn't work due to wrong path while loading
{{filter.js}}. PFA image for the errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-544) Ability to Pause/Resume a DAG from its details page
[ https://issues.apache.org/jira/browse/AIRFLOW-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-544: - External issue URL: https://github.com/apache/incubator-airflow/pull/1818 > Ability to Pause/Resume a DAG from its details page > --- > > Key: AIRFLOW-544 > URL: https://issues.apache.org/jira/browse/AIRFLOW-544 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > Labels: ui, usability > > Almost everytime I've to go back and forth on main page to a dag page to > pause/resume it. It'll be great if I can view details and pause/resume from > there itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (AIRFLOW-544) Ability to Pause/Resume a DAG from its details page
[ https://issues.apache.org/jira/browse/AIRFLOW-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-544 started by Sumit Maheshwari. > Ability to Pause/Resume a DAG from its details page > --- > > Key: AIRFLOW-544 > URL: https://issues.apache.org/jira/browse/AIRFLOW-544 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > Labels: ui, usability > > Almost everytime I've to go back and forth on main page to a dag page to > pause/resume it. It'll be great if I can view details and pause/resume from > there itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-544) Ability to Pause/Resume a DAG from its details page
Sumit Maheshwari created AIRFLOW-544: Summary: Ability to Pause/Resume a DAG from its details page Key: AIRFLOW-544 URL: https://issues.apache.org/jira/browse/AIRFLOW-544 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari Almost everytime I've to go back and forth on main page to a dag page to pause/resume it. It'll be great if I can view details and pause/resume from there itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-543) Tailing of logs for running instances
Sumit Maheshwari created AIRFLOW-543:
Summary: Tailing of logs for running instances
Key: AIRFLOW-543
URL: https://issues.apache.org/jira/browse/AIRFLOW-543
Project: Apache Airflow
Issue Type: Wish
Components: logging, webserver
Reporter: Sumit Maheshwari
As of now one has to refresh the task log viewer page to see the latest logs.
It would've been great if logs get automatically pushed to browser for running
instances in {{tail -f}} manner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-534) Add test coverage for Qubole operator
Sumit Maheshwari created AIRFLOW-534: Summary: Add test coverage for Qubole operator Key: AIRFLOW-534 URL: https://issues.apache.org/jira/browse/AIRFLOW-534 Project: Apache Airflow Issue Type: Task Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-437) Provide task context while calling ti.handle_failure during kill_zombies
[
https://issues.apache.org/jira/browse/AIRFLOW-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari updated AIRFLOW-437:
-
External issue URL: https://github.com/apache/incubator-airflow/pull/1796
> Provide task context while calling ti.handle_failure during kill_zombies
>
>
> Key: AIRFLOW-437
> URL: https://issues.apache.org/jira/browse/AIRFLOW-437
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>
> Without proper context {{ti.handler_failure}} is of no use and its as good as
> marking those TIs failed directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-463) Airflow image should take users to landing page
Sumit Maheshwari created AIRFLOW-463: Summary: Airflow image should take users to landing page Key: AIRFLOW-463 URL: https://issues.apache.org/jira/browse/AIRFLOW-463 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari As of now the Airflow image icon on top left doesn't leads users to anywhere. It should take users to initial landing page, which is generally happened on most of the other sites. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-437) Provide task context while calling ti.handle_failure during kill_zombies
Sumit Maheshwari created AIRFLOW-437:
Summary: Provide task context while calling ti.handle_failure
during kill_zombies
Key: AIRFLOW-437
URL: https://issues.apache.org/jira/browse/AIRFLOW-437
Project: Apache Airflow
Issue Type: Bug
Reporter: Sumit Maheshwari
Assignee: Sumit Maheshwari
Without proper context {{ti.handler_failure}} is of no use and its as good as
marking those TIs failed directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-436) Incorrect templating when used with .format
Sumit Maheshwari created AIRFLOW-436:
Summary: Incorrect templating when used with .format
Key: AIRFLOW-436
URL: https://issues.apache.org/jira/browse/AIRFLOW-436
Project: Apache Airflow
Issue Type: Bug
Reporter: Sumit Maheshwari
Found that when {{.format}} is used in a templatable field, it produces weird
results. For example:
{code:none}
t3 = BashOperator(
task_id='wget',
bash_command="wget {0}/{{ ds }}".format('google.com'),
dag=dag)
{code}
produces following result:
{code:none}
{bash_operator.py:79} INFO - Running command: wget google.com/{ ds }
{code}
But if we change {{bash_command}} to following, it works as expected:
{code:none}
bash_command="wget %s/{{ ds }}"%('google.com')
{code}
{code:none}
{bash_operator.py:79} INFO - Running command: wget google.com/2016-08-05
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-361) Default failure and retry handler in Qubole Operator
Sumit Maheshwari created AIRFLOW-361: Summary: Default failure and retry handler in Qubole Operator Key: AIRFLOW-361 URL: https://issues.apache.org/jira/browse/AIRFLOW-361 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari It would be nice if there is a default handler for failures and retries in all operators. I've a working model for Qubole Operator and will open a PR for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-339) Passing extra params to Airflow Flower
[
https://issues.apache.org/jira/browse/AIRFLOW-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari reassigned AIRFLOW-339:
Assignee: Sumit Maheshwari
> Passing extra params to Airflow Flower
> --
>
> Key: AIRFLOW-339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-339
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>
> Flower accepts around 2 dozen different parameters, but via {{airflow
> flower}} I can override only {{port}} and {{broker_api}}.
> There should be some way to pass on all {{flower}} supported params via
> {{airflow}} as well. For now I must need to pass {{url_prefix}} to
> {{flower}}, someone might want to pass {{ca_certs}} or other params in
> future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AIRFLOW-296) Getting TemplateNotFound Error while using QuboleOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumit Maheshwari updated AIRFLOW-296:
-
Description:
Getting following error while using Qubole operator.
{{jinja2.exceptions.TemplateNotFound: select count(*) from default.payment}}
Found the error and fix, will open a PR soon.
was:
Getting following error while using Qubole operator.
{{ jinja2.exceptions.TemplateNotFound: select count(*) from default.payment }}
Found the error and fix, will open a PR soon.
> Getting TemplateNotFound Error while using QuboleOperator
> -
>
> Key: AIRFLOW-296
> URL: https://issues.apache.org/jira/browse/AIRFLOW-296
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>
> Getting following error while using Qubole operator.
> {{jinja2.exceptions.TemplateNotFound: select count(*) from default.payment}}
> Found the error and fix, will open a PR soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-296) Getting TemplateNotFound Error while using QuboleOperator
Sumit Maheshwari created AIRFLOW-296:
Summary: Getting TemplateNotFound Error while using QuboleOperator
Key: AIRFLOW-296
URL: https://issues.apache.org/jira/browse/AIRFLOW-296
Project: Apache Airflow
Issue Type: Bug
Reporter: Sumit Maheshwari
Assignee: Sumit Maheshwari
Getting following error while using Qubole operator.
{{ jinja2.exceptions.TemplateNotFound: select count(*) from default.payment }}
Found the error and fix, will open a PR soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari reassigned AIRFLOW-161: Assignee: Sumit Maheshwari (was: Sumit Maheshwari) > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307235#comment-15307235
]
Sumit Maheshwari edited comment on AIRFLOW-184 at 5/31/16 5:40 AM:
---
Was checking {{airflow clear}} and it seems like its doable by saying {{airflow
-f -s start_date -e end_date dag_id}} and cleverly choosing start_date and
end_date at the boundaries of desire execution date, which is tricky and risky
for automated scripts.
Also seems that it clears task instances with {{failed}} state only, not sure
how does it affects instances with state {{upstream_failed}}.
was (Author: sumitm):
Was checking "airflow clear" and it seems like its doable by saying "airflow -f
-s start_date -e end_date dag_id" and cleverly choosing start_date and end_date
at the boundaries of desire execution date, which is tricky and risky for
automated scripts.
Also seems that it clears task instances with "failed" state only, not sure how
does it affects instances with state "upstream_failed".
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307235#comment-15307235
]
Sumit Maheshwari edited comment on AIRFLOW-184 at 5/31/16 5:39 AM:
---
Was checking "airflow clear" and it seems like its doable by saying "airflow -f
-s start_date -e end_date dag_id" and cleverly choosing start_date and end_date
at the boundaries of desire execution date, which is tricky and risky for
automated scripts.
Also seems that it clears task instances with "failed" state only, not sure how
does it affects instances with state "upstream_failed".
was (Author: sumitm):
Was checking {{ airflow clear }} and it seems like its doable by saying {{
airflow -f -s start_date -e end_date dag_id }} and cleverly choosing start_date
and end_date at the boundaries of desire execution date, which is tricky and
risky for automated scripts.
Also seems that it clears task instances with {{ failed }} state only, not sure
how does it affects instances with state {{ upstream_failed }}.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-184) Add clear/mark success to CLI
[
https://issues.apache.org/jira/browse/AIRFLOW-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307235#comment-15307235
]
Sumit Maheshwari commented on AIRFLOW-184:
--
Was checking {{ airflow clear }} and it seems like its doable by saying {{
airflow -f -s start_date -e end_date dag_id }} and cleverly choosing start_date
and end_date at the boundaries of desire execution date, which is tricky and
risky for automated scripts.
Also seems that it clears task instances with {{ failed }} state only, not sure
how does it affects instances with state {{ upstream_failed }}.
> Add clear/mark success to CLI
> -
>
> Key: AIRFLOW-184
> URL: https://issues.apache.org/jira/browse/AIRFLOW-184
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Reporter: Chris Riccomini
>
> AIRFLOW-177 pointed out that the current CLI does not allow us to clear or
> mark success a task (including upstream, downstream, past, future, and
> recursive) the way that the UI widget does. Given a goal of keeping parity
> between the UI and CLI, it seems like we should support this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (AIRFLOW-177) Resume a failed dag
[ https://issues.apache.org/jira/browse/AIRFLOW-177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari reopened AIRFLOW-177: -- > Resume a failed dag > --- > > Key: AIRFLOW-177 > URL: https://issues.apache.org/jira/browse/AIRFLOW-177 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Reporter: Sumit Maheshwari > > Say I've a dag with 10 nodes and one of the dag run got failed at 5th node. > Now if I want to resume that dag, I can go and run individual task one by > one. Is there any way by which I can just tell dag_id and execution_date (or > run_id) and it automatically retries only failed tasks? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-177) Resume a failed dag
[
https://issues.apache.org/jira/browse/AIRFLOW-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303644#comment-15303644
]
Sumit Maheshwari commented on AIRFLOW-177:
--
[~criccomini] I was thinking that would it be helpful to have a cli support for
that.. say {{ resume_dag }} which takes a execution_date as input and rerun
only failed tasks?
> Resume a failed dag
> ---
>
> Key: AIRFLOW-177
> URL: https://issues.apache.org/jira/browse/AIRFLOW-177
> Project: Apache Airflow
> Issue Type: New Feature
> Components: core
>Reporter: Sumit Maheshwari
>
> Say I've a dag with 10 nodes and one of the dag run got failed at 5th node.
> Now if I want to resume that dag, I can go and run individual task one by
> one. Is there any way by which I can just tell dag_id and execution_date (or
> run_id) and it automatically retries only failed tasks?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-177) Resume a failed dag
[ https://issues.apache.org/jira/browse/AIRFLOW-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302317#comment-15302317 ] Sumit Maheshwari commented on AIRFLOW-177: -- Thanks Chris. > Resume a failed dag > --- > > Key: AIRFLOW-177 > URL: https://issues.apache.org/jira/browse/AIRFLOW-177 > Project: Apache Airflow > Issue Type: New Feature > Components: core >Reporter: Sumit Maheshwari > > Say I've a dag with 10 nodes and one of the dag run got failed at 5th node. > Now if I want to resume that dag, I can go and run individual task one by > one. Is there any way by which I can just tell dag_id and execution_date (or > run_id) and it automatically retries only failed tasks? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari reassigned AIRFLOW-161: Assignee: Sumit Maheshwari > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-177) Resume a failed dag
Sumit Maheshwari created AIRFLOW-177: Summary: Resume a failed dag Key: AIRFLOW-177 URL: https://issues.apache.org/jira/browse/AIRFLOW-177 Project: Apache Airflow Issue Type: New Feature Components: core Reporter: Sumit Maheshwari Say I've a dag with 10 nodes and one of the dag run got failed at 5th node. Now if I want to resume that dag, I can go and run individual task one by one. Is there any way by which I can just tell dag_id and execution_date (or run_id) and it automatically retries only failed tasks? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-167) Get dag state for a given execution date.
[ https://issues.apache.org/jira/browse/AIRFLOW-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari reassigned AIRFLOW-167: Assignee: Sumit Maheshwari > Get dag state for a given execution date. > - > > Key: AIRFLOW-167 > URL: https://issues.apache.org/jira/browse/AIRFLOW-167 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > > I was trying to get state for a particular dag-run programmatically, but > couldn't find a way. > If we could have a rest call like > `/admin/dagrun?dag_id=_date=` and get the output that > would be best. Currently we've to do html parsing to get the same. > Other (and easier) way is to add a cli support like we have for `task_state`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice
[ https://issues.apache.org/jira/browse/AIRFLOW-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-168: - Description: I was looking at example_xcom example and found that it got scheduled twice. Ones at the start_time and ones at the current time. To be correct I tried multiple times (by reloading db) and its same. I am on airflow master, using sequential executor with sqlite3. Though it works as expected on a prod env which is running v1.7 with celery workers and mysql backend. was: I was looking at example_xcom example and found that it got scheduled twice. Ones at the start_time and ones at the current time. To be correct I tried multiple times (by reloading db) and its same. I am on airflow master, using sequential executor with sqlite3. > schedule_interval @once scheduling dag atleast twice > > > Key: AIRFLOW-168 > URL: https://issues.apache.org/jira/browse/AIRFLOW-168 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: Airflow 1.7.1.2 >Reporter: Sumit Maheshwari > Attachments: Screen Shot 2016-05-24 at 9.51.50 PM.png, > screenshot-1.png > > > I was looking at example_xcom example and found that it got scheduled twice. > Ones at the start_time and ones at the current time. To be correct I tried > multiple times (by reloading db) and its same. > I am on airflow master, using sequential executor with sqlite3. Though it > works as expected on a prod env which is running v1.7 with celery workers and > mysql backend. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-161: - External issue URL: https://github.com/apache/incubator-airflow/pull/1538/files > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301450#comment-15301450 ] Sumit Maheshwari commented on AIRFLOW-161: -- Sure, can you please cc top contributors to PR or here. > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301400#comment-15301400 ] Sumit Maheshwari commented on AIRFLOW-161: -- Fair enough, I can't challenge that decision, as Qubole is not as big as aws or gce :). However as that link will be visible only for qubole_operator type tasks, which implies that user is using qubole and having that link will help him. Also I think that airflow gonna need /redirect (or similar) route in near future. > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-169) Hide expire dags in UI
[ https://issues.apache.org/jira/browse/AIRFLOW-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301346#comment-15301346 ] Sumit Maheshwari commented on AIRFLOW-169: -- I am referring to landing page, ie. /admin. Expired dags means dags which are supposed to run @once and already ran, or dags with end_time which is in past to current time. Similarly if in cli, we can pass some option (say -e) to list_dags command, which will ignores those expired dags. > Hide expire dags in UI > -- > > Key: AIRFLOW-169 > URL: https://issues.apache.org/jira/browse/AIRFLOW-169 > Project: Apache Airflow > Issue Type: Wish > Components: ui >Reporter: Sumit Maheshwari > > It would be great if we've option to hide expired schedules from UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-167) Get dag state for a given execution date.
[ https://issues.apache.org/jira/browse/AIRFLOW-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301338#comment-15301338 ] Sumit Maheshwari commented on AIRFLOW-167: -- Yup, already opened https://github.com/apache/incubator-airflow/pull/1541/files. > Get dag state for a given execution date. > - > > Key: AIRFLOW-167 > URL: https://issues.apache.org/jira/browse/AIRFLOW-167 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Reporter: Sumit Maheshwari > > I was trying to get state for a particular dag-run programmatically, but > couldn't find a way. > If we could have a rest call like > `/admin/dagrun?dag_id=_date=` and get the output that > would be best. Currently we've to do html parsing to get the same. > Other (and easier) way is to add a cli support like we have for `task_state`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-169) Hide expire dags in UI
Sumit Maheshwari created AIRFLOW-169: Summary: Hide expire dags in UI Key: AIRFLOW-169 URL: https://issues.apache.org/jira/browse/AIRFLOW-169 Project: Apache Airflow Issue Type: Wish Reporter: Sumit Maheshwari It would be great if we've option to hide expired schedules from UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice
[ https://issues.apache.org/jira/browse/AIRFLOW-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-168: - Attachment: Screen Shot 2016-05-24 at 9.51.50 PM.png > schedule_interval @once scheduling dag atleast twice > > > Key: AIRFLOW-168 > URL: https://issues.apache.org/jira/browse/AIRFLOW-168 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Sumit Maheshwari > Attachments: Screen Shot 2016-05-24 at 9.51.50 PM.png > > > I was looking at example_xcom example and found that it got scheduled twice. > Ones at the start_time and ones at the current time. To be correct I tried > multiple times (by reloading db) and its same. > I am on airflow master, using sequential executor with sqlite3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice
[ https://issues.apache.org/jira/browse/AIRFLOW-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit Maheshwari updated AIRFLOW-168: - Attachment: Screen Shot 2016-05-24 at 9.39.17 PM.png > schedule_interval @once scheduling dag atleast twice > > > Key: AIRFLOW-168 > URL: https://issues.apache.org/jira/browse/AIRFLOW-168 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Reporter: Sumit Maheshwari > Attachments: Screen Shot 2016-05-24 at 9.39.17 PM.png > > > I was looking at example_xcom example and found that it got scheduled twice. > Ones at the start_time and ones at the current time. To be correct I tried > multiple times (by reloading db) and its same. > I am on airflow master, using sequential executor with sqlite3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-168) schedule_interval @once scheduling dag atleast twice
Sumit Maheshwari created AIRFLOW-168: Summary: schedule_interval @once scheduling dag atleast twice Key: AIRFLOW-168 URL: https://issues.apache.org/jira/browse/AIRFLOW-168 Project: Apache Airflow Issue Type: Bug Components: scheduler Reporter: Sumit Maheshwari I was looking at example_xcom example and found that it got scheduled twice. Ones at the start_time and ones at the current time. To be correct I tried multiple times (by reloading db) and its same. I am on airflow master, using sequential executor with sqlite3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-167) Get dag state for a given execution date.
Sumit Maheshwari created AIRFLOW-167: Summary: Get dag state for a given execution date. Key: AIRFLOW-167 URL: https://issues.apache.org/jira/browse/AIRFLOW-167 Project: Apache Airflow Issue Type: Improvement Reporter: Sumit Maheshwari I was trying to get state for a particular dag-run programmatically, but couldn't find a way. If we could have a rest call like `/admin/dagrun?dag_id=_date=` and get the output that would be best. Currently we've to do html parsing to get the same. Other (and easier) way is to add a cli support like we have for `task_state`. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-161) Redirection to external url
[ https://issues.apache.org/jira/browse/AIRFLOW-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297652#comment-15297652 ] Sumit Maheshwari commented on AIRFLOW-161: -- Hi Chris, Yes, for offline things like mail or slack this is what we are using as well, but for real time cases like mine, it doesn't suit. Please take a look on this PR (https://github.com/apache/incubator-airflow/pull/1538) and let me know if I can achieve similar using some other way. Thanks, Sumit > Redirection to external url > --- > > Key: AIRFLOW-161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-161 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Reporter: Sumit Maheshwari > > Hi, > I am not able to find a good way (apart from loading everything upfront), > where I can redirect someone to a external service url, using the information > stored in airflow. There could be many use cases like downloading a signed > file from s3, redirecting to hadoop job tracker, or a direct case on which I > am working which is linking airflow tasks to qubole commands. > I already have a working model and will open a PR soon. Please let me know if > there existing ways already. > Thanks, > Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-161) Redirection to external url
Sumit Maheshwari created AIRFLOW-161: Summary: Redirection to external url Key: AIRFLOW-161 URL: https://issues.apache.org/jira/browse/AIRFLOW-161 Project: Apache Airflow Issue Type: Improvement Components: webserver Reporter: Sumit Maheshwari Hi, I am not able to find a good way (apart from loading everything upfront), where I can redirect someone to a external service url, using the information stored in airflow. There could be many use cases like downloading a signed file from s3, redirecting to hadoop job tracker, or a direct case on which I am working which is linking airflow tasks to qubole commands. I already have a working model and will open a PR soon. Please let me know if there existing ways already. Thanks, Sumit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293541#comment-15293541 ] Sumit Maheshwari commented on AIRFLOW-119: -- Thanks. Opened a jira for documentation (https://issues.apache.org/jira/browse/AIRFLOW-155). > "Template Not Found" error from QuboleOperator > -- > > Key: AIRFLOW-119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-119 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.7.0 >Reporter: Sumit Maheshwari >Assignee: Sumit Maheshwari > Labels: qds > > From the given example set, when I am using qbol operator for a hive > workload, whose script resides in s3 and ends with ".qbl", I am getting > "Template Not Found" error. > Also it would be nice, if airflow always tags commands going from airflow to > qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292680#comment-15292680 ] Sumit Maheshwari commented on AIRFLOW-119: -- But I'll need some time, maybe by next week I can open another PR. So until then can we merge this change? > "Template Not Found" error from QuboleOperator > -- > > Key: AIRFLOW-119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-119 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.7.0 >Reporter: Sumit Maheshwari > Labels: qds > > From the given example set, when I am using qbol operator for a hive > workload, whose script resides in s3 and ends with ".qbl", I am getting > "Template Not Found" error. > Also it would be nice, if airflow always tags commands going from airflow to > qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291659#comment-15291659 ] Sumit Maheshwari commented on AIRFLOW-119: -- Yeah, sure. I'll add relevant documentation about qubole operator in "Community-contributed Operators" section soon. > "Template Not Found" error from QuboleOperator > -- > > Key: AIRFLOW-119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-119 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.7.0 >Reporter: Sumit Maheshwari > Labels: qds > > From the given example set, when I am using qbol operator for a hive > workload, whose script resides in s3 and ends with ".qbl", I am getting > "Template Not Found" error. > Also it would be nice, if airflow always tags commands going from airflow to > qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291382#comment-15291382
]
Sumit Maheshwari edited comment on AIRFLOW-119 at 5/19/16 4:12 PM:
---
Let me explain my point, if its a hive_operator than exts would be .hql/.sql,
if its bash_operator than it would be .sh/.bash, but a qubole operator is
combination of many such operators, so we would need a generic ext. I thought
of .qbl initially, but .txt seems to be more generic and known to everyone.
was (Author: sumitm):
Let me explain my point, if its a hive_operator than exts would be .hql/.sql,
if its bash_operator than it would be .sh/.bash, but a qubole operator is
combination of many such operators, so we would need a generic ext. I thought
of {{ .qbl }} initially, but {{ .txt }} seems to be more generic and know to
everyone.
> "Template Not Found" error from QuboleOperator
> --
>
> Key: AIRFLOW-119
> URL: https://issues.apache.org/jira/browse/AIRFLOW-119
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.7.0
>Reporter: Sumit Maheshwari
> Labels: qds
>
> From the given example set, when I am using qbol operator for a hive
> workload, whose script resides in s3 and ends with ".qbl", I am getting
> "Template Not Found" error.
> Also it would be nice, if airflow always tags commands going from airflow to
> qds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291382#comment-15291382
]
Sumit Maheshwari commented on AIRFLOW-119:
--
Let me explain my point, if its a hive_operator than exts would be .hql/.sql,
if its bash_operator than it would be .sh/.bash, but a qubole operator is
combination of many such operators, so we would need a generic ext. I thought
of {{ .qbl }} initially, but {{ .txt }} seems to be more generic and know to
everyone.
> "Template Not Found" error from QuboleOperator
> --
>
> Key: AIRFLOW-119
> URL: https://issues.apache.org/jira/browse/AIRFLOW-119
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.7.0
>Reporter: Sumit Maheshwari
> Labels: qds
>
> From the given example set, when I am using qbol operator for a hive
> workload, whose script resides in s3 and ends with ".qbl", I am getting
> "Template Not Found" error.
> Also it would be nice, if airflow always tags commands going from airflow to
> qds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290386#comment-15290386 ] Sumit Maheshwari commented on AIRFLOW-119: -- +1 for catching the exception. But it might impact other people who are relied on that exception in their workloads. > "Template Not Found" error from QuboleOperator > -- > > Key: AIRFLOW-119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-119 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.7.0 >Reporter: Sumit Maheshwari > Labels: qds > > From the given example set, when I am using qbol operator for a hive > workload, whose script resides in s3 and ends with ".qbl", I am getting > "Template Not Found" error. > Also it would be nice, if airflow always tags commands going from airflow to > qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289642#comment-15289642 ] Sumit Maheshwari commented on AIRFLOW-119: -- Inline for qubole, not for airflow. Airflow can treat it just like any other param, which can be templatify. So airflow will read the file, templatify it and send the text as inline query to qubole. > "Template Not Found" error from QuboleOperator > -- > > Key: AIRFLOW-119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-119 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.7.0 >Reporter: Sumit Maheshwari > Labels: qds > > From the given example set, when I am using qbol operator for a hive > workload, whose script resides in s3 and ends with ".qbl", I am getting > "Template Not Found" error. > Also it would be nice, if airflow always tags commands going from airflow to > qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-119) "Template Not Found" error from QuboleOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287012#comment-15287012
]
Sumit Maheshwari commented on AIRFLOW-119:
--
Actually there may be a use case where someone would like to templatify
script_location as well, like diff envs (prod, staging etc) having same named
scripts in respective buckets. Also leaving one extension (here .txt) will keep
a way to use file templating. For example I can write a template driven hql in
txt file and use that in {{query}} param.
> "Template Not Found" error from QuboleOperator
> --
>
> Key: AIRFLOW-119
> URL: https://issues.apache.org/jira/browse/AIRFLOW-119
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: Airflow 1.7.0
>Reporter: Sumit Maheshwari
> Labels: qds
>
> From the given example set, when I am using qbol operator for a hive
> workload, whose script resides in s3 and ends with ".qbl", I am getting
> "Template Not Found" error.
> Also it would be nice, if airflow always tags commands going from airflow to
> qds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AIRFLOW-120) "Template Not Found" error from QuboleOperator
Sumit Maheshwari created AIRFLOW-120: Summary: "Template Not Found" error from QuboleOperator Key: AIRFLOW-120 URL: https://issues.apache.org/jira/browse/AIRFLOW-120 Project: Apache Airflow Issue Type: Bug Components: operators Affects Versions: Airflow 1.7.0 Reporter: Sumit Maheshwari >From the given example set, when I am using qbol operator for a hive workload, >whose script resides in s3 and ends with ".qbl", I am getting "Template Not >Found" error. Also it would be nice, if airflow always tags commands going from airflow to qds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
