[jira] [Created] (AIRFLOW-2355) Airflow trigger tag parameters in subdag
Mohammed Tameem created AIRFLOW-2355:
Summary: Airflow trigger tag parameters in subdag
Key: AIRFLOW-2355
URL: https://issues.apache.org/jira/browse/AIRFLOW-2355
Project: Apache Airflow
Issue Type: Bug
Components: api
Affects Versions: 1.9.0
Reporter: Mohammed Tameem
Fix For: 1.9.0
The command airflow {color:#8eb021}+_trigger_dag -c
"\{'name':'value'}"_+{color} sends conf parameters only to the parent DAG. I'm
using SubDags that are dependent on these parameters. And no parameters are
recieved by the SubDag.
>From source code of SubDag operator I see that there is no way of passing
>these trigger parameters to a Subdag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-852) Docs: Cancel a triggered dag
[ https://issues.apache.org/jira/browse/AIRFLOW-852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446689#comment-16446689 ] Mohammed Tameem commented on AIRFLOW-852: - Turn off the DAG. It will stop Airflow from scheduling those tasks. After turning off mark success to downstream tasks. Then turn your DAG on again. > Docs: Cancel a triggered dag > > > Key: AIRFLOW-852 > URL: https://issues.apache.org/jira/browse/AIRFLOW-852 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Andi Pl >Priority: Major > > Issue: Cancel an externally triggered dag. > The dag is running and contains 200 parallel tasks. Only 3 are run at once, > as the celery executor is limited to 3 in parallel. > 23 tasks are finished till now. > We want to cancel the execution of the missing tasks. But they are not queued > yet. How can we cancel them? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-859) airflow trigger_dag not working
[
https://issues.apache.org/jira/browse/AIRFLOW-859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446690#comment-16446690
]
Mohammed Tameem commented on AIRFLOW-859:
-
Make sure you have AIRFLOW_HOME environment variable pointing to your Airflow
directory. Once you confirm, restart scheduler and webserver
> airflow trigger_dag not working
> --
>
> Key: AIRFLOW-859
> URL: https://issues.apache.org/jira/browse/AIRFLOW-859
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 1.7.1.3
> Environment: Using Airflow 1.7.1.3 on Amazon Web Services (date +%Z #
> timezone name
> displays UTC).
> Airflow using Mariadb also on AWS (maria db system_time_zone set to UTC)
>Reporter: Stefano Tiani-Tanzi
>Priority: Major
>
> Hi,
> I have a simple DAG ( code at end of this message ).
> The DAG loads OK using CLI "airflow list_dags".
> In Airflow webpage the DAG is set to "On".
> I try and trigger the dag using CLI "airflow trigger_dag trigger_dag_v1" and
> get feedback as follows:
> [2017-02-10 18:20:12,969] {__init__.py:36} INFO - Using executor LocalExecutor
> [2017-02-10 18:20:13,374] {cli.py:142} INFO - Created @ 2017-02-10 18:20:13.342199: manual__2017-02-10T18:20:13.342199, externally
> triggered: True>
> However, the dag does not start/run.
> I can see in the web interface ( using Browse|Dag Runs ) that the run is at
> "Failed" status but there are no logs associated with the run. Also in the
> Tree view no runs appear.
> Airflow server and db are both set to UTC and the DAG is switched on in the
> web UI.
> Have I missed something or am I doing something incorrectly?
> Any help would be much appreciated.
> Many thanks
> Stefano
> import json
> from airflow import DAG
> from airflow.models import Variable
> from airflow.operators import DummyOperator
> from datetime import datetime, timedelta
> default_args = {
> 'owner': 'stef',
> 'start_date': datetime(2017, 01, 01),
> 'retries': 1,
> 'retry_delay': timedelta(minutes=3),
> 'depends_on_past': False,
> 'wait_for_downstream': True,
> 'provide_context': True
> }
> dag = DAG('trigger_dag_v1', default_args=default_args, schedule_interval=None)
> task_dummy = DummyOperator(
> task_id='dummy',
> dag=dag)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446696#comment-16446696
]
Mohammed Tameem commented on AIRFLOW-1252:
--
Use Json client for post request and pass your conf as below
{"conf": "{'conf':{'name':'value'}}"}
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2353) Separate kerberos config to use per impersonated user
[ https://issues.apache.org/jira/browse/AIRFLOW-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slawomir Krysiak updated AIRFLOW-2353: -- Description: Would be useful to use various Kerberos configs per an impersonated Unix user (via sudo). (was: Would be useful to use various Kerberos configs per an impersonated Unix user (via sudo). Maybe some integrated security config that would allow Airflow to define roles and responsibilities in generic way, with ability to configure various security subsystems?) > Separate kerberos config to use per impersonated user > - > > Key: AIRFLOW-2353 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2353 > Project: Apache Airflow > Issue Type: Improvement > Components: security >Affects Versions: 1.9.0 >Reporter: Slawomir Krysiak >Priority: Major > > Would be useful to use various Kerberos configs per an impersonated Unix user > (via sudo). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2353) Separate kerberos config to use per impersonated user
[ https://issues.apache.org/jira/browse/AIRFLOW-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Slawomir Krysiak updated AIRFLOW-2353: -- Description: Would be useful to be able to use different Kerberos principals per an impersonated Unix user (each user could have different kerberos keytab and other settings etc). Currently Airflow doesn't seem to support it as there is single Kerberos config section in airflow.cfg. was:Would be useful to use various Kerberos configs per an impersonated Unix user (via sudo). > Separate kerberos config to use per impersonated user > - > > Key: AIRFLOW-2353 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2353 > Project: Apache Airflow > Issue Type: Improvement > Components: security >Affects Versions: 1.9.0 >Reporter: Slawomir Krysiak >Priority: Major > > Would be useful to be able to use different Kerberos principals per an > impersonated Unix user (each user could have different kerberos keytab and > other settings etc). > Currently Airflow doesn't seem to support it as there is single Kerberos > config section in airflow.cfg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446712#comment-16446712
]
Sergio Herrera commented on AIRFLOW-1252:
-
That is so redundant. With the PR I did that solves the problem for API Rest
and the cli, the JSON you pass hasn't that redundance
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446713#comment-16446713
]
Mohammed Tameem commented on AIRFLOW-1252:
--
I went through the source code of API trigger and noticed that they were
parsing the json after reading conf from the POST request. That made it obvious
to me that if I wanted to achieve the API trigger with parameters I had to send
the JSON in the format I've shown above. The code then picks up this JSON
string as you have explained above and parses it using json.loads().
For triggering on cli, we can go ahead with other simple json.dumps().
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446720#comment-16446720
]
Sergio Herrera commented on AIRFLOW-1252:
-
Yeah, as you can see in the PR, the solution is apply a json.dumps to the
specific key in the data dict.
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446721#comment-16446721
]
Mohammed Tameem commented on AIRFLOW-1252:
--
Yes. Exactly. Correct me if I'm wrong, I'm trying showing the JSON format that
is actually sent to API in the post request.
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446782#comment-16446782
]
Sergio Herrera commented on AIRFLOW-1252:
-
I usually use a Python Operator where you can show the JSON that reach the DAG
(do not forget to enable the provide context for this operator). Later I will
give you an example of a DAG to analyze that ;)
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1929) TriggerDagRunOperator should allow to set the execution_date
[ https://issues.apache.org/jira/browse/AIRFLOW-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath Kamath reassigned AIRFLOW-1929: Assignee: Sreenath Kamath (was: Bolke de Bruin) > TriggerDagRunOperator should allow to set the execution_date > > > Key: AIRFLOW-1929 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1929 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.9.0, 1.8.2 >Reporter: Bolke de Bruin >Assignee: Sreenath Kamath >Priority: Major > Fix For: 1.9.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1929) TriggerDagRunOperator should allow to set the execution_date
[ https://issues.apache.org/jira/browse/AIRFLOW-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446803#comment-16446803 ] Sreenath Kamath commented on AIRFLOW-1929: -- [~bolke] I have raised a PR for this https://github.com/apache/incubator-airflow/pull/3246 . Can you please take a look when you have some time > TriggerDagRunOperator should allow to set the execution_date > > > Key: AIRFLOW-1929 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1929 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: 1.9.0, 1.8.2 >Reporter: Bolke de Bruin >Assignee: Sreenath Kamath >Priority: Major > Fix For: 1.9.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2355) Airflow trigger tag parameters in subdag
[
https://issues.apache.org/jira/browse/AIRFLOW-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446859#comment-16446859
]
Sreenath Kamath commented on AIRFLOW-2355:
--
One workaround is to have a task in the main DAG that will push all the config
values as an XCOM and then the sub dag tasks consume the config from there.
> Airflow trigger tag parameters in subdag
>
>
> Key: AIRFLOW-2355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2355
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: 1.9.0
>Reporter: Mohammed Tameem
>Priority: Blocker
> Fix For: 1.9.0
>
>
> The command airflow {color:#8eb021}+_trigger_dag -c
> "\{'name':'value'}"_+{color} sends conf parameters only to the parent DAG.
> I'm using SubDags that are dependent on these parameters. And no parameters
> are recieved by the SubDag.
> From source code of SubDag operator I see that there is no way of passing
> these trigger parameters to a Subdag.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2355) Airflow trigger tag parameters in subdag
[
https://issues.apache.org/jira/browse/AIRFLOW-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446862#comment-16446862
]
Mohammed Tameem commented on AIRFLOW-2355:
--
Ok thanks for your inputs
> Airflow trigger tag parameters in subdag
>
>
> Key: AIRFLOW-2355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2355
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: 1.9.0
>Reporter: Mohammed Tameem
>Priority: Blocker
> Fix For: 1.9.0
>
>
> The command airflow {color:#8eb021}+_trigger_dag -c
> "\{'name':'value'}"_+{color} sends conf parameters only to the parent DAG.
> I'm using SubDags that are dependent on these parameters. And no parameters
> are recieved by the SubDag.
> From source code of SubDag operator I see that there is no way of passing
> these trigger parameters to a Subdag.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1252) Experimental API - exception when conf is present in JSON body
[
https://issues.apache.org/jira/browse/AIRFLOW-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446782#comment-16446782
]
Sergio Herrera edited comment on AIRFLOW-1252 at 4/21/18 3:55 PM:
--
I usually use a Python Operator where you can show the JSON that reach the DAG
(do not forget to enable the provide context for this operator).
was (Author: sergiohgz):
I usually use a Python Operator where you can show the JSON that reach the DAG
(do not forget to enable the provide context for this operator). Later I will
give you an example of a DAG to analyze that ;)
> Experimental API - exception when conf is present in JSON body
> --
>
> Key: AIRFLOW-1252
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1252
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: Airflow 1.8, 1.8.1
>Reporter: Sergio Herrera
>Assignee: Sergio Herrera
>Priority: Major
> Labels: api
>
> When someones calls to the endpoint _POST
> :/api/experimental/dags//dag_runs {}_, Airflow never run
> that request if the body of that contains _conf_.
> This occurs due to a mismatch between types when calling function
> _trigger_dag()_, which is also used by *CLI*. That function perform a
> _json.loads(conf)_ because from CLI the type of conf is _string_, but, in the
> other side, from *experimental API*, that type is _dict_ (because _Json_ is
> processed before to get all data, such as execution_date).
> There are two possibilities:
> 1. Look for every use of _trigger_dag()_ function and put _Json_ formatting
> from outside the function.
> 2. In the *experimental API*, put the conf in a string (with _json.dumps()_)
> to allow _trigger_dag()_ transform into _dict_.
> I have implemented the second option, so I can make a PR with that if you
> want.
> Thank you a lot
> EDIT: Also, there are no tests which uses conf in the Json passed through
> request currently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2356) Support queues in dask using worker resources
Christoph Paulik created AIRFLOW-2356: - Summary: Support queues in dask using worker resources Key: AIRFLOW-2356 URL: https://issues.apache.org/jira/browse/AIRFLOW-2356 Project: Apache Airflow Issue Type: Improvement Reporter: Christoph Paulik Dask in Airflow does at the moment not support queues. I suggest to implement this by using [worker resources](https://distributed.readthedocs.io/en/latest/resources.html#example). A queue name would be a necessary worker resource that a dask worker must have to be able to compute a specific task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2356) Support queues in dask using worker resources
[ https://issues.apache.org/jira/browse/AIRFLOW-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446929#comment-16446929 ] Christoph Paulik commented on AIRFLOW-2356: --- I could implement this if the idea is accepted. > Support queues in dask using worker resources > - > > Key: AIRFLOW-2356 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2356 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Christoph Paulik >Priority: Minor > > Dask in Airflow does at the moment not support queues. I suggest to implement > this by using [worker > resources](https://distributed.readthedocs.io/en/latest/resources.html#example). > > A queue name would be a necessary worker resource that a dask worker must > have to be able to compute a specific task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2352) Airflow isn't picking up earlier periods after DAG definition update
[ https://issues.apache.org/jira/browse/AIRFLOW-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anonymous reassigned AIRFLOW-2352: -- Assignee: Alex Lumpov > Airflow isn't picking up earlier periods after DAG definition update > > > Key: AIRFLOW-2352 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2352 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: Slawomir Krysiak >Assignee: Alex Lumpov >Priority: Major > Attachments: Screen Shot 2018-04-20 at 5.04.12 PM.png > > > Hi, > > It would be nice to be able to modify the period range (a.k.a start_date) per > dag/subdag and have scheduler pick it up. Not sure if that should be a > feature request or a bug, but I was under the assumption that it works that > way already. But for some reason it doesn't seem to be the case in 1.9.0 > which I'm using for my POC. Attaching my message from gitter... BTW, it seems > that there's so many questions coming up on that channel but they don't seem > to be addressed promptly. > Thanks, > Slawomir > > P.S. It would probably be helpful to be able to submit an 'end_date' > parameter to DAG/SubDAG... there may be datasets that are no longer produced, > yet they still have some period range extracted. Evolving transformation > pipelines would definitely benefit from this kind of option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AIRFLOW-2222) GoogleCloudStorageHook.copy fails for large files between locations
[
https://issues.apache.org/jira/browse/AIRFLOW-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Berislav Lopac reopened AIRFLOW-:
-
This specific issue was not resolved by PR 3232; {{rewrite}} was not
implemented there.
> GoogleCloudStorageHook.copy fails for large files between locations
> ---
>
> Key: AIRFLOW-
> URL: https://issues.apache.org/jira/browse/AIRFLOW-
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Berislav Lopac
>Assignee: Berislav Lopac
>Priority: Major
> Fix For: 2.0.0
>
>
> When copying large files (confirmed for around 3GB) between buckets in
> different projects, the operation fails and the Google API returns error
> [413—Payload Too
> Large|https://cloud.google.com/storage/docs/json_api/v1/status-codes#413_Payload_Too_Large].
> The documentation for the error says:
> {quote}The Cloud Storage JSON API supports up to 5 TB objects.
> This error may, alternatively, arise if copying objects between locations
> and/or storage classes can not complete within 30 seconds. In this case, use
> the
> [Rewrite|https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite]
> method instead.{quote}
> The reason seems to be that the {{GoogleCloudStorageHook.copy}} is using the
> API {{copy}} method.
> h3. Proposed Solution
> There are two potential solutions:
> # Implement {{GoogleCloudStorageHook.rewrite}} method which can be called
> from operators and other objects to ensure successful execution. This method
> is more flexible but requires changes both in the {{GoogleCloudStorageHook}}
> class and any other classes that use it for copying files to ensure that they
> explicitly call {{rewrite}} when needed.
> # Modify {{GoogleCloudStorageHook.copy}} to determine when to use {{rewrite}}
> instead of {{copy}} underneath. This requires updating only the
> {{GoogleCloudStorageHook}} class, but the logic might not cover all the edge
> cases and could be difficult to implement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-1314) Airflow kubernetes integration
[ https://issues.apache.org/jira/browse/AIRFLOW-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-1314: -- Affects Version/s: (was: Airflow 2.0) 1.10.0 Fix Version/s: (was: Airflow 2.0) 1.10.0 > Airflow kubernetes integration > -- > > Key: AIRFLOW-1314 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1314 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Affects Versions: 1.10.0 >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Major > Labels: features > Fix For: 1.10.0 > > > Kubernetes is a container-based cluster management system designed by google > for easy application deployment. Companies such as Airbnb, Bloomberg, > Palantir, and Google use kubernetes for a variety of large-scale solutions > including data science, ETL, and app deployment. Integrating airflow into > Kubernetes would increase viable use cases for airflow, promote airflow as a > de facto workflow scheduler for Kubernetes, and create possibilities for > improved security and robustness within airflow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
