[jira] [Comment Edited] (AIRFLOW-4878) Allow alternate docker image for kubernetes xcom sidecar
[
https://issues.apache.org/jira/browse/AIRFLOW-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236617#comment-17236617
]
Kamil Bregula edited comment on AIRFLOW-4878 at 11/21/20, 9:07 AM:
---
we can set your own image for sidecar container with airflow_local_settings.py.
More information:
https://hackmd.io/bhMoDP-zQgOKzBFr4MlB8g
was (Author: kamil.bregula):
You can set your own image with airflow_local_settings.py. More information:
https://hackmd.io/bhMoDP-zQgOKzBFr4MlB8g
> Allow alternate docker image for kubernetes xcom sidecar
>
>
> Key: AIRFLOW-4878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4878
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators, xcom
>Affects Versions: 2.0.0
>Reporter: Eric Carlson
>Priority: Minor
>
> Currently the image used for xcom push is hardcoded to `python:3.5-alpine`:
> -
> [https://github.com/apache/airflow/blob/master/airflow/kubernetes/kubernetes_request_factory/pod_request_factory.py#L98]
> This is a problem in environments that are locked down and do not have access
> to the public docker registry.
> I propose adding a similar set of config variables and method arguments that
> specify the image and pull policy of the base container. Specifically:
> * in the config file, adding \{xcom_container_repository,
> xcom_container_tag, xcom_container_image_pull_policy} to complement the
> existing \{worker_container_*} variables
> * In the pod operator, adding a xcom_image_* arguments to complement the
> existing image_* arguments (image, image_pull_policy)
> If this proposal is acceptable I have some locally working code that I'll
> create a PR from
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4878) Allow alternate docker image for kubernetes xcom sidecar
[
https://issues.apache.org/jira/browse/AIRFLOW-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236617#comment-17236617
]
Kamil Bregula commented on AIRFLOW-4878:
You can set your own image with airflow_local_settings.py. More information:
https://hackmd.io/bhMoDP-zQgOKzBFr4MlB8g
> Allow alternate docker image for kubernetes xcom sidecar
>
>
> Key: AIRFLOW-4878
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4878
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators, xcom
>Affects Versions: 2.0.0
>Reporter: Eric Carlson
>Priority: Minor
>
> Currently the image used for xcom push is hardcoded to `python:3.5-alpine`:
> -
> [https://github.com/apache/airflow/blob/master/airflow/kubernetes/kubernetes_request_factory/pod_request_factory.py#L98]
> This is a problem in environments that are locked down and do not have access
> to the public docker registry.
> I propose adding a similar set of config variables and method arguments that
> specify the image and pull policy of the base container. Specifically:
> * in the config file, adding \{xcom_container_repository,
> xcom_container_tag, xcom_container_image_pull_policy} to complement the
> existing \{worker_container_*} variables
> * In the pod operator, adding a xcom_image_* arguments to complement the
> existing image_* arguments (image, image_pull_policy)
> If this proposal is acceptable I have some locally working code that I'll
> create a PR from
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235970#comment-17235970 ] Kamil Bregula commented on AIRFLOW-2221: No, and it is not planned for any future versions. > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2075) Extract and create a data access layer
[ https://issues.apache.org/jira/browse/AIRFLOW-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222910#comment-17222910 ] Kamil Bregula commented on AIRFLOW-2075: [~richard.williamson] Hello, We no longer use Jira for new tickets. Please, use Github Issue. https://github.com/apache/airflow/issues Jira is just an archive for old tickets. You can no longer create new tickets. You can still comment on old tickets, but you can't expect a response. If the ticket is important to you, please create a new ticket in Github Issue and give the link to the Jira ticket. Big love, See: https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-7120?filter=allopenissues > Extract and create a data access layer > -- > > Key: AIRFLOW-2075 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2075 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Yang Pan >Priority: Major > > Currently Airflow is tightly coupled with sqlalchemy. > [https://github.com/search?p=2&q=%22from+sqlalchemy%22+repo%3Aapache%2Fincubator-airflow+path%3A%2Fairflow&type=Code&utf8=%E2%9C%93] > While it's convenient for sqlalchemy compatible database, it makes non > compatible ones, such as google cloud spanner, harder to integrate. > It would be nice if the data access layer can be decoupled from the airflow > core, and developers have option to use sqlalchemy or swap to talk to db > directly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4481) DummyOperator performance improvement - currently needs +87 seconds to do nothing
[
https://issues.apache.org/jira/browse/AIRFLOW-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164391#comment-17164391
]
Kamil Bregula commented on AIRFLOW-4481:
In Airflow 1.10.10, dummy operators is evaluated by the scheduler but never
processed by the executor.
https://github.com/apache/airflow/pull/7880/files
> DummyOperator performance improvement - currently needs +87 seconds to do
> nothing
> -
>
> Key: AIRFLOW-4481
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4481
> Project: Apache Airflow
> Issue Type: Improvement
> Components: DAG, operators
>Affects Versions: 1.10.3
>Reporter: t oo
>Priority: Trivial
>
> I don't know why a DummyOperator can't finish in sub-second time, currently
> taking >4 seconds.
> *** Reading local file:
> /home/ec2-user/airflow/logs/stream_run/JOIN_tsk_/2019-01-01T00:00:00+00:00/1.log
> [2019-05-08 15:33:53,982] {__init__.py:1139} INFO - Dependencies all met for
>
> [2019-05-08 15:33:54,009] {__init__.py:1139} INFO - Dependencies all met for
>
> [2019-05-08 15:33:54,009] {__init__.py:1353} INFO -
>
> [2019-05-08 15:33:54,009] {__init__.py:1354} INFO - Starting attempt 1 of 1
> [2019-05-08 15:33:54,009] {__init__.py:1355} INFO -
>
> [2019-05-08 15:33:54,026] {__init__.py:1374} INFO - Executing
> on 2019-01-01T00:00:00+00:00
> [2019-05-08 15:33:54,026] {base_task_runner.py:119} INFO - Running:
> [u'airflow', u'run', 'stream_run', 'JOIN_tsk_', '2019-01-01T00:00:00+00:00',
> u'--job_id', '2550', u'--raw', u'-sd',
> u'DAGS_FOLDER/stream_dag_generation.py', u'--cfg_path', '/tmp/tmpJR_rQX']
> [2019-05-08 15:33:56,117] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,117] {settings.py:182} INFO -
> settings.configure_orm(): Using pool settings. pool_size=5,
> pool_recycle=1800, pid=16385
> [2019-05-08 15:33:56,317] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,317] {__init__.py:51} INFO - Using executor
> LocalExecutor
> [2019-05-08 15:33:56,582] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:56,581] {__init__.py:305} INFO - Filling up the
> DagBag from /home/ec2-user/airflow/dags/stream_dag_generation.py
> [2019-05-08 15:33:58,002] {base_task_runner.py:101} INFO - Job 2550: Subtask
> JOIN_tsk_ [2019-05-08 15:33:58,001] {cli.py:517} INFO - Running
> on
> host domain
> [2019-05-08 15:33:58,279] {logging_mixin.py:95} INFO - [2019-05-08
> 15:33:58,279] {jobs.py:2562} INFO - Task exited with return code 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-822) Close the connection before throwing exception in BaseHook
[ https://issues.apache.org/jira/browse/AIRFLOW-822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-822: - Assignee: Fokko Driesprong > Close the connection before throwing exception in BaseHook > -- > > Key: AIRFLOW-822 > URL: https://issues.apache.org/jira/browse/AIRFLOW-822 > Project: Apache Airflow > Issue Type: Bug > Components: hooks >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.9.0 > > > Hi Guys, > The basehook contains functionality to retrieve connections from the > database. If a connection doesn't exists it will throw an exception. This > exception will be thrown before the connection to the database is closed. > Therefore the session to the db might stay open. > Cheers, Fokko -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4425) Add FacebookAdsHook
[ https://issues.apache.org/jira/browse/AIRFLOW-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-4425. -- Resolution: Fixed > Add FacebookAdsHook > --- > > Key: AIRFLOW-4425 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4425 > Project: Apache Airflow > Issue Type: Wish > Components: hooks >Reporter: jack >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Add hook to interact with FacebookAds -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6463) Mock Cassandra in tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151437#comment-17151437
]
Kamil Bregula commented on AIRFLOW-6463:
Hi.
Hi. I'm afraid this ticket is no longer valid. We have made changes to the
infrastructure and Cassandra is no longer a problem for us.
Best regards,
> Mock Cassandra in tests
> ---
>
> Key: AIRFLOW-6463
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6463
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
>Affects Versions: 1.10.7
>Reporter: Kamil Bregula
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> Cassandra consume 1.173GiB of memory. Travis does not have very efficient
> machines, so we should limit system/integration tests of components that do
> not require much attention, e.g. they are not changed often. Cassandra is a
> good candidate for this. This will allow the machine power to be used for
> more needed work.
> {code:java}
> CONTAINER IDNAME CPU %
> MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
> PIDS
> 8aa37ca50f7cci_airflow-testing_run_1f3aeb6d1052 0.00%
> 5.715MiB / 3.855GiB 0.14% 1.14kB / 0B 2.36MB / 0B
> 2
> f2b3be15558fci_cassandra_10.69%
> 1.173GiB / 3.855GiB 30.42% 2.39kB / 0B 75.3MB / 9.95MB
> 50
> ef1de3981ca6ci_krb5-kdc-server_1 0.02%
> 12.15MiB / 3.855GiB 0.31% 2.46kB / 0B 18.9MB / 184kB
> 4
> be808233eb91ci_mongo_10.31%
> 36.71MiB / 3.855GiB 0.93% 2.39kB / 0B 43.2MB / 19.1MB
> 24
> 667e047be097ci_rabbitmq_1 0.77%
> 69.95MiB / 3.855GiB 1.77% 2.39kB / 0B 43.2MB / 508kB
> 92
> 2453dd6e7ccaci_postgres_1 0.00%
> 7.547MiB / 3.855GiB 0.19% 1.05MB / 889kB 35.4MB / 145MB
> 6
> 78050c5c61ccci_redis_10.29%
> 1.695MiB / 3.855GiB 0.04% 2.46kB / 0B 6.94MB / 0B
> 4
> c117eb0a0d43ci_mysql_10.13%
> 452MiB / 3.855GiB 11.45% 2.21kB / 0B 33.9MB / 548MB
> 21
> 131427b19282ci_openldap_1 0.00%
> 45.68MiB / 3.855GiB 1.16% 2.64kB / 0B 32.8MB / 16.1MB
> 4
> 8c2549c010b1ci_docker_1 0.59%
> 22.06MiB / 3.855GiB 0.56% 2.39kB / 0B 95.9MB / 291kB
> 30
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6463) Mock Cassandra in tests
[
https://issues.apache.org/jira/browse/AIRFLOW-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151437#comment-17151437
]
Kamil Bregula edited comment on AIRFLOW-6463 at 7/4/20, 10:21 PM:
--
Hi.
I'm afraid this ticket is no longer valid. We have made changes to the
infrastructure and Cassandra is no longer a problem for us.
Best regards,
was (Author: kamil.bregula):
Hi.
Hi. I'm afraid this ticket is no longer valid. We have made changes to the
infrastructure and Cassandra is no longer a problem for us.
Best regards,
> Mock Cassandra in tests
> ---
>
> Key: AIRFLOW-6463
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6463
> Project: Apache Airflow
> Issue Type: Improvement
> Components: ci
>Affects Versions: 1.10.7
>Reporter: Kamil Bregula
>Priority: Major
> Labels: gsoc, gsoc2020, mentor
>
> Cassandra consume 1.173GiB of memory. Travis does not have very efficient
> machines, so we should limit system/integration tests of components that do
> not require much attention, e.g. they are not changed often. Cassandra is a
> good candidate for this. This will allow the machine power to be used for
> more needed work.
> {code:java}
> CONTAINER IDNAME CPU %
> MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O
> PIDS
> 8aa37ca50f7cci_airflow-testing_run_1f3aeb6d1052 0.00%
> 5.715MiB / 3.855GiB 0.14% 1.14kB / 0B 2.36MB / 0B
> 2
> f2b3be15558fci_cassandra_10.69%
> 1.173GiB / 3.855GiB 30.42% 2.39kB / 0B 75.3MB / 9.95MB
> 50
> ef1de3981ca6ci_krb5-kdc-server_1 0.02%
> 12.15MiB / 3.855GiB 0.31% 2.46kB / 0B 18.9MB / 184kB
> 4
> be808233eb91ci_mongo_10.31%
> 36.71MiB / 3.855GiB 0.93% 2.39kB / 0B 43.2MB / 19.1MB
> 24
> 667e047be097ci_rabbitmq_1 0.77%
> 69.95MiB / 3.855GiB 1.77% 2.39kB / 0B 43.2MB / 508kB
> 92
> 2453dd6e7ccaci_postgres_1 0.00%
> 7.547MiB / 3.855GiB 0.19% 1.05MB / 889kB 35.4MB / 145MB
> 6
> 78050c5c61ccci_redis_10.29%
> 1.695MiB / 3.855GiB 0.04% 2.46kB / 0B 6.94MB / 0B
> 4
> c117eb0a0d43ci_mysql_10.13%
> 452MiB / 3.855GiB 11.45% 2.21kB / 0B 33.9MB / 548MB
> 21
> 131427b19282ci_openldap_1 0.00%
> 45.68MiB / 3.855GiB 1.16% 2.64kB / 0B 32.8MB / 16.1MB
> 4
> 8c2549c010b1ci_docker_1 0.59%
> 22.06MiB / 3.855GiB 0.56% 2.39kB / 0B 95.9MB / 291kB
> 30
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6648) Timeout Feature - Provided statistical solution to long running/stuck jobs and take appropriate actions
[ https://issues.apache.org/jira/browse/AIRFLOW-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139294#comment-17139294 ] Kamil Bregula commented on AIRFLOW-6648: Only people with write access to the repository must have a CLA Most contributors do not need CLA > Timeout Feature - Provided statistical solution to long running/stuck jobs > and take appropriate actions > --- > > Key: AIRFLOW-6648 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6648 > Project: Apache Airflow > Issue Type: Improvement > Components: aws, DAG, database, operators >Affects Versions: 1.10.0 > Environment: AWS Linux AMI - Ubuntu 18.04.1 LTS (GNU/Linux > 4.15.0-1027-aws x86_64) >Reporter: Golokesh Patra >Assignee: Golokesh Patra >Priority: Minor > Attachments: image-2020-01-27-17-07-51-822.png, > image-2020-01-27-17-08-09-867.png, image-2020-01-27-17-08-33-088.png, > image-2020-01-27-17-22-07-433.png, image2019-3-25_12-33-57.png > > Original Estimate: 336h > Remaining Estimate: 336h > > Sometimes, across different type of tasks/jobs , > one might encounter issues where airflow jobs/tasks get stuck while they are > in running state. > Such issues will cause - Pipeline being stuck for no reason stalling other > jobs/tasks which will be a disaster when such issues happen on Production. > This particular improvement aims to not only improve upon the TIMEOUT logic > already in airflow, but to make it more functional and automated. > *Diagrammatically Explanation of the solution -* > !image-2020-01-27-17-22-07-433.png! > *Detailed Theoretical Explanation -* > With increasing Data & Complexity of tasks/job , besides the increasing load, > the chances of memory leaks/stuck jobs/some infrastructural issues etc may > occur thereby creating some unwanted results. > Maybe on some day there was more data which resulted in a steep jump in the > duration of the job; otherwise, the growth is expected to be gradual. > And sometimes, the Jobs get stuck because of various issues and often > requires termination followed by a restart. > So, we are trying to make a logic which will automatically decide whether to > * _terminate the Job_ > * _Terminate and Restart_ > * _Terminate and Mark as a failure so that downstream jobs don't get > triggered._ > * _Take no action and inform DevOps regarding the issue ( Manual Action )_ > So, I just want to know, statistically, what will be the effective way to > achieve the above outcomes. > Lets Consider 2 Jobs X & Y. > Jobs related Info - > !image-2020-01-27-17-07-51-822.png! > !image-2020-01-27-17-08-09-867.png! > Then I was thinking of having a New Table which would be structured as - > +Derived table-+ > !image-2020-01-27-17-08-33-088.png! > ( The above Example is theoretical and actual implementation might differ ) > *LIMITATION -* > # For now , we have only tested the above on EMR ( Personal Usecase ) > # Testing Pending for Databricks. ( Personal Usecase ) > Please do suggest any other services where this needs/can be used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6458) Deprecation warning about dropping support for Python 2
[
https://issues.apache.org/jira/browse/AIRFLOW-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094461#comment-17094461
]
Kamil Bregula commented on AIRFLOW-6458:
Go ahead and do it.
> Deprecation warning about dropping support for Python 2
> ---
>
> Key: AIRFLOW-6458
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6458
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>
> We should add a deprecation warning in Airflow 1.10.x to inform users that
> Python 3 is required in the Airflow 2.0.
> We should add something similar to {{airflow/__init__.py}} file
> {code:python}
> if sys.version_info[0] < 3:
> message = (
> "A future version of this application will drop support for Python
> 2.7."
> "More details about Python 2 support for Airflow can be found at "
> "https://cloud.google.com/python/docs/python2-sunset/";
> )
> warnings.warn(message, DeprecationWarning)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6458) Deprecation warning about dropping support for Python 2
[
https://issues.apache.org/jira/browse/AIRFLOW-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula reassigned AIRFLOW-6458:
--
Assignee: Will Hudgins
> Deprecation warning about dropping support for Python 2
> ---
>
> Key: AIRFLOW-6458
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6458
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Assignee: Will Hudgins
>Priority: Major
>
> We should add a deprecation warning in Airflow 1.10.x to inform users that
> Python 3 is required in the Airflow 2.0.
> We should add something similar to {{airflow/__init__.py}} file
> {code:python}
> if sys.version_info[0] < 3:
> message = (
> "A future version of this application will drop support for Python
> 2.7."
> "More details about Python 2 support for Airflow can be found at "
> "https://cloud.google.com/python/docs/python2-sunset/";
> )
> warnings.warn(message, DeprecationWarning)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6242) Introduce run_type in DAG Run table
[ https://issues.apache.org/jira/browse/AIRFLOW-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094460#comment-17094460 ] Kamil Bregula commented on AIRFLOW-6242: This is not a change in the architecture of the project. It also doesn't change any public interface. This does not require AIP. We have PR with this change: https://github.com/apache/airflow/pull/8227 > Introduce run_type in DAG Run table > --- > > Key: AIRFLOW-6242 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6242 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Hello, > Before you start reading this ticket, I want to tell you one thing - you are > wonderful. > We use queries that use like expressions to filter two types of DAG Run. > * Backfill: > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/jobs/backfill_job.py#L52] > * Scheduled > > [https://github.com/apache/airflow/blob/699aea8ee368abcba29d717daf2580f897ab9d93/airflow/models/dagrun.py#L46] > This type of expression can affect performance and it seems a better idea to > store this information in a separate column. This may allow a query planner > to create a better query plan in the database engine. > Thanks for reading this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7120) WE HAVE MOVED TO GITHUB ISSUE
[ https://issues.apache.org/jira/browse/AIRFLOW-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-7120: --- Priority: Blocker (was: Critical) > WE HAVE MOVED TO GITHUB ISSUE > - > > Key: AIRFLOW-7120 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7120 > Project: Apache Airflow > Issue Type: Bug > Components: api, authentication, aws, backfill, backport-packages, > breeze, build, celery, ci, cli, configuration, contrib, core, DAG, DagRun, > database, dependencies, documentation, examples, executor-kubernetes, > executors, gcp, hooks, lineage, logging, models, mysql, operators, packages, > plugins, pre-commit, project-management, pylint, scheduler, security, > serialization, tests, ui, utils, webserver, worker, xcom >Affects Versions: 1.10.10 >Reporter: Kamil Bregula >Assignee: Daniel Imberman >Priority: Blocker > > Hello, > We no longer use Jira for new tickets. Please, use Github Issue. > https://github.com/apache/airflow/issues > Jira is just an archive for old tickets. You can no longer create new > tickets. You can still comment on old tickets, but you can't expect a > response. If the ticket is important to you, please create a new ticket in > Github Issue and give the link to the Jira ticket. > Big love, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7120) WE HAVE MOVED TO GITHUB ISSUE
[ https://issues.apache.org/jira/browse/AIRFLOW-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-7120: --- Priority: Critical (was: Major) > WE HAVE MOVED TO GITHUB ISSUE > - > > Key: AIRFLOW-7120 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7120 > Project: Apache Airflow > Issue Type: Bug > Components: api, authentication, aws, backfill, backport-packages, > breeze, build, celery, ci, cli, configuration, contrib, core, DAG, DagRun, > database, dependencies, documentation, examples, executor-kubernetes, > executors, gcp, hooks, lineage, logging, models, mysql, operators, packages, > plugins, pre-commit, project-management, pylint, scheduler, security, > serialization, tests, ui, utils, webserver, worker, xcom >Affects Versions: 1.10.10 >Reporter: Kamil Bregula >Assignee: Daniel Imberman >Priority: Critical > > Hello, > We no longer use Jira for new tickets. Please, use Github Issue. > https://github.com/apache/airflow/issues > Jira is just an archive for old tickets. You can no longer create new > tickets. You can still comment on old tickets, but you can't expect a > response. If the ticket is important to you, please create a new ticket in > Github Issue and give the link to the Jira ticket. > Big love, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7120) WE HAVE MOVED TO GITHUB ISSUE
Kamil Bregula created AIRFLOW-7120: -- Summary: WE HAVE MOVED TO GITHUB ISSUE Key: AIRFLOW-7120 URL: https://issues.apache.org/jira/browse/AIRFLOW-7120 Project: Apache Airflow Issue Type: Bug Components: api, authentication, aws, backfill, backport-packages, breeze, build, celery, ci, cli, configuration, contrib, core, DAG, DagRun, database, dependencies, documentation, examples, executor-kubernetes, executors, gcp, hooks, lineage, logging, models, mysql, operators, packages, plugins, pre-commit, project-management, pylint, scheduler, security, serialization, tests, ui, utils, webserver, worker, xcom Affects Versions: 1.10.10 Reporter: Kamil Bregula Assignee: Daniel Imberman Hello, We no longer use Jira for new tickets. Please, use Github Issue. https://github.com/apache/airflow/issues Jira is just an archive for old tickets. You can no longer create new tickets. You can still comment on old tickets, but you can't expect a response. If the ticket is important to you, please create a new ticket in Github Issue and give the link to the Jira ticket. Big love, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-2153) Dataflow Operator doesn't allow multiple extra_packages
[
https://issues.apache.org/jira/browse/AIRFLOW-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula resolved AIRFLOW-2153.
Fix Version/s: 2.0.0
Resolution: Fixed
> Dataflow Operator doesn't allow multiple extra_packages
> ---
>
> Key: AIRFLOW-2153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2153
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.9.0
>Reporter: Maximilian Roos
>Priority: Minor
> Fix For: 2.0.0
>
>
> If we have multiple `extra_packages` to pass to DataFlow - for example we'd
> want to pass:
> {{python run_dataflow.py --extra-package=a.whl --extra-package=b.whl}}
>
> ...I don't think this currently isn't possible. Because the args are passed
> as dictionary, you can't have multiple identical keys.
>
> A hacky workaround is to append a key containing everything you need, with an
> empty value:
>
> {{arg_dict['--extra-package=a.whl --extra-package=b.whl'] = ' '}}
>
> I think ideally we would be able to pass a list of 2-tuples as well as a
> dict, or pass a list as a dict value which would then be repeated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2153) Dataflow Operator doesn't allow multiple extra_packages
[
https://issues.apache.org/jira/browse/AIRFLOW-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17087024#comment-17087024
]
Kamil Bregula commented on AIRFLOW-2153:
I prepared fix: https://github.com/apache/airflow/pull/8394
> Dataflow Operator doesn't allow multiple extra_packages
> ---
>
> Key: AIRFLOW-2153
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2153
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
>Affects Versions: 1.9.0
>Reporter: Maximilian Roos
>Priority: Minor
>
> If we have multiple `extra_packages` to pass to DataFlow - for example we'd
> want to pass:
> {{python run_dataflow.py --extra-package=a.whl --extra-package=b.whl}}
>
> ...I don't think this currently isn't possible. Because the args are passed
> as dictionary, you can't have multiple identical keys.
>
> A hacky workaround is to append a key containing everything you need, with an
> empty value:
>
> {{arg_dict['--extra-package=a.whl --extra-package=b.whl'] = ' '}}
>
> I think ideally we would be able to pass a list of 2-tuples as well as a
> dict, or pass a list as a dict value which would then be repeated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-603) db.py pessimistic_connection_handling not compatible with Oracle
[
https://issues.apache.org/jira/browse/AIRFLOW-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083114#comment-17083114
]
Kamil Bregula commented on AIRFLOW-603:
---
In Oracle, it should be `SELECT 1 FROM dual;`
> db.py pessimistic_connection_handling not compatible with Oracle
>
>
> Key: AIRFLOW-603
> URL: https://issues.apache.org/jira/browse/AIRFLOW-603
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
>Affects Versions: 1.7.1.3
> Environment: Debian Jessie, Python 2.7
>Reporter: Vincent BENOIT
>Priority: Major
> Labels: db, oracle
>
> There is a problem with airflow's implementation of sqlalchemy.
> Current **pessimistic_connection_handling()** code in /utils/db.py is not
> compatible with Oracle ("SELECT 1" is non functional in Oracle).
> According to sqlalchemy documentation,
> [http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic]
> , the current code in Airflow is the older approach.
> I updated my Airflow source code with the following, recommended by
> sqlalchemy, and it seems fully functional.
> {code:title=db.py|borderStyle=solid}
> def pessimistic_connection_handling():
> # @event.listens_for(Pool, "checkout")
> # def ping_connection(dbapi_connection, connection_record,
> connection_proxy):
> # '''
> # Disconnect Handling - Pessimistic, taken from:
> # http://docs.sqlalchemy.org/en/rel_0_9/core/pooling.html
> # '''
> # cursor = dbapi_connection.cursor()
> # try:
> # cursor.execute("SELECT 1")
> # except:
> # raise exc.DisconnectionError()
> # cursor.close()
> from sqlalchemy import select
> @event.listens_for(settings.engine, "engine_connect")
> def ping_connection(connection, branch):
> if branch:
> # "branch" refers to a sub-connection of a connection,
> # we don't want to bother pinging on these.
> return
> # turn off "close with result". This flag is only used with
> # "connectionless" execution, otherwise will be False in any
> case
> save_should_close_with_result =
> connection.should_close_with_result
> connection.should_close_with_result = False
> try:
> # run a SELECT 1. use a core select() so that
> # the SELECT of a scalar value without a table is
> # appropriately formatted for the backend
> connection.scalar(select([1]))
> except exc.DBAPIError as err:
> # catch SQLAlchemy's DBAPIError, which is a wrapper
> # for the DBAPI's exception. It includes a
> .connection_invalidated
> # attribute which specifies if this connection is a
> "disconnect"
> # condition, which is based on inspection of the
> original exception
> # by the dialect in use.
> if err.connection_invalidated:
> # run the same SELECT again - the connection
> will re-validate
> # itself and establish a new connection. The
> disconnect detection
> # here also causes the whole connection pool to
> be invalidated
> # so that all stale connections are discarded.
> connection.scalar(select([1]))
> else:
> raise
> finally:
> # restore "close with result"
> connection.should_close_with_result =
> save_should_close_with_result
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5308) Pass credentials object to pandas_gbq
[ https://issues.apache.org/jira/browse/AIRFLOW-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5308. Fix Version/s: 2.0.0 Resolution: Fixed > Pass credentials object to pandas_gbq > - > > Key: AIRFLOW-5308 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5308 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5308) Pass credentials object to pandas_gbq
[ https://issues.apache.org/jira/browse/AIRFLOW-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reopened AIRFLOW-5308: > Pass credentials object to pandas_gbq > - > > Key: AIRFLOW-5308 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5308 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6294) Create guide for Dataflow service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6294. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Dataflow service operator > -- > > Key: AIRFLOW-6294 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6294 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Dataflow service perators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6292) Create guide for Datastore service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6292. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Datastore service operator > --- > > Key: AIRFLOW-6292 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6292 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Datastore service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6290) Create guide for GKE service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6290. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for GKE service operator > - > > Key: AIRFLOW-6290 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6290 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GKE service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6289) Create guide for ML Engine service operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6289. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for ML Engine service operators > > > Key: AIRFLOW-6289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6289 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use ML Engine service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6291) Create guide for DLP service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6291. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for DLP service operator > - > > Key: AIRFLOW-6291 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6291 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use DLP service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6288) Create guide for Cloud Tasks service operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6288. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Cloud Tasks service operators > -- > > Key: AIRFLOW-6288 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6288 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Cloud Tasks service operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6293) Create guide for Dataproc service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6293. Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Dataproc service operator > -- > > Key: AIRFLOW-6293 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6293 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Dataproc service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6295) Create guide for BigQuery service operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6295. -- Resolution: Fixed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ [~alexaabbas] Can you write a comment under the new ticket and will I assign you to this task? > Create guide for BigQuery service operator > -- > > Key: AIRFLOW-6295 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6295 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Assignee: Alexandra Abbas >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use BigQuery service operators would be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6298) Create guide for Apache Cassandra software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6298. -- Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Cassandra software operator > --- > > Key: AIRFLOW-6298 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6298 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Cassandra software operators would > be useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6299) Create guide for Apache Druid software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6299. -- Resolution: Auto Closed I created this ticket on Github: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Druid software operator > --- > > Key: AIRFLOW-6299 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6299 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Druid software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6304) Create guide for Apache Sqoop software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6304. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Sqoop software operator > --- > > Key: AIRFLOW-6304 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6304 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Sqoop software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6300) Create guide for Apache Hive software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6300. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Hive software operator > -- > > Key: AIRFLOW-6300 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6300 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Hive software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6301) Create guide for Apache Pig software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6301. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Pig software operator > - > > Key: AIRFLOW-6301 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6301 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Pig software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6302) Create guide for Apache Pinot software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6302. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Pinot software operator > --- > > Key: AIRFLOW-6302 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6302 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Pinot software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6303) Create guide for Apache Spark software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6303. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache Spark software operator > --- > > Key: AIRFLOW-6303 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6303 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache Spark software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6305) Create guide for Apache HDFS software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6305. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache HDFS software operator > -- > > Key: AIRFLOW-6305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6305 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache HDFS software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (AIRFLOW-6275) Create guide for SQL to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-6275: --- Comment: was deleted (was: I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+) > Create guide for SQL to GCS transfer operator > -- > > Key: AIRFLOW-6275 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6275 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use SQL to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6275) Create guide for SQL to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1709#comment-1709 ] Kamil Bregula commented on AIRFLOW-6275: I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for SQL to GCS transfer operator > -- > > Key: AIRFLOW-6275 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6275 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use SQL to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6306) Create guide for Apache WebHDFS software operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6306. -- Resolution: Fixed I created this ticket on Github also: https://github.com/apache/airflow/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue+label%3Aarea%3Adocs+ > Create guide for Apache WebHDFS software operator > - > > Key: AIRFLOW-6306 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6306 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use Apache WebHDFS software operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/index.html > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4060) Enforce import order
[ https://issues.apache.org/jira/browse/AIRFLOW-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-4060. -- Resolution: Done > Enforce import order > > > Key: AIRFLOW-4060 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4060 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5305) Sort extra links by name
[ https://issues.apache.org/jira/browse/AIRFLOW-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5305. Fix Version/s: (was: 2.0.0) 1.10.11 Resolution: Fixed > Sort extra links by name > > > Key: AIRFLOW-5305 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5305 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.4 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 1.10.11 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5706) Add KubernetesPodYamlOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5706. -- Resolution: Abandoned > Add KubernetesPodYamlOperator > - > > Key: AIRFLOW-5706 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5706 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Affects Versions: 1.10.5 >Reporter: Kamil Bregula >Priority: Major > > I would like to create an operator that allows us to run pods using the yaml > file definition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6307) Improve support for a special characters in DAG's name in stats view
[ https://issues.apache.org/jira/browse/AIRFLOW-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6307. -- Resolution: Won't Fix > Improve support for a special characters in DAG's name in stats view > > > Key: AIRFLOW-6307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6307 > Project: Apache Airflow > Issue Type: New Feature > Components: webserver >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6659) Move AWS Transfer operators to providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6659. Fix Version/s: 2.0.0 Resolution: Fixed > Move AWS Transfer operators to providers package > > > Key: AIRFLOW-6659 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6659 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-6307) Improve support for a special characters in DAG's name in stats view
[ https://issues.apache.org/jira/browse/AIRFLOW-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reopened AIRFLOW-6307: > Improve support for a special characters in DAG's name in stats view > > > Key: AIRFLOW-6307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6307 > Project: Apache Airflow > Issue Type: New Feature > Components: webserver >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6658) Remove contrib/{hooks,sensors} directory
[
https://issues.apache.org/jira/browse/AIRFLOW-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula resolved AIRFLOW-6658.
Fix Version/s: 2.0.0
Resolution: Fixed
> Remove contrib/{hooks,sensors} directory
>
>
> Key: AIRFLOW-6658
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6658
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks, operators
>Affects Versions: 1.10.7
>Reporter: Kamil Bregula
>Priority: Major
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (AIRFLOW-6307) Improve support for a special characters in DAG's name in stats view
[ https://issues.apache.org/jira/browse/AIRFLOW-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6307. -- Resolution: Fixed > Improve support for a special characters in DAG's name in stats view > > > Key: AIRFLOW-6307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6307 > Project: Apache Airflow > Issue Type: New Feature > Components: webserver >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6669) Move Apache Transfer operators to providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6669. Resolution: Fixed > Move Apache Transfer operators to providers package > --- > > Key: AIRFLOW-6669 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6669 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6691) Add tests that protect the deprecated packages
[ https://issues.apache.org/jira/browse/AIRFLOW-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6691. Fix Version/s: 2.0.0 Resolution: Fixed > Add tests that protect the deprecated packages > -- > > Key: AIRFLOW-6691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6691 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6670) Move contrib operators to providers package
[ https://issues.apache.org/jira/browse/AIRFLOW-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6670. Resolution: Fixed > Move contrib operators to providers package > --- > > Key: AIRFLOW-6670 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6670 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6897) Simplify DagFileProcessorManager
[ https://issues.apache.org/jira/browse/AIRFLOW-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6897. Resolution: Fixed > Simplify DagFileProcessorManager > > > Key: AIRFLOW-6897 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6897 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6938) Don't read dag_directory in Scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6938. Fix Version/s: 2.0.0 Resolution: Fixed > Don't read dag_directory in Scheduler > - > > Key: AIRFLOW-6938 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6938 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6708) Set unique logger names
[ https://issues.apache.org/jira/browse/AIRFLOW-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-6708. Fix Version/s: 2.0.0 Resolution: Fixed > Set unique logger names > --- > > Key: AIRFLOW-6708 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6708 > Project: Apache Airflow > Issue Type: Improvement > Components: core, logging >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7073) GKEStartPodOperator uses user credentials instead of ADC
[ https://issues.apache.org/jira/browse/AIRFLOW-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7073. Fix Version/s: 2.0.0 Resolution: Fixed > GKEStartPodOperator uses user credentials instead of ADC > > > Key: AIRFLOW-7073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7073 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4425) Add FacebookAdsHook
[ https://issues.apache.org/jira/browse/AIRFLOW-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077294#comment-17077294 ] Kamil Bregula commented on AIRFLOW-4425: Can you check if someone is already working on this community and there is no Github issue or Github Pull Request? > Add FacebookAdsHook > --- > > Key: AIRFLOW-4425 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4425 > Project: Apache Airflow > Issue Type: Wish > Components: hooks >Reporter: jack >Assignee: Omair Khan >Priority: Major > Labels: gsoc, gsoc2020, mentor > Fix For: 2.0.0 > > > Add hook to interact with FacebookAds -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-3920) Moving everything out of contrib
[ https://issues.apache.org/jira/browse/AIRFLOW-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-3920. -- Resolution: Fixed > Moving everything out of contrib > > > Key: AIRFLOW-3920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3920 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Felix Uellendall >Assignee: Kamil Bregula >Priority: Major > > At the moment airflow has a contrib folder that contains a lot of modules > that were contributed by the community. > As this repo grows I think we shouldn't differ committers/maintainers work > from contributors work anymore. We as a community should strive for a unified > code base where everyone has to follow the same rules. > *To drive this change..* > I think we should create multiple PRs to completely move all modules out of > contrib. One PR would probably take too long and be uncomfortable to review. > I suggest that we create a SubTask/PR per subfolder of contrib (or maybe even > smaller ones). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5784) Migrate AWS S3 to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5784. -- Resolution: Fixed > Migrate AWS S3 to /providers/aws [AIP-21] > - > > Key: AIRFLOW-5784 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5784 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5776) Migrate AWS Cloudwatch logs to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5776. -- Resolution: Fixed > Migrate AWS Cloudwatch logs to /providers/aws [AIP-21] > -- > > Key: AIRFLOW-5776 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5776 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5779) Migrate AWS EMR to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5779. -- Resolution: Fixed > Migrate AWS EMR to /providers/aws [AIP-21] > -- > > Key: AIRFLOW-5779 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5779 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5778) Migrate AWS ECS to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5778. -- Resolution: Fixed > Migrate AWS ECS to /providers/aws [AIP-21] > -- > > Key: AIRFLOW-5778 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5778 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5790) Migrate AwsHook to /providers/aws
[ https://issues.apache.org/jira/browse/AIRFLOW-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-5790. -- Resolution: Fixed > Migrate AwsHook to /providers/aws > - > > Key: AIRFLOW-5790 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5790 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074498#comment-17074498 ] Kamil Bregula edited comment on AIRFLOW-6281 at 4/3/20, 11:29 AM: -- Yes. This is the operator. Can you speak on Slack or e-mail? I will send you a document with the guide e-mail: kamil.breg...@polidea.com was (Author: kamil.bregula): Yes. This is the operator. You can speak on Slack or e-mail so that he can send you a document with the guide e-mail: kamil.breg...@polidea.com > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6281) Create guide for GCS to GCS transfer operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074498#comment-17074498 ] Kamil Bregula commented on AIRFLOW-6281: Yes. This is the operator. You can speak on Slack or e-mail so that he can send you a document with the guide e-mail: kamil.breg...@polidea.com > Create guide for GCS to GCS transfer operator > -- > > Key: AIRFLOW-6281 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6281 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation, gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Minor > Labels: GoodForNewContributors > > Hello, > A guide that describes how to use GCS to GCS transfer operators would be > useful. > Other guides are available: > https://airflow.readthedocs.io/en/latest/howto/operator/gcp/index.html > If anyone is interested in this task, I am willing to provide all the > necessary tips and information. > Best regards, > Kamil -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7119) is there has plan to translate the document to chinese ?
[ https://issues.apache.org/jira/browse/AIRFLOW-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074253#comment-17074253 ] Kamil Bregula commented on AIRFLOW-7119: No one who maintains this Chinese translation was interested to make it official. We cannot provide this translation without the consent of the original maintainers. [https://github.com/apachecn/airflow-doc-zh/issues/65] It is worth noting that first the Chinese documentation was created, and then the official website. > is there has plan to translate the document to chinese ? > > > Key: AIRFLOW-7119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7119 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.9 >Reporter: Zhou Yao >Priority: Minor > > I think airflow is a great tool to schedule jobs and task every day.if the > comunity want to translate the document into chinese, Maybe i can help to do > this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6797) Create policy hooks for DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073603#comment-17073603 ] Kamil Bregula commented on AIRFLOW-6797: You can access DAG via task.dag property. So the cluster policy option also allows you to specify the DAG policy as well. > Create policy hooks for DAGs > > > Key: AIRFLOW-6797 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6797 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Matthew Bruce >Assignee: Kaxil Naik >Priority: Minor > > Policy hooks exist to modify task objects just before they are run: > [https://airflow.apache.org/docs/stable/concepts.html?highlight=policy#cluster-policy] > > Similar functionality for DAGs at loading time so that they could be rejected > or modified would be useful (i.e. to validate DAG naming, etc.) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7119) is there has plan to translate the document to chinese ?
[ https://issues.apache.org/jira/browse/AIRFLOW-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073588#comment-17073588 ] Kamil Bregula commented on AIRFLOW-7119: Community working on it: [https://airflow.apachecn.org/#/] > is there has plan to translate the document to chinese ? > > > Key: AIRFLOW-7119 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7119 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.9 >Reporter: Zhou Yao >Priority: Minor > > I think airflow is a great tool to schedule jobs and task every day.if the > comunity want to translate the document into chinese, Maybe i can help to do > this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-849) Show whether spark job is "running" or "waiting"
[ https://issues.apache.org/jira/browse/AIRFLOW-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070615#comment-17070615 ] Kamil Bregula commented on AIRFLOW-849: --- I think it should be a completely element. This task is constantly being execute and consumes worker resources. We can add a new column, e.g. external, which can be controlled by the task. The external state will not be used by the scheduler and will be able to have any label set by the task. WDYT? > Show whether spark job is "running" or "waiting" > > > Key: AIRFLOW-849 > URL: https://issues.apache.org/jira/browse/AIRFLOW-849 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Daniel Blazevski >Priority: Minor > > As of now, Airflow shows that a task is "running" in Airflow when it could be > listed as "WAITING" in the Spark UI (and can be listed as "WAITING" for hours > when the Airflow UI might indicate it is running if I forget to check the > Spark UI). > Would be awesome once ticket 802 for spark integration is complete to add > this enhancement of more granular detail about the task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-7039) Specific DAG Schedule & DST Results in Skipped DAG Run
[
https://issues.apache.org/jira/browse/AIRFLOW-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070573#comment-17070573
]
Kamil Bregula commented on AIRFLOW-7039:
I know where the problem is. is_fixed_time_schedule returns incorrect values
sometimes,...but the problem is that this method exists. Cron does not support
relative time expression.
> Specific DAG Schedule & DST Results in Skipped DAG Run
> --
>
> Key: AIRFLOW-7039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7039
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.7
> Environment: Amazon Linux 2 AMI
>Reporter: Peter Kim
>Priority: Critical
> Labels: timezone
>
> *Scenario:*
> EC2 running airflow is in Eastern Time (America/New_York),
> airflow.cfg>[core]>default_timezone=America/New_York (automatically changes
> correctly)
> Monday morning after Daylight Savings Time applied a handful of DAG runs were
> not executed as expected. The strange part is that these DAGs were the only
> jobs that did not behave as expected, all other DAGs ran normally.
> Additionally, only the first expected run after DST was skipped, subsequent
> runs later that day were scheduled successfully.
> Here is the pattern observed:
> DAG Schedule which skipped first run: (0 , * * 1,2,3,4,5)
> e.g. Schedules M-F, with two distinct runs per day.
> DAGs that run at one time, M-F & DAGs that run at two times, not M-F did not
> experience this issue.
>
> Based on the logs, it appears as if the expected run that was missed was not
> seen by the scheduler whatsoever (see below):
>
>
> 2020 03 06 6:30 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
> [2020-03-06 06:31:01,220] \{logging_mixin.py:112} INFO - [2020-03-06
> 06:31:01,220] \{settings.py:254} INFO - settings.configure_orm(): Using pool
> settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=697
> [2020-03-06 06:31:01,222] \{scheduler_job.py:153} INFO - Started process
> (PID=697) to work on /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,228] \{scheduler_job.py:1539} INFO - Processing file
> /home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
> [2020-03-06 06:31:01,228] \{logging_mixin.py:112} INFO - [2020-03-06
> 06:31:01,228] \{dagbag.py:403} INFO - Filling up the DagBag from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,238] \{scheduler_job.py:1551} INFO - DAG(s)
> dict_keys(['']) retrieved from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,305] \{scheduler_job.py:1262} INFO - Processing
>
> [2020-03-06 06:31:01,348] \{logging_mixin.py:112} WARNING -
> /home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170:
> Warning: (1300, "Invalid utf8mb4 character string: '80047D'") result =
> self._query(query)
> [2020-03-06 06:31:01,362] \{scheduler_job.py:1272} INFO - Created @ 2020-03-05T15:30:00+00:00: scheduled__2020-03-05T15:30:00+00:00,
> externally triggered: False>
> [2020-03-06 06:31:01,366] \{scheduler_job.py:740} INFO - Examining DAG run
> @ 2020-03-05 15:30:00+00:00:
> scheduled__2020-03-05T15:30:00+00:00, externally triggered: False>
> [2020-03-06 06:31:01,389] \{scheduler_job.py:440} INFO - Skipping SLA check
> for > because no tasks in DAG have SLAs
> [2020-03-06 06:31:01,395] \{scheduler_job.py:1613} INFO - Creating / updating
> . 2020-03-05 15:30:00+00:00 [scheduled]>
> in ORM
> [2020-03-06 06:31:01,414] \{scheduler_job.py:161} INFO - Processing
> /home/ec2-user/airflow/s3fuse/dags/.py took 0.192 seconds
> 20200306 10 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
> [2020-03-06 10:30:00,083] \{logging_mixin.py:112} INFO - [2020-03-06
> 10:30:00,082] \{settings.py:254} INFO - settings.configure_orm(): Using pool
> settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=16194
> [2020-03-06 10:30:00,085] \{scheduler_job.py:153} INFO - Started process
> (PID=16194) to work on /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,090] \{scheduler_job.py:1539} INFO - Processing file
> /home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
> [2020-03-06 10:30:00,090] \{logging_mixin.py:112} INFO - [2020-03-06
> 10:30:00,090] \{dagbag.py:403} INFO - Filling up the DagBag from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,099] \{scheduler_job.py:1551} INFO - DAG(s)
> dict_keys(['']) retrieved from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,159] \{scheduler_job.py:1262} INFO - Processing
>
> [2020-03-06 10:30:00,193] \{logging_mixin.py:112} WARNING -
> /home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170:
> Warning: (1300, "Invalid utf8mb4 character string: '80047D'")
> result = self._query(query)
> [2020-03-06 10:30:00
[jira] [Commented] (AIRFLOW-7039) Specific DAG Schedule & DST Results in Skipped DAG Run
[
https://issues.apache.org/jira/browse/AIRFLOW-7039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070545#comment-17070545
]
Kamil Bregula commented on AIRFLOW-7039:
[~potiuk]You are assigned to this task. What is his progress?
> Specific DAG Schedule & DST Results in Skipped DAG Run
> --
>
> Key: AIRFLOW-7039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7039
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.7
> Environment: Amazon Linux 2 AMI
>Reporter: Peter Kim
>Assignee: Jarek Potiuk
>Priority: Critical
> Labels: timezone
>
> *Scenario:*
> EC2 running airflow is in Eastern Time (America/New_York),
> airflow.cfg>[core]>default_timezone=America/New_York (automatically changes
> correctly)
> Monday morning after Daylight Savings Time applied a handful of DAG runs were
> not executed as expected. The strange part is that these DAGs were the only
> jobs that did not behave as expected, all other DAGs ran normally.
> Additionally, only the first expected run after DST was skipped, subsequent
> runs later that day were scheduled successfully.
> Here is the pattern observed:
> DAG Schedule which skipped first run: (0 , * * 1,2,3,4,5)
> e.g. Schedules M-F, with two distinct runs per day.
> DAGs that run at one time, M-F & DAGs that run at two times, not M-F did not
> experience this issue.
>
> Based on the logs, it appears as if the expected run that was missed was not
> seen by the scheduler whatsoever (see below):
>
>
> 2020 03 06 6:30 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
> [2020-03-06 06:31:01,220] \{logging_mixin.py:112} INFO - [2020-03-06
> 06:31:01,220] \{settings.py:254} INFO - settings.configure_orm(): Using pool
> settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=697
> [2020-03-06 06:31:01,222] \{scheduler_job.py:153} INFO - Started process
> (PID=697) to work on /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,228] \{scheduler_job.py:1539} INFO - Processing file
> /home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
> [2020-03-06 06:31:01,228] \{logging_mixin.py:112} INFO - [2020-03-06
> 06:31:01,228] \{dagbag.py:403} INFO - Filling up the DagBag from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,238] \{scheduler_job.py:1551} INFO - DAG(s)
> dict_keys(['']) retrieved from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 06:31:01,305] \{scheduler_job.py:1262} INFO - Processing
>
> [2020-03-06 06:31:01,348] \{logging_mixin.py:112} WARNING -
> /home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170:
> Warning: (1300, "Invalid utf8mb4 character string: '80047D'") result =
> self._query(query)
> [2020-03-06 06:31:01,362] \{scheduler_job.py:1272} INFO - Created @ 2020-03-05T15:30:00+00:00: scheduled__2020-03-05T15:30:00+00:00,
> externally triggered: False>
> [2020-03-06 06:31:01,366] \{scheduler_job.py:740} INFO - Examining DAG run
> @ 2020-03-05 15:30:00+00:00:
> scheduled__2020-03-05T15:30:00+00:00, externally triggered: False>
> [2020-03-06 06:31:01,389] \{scheduler_job.py:440} INFO - Skipping SLA check
> for > because no tasks in DAG have SLAs
> [2020-03-06 06:31:01,395] \{scheduler_job.py:1613} INFO - Creating / updating
> . 2020-03-05 15:30:00+00:00 [scheduled]>
> in ORM
> [2020-03-06 06:31:01,414] \{scheduler_job.py:161} INFO - Processing
> /home/ec2-user/airflow/s3fuse/dags/.py took 0.192 seconds
> 20200306 10 AM ET (BEFORE DST, EXPECTED BEHAVIOR):
> [2020-03-06 10:30:00,083] \{logging_mixin.py:112} INFO - [2020-03-06
> 10:30:00,082] \{settings.py:254} INFO - settings.configure_orm(): Using pool
> settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=16194
> [2020-03-06 10:30:00,085] \{scheduler_job.py:153} INFO - Started process
> (PID=16194) to work on /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,090] \{scheduler_job.py:1539} INFO - Processing file
> /home/ec2-user/airflow/s3fuse/dags/.py for tasks to queue
> [2020-03-06 10:30:00,090] \{logging_mixin.py:112} INFO - [2020-03-06
> 10:30:00,090] \{dagbag.py:403} INFO - Filling up the DagBag from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,099] \{scheduler_job.py:1551} INFO - DAG(s)
> dict_keys(['']) retrieved from
> /home/ec2-user/airflow/s3fuse/dags/.py
> [2020-03-06 10:30:00,159] \{scheduler_job.py:1262} INFO - Processing
>
> [2020-03-06 10:30:00,193] \{logging_mixin.py:112} WARNING -
> /home/ec2-user/venv/lib64/python3.7/site-packages/pymysql/cursors.py:170:
> Warning: (1300, "Invalid utf8mb4 character string: '80047D'")
> result = self._query(query)
> [2020-03-06 10:30:00,207] \{scheduler_job.py:1272} INFO - Created @ 2020-03-06T11:30:00+00:00: scheduled
[jira] [Closed] (AIRFLOW-6160) Move _check_and_change_state_before_execution to LocalTaskJob
[ https://issues.apache.org/jira/browse/AIRFLOW-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula closed AIRFLOW-6160. -- Resolution: Not A Problem > Move _check_and_change_state_before_execution to LocalTaskJob > - > > Key: AIRFLOW-6160 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6160 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Hello > I think that LocaclTaskJob should control which tasks are to be executed. > SchedulerJob and BackfillingJob behave similarly. > Best regards -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3116) Import / export connections from CLI (like for Variables)
[ https://issues.apache.org/jira/browse/AIRFLOW-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17063292#comment-17063292 ] Kamil Bregula commented on AIRFLOW-3116: [~lovk] for development, we use an environment variable. [https://airflow.readthedocs.io/en/latest/howto/connection/index.html#storing-a-connection-in-environment-variables] > Import / export connections from CLI (like for Variables) > - > > Key: AIRFLOW-3116 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3116 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.3 >Reporter: jack >Priority: Minor > > Variables have import/export from Json: > airflow variables > |-i, --import|Import variables from JSON file| > |-e, --export|Export variables to JSON file| > This behavior doesn't exist for connections > [https://airflow.apache.org/cli.html] > > Add this behavior also for connections. > Also requested on stackoverflow > https://stackoverflow.com/questions/48636783/how-to-migrate-airflow-variables-between-dev-and-prod-environments > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7084) Lazy initialize plugins for each entrypoint
[ https://issues.apache.org/jira/browse/AIRFLOW-7084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7084. Resolution: Fixed > Lazy initialize plugins for each entrypoint > --- > > Key: AIRFLOW-7084 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7084 > Project: Apache Airflow > Issue Type: Improvement > Components: plugins >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7087) Rename _execute_helper to _run_scheduler_loop
[ https://issues.apache.org/jira/browse/AIRFLOW-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7087. Fix Version/s: 2.0.0 Resolution: Fixed > Rename _execute_helper to _run_scheduler_loop > - > > Key: AIRFLOW-7087 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7087 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7085) Cache credentials, project_id in GCP Base Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7085. Fix Version/s: 2.0.0 Resolution: Fixed > Cache credentials, project_id in GCP Base Hook > -- > > Key: AIRFLOW-7085 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7085 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7089) Create kill method in AbstractDagFileProcessorProcess
[ https://issues.apache.org/jira/browse/AIRFLOW-7089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7089. Resolution: Fixed > Create kill method in AbstractDagFileProcessorProcess > - > > Key: AIRFLOW-7089 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7089 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7088) Avoid reading store_dag_code option in loop
[ https://issues.apache.org/jira/browse/AIRFLOW-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7088. Fix Version/s: 2.0.0 Resolution: Fixed > Avoid reading store_dag_code option in loop > --- > > Key: AIRFLOW-7088 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7088 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7089) Create kill method in AbstractDagFileProcessorProcess
Kamil Bregula created AIRFLOW-7089: -- Summary: Create kill method in AbstractDagFileProcessorProcess Key: AIRFLOW-7089 URL: https://issues.apache.org/jira/browse/AIRFLOW-7089 Project: Apache Airflow Issue Type: Improvement Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7088) Avoid reading store_dag_code option in loop
Kamil Bregula created AIRFLOW-7088: -- Summary: Avoid reading store_dag_code option in loop Key: AIRFLOW-7088 URL: https://issues.apache.org/jira/browse/AIRFLOW-7088 Project: Apache Airflow Issue Type: Improvement Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7087) Rename _execute_helper to _run_scheduler_loop
Kamil Bregula created AIRFLOW-7087: -- Summary: Rename _execute_helper to _run_scheduler_loop Key: AIRFLOW-7087 URL: https://issues.apache.org/jira/browse/AIRFLOW-7087 Project: Apache Airflow Issue Type: Improvement Components: scheduler Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7085) Cache credentials, project_id in GCP Base Hook
[ https://issues.apache.org/jira/browse/AIRFLOW-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-7085: --- Summary: Cache credentials, project_id in GCP Base Hook (was: Cached credentials, project_id in GCP Base Hook) > Cache credentials, project_id in GCP Base Hook > -- > > Key: AIRFLOW-7085 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7085 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7085) Cached credentials, project_id in GCP Base Hook
Kamil Bregula created AIRFLOW-7085: -- Summary: Cached credentials, project_id in GCP Base Hook Key: AIRFLOW-7085 URL: https://issues.apache.org/jira/browse/AIRFLOW-7085 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7084) Lazy initialize plugins for each entrypoint
Kamil Bregula created AIRFLOW-7084: -- Summary: Lazy initialize plugins for each entrypoint Key: AIRFLOW-7084 URL: https://issues.apache.org/jira/browse/AIRFLOW-7084 Project: Apache Airflow Issue Type: Improvement Components: plugins Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7064) Add CloudFirestoreExportDatabaseOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7064. Fix Version/s: 2.0.0 Resolution: Fixed > Add CloudFirestoreExportDatabaseOperator > > > Key: AIRFLOW-7064 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7064 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5421) Request for PrestoToGoogleStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reopened AIRFLOW-5421: > Request for PrestoToGoogleStorageOperator > - > > Key: AIRFLOW-5421 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5421 > Project: Apache Airflow > Issue Type: Wish > Components: gcp >Affects Versions: 1.10.2 >Reporter: HaloKu >Priority: Major > Fix For: 2.0.0 > > > I need to load data from presto to google storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5421) Request for PrestoToGoogleStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5421. Assignee: Kamil Bregula Resolution: Fixed > Request for PrestoToGoogleStorageOperator > - > > Key: AIRFLOW-5421 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5421 > Project: Apache Airflow > Issue Type: Wish > Components: gcp >Affects Versions: 1.10.2 >Reporter: HaloKu >Assignee: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > > I need to load data from presto to google storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5421) Request for PrestoToGoogleStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-5421. Fix Version/s: 2.0.0 Resolution: Fixed > Request for PrestoToGoogleStorageOperator > - > > Key: AIRFLOW-5421 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5421 > Project: Apache Airflow > Issue Type: Wish > Components: gcp >Affects Versions: 1.10.2 >Reporter: HaloKu >Priority: Major > Fix For: 2.0.0 > > > I need to load data from presto to google storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7081) Remove env variable from GCP guide
[ https://issues.apache.org/jira/browse/AIRFLOW-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7081. Resolution: Fixed > Remove env variable from GCP guide > -- > > Key: AIRFLOW-7081 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7081 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation, gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7081) Remove env variable from GCP guide
Kamil Bregula created AIRFLOW-7081: -- Summary: Remove env variable from GCP guide Key: AIRFLOW-7081 URL: https://issues.apache.org/jira/browse/AIRFLOW-7081 Project: Apache Airflow Issue Type: Improvement Components: documentation, gcp Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7073) GKEStartPodOperator uses user credentials instead of ADC
Kamil Bregula created AIRFLOW-7073: -- Summary: GKEStartPodOperator uses user credentials instead of ADC Key: AIRFLOW-7073 URL: https://issues.apache.org/jira/browse/AIRFLOW-7073 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-7064) Add CloudFirestoreExportDatabaseOperator
Kamil Bregula created AIRFLOW-7064: -- Summary: Add CloudFirestoreExportDatabaseOperator Key: AIRFLOW-7064 URL: https://issues.apache.org/jira/browse/AIRFLOW-7064 Project: Apache Airflow Issue Type: Improvement Components: gcp Affects Versions: 1.10.9 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-7064) Add CloudFirestoreExportDatabaseOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-7064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula updated AIRFLOW-7064: --- Issue Type: New Feature (was: Improvement) > Add CloudFirestoreExportDatabaseOperator > > > Key: AIRFLOW-7064 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7064 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5421) Request for PrestoToGoogleStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059174#comment-17059174 ] Kamil Bregula commented on AIRFLOW-5421: [~HaloKu] I created the PR. [https://github.com/apache/airflow/pull/7718] Can I ask for the review? > Request for PrestoToGoogleStorageOperator > - > > Key: AIRFLOW-5421 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5421 > Project: Apache Airflow > Issue Type: Wish > Components: gcp >Affects Versions: 1.10.2 >Reporter: HaloKu >Priority: Major > > I need to load data from presto to google storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5310) Add PrestoToGoogleStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059173#comment-17059173 ] Kamil Bregula commented on AIRFLOW-5310: [~lovk] I created the PR. [https://github.com/apache/airflow/pull/7718] Can I ask for the review? > Add PrestoToGoogleStorageOperator > - > > Key: AIRFLOW-5310 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5310 > Project: Apache Airflow > Issue Type: Wish > Components: contrib, gcp, operators >Affects Versions: 1.10.4 >Reporter: lovk korm >Priority: Major > > Please add PrestoToGoogleStorageOperator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7003) Lazy load all plguins
[ https://issues.apache.org/jira/browse/AIRFLOW-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7003. Fix Version/s: 2.0.0 Resolution: Fixed > Lazy load all plguins > - > > Key: AIRFLOW-7003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7003 > Project: Apache Airflow > Issue Type: Improvement > Components: plugins >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7022) Simplify DagFileProcessor.process_file method
[ https://issues.apache.org/jira/browse/AIRFLOW-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7022. Fix Version/s: 2.0.0 Resolution: Fixed > Simplify DagFileProcessor.process_file method > - > > Key: AIRFLOW-7022 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7022 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7021) Avoid multiple calls to conf.getboolean in _process_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7021. Fix Version/s: 2.0.0 Resolution: Fixed > Avoid multiple calls to conf.getboolean in _process_dags > > > Key: AIRFLOW-7021 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7021 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-7021) Avoid multiple calls to conf.getboolean in _process_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reassigned AIRFLOW-7021: -- Assignee: Kamil Bregula > Avoid multiple calls to conf.getboolean in _process_dags > > > Key: AIRFLOW-7021 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7021 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-7021) Avoid multiple calls to conf.getboolean in _process_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula reopened AIRFLOW-7021: > Avoid multiple calls to conf.getboolean in _process_dags > > > Key: AIRFLOW-7021 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7021 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-7021) Avoid multiple calls to conf.getboolean in _process_dags
[ https://issues.apache.org/jira/browse/AIRFLOW-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamil Bregula resolved AIRFLOW-7021. Resolution: Fixed > Avoid multiple calls to conf.getboolean in _process_dags > > > Key: AIRFLOW-7021 > URL: https://issues.apache.org/jira/browse/AIRFLOW-7021 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.9 >Reporter: Kamil Bregula >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
