[GitHub] [airflow] tfindlay-au commented on a change in pull request #6604: [AIRFLOW-5920] Neo4j operator and hook
tfindlay-au commented on a change in pull request #6604: [AIRFLOW-5920] Neo4j operator and hook URL: https://github.com/apache/airflow/pull/6604#discussion_r358022006 ## File path: airflow/contrib/hooks/neo4j_hook.py ## @@ -0,0 +1,101 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""This hook provides minimal thin wrapper around the neo4j python library to provide query execution""" +from neo4j import BoltStatementResult, Driver, GraphDatabase, Session + +from airflow.hooks.base_hook import BaseHook + + +class Neo4JHook(BaseHook): +"""This class enables the neo4j operator to execute queries against a configured neo4j server. +It requires the configuration name as set in Airflow -> Connections -> +:param n4j_conn_id: +:type str: +""" +n4j_conn_id: str Review comment: @feluelle Waiting for your advice on this please. Where would you have defined this ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r358007082
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
Oh, I see. 👍
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Comment Edited] (AIRFLOW-4424) Scheduler does not terminate after num_runs when executor is KubernetesExecutor
[
https://issues.apache.org/jira/browse/AIRFLOW-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996849#comment-16996849
]
Gaurav Sehgal edited comment on AIRFLOW-4424 at 12/15/19 7:04 PM:
--
Hi, At GoJek, we are facing the same issue with the local executor. Here's the
thread dump.
~~
{code:java}
ThreadID: 140356901611264
File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File: "", line 167, in run
File: "/usr/local/lib/python3.7/code.py", line 232, in interact
more = self.push(line)
File: "/usr/local/lib/python3.7/code.py", line 258, in push
more = self.runsource(source, self.filename)
File: "/usr/local/lib/python3.7/code.py", line 74, in runsource
self.runcode(code)
File: "/usr/local/lib/python3.7/code.py", line 90, in runcode
exec(code, self.locals)
File: "", line 3, in
ThreadID: 140358376056576
File: "/usr/local/bin/airflow", line 37, in
args.func(args)
File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74,
in wrapper
return f(*args, **kwargs)
File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042,
in scheduler
job.run()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1356, in _execute
self._execute_helper()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1496, in _execute_helper
self.executor.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 233, in end
self.impl.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 212, in end
self.queue.join()
File: "", line 2, in join
File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in
_callmethod
kind, result = conn.recv()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in
recv
buf = self._recv_bytes()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in
_recv_bytes
buf = self._recv(4)
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in
_recv
chunk = read(handle, remaining){code}
was (Author: gaurav123):
Hi, At GoJek, we are facing the same issue with the local executor. Here's the
thread dump.
```
ThreadID: 140356901611264
File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File: "", line 167, in run
File: "/usr/local/lib/python3.7/code.py", line 232, in interact
more = self.push(line)
File: "/usr/local/lib/python3.7/code.py", line 258, in push
more = self.runsource(source, self.filename)
File: "/usr/local/lib/python3.7/code.py", line 74, in runsource
self.runcode(code)
File: "/usr/local/lib/python3.7/code.py", line 90, in runcode
exec(code, self.locals)
File: "", line 3, in
ThreadID: 140358376056576
File: "/usr/local/bin/airflow", line 37, in
args.func(args)
File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74,
in wrapper
return f(*args, **kwargs)
File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042,
in scheduler
job.run()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1356, in _execute
self._execute_helper()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1496, in _execute_helper
self.executor.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 233, in end
self.impl.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 212, in end
self.queue.join()
File: "", line 2, in join
File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in
_callmethod
kind, result = conn.recv()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in
recv
buf = self._recv_bytes()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in
_recv_bytes
buf = self._recv(4)
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in
_recv
chunk = read(handle, remaining)
```
> Scheduler does not terminate after num_runs when executor is
> KubernetesExecutor
> ---
>
> Key: AIRFLOW-4424
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4424
> Project: Apache Airflow
> Issue Type: Bug
> Comp
[jira] [Comment Edited] (AIRFLOW-4424) Scheduler does not terminate after num_runs when executor is KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996849#comment-16996849 ] Gaurav Sehgal edited comment on AIRFLOW-4424 at 12/15/19 7:03 PM: -- Hi, At GoJek, we are facing the same issue with the local executor. Here's the thread dump. ``` ThreadID: 140356901611264 File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner() File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File: "", line 167, in run File: "/usr/local/lib/python3.7/code.py", line 232, in interact more = self.push(line) File: "/usr/local/lib/python3.7/code.py", line 258, in push more = self.runsource(source, self.filename) File: "/usr/local/lib/python3.7/code.py", line 74, in runsource self.runcode(code) File: "/usr/local/lib/python3.7/code.py", line 90, in runcode exec(code, self.locals) File: "", line 3, in ThreadID: 140358376056576 File: "/usr/local/bin/airflow", line 37, in args.func(args) File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042, in scheduler job.run() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 222, in run self._execute() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1356, in _execute self._execute_helper() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1496, in _execute_helper self.executor.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 233, in end self.impl.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 212, in end self.queue.join() File: "", line 2, in join File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod kind, result = conn.recv() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ``` was (Author: gaurav123): Hi, At GoJek, we are facing the same issue with the local executor. Here's the thread dump. ``` # ThreadID: 140356901611264 File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner() File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File: "", line 167, in run File: "/usr/local/lib/python3.7/code.py", line 232, in interact more = self.push(line) File: "/usr/local/lib/python3.7/code.py", line 258, in push more = self.runsource(source, self.filename) File: "/usr/local/lib/python3.7/code.py", line 74, in runsource self.runcode(code) File: "/usr/local/lib/python3.7/code.py", line 90, in runcode exec(code, self.locals) File: "", line 3, in # ThreadID: 140358376056576 File: "/usr/local/bin/airflow", line 37, in args.func(args) File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042, in scheduler job.run() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 222, in run self._execute() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1356, in _execute self._execute_helper() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1496, in _execute_helper self.executor.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 233, in end self.impl.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 212, in end self.queue.join() File: "", line 2, in join File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod kind, result = conn.recv() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ``` > Scheduler does not terminate after num_runs when executor is > KubernetesExecutor > --- > > Key: AIRFLOW-4424 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4424 > Project: Apache Airflow > Issue Type: Bug > Components: exe
[jira] [Comment Edited] (AIRFLOW-4424) Scheduler does not terminate after num_runs when executor is KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996849#comment-16996849 ] Gaurav Sehgal edited comment on AIRFLOW-4424 at 12/15/19 7:01 PM: -- Hi, At GoJek, we are facing the same issue with the local executor. Here's the thread dump. ``` # ThreadID: 140356901611264 File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner() File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File: "", line 167, in run File: "/usr/local/lib/python3.7/code.py", line 232, in interact more = self.push(line) File: "/usr/local/lib/python3.7/code.py", line 258, in push more = self.runsource(source, self.filename) File: "/usr/local/lib/python3.7/code.py", line 74, in runsource self.runcode(code) File: "/usr/local/lib/python3.7/code.py", line 90, in runcode exec(code, self.locals) File: "", line 3, in # ThreadID: 140358376056576 File: "/usr/local/bin/airflow", line 37, in args.func(args) File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042, in scheduler job.run() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 222, in run self._execute() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1356, in _execute self._execute_helper() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1496, in _execute_helper self.executor.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 233, in end self.impl.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 212, in end self.queue.join() File: "", line 2, in join File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod kind, result = conn.recv() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ``` was (Author: gaurav123): Hi, At GoJek, we are facing the same issue with the local executor. Here's the thread dump. ``` # ThreadID: 140356901611264 File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner() File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File: "", line 167, in run File: "/usr/local/lib/python3.7/code.py", line 232, in interact more = self.push(line) File: "/usr/local/lib/python3.7/code.py", line 258, in push more = self.runsource(source, self.filename) File: "/usr/local/lib/python3.7/code.py", line 74, in runsource self.runcode(code) File: "/usr/local/lib/python3.7/code.py", line 90, in runcode exec(code, self.locals) File: "", line 3, in # ThreadID: 140358376056576 File: "/usr/local/bin/airflow", line 37, in args.func(args) File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042, in scheduler job.run() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 222, in run self._execute() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1356, in _execute self._execute_helper() File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1496, in _execute_helper self.executor.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 233, in end self.impl.end() File: "/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 212, in end self.queue.join() File: "", line 2, in join File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod kind, result = conn.recv() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ``` > Scheduler does not terminate after num_runs when executor is > KubernetesExecutor > --- > > Key: AIRFLOW-4424 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4424 > Project: Apache Airflow > Issue Type: Bug > Components: execu
[jira] [Commented] (AIRFLOW-4424) Scheduler does not terminate after num_runs when executor is KubernetesExecutor
[
https://issues.apache.org/jira/browse/AIRFLOW-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996849#comment-16996849
]
Gaurav Sehgal commented on AIRFLOW-4424:
Hi, At GoJek, we are facing the same issue with the local executor. Here's the
thread dump.
```
# ThreadID: 140356901611264
File: "/usr/local/lib/python3.7/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File: "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File: "", line 167, in run
File: "/usr/local/lib/python3.7/code.py", line 232, in interact
more = self.push(line)
File: "/usr/local/lib/python3.7/code.py", line 258, in push
more = self.runsource(source, self.filename)
File: "/usr/local/lib/python3.7/code.py", line 74, in runsource
self.runcode(code)
File: "/usr/local/lib/python3.7/code.py", line 90, in runcode
exec(code, self.locals)
File: "", line 3, in # ThreadID: 140358376056576
File: "/usr/local/bin/airflow", line 37, in
args.func(args)
File: "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 74,
in wrapper
return f(*args, **kwargs)
File: "/usr/local/lib/python3.7/site-packages/airflow/bin/cli.py", line 1042,
in scheduler
job.run()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1356, in _execute
self._execute_helper()
File: "/usr/local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py",
line 1496, in _execute_helper
self.executor.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 233, in end
self.impl.end()
File:
"/usr/local/lib/python3.7/site-packages/airflow/executors/local_executor.py",
line 212, in end
self.queue.join()
File: "", line 2, in join
File: "/usr/local/lib/python3.7/multiprocessing/managers.py", line 819, in
_callmethod
kind, result = conn.recv()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 250, in
recv
buf = self._recv_bytes()
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 407, in
_recv_bytes
buf = self._recv(4)
File: "/usr/local/lib/python3.7/multiprocessing/connection.py", line 379, in
_recv
chunk = read(handle, remaining)
```
> Scheduler does not terminate after num_runs when executor is
> KubernetesExecutor
> ---
>
> Key: AIRFLOW-4424
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4424
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors, scheduler
>Affects Versions: 1.10.3
> Environment: EKS, deployed with stable airflow helm chart
>Reporter: Brian Nutt
>Priority: Blocker
> Labels: kubernetes
> Fix For: 2.0.0
>
>
> When using the executor like the CeleryExecutor and num_runs is set on the
> scheduler, the scheduler pod restarts after num runs have completed. After
> switching to KubernetesExecutor, the scheduler logs:
> [2019-04-26 19:20:43,562] \{{kubernetes_executor.py:770}} INFO - Shutting
> down Kubernetes executor
> However, the scheduler process does not complete. This leads to the scheduler
> pod never restarting and running num_runs again. Resulted in having to roll
> back to CeleryExecutor because if num_runs is -1, the scheduler builds up
> tons of defunct processes, which is eventually making tasks not able to be
> scheduled as the underlying nodes have run out of file descriptors.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions. URL: https://github.com/apache/airflow/pull/6822#issuecomment-565816371 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=h1) Report > Merging [#6822](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84.61%`. [](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6822 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138335 +4 = - Hits3243132321-110 - Misses 5900 6014+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/presto\_hook.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wcmVzdG9faG9vay5weQ==) | `75.47% <84.61%> (-2.08%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.8% <0%> (-0.2%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=footer). Last update [8107651...a188522](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions. URL: https://github.com/apache/airflow/pull/6822#issuecomment-565816371 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=h1) Report > Merging [#6822](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84.61%`. [](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6822 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138335 +4 = - Hits3243132321-110 - Misses 5900 6014+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/presto\_hook.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wcmVzdG9faG9vay5weQ==) | `75.47% <84.61%> (-2.08%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.8% <0%> (-0.2%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=footer). Last update [8107651...a188522](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3282) Implement a Azure Kubernetes Service Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996831#comment-16996831 ] ASF GitHub Bot commented on AIRFLOW-3282: - stale[bot] commented on pull request #4530: [AIRFLOW-3282] Implement Azure Kubernetes Service Operator URL: https://github.com/apache/airflow/pull/4530 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement a Azure Kubernetes Service Operator > - > > Key: AIRFLOW-3282 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3282 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Aparna >Assignee: Aparna >Priority: Major > > Add AKS Operator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] commented on issue #6451: [AIRFLOW-5796] Fix case when slot_pool relation is missing
stale[bot] commented on issue #6451: [AIRFLOW-5796] Fix case when slot_pool relation is missing URL: https://github.com/apache/airflow/pull/6451#issuecomment-565833973 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] closed pull request #4530: [AIRFLOW-3282] Implement Azure Kubernetes Service Operator
stale[bot] closed pull request #4530: [AIRFLOW-3282] Implement Azure Kubernetes Service Operator URL: https://github.com/apache/airflow/pull/4530 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994715
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I think `IsolationLevel.isolation_level` won't work because isolation_level
is user-defined and it's in a string format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994715
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I think `IsolationLevel.isolation_level` won't work because isolation_level
is user defined and it's in string format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994715
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I think `IsolationLevel. isolation_level` won't work because isolation_level
is user defined and it's in string format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994715
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I think `IsolationLevel.isolation_level` won't work because isolation_level
is user-defined and it's in string format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994715
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I think `IsolationLevel. isolation_level` won't work because isolation_level
is user input and it's in string format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994655
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357994649
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
Will replace it with IsolationLevel.AUTOCOMMIT.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357992572
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
I would suggest `isolation_level.IsolationLevel or 0` instead. IMO `getattr`
often makes code a bit harder to read.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357992882
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
+
+isolation_level = db.extra_dejson.get('isolation_level',
'AUTOCOMMIT').upper()
+return prestodb.dbapi.connect(
host=db.host,
port=db.port,
-username=db.login,
+user=db.login,
source=db.extra_dejson.get('source', 'airflow'),
-protocol=db.extra_dejson.get('protocol', 'http'),
+http_scheme=db.extra_dejson.get('protocol', 'http'),
catalog=db.extra_dejson.get('catalog', 'hive'),
-requests_kwargs=reqkwargs,
-schema=db.schema)
+schema=db.schema,
+auth=auth,
+isolation_level=getattr(IsolationLevel, isolation_level, 0)
Review comment:
`0` as default feels a bit like a magic value. Is there a more descriptive
alternative?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
pgagnon commented on a change in pull request #6822: [AIRFLOW-5616] Refactor
presto hook to support transactions.
URL: https://github.com/apache/airflow/pull/6822#discussion_r357992816
##
File path: airflow/hooks/presto_hook.py
##
@@ -44,18 +45,22 @@ class PrestoHook(DbApiHook):
def get_conn(self):
"""Returns a connection object"""
db = self.get_connection(self.presto_conn_id)
-reqkwargs = None
+auth = None
if db.password is not None:
-reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
-return presto.connect(
+auth = prestodb.auth.BasicAuthentication(db.login, db.password)
Review comment:
This block could be replaced with `auth =
prestodb.auth.BasicAuthentication(db.login, db.password) if db.password else
None`. This would also allow you to inline this at line 61.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
codecov-io edited a comment on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions. URL: https://github.com/apache/airflow/pull/6822#issuecomment-565816371 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=h1) Report > Merging [#6822](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `85.71%`. [](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6822 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138337 +6 = - Hits3243132323-108 - Misses 5900 6014+114 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/presto\_hook.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wcmVzdG9faG9vay5weQ==) | `76.36% <85.71%> (-1.19%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.8% <0%> (-0.2%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=footer). Last update [8107651...da5d58c](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6812: [AIRFLOW-6250] handle_failure needs better default for context and test_mode
codecov-io commented on issue #6812: [AIRFLOW-6250] handle_failure needs better default for context and test_mode URL: https://github.com/apache/airflow/pull/6812#issuecomment-565825685 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=h1) Report > Merging [#6812](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#6812 +/- ## == - Coverage84.6% 84.31% -0.29% == Files 676 676 Lines 3833138335 +4 == - Hits3243132324 -107 - Misses 5900 6011 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.98% <100%> (+0.18%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | | [airflow/models/taskfail.py](https://codecov.io/gh/apache/airflow/pull/6812/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2ZhaWwucHk=) | `100% <0%> (+4.34%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=footer). Last update [8107651...8e7305e](https://codecov.io/gh/apache/airflow/pull/6812?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/6633#issuecomment-560395451 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=h1) Report > Merging [#6633](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `86.88%`. [](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6633 +/- ## = - Coverage84.6% 84.6% -0.01% = Files 676 677 +1 Lines 38331 38390 +59 = + Hits32431 32478 +47 - Misses 59005912 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...rflow/example\_dags/example\_external\_task\_marker.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9leHRlcm5hbF90YXNrX21hcmtlci5weQ==) | `100% <100%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.95% <50%> (-0.13%)` | :arrow_down: | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.71% <85.71%> (-0.26%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.95% <90.9%> (-0.01%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=footer). Last update [8107651...f1f173a](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/6633#issuecomment-560395451 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=h1) Report > Merging [#6633](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `86.88%`. [](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6633 +/- ## = - Coverage84.6% 84.6% -0.01% = Files 676 677 +1 Lines 38331 38390 +59 = + Hits32431 32478 +47 - Misses 59005912 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...rflow/example\_dags/example\_external\_task\_marker.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9leHRlcm5hbF90YXNrX21hcmtlci5weQ==) | `100% <100%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.95% <50%> (-0.13%)` | :arrow_down: | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.71% <85.71%> (-0.26%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.95% <90.9%> (-0.01%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=footer). Last update [8107651...f1f173a](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
codecov-io edited a comment on issue #6633: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/6633#issuecomment-560395451 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=h1) Report > Merging [#6633](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `86.88%`. [](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #6633 +/- ## = - Coverage84.6% 84.6% -0.01% = Files 676 677 +1 Lines 38331 38390 +59 = + Hits32431 32478 +47 - Misses 59005912 +12 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [...rflow/example\_dags/example\_external\_task\_marker.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9leHRlcm5hbF90YXNrX21hcmtlci5weQ==) | `100% <100%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `75.95% <50%> (-0.13%)` | :arrow_down: | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.71% <85.71%> (-0.26%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.95% <90.9%> (-0.01%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6633/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=footer). Last update [8107651...f1f173a](https://codecov.io/gh/apache/airflow/pull/6633?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
codecov-io commented on issue #6822: [AIRFLOW-5616] Refactor presto hook to support transactions. URL: https://github.com/apache/airflow/pull/6822#issuecomment-565816371 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=h1) Report > Merging [#6822](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `85.71%`. [](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6822 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138337 +6 = - Hits3243132322-109 - Misses 5900 6015+115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/hooks/presto\_hook.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9ob29rcy9wcmVzdG9faG9vay5weQ==) | `76.36% <85.71%> (-1.19%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6822/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.42% <0%> (-0.58%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=footer). Last update [8107651...4b9ed7a](https://codecov.io/gh/apache/airflow/pull/6822?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=h1) Report > Merging [#4751](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#4751 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138339 +8 = - Hits3243132326-105 - Misses 5900 6013+113 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.15% <100%> (-0.12%)` | :arrow_down: | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `91.25% <100%> (+0.46%)` | :arrow_up: | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.27% <100%> (-0.31%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=footer). Last update [8107651...b62e621](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@e17b2cc`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#4751 +/- ## = Coverage ? 84.31% = Files ? 676 Lines ?38339 Branches ?0 = Hits ?32327 Misses? 6012 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.15% <100%> (ø)` | | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `91.25% <100%> (ø)` | | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.27% <100%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=footer). Last update [e17b2cc...7babbe3](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
codecov-io edited a comment on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-466029246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@e17b2cc`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#4751 +/- ## = Coverage ? 84.31% = Files ? 676 Lines ?38339 Branches ?0 = Hits ?32327 Misses? 6012 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.15% <100%> (ø)` | | | [airflow/ti\_deps/deps/trigger\_rule\_dep.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvdHJpZ2dlcl9ydWxlX2RlcC5weQ==) | `91.25% <100%> (ø)` | | | [airflow/ti\_deps/dep\_context.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcF9jb250ZXh0LnB5) | `100% <100%> (ø)` | | | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/4751/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `96.27% <100%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=footer). Last update [e17b2cc...7babbe3](https://codecov.io/gh/apache/airflow/pull/4751?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6258) CloudFormation create_stack and delete_stack operators
[
https://issues.apache.org/jira/browse/AIRFLOW-6258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996712#comment-16996712
]
ASF GitHub Bot commented on AIRFLOW-6258:
-
aviemzur commented on pull request #6824: [AIRFLOW-6258] add CloudFormation
operators to contrib
URL: https://github.com/apache/airflow/pull/6824
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add 2 new operators: `CloudFormationCreateStackOperator` and
`CloudFormationDeleteStackOperator`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
`TestCloudFormationCreateStackOperator` and
`TestCloudFormationDeleteStackOperator`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> CloudFormation create_stack and delete_stack operators
> --
>
> Key: AIRFLOW-6258
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6258
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
>Affects Versions: 1.10.6
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Major
>
> Add CloudFormation create_stack and delete_stack operators.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] aviemzur opened a new pull request #6824: [AIRFLOW-6258] add CloudFormation operators to contrib
aviemzur opened a new pull request #6824: [AIRFLOW-6258] add CloudFormation
operators to contrib
URL: https://github.com/apache/airflow/pull/6824
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add 2 new operators: `CloudFormationCreateStackOperator` and
`CloudFormationDeleteStackOperator`
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
`TestCloudFormationCreateStackOperator` and
`TestCloudFormationDeleteStackOperator`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6258) CloudFormation create_stack and delete_stack operators
Aviem Zur created AIRFLOW-6258: -- Summary: CloudFormation create_stack and delete_stack operators Key: AIRFLOW-6258 URL: https://issues.apache.org/jira/browse/AIRFLOW-6258 Project: Apache Airflow Issue Type: New Feature Components: contrib Affects Versions: 1.10.6 Reporter: Aviem Zur Assignee: Aviem Zur Add CloudFormation create_stack and delete_stack operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6820 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138354 +23 = - Hits3243132337 -94 - Misses 5900 6017+117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=footer). Last update [8107651...7a97d74](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6820 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138354 +23 = - Hits3243132337 -94 - Misses 5900 6017+117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=footer). Last update [8107651...7a97d74](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor
codecov-io edited a comment on issue #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#issuecomment-565753224 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=h1) Report > Merging [#6820](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8107651f8f578cc98801f37ae50ebc6be9e9d035?src=pr&el=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master#6820 +/- ## = - Coverage84.6% 84.31% -0.3% = Files 676 676 Lines 3833138354 +23 = - Hits3243132337 -94 - Misses 5900 6017+117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/redis\_key\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfa2V5X3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [...irflow/contrib/operators/redis\_publish\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9yZWRpc19wdWJsaXNoX29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/redis\_hook.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3JlZGlzX2hvb2sucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/redis\_pub\_sub\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcmVkaXNfcHViX3N1Yl9zZW5zb3IucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/6820/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=footer). Last update [8107651...7a97d74](https://codecov.io/gh/apache/airflow/pull/6820?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ohadmata opened a new pull request #6823: Update ti_log.html - add colored log support
ohadmata opened a new pull request #6823: Update ti_log.html - add colored log
support
URL: https://github.com/apache/airflow/pull/6823
Add colored log support to the airflow log screen
print('[class=red]Test log RED')
print('[class=orange]Test log ORANGE')
print('[class=green]Test log GREEN')
Screenshot : http://imgur.com/a/XJt7hal
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dggs123 opened a new pull request #6822: [AIRFLOW-5616] Refactor presto hook to support transactions.
dggs123 opened a new pull request #6822: [AIRFLOW-5616] Refactor presto hook to
support transactions.
URL: https://github.com/apache/airflow/pull/6822
- Replace the pyhive client with presto official client
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
This PR aims to refactor presto hook by replacing the pyhive client with
prestosql official client. Now presto hook also supports transactions. Users
can specify the type of transaction in the presto connection by setting the
extra field isolation_level.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
I've modified some tests of presto_hook to support the new client.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5616) PrestoHook to use prestodb
[
https://issues.apache.org/jira/browse/AIRFLOW-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996703#comment-16996703
]
ASF GitHub Bot commented on AIRFLOW-5616:
-
dggs123 commented on pull request #6822: [AIRFLOW-5616] Refactor presto hook to
support transactions.
URL: https://github.com/apache/airflow/pull/6822
- Replace the pyhive client with presto official client
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
This PR aims to refactor presto hook by replacing the pyhive client with
prestosql official client. Now presto hook also supports transactions. Users
can specify the type of transaction in the presto connection by setting the
extra field isolation_level.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
I've modified some tests of presto_hook to support the new client.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> PrestoHook to use prestodb
> --
>
> Key: AIRFLOW-5616
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5616
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.5
>Reporter: Alexandre Brilhante
>Assignee: Gaurav Sehgal
>Priority: Minor
>
> PrestoHook currently uses PyHive which doesn't support transactions whereas
> prestodb
> ([https://github.com/prestodb/presto-python-client)|https://github.com/prestodb/presto-python-client]
> does. I think it would more flexible to use prestodb as client. I can work
> on a PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5616) PrestoHook to use prestodb
[ https://issues.apache.org/jira/browse/AIRFLOW-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996699#comment-16996699 ] ASF GitHub Bot commented on AIRFLOW-5616: - dggs123 commented on pull request #6821: [AIRFLOW-5616] Replace pyhive with presto official client in presto hook. URL: https://github.com/apache/airflow/pull/6821 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > PrestoHook to use prestodb > -- > > Key: AIRFLOW-5616 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5616 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 1.10.5 >Reporter: Alexandre Brilhante >Assignee: Gaurav Sehgal >Priority: Minor > > PrestoHook currently uses PyHive which doesn't support transactions whereas > prestodb > ([https://github.com/prestodb/presto-python-client)|https://github.com/prestodb/presto-python-client] > does. I think it would more flexible to use prestodb as client. I can work > on a PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] dggs123 closed pull request #6821: [AIRFLOW-5616] Replace pyhive with presto official client in presto hook.
dggs123 closed pull request #6821: [AIRFLOW-5616] Replace pyhive with presto official client in presto hook. URL: https://github.com/apache/airflow/pull/6821 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook
Refactor
URL: https://github.com/apache/airflow/pull/6820#discussion_r357968196
##
File path: airflow/contrib/operators/redis_publish_operator.py
##
@@ -61,6 +61,6 @@ def execute(self, context):
self.log.info('Sending messsage %s to Redis on channel %s',
self.message, self.channel)
-result = redis_hook.get_conn().publish(channel=self.channel,
message=self.message)
+result = redis_hook.publish_message(channel=self.channel,
message=self.message)
Review comment:
i've changed publish_message to return int (as mentioned above) so now it
makes sense to capture result
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357967305 ## File path: airflow/contrib/hooks/redis_hook.py ## @@ -66,3 +66,47 @@ def get_conn(self): db=self.db) return self.redis + +def set_key_value(self, k, v): Review comment: hear hear, will do This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
nuclearpinguin commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357965814 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: That's interesting and I can agree. Additionally this could be a problem when used with `reschedule` mode. WDYT @mik-laj ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357964004 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It made sense when we discussed it but after playing around with Redis for a little bit I think that maybe it should stay like that. The reason for this is when you create subscription, it only reads messages published after it's created. So if message was published between two pokes it would be lost. Please refer to docstring of **get_message** method of PubSub class: > Get the **next** message if one is available, otherwise None. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357964004 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It made sense when we discussed it but after playing around with Redis for a little bit I think that maybe it should stay like that. The reason for this is when you create subscription, it only reads messages published after it's created. So if message was published between two pokes. Please refer to docstring of get_message method of redis pubsub: > Get the **next** message if one is available, otherwise None. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357964004 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It made sense when we discussed it but after playing around with Redis for a little bit I think that maybe it should stay like that. The reason for this is when you create subscription, it only reads messages published after it's created. So if message was published between two pokes it would be lost. Please refer to docstring of get_message method of redis pubsub: > Get the **next** message if one is available, otherwise None. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357964004 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It made sense when we discussed it but after playing around with Redis for a little bit I think that maybe it should stay like that. The reason for this is when you create subscription, it only reads messages published after it's created. So if message was published between two pokes, it would be lost. What do you think ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor URL: https://github.com/apache/airflow/pull/6820#discussion_r357964004 ## File path: airflow/contrib/sensors/redis_pub_sub_sensor.py ## @@ -40,7 +40,7 @@ def __init__(self, channels, redis_conn_id, *args, **kwargs): super().__init__(*args, **kwargs) self.channels = channels self.redis_conn_id = redis_conn_id -self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub() +self.pubsub = RedisHook(redis_conn_id=self.redis_conn_id).create_pubsub() Review comment: It made sense when we discussed it but playing around with Redis for a little bit it actually turns out, that maybe it should stay like that. The reason for this is when you create subscription it only reads messages published after subscription is created. So if message was published between two pokes, it would be lost. What do you think ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook Refactor
mgorsk1 commented on a change in pull request #6820: [AIRFLOW-6255] Redis Hook
Refactor
URL: https://github.com/apache/airflow/pull/6820#discussion_r357962509
##
File path: airflow/contrib/hooks/redis_hook.py
##
@@ -66,3 +66,47 @@ def get_conn(self):
db=self.db)
return self.redis
+
+def set_key_value(self, k, v):
+"""
+Create redis key ${k} with value ${v}
+"""
+client = self.get_conn()
+
+return client.set(k, v)
+
+def get_key(self, k):
+""" Get value of key ${key} stored in Redis """
+client = self.get_conn()
+
+return client.get(k)
+
+def delete_key(self, k):
+""" Delete key ${k} from redis """
+client = self.get_conn()
+
+return client.delete(k)
+
+def publish_message(self, channel, message):
+""" Publish message ${message} to channel ${channel}"""
+client = self.get_conn()
+
+client.publish(channel=channel, message=message)
Review comment:
returns int - number of subscribes that received the message. so will return
that ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
