[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6:

(1 comment)

Thank Sailesh for review. fixed now.
I will use clang-format-diff.py for formatting next time.
> Uploaded patch set 6.

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@107
PS2, Line 107:   const EVP_CIPHER* evpCipher = IsCtrSupported() ? 
EVP_aes_256_ctr() : EVP_aes_256_cfb();
> long line
thank you for review. these two issues are fixed now



--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 05:03:04 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

[Xianda Ke (Code Review)]

Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#6).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/6
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 

[Impala-ASF-CR] IMPALA-1575: part 2: yield admission control resources

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8581 )

Change subject: IMPALA-1575: part 2: yield admission control resources
..


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1498/


--
To view, visit http://gerrit.cloudera.org:8080/8581
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib1fae8dc1c4b0eca7bfa8fadae4a56ef2b37947a
Gerrit-Change-Number: 8581
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sun, 19 Nov 2017 04:31:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5053: [SECURITY] Make KRPC work with Kerberos

[Sailesh Mukil (Code Review)]

Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8270 )

Change subject: IMPALA-5053: [SECURITY] Make KRPC work with Kerberos
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8270/4/be/src/rpc/authentication.cc
File be/src/rpc/authentication.cc:

http://gerrit.cloudera.org:8080/#/c/8270/4/be/src/rpc/authentication.cc@593
PS4, Line 593: !FLAGS_principal.empty()
> What do you think about wrapping this inside an inline function ?
I wrapped it in a function, but I can't keep it inline, because inline 
functions need to be defined in the .h file (unless there's some other way to 
do it) and we can't access gflags members in the .h file.


http://gerrit.cloudera.org:8080/#/c/8270/3/be/src/util/auth-util.cc
File be/src/util/auth-util.cc:

http://gerrit.cloudera.org:8080/#/c/8270/3/be/src/util/auth-util.cc@90
PS3, Line 90: }
> It seems better to at least make sure / appears before @.
Done



--
To view, visit http://gerrit.cloudera.org:8080/8270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8cec5cca5fdb4b1d46bab19e86cb1a8a3ad718fd
Gerrit-Change-Number: 8270
Gerrit-PatchSet: 4
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Sun, 19 Nov 2017 04:14:14 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5053: [SECURITY] Make KRPC work with Kerberos

[Sailesh Mukil (Code Review)]

Hello Michael Ho,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8270

to look at the new patch set (#5).

Change subject: IMPALA-5053: [SECURITY] Make KRPC work with Kerberos
..

IMPALA-5053: [SECURITY] Make KRPC work with Kerberos

KuduRPC has support for Kerberos. However, since Impala's client transport
still uses the Thrift transport stack, we need to make sure that a single
security configuration applies to both internal communication (KuduRPC)
and external communication (Thrift's TSaslTransport).

This patch changes InitAuth() to start Sasl regardless of security
configuration, since KRPC uses plain SASL for negotiation on insecure
clusters.

It also moves some utility code out of authentication.cc into
auth-util.cc for resuse by the RpcMgr while enabling kerberos.

The MiniKDC related code is moved out of thrift-server-test.cc into a
new file called mini-kdc-wrapper.h/cc. This file exposes a new class
MiniKdcWrapper which can be easily used by the tests to configure the
kerberos environment, create the keytab, start the KDC and also
initialize the Impala security library.

Tests are added to rpc-mgr-test for kerberos tests over KRPC.
thrift-server-test also has a mechanical change to use MiniKdcWrapper.
Also tested on a live cluster configured to use kerberos.

Change-Id: I8cec5cca5fdb4b1d46bab19e86cb1a8a3ad718fd
---
M be/src/common/global-flags.cc
M be/src/rpc/CMakeLists.txt
M be/src/rpc/auth-provider.h
M be/src/rpc/authentication-test.cc
M be/src/rpc/authentication.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/thrift-server-test.cc
M be/src/testutil/CMakeLists.txt
A be/src/testutil/mini-kdc-wrapper.cc
A be/src/testutil/mini-kdc-wrapper.h
M be/src/util/auth-util.cc
M be/src/util/auth-util.h
13 files changed, 448 insertions(+), 172 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/8270/5
--
To view, visit http://gerrit.cloudera.org:8080/8270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8cec5cca5fdb4b1d46bab19e86cb1a8a3ad718fd
Gerrit-Change-Number: 8270
Gerrit-PatchSet: 5
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 

[Impala-ASF-CR] IMPALA-6217: fix DCHECK in Parquet fuzz test

[Tim Armstrong (Code Review)]

Tim Armstrong has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/8594 )

Change subject: IMPALA-6217: fix DCHECK in Parquet fuzz test
..

IMPALA-6217: fix DCHECK in Parquet fuzz test

The IMPALA-4177 change accidentally removed a Status check that could be
hit with a corrupt parquet file.

Change-Id: I6ceca7de31f602b75d744dacbdf37afa75983344
---
M be/src/exec/parquet-column-readers.cc
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/8594/2
--
To view, visit http://gerrit.cloudera.org:8080/8594
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6ceca7de31f602b75d744dacbdf37afa75983344
Gerrit-Change-Number: 8594
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 

[Impala-ASF-CR] IMPALA-6092: avoid drop/create function interactions in e2e tests

[Bharath Vissapragada (Code Review)]

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8593 )

Change subject: IMPALA-6092: avoid drop/create function interactions in e2e 
tests
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8593/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8593/1//COMMIT_MSG@10
PS1, Line 10: IMPALA-6215 explains a race between the lib_cache
Shouldn't we attempt a fix for IMPALA-6215 rather than fixing the tests? or is 
this just to unblock the builds?



--
To view, visit http://gerrit.cloudera.org:8080/8593
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ica3538788b1d2ab5e361261e2ade62780b838e65
Gerrit-Change-Number: 8593
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Comment-Date: Sat, 18 Nov 2017 20:24:46 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6092: avoid drop/create function interactions in e2e tests

[Vuk Ercegovac (Code Review)]

Vuk Ercegovac has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/8593


Change subject: IMPALA-6092: avoid drop/create function interactions in e2e 
tests
..

IMPALA-6092: avoid drop/create function interactions in e2e tests

The e2e unit tests for udfs can interact via the backend lib_cache,
causing test flakes. IMPALA-6215 explains a race between the lib_cache
and UdfExecutor in the frontend which is the likely the root cause.
Two e2e tests use the same jar (test_java_udfs and test_udf_invalid_symbol),
test_udf_invalid_symbol drops a function from that jar, which causes the use
of that jar to fail in the test_java_udfs test. Since the state of lib_cache
is per process, its state causes these interactions across unit tests.
This change avoids the interactions by using separate jars for the separate 
tests.

Change-Id: Ica3538788b1d2ab5e361261e2ade62780b838e65
---
M be/src/runtime/lib-cache.h
M fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java
M tests/query_test/test_udfs.py
3 files changed, 12 insertions(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/8593/1
--
To view, visit http://gerrit.cloudera.org:8080/8593
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ica3538788b1d2ab5e361261e2ade62780b838e65
Gerrit-Change-Number: 8593
Gerrit-PatchSet: 1
Gerrit-Owner: Vuk Ercegovac 

[Impala-ASF-CR](asf-site) Remove outdated incubator reference from bylaws.

[Jim Apple (Code Review)]

Jim Apple has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/8592


Change subject: Remove outdated incubator reference from bylaws.
..

Remove outdated incubator reference from bylaws.

This must get lazy consensus from PMC members before being committed.

Change-Id: I3268be8cc4103e2548287020560a3292c8f124c3
---
M bylaws.html
1 file changed, 3 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/8592/1
--
To view, visit http://gerrit.cloudera.org:8080/8592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: asf-site
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3268be8cc4103e2548287020560a3292c8f124c3
Gerrit-Change-Number: 8592
Gerrit-PatchSet: 1
Gerrit-Owner: Jim Apple 

[Impala-ASF-CR] IMPALA-5976: Remove equivalence class computation in FE

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8317 )

Change subject: IMPALA-5976: Remove equivalence class computation in FE
..


Patch Set 9: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/8317
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4cb1d8be46efa8fd61a97048cc79dabe2ffa51a
Gerrit-Change-Number: 8317
Gerrit-PatchSet: 9
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Sat, 18 Nov 2017 09:07:05 +
Gerrit-HasComments: No