[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

2017-12-27 Thread Bikramjeet Vig (Code Review) 
Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8861 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..


Patch Set 2:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@7
PS2, Line 7: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
nit, how about this?:
Add support for AES-CTR encryption when spilling to disk


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@8
PS2, Line 8:
nit:
is a stream


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@8
PS2, Line 8: use
nit: used with a


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@9
PS2, Line 9: would
nit: can


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@14
PS2, Line 14: CTR mode is used if OpenSSL version>=1.0.1 at runtime, otherwise
nit,add:
With this patch


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@15
PS2, Line 15:
nit: using


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@18
PS2, Line 18: run runtime tmp-file-mgr-test, openssl-util-test, 
buffer-pool-test and buffered-tuple-stream-test
nit: long line, wrap around after 72 characters.


http://gerrit.cloudera.org:8080/#/c/8861/2//COMMIT_MSG@19
PS2, Line 19:
please also mention that you added a test that tests encryption in both modes


http://gerrit.cloudera.org:8080/#/c/8861/2/be/src/util/openssl-util.h
File be/src/util/openssl-util.h:

http://gerrit.cloudera.org:8080/#/c/8861/2/be/src/util/openssl-util.h@59
PS2, Line 59: CTR/CFB
maybe add a line explaining when each mode is used.


http://gerrit.cloudera.org:8080/#/c/8861/2/be/src/util/openssl-util.h@89
PS2, Line 89: Maybe
nit, add:
Currently used only for testing but


http://gerrit.cloudera.org:8080/#/c/8861/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8861/2/be/src/util/openssl-util.cc@104
PS2, Line 104:
nit:
either CTR or CFB(stream cipher), both of which support arbitrary length 
ciphertexts - it doesn't have to be a multiple of 16 bytes. Additionally, CTR 
mode is well-optimized(instruction level parallelism) with hardware 
acceleration on x86 and PowerPC.



--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Thu, 28 Dec 2017 00:53:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

2017-12-24 Thread Xianda Ke (Code Review) 
Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8861 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..


Patch Set 2:

Thanks for you comments.  Merry Xmax!


--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 24 Dec 2017 14:45:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

2017-12-23 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8861 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..


Patch Set 2:

Sailesh and I are both AFK until January but Bikram might be able to look in 
the meantime. IIRC the two changes were that we wanted test coverage for both 
code paths, and we wanted to make sure it worked with older OpenSSL versions 
(e.g. on CentOs 6)


--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Sun, 24 Dec 2017 03:33:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

2017-12-23 Thread Tim Armstrong (Code Review) 
Tim Armstrong has removed Bharath Vissapragada from this change.  ( 
http://gerrit.cloudera.org:8080/8861 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..


Removed reviewer Bharath Vissapragada.
--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow CFB mode is stream cipher and is secure when use different nonce/IV for every message. However it would be a performance

2017-12-18 Thread Xianda Ke (Code Review) 
Xianda Ke has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/8861


Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow 
CFB mode is stream cipher and is secure when use different nonce/IV for every 
message. However it would be a performance bottleneck. CTR mode is also stream 
cipher and is secure, 4~6x faster
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL. AES-CTR+SHA256 is about 40~70% faster
than AES-CFB+SHA256.

CTR mode is used if OpenSSL version>=1.0.1 at runtime, otherwise
fall back to CFB mode.

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util-test.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
4 files changed, 70 insertions(+), 24 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/8861/2
--
To view, visit http://gerrit.cloudera.org:8080/8861
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9debc240615dd8cdbf00ec8730cff62ffef52aff
Gerrit-Change-Number: 8861
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-20 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8510/7/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8510/7/be/src/util/openssl-util.cc@147
PS7, Line 147:   return (SSLeay() >= OPENSSL_VERSION_1_0_1);
When we re-do this patch we should also add a test option to force use of CFB 
mode and make sure that the unit test tests both encryption modes.



--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Mon, 20 Nov 2017 17:57:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-19 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 7:

> Change has been successfully cherry-picked as 
> fb4c3b01240d8f65fc2c45bf27b668ae9b1fa5d2
 > by Impala Public Jenkins

Thanks for the patch Xianda. The patch has been merged.


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 22:30:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Reviewed-on: http://gerrit.cloudera.org:8080/8510
Reviewed-by: Sailesh Mukil 
Tested-by: Impala Public Jenkins
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)

Approvals:
  Sailesh Mukil: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 7
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 22:21:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-19 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1499/


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 18:53:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-19 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 18:52:40 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-18 Thread Xianda Ke (Code Review) 
Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 6:

(1 comment)

Thank Sailesh for review. fixed now.
I will use clang-format-diff.py for formatting next time.
> Uploaded patch set 6.

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8510/2/be/src/util/openssl-util.cc@107
PS2, Line 107:   const EVP_CIPHER* evpCipher = IsCtrSupported() ? 
EVP_aes_256_ctr() : EVP_aes_256_cfb();
> long line
thank you for review. these two issues are fixed now



--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sun, 19 Nov 2017 05:03:04 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-18 Thread Xianda Ke (Code Review) 
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#6).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/6
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 6
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-18 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.h
File be/src/util/openssl-util.h:

http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.h@86
PS5, Line 86:
nit:whitespace


http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.cc
File be/src/util/openssl-util.cc:

http://gerrit.cloudera.org:8080/#/c/8510/5/be/src/util/openssl-util.cc@109
PS5, Line 109: EVP_DecryptInit_ex(&ctx, evpCipher, NULL, key_, iv_);
nit: formatting



--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 5
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Sat, 18 Nov 2017 17:42:46 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-18 Thread Xianda Ke (Code Review) 
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#5).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 22 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/5
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 5
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-16 Thread Sailesh Mukil (Code Review) 
Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 4:

> > Thanks for doing this. It seems reasonable to me to switch to
 > CTR.
 > >
 > > One concern I have is were you able to test this on a system with
 > > OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure
 > if
 > > this API is supported in that version.
 > > The best I could see was this line of code that was under an #if
 > 0:
 > > https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
 > >
 > > It would be great if you could verify that.
 >
 > Thank Sailesh for your comments. aes_256_ctr was supported since
 > openssl 1.0.1. Luckily, Impala add openssl to toolchain and upgrade
 > to version 1.0.1p (see IMPALA-5714)

Xianda, we don't statically link against OpenSSL. So the version you see in the 
toolchain is just what we build against, but we may be running against an older 
version of OpenSSL.

Impala supports OpenSSL 1.0.0 onwards, so we wouldn't want this to break for 
users that are still running against that version.

One way to fix this would be to use SSLeay() to check  the runtime version of 
OpenSSL, like we do here, and use  CTR mode or CFB mode accordingly:

https://github.com/apache/incubator-impala/blob/1ca3adf46c5ef5055c13fd3ce57e7c53218c219c/be/src/rpc/thrift-server.cc#L92

It isn't ideal to add more runtime version checks in the code, but this would 
be nice to have for the majority of the users that run on newer OpenSSL 
versions.


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Fri, 17 Nov 2017 00:15:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-16 Thread Tim Armstrong (Code Review) 
Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 4: Code-Review+1

Change makes sense to me. I don't know the story with what OpenSSL versions we 
have to interoperate with, so I'll let Sailesh weigh in on that.


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Thu, 16 Nov 2017 23:55:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-16 Thread Xianda Ke (Code Review) 
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#4).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 12 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/4
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 4
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-16 Thread Xianda Ke (Code Review) 
Xianda Ke has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8510 )

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..


Patch Set 2:

> Thanks for doing this. It seems reasonable to me to switch to CTR.
 >
 > One concern I have is were you able to test this on a system with
 > OpenSSL 1.0.0? Their documentation isn't clear, so I'm not sure if
 > this API is supported in that version.
 > The best I could see was this line of code that was under an #if 0:
 > https://github.com/openssl/openssl/blob/OpenSSL_1_0_0-stable/crypto/evp/evp.h#L782-L783
 >
 > It would be great if you could verify that.

Thank Sailesh for your comments. aes_256_ctr was supported since openssl 1.0.1. 
Luckily, Impala add openssl to toolchain and upgrade to version 1.0.1p (see 
IMPALA-5714)


--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Thu, 16 Nov 2017 14:55:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

2017-11-16 Thread Xianda Ke (Code Review) 
Hello Jim Apple, Mike Yoder, Sailesh Mukil, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8510

to look at the new patch set (#3).

Change subject: IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow
..

IMPALA-6128: Spill-to-disk Encryption(AES-CFB + SHA256) is slow

CFB mode is stream cipher and is secure when use different nonce/IV
for every message. However it would be a performance bottleneck.
CTR mode is also stream cipher and is secure, 4~6x faster
than CFB mode in OpenSSL.

AES-CTR+SHA256 is about 40~70% faster than AES-CFB+SHA256

Testing:
run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and 
buffered-tuple-stream-test

Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
---
M be/src/runtime/tmp-file-mgr.cc
M be/src/util/openssl-util.cc
M be/src/util/openssl-util.h
3 files changed, 11 insertions(+), 11 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/8510/3
--
To view, visit http://gerrit.cloudera.org:8080/8510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib97939f2334838263364b53ef3413871638bf53e
Gerrit-Change-Number: 8510
Gerrit-PatchSet: 3
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Mike Yoder 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tim Armstrong