[kudu-CR] [ranger] update description of Ranger integration related flags
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15670 Change subject: [ranger] update description of Ranger integration related flags .. [ranger] update description of Ranger integration related flags This patch updates the description of Ranger related flags to make it more clear how to enable Ranger integration. Change-Id: If782f022ada6606b31d720f1dba793af45e8003e --- M src/kudu/ranger/ranger_client.cc 1 file changed, 15 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/15670/1 -- To view, visit http://gerrit.cloudera.org:8080/15670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If782f022ada6606b31d720f1dba793af45e8003e Gerrit-Change-Number: 15670 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao
[kudu-CR] util: remove duplicate results from DNS resolution
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15665 ) Change subject: util: remove duplicate results from DNS resolution .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG@13 PS1, Line 13: $ kudu table list localhost : W0404 20:35:05.511526 31378 client-internal.cc:597] Specified master : server address 'localhost' resolved to multiple IPs. Using : 127.0.0.1:7051 > seems to be reported here: https://bugzilla.redhat.com/show_bug.cgi?id=4963 thank you for the reference -- To view, visit http://gerrit.cloudera.org:8080/15665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 Gerrit-Change-Number: 15665 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 07 Apr 2020 04:46:24 + Gerrit-HasComments: Yes
[kudu-CR] [python] KUDU-2632 Add DATE type support
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 5: Code-Review+2 Looks good to me, but maybe other people have more feedback. -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 5 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Tue, 07 Apr 2020 04:32:49 + Gerrit-HasComments: No
[kudu-CR] util: remove duplicate results from DNS resolution
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15665 ) Change subject: util: remove duplicate results from DNS resolution .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG@13 PS1, Line 13: $ kudu table list localhost : W0404 20:35:05.511526 31378 client-internal.cc:597] Specified master : server address 'localhost' resolved to multiple IPs. Using : 127.0.0.1:7051 > Maybe, that was due to multiple entries in /etc/hosts? seems to be reported here: https://bugzilla.redhat.com/show_bug.cgi?id=496300 with some discussion about whether it's really a bug -- To view, visit http://gerrit.cloudera.org:8080/15665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 Gerrit-Change-Number: 15665 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 07 Apr 2020 04:12:09 + Gerrit-HasComments: Yes
[kudu-CR] allow skip block manager in some ops of local replica tools
wangning has posted comments on this change. ( http://gerrit.cloudera.org:8080/15656 ) Change subject: allow skip block manager in some ops of local_replica tools .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc File src/kudu/tools/tool_action_local_replica.cc: http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc@158 PS2, Line 158: bool skip_block_manager, unique_ptr* fs_manager > nit: per the guidance of the Google style guide, could you reverse the orde Thx for point out, I will read about it http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc@242 PS2, Line 242: (/*skip_block_manag > nit: for boolean arguments, to improve readability, would you mind inline-a Done -- To view, visit http://gerrit.cloudera.org:8080/15656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I01e22354fdb76596008cd0824e240d24a8f20099 Gerrit-Change-Number: 15656 Gerrit-PatchSet: 3 Gerrit-Owner: wangning <1994wangn...@gmail.com> Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: wangning <1994wangn...@gmail.com> Gerrit-Comment-Date: Tue, 07 Apr 2020 02:12:19 + Gerrit-HasComments: Yes
[kudu-CR] allow skip block manager in some ops of local replica tools
Hello Kudu Jenkins, Andrew Wong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15656 to look at the new patch set (#3). Change subject: allow skip block manager in some ops of local_replica tools .. allow skip block manager in some ops of local_replica tools These cli ops could be speed up via skip opening block manager. - local_replica cmeta print_replica_uuids - local_replica cmeta rewrite_raft_config - local_replica cmeta set_term - local_replica dump wals - local_replica list Change-Id: I01e22354fdb76596008cd0824e240d24a8f20099 --- M src/kudu/tools/tool_action_local_replica.cc 1 file changed, 13 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/56/15656/3 -- To view, visit http://gerrit.cloudera.org:8080/15656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I01e22354fdb76596008cd0824e240d24a8f20099 Gerrit-Change-Number: 15656 Gerrit-PatchSet: 3 Gerrit-Owner: wangning <1994wangn...@gmail.com> Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15657 ) Change subject: [master] KUDU-2798 fix logging on deleted TSK entries .. [master] KUDU-2798 fix logging on deleted TSK entries Decode the identifiers of the deleted TSK entries while logging. Prior to this patch, the raw value of the system catalog's 'entry_id' column was written into the log. Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Reviewed-on: http://gerrit.cloudera.org:8080/15657 Reviewed-by: Adar Dembo Tested-by: Kudu Jenkins --- M src/kudu/master/catalog_manager.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h 3 files changed, 23 insertions(+), 4 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Gerrit-Change-Number: 15657 Gerrit-PatchSet: 4 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Volodymyr Verovkin
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15657 ) Change subject: [master] KUDU-2798 fix logging on deleted TSK entries .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Gerrit-Change-Number: 15657 Gerrit-PatchSet: 3 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Tue, 07 Apr 2020 01:04:02 + Gerrit-HasComments: No
[kudu-CR] [python] KUDU-2632 Add DATE type support
Volodymyr Verovkin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py File python/kudu/util.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@197 PS3, Line 197: nday > nit: ndays Done -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 5 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Tue, 07 Apr 2020 01:01:54 + Gerrit-HasComments: Yes
[kudu-CR] [python] KUDU-2632 Add DATE type support
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15645 to look at the new patch set (#5). Change subject: [python] KUDU-2632 Add DATE type support .. [python] KUDU-2632 Add DATE type support There are two utility functions which perfrom conversion between Kudu DATE and Python datetime.date: unix_epoch_days_to_date() - converts number of days since Unix ecpoch to datetime.date() date_to_unix_epoch_days() - converts datetime.date() to number of days since Unix ecpoch Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 --- M python/kudu/__init__.py M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M python/kudu/tests/test_scanner.py M python/kudu/tests/test_scantoken.py M python/kudu/tests/test_schema.py M python/kudu/tests/util.py M python/kudu/util.py 9 files changed, 113 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/15645/5 -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 5 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin
[kudu-CR] [python] KUDU-2632 Add DATE type support
Volodymyr Verovkin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/tests/test_scanner.py File python/kudu/tests/test_scanner.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/tests/test_scanner.py@a365 PS3, Line 365: > What happened to types[8] here? Should the change to be to add types[9]? Done http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py File python/kudu/util.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@201 PS3, Line 201: dat > nit: date Done http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@208 PS3, Line 208: dat > here too Done -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 4 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Tue, 07 Apr 2020 01:00:32 + Gerrit-HasComments: Yes
[kudu-CR] [python] KUDU-2632 Add DATE type support
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15645 to look at the new patch set (#4). Change subject: [python] KUDU-2632 Add DATE type support .. [python] KUDU-2632 Add DATE type support There are two utility functions which perfrom conversion between Kudu DATE and Python datetime.date: unix_epoch_days_to_date() - converts number of days since Unix ecpoch to datetime.date() date_to_unix_epoch_days() - converts datetime.date() to number of days since Unix ecpoch Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 --- M python/kudu/__init__.py M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M python/kudu/tests/test_scanner.py M python/kudu/tests/test_scantoken.py M python/kudu/tests/test_schema.py M python/kudu/tests/util.py M python/kudu/util.py 9 files changed, 113 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/15645/4 -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 4 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin
[kudu-CR] cfile: clean up encoding-test to use fewer templates
Bankim Bhavsar has uploaded a new patch set (#3) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/15044 ) Change subject: cfile: clean up encoding-test to use fewer templates .. cfile: clean up encoding-test to use fewer templates This test made heavy use of templates, which made things overly complicated and hard to follow. All of the block builders/decoders already implement common interfaces, so we can use runtime polymorphism instead for the majority of code here. Change-Id: Iba4464c2ea41107df96c68ea61576a0ea269277a --- M src/kudu/cfile/cfile_reader.cc M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/encoding-test.cc M src/kudu/cfile/type_encodings.cc M src/kudu/cfile/type_encodings.h 5 files changed, 266 insertions(+), 316 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/44/15044/3 -- To view, visit http://gerrit.cloudera.org:8080/15044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iba4464c2ea41107df96c68ea61576a0ea269277a Gerrit-Change-Number: 15044 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] cfile: change BlockBuilder API to yield a vector of Slices
Bankim Bhavsar has uploaded a new patch set (#6) to the change originally created by Todd Lipcon. ( http://gerrit.cloudera.org:8080/15042 ) Change subject: cfile: change BlockBuilder API to yield a vector of Slices .. cfile: change BlockBuilder API to yield a vector of Slices When blocks are appended to cfiles at the IO layer, we already have the ability to write multiple slices using a vectored IO. Prior to this patch, the BlockBuilder API was restricted to returning a single slice, whereas it would be more convenient in some cases to be able to return multiple slices (eg separating the header from the data). This new functionality is used by BinaryDictBlockBuilder to avoid an extra copy in Finish(). Change-Id: Ifc7a5f148a4a43cedac2428f4c1a18d0f93a10db --- M src/kudu/cfile/binary_dict_block.cc M src/kudu/cfile/binary_dict_block.h M src/kudu/cfile/binary_plain_block.cc M src/kudu/cfile/binary_plain_block.h M src/kudu/cfile/binary_prefix_block.cc M src/kudu/cfile/binary_prefix_block.h M src/kudu/cfile/block_encodings.h M src/kudu/cfile/bshuf_block.cc M src/kudu/cfile/bshuf_block.h M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/encoding-test.cc M src/kudu/cfile/plain_bitmap_block.h M src/kudu/cfile/plain_block.h M src/kudu/cfile/rle_block.h 14 files changed, 116 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/42/15042/6 -- To view, visit http://gerrit.cloudera.org:8080/15042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifc7a5f148a4a43cedac2428f4c1a18d0f93a10db Gerrit-Change-Number: 15042 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon
[kudu-CR] cfile: clean up encoding-test to use fewer templates
Bankim Bhavsar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15044 ) Change subject: cfile: clean up encoding-test to use fewer templates .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15044/1/src/kudu/cfile/type_encodings.h File src/kudu/cfile/type_encodings.h: http://gerrit.cloudera.org:8080/#/c/15044/1/src/kudu/cfile/type_encodings.h@46 PS1, Line 46: > nit: while you are here, maybe stick * and & to the type in the changes lin Done -- To view, visit http://gerrit.cloudera.org:8080/15044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4464c2ea41107df96c68ea61576a0ea269277a Gerrit-Change-Number: 15044 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Tue, 07 Apr 2020 00:55:41 + Gerrit-HasComments: Yes
[kudu-CR] [python] KUDU-2632 Add DATE type support
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py File python/kudu/tests/util.py: http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py@112 PS1, Line 112: datetime.date(2020,1,1) > This could be done in a follow up patch. +1 for doing this in a follow-up patch. http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py File python/kudu/util.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@197 PS3, Line 197: nday nit: ndays -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Tue, 07 Apr 2020 00:52:21 + Gerrit-HasComments: Yes
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Alexey Serbin has posted comments on this change. (
http://gerrit.cloudera.org:8080/15657 )
Change subject: [master] KUDU-2798 fix logging on deleted TSK entries
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/catalog_manager.cc@4592
PS2, Line 4592: for (const auto& entry_id : entry_ids) {
: msg += Substitute(" $0",
SysCatalogTable::TskEntryIdToSeqNumber(entry_id));
: }
> JoinMapped() is a pretty handy way to do loop transformations like these.
Done
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/sys_catalog.cc@636
PS2, Line 636: false, nullptr,
> Nit: annotate with comments what these mean?
Done
--
To view, visit http://gerrit.cloudera.org:8080/15657
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4
Gerrit-Change-Number: 15657
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Alexey Serbin
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Volodymyr Verovkin
Gerrit-Comment-Date: Tue, 07 Apr 2020 00:45:46 +
Gerrit-HasComments: Yes
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, Volodymyr Verovkin, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15657 to look at the new patch set (#3). Change subject: [master] KUDU-2798 fix logging on deleted TSK entries .. [master] KUDU-2798 fix logging on deleted TSK entries Decode the identifiers of the deleted TSK entries while logging. Prior to this patch, the raw value of the system catalog's 'entry_id' column was written into the log. Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 --- M src/kudu/master/catalog_manager.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h 3 files changed, 23 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/15657/3 -- To view, visit http://gerrit.cloudera.org:8080/15657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Gerrit-Change-Number: 15657 Gerrit-PatchSet: 3 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Volodymyr Verovkin
[kudu-CR] [python] KUDU-2632 Add DATE type support
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/tests/test_scanner.py File python/kudu/tests/test_scanner.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/tests/test_scanner.py@a365 PS3, Line 365: What happened to types[8] here? Should the change to be to add types[9]? http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py File python/kudu/tests/util.py: http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py@112 PS1, Line 112: datetime.date(2020,1,1) > I think for this kind of testing we just need to write tests for util funct This could be done in a follow up patch. http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py File python/kudu/util.py: http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@201 PS3, Line 201: dat nit: date http://gerrit.cloudera.org:8080/#/c/15645/3/python/kudu/util.py@208 PS3, Line 208: dat here too -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Mon, 06 Apr 2020 23:24:38 + Gerrit-HasComments: Yes
[kudu-CR] util: remove duplicate results from DNS resolution
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15665 ) Change subject: util: remove duplicate results from DNS resolution .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG@13 PS1, Line 13: $ kudu table list localhost : W0404 20:35:05.511526 31378 client-internal.cc:597] Specified master : server address 'localhost' resolved to multiple IPs. Using : 127.0.0.1:7051 > I've definitely seen this before and always wondered what caused it. Is it Maybe, that was due to multiple entries in /etc/hosts? -- To view, visit http://gerrit.cloudera.org:8080/15665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 Gerrit-Change-Number: 15665 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 23:19:23 + Gerrit-HasComments: Yes
[kudu-CR] [docker] Fix the docker thirdparty build
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15659 ) Change subject: [docker] Fix the docker thirdparty build .. [docker] Fix the docker thirdparty build After 458e6ce, the docker thirdparty build was failing because `/build-support/ccache-clang` was not available in the thirdparty image. I tested this fix by running builds based on ubuntu:xenial, centos:6, and centos:7 base images. Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed Reviewed-on: http://gerrit.cloudera.org:8080/15659 Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong --- M docker/Dockerfile 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Kudu Jenkins: Verified Andrew Wong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed Gerrit-Change-Number: 15659 Gerrit-PatchSet: 2 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [security] KUDU-3106 update on getEndpointChannelBindings()
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15664 ) Change subject: [security] KUDU-3106 update on getEndpointChannelBindings() .. [security] KUDU-3106 update on getEndpointChannelBindings() Fixed an issue with SecurityUtil::getEndpointChannelBindings() when Java security provider converts the certificate signature algorithm's name to the upper case. At least, the upper-case conversion behavior was observed when using BouncyCastle 1.65. Change-Id: Ia788d6d72cfee7958e9db7293740aa22ede8fabd Reviewed-on: http://gerrit.cloudera.org:8080/15664 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Kudu Jenkins: Verified Grant Henke: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia788d6d72cfee7958e9db7293740aa22ede8fabd Gerrit-Change-Number: 15664 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin
[kudu-CR] util: remove duplicate results from DNS resolution
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15665 ) Change subject: util: remove duplicate results from DNS resolution .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15665/1//COMMIT_MSG@13 PS1, Line 13: $ kudu table list localhost : W0404 20:35:05.511526 31378 client-internal.cc:597] Specified master : server address 'localhost' resolved to multiple IPs. Using : 127.0.0.1:7051 I've definitely seen this before and always wondered what caused it. Is it due to poor DNS configuration? A bug in glibc? Something else entirely? -- To view, visit http://gerrit.cloudera.org:8080/15665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 Gerrit-Change-Number: 15665 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 22:49:08 + Gerrit-HasComments: Yes
[kudu-CR] [security] KUDU-3106 update on getEndpointChannelBindings()
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15664 ) Change subject: [security] KUDU-3106 update on getEndpointChannelBindings() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia788d6d72cfee7958e9db7293740aa22ede8fabd Gerrit-Change-Number: 15664 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Mon, 06 Apr 2020 22:43:34 + Gerrit-HasComments: No
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Adar Dembo has posted comments on this change. (
http://gerrit.cloudera.org:8080/15657 )
Change subject: [master] KUDU-2798 fix logging on deleted TSK entries
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/catalog_manager.cc@4592
PS2, Line 4592: for (const auto& entry_id : entry_ids) {
: msg += Substitute(" $0",
SysCatalogTable::TskEntryIdToSeqNumber(entry_id));
: }
JoinMapped() is a pretty handy way to do loop transformations like these.
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/sys_catalog.cc
File src/kudu/master/sys_catalog.cc:
http://gerrit.cloudera.org:8080/#/c/15657/2/src/kudu/master/sys_catalog.cc@636
PS2, Line 636: false, nullptr,
Nit: annotate with comments what these mean?
--
To view, visit http://gerrit.cloudera.org:8080/15657
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4
Gerrit-Change-Number: 15657
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin
Gerrit-Reviewer: Adar Dembo
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Volodymyr Verovkin
Gerrit-Comment-Date: Mon, 06 Apr 2020 22:37:19 +
Gerrit-HasComments: Yes
[kudu-CR] util: remove duplicate results from DNS resolution
Hello Alexey Serbin, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15665 to review the following change. Change subject: util: remove duplicate results from DNS resolution .. util: remove duplicate results from DNS resolution On some systems it seems that our DNS resolution code can end up yielding multiple copies of the same address. That would produce annoying log messages like: $ kudu table list localhost W0404 20:35:05.511526 31378 client-internal.cc:597] Specified master server address 'localhost' resolved to multiple IPs. Using 127.0.0.1:7051 This patch ensures that any given address is only appended to the result vector once. Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 --- M src/kudu/util/net/net_util.cc 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/65/15665/1 -- To view, visit http://gerrit.cloudera.org:8080/15665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7d9b9f9839a899d8022f5ac6496555ff84583192 Gerrit-Change-Number: 15665 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15660 ) Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. Patch Set 3: (7 comments) http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/client/columnar_scan_batch.h File src/kudu/client/columnar_scan_batch.h: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/client/columnar_scan_batch.h@45 PS3, Line 45: but without the alignment and padding guarantees that are made by : /// the Arrow IPC serialization. nit: just to be sure, is this to say that the guarantees are only important for data transfer? http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/client/scanner-internal.cc File src/kudu/client/scanner-internal.cc: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/client/scanner-internal.cc@882 PS3, Line 882: resp_data_.columns(idx) nit: why not use `col`? http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/columnar_serialization.cc File src/kudu/common/columnar_serialization.cc: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/columnar_serialization.cc@513 PS3, Line 513: total_size nit: seems a little odd to call this the total size, given old_size + total_size will be used to get the total size of the new buffer. Maybe added_offsets_size? additional_size? http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/columnar_serialization.cc@615 PS3, Line 615: SelectedRows sel = block.selection_vector()->GetSelectedRows(); Can we short circuit if we selected 0 rows in this row block? If so, maybe add a DCHECK to CopySelectedVarlenCellsFromColumn that we selected some rows. http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/rowblock.h File src/kudu/common/rowblock.h: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/rowblock.h@265 PS3, Line 265: num_selected nit: could use sel_vector_->nrows() to avoid the extra all_selected_ evaluation http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/wire_protocol-test.cc File src/kudu/common/wire_protocol-test.cc: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/wire_protocol-test.cc@341 PS3, Line 341: sizeof(uint32_t)*dst_row_idx nit: spacing http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/wire_protocol.proto File src/kudu/common/wire_protocol.proto: http://gerrit.cloudera.org:8080/#/c/15660/3/src/kudu/common/wire_protocol.proto@161 PS3, Line 161: num_rows+1 nit: for consistency with elsewhere in this patch, separate these with spaces? -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 06 Apr 2020 21:39:17 + Gerrit-HasComments: Yes
[kudu-CR] [security] KUDU-3106 update on getEndpointChannelBindings()
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15664 Change subject: [security] KUDU-3106 update on getEndpointChannelBindings() .. [security] KUDU-3106 update on getEndpointChannelBindings() Fixed an issue with SecurityUtil::getEndpointChannelBindings() when Java security provider converts the certificate signature algorithm's name to the upper case. At least, the upper-case conversion behavior was observed when using BouncyCastle 1.65. Change-Id: Ia788d6d72cfee7958e9db7293740aa22ede8fabd --- M java/kudu-client/src/main/java/org/apache/kudu/util/SecurityUtil.java 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/64/15664/1 -- To view, visit http://gerrit.cloudera.org:8080/15664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia788d6d72cfee7958e9db7293740aa22ede8fabd Gerrit-Change-Number: 15664 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin
[kudu-CR] WIP: support for passing arrow data to python
Hello Tidy Bot, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15661 to look at the new patch set (#2). Change subject: WIP: support for passing arrow data to python .. WIP: support for passing arrow data to python This adds some support to pass arrow-formatted data to pyarrow. WIP because it needs some cleanup, and also need to figure out how to test this, given it depends on pyarrow nightly builds. The required feature should be in the next pyarrow release due in a "couple weeks" so will probably hold off until then. Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed --- M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M src/kudu/client/CMakeLists.txt A src/kudu/client/arrow-internal.h A src/kudu/client/arrow.h M src/kudu/client/client-test.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/util/memory/arena.h 16 files changed, 399 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/15661/2 -- To view, visit http://gerrit.cloudera.org:8080/15661 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed Gerrit-Change-Number: 15661 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15660 ) Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG@10 PS1, Line 10: match Apache Arrow > well, a later commit in this series will use pyarrow to pass these batches Oh yeah, that works. So long as we have coverage in the repo. We might get coverage from Spark/Java support as well. -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 06 Apr 2020 19:55:27 + Gerrit-HasComments: Yes
[kudu-CR] WIP: support for passing arrow data to python
Hello Grant Henke, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15661 to review the following change. Change subject: WIP: support for passing arrow data to python .. WIP: support for passing arrow data to python This adds some support to pass arrow-formatted data to pyarrow. WIP because it needs some cleanup, and also need to figure out how to test this, given it depends on pyarrow nightly builds. The required feature should be in the next pyarrow release due in a "couple weeks" so will probably hold off until then. Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed --- M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M src/kudu/client/CMakeLists.txt M src/kudu/client/client-test.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/util/memory/arena.h 14 files changed, 254 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/15661/1 -- To view, visit http://gerrit.cloudera.org:8080/15661 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5df4714bf5ac339d675f2a9169f8aeff06d30eed Gerrit-Change-Number: 15661 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Grant Henke
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15660 ) Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h File src/kudu/client/columnar_scan_batch.h: http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h@67 PS1, Line 67: Status GetFixedLengthColumn(int idx, Slice* data) const; > hm I suppose we could add a bunch of per-type APIs but since this is a pret Done http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/common/columnar_serialization.h File src/kudu/common/columnar_serialization.h: http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/common/columnar_serialization.h@38 PS1, Line 38: BINARY > nit: maybe be explicit about binary physical type. Done -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 06 Apr 2020 19:45:19 + Gerrit-HasComments: Yes
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Hello Andrew Wong, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15660 to look at the new patch set (#3). Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. wire_protocol: change columnar serialization of varlen data to match Arrow This changes the format of variable-length columns serialized on the wire to match Apache Arrow instead of our internal column format. The Arrow format consists of an array of n+1 offsets for n rows, such that the data for cell 'n' spans offset offsets[n]...offsets[n+1]. The obvious advantage here is that clients can zero-copy into Arrow structures since the format is compatible. The less obvious advantage is that we are going from 16 bytes (sizeof(Slice)) to 4 bytes (sizeof(uint32_t offset)) for each serialized string, so this should be a savings even for non-arrow users of the API. This patch also adds some more sanity checking of the wire format in the client API so that it's not the responsibility of the caller to guard against malicious servers. Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 --- M src/kudu/client/client-test.cc M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/common/columnar_serialization.cc M src/kudu/common/columnar_serialization.h M src/kudu/common/rowblock.h M src/kudu/common/wire_protocol-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc 12 files changed, 245 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/15660/3 -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/15660 )
Change subject: wire_protocol: change columnar serialization of varlen data to
match Arrow
..
Patch Set 2:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG@10
PS1, Line 10: match Apache Arrow
> I think it might be important to add an integration test that verifies we m
well, a later commit in this series will use pyarrow to pass these batches
there, so I think end-to-end testing should cover it, right? I'd rather avoid
adding arrow itself to thirdparty since it's a big build with its own
thirdparty stuff
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h
File src/kudu/client/columnar_scan_batch.h:
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h@45
PS1, Line 45: but with no alignment or padding guarantees
> Are there potential issues by other arrow integrations based on this differ
I clarified with Wes that the C interface doesn't specify padding/alignment
(the serializatoin format does). I'll ask on the dev list to get that added to
the spec for the C interface.
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h@67
PS1, Line 67: Status GetFixedLengthColumn(int idx, Slice* data) const;
> Do you think we should use an API similar to the row based API where we hav
hm I suppose we could add a bunch of per-type APIs but since this is a pretty
"raw" interface I had decided not to, and given we don't currently provide
alignment guarantees it would be wrong to have out-params like 'uint32_t**'.
I'll improve docs.
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc
File src/kudu/client/scanner-internal.cc:
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@815
PS1, Line 815: return Status::InvalidArgument("column is variable-length",
col.ToString());
> nit: Maybe add the type info here too.
col.ToString() includes it
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@820
PS1, Line 820: return Status::Corruption("server did not send data for
column", col.ToString());
> Is there a case where this is expected to happen or is this just being defe
defensive
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@840
PS1, Line 840: return Status::InvalidArgument("column is not
variable-length", col.ToString());
> nit: Maybe add the type info here too.
see baove
--
To view, visit http://gerrit.cloudera.org:8080/15660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467
Gerrit-Change-Number: 15660
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 06 Apr 2020 19:38:29 +
Gerrit-HasComments: Yes
[kudu-CR] [docker] Fix the docker thirdparty build
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15659 ) Change subject: [docker] Fix the docker thirdparty build .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed Gerrit-Change-Number: 15659 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 19:35:50 + Gerrit-HasComments: No
[kudu-CR] [docker] Fix the docker thirdparty build
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15659 ) Change subject: [docker] Fix the docker thirdparty build .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/15659/1/docker/Dockerfile File docker/Dockerfile: http://gerrit.cloudera.org:8080/#/c/15659/1/docker/Dockerfile@108 PS1, Line 108: COPY ./build-support/ccache-clang build-support/ccache-clang : COPY ./build-support/ccache-devtoolset-3 build-support/ccache-devtoolset-3 Just curious, would it also have worked to merge this with L105..107? -- To view, visit http://gerrit.cloudera.org:8080/15659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed Gerrit-Change-Number: 15659 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 19:32:48 + Gerrit-HasComments: Yes
[kudu-CR] KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15638 ) Change subject: KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore .. KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore The System.exit() calls have a side effect that can cause Spark to fail even if the run function returns 0 on success. Rather than call System.exit() the run() method will return true on a successful run. We then throw a RuntimeException() in main if we find that run() failed, otherwise we call SparkSession's stop() method to cleanly shutdown Spark. Unfortunately the issue isn't easy to reproduce but we had one environment exhibiting the problem and we confirmed that this patch fixes the issue. TestKuduBackup.scala was modified where assertFalse() is used to check for failure and assertTrue() for success. Change-Id: I7d1b4796b6280adecd7dab685a0281af6b2570ce Reviewed-on: http://gerrit.cloudera.org:8080/15638 Tested-by: Grant Henke Reviewed-by: Grant Henke --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 3 files changed, 37 insertions(+), 33 deletions(-) Approvals: Grant Henke: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7d1b4796b6280adecd7dab685a0281af6b2570ce Gerrit-Change-Number: 15638 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15638 ) Change subject: KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore .. Patch Set 3: Code-Review+2 Thanks for the contribution! -- To view, visit http://gerrit.cloudera.org:8080/15638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d1b4796b6280adecd7dab685a0281af6b2570ce Gerrit-Change-Number: 15638 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 19:06:18 + Gerrit-HasComments: No
[kudu-CR] KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore
Grant Henke has removed a vote on this change. Change subject: KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore .. Removed Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/15638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I7d1b4796b6280adecd7dab685a0281af6b2570ce Gerrit-Change-Number: 15638 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15638 ) Change subject: KUDU-3099: Remove System.exit() calls from KuduBackup/KuduRestore .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7d1b4796b6280adecd7dab685a0281af6b2570ce Gerrit-Change-Number: 15638 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 19:05:35 + Gerrit-HasComments: No
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Grant Henke has posted comments on this change. (
http://gerrit.cloudera.org:8080/15660 )
Change subject: wire_protocol: change columnar serialization of varlen data to
match Arrow
..
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15660/1//COMMIT_MSG@10
PS1, Line 10: match Apache Arrow
I think it might be important to add an integration test that verifies we
maintain arrow compatibility. That may mean adding arrow in the thirdparty
build.
This could be a follow on change and you can open a jira if you want to do it
later or allow someone else to do it.
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h
File src/kudu/client/columnar_scan_batch.h:
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h@45
PS1, Line 45: but with no alignment or padding guarantees
Are there potential issues by other arrow integrations based on this difference?
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/columnar_scan_batch.h@67
PS1, Line 67: Status GetFixedLengthColumn(int idx, Slice* data) const;
Do you think we should use an API similar to the row based API where we have a
method for each column type? If not I suppose we should at least doc all the
variable length types here (bytes and varchar).
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc
File src/kudu/client/scanner-internal.cc:
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@815
PS1, Line 815: return Status::InvalidArgument("column is variable-length",
col.ToString());
nit: Maybe add the type info here too.
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@820
PS1, Line 820: return Status::Corruption("server did not send data for
column", col.ToString());
Is there a case where this is expected to happen or is this just being
defensive?
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/client/scanner-internal.cc@840
PS1, Line 840: return Status::InvalidArgument("column is not
variable-length", col.ToString());
nit: Maybe add the type info here too.
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/common/columnar_serialization.h
File src/kudu/common/columnar_serialization.h:
http://gerrit.cloudera.org:8080/#/c/15660/1/src/kudu/common/columnar_serialization.h@38
PS1, Line 38: BINARY
nit: maybe be explicit about binary physical type.
--
To view, visit http://gerrit.cloudera.org:8080/15660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467
Gerrit-Change-Number: 15660
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Andrew Wong
Gerrit-Reviewer: Grant Henke
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 06 Apr 2020 19:04:34 +
Gerrit-HasComments: Yes
[kudu-CR] allow skip block manager in some ops of local replica tools
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15656 ) Change subject: allow skip block manager in some ops of local_replica tools .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc File src/kudu/tools/tool_action_local_replica.cc: http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc@158 PS2, Line 158: unique_ptr* fs_manager, bool skip_block_manager nit: per the guidance of the Google style guide, could you reverse the order of these arguments so the input parameter (skip_block_manager) is first? >When ordering function parameters, put all input-only parameters before any >output parameters. In particular, do not add new parameters to the end of the >function just because they are new; place new input-only parameters before the >output parameters. from https://google.github.io/styleguide/cppguide.html#Output_Parameters http://gerrit.cloudera.org:8080/#/c/15656/2/src/kudu/tools/tool_action_local_replica.cc@242 PS2, Line 242: (_manager, true) nit: for boolean arguments, to improve readability, would you mind inline-annotating the variable name? E.g. RETURN_NOT_OK(FsInit(_manager, /*skip_block_manager*/true)); -- To view, visit http://gerrit.cloudera.org:8080/15656 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I01e22354fdb76596008cd0824e240d24a8f20099 Gerrit-Change-Number: 15656 Gerrit-PatchSet: 2 Gerrit-Owner: wangning <1994wangn...@gmail.com> Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 18:43:02 + Gerrit-HasComments: Yes
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Hello Andrew Wong, Kudu Jenkins, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15660 to look at the new patch set (#2). Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. wire_protocol: change columnar serialization of varlen data to match Arrow This changes the format of variable-length columns serialized on the wire to match Apache Arrow instead of our internal column format. The Arrow format consists of an array of n+1 offsets for n rows, such that the data for cell 'n' spans offset offsets[n]...offsets[n+1]. The obvious advantage here is that clients can zero-copy into Arrow structures since the format is compatible. The less obvious advantage is that we are going from 16 bytes (sizeof(Slice)) to 4 bytes (sizeof(uint32_t offset)) for each serialized string, so this should be a savings even for non-arrow users of the API. This patch also adds some more sanity checking of the wire format in the client API so that it's not the responsibility of the caller to guard against malicious servers. Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 --- M src/kudu/client/client-test.cc M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/common/columnar_serialization.cc M src/kudu/common/columnar_serialization.h M src/kudu/common/rowblock.h M src/kudu/common/wire_protocol-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/tserver/tablet_server-test.cc M src/kudu/tserver/tablet_service.cc 12 files changed, 242 insertions(+), 120 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/15660/2 -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] wire protocol: change columnar serialization of varlen data to match Arrow
Hello Andrew Wong, Grant Henke, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/15660 to review the following change. Change subject: wire_protocol: change columnar serialization of varlen data to match Arrow .. wire_protocol: change columnar serialization of varlen data to match Arrow This changes the format of variable-length columns serialized on the wire to match Apache Arrow instead of our internal column format. The Arrow format consists of an array of n+1 offsets for n rows, such that the data for cell 'n' spans offset offsets[n]...offsets[n+1]. The obvious advantage here is that clients can zero-copy into Arrow structures since the format is compatible. The less obvious advantage is that we are going from 16 bytes (sizeof(Slice)) to 4 bytes (sizeof(uint32_t offset)) for each serialized string, so this should be a savings even for non-arrow users of the API. This patch also adds some more sanity checking of the wire format in the client API so that it's not the responsibility of the caller to guard against malicious servers. Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 --- M src/kudu/client/client-test.cc M src/kudu/client/columnar_scan_batch.cc M src/kudu/client/columnar_scan_batch.h M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/common/columnar_serialization.cc M src/kudu/common/columnar_serialization.h M src/kudu/common/rowblock.h M src/kudu/common/wire_protocol-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/tserver/tablet_service.cc 11 files changed, 234 insertions(+), 112 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/15660/1 -- To view, visit http://gerrit.cloudera.org:8080/15660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iadf728744feb83f5980e62bea4fd7634a1a52467 Gerrit-Change-Number: 15660 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke
[kudu-CR] columnar serialization: use AVX2 for int32 and int64 copying
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15634 ) Change subject: columnar_serialization: use AVX2 for int32 and int64 copying .. columnar_serialization: use AVX2 for int32 and int64 copying This uses the AVX2 "gather" instructions to do the copying of selected int32s and int64s. The following improvements were observed: Int32: Converting 10_int32_non_null to PB (method columnar) row select rate 1: 0.8829691 cycles/cell -> 0.8386091 cycles/cell Converting 10_int32_non_null to PB (method columnar) row select rate 0.8: 1.86863074 cycles/cell -> 1.61456746 cycles/cell Converting 10_int32_non_null to PB (method columnar) row select rate 0.5: 2.3829623 cycles/cell -> 2.05157198 cycles/cell Converting 10_int32_non_null to PB (method columnar) row select rate 0.2: 4.15909214 cycles/cell -> 3.82449024 cycles/cell Converting 10_int32_0pct_null to PB (method columnar) row select rate 1: 1.04652828 cycles/cell -> 1.01822806 cycles/cell Converting 10_int32_0pct_null to PB (method columnar) row select rate 0.8: 2.10860372 cycles/cell -> 1.85333702 cycles/cell Converting 10_int32_0pct_null to PB (method columnar) row select rate 0.5: 2.75141002 cycles/cell -> 2.39638206 cycles/cell Converting 10_int32_0pct_null to PB (method columnar) row select rate 0.2: 4.6968821 cycles/cell -> 4.40193506 cycles/cell Converting 10_int32_10pct_null to PB (method columnar) row select rate 1: 1.31809924 cycles/cell -> 1.31851512 cycles/cell Converting 10_int32_10pct_null to PB (method columnar) row select rate 0.8: 2.36648378 cycles/cell -> 2.12030662 cycles/cell Converting 10_int32_10pct_null to PB (method columnar) row select rate 0.5: 2.98480266 cycles/cell -> 2.7476185 cycles/cell Converting 10_int32_10pct_null to PB (method columnar) row select rate 0.2: 5.0439634 cycles/cell -> 4.5842071 cycles/cell Int64: Converting 10_int64_non_null to PB (method columnar) row select rate 1: 1.32330358 cycles/cell -> 1.24855148 cycles/cell Converting 10_int64_non_null to PB (method columnar) row select rate 0.8: 2.04848734 cycles/cell -> 2.12979712 cycles/cell Converting 10_int64_non_null to PB (method columnar) row select rate 0.5: 2.50150968 cycles/cell -> 2.5724664 cycles/cell Converting 10_int64_non_null to PB (method columnar) row select rate 0.2: 4.4513395 cycles/cell -> 4.35936382 cycles/cell Converting 10_int64_0pct_null to PB (method columnar) row select rate 1: 1.5080423 cycles/cell -> 1.51448434 cycles/cell Converting 10_int64_0pct_null to PB (method columnar) row select rate 0.8: 2.34286302 cycles/cell -> 2.26529584 cycles/cell Converting 10_int64_0pct_null to PB (method columnar) row select rate 0.5: 2.99375316 cycles/cell -> 2.7263687 cycles/cell Converting 10_int64_0pct_null to PB (method columnar) row select rate 0.2: 5.01722324 cycles/cell -> 4.71793008 cycles/cell Converting 10_int64_10pct_null to PB (method columnar) row select rate 1: 1.7227708 cycles/cell -> 1.67661726 cycles/cell Converting 10_int64_10pct_null to PB (method columnar) row select rate 0.8: 2.68160422 cycles/cell -> 2.50480846 cycles/cell Converting 10_int64_10pct_null to PB (method columnar) row select rate 0.5: 3.29833934 cycles/cell -> 3.05940708 cycles/cell Converting 10_int64_10pct_null to PB (method columnar) row select rate 0.2: 5.42127834 cycles/cell -> 4.99359244 cycles/cell In the few places that the above indicates a regression, I looped that same test case and found that the "after" was indeed either indistinguishable or slightly faster. The test results just have a little bit of noise. Change-Id: I6c9a536b78a524e8178f5d4a0d2dea04deedbd78 Reviewed-on: http://gerrit.cloudera.org:8080/15634 Tested-by: Todd Lipcon Reviewed-by: Andrew Wong --- M src/kudu/common/columnar_serialization.cc 1 file changed, 93 insertions(+), 9 deletions(-) Approvals: Todd Lipcon: Verified Andrew Wong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6c9a536b78a524e8178f5d4a0d2dea04deedbd78 Gerrit-Change-Number: 15634 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Todd Lipcon
[kudu-CR] columnar serialization: use AVX2 for int32 and int64 copying
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15634 ) Change subject: columnar_serialization: use AVX2 for int32 and int64 copying .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c9a536b78a524e8178f5d4a0d2dea04deedbd78 Gerrit-Change-Number: 15634 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 06 Apr 2020 17:42:23 + Gerrit-HasComments: No
[kudu-CR] [python] KUDU-2632 Add DATE type support
Volodymyr Verovkin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py File python/kudu/tests/util.py: http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/util.py@112 PS1, Line 112: datetime.date(2020,1,1) > For the sake of covering corner cases and making sure datetime.date() prope I think for this kind of testing we just need to write tests for util functions unix_epoch_days_to_date() and date_to_unix_epoch_days() -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Mon, 06 Apr 2020 17:34:19 + Gerrit-HasComments: Yes
[kudu-CR] ranger: remove experimental flags
Hao Hao has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15658 ) Change subject: ranger: remove experimental flags .. ranger: remove experimental flags This patch removes the experimental flags in Ranger, given I did integration tests on a real cluster. Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Reviewed-on: http://gerrit.cloudera.org:8080/15658 Reviewed-by: Andrew Wong Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M src/kudu/ranger/ranger_client.cc 1 file changed, 0 insertions(+), 3 deletions(-) Approvals: Andrew Wong: Looks good to me, approved Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Gerrit-Change-Number: 15658 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] ranger: remove experimental flags
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15658 ) Change subject: ranger: remove experimental flags .. Patch Set 1: (1 comment) > Patch Set 1: Code-Review+2 > > (1 comment) http://gerrit.cloudera.org:8080/#/c/15658/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15658/1//COMMIT_MSG@10 PS1, Line 10: integration tests on a real cluster > So, the main driver of this change is to avoid using --unlock_experimental_ Yes, that's right. -- To view, visit http://gerrit.cloudera.org:8080/15658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Gerrit-Change-Number: 15658 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 17:30:40 + Gerrit-HasComments: Yes
[kudu-CR] [docker] Fix the docker thirdparty build
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15659 Change subject: [docker] Fix the docker thirdparty build .. [docker] Fix the docker thirdparty build After 458e6ce, the docker thirdparty build was failing because `/build-support/ccache-clang` was not available in the thirdparty image. I tested this fix by running builds based on ubuntu:xenial, centos:6, and centos:7 base images. Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed --- M docker/Dockerfile 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/59/15659/1 -- To view, visit http://gerrit.cloudera.org:8080/15659 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If25d7b07db4cc20de53b70fe0ab097be6f3756ed Gerrit-Change-Number: 15659 Gerrit-PatchSet: 1 Gerrit-Owner: Grant Henke
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Hello Tidy Bot, Kudu Jenkins, Adar Dembo, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15657 to look at the new patch set (#2). Change subject: [master] KUDU-2798 fix logging on deleted TSK entries .. [master] KUDU-2798 fix logging on deleted TSK entries Decode the identifiers of the deleted TSK entries while logging. Prior to this patch, the raw value of the system catalog's 'entry_id' column was written into the log. Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 --- M src/kudu/master/catalog_manager.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h 3 files changed, 19 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/15657/2 -- To view, visit http://gerrit.cloudera.org:8080/15657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Gerrit-Change-Number: 15657 Gerrit-PatchSet: 2 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [python] KUDU-2632 Add DATE type support
Volodymyr Verovkin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15645 ) Change subject: [python] KUDU-2632 Add DATE type support .. Patch Set 3: (8 comments) http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/test_scanner.py File python/kudu/tests/test_scanner.py: http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/tests/test_scanner.py@349 PS1, Line 349: self.assertEqual(types[0], np.int64) : self.assertEqual(types[1], 'datetime64[ns, UTC]') : self.assertEqual(types[2], np.object) : self.assertEqual(types[3], np.object) : self.assertEqual(types[4], np.bool) : self.assertEqual(types[5], np.float64) : self.assertEqual(types[6], np.int8) : self.assertEqual(types[7], np.object) : self.assertEqual(types[8], np.object) : self.assertEqual(types[9], np.object) : self.assertEqual(types[10], np.float32) : else: : self.assertEqual(types[0], np.int64) : self.assertEqual(types[1], 'datetime64[ns, UTC]') : self.assertEqual(types[2], np.object) : self.assertEqual(types[3], np.bool) : self.assertEqual(types[4], np.float64) : self.assertEqual(types[5], np.int8) : self.assertEqual(types[6], np.object) > It seems this needs to be updated to accommodate for newly added columns of Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py File python/kudu/util.py: http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@186 PS1, Line 186: Convert > If following the same style of the docs as for the other functions in this Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@188 PS1, Line 188: of seconds per day (86400). > nit: add a period Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@189 PS1, Line 189: > extra spaces Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@190 PS1, Line 190: Parameters > Could you add information about the parameter of this function into the doc Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@198 PS1, Line 198: """ > Convert Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@199 PS1, Line 199: return datetime.datetime.utcfromtimestamp(ndays * 86400).date() > nit: add a period Done http://gerrit.cloudera.org:8080/#/c/15645/1/python/kudu/util.py@203 PS1, Line 203: Convert d > Is this a mismatch between the actual name of the parameter and the documen Done -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Mon, 06 Apr 2020 16:53:20 + Gerrit-HasComments: Yes
[kudu-CR] columnar serialization: use AVX2 for int32 and int64 copying
Todd Lipcon has removed Kudu Jenkins from this change. ( http://gerrit.cloudera.org:8080/15634 ) Change subject: columnar_serialization: use AVX2 for int32 and int64 copying .. Removed reviewer Kudu Jenkins with the following votes: * Verified-1 by Kudu Jenkins (120) -- To view, visit http://gerrit.cloudera.org:8080/15634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I6c9a536b78a524e8178f5d4a0d2dea04deedbd78 Gerrit-Change-Number: 15634 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Todd Lipcon
[kudu-CR] rowblock: use BMI instruction set when available for GetSelectedRows
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15635 ) Change subject: rowblock: use BMI instruction set when available for GetSelectedRows .. rowblock: use BMI instruction set when available for GetSelectedRows This enables a BMI variant of SelectionVector::GetSelectedRows which has a higher throughput. I disassembled the resulting hot loop as follows: BMI: L: tzcnt %rsi,%rbx or %r11d,%ebx mov%bx,(%rdx) blsr %rsi,%rsi tzcnt %rsi,%rbx or %r11d,%ebx mov%bx,0x2(%rdx) blsr %rsi,%rsi tzcnt %rsi,%rbx or %r11d,%ebx mov%bx,0x4(%rdx) add$0x6,%rdx blsr %rsi,%rsi add$0xfffd,%ecx jneL non-BMI: L: bsf%rsi,%rax or %r12d,%eax mov%ax,(%rdx) lea-0x1(%rsi),%rax and%rsi,%rax bsf%rax,%rsi or %r12d,%esi mov%si,0x2(%rdx) lea-0x1(%rax),%rbx and%rax,%rbx bsf%rbx,%rax or %r12d,%eax mov%ax,0x4(%rdx) add$0x6,%rdx lea-0x1(%rbx),%rsi and%rbx,%rsi add$0xfffd,%ecx jneL ... and then used llvm-mca on these assembly files across a few common architectures to see how many cycles were required for 100 iterations of the loop. Results are as follows: haswell non-bmi.s: Total Cycles: 606 haswell bmi.s: Total Cycles: 382 broadwell non-bmi.s: Total Cycles 606 broadwell bmi.s: Total Cycles:382 skylake non-bmi.s: Total Cycles: 606 skylake bmi.s: Total Cycles: 307 So, on the most recent chips, this should be about a 2x improvement in this function. This function made up a few percent of overall CPU consumption in some TSBS workloads, so this patch had some small but measurable improvement on end-to-end throughput. Change-Id: I8ec74bc5db07c18d0e36de14a2343f49fc5c2859 Reviewed-on: http://gerrit.cloudera.org:8080/15635 Tested-by: Kudu Jenkins Reviewed-by: Alexey Serbin --- M src/kudu/common/rowblock.cc 1 file changed, 19 insertions(+), 3 deletions(-) Approvals: Kudu Jenkins: Verified Alexey Serbin: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8ec74bc5db07c18d0e36de14a2343f49fc5c2859 Gerrit-Change-Number: 15635 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Todd Lipcon
[kudu-CR] columnar serialization: use AVX2 for int32 and int64 copying
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/15634 ) Change subject: columnar_serialization: use AVX2 for int32 and int64 copying .. Patch Set 3: Verified+1 the release build failed downloading numpy due to a network error. Since the previous build passed (only a lint issue) I'm overriding -- To view, visit http://gerrit.cloudera.org:8080/15634 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c9a536b78a524e8178f5d4a0d2dea04deedbd78 Gerrit-Change-Number: 15634 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 06 Apr 2020 16:36:27 + Gerrit-HasComments: No
[kudu-CR] ranger: remove experimental flags
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15658 ) Change subject: ranger: remove experimental flags .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/15658/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15658/1//COMMIT_MSG@10 PS1, Line 10: integration tests on a real cluster So, the main driver of this change is to avoid using --unlock_experimental_flags when running the Ranger client? And we don't expect these flags to be changed or removed in future, right? -- To view, visit http://gerrit.cloudera.org:8080/15658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Gerrit-Change-Number: 15658 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 16:00:03 + Gerrit-HasComments: Yes
[kudu-CR] ranger: remove experimental flags
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15658 ) Change subject: ranger: remove experimental flags .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Gerrit-Change-Number: 15658 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Comment-Date: Mon, 06 Apr 2020 07:50:02 + Gerrit-HasComments: No
[kudu-CR] [python] KUDU-2632 Add DATE type support
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15645 to look at the new patch set (#3). Change subject: [python] KUDU-2632 Add DATE type support .. [python] KUDU-2632 Add DATE type support There are two utility functions which perfrom conversion between Kudu DATE and Python datetime.date: unix_epoch_days_to_date() - converts number of days since Unix ecpoch to datetime.date() date_to_unix_epoch_days() - converts datetime.date() to number of days since Unix ecpoch Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 --- M python/kudu/__init__.py M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M python/kudu/tests/test_scanner.py M python/kudu/tests/test_scantoken.py M python/kudu/tests/test_schema.py M python/kudu/tests/util.py M python/kudu/util.py 9 files changed, 113 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/15645/3 -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] ranger: remove experimental flags
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15658 Change subject: ranger: remove experimental flags .. ranger: remove experimental flags This patch removes the experimental flags in Ranger, given I did integration tests on a real cluster. Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 --- M src/kudu/ranger/ranger_client.cc 1 file changed, 0 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/58/15658/1 -- To view, visit http://gerrit.cloudera.org:8080/15658 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I87e36f973730f54e6e3984e34a8216eeeacc1088 Gerrit-Change-Number: 15658 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao
[kudu-CR] [master] KUDU-2798 fix logging on deleted TSK entries
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15657 Change subject: [master] KUDU-2798 fix logging on deleted TSK entries .. [master] KUDU-2798 fix logging on deleted TSK entries Decode the identifiers of the deleted TSK entries while logging. Prior to this patch, the raw value of the system catalog's 'entry_id' column was written into the log. Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 --- M src/kudu/master/catalog_manager.cc M src/kudu/master/sys_catalog.cc M src/kudu/master/sys_catalog.h 3 files changed, 17 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/57/15657/1 -- To view, visit http://gerrit.cloudera.org:8080/15657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I98c7ba7fd2277fff1176eca51d59404deebe38c4 Gerrit-Change-Number: 15657 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin
[kudu-CR] ranger: allow overwriting of the log4j2 properties file
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15650 ) Change subject: ranger: allow overwriting of the log4j2 properties file .. ranger: allow overwriting of the log4j2 properties file It seems important for users to have the ability to have subsequent runs of the master honor any new logging configurations specified via gflag. As such, it seems important to allow users to recreate the log4j2 properties file used by the Ranger client, even if one exists. This patch enables this by introducing the --ranger_overwrite_log_config gflag, which is set to true by default. Change-Id: I4a06f8a1b3328cfd4029295527b5ba61a03efbfa Reviewed-on: http://gerrit.cloudera.org:8080/15650 Tested-by: Andrew Wong Reviewed-by: Alexey Serbin Reviewed-by: Hao Hao --- M src/kudu/ranger/ranger_client-test.cc M src/kudu/ranger/ranger_client.cc 2 files changed, 132 insertions(+), 31 deletions(-) Approvals: Andrew Wong: Verified Alexey Serbin: Looks good to me, approved Hao Hao: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/15650 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4a06f8a1b3328cfd4029295527b5ba61a03efbfa Gerrit-Change-Number: 15650 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] [python] KUDU-2632 Add DATE type support
Hello Alexey Serbin, Attila Bukor, Kudu Jenkins, Adar Dembo, Grant Henke, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15645 to look at the new patch set (#2). Change subject: [python] KUDU-2632 Add DATE type support .. [python] KUDU-2632 Add DATE type support There are two utility functions which perfrom conversion between Kudu DATE and Python datetime.date: unix_epoch_days_to_date() - converts number of days since Unix ecpoch to datetime.date() date_to_unix_epoch_days() - converts datetime.date() to number of days since Unix ecpoch Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 --- M python/kudu/__init__.py M python/kudu/client.pyx M python/kudu/libkudu_client.pxd M python/kudu/schema.pyx M python/kudu/tests/test_scanner.py M python/kudu/tests/test_scantoken.py M python/kudu/tests/test_schema.py M python/kudu/tests/util.py M python/kudu/util.py 9 files changed, 108 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/15645/2 -- To view, visit http://gerrit.cloudera.org:8080/15645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1f08946e9ba56dab5e5b43e2bf65bc535c26ab25 Gerrit-Change-Number: 15645 Gerrit-PatchSet: 2 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] ranger: allow overwriting of the log4j2 properties file
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15650 ) Change subject: ranger: allow overwriting of the log4j2 properties file .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15650 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4a06f8a1b3328cfd4029295527b5ba61a03efbfa Gerrit-Change-Number: 15650 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Wong Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Tidy Bot (241) Gerrit-Comment-Date: Mon, 06 Apr 2020 06:27:21 + Gerrit-HasComments: No
