[kudu-CR] KUDU-1943: Add BlockTransaction to Block Manager
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7207 to look at the new patch set (#11). Change subject: KUDU-1943: Add BlockTransaction to Block Manager .. KUDU-1943: Add BlockTransaction to Block Manager This adds a new layer of abstraction 'BlockTransaction' to Block Manager, to accumulate creates which happen in a logical operation, e.g. flush/compaction. Avoid using fsync() per block can potentionaly reduce a lot I/O overhead. This patch also add a new API FinalizeWrite() to Block Manager, so that for LBM container can be resued, without flushing in-flight writable blocks to disk. A design doc can be found here: https://docs.google.com/a/cloudera.com/document/d/15zZaKeLENRGC_AdjF2N3hxtM_6_OuydXA1reWvDbbK0/edit?usp=sharing Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd --- M src/kudu/cfile/bloomfile.cc M src/kudu/cfile/bloomfile.h M src/kudu/cfile/cfile-test.cc M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/cfile_writer.h M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager.cc M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/multi_column_writer.cc M src/kudu/tablet/multi_column_writer.h M src/kudu/util/scoped_cleanup-test.cc 19 files changed, 627 insertions(+), 268 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/07/7207/11 -- To view, visit http://gerrit.cloudera.org:8080/7207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd Gerrit-PatchSet: 11 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has uploaded a new change for review. http://gerrit.cloudera.org:8080/7584 Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 --- M java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportCsv.java M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 2 files changed, 11 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/84/7584/1 -- To view, visit http://gerrit.cloudera.org:8080/7584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has uploaded a new change for review. http://gerrit.cloudera.org:8080/7585 Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 --- M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/7585/1 -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has uploaded a new patch set (#2). Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 --- D java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 1 file changed, 0 insertions(+), 127 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/7585/2 -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has abandoned this change. Change subject: Kudu-client ImportCsv small fix's .. Abandoned NA -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has uploaded a new patch set (#2). Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 --- M java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportCsv.java M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 2 files changed, 10 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/84/7584/2 -- To view, visit http://gerrit.cloudera.org:8080/7584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7584 to look at the new patch set (#3). Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 --- M java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportCsv.java M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 2 files changed, 10 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/84/7584/3 -- To view, visit http://gerrit.cloudera.org:8080/7584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7584 to look at the new patch set (#4). Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 --- M java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportCsv.java M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 2 files changed, 10 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/84/7584/4 -- To view, visit http://gerrit.cloudera.org:8080/7584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I422e83d99db38be7a011ddc00b3412d542e2c181 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has restored this change. Change subject: Kudu-client ImportCsv small fix's .. Restored -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: restore Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7585 to look at the new patch set (#3). Change subject: Kudu-client ImportCsv small fix's .. Kudu-client ImportCsv small fix's Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/7585/3 -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Kudu-client ImportCsv small fix's
Sandish Kumar HN has abandoned this change. Change subject: Kudu-client ImportCsv small fix's .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/7585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I54ab38cd00d4aba4d899584e446e9c6e11217550 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Revert "Kudu-client ImportCsv small fix's"
Sandish Kumar HN has uploaded a new change for review. http://gerrit.cloudera.org:8080/7586 Change subject: Revert "Kudu-client ImportCsv small fix's" .. Revert "Kudu-client ImportCsv small fix's" This reverts commit 6163c0e1442a92f9029de26d1298a4d2cd50ec69. Change-Id: Idb7c0359e24b36ca4e8471149327a368ade55765 --- M java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportCsv.java 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/86/7586/1 -- To view, visit http://gerrit.cloudera.org:8080/7586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Idb7c0359e24b36ca4e8471149327a368ade55765 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN
[kudu-CR] Revert "Kudu-client ImportCsv small fix's"
Sandish Kumar HN has abandoned this change. Change subject: Revert "Kudu-client ImportCsv small fix's" .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/7586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: Idb7c0359e24b36ca4e8471149327a368ade55765 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Revert "Kudu-client ImportCsv small fix's"
Sandish Kumar HN has uploaded a new patch set (#2). Change subject: Revert "Kudu-client ImportCsv small fix's" .. Revert "Kudu-client ImportCsv small fix's" This reverts commit 6163c0e1442a92f9029de26d1298a4d2cd50ec69. Change-Id: Idb7c0359e24b36ca4e8471149327a368ade55765 --- 0 files changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/86/7586/2 -- To view, visit http://gerrit.cloudera.org:8080/7586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb7c0359e24b36ca4e8471149327a368ade55765 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [build] Fix Python build failures on RHEL 6
Hello David Ribeiro Alves, Adar Dembo, Todd Lipcon, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/7589 to review the following change. Change subject: [build] Fix Python build failures on RHEL 6 .. [build] Fix Python build failures on RHEL 6 Internal build machines running RHEL 6 started failing recently on something that looks a lot like: https://github.com/pypa/pip/issues/3045 But we already have the fix in. This patch works around the problem by upgrading setuptools separately which somehow works. Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb --- M build-support/jenkins/build-and-test.sh M python/requirements.txt 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/89/7589/1 -- To view, visit http://gerrit.cloudera.org:8080/7589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Jean-Daniel Cryans Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [build] Fix Python build failures on RHEL 6
David Ribeiro Alves has posted comments on this change. Change subject: [build] Fix Python build failures on RHEL 6 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/7589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Jean-Daniel Cryans Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] Expose running maintenance op info
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7537 to look at the new patch set (#6). Change subject: Expose running maintenance op info .. Expose running maintenance op info Previously, the maintenance manager stored the number of instances a given operation has running, but information like the start time of an instance was not made available until the instance completed. However, the start times of running instances can be helpful information to display on the web UI. This commit adds a collection of running instances to the MaintenanceManager so the information mentioned above can be accessed while operations are running. Additionally, instances of operations now store the id of the thread on which they run. Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h M src/kudu/util/maintenance_manager.proto 5 files changed, 114 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/7537/6 -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 6 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [build] Fix Python build failures on RHEL 6
David Ribeiro Alves has posted comments on this change. Change subject: [build] Fix Python build failures on RHEL 6 .. Patch Set 1: I'll keep looking for a "non-workaround" version. -- To view, visit http://gerrit.cloudera.org:8080/7589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Jean-Daniel Cryans Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] Expose running maintenance op info
Sam Okrent has posted comments on this change. Change subject: Expose running maintenance op info .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/7537/5/src/kudu/util/maintenance_manager.cc File src/kudu/util/maintenance_manager.cc: PS5, Line 501: MaintenanceManagerStatusPB_OpInstancePB* running_pb = out_pb->add_running_operations(); : running_op->DumpToPB(running_pb); > How about making DumpToPB return the pb directly and maybe changing this to Done http://gerrit.cloudera.org:8080/#/c/7537/5/src/kudu/util/maintenance_manager.h File src/kudu/util/maintenance_manager.h: PS5, Line 155: Value is negative if instance is still running. > Can you add a similar comment to the .proto? Done PS5, Line 332: std::map running_instances_; > Maybe mention locking requirements? Done http://gerrit.cloudera.org:8080/#/c/7537/5/src/kudu/util/maintenance_manager.proto File src/kudu/util/maintenance_manager.proto: PS5, Line 37: required int32 duration_millis = 3; > Did you consider making this optional in the case of currently-running inst Done -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] [build] Fix Python build failures on RHEL 6
Jean-Daniel Cryans has posted comments on this change. Change subject: [build] Fix Python build failures on RHEL 6 .. Patch Set 1: Verified+1 Overriding jenkins, I doubt a Python build change had an impact on Java tests. -- To view, visit http://gerrit.cloudera.org:8080/7589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Jean-Daniel Cryans Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] [build] Fix Python build failures on RHEL 6
Jean-Daniel Cryans has submitted this change and it was merged. Change subject: [build] Fix Python build failures on RHEL 6 .. [build] Fix Python build failures on RHEL 6 Internal build machines running RHEL 6 started failing recently on something that looks a lot like: https://github.com/pypa/pip/issues/3045 But we already have the fix in. This patch works around the problem by upgrading setuptools separately which somehow works. Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Reviewed-on: http://gerrit.cloudera.org:8080/7589 Reviewed-by: David Ribeiro Alves Tested-by: Jean-Daniel Cryans --- M build-support/jenkins/build-and-test.sh M python/requirements.txt 2 files changed, 5 insertions(+), 1 deletion(-) Approvals: David Ribeiro Alves: Looks good to me, approved Jean-Daniel Cryans: Verified -- To view, visit http://gerrit.cloudera.org:8080/7589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I05ddecc4f903607f1131015277c1f36f2d766acb Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Jean-Daniel Cryans Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Todd Lipcon
[kudu-CR] Expose running maintenance op info
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7537 to look at the new patch set (#7). Change subject: Expose running maintenance op info .. Expose running maintenance op info Previously, the maintenance manager stored the number of instances a given operation has running, but information like the start time of an instance was not made available until the instance completed. However, the start times of running instances can be helpful information to display on the web UI. This commit adds a collection of running instances to the MaintenanceManager so the information mentioned above can be accessed while operations are running. Additionally, instances of operations now store the id of the thread on which they run. Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h M src/kudu/util/maintenance_manager.proto 5 files changed, 114 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/7537/7 -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 7 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] kudu client tools for hadoop and spark import/export(csv,parquet,avro)
Jean-Daniel Cryans has posted comments on this change. Change subject: kudu client tools for hadoop and spark import/export(csv,parquet,avro) .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/7421/3/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportParquet.java File java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportParquet.java: Line 56: private static final Log LOG = Log.getLog(ImportParquet.class); I just noticed, you shouldn't use this Log that comes from Parquet. It's best to use System.err.println like ImportCsv does. Line 101: System.exit(0); > what should I use instead of system.exit(0). just LOG.error("") enough?? I'd suggestion throwing an exception and catching it in run() below. http://gerrit.cloudera.org:8080/#/c/7421/3/java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportParquet.java File java/kudu-client-tools/src/test/java/org/apache/kudu/mapreduce/tools/ITImportParquet.java: Line 50: public class ITImportParquet extends BaseKuduTest { > separate test-case for pre-flight checks? and what should be the expected r The goal is to test all the major code paths, right now your patch isn't testing error cases. -- To view, visit http://gerrit.cloudera.org:8080/7421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: If462af948651f3869b444e82151c3559fde19142 Gerrit-PatchSet: 3 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sandish Kumar HN Gerrit-HasComments: Yes
[kudu-CR] kudu client tools for hadoop and spark import/export(csv,parquet,avro)
Jean-Daniel Cryans has posted comments on this change. Change subject: kudu client tools for hadoop and spark import/export(csv,parquet,avro) .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/7421/5/java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportParquet.java File java/kudu-client-tools/src/main/java/org/apache/kudu/mapreduce/tools/ImportParquet.java: Line 99: if (!schema.containsField(columnSchema.getName())) { > Kudu has primitive-types, parquet has just Type. Example: In Kudu binary ty The goal is to not hit weird errors in ImportParquetMapper#map when doing data conversions, so you have to find the equivalences. -- To view, visit http://gerrit.cloudera.org:8080/7421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: If462af948651f3869b444e82151c3559fde19142 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sandish Kumar HN Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sandish Kumar HN Gerrit-HasComments: Yes
[kudu-CR] Expose running maintenance op info
Andrew Wong has posted comments on this change. Change subject: Expose running maintenance op info .. Patch Set 7: (2 comments) Couple more nits. http://gerrit.cloudera.org:8080/#/c/7537/7/src/kudu/tserver/tserver-path-handlers.cc File src/kudu/tserver/tserver-path-handlers.cc: PS7, Line 52: using kudu::MaintenanceManagerStatusPB_OpInstancePB; nit: fix ordering http://gerrit.cloudera.org:8080/#/c/7537/7/src/kudu/util/maintenance_manager-test.cc File src/kudu/util/maintenance_manager-test.cc: PS7, Line 307: MaintenanceManagerStatusPB status_pb; : manager_->GetMaintenanceManagerStatusDump(&status_pb); : ASSERT_EQ(status_pb.running_operations_size(), 2); : const MaintenanceManagerStatusPB_OpInstancePB& instance1 = status_pb.running_operations(0); : const MaintenanceManagerStatusPB_OpInstancePB& instance2 = status_pb.running_operations(1); : ASSERT_EQ(instance1.name(), op.name()); : ASSERT_NE(instance1.thread_id(), instance2.thread_id()); : nit: can you move this back a couple of spaces? -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 7 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] Expose running maintenance op info
Sam Okrent has posted comments on this change. Change subject: Expose running maintenance op info .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/7537/7/src/kudu/tserver/tserver-path-handlers.cc File src/kudu/tserver/tserver-path-handlers.cc: PS7, Line 52: using kudu::MaintenanceManagerStatusPB_OpInstancePB; > nit: fix ordering Done http://gerrit.cloudera.org:8080/#/c/7537/7/src/kudu/util/maintenance_manager-test.cc File src/kudu/util/maintenance_manager-test.cc: PS7, Line 307: MaintenanceManagerStatusPB status_pb; : manager_->GetMaintenanceManagerStatusDump(&status_pb); : ASSERT_EQ(status_pb.running_operations_size(), 2); : const MaintenanceManagerStatusPB_OpInstancePB& instance1 = status_pb.running_operations(0); : const MaintenanceManagerStatusPB_OpInstancePB& instance2 = status_pb.running_operations(1); : ASSERT_EQ(instance1.name(), op.name()); : ASSERT_NE(instance1.thread_id(), instance2.thread_id()); : > nit: can you move this back a couple of spaces? The other ASSERT_EVENTUALLY blocks are formatted like this, so it does follow the style of the file. -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 7 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] Expose running maintenance op info
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7537 to look at the new patch set (#8). Change subject: Expose running maintenance op info .. Expose running maintenance op info Previously, the maintenance manager stored the number of instances a given operation has running, but information like the start time of an instance was not made available until the instance completed. However, the start times of running instances can be helpful information to display on the web UI. This commit adds a collection of running instances to the MaintenanceManager so the information mentioned above can be accessed while operations are running. Additionally, instances of operations now store the id of the thread on which they run. Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h M src/kudu/util/maintenance_manager.proto 5 files changed, 114 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/37/7537/8 -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 8 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] Add Maintenance Manager visualizer
Andrew Wong has posted comments on this change. Change subject: Add Maintenance Manager visualizer .. Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/7570/3//COMMIT_MSG Commit Message: PS3, Line 7: Add Maintenance Manager visualizer Could you add a link to some publicly-hosted screenshot? E.g. upload the image from your slides to your own local github branch and link to that, or somesuch. PS3, Line 11: Previously, completed and : running ops were displayed in two tables. Are running ops displayed differently from completed ops? In addition to an image, a description of what is newly shown would be nice. http://gerrit.cloudera.org:8080/#/c/7570/3/www/maintenance-manager.mustache File www/maintenance-manager.mustache: PS3, Line 69: } nit: Got some whitespaces throughout this file. PS3, Line 181: : function inspectOp(op, event) { While some of these functions are fairly straightforward, it'd still be nice to have some docs explaining what these do. PS3, Line 202: ""; Do we expect these? Might be worth documenting what an expected op.name/output are. -- To view, visit http://gerrit.cloudera.org:8080/7570 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] WIP [thirdparty]: added include-what-you-use
Alexey Serbin has uploaded a new change for review. http://gerrit.cloudera.org:8080/7593 Change subject: WIP [thirdparty]: added include-what-you-use .. WIP [thirdparty]: added include-what-you-use Build the include-what-you-use utility along with the LLVM toolchain. The project URL: https://include-what-you-use.org/ WIP: it's necessary to remove custom prefix once the tarball is uploaded into the cloudfront bucket. Change-Id: I041ad9e4bbff410f1760c7abd8cd173e5e9cc564 --- M thirdparty/download-thirdparty.sh A thirdparty/patches/llvm-add-iwyu.patch M thirdparty/vars.sh 3 files changed, 34 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/7593/1 -- To view, visit http://gerrit.cloudera.org:8080/7593 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I041ad9e4bbff410f1760c7abd8cd173e5e9cc564 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Alexey Serbin
[kudu-CR] WIP: [iwyu] first pass
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/4738 to look at the new patch set (#9). Change subject: WIP: [iwyu] first pass .. WIP: [iwyu] first pass Updated C++ source files in accordance with include-what-you-use recommendations: * remove unused header files * add missing header files * use forward declarations when possible As a result, time of compilation reduced ~10% if building with GNU make in parallel with 48 jobs (make -j48) at ve0518.halxg.cloudera.com: prior: real 1m21.858s user 33m26.685s sys 2m19.831s after: real 1m12.686s user 30m26.891s sys 2m7.627s The next step is automating the checks, so the IWYU check would run automatically (like the LINT build). Change-Id: I6923091be36a1c03fac34dd73e6b17e17ac9ceaa --- M src/kudu/benchmarks/rle.cc M src/kudu/benchmarks/tpch/line_item_tsv_importer.h M src/kudu/benchmarks/tpch/rpc_line_item_dao-test.cc M src/kudu/benchmarks/tpch/rpc_line_item_dao.cc M src/kudu/benchmarks/tpch/rpc_line_item_dao.h M src/kudu/benchmarks/tpch/tpch-schemas.h M src/kudu/benchmarks/tpch/tpch1.cc M src/kudu/benchmarks/tpch/tpch_real_world.cc M src/kudu/benchmarks/wal_hiccup.cc M src/kudu/cfile/binary_dict_block.cc M src/kudu/cfile/binary_dict_block.h M src/kudu/cfile/binary_plain_block.cc M src/kudu/cfile/binary_plain_block.h M src/kudu/cfile/binary_prefix_block.cc M src/kudu/cfile/binary_prefix_block.h M src/kudu/cfile/block_cache-test.cc M src/kudu/cfile/block_cache.cc M src/kudu/cfile/block_cache.h M src/kudu/cfile/block_compression.cc M src/kudu/cfile/block_compression.h M src/kudu/cfile/bloomfile-test-base.h M src/kudu/cfile/bloomfile-test.cc M src/kudu/cfile/bloomfile.cc M src/kudu/cfile/bloomfile.h M src/kudu/cfile/bshuf_block.h M src/kudu/cfile/cfile-test-base.h M src/kudu/cfile/cfile-test.cc M src/kudu/cfile/cfile_reader.cc M src/kudu/cfile/cfile_reader.h M src/kudu/cfile/cfile_util.cc M src/kudu/cfile/cfile_util.h M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/cfile_writer.h M src/kudu/cfile/encoding-test.cc M src/kudu/cfile/index-test.cc M src/kudu/cfile/index_block.cc M src/kudu/cfile/index_block.h M src/kudu/cfile/index_btree.cc M src/kudu/cfile/index_btree.h M src/kudu/cfile/mt-bloomfile-test.cc M src/kudu/cfile/type_encodings.cc M src/kudu/cfile/type_encodings.h M src/kudu/client/batcher.cc M src/kudu/client/batcher.h M src/kudu/client/client-internal.cc M src/kudu/client/client-internal.h M src/kudu/client/client-test-util.cc M src/kudu/client/client-test-util.h M src/kudu/client/client-test.cc M src/kudu/client/client-unittest.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/client_builder-internal.h M src/kudu/client/error-internal.cc M src/kudu/client/error-internal.h M src/kudu/client/error_collector.cc M src/kudu/client/error_collector.h M src/kudu/client/master_rpc.h M src/kudu/client/meta_cache.cc M src/kudu/client/meta_cache.h M src/kudu/client/partitioner-internal.cc M src/kudu/client/predicate-test.cc M src/kudu/client/replica-internal.cc M src/kudu/client/replica-internal.h M src/kudu/client/resource_metrics.cc M src/kudu/client/resource_metrics.h M src/kudu/client/samples/sample.cc M src/kudu/client/scan_batch.cc M src/kudu/client/scan_batch.h M src/kudu/client/scan_configuration.cc M src/kudu/client/scan_configuration.h M src/kudu/client/scan_predicate.cc M src/kudu/client/scan_predicate.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/scan_token-internal.h M src/kudu/client/scan_token-test.cc M src/kudu/client/scanner-internal.cc M src/kudu/client/scanner-internal.h M src/kudu/client/schema-internal.h M src/kudu/client/schema.cc M src/kudu/client/schema.h M src/kudu/client/session-internal.cc M src/kudu/client/session-internal.h M src/kudu/client/shared_ptr.h M src/kudu/client/table-internal.cc M src/kudu/client/table-internal.h M src/kudu/client/table_alterer-internal.cc M src/kudu/client/table_alterer-internal.h M src/kudu/client/table_creator-internal.cc M src/kudu/client/table_creator-internal.h M src/kudu/client/tablet-internal.h M src/kudu/client/tablet_server-internal.cc M src/kudu/client/tablet_server-internal.h M src/kudu/client/value-internal.h M src/kudu/client/value.cc M src/kudu/client/value.h M src/kudu/client/write_op.cc M src/kudu/client/write_op.h M src/kudu/clock/clock.h M src/kudu/clock/hybrid_clock-test.cc M src/kudu/clock/hybrid_clock.cc M src/kudu/clock/hybrid_clock.h M src/kudu/clock/logical_clock-test.cc M src/kudu/clock/logical_clock.cc M src/kudu/clock/logical_clock.h M src/kudu/codegen/code_cache.cc M src/kudu/codegen/code_cache.h M src/kudu/codegen/code_generator.cc M src/kudu/codegen/code_generator.h M src/kudu/codegen/codegen-test.cc M src/kudu/codegen/compilation_manager.cc M src/kudu/codegen/compilation_manager.h M src/kudu/codegen/jit_wrapper.cc M src/kudu/codegen/jit_wrapper.h M src/kudu/codegen/module_builder.cc M
[kudu-CR] KUDU-1943: Add BlockTransaction to Block Manager
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7207 to look at the new patch set (#12). Change subject: KUDU-1943: Add BlockTransaction to Block Manager .. KUDU-1943: Add BlockTransaction to Block Manager This adds a new layer of abstraction 'BlockTransaction' to Block Manager, to accumulate creates which happen in a logical operation, e.g. flush/compaction. Avoid using fsync() per block can potentionaly reduce a lot I/O overhead. This patch also add a new API FinalizeWrite() to Block Manager, so that for LBM container can be resued, without flushing in-flight writable blocks to disk. A design doc can be found here: https://docs.google.com/a/cloudera.com/document/d/15zZaKeLENRGC_AdjF2N3hxtM_6_OuydXA1reWvDbbK0/edit?usp=sharing Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd --- M src/kudu/cfile/bloomfile.cc M src/kudu/cfile/bloomfile.h M src/kudu/cfile/cfile-test.cc M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/cfile_writer.h M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager.cc M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/multi_column_writer.cc M src/kudu/tablet/multi_column_writer.h 18 files changed, 662 insertions(+), 267 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/07/7207/12 -- To view, visit http://gerrit.cloudera.org:8080/7207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd Gerrit-PatchSet: 12 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] KUDU-1943: Add BlockTransaction to Block Manager
Hao Hao has posted comments on this change. Change subject: KUDU-1943: Add BlockTransaction to Block Manager .. Patch Set 12: (38 comments) http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/cfile/cfile-test.cc File src/kudu/cfile/cfile-test.cc: Line 900: } > Could you further parameterize the test so that we run with both values of Done http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/cfile/cfile_writer.cc File src/kudu/cfile/cfile_writer.cc: PS10, Line 62: "Whether to finalize or close cfile blocks when writing " : "is finished. "); > "Whether to finalize or close cfile blocks when writing is finished." Done http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/block_manager-test.cc File src/kudu/fs/block_manager-test.cc: Line 998: TYPED_TEST(BlockManagerTest, ConcurrentCloseFinalizedWritableBlockTest) { > kTestData for this. Above and below too. Done Line 1002: // written, and then Close() it. > Since these operations happen on other threads, you should use CHECK_OK ins Done Line 1007: CHECK_OK(writer->Append(kTestData)); > Maybe inject a tiny (random) bit of sleeping between Finalize() and Close() Done PS10, Line 1010: CHECK_OK(writer->Close()); : } : }; : : vector threads; : for (int i = 0; i < 100; i++) { : threads.emplace_back(write_data); : } : for (auto& t : threads) { : > Use std::thread; they're less LOC and thus better suited for tests. Done Line 1032: ASSERT_OK(block->Read(0, &slice)); > ASSERT_EQ Done PS10, Line 1043: a > "some" instead, so that if 20 changes to 30 or 40, we don't have to update Done Line 1053: ASSERT_OK(writer->Append(kTestData)); > What's this about? My bad, comment it out for testing, but forget to uncomment back. http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/block_manager.h File src/kudu/fs/block_manager.h: Line 22: #include > What happened here? The older style (and location) was correct. Done PS10, Line 73: 1) flushing of dirty : //blocks a > Not sure what this is saying; please reword. Done Line 88: // No more data may be written to the block, but it is not yet closed. > "No more data may be written to the block, but it is not yet closed." Done PS10, Line 123: // Signals that the block will no longer receive writes. Does not guarantee : // durability; Close() must still be called for t > "Signals that the block will no longer receive writes. Does not guarantee d Done PS10, Line 275: block > block Done PS10, Line 276: 1) the underlying block manager can optimize > Expand on this a bit (by describing in vagaries that the underlying block m Done PS10, Line 277: possi > logical Done Line 279: class BlockTransaction { > Nit: indent by one char. Done Line 312: } > Nit: indent by one char. Done http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/file_block_manager.cc File src/kudu/fs/file_block_manager.cc: PS10, Line 335: : if (state_ == FINALIZED) { : return Status::OK(); : } : VLOG(3) << "Finalizing block " << id(); : if (state_ == DIRTY) { : if (FLAGS_block_manager_flush_on_finalize) { : > How about inverting this: Done http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/fs_manager.cc File src/kudu/fs/fs_manager.cc: Line 38: #include "kudu/gutil/stringprintf.h" > Why is this include needed? Done http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/log_block_manager-test.cc File src/kudu/fs/log_block_manager-test.cc: Line 1274: } > I'd also close all the blocks in 'blocks' and check all_containers_by_name_ Done Line 1277: for (const auto& block : blocks) { > Isn't this state transition already tested by BlockManagerTest.WritableBloc I combined this test with the test case above. Let me know if you still think it is redundant. Thanks! Line 1288: } // namespace fs > How does L1289 check this? I was trying to check in the available_containers_by_data_dir_ map, there is only one container for the corresponding datadir(should be the first element of the map). Maybe there is a more elegant way to do so? http://gerrit.cloudera.org:8080/#/c/7207/10/src/kudu/fs/log_block_manager.cc File src/kudu/fs/log_block_manager.cc: Line 129: using pb_util::WritablePBContainerFile; > There's an existing internal::LogWritableBlock reference in the file; pleas Done Line 513: // The metrics. Not owned by the log container; it has the same lifespan > Please use AtomicInt rather than std::atomic, since that's what's already b Done Line 984: // is we sync the data/metadata more often than needed. > We shouldn't hold spinlocks while doing IO, because
[kudu-CR] Expose running maintenance op info
Andrew Wong has posted comments on this change. Change subject: Expose running maintenance op info .. Patch Set 8: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 8 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] Expose running maintenance op info
Jean-Daniel Cryans has posted comments on this change. Change subject: Expose running maintenance op info .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 8 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] Expose running maintenance op info
Jean-Daniel Cryans has submitted this change and it was merged. Change subject: Expose running maintenance op info .. Expose running maintenance op info Previously, the maintenance manager stored the number of instances a given operation has running, but information like the start time of an instance was not made available until the instance completed. However, the start times of running instances can be helpful information to display on the web UI. This commit adds a collection of running instances to the MaintenanceManager so the information mentioned above can be accessed while operations are running. Additionally, instances of operations now store the id of the thread on which they run. Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Reviewed-on: http://gerrit.cloudera.org:8080/7537 Tested-by: Kudu Jenkins Reviewed-by: Andrew Wong Reviewed-by: Jean-Daniel Cryans --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/util/maintenance_manager-test.cc M src/kudu/util/maintenance_manager.cc M src/kudu/util/maintenance_manager.h M src/kudu/util/maintenance_manager.proto 5 files changed, 114 insertions(+), 46 deletions(-) Approvals: Jean-Daniel Cryans: Looks good to me, approved Andrew Wong: Looks good to me, but someone else must approve Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/7537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ide228d7e70deae3ae89d108cbd270f3f0f2580ca Gerrit-PatchSet: 9 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon
[kudu-CR] [tests] fix compilation warning on OS X
Alexey Serbin has uploaded a new change for review. http://gerrit.cloudera.org:8080/7594 Change subject: [tests] fix compilation warning on OS X .. [tests] fix compilation warning on OS X Fixed compilation warning on OS X due to unused function: CountBlocksUnderManagement(). This changelist does not contain any functional modifications. Change-Id: I836453fd986589e53fcceaa19599eba615074e53 --- M src/kudu/integration-tests/tablet_copy-itest.cc 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/7594/1 -- To view, visit http://gerrit.cloudera.org:8080/7594 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I836453fd986589e53fcceaa19599eba615074e53 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Alexey Serbin
[kudu-CR] [tests] fix compilation warning on OS X
Hao Hao has posted comments on this change. Change subject: [tests] fix compilation warning on OS X .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/7594 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I836453fd986589e53fcceaa19599eba615074e53 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-HasComments: No
[kudu-CR] Add Maintenance Manager visualizer
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7570 to look at the new patch set (#5). Change subject: Add Maintenance Manager visualizer .. Add Maintenance Manager visualizer Adds a timeline/swimlane chart to display maintenance manager operations to the /maintenance-manager page of the tserver web UI. Previously, completed and running ops were displayed in two tables. Now, both running and completed ops are displayed as color-coded bars in parallel lanes representing threads. A screenshot of the new maintenance-manager page can be found here: http://imgur.com/gallery/BVJAV. Also switches the /maintenance-manager page over to a mustache template. Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/tserver/tserver-path-handlers.h A www/maintenance-manager.mustache 3 files changed, 409 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/7570/5 -- To view, visit http://gerrit.cloudera.org:8080/7570 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Todd Lipcon
[kudu-CR] Add Maintenance Manager visualizer
Jean-Daniel Cryans has posted comments on this change. Change subject: Add Maintenance Manager visualizer .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/7570/5/www/maintenance-manager.mustache File www/maintenance-manager.mustache: Line 326: // If new operation is added, its name should be added to this list, Mmm do we have a comment where those classes are defined about not forgetting to modify this file? -- To view, visit http://gerrit.cloudera.org:8080/7570 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: Yes
[kudu-CR] log block manager: Reorder class declaration of LogWritableBlock
Hao Hao has uploaded a new change for review. http://gerrit.cloudera.org:8080/7595 Change subject: log block manager: Reorder class declaration of LogWritableBlock .. log block manager: Reorder class declaration of LogWritableBlock Reordered class declaration of LogWritableBlock to avoid the intermingling of LogBlockContainer and LogWritableBlock definitions in future when adding new methods. Change-Id: I0a5192cccac1931fd1f65b11fa0739090e8368e2 --- M src/kudu/fs/log_block_manager.cc 1 file changed, 71 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/7595/1 -- To view, visit http://gerrit.cloudera.org:8080/7595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0a5192cccac1931fd1f65b11fa0739090e8368e2 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Hao Hao
[kudu-CR] log block manager: Reorder class declaration of LogWritableBlock
Hao Hao has uploaded a new patch set (#2). Change subject: log block manager: Reorder class declaration of LogWritableBlock .. log block manager: Reorder class declaration of LogWritableBlock Reordered class declaration of LogWritableBlock to avoid the intermingling of LogBlockContainer and LogWritableBlock definitions in future when adding new methods. Change-Id: I0a5192cccac1931fd1f65b11fa0739090e8368e2 --- M src/kudu/fs/log_block_manager.cc 1 file changed, 71 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/95/7595/2 -- To view, visit http://gerrit.cloudera.org:8080/7595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0a5192cccac1931fd1f65b11fa0739090e8368e2 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Hao Hao Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] KUDU-1943: Add BlockTransaction to Block Manager
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7207 to look at the new patch set (#13). Change subject: KUDU-1943: Add BlockTransaction to Block Manager .. KUDU-1943: Add BlockTransaction to Block Manager This adds a new layer of abstraction 'BlockTransaction' to Block Manager, to accumulate creates which happen in a logical operation, e.g. flush/compaction. Avoid using fsync() per block can potentionaly reduce a lot I/O overhead. This patch also add a new API FinalizeWrite() to Block Manager, so that for LBM container can be resued, without flushing in-flight writable blocks to disk. A design doc can be found here: https://docs.google.com/a/cloudera.com/document/d/15zZaKeLENRGC_AdjF2N3hxtM_6_OuydXA1reWvDbbK0/edit?usp=sharing Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd --- M src/kudu/cfile/bloomfile.cc M src/kudu/cfile/bloomfile.h M src/kudu/cfile/cfile-test.cc M src/kudu/cfile/cfile_writer.cc M src/kudu/cfile/cfile_writer.h M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager.cc M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/tablet/deltafile.cc M src/kudu/tablet/deltafile.h M src/kudu/tablet/diskrowset.cc M src/kudu/tablet/diskrowset.h M src/kudu/tablet/multi_column_writer.cc M src/kudu/tablet/multi_column_writer.h 18 files changed, 657 insertions(+), 262 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/07/7207/13 -- To view, visit http://gerrit.cloudera.org:8080/7207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I01f3ed830cc1f86d7bb8a66b2f780914a4bd Gerrit-PatchSet: 13 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Dan Burkert Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot
[kudu-CR] Add Maintenance Manager visualizer
Hello Kudu Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/7570 to look at the new patch set (#6). Change subject: Add Maintenance Manager visualizer .. Add Maintenance Manager visualizer Adds a timeline/swimlane chart to display maintenance manager operations to the /maintenance-manager page of the tserver web UI. Previously, completed and running ops were displayed in two tables. Now, both running and completed ops are displayed as color-coded bars in parallel lanes representing threads. A screenshot of the new maintenance-manager page can be found here: http://imgur.com/gallery/qxiHa. Also switches the /maintenance-manager page over to a mustache template. Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 --- M src/kudu/tserver/tserver-path-handlers.cc M src/kudu/tserver/tserver-path-handlers.h M src/kudu/util/maintenance_manager.h A www/maintenance-manager.mustache 4 files changed, 413 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/7570/6 -- To view, visit http://gerrit.cloudera.org:8080/7570 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: If2e7f18ac4834791a94d935b540930b11ea14532 Gerrit-PatchSet: 6 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Sam Okrent Gerrit-Reviewer: Todd Lipcon
[kudu-CR] Test Prepare() failed codepath
Sam Okrent has abandoned this change. Change subject: Test Prepare() failed codepath .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/7522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: abandon Gerrit-Change-Id: I8e69700121c78b3590dcc48743a8a43e427c862e Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sam Okrent Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] Clean up spurious declarations in messenger.cc
Todd Lipcon has posted comments on this change. Change subject: Clean up spurious declarations in messenger.cc .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/7583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I105524bab85bef9c08205e3d1f262fa5234745d9 Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-HasComments: No
[kudu-CR] Clean up spurious declarations in messenger.cc
Todd Lipcon has submitted this change and it was merged. Change subject: Clean up spurious declarations in messenger.cc .. Clean up spurious declarations in messenger.cc Change-Id: I105524bab85bef9c08205e3d1f262fa5234745d9 Reviewed-on: http://gerrit.cloudera.org:8080/7583 Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon --- M src/kudu/rpc/messenger.cc 1 file changed, 0 insertions(+), 3 deletions(-) Approvals: Todd Lipcon: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/7583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I105524bab85bef9c08205e3d1f262fa5234745d9 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon