[kudu] branch master updated: [docs] add blurb about automatic master addition
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new 0335f3e [docs] add blurb about automatic master addition 0335f3e is described below commit 0335f3ea6356011ac88c25a9a353b4fbec952a42 Author: Andrew Wong AuthorDate: Fri Feb 25 13:54:37 2022 -0800 [docs] add blurb about automatic master addition The steps are updated in the latest version of Kudu, and this patch adds a note indicating to readers that they do not need to go through the administrative docs. Change-Id: I15ada3ce0a4319a1e1b5340bdfb7a261ef260f8d Reviewed-on: http://gerrit.cloudera.org:8080/18278 Tested-by: Kudu Jenkins Reviewed-by: Attila Bukor --- docs/administration.adoc | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/administration.adoc b/docs/administration.adoc index 113c1c3..2ba1525 100644 --- a/docs/administration.adoc +++ b/docs/administration.adoc @@ -521,6 +521,13 @@ or because Kudu multi-master support was still experimental at the time. This wo how to migrate to a multi-master configuration. It can also be used to migrate from two masters to three, with straightforward modifications. Note that the number of masters must be odd. +NOTE: From Kudu version 1.16.0 onwards, new Kudu Masters automatically attempt to contact other +masters, determine whether a quorum has already been established, and if so, attempt to join the +quorum. The below steps do not need to be performed in these versions. + +NOTE: From Kudu version 1.15.0 onwards, the `kudu master add` command has been added that simplifies +the orchestration to migrate an existing Kudu cluster to multiple masters. + WARNING: The workflow is unsafe for adding new masters to an existing configuration that already has three or more masters. Do not use it for that purpose. @@ -534,9 +541,6 @@ WARNING: The workflow presupposes at least basic familiarity with Kudu configura using vendor-specific tools the workflow also presupposes familiarity with it and the vendor's instructions should be used instead as details may differ. -NOTE: From Kudu version 1.15.0, a new `kudu master add` command has been added that -simplifies the orchestration to migrate an existing Kudu cluster to multiple masters. - Prepare for the migration . Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster
[kudu] branch branch-1.16.x updated: [docs] add blurb about automatic master addition
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch branch-1.16.x in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/branch-1.16.x by this push: new baa5c8b [docs] add blurb about automatic master addition baa5c8b is described below commit baa5c8b23c1d49b48022337a082fc1c69c81d3e0 Author: Andrew Wong AuthorDate: Fri Feb 25 13:54:37 2022 -0800 [docs] add blurb about automatic master addition The steps are updated in the latest version of Kudu, and this patch adds a note indicating to readers that they do not need to go through the administrative docs. Change-Id: I15ada3ce0a4319a1e1b5340bdfb7a261ef260f8d Reviewed-on: http://gerrit.cloudera.org:8080/18278 Tested-by: Kudu Jenkins Reviewed-by: Attila Bukor (cherry picked from commit cb016a09994f4b6cf89e62f76b5a5b4bc2a0a3ad) Reviewed-on: http://gerrit.cloudera.org:8080/18279 --- docs/administration.adoc | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/administration.adoc b/docs/administration.adoc index 456665f..995fb12 100644 --- a/docs/administration.adoc +++ b/docs/administration.adoc @@ -515,6 +515,13 @@ or because Kudu multi-master support was still experimental at the time. This wo how to migrate to a multi-master configuration. It can also be used to migrate from two masters to three, with straightforward modifications. Note that the number of masters must be odd. +NOTE: From Kudu version 1.16.0 onwards, new Kudu Masters automatically attempt to contact other +masters, determine whether a quorum has already been established, and if so, attempt to join the +quorum. The below steps do not need to be performed in these versions. + +NOTE: From Kudu version 1.15.0 onwards, the `kudu master add` command has been added that simplifies +the orchestration to migrate an existing Kudu cluster to multiple masters. + WARNING: The workflow is unsafe for adding new masters to an existing configuration that already has three or more masters. Do not use it for that purpose. @@ -528,9 +535,6 @@ WARNING: The workflow presupposes at least basic familiarity with Kudu configura using vendor-specific tools the workflow also presupposes familiarity with it and the vendor's instructions should be used instead as details may differ. -NOTE: From Kudu version 1.15.0, a new `kudu master add` command has been added that -simplifies the orchestration to migrate an existing Kudu cluster to multiple masters. - Prepare for the migration . Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster
[kudu] branch master updated: KUDU-2915: add tool to unregister a tablet server
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 9d01e10 KUDU-2915: add tool to unregister a tablet server
9d01e10 is described below
commit 9d01e1046249a815f26c7b5ebb1ceb2b67f72b9e
Author: zhangyifan27
AuthorDate: Wed Dec 29 15:38:40 2021 +0800
KUDU-2915: add tool to unregister a tablet server
Add a 'kudu tserver unregister' tool to unregister a tserver from the
master. This tool will be useful when we want to decommission a tserver
without restarting masters.
This tool unregisters the dead tserver from master's in-memory map and
removes its persisted state from catalog table by default. It's also
possible to unregister a tserver which is not presumed dead by adding
'-force_unregister_live_tserver', or keep tserver's persisted state
by adding '-remove_tserver_state=false'.
Change-Id: If1f5c2979a8d14428f4bcc8e850c57ce228c793a
Reviewed-on: http://gerrit.cloudera.org:8080/18124
Reviewed-by: Alexey Serbin
Reviewed-by: Andrew Wong
Tested-by: Kudu Jenkins
---
src/kudu/master/catalog_manager.cc| 1 +
src/kudu/master/master.proto | 20 +
src/kudu/master/master_service.cc | 16
src/kudu/master/master_service.h | 6 ++
src/kudu/master/ts_manager.cc | 15
src/kudu/master/ts_manager.h | 3 +
src/kudu/tools/kudu-tool-test.cc | 137 ++
src/kudu/tools/tool_action_common.cc | 62 +++
src/kudu/tools/tool_action_common.h | 5 ++
src/kudu/tools/tool_action_master.cc | 63
src/kudu/tools/tool_action_tserver.cc | 54 ++
11 files changed, 319 insertions(+), 63 deletions(-)
diff --git a/src/kudu/master/catalog_manager.cc
b/src/kudu/master/catalog_manager.cc
index 94fb6ac..0a2a830 100644
--- a/src/kudu/master/catalog_manager.cc
+++ b/src/kudu/master/catalog_manager.cc
@@ -6231,6 +6231,7 @@ bool
CatalogManager::ScopedLeaderSharedLock::CheckIsInitializedAndIsLeaderOrResp
INITTED_OR_RESPOND(ConnectToMasterResponsePB);
INITTED_OR_RESPOND(GetMasterRegistrationResponsePB);
+INITTED_OR_RESPOND(UnregisterTServerResponsePB);
INITTED_OR_RESPOND(TSHeartbeatResponsePB);
INITTED_AND_LEADER_OR_RESPOND(AddMasterResponsePB);
INITTED_AND_LEADER_OR_RESPOND(AlterTableResponsePB);
diff --git a/src/kudu/master/master.proto b/src/kudu/master/master.proto
index b9ded52..2967f24 100644
--- a/src/kudu/master/master.proto
+++ b/src/kudu/master/master.proto
@@ -1070,6 +1070,22 @@ message RefreshAuthzCacheResponsePB {
optional MasterErrorPB error = 1;
}
+// RemoveTabletServerRequest/Response: remove a tablet server from master's
+// in-memory map and persisted catalog.
+message UnregisterTServerRequestPB {
+ // The tserver UUID to be unregistered.
+ optional string uuid = 1;
+
+ // Whether to return an error in case the tserver is not presumed to be dead,
+ // per --tserver_unresponsive_timeout_ms.
+ // Disable this by default to make sure the tserver has been brought down.
+ optional bool force_unregister_live_tserver = 2 [default = false];
+}
+
+message UnregisterTServerResponsePB {
+ optional MasterErrorPB error = 1;
+}
+
enum MasterFeatures {
UNKNOWN_FEATURE = 0;
// The master supports creating tables with non-covering range partitions.
@@ -1184,6 +1200,10 @@ service MasterService {
option (kudu.rpc.authz_method) = "AuthorizeSuperUser";
}
+ rpc UnregisterTServer(UnregisterTServerRequestPB) returns
(UnregisterTServerResponsePB) {
+option (kudu.rpc.authz_method) = "AuthorizeSuperUser";
+ }
+
// Master->Master RPCs
//
diff --git a/src/kudu/master/master_service.cc
b/src/kudu/master/master_service.cc
index 71d84f2..0d0305d 100644
--- a/src/kudu/master/master_service.cc
+++ b/src/kudu/master/master_service.cc
@@ -331,6 +331,22 @@ void MasterServiceImpl::RemoveMaster(const
RemoveMasterRequestPB* req,
// See completion_cb in CatalogManager::InitiateMasterChangeConfig().
}
+void MasterServiceImpl::UnregisterTServer(const UnregisterTServerRequestPB*
req,
+ UnregisterTServerResponsePB* resp,
+ rpc::RpcContext* rpc) {
+ const auto& ts_uuid = req->uuid();
+ bool force_unregister_live_tserver = req->force_unregister_live_tserver();
+
+ Status s = server_->ts_manager()->UnregisterTServer(ts_uuid,
force_unregister_live_tserver);
+ if (PREDICT_FALSE(!s.ok() && !s.IsNotFound())) {
+// Ignore the NotFound error to make this RPC retriable and effectively
idempotent.
+StatusToPB(s, resp->mutable_error()->mutable_status());
+resp->mutable_error()->set_code(MasterErrorPB::
[kudu] 02/02: [dense_node-itest] enable on macOS
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit cac9b48ccee922b5d88cac20f496cd77b409f0bc
Author: Andrew Wong
AuthorDate: Thu Nov 18 18:38:43 2021 -0800
[dense_node-itest] enable on macOS
This patch addresses a TODO to enable this test on macOS once the file
cache is used for WAL segments. This occurred in commit
fc4ab691a502067bc4d5bdff30507cac7feb7cfe.
Since macOS doesn't currently support the LogBlockManager, this also
updates the test to look for some BlockManager-agnostic metrics if not
using the LogBlockManager.
I also found we can add more block-related stress to the test scenario
by disabling compactions, which I've done optionally (though leaving the
default behavior the same).
Change-Id: I240331c4c9293d4f449d52b6a4ed2392a6ec9f3c
Reviewed-on: http://gerrit.cloudera.org:8080/18055
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/integration-tests/CMakeLists.txt | 5 +---
src/kudu/integration-tests/dense_node-itest.cc | 36 +-
2 files changed, 30 insertions(+), 11 deletions(-)
diff --git a/src/kudu/integration-tests/CMakeLists.txt
b/src/kudu/integration-tests/CMakeLists.txt
index c687930..9e2f8ad 100644
--- a/src/kudu/integration-tests/CMakeLists.txt
+++ b/src/kudu/integration-tests/CMakeLists.txt
@@ -76,6 +76,7 @@ ADD_KUDU_TEST(create-table-stress-test RUN_SERIAL true)
ADD_KUDU_TEST(decimal-itest)
ADD_KUDU_TEST(delete_table-itest NUM_SHARDS 8 PROCESSORS 4)
ADD_KUDU_TEST(delete_tablet-itest PROCESSORS 2)
+ADD_KUDU_TEST(dense_node-itest RUN_SERIAL true)
ADD_KUDU_TEST(disk_failure-itest PROCESSORS 2)
ADD_KUDU_TEST(disk_reservation-itest)
ADD_KUDU_TEST(dns_alias-itest)
@@ -151,10 +152,6 @@ ADD_KUDU_TEST(write_throttling-itest)
if (NOT APPLE)
ADD_KUDU_TEST(minidump_generation-itest)
- # Opens too many files on macOS because we're not using a file cache for
- # reading log segments.
- # TODO(wdb): Re-enable once a file cache is being used.
- ADD_KUDU_TEST(dense_node-itest RUN_SERIAL true)
endif()
# Tests that should not be run automatically by ctest.
diff --git a/src/kudu/integration-tests/dense_node-itest.cc
b/src/kudu/integration-tests/dense_node-itest.cc
index b75cd09..2e37625 100644
--- a/src/kudu/integration-tests/dense_node-itest.cc
+++ b/src/kudu/integration-tests/dense_node-itest.cc
@@ -19,7 +19,6 @@
#include
#include
-#include
#include
#include
#include
@@ -31,6 +30,7 @@
#include
#include "kudu/client/schema.h"
+#include "kudu/gutil/integral_types.h"
#include "kudu/gutil/strings/substitute.h"
#include "kudu/integration-tests/cluster_itest_util.h"
#include "kudu/integration-tests/external_mini_cluster-itest-base.h"
@@ -44,12 +44,18 @@
#include "kudu/util/test_macros.h"
METRIC_DECLARE_entity(server);
+METRIC_DECLARE_gauge_uint64(block_manager_total_blocks_created);
+METRIC_DECLARE_gauge_uint64(block_manager_total_bytes_written);
+METRIC_DECLARE_gauge_uint64(block_manager_total_disk_sync);
METRIC_DECLARE_gauge_uint64(log_block_manager_blocks_under_management);
METRIC_DECLARE_gauge_uint64(log_block_manager_bytes_under_management);
METRIC_DECLARE_gauge_uint64(log_block_manager_containers);
METRIC_DECLARE_gauge_uint64(log_block_manager_full_containers);
METRIC_DECLARE_gauge_uint64(threads_running);
+DECLARE_bool(enable_rowset_compaction);
+DECLARE_string(block_manager);
+
DEFINE_bool(measure_startup_drop_caches, false,
"Whether to drop kernel caches before measuring startup time. Must
be root");
DEFINE_bool(measure_startup_sync, false,
@@ -109,9 +115,15 @@ TEST_F(DenseNodeTest, RunTest) {
FLAGS_max_blocks_per_container),
// UNDO delta block GC runs a lot to eagerly open newly created cfiles.
- // Disable it so we can maximize flushes and compactions.
+ // Disable it so we can maximize flushes.
"--enable_undo_delta_block_gc=false",
+ // Compactions aim to reduce the number of rowsets and thus reduce the
+ // number of cfiles. Potentially disable them so we can maximize the
+ // number of files. NOTE: this also means we don't exercise the block
+ // deletions that come with compactions.
+ Substitute("--enable_rowset_compaction=$0",
FLAGS_enable_rowset_compaction),
+
// Allow our single tserver to service many, many RPCs.
"--rpc_service_queue_length=1000",
@@ -177,11 +189,21 @@ TEST_F(DenseNodeTest, RunTest) {
// Collect some interesting metrics. The cluster is shut down before the
// metrics are logged so that they're easier to find in the log output.
vector> metrics;
- for (const auto* m : { _log_block_manager_blocks_under_management,
-
[kudu] branch master updated (e266f71 -> cac9b48)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from e266f71 [benchmarks] add MetaCacheLookupStressTest scenarios new 6d81fe4 KUDU-1959 - Add tests for /startup page and metrics for tservers new cac9b48 [dense_node-itest] enable on macOS The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/integration-tests/CMakeLists.txt | 5 +- src/kudu/integration-tests/dense_node-itest.cc | 36 - src/kudu/mini-cluster/webui_checker.h | 2 + src/kudu/server/webserver.cc | 14 +- src/kudu/tserver/tablet_server-test.cc | 207 + 5 files changed, 249 insertions(+), 15 deletions(-)
[kudu] 01/02: KUDU-1959 - Add tests for /startup page and metrics for tservers
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 6d81fe44b51844942bc8433931d663531547c4b8
Author: Abhishek Chennaka
AuthorDate: Mon Nov 1 21:25:33 2021 -0400
KUDU-1959 - Add tests for /startup page and metrics for tservers
This patch implements the tests for the startup page using mini tablet
server.
- We inject latency to bootstrap tablets while reading the webpage every
10 milliseconds and validating the status for each step.
- Fail a data directory and validate the status of each startup step.
We also validate the below startup metrics in the above scenarios
(log_block_manager* metrics in the case of using log block manager):
- log_block_manager_total_containers_startup
- log_block_manager_processed_containers_startup
- log_block_manager_containers_processing_time_startup
- tablets_num_total_startup
- tablets_num_opened_startup
- tablets_opening_time_startup
Additionally we also fix a race condition in the Kudu tablet server
WebUI. This race condition occurs if the tablet server is started while
the WebUI is continuously curled. The reason appears to be starting up
of the webserver before registering the path handlers as a part of
the change https://gerrit.cloudera.org/#/c/17730/
Change-Id: I9f432b4eb813e51214b4d6b3c5b7b4c89426f47f
Reviewed-on: http://gerrit.cloudera.org:8080/17990
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/mini-cluster/webui_checker.h | 2 +
src/kudu/server/webserver.cc | 14 ++-
src/kudu/tserver/tablet_server-test.cc | 207 +
3 files changed, 219 insertions(+), 4 deletions(-)
diff --git a/src/kudu/mini-cluster/webui_checker.h
b/src/kudu/mini-cluster/webui_checker.h
index f9a46ee..315fe68 100644
--- a/src/kudu/mini-cluster/webui_checker.h
+++ b/src/kudu/mini-cluster/webui_checker.h
@@ -40,6 +40,7 @@ class PeriodicWebUIChecker {
"/mem-trackers",
"/metrics",
"/stacks",
+ "/startup",
"/tables",
"/tablet-servers",
"/threadz",
@@ -50,6 +51,7 @@ class PeriodicWebUIChecker {
"/metrics",
"/scans",
"/stacks",
+ "/startup",
"/tablets",
"/threadz",
"/threadz?group=all",
diff --git a/src/kudu/server/webserver.cc b/src/kudu/server/webserver.cc
index 2f57490..a9253de 100644
--- a/src/kudu/server/webserver.cc
+++ b/src/kudu/server/webserver.cc
@@ -72,6 +72,7 @@ typedef sig_t sighandler_t;
using mustache::RenderTemplate;
using std::ostringstream;
+using std::pair;
using std::stringstream;
using std::string;
using std::unordered_set;
@@ -806,17 +807,22 @@ void Webserver::RenderMainTemplate(
ej["static_pages_available"] = static_pages_available();
ej["content"] = content;
AddKnoxVariables(req, );
+ std::vector> paths_and_handlers;
{
shared_lock l(lock_);
ej["footer_html"] = footer_html_;
+paths_and_handlers.reserve(path_handlers_.size());
+for (const auto& [path, handler] : path_handlers_) {
+ paths_and_handlers.emplace_back(path, handler);
+}
}
EasyJson path_handlers = ej.Set("path_handlers", EasyJson::kArray);
- for (const PathHandlerMap::value_type& handler : path_handlers_) {
-if (handler.second->is_on_nav_bar()) {
+ for (const auto& [path, handler] : paths_and_handlers) {
+if (handler->is_on_nav_bar()) {
EasyJson path_handler = path_handlers.PushBack(EasyJson::kObject);
- path_handler["path"] = handler.first;
- path_handler["alias"] = handler.second->alias();
+ path_handler["path"] = path;
+ path_handler["alias"] = handler->alias();
}
}
RenderTemplate(kMainTemplate, opts_.doc_root, ej.value(), output);
diff --git a/src/kudu/tserver/tablet_server-test.cc
b/src/kudu/tserver/tablet_server-test.cc
index ffd3849..4ea9ac2 100644
--- a/src/kudu/tserver/tablet_server-test.cc
+++ b/src/kudu/tserver/tablet_server-test.cc
@@ -199,6 +199,7 @@ DECLARE_int32(rpc_service_queue_length);
DECLARE_int32(scanner_batch_size_rows);
DECLARE_int32(scanner_gc_check_interval_us);
DECLARE_int32(scanner_ttl_ms);
+DECLARE_int32(tablet_bootstrap_inject_latency_ms);
DECLARE_int32(tablet_inject_latency_on_apply_write_op_ms);
DECLARE_int32(workload_stats_rate_collection_min_interv
[kudu] branch master updated: [master] KUDU-3311 Allow to start with diff num of masters
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new e3bcd74 [master] KUDU-3311 Allow to start with diff num of masters
e3bcd74 is described below
commit e3bcd743d6c2d8791591c69260e7d9e6bc59fef9
Author: Zoltan Chovan
AuthorDate: Thu Oct 28 14:32:40 2021 +0200
[master] KUDU-3311 Allow to start with diff num of masters
Modified SysCatalogTable:Load to allow cluster startup if +1
master address is present in the flags.
The startup is halted, if the number of masters on-disk is higher than
what was provided in the flags, discouraging decommissioning masters
this way.
Additional manual test was done with the following steps in order to
ensure, that the documented way of removing a master works:
- ./start_kudu.sh -m 4 -t 3 -b $KUDU_BUILD_DIR -c /tmp/data --rpc-master
8800
- looked up the "last" master which was listening on :8006 and removed
it by running:
./kudu master remove
127.0.0.1:8800,127.0.0.1:8802,127.0.0.1:8804,127.0.0.1:8806 127.0.0.1:8806
- stopped the cluster by running ./stop_kudu.sh
- restarted with 3 masters by running
./start_kudu.sh -m 3 -t 3 -b $KUDU_BUILD_DIR -c /tmp/data --rpc-master
8800
Change-Id: I39aeee2f52a55a8c29770f748895d38c9adff8a2
Reviewed-on: http://gerrit.cloudera.org:8080/17995
Reviewed-by: Andrew Wong
Tested-by: Kudu Jenkins
---
.../integration-tests/master_replication-itest.cc | 4 +-
src/kudu/master/dynamic_multi_master-test.cc | 57 ++
src/kudu/master/master_options-test.cc | 55 -
src/kudu/master/sys_catalog.cc | 33 ++---
src/kudu/mini-cluster/external_mini_cluster.h | 7 ++-
5 files changed, 141 insertions(+), 15 deletions(-)
diff --git a/src/kudu/integration-tests/master_replication-itest.cc
b/src/kudu/integration-tests/master_replication-itest.cc
index b7732ea..5995fa8 100644
--- a/src/kudu/integration-tests/master_replication-itest.cc
+++ b/src/kudu/integration-tests/master_replication-itest.cc
@@ -307,17 +307,19 @@ TEST_F(MasterReplicationTest,
TestHeartbeatAcceptedByAnyMaster) {
}
TEST_F(MasterReplicationTest, TestMasterPeerSetsDontMatch) {
- // Restart one master with an additional entry in --master_addresses. The
+ // Restart one master with two additional entries in --master_addresses. The
// discrepancy with the on-disk list of masters should trigger a failure.
vector master_rpc_addrs = cluster_->master_rpc_addrs();
cluster_->mini_master(0)->Shutdown();
master_rpc_addrs.emplace_back("127.0.0.1", 5);
+ master_rpc_addrs.emplace_back("127.0.0.1", 6);
cluster_->mini_master(0)->SetMasterAddresses(master_rpc_addrs);
ASSERT_OK(cluster_->mini_master(0)->Start());
Status s = cluster_->mini_master(0)->WaitForCatalogManagerInit();
SCOPED_TRACE(s.ToString());
ASSERT_TRUE(s.IsInvalidArgument());
ASSERT_STR_CONTAINS(s.ToString(), "5");
+ ASSERT_STR_CONTAINS(s.ToString(), "6");
}
TEST_F(MasterReplicationTest, TestConnectToClusterReturnsAddresses) {
diff --git a/src/kudu/master/dynamic_multi_master-test.cc
b/src/kudu/master/dynamic_multi_master-test.cc
index 71b278f..147707c 100644
--- a/src/kudu/master/dynamic_multi_master-test.cc
+++ b/src/kudu/master/dynamic_multi_master-test.cc
@@ -35,6 +35,7 @@
#include
#include
#include
+#include
#include "kudu/client/client.h"
#include "kudu/client/schema.h"
@@ -1713,6 +1714,62 @@ TEST_F(AutoAddMasterTest, TestAddWithOnGoingDdl) {
}
}
+TEST_F(AutoAddMasterTest, TestAddNewMaster) {
+ SKIP_IF_SLOW_NOT_ALLOWED();
+
+ // Let's get the current master addresses and add a new one to them
+ vector master_addrs = cluster_->master_rpc_addrs();
+ unique_ptr reserved_socket;
+ ASSERT_OK(MiniCluster::ReserveDaemonSocket(
+ MiniCluster::DaemonType::MASTER,
+ master_addrs.size(),
+ opts_.bind_mode,
+ _socket));
+
+ Sockaddr addr;
+ ASSERT_OK(reserved_socket->GetSocketAddress());
+ master_addrs.emplace_back(addr.host(), addr.port());
+
+ const auto& new_master_addrs_list =
+ HostPort::ToCommaSeparatedString(master_addrs);
+
+ cluster_->Shutdown();
+
+ // We have to remove the previous '--master_addresses' flag to ensure there
+ // are no duplicates, then we set the new addresses on the existing masters.
+ for (int i = 0; i < cluster_->num_masters(); i++) {
+auto* flags = cluster_->master(i)->mutable_flags();
+flags->erase(
+remove_if(flags->begin(),
+ flags->end(),
+ [](const string ) {
+return HasPrefixString(s, "--master_addresses=")
[kudu] 02/02: [LBM] Speed up server bootstrap by using multi-thread to compact containers
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 032d1d90147139db62a7cea8700d5373332bde9a
Author: Yingchun Lai
AuthorDate: Mon Oct 18 18:53:06 2021 +0800
[LBM] Speed up server bootstrap by using multi-thread to compact containers
When a data directory has many low live block containers, a single thread to
compact these container metadata files is low efficiency, we can improve it
by
multi-threads.
In some test cases, e.g. one or very few data directories, high latency disk
drivers, a large amount of low live block containers, and with a multi-core
CPU, this patch could reduce about 85% time in metadata files compact
procedure when bootstrap.
Change-Id: Ie48211d9e8c1d74e520fcb04df25c1d681261bb5
Reviewed-on: http://gerrit.cloudera.org:8080/17942
Reviewed-by: Alexey Serbin
Tested-by: Alexey Serbin
---
src/kudu/fs/log_block_manager-test.cc | 52 ---
src/kudu/fs/log_block_manager.cc | 80 ++-
2 files changed, 106 insertions(+), 26 deletions(-)
diff --git a/src/kudu/fs/log_block_manager-test.cc
b/src/kudu/fs/log_block_manager-test.cc
index 6af5992..9bd6e24 100644
--- a/src/kudu/fs/log_block_manager-test.cc
+++ b/src/kudu/fs/log_block_manager-test.cc
@@ -92,6 +92,12 @@ DEFINE_int32(startup_benchmark_block_count_for_testing,
100,
"Block count to do startup benchmark.");
DEFINE_int32(startup_benchmark_data_dir_count_for_testing, 8,
"Data directories to do startup benchmark.");
+DEFINE_int32(startup_benchmark_reopen_times, 10,
+ "Block manager reopen times.");
+DEFINE_int32(startup_benchmark_deleted_block_percentage, 90,
+ "Percentage of deleted blocks in containers.");
+DEFINE_validator(startup_benchmark_deleted_block_percentage,
+ [](const char* /*n*/, int32_t v) { return 0 <= v && v <= 100;
});
// Block manager metrics.
METRIC_DECLARE_counter(block_manager_total_blocks_deleted);
@@ -1008,15 +1014,28 @@ TEST_F(LogBlockManagerTest, TestParseKernelRelease) {
// Simple micro-benchmark which creates a large number of blocks and then
// times the startup of the LBM.
//
-// This is simplistic in several ways compared to a typical workload:
-// - minimal number of containers, each of which is entirely full
+// This is simplistic in several ways compared to two typical workloads:
+// 1. minimal number of containers, each of which is entirely full
+//without any deleted blocks.
// (typical workloads end up writing to several containers at once
//due to concurrent write operations such as multiple MM threads
//flushing)
-// - no deleted blocks to process
+// 2. minimal number of containers, each of which is entirely full
+//with about --startup_benchmark_deleted_block_percentage percent
+//deleted blocks.
+// (typical workloads of write, alter operations, and background MM
+//threads running a long time since last bootstrap)
//
// However it still can be used to micro-optimize the startup process.
-TEST_F(LogBlockManagerTest, StartupBenchmark) {
+class LogBlockManagerStartupBenchmarkTest:
+public LogBlockManagerTest,
+public ::testing::WithParamInterface {
+};
+INSTANTIATE_TEST_SUITE_P(StartupBenchmarkSuite,
LogBlockManagerStartupBenchmarkTest,
+ ::testing::Values(false, true));
+
+TEST_P(LogBlockManagerStartupBenchmarkTest, StartupBenchmark) {
+ bool delete_blocks = GetParam();
std::vector test_dirs;
for (int i = 0; i < FLAGS_startup_benchmark_data_dir_count_for_testing; ++i)
{
test_dirs.emplace_back(test_dir_ + "/" + std::to_string(i));
@@ -1034,7 +1053,9 @@ TEST_F(LogBlockManagerTest, StartupBenchmark) {
// for details.
FLAGS_block_manager_preflush_control = "never";
const int kNumBlocks = AllowSlowTests() ?
FLAGS_startup_benchmark_block_count_for_testing : 1000;
+
// Creates 'kNumBlocks' blocks with minimal data.
+ vector block_ids;
{
unique_ptr transaction =
bm_->NewCreationTransaction();
for (int i = 0; i < kNumBlocks; i++) {
@@ -1042,11 +1063,32 @@ TEST_F(LogBlockManagerTest, StartupBenchmark) {
ASSERT_OK_FAST(bm_->CreateBlock(test_block_opts_, ));
ASSERT_OK_FAST(block->Append("x"));
ASSERT_OK_FAST(block->Finalize());
+ block_ids.emplace_back(block->id());
transaction->AddCreatedBlock(std::move(block));
}
ASSERT_OK(transaction->CommitCreatedBlocks());
}
- for (int i = 0; i < 10; i++) {
+
+ if (delete_blocks) {
+std::mt19937 gen(SeedRandom());
+std::shuffle(block_ids.begin(), block_ids.end(), gen);
+{
+ int to_delete_count =
+ block_ids.size() * FLAGS_startup_benchmark_deleted_block_percentage
/ 100;
[kudu] branch master updated (3e24e1b -> 032d1d9)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 3e24e1b KUDU-1959 - Implement startup progress metrics related to containers and tablets new f334e6e [txns] fix UB in TxnSystemClient when adding max timeout to now new 032d1d9 [LBM] Speed up server bootstrap by using multi-thread to compact containers The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/fs/log_block_manager-test.cc | 52 ++-- src/kudu/fs/log_block_manager.cc | 80 - src/kudu/integration-tests/txn_commit-itest.cc | 2 +- .../integration-tests/txn_participant-itest.cc | 99 +- .../integration-tests/txn_status_table-itest.cc| 12 +-- src/kudu/master/master.cc | 6 +- src/kudu/master/master.h | 3 +- src/kudu/master/txn_manager-test.cc| 2 +- src/kudu/master/txn_manager.cc | 24 ++ src/kudu/tools/tool_action_txn.cc | 3 +- src/kudu/transactions/txn_status_manager.cc| 6 +- src/kudu/transactions/txn_system_client.cc | 58 ++--- src/kudu/transactions/txn_system_client.h | 22 ++--- src/kudu/tserver/ts_tablet_manager.cc | 4 +- 14 files changed, 227 insertions(+), 146 deletions(-)
[kudu] 01/02: [txns] fix UB in TxnSystemClient when adding max timeout to now
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f334e6e083203c96e56957e3d331adbc33d3c8a4
Author: Andrew Wong
AuthorDate: Tue Nov 2 17:14:04 2021 -0700
[txns] fix UB in TxnSystemClient when adding max timeout to now
This patch addresses the following UB found in a pre-commit:
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/monotime.cc:220:10:
runtime error: signed integer overflow: 271833850110 + 9223372036854775807
cannot be represented in type 'long'
#0 0x7f225fca9b31 in kudu::MonoTime::AddDelta(kudu::MonoDelta const&)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/monotime.cc:220:10
#1 0x7f225fcaaafe in kudu::operator+(kudu::MonoTime const&,
kudu::MonoDelta const&)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/monotime.cc:335:7
#2 0x7f226fa3d6ff in
kudu::transactions::TxnSystemClient::CoordinateTransactionAsync(kudu::tserver::CoordinatorOpPB,
kudu::MonoDelta, std::function const&,
kudu::tserver::CoordinatorOpResultPB*)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/transactions/txn_system_client.cc:331:45
#3 0x7f226fa3feca in
kudu::transactions::TxnSystemClient::KeepTransactionAlive(long,
std::__cxx11::basic_string, std::allocator >
const&, kudu::MonoDelta)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/transactions/txn_system_client.cc:320:3
#4 0x7f2271211629 in
kudu::transactions::TxnManager::KeepTransactionAlive(long,
std::__cxx11::basic_string, std::allocator >
const&, kudu::MonoTime const&)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/txn_manager.cc:238:27
#5 0x7f227121535f in
kudu::transactions::TxnManagerServiceImpl::KeepTransactionAlive(kudu::transactions::KeepTransactionAliveRequestPB
const*, kudu::transactions::KeepTransactionAliveResponsePB*,
kudu::rpc::RpcContext*)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/master/txn_manager_service.cc:159:42
#6 0x7f2265d5749e in std::function::operator()(google::protobuf::Message const*,
google::protobuf::Message*, kudu::rpc::RpcContext*) const
../../../include/c++/7.5.0/bits/std_function.h:706:14
#7 0x7f2265d5648c in
kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_if.cc:137:3
#8 0x7f2265d5c42d in kudu::rpc::ServicePool::RunThread()
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/rpc/service_pool.cc:232:15
#9 0x7f225fd596ba in kudu::Thread::SuperviseThread(void*)
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/thread.cc:674:3
#10 0x7f22625026da in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#11 0x7f225bfea71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
/home/jenkins-slave/workspace/kudu-master/1/src/kudu/util/monotime.cc:220:10 in
Previously, we converted an initial deadline to a timeout, potentially
rejiggering the value in case of the maximal timeout, and then
recomputed the deadline. This patch addresses the UB by addressing a
TODO to pass deadlines in the context of the TxnSystemClient instead of
timeouts.
Change-Id: I1e5d4d06e8c0801c7f6b2399f7622e6f039f988e
Reviewed-on: http://gerrit.cloudera.org:8080/17993
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
Reviewed-by: Abhishek Chennaka
---
src/kudu/integration-tests/txn_commit-itest.cc | 2 +-
.../integration-tests/txn_participant-itest.cc | 99 +-
.../integration-tests/txn_status_table-itest.cc| 12 +--
src/kudu/master/master.cc | 6 +-
src/kudu/master/master.h | 3 +-
src/kudu/master/txn_manager-test.cc| 2 +-
src/kudu/master/txn_manager.cc | 24 ++
src/kudu/tools/tool_action_txn.cc | 3 +-
src/kudu/transactions/txn_status_manager.cc| 6 +-
src/kudu/transactions/txn_system_client.cc | 58 ++---
src/kudu/transactions/txn_system_client.h | 22 ++---
src/kudu/tserver/ts_tablet_manager.cc | 4 +-
12 files changed, 121 insertions(+), 120 deletions(-)
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index b833959..9b52867 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -676,7 +676,7 @@ TEST_F(TxnCommitITest, TestCommitAfterParticipantAbort) {
op_pb.set_txn_id(0);
op_pb.set_type(ParticipantOpPB::ABORT_TXN);
ASSERT_OK(txn_client_->ParticipateInTransaction(
- participant_ids_[0], op_pb, MonoDelta::FromSeconds(3)));
+ participant_ids_[0], op_pb, MonoTi
[kudu] branch master updated (a31b96e -> 3e24e1b)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from a31b96e KUDU-3332 [tools] Fix too large next_column_id after unsafe_rebuilding master new 59070bf KUDU-1959 - Implement aggregate startup progress metrics new 3e24e1b KUDU-1959 - Implement startup progress metrics related to containers and tablets The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/fs/fs_manager.cc | 12 src/kudu/fs/log_block_manager.cc| 26 + src/kudu/server/server_base.cc | 2 +- src/kudu/server/startup_path_handler.cc | 50 - src/kudu/server/startup_path_handler.h | 11 +++- src/kudu/tserver/ts_tablet_manager.cc | 25 + src/kudu/tserver/ts_tablet_manager.h| 4 +++ 7 files changed, 127 insertions(+), 3 deletions(-)
[kudu] 02/02: KUDU-1959 - Implement startup progress metrics related to containers and tablets
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3e24e1be4362ba9efb6d295ff96d3a18893f2733
Author: Abhishek Chennaka
AuthorDate: Mon Oct 18 22:03:23 2021 -0400
KUDU-1959 - Implement startup progress metrics related to containers and
tablets
This patch implements the metrics related to the server startup.
* In case of log block manager, we expose:
- log_block_manager_total_containers_startup : total containers present,
- log_block_manager_processed_containers_startup : count of containers
opened/processed until the requested instant of time and
- log_block_manager_containers_processing_time_startup : time elapsed
for opening the containers. If the containers are not yet opened, we
provide the time elapsed so far.
* In case of tablet server, we expose:
- tablets_num_total_startup : total tablets present,
- tablets_num_opened_startup : count of tablets opened/processed until
the requested instant of time and
- tablets_opening_time_startup : time elapsed for opening the tablets.
If the tablets are not yet opened, we provide the time elapsed so
far.
All the times are in milliseconds and the time metrics are in debug
level.
Change-Id: I9d1aa85b0585214475a6bdb8c0e5d7343c5bc3c9
Reviewed-on: http://gerrit.cloudera.org:8080/17947
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/fs/fs_manager.cc | 12
src/kudu/fs/log_block_manager.cc | 26 ++
src/kudu/tserver/ts_tablet_manager.cc | 25 +
src/kudu/tserver/ts_tablet_manager.h | 4
4 files changed, 67 insertions(+)
diff --git a/src/kudu/fs/fs_manager.cc b/src/kudu/fs/fs_manager.cc
index 2ecca51..d58a195 100644
--- a/src/kudu/fs/fs_manager.cc
+++ b/src/kudu/fs/fs_manager.cc
@@ -50,6 +50,7 @@
#include "kudu/util/env_util.h"
#include "kudu/util/flag_tags.h"
#include "kudu/util/metrics.h"
+#include "kudu/util/monotime.h"
#include "kudu/util/net/net_util.h"
#include "kudu/util/oid_generator.h"
#include "kudu/util/path_util.h"
@@ -108,6 +109,13 @@ DEFINE_int64(fs_wal_dir_reserved_bytes, -1,
DEFINE_validator(fs_wal_dir_reserved_bytes, [](const char* /*n*/, int64_t v) {
return v >= -1; });
TAG_FLAG(fs_wal_dir_reserved_bytes, runtime);
+METRIC_DEFINE_gauge_int64(server,
log_block_manager_containers_processing_time_startup,
+ "Time taken to open all log block containers during
server startup",
+ kudu::MetricUnit::kMilliseconds,
+ "The total time taken by the server to open all the
container"
+ "files during the startup",
+ kudu::MetricLevel::kDebug);
+
using kudu::fs::BlockManagerOptions;
using kudu::fs::CreateBlockOptions;
using kudu::fs::DataDirManager;
@@ -463,6 +471,10 @@ Status FsManager::Open(FsReport* report, Timer*
read_instance_metadata_files,
}
if (read_data_directories) {
read_data_directories->Stop();
+ if (opts_.metric_entity && opts_.block_manager_type == "log") {
+
METRIC_log_block_manager_containers_processing_time_startup.Instantiate(opts_.metric_entity,
+(read_data_directories->TimeElapsed()).ToMilliseconds());
+ }
}
}
// Report wal and metadata directories.
diff --git a/src/kudu/fs/log_block_manager.cc b/src/kudu/fs/log_block_manager.cc
index 8fcae4a..9a17cdb 100644
--- a/src/kudu/fs/log_block_manager.cc
+++ b/src/kudu/fs/log_block_manager.cc
@@ -159,6 +159,20 @@ METRIC_DEFINE_counter(server,
log_block_manager_dead_containers_deleted,
"Number of full (but dead) block containers that were
deleted",
kudu::MetricLevel::kDebug);
+METRIC_DEFINE_gauge_uint64(server, log_block_manager_total_containers_startup,
+ "Total number of Log Block Containers during
startup",
+ kudu::MetricUnit::kLogBlockContainers,
+ "Number of log block containers which were present
during the server "
+ "startup",
+ kudu::MetricLevel::kInfo);
+
+METRIC_DEFINE_gauge_uint64(server,
log_block_manager_processed_containers_startup,
+ "Number of Log Block Containers opened during
startup",
+ kudu::MetricUnit::kLogBlockContainers,
+ "Number of log block containers which were
opened/processed during "
+ "the server startup",
+ kud
[kudu] 01/02: KUDU-1959 - Implement aggregate startup progress metrics
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 59070bf5bd5924c6e4deb68434744cac3b062dcc
Author: Abhishek Chennaka
AuthorDate: Mon Oct 11 11:36:55 2021 -0700
KUDU-1959 - Implement aggregate startup progress metrics
We expose the below metrics as a part of this commit:
* startup_progress_steps_remaining : count of server startup steps which
are yet to be completed. This value is in the range [0,4].
* startup_progress_time_elapsed : the time elapsed so far for the server to
startup. If the startup is completed, this is the total time taken for the
startup. This is in milliseconds.
These metrics are primarily expected to be used by third party monitoring
tools
to see how long has the server taken to startup historically for any sort of
trend analysis.
The startup_progress_time_elapsed metric can also be used to check the
previous startup time as an alternative to the startup page in the WebUI.
Change-Id: I0a508c3baf0a0d77baf75f36f7bb305a6ad821e1
Reviewed-on: http://gerrit.cloudera.org:8080/17903
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/server/server_base.cc | 2 +-
src/kudu/server/startup_path_handler.cc | 50 -
src/kudu/server/startup_path_handler.h | 11 +++-
3 files changed, 60 insertions(+), 3 deletions(-)
diff --git a/src/kudu/server/server_base.cc b/src/kudu/server/server_base.cc
index bbbcefa..ec44729 100644
--- a/src/kudu/server/server_base.cc
+++ b/src/kudu/server/server_base.cc
@@ -495,7 +495,7 @@ ServerBase::ServerBase(string name, const
ServerBaseOptions& options,
file_cache_(new FileCache("file cache", options.env,
GetFileCacheCapacity(options.env),
metric_entity_)),
rpc_server_(new RpcServer(options.rpc_opts)),
- startup_path_handler_(new StartupPathHandler),
+ startup_path_handler_(new StartupPathHandler(metric_entity_)),
result_tracker_(new rpc::ResultTracker(shared_ptr(
MemTracker::CreateTracker(-1, "result-tracker", mem_tracker_,
is_first_run_(false),
diff --git a/src/kudu/server/startup_path_handler.cc
b/src/kudu/server/startup_path_handler.cc
index 93f839e..6e1948f 100644
--- a/src/kudu/server/startup_path_handler.cc
+++ b/src/kudu/server/startup_path_handler.cc
@@ -24,10 +24,24 @@
#include "kudu/gutil/strings/human_readable.h"
#include "kudu/server/webserver.h"
#include "kudu/util/easy_json.h"
+#include "kudu/util/metrics.h"
#include "kudu/util/monotime.h"
#include "kudu/util/timer.h"
#include "kudu/util/web_callback_registry.h"
+METRIC_DEFINE_gauge_int32(server, startup_progress_steps_remaining,
+ "Server Startup Steps Remaining",
+ kudu::MetricUnit::kUnits,
+ "Server startup progress steps remaining ",
+ kudu::MetricLevel::kWarn);
+
+METRIC_DEFINE_gauge_int64(server, startup_progress_time_elapsed,
+ "Server Startup Progress Time Elapsed",
+ kudu::MetricUnit::kMilliseconds,
+ "Time taken by the server to complete the startup or"
+ "time elapsed so far for the server to startup",
+ kudu::MetricLevel::kInfo);
+
using std::ifstream;
using std::ostringstream;
using std::string;
@@ -43,13 +57,19 @@ void SetWebResponse(EasyJson* output, const string& step,
(startup_step.TimeElapsed()).ToSeconds()));
}
-StartupPathHandler::StartupPathHandler():
+StartupPathHandler::StartupPathHandler(const scoped_refptr&
entity):
tablets_processed_(0),
tablets_total_(0),
containers_processed_(0),
containers_total_(0),
is_tablet_server_(false),
is_using_lbm_(true) {
+ METRIC_startup_progress_steps_remaining.InstantiateFunctionGauge(entity,
+ [this]() {return StartupProgressStepsRemainingMetric();})
+ ->AutoDetachToLastValue(_detacher_);
+ METRIC_startup_progress_time_elapsed.InstantiateFunctionGauge(entity,
+ [this]() {return StartupProgressTimeElapsedMetric().ToMilliseconds();})
+ ->AutoDetachToLastValue(_detacher_);
}
void StartupPathHandler::Startup(const Webserver::WebRequest& /*req*/,
@@ -116,5 +136,33 @@ void StartupPathHandler::set_is_tablet_server(bool
is_tablet_server) {
void StartupPathHandler::set_is_using_lbm(bool is_using_lbm) {
is_using_lbm_ = is_using_lbm;
}
+
+int StartupPathHandler::StartupProgressStepsRemainingMetric() {
+ int counter = 0;
+ counter += (init_progress_.IsStopped() ? 0 : 1);
+ counter += (read_filesystem_progress_.IsStopped() ? 0 : 1);
+ counter +=
[kudu] branch master updated: KUDU-1959 - Implement server startup progress page for tablet and master servers
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new bd5f72e KUDU-1959 - Implement server startup progress page for tablet
and master servers
bd5f72e is described below
commit bd5f72e0ed03b614d49b8125160bf555d11dcf9c
Author: Abhishek Chennaka
AuthorDate: Tue Jul 27 11:11:06 2021 -0400
KUDU-1959 - Implement server startup progress page for tablet and master
servers
This patch implements adjusting the server startup sequence to start the
web server and load the startup page which shows the progress of the
startup.
This progress is broken down into the below steps:
Initializing
Reading Filesystem
Reading instance metadata files
Opening container files
Bootstrapping and opening the tablets
Starting RPC server
Sample screenshot of the page can be found here:
https://i.imgur.com/J4NJD5i.png
Each of the steps above has a progress status of either 0 or 100 except
for the step “Bootstrapping and opening the tablets” which tracks the
processing of each of the tablet and the step “Opening container files”.
For log block manager this step shows the progress of opening the log
block containers. For file block manager, this is renamed to
“Reporting Filesystem” and has only 0 or 100 progress status. All of
the steps have time elapsed presented in seconds.
For the master startup page, the step “Bootstrapping and opening the
tablets” is replaced by “Initializing master catalog”.
Along with the above startup page the web server will also display
the pages - /home, /config, /logs, /mem-trackers, /memz, /threadz, /varz.
During the initialization since we expect the user's primary reason to
open the web UI to be to check the startup progress, /home will be
redirected to /startup until the startup is completed. The usual homepage
contents and the other remaining web pages will be available once the rpc
server is started due to dependency. Manually validated the startup when
security is enabled and filesystem is not present.
The footer of the WebUI has the UUID by default. In case of new
instances, since we are starting the Webserver before the UUID is
generated the footer doesn’t contain UUID but it is set later when
the rpc server is started.
The next steps for this patch include writing tests for the startup
page. So far all the testing has been manual.
Change-Id: I1db1fcf16261d4ced1b3657a697766a5335271b4
Reviewed-on: http://gerrit.cloudera.org:8080/17730
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/fs/block_manager.h | 7 +-
src/kudu/fs/file_block_manager.cc| 3 +-
src/kudu/fs/file_block_manager.h | 4 +-
src/kudu/fs/fs_manager.cc| 25 ++-
src/kudu/fs/fs_manager.h | 17 -
src/kudu/fs/log_block_manager.cc | 23 --
src/kudu/fs/log_block_manager.h | 15 +++-
src/kudu/master/master.cc| 8 ++-
src/kudu/server/CMakeLists.txt | 4 +-
src/kudu/server/default_path_handlers.cc | 10 +--
src/kudu/server/default_path_handlers.h | 8 ++-
src/kudu/server/server_base.cc | 57 ---
src/kudu/server/server_base.h| 2 +
src/kudu/server/startup_path_handler.cc | 120 +++
src/kudu/server/startup_path_handler.h | 97 +
src/kudu/server/tracing_path_handlers.h | 4 +-
src/kudu/server/webserver-test.cc| 6 +-
src/kudu/server/webserver.cc | 40 +++
src/kudu/server/webserver.h | 14
src/kudu/tserver/tablet_server.cc| 22 --
src/kudu/tserver/ts_tablet_manager.cc| 51 ++---
src/kudu/tserver/ts_tablet_manager.h | 24 ++-
src/kudu/util/timer.h| 67 +
src/kudu/util/web_callback_registry.h| 1 +
www/startup.mustache | 79
25 files changed, 637 insertions(+), 71 deletions(-)
diff --git a/src/kudu/fs/block_manager.h b/src/kudu/fs/block_manager.h
index 87239c3..1c74e40 100644
--- a/src/kudu/fs/block_manager.h
+++ b/src/kudu/fs/block_manager.h
@@ -219,7 +219,12 @@ class BlockManager {
//
// Returns an error if an on-disk representation does not exist or cannot be
// opened.
- virtual Status Open(FsReport* report) = 0;
+ //
+ // If 'containers_processed' and 'containers_total' are not nullptr, they
will
+ // be populated with total containers attempted to be opened/processed and
+ // total containers present respectively.
+ virtual Status Open(FsReport* report, std::atomic* containers_processed
= nullptr,
+ std
[kudu] 02/02: [tablet] fix non-handled switch warning
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 9015ccc162122b0d0a55431957c03fb085bc26c3
Author: Andrew Wong
AuthorDate: Mon Oct 25 19:24:10 2021 -0700
[tablet] fix non-handled switch warning
This patch addresses the following build warning:
../../src/kudu/tablet/ops/participant_op.cc: In member function
‘kudu::Status kudu::tablet::ParticipantOpState::PerformOp(const
kudu::consensus::OpId&, kudu::tablet::Tablet*)’:
../../src/kudu/tablet/ops/participant_op.cc:220:10: warning: enumeration
value ‘ParticipantOpPB_ParticipantOpType_GET_METADATA’ not handled in switch
[-Wswitch]
switch (op_type) {
^
Change-Id: I25c05d90dd3716f755ddcd9e842c4080fea277e1
Reviewed-on: http://gerrit.cloudera.org:8080/17970
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
---
src/kudu/tablet/ops/participant_op.cc | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/src/kudu/tablet/ops/participant_op.cc
b/src/kudu/tablet/ops/participant_op.cc
index a5ed7eb..0bd305d 100644
--- a/src/kudu/tablet/ops/participant_op.cc
+++ b/src/kudu/tablet/ops/participant_op.cc
@@ -252,9 +252,10 @@ Status ParticipantOpState::PerformOp(const
consensus::OpId& op_id, Tablet* table
txn_->ReleasePartitionLock();
break;
}
-case ParticipantOpPB::UNKNOWN: {
- return Status::InvalidArgument("unknown op type");
-}
+default:
+ return Status::InvalidArgument(
+ Substitute("bad op type $0 ($1)",
+ op_type,
ParticipantOpPB::ParticipantOpType_Name(op_type)));
}
return Status::OK();
}
[kudu] branch master updated (de02a34 -> 9015ccc)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from de02a34 [encryption] Add encryption support to Env new 5ce41d9 [rpc] re-add methods_by_name() new 9015ccc [tablet] fix non-handled switch warning The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/rpc/rpc-test.cc | 10 ++ src/kudu/rpc/service_if.h | 5 - src/kudu/tablet/ops/participant_op.cc | 7 --- 3 files changed, 18 insertions(+), 4 deletions(-)
[kudu] 01/02: [rpc] re-add methods_by_name()
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 5ce41d9075b04a39014ec8445b7cdf0cd2e64b10
Author: Andrew Wong
AuthorDate: Tue Oct 5 13:34:55 2021 -0700
[rpc] re-add methods_by_name()
This patch reverts the removal of the methods_by_name() accessor method
from patch 7f794e3ac8c389af4c74faeac11274e4ef0a55c0. Turns out it may be
used outside the project, in the Apache Impala project[1]. To maintain
consistency between the two projects, this re-adds the method.
This patch also adds a simple test with this context to deter further
removals of the call.
[1]
https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/rpc/impala-service-pool.cc#L77
Change-Id: I2cbccdbbd119ebd53544d951e45b8bb4d34abae7
Reviewed-on: http://gerrit.cloudera.org:8080/17969
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/rpc/rpc-test.cc | 10 ++
src/kudu/rpc/service_if.h | 5 -
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/src/kudu/rpc/rpc-test.cc b/src/kudu/rpc/rpc-test.cc
index bd7adf8..4a3aedd 100644
--- a/src/kudu/rpc/rpc-test.cc
+++ b/src/kudu/rpc/rpc-test.cc
@@ -48,6 +48,7 @@
#include "kudu/rpc/outbound_call.h"
#include "kudu/rpc/proxy.h"
#include "kudu/rpc/reactor.h"
+#include "kudu/rpc/result_tracker.h"
#include "kudu/rpc/rpc-test-base.h"
#include "kudu/rpc/rpc_controller.h"
#include "kudu/rpc/rpc_header.pb.h"
@@ -1059,6 +1060,15 @@ static void AcceptAndReadForever(Socket* listen_sock) {
}
}
+// Basic test for methods_by_name(). At the time of writing, this isn't used by
+// Kudu, but is used in other projects like Apache Impala.
+TEST_F(TestRpc, TestMethodsByName) {
+ std::unique_ptr service(
+ new CalculatorService(metric_entity_, result_tracker_));
+ const auto& methods = service->methods_by_name();
+ ASSERT_EQ(8, methods.size());
+}
+
// Starts a fake listening socket which never actually negotiates.
// Ensures that the client gets a reasonable status code in this case.
TEST_F(TestRpc, TestNegotiationTimeout) {
diff --git a/src/kudu/rpc/service_if.h b/src/kudu/rpc/service_if.h
index 71226db..185e291 100644
--- a/src/kudu/rpc/service_if.h
+++ b/src/kudu/rpc/service_if.h
@@ -114,6 +114,10 @@ class GeneratedServiceIf : public ServiceIf {
RpcMethodInfo* LookupMethod(const RemoteMethod& method) override;
+ // Returns the mapping from method names to method infos.
+ typedef std::unordered_map>
MethodInfoMap;
+ const MethodInfoMap& methods_by_name() const { return methods_by_name_; }
+
protected:
explicit GeneratedServiceIf(const scoped_refptr& tracker);
@@ -124,7 +128,6 @@ class GeneratedServiceIf : public ServiceIf {
// call. Methods are inserted by the constructor of the generated subclass.
// After construction, this map is accessed by multiple threads and therefore
// must not be modified.
- typedef std::unordered_map>
MethodInfoMap;
MethodInfoMap methods_by_name_;
};
[kudu] branch master updated: KUDU-1260: Fix prefetching bug on Java scanner
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 345fce8 KUDU-1260: Fix prefetching bug on Java scanner
345fce8 is described below
commit 345fce81aa52fa378a57d6b610d42e751a1387a4
Author: Hongjiang Zhang
AuthorDate: Mon Aug 16 11:57:54 2021 +0800
KUDU-1260: Fix prefetching bug on Java scanner
Add a UT to test prefetching. The UT has two concurrent threads: writing
thread and scanner thread. The writing thread records the timestamp of
its write, and the scanner thread creates two scanners (w and w/o
prefetching),
by comparing the scan result of the two scanners, we can verify the
prefetching result.
When prefetching is enabled, there is a RowResultIterator prefetched and
it will override the one which has not yet been consumed in current
implementation, as a result, some data will loss. The fix is simple:
use an atomic reference to cache the result and avoid value overriding.
Furthermore, there are at most two concurrent ScanRequests
sent to the tserver. But if the scan data reached the end, only one
hasMore=false is returned.
As a result, one of the ScanRequests got "scanner not found (it may have
expired)" exception.
The same issue occurs for KeepAliveRequest. This patch also addressed this
issue.
Change-Id: I853a041d86c75ec196d7d4ff45af4673c5c5f5cd
Reviewed-on: http://gerrit.cloudera.org:8080/17773
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
.../org/apache/kudu/client/AsyncKuduScanner.java | 117 +---
.../kudu/client/TestKuduScannerPrefetching.java| 307 +
2 files changed, 389 insertions(+), 35 deletions(-)
diff --git
a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java
b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java
index 9206495..79e18f8 100644
---
a/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java
+++
b/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduScanner.java
@@ -38,6 +38,7 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
import com.google.common.collect.ImmutableList;
import com.google.protobuf.Message;
@@ -251,7 +252,7 @@ public final class AsyncKuduScanner {
private boolean closed = false;
- private boolean hasMore = true;
+ private boolean canRequestMore = true;
private long numRowsReturned = 0;
@@ -281,10 +282,29 @@ public final class AsyncKuduScanner {
*/
private int sequenceId;
- private Deferred prefetcherDeferred;
-
final long scanRequestTimeout;
+ /**
+ * The prefetching result is cached in memory. This atomic reference is used
to avoid
+ * two concurrent prefetchings occur and the latest one overrides the
previous one.
+ */
+ private AtomicReference>
cachedPrefetcherDeferred =
+ new AtomicReference<>();
+
+ /**
+ * When scanner's prefetching is enabled, there are at most two concurrent
ScanRequests
+ * sent to the tserver. But if the scan data reached the end, only one
hasMore=false is returned.
+ * As a result, one of the ScanRequests got "scanner not found (it may have
expired)" exception.
+ * The same issue occurs for KeepAliveRequest.
+ *
+ * @param errorCode error code returned from tserver
+ * @return true if this can be ignored
+ */
+ boolean canBeIgnored(TabletServerErrorPB.Code errorCode) {
+return errorCode == TabletServerErrorPB.Code.SCANNER_EXPIRED &&
+prefetching && closed;
+ }
+
AsyncKuduScanner(AsyncKuduClient client, KuduTable table, List
projectedNames,
List projectedIndexes, ReadMode readMode, boolean
isFaultTolerant,
long scanRequestTimeout,
@@ -362,7 +382,7 @@ public final class AsyncKuduScanner {
// short circuited without contacting any tablet servers.
if (!pruner.hasMorePartitionKeyRanges()) {
LOG.debug("Short circuiting scan");
- this.hasMore = false;
+ this.canRequestMore = false;
this.closed = true;
}
@@ -421,11 +441,11 @@ public final class AsyncKuduScanner {
}
/**
- * Tells if the last rpc returned that there might be more rows to scan.
+ * Tells if there is data to scan, including both rpc or cached rpc result.
* @return true if there might be more data to scan, else false
*/
public boolean hasMoreRows() {
-return this.hasMore;
+return this.canRequestMore || cachedPrefetcherDeferred.get() != null;
}
/**
@@ -529,6 +549,10 @@ public final class AsyncKuduScanner {
*/
public Deferred nextRows() {
if (closed) { // We're already done scanning.
[kudu] branch master updated: [master] KUDU-1885: re-resolve TSDescriptor proxies on network error
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 1584e43 [master] KUDU-1885: re-resolve TSDescriptor proxies on
network error
1584e43 is described below
commit 1584e4325cfce37f89fd1dbebff5f24ed980780f
Author: Andrew Wong
AuthorDate: Wed Sep 15 19:50:34 2021 -0700
[master] KUDU-1885: re-resolve TSDescriptor proxies on network error
This patch uses the capabilities added in KUDU-75 to add the ability for
the master to re-resolve the addresses of tablet servers if its requests
fail.
Change-Id: I6245f75a232fd4827de684cfc04d6b6e53b7ddef
Reviewed-on: http://gerrit.cloudera.org:8080/17909
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/integration-tests/dns_alias-itest.cc | 82 +--
src/kudu/integration-tests/test_workload.cc | 2 +
src/kudu/integration-tests/test_workload.h| 5 ++
src/kudu/master/ts_descriptor.cc | 12 ++--
4 files changed, 92 insertions(+), 9 deletions(-)
diff --git a/src/kudu/integration-tests/dns_alias-itest.cc
b/src/kudu/integration-tests/dns_alias-itest.cc
index a801005..58735bd 100644
--- a/src/kudu/integration-tests/dns_alias-itest.cc
+++ b/src/kudu/integration-tests/dns_alias-itest.cc
@@ -35,11 +35,15 @@
#include "kudu/integration-tests/test_workload.h"
#include "kudu/mini-cluster/external_mini_cluster.h"
#include "kudu/mini-cluster/mini_cluster.h"
+#include "kudu/rpc/rpc_controller.h"
+#include "kudu/tserver/tserver.pb.h"
+#include "kudu/tserver/tserver_service.proxy.h"
#include "kudu/util/metrics.h"
#include "kudu/util/monotime.h"
#include "kudu/util/net/net_util.h"
#include "kudu/util/net/sockaddr.h"
#include "kudu/util/net/socket.h"
+#include "kudu/util/pb_util.h"
#include "kudu/util/status.h"
#include "kudu/util/test_macros.h"
#include "kudu/util/test_util.h"
@@ -56,6 +60,7 @@ using kudu::client::KuduTabletServer;
using kudu::cluster::ExternalDaemon;
using kudu::cluster::ExternalMiniCluster;
using kudu::cluster::ExternalMiniClusterOptions;
+using kudu::pb_util::SecureShortDebugString;
using std::string;
using std::unique_ptr;
using std::vector;
@@ -76,8 +81,10 @@ class DnsAliasITest : public KuduTest {
SetUpCluster();
}
- void SetUpCluster(vector extra_master_flags = {},
-vector extra_tserver_flags = {}) {
+ // TODO(awong): more plumbing is needed to allow the server to be restarted
+ // bound to a different address with the webserver, so just disable it.
+ void SetUpCluster(vector extra_master_flags = {
"--webserver_enabled=false" },
+vector extra_tserver_flags = {
"--webserver_enabled=false" }) {
ExternalMiniClusterOptions opts;
opts.num_masters = 3;
opts.num_tablet_servers = 3;
@@ -133,9 +140,6 @@ class DnsAliasITest : public KuduTest {
HostPort new_ip_hp(new_addr.host(), new_addr.port());
daemon->SetRpcBindAddress(new_ip_hp);
daemon->mutable_flags()->emplace_back("--rpc_reuseport=true");
-// TODO(awong): more plumbing is needed to allow the server to startup with
-// the webserver, so just disable it.
-daemon->mutable_flags()->emplace_back("--webserver_enabled=false");
daemon->mutable_flags()->emplace_back(
Substitute("--dns_addr_resolution_override=$0", new_overrides_str));
}
@@ -356,6 +360,74 @@ TEST_F(DnsAliasITest, TestMasterReresolveOnStartup) {
});
}
+// Regression test for KUDU-1885, wherein tserver proxies on the masters don't
+// eventually succeed when the tserver's address changes.
+TEST_F(DnsAliasITest, Kudu1885) {
+ // First, wait for all tablet servers to report to the masters.
+ ASSERT_EVENTUALLY([&] {
+vector tservers;
+ElementDeleter deleter();
+ASSERT_OK(client_->ListTabletServers());
+ASSERT_EQ(cluster_->num_tablet_servers(), tservers.size());
+ });
+ auto* tserver = cluster_->tablet_server(cluster_->num_tablet_servers() - 1);
+ // Shut down a tablet server so we can start it up with a different address.
+ tserver->Shutdown();
+ unique_ptr reserved_socket;
+
ASSERT_OK(cluster_->ReserveDaemonSocket(cluster::ExternalMiniCluster::DaemonType::TSERVER,
3,
+ kDefaultBindMode, _socket,
+
tserver->bound_rpc_hostport().port()));
+ Sockaddr new_addr;
+ ASSERT_OK(reserved_socket->GetSocketAddress(_addr));
+ auto new_overrides_str =
GetNewOverridesFlag(ExternalMiniCluster::DaemonType::TSERVER, new_addr);
+ SetUpDaemonForNewAddr(new_addr, new_overrides_str, tserver);
+
+
[kudu] branch master updated: [mini-cluster] allow using aliases instead of addresses in EMC
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new acf6de8 [mini-cluster] allow using aliases instead of addresses in EMC acf6de8 is described below commit acf6de866c878dc2f09fb5bd7cccf0312375e7c7 Author: Andrew Wong AuthorDate: Wed Sep 15 19:36:16 2021 -0700 [mini-cluster] allow using aliases instead of addresses in EMC This patch adds a configuration to ExternalMiniClusters to allow masters and tablet servers to use and broadcast an alias instead of a bound address. This is useful to simulate the default deployment where the configuration of --rpc_bind_addresses results in servers broadcasting their FQDN instead of a bound address. This entails a couple things: - The EMC may now use the --dns_addr_resolution_override flag to map aliases to the reserved addresses. - Previously we used Sockaddrs internally and converted them to HostPorts to pass around to various functions for remote callers to resolve (e.g. TServers pass these to Masters and Masters would resolve the HostPort to communicate with TServers). Since the HostPorts need to be resolvable remotely, wildcards would be substituted with the FQDN. In ExternalMiniClusters, we don't bind to the wildcard, but we still need to allow a server to broadcast its alias in lieu of an FQDN. A new --host_for_test is added for this purpose; all ExternalMasters and ExternalTabletServers set this when the configured to use aliases. - This also refactors our current Sockaddr-to-HostPort conversions to: 1) fetch HostPorts directly at call-sites, rather than getting Sockaddrs and doing a conversion, and 2) also account for --host_for_test. NOTE: this mini-cluster aliasing is compatible with the existing --rpc_listen_on_unix_domain_socket flag. Client processes need only set --host_for_test, and whatever server has the same --host_for_test will treat that client as local and communicate over a UNIX socket. A test is added that exercises this new functionality. Change-Id: I09a22636748a13f282406119b52021184d92a76f Reviewed-on: http://gerrit.cloudera.org:8080/17850 Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins --- src/kudu/integration-tests/CMakeLists.txt | 1 + src/kudu/integration-tests/dns_alias-itest.cc | 184 + src/kudu/master/master.cc | 10 +- src/kudu/master/master_path_handlers.cc| 16 +-- src/kudu/master/sys_catalog.cc | 10 +- src/kudu/mini-cluster/external_mini_cluster.cc | 38 - src/kudu/mini-cluster/external_mini_cluster.h | 9 ++ src/kudu/server/rpc_server.cc | 12 ++ src/kudu/server/rpc_server.h | 3 + src/kudu/server/server_base.cc | 20 +-- src/kudu/server/webserver.cc | 12 ++ src/kudu/server/webserver.h| 3 + src/kudu/tserver/heartbeater.cc| 11 +- src/kudu/tserver/ts_tablet_manager.cc | 10 +- src/kudu/util/net/net_util-test.cc | 17 +-- src/kudu/util/net/net_util.cc | 43 -- src/kudu/util/net/net_util.h | 9 +- 17 files changed, 350 insertions(+), 58 deletions(-) diff --git a/src/kudu/integration-tests/CMakeLists.txt b/src/kudu/integration-tests/CMakeLists.txt index f7208c3..c687930 100644 --- a/src/kudu/integration-tests/CMakeLists.txt +++ b/src/kudu/integration-tests/CMakeLists.txt @@ -78,6 +78,7 @@ ADD_KUDU_TEST(delete_table-itest NUM_SHARDS 8 PROCESSORS 4) ADD_KUDU_TEST(delete_tablet-itest PROCESSORS 2) ADD_KUDU_TEST(disk_failure-itest PROCESSORS 2) ADD_KUDU_TEST(disk_reservation-itest) +ADD_KUDU_TEST(dns_alias-itest) ADD_KUDU_TEST(exactly_once_writes-itest) ADD_KUDU_TEST(flex_partitioning-itest TIMEOUT 1800 NUM_SHARDS 8 PROCESSORS 2) ADD_KUDU_TEST(full_stack-insert-scan-test RUN_SERIAL true) diff --git a/src/kudu/integration-tests/dns_alias-itest.cc b/src/kudu/integration-tests/dns_alias-itest.cc new file mode 100644 index 000..0e92cc3 --- /dev/null +++ b/src/kudu/integration-tests/dns_alias-itest.cc @@ -0,0 +1,184 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software d
[kudu] 02/02: [rpc] KUDU-75: refresh DNS entries if proxies hit a network error
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 41ebabf2eb618b33fd30ad1821ccbda9d6390010
Author: Andrew Wong
AuthorDate: Sat Aug 28 13:55:47 2021 -0700
[rpc] KUDU-75: refresh DNS entries if proxies hit a network error
This patch aims to tackle the following issues that revolve around
changes in addresses at runtime.
- KUDU-1885: master long-lived tserver proxies need to be re-resolved in
case nodes are assigned different addresses; today we just retry at
the same location forever.
- KUDU-1620: tablet consensus long-lived proxies need to be re-resolved
on failure.
- C++ clients' usages of RemoteTabletServer also have long-lived proxies
and are likely to run into similar problems if tservers are restarted
and assigned new physical addresses.
It addresses this by plumbing a DnsResolver into the rpc::Proxy class,
and chaining the asynchronous callback to an asynchronous refresh of the
address with the newly introduced refreshing capabilities of the
DnsResolver.
The new style of proxy isn't currently used, but a test is added
exercising the new functionality.
Change-Id: I777d169bd3a461294e5721f05071b726ced70f7e
Reviewed-on: http://gerrit.cloudera.org:8080/17839
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/rpc/CMakeLists.txt | 1 +
src/kudu/rpc/connection_id.cc | 4 +-
src/kudu/rpc/connection_id.h| 4 +
src/kudu/rpc/mt-rpc-test.cc | 4 +-
src/kudu/rpc/protoc-gen-krpc.cc | 19 +++-
src/kudu/rpc/proxy-test.cc | 221
src/kudu/rpc/proxy.cc | 156 ++--
src/kudu/rpc/proxy.h| 67 +++-
src/kudu/rpc/rpc-test-base.h| 26 ++---
src/kudu/rpc/rpc-test.cc| 82 +++
src/kudu/rpc/rpc_stub-test.cc | 2 +-
src/kudu/util/net/sockaddr.cc | 3 +
src/kudu/util/net/sockaddr.h| 2 +-
13 files changed, 517 insertions(+), 74 deletions(-)
diff --git a/src/kudu/rpc/CMakeLists.txt b/src/kudu/rpc/CMakeLists.txt
index 273842d..c8d831e 100644
--- a/src/kudu/rpc/CMakeLists.txt
+++ b/src/kudu/rpc/CMakeLists.txt
@@ -129,6 +129,7 @@ ADD_KUDU_TEST(exactly_once_rpc-test PROCESSORS 10)
ADD_KUDU_TEST(mt-rpc-test RUN_SERIAL true)
ADD_KUDU_TEST(negotiation-test)
ADD_KUDU_TEST(periodic-test)
+ADD_KUDU_TEST(proxy-test)
ADD_KUDU_TEST(reactor-test)
ADD_KUDU_TEST(request_tracker-test)
ADD_KUDU_TEST(rpc-bench RUN_SERIAL true)
diff --git a/src/kudu/rpc/connection_id.cc b/src/kudu/rpc/connection_id.cc
index 9728f01..8a14d83 100644
--- a/src/kudu/rpc/connection_id.cc
+++ b/src/kudu/rpc/connection_id.cc
@@ -20,7 +20,7 @@
#include
#include
-#include
+#include
#include
#include "kudu/gutil/strings/substitute.h"
@@ -52,7 +52,7 @@ void ConnectionId::set_network_plane(string network_plane) {
string ConnectionId::ToString() const {
string remote;
- if (remote_.is_ip() && hostname_ != remote_.host()) {
+ if (remote_.is_initialized() && remote_.is_ip() && hostname_ !=
remote_.host()) {
remote = strings::Substitute("$0 ($1)", remote_.ToString(), hostname_);
} else {
remote = remote_.ToString();
diff --git a/src/kudu/rpc/connection_id.h b/src/kudu/rpc/connection_id.h
index 6ec98f7..0aed953 100644
--- a/src/kudu/rpc/connection_id.h
+++ b/src/kudu/rpc/connection_id.h
@@ -45,6 +45,10 @@ class ConnectionId {
const std::string& hostname() const { return hostname_; }
+ void set_remote(const Sockaddr& remote) {
+remote_ = remote;
+ }
+
// The credentials of the user associated with this connection, if any.
void set_user_credentials(UserCredentials user_credentials);
diff --git a/src/kudu/rpc/mt-rpc-test.cc b/src/kudu/rpc/mt-rpc-test.cc
index 09bda4b..3496e26 100644
--- a/src/kudu/rpc/mt-rpc-test.cc
+++ b/src/kudu/rpc/mt-rpc-test.cc
@@ -71,7 +71,7 @@ class MultiThreadedRpcTest : public RpcTestBase {
CHECK_OK(CreateMessenger("ClientSC", _messenger));
Proxy p(client_messenger, server_addr, server_addr.host(),
GenericCalculatorService::static_service_name());
-*result = DoTestSyncCall(p, method_name);
+*result = DoTestSyncCall(, method_name);
latch->CountDown();
}
@@ -93,7 +93,7 @@ class MultiThreadedRpcTest : public RpcTestBase {
int i = 0;
while (true) {
i++;
- Status s = DoTestSyncCall(p, method_name);
+ Status s = DoTestSyncCall(, method_name);
if (!s.ok()) {
// Return on first failure.
LOG(INFO) << "Call failed. Shutting down client thread. Ran " << i <<
" calls: "
diff --git a/src/kudu/rpc/protoc-gen-krpc.cc b/src/kudu/rpc/protoc-gen-krpc.cc
index 2226160..e8d8e55 100644
--- a/src/kudu/rpc/protoc-gen-kr
[kudu] 01/02: [util] allow DnsResolver to refresh DNS addresses
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 246505b538a6449eda7a9936609e9fec4632fe5d
Author: Andrew Wong
AuthorDate: Wed Sep 15 19:45:09 2021 -0700
[util] allow DnsResolver to refresh DNS addresses
This patch introduces functionality to the DnsResolver to refresh an
address, rather than looking it up in the cache. It does this by
removing any cached entry and performing the lookup.
This will be used in a follow-up change to refresh the address on
certain transient failures.
A new --dns_addr_resolution_override flag is also introduced for testing
purposes.
Change-Id: I0616f3e6fb50aba271f106b05d1926fc46a53ed0
Reviewed-on: http://gerrit.cloudera.org:8080/17849
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/util/net/dns_resolver-test.cc | 94 +-
src/kudu/util/net/dns_resolver.cc | 20
src/kudu/util/net/dns_resolver.h | 6 +++
src/kudu/util/net/net_util.cc | 23 +
src/kudu/util/ttl_cache.h | 4 ++
5 files changed, 145 insertions(+), 2 deletions(-)
diff --git a/src/kudu/util/net/dns_resolver-test.cc
b/src/kudu/util/net/dns_resolver-test.cc
index f53df7d..b861c68 100644
--- a/src/kudu/util/net/dns_resolver-test.cc
+++ b/src/kudu/util/net/dns_resolver-test.cc
@@ -21,9 +21,10 @@
#include
#include
#include
+#include
#include
-#include
+#include
#include
#include
@@ -34,10 +35,13 @@
#include "kudu/util/monotime.h"
#include "kudu/util/net/net_util.h"
#include "kudu/util/net/sockaddr.h"
+#include "kudu/util/scoped_cleanup.h"
#include "kudu/util/test_macros.h"
+DECLARE_string(dns_addr_resolution_override);
DECLARE_uint32(dns_resolver_cache_capacity_mb);
+using std::thread;
using std::vector;
using strings::Substitute;
@@ -59,6 +63,92 @@ TEST(DnsResolverTest, AsyncResolution) {
}
}
+TEST(DnsResolverTest, RefreshCachedEntry) {
+ gflags::FlagSaver saver;
+ vector addrs;
+ DnsResolver resolver(1/* max_threads_num */, 1024 * 1024/*
cache_capacity_bytes */);
+ ASSERT_OK(resolver.ResolveAddresses(HostPort("localhost", 12345), ));
+ ASSERT_TRUE(!addrs.empty());
+ for (const Sockaddr& addr : addrs) {
+LOG(INFO) << "Address: " << addr.ToString();
+EXPECT_TRUE(HasPrefixString(addr.ToString(), "127."));
+EXPECT_TRUE(HasSuffixString(addr.ToString(), ":12345"));
+ }
+ // If we override the DNS lookup address, when we refresh the address, the
+ // cached entry gets reset.
+ constexpr const char* kFakeAddr = "1.1.1.1";
+ FLAGS_dns_addr_resolution_override = Substitute("localhost=$0", kFakeAddr);
+ Synchronizer s;
+ resolver.RefreshAddressesAsync(HostPort("localhost", ), ,
+ s.AsStatusCallback());
+ ASSERT_OK(s.Wait());
+ ASSERT_EQ(1, addrs.size());
+ ASSERT_EQ(Substitute("$0:", kFakeAddr), addrs[0].ToString());
+ ASSERT_EQ(, addrs[0].port());
+
+ // Once we stop overriding DNS lookups, simply getting the address from the
+ // resolver will read from the cache.
+ FLAGS_dns_addr_resolution_override = "";
+ ASSERT_OK(resolver.ResolveAddresses(HostPort("localhost", 12345), ));
+ ASSERT_EQ(1, addrs.size());
+ ASSERT_EQ(Substitute("$0:12345", kFakeAddr), addrs[0].ToString());
+ ASSERT_EQ(12345, addrs[0].port());
+
+ // But a refresh should return the original address.
+ Synchronizer s2;
+ resolver.RefreshAddressesAsync(HostPort("localhost", 12345), ,
+ s2.AsStatusCallback());
+ ASSERT_OK(s2.Wait());
+ EXPECT_FALSE(addrs.empty());
+ for (const Sockaddr& addr : addrs) {
+LOG(INFO) << "Address: " << addr.ToString();
+EXPECT_TRUE(HasPrefixString(addr.ToString(), "127."));
+EXPECT_TRUE(HasSuffixString(addr.ToString(), ":12345"));
+ }
+}
+
+TEST(DnsResolverTest, ConcurrentRefreshesAndResolutions) {
+ constexpr int kNumThreads = 3;
+ constexpr int kNumResolutionsPerThread = 10;
+ DnsResolver resolver(1/* max_threads_num */, 1024 * 1024/*
cache_capacity_bytes */);
+ vector threads;
+ auto cancel_threads = MakeScopedCleanup([&] {
+for (auto& t : threads) {
+ t.join();
+}
+ });
+ const auto validate_addrs = [] (const vector& addrs) {
+ASSERT_FALSE(addrs.empty());
+for (const Sockaddr& addr : addrs) {
+ EXPECT_TRUE(HasPrefixString(addr.ToString(), "127."));
+ EXPECT_TRUE(HasSuffixString(addr.ToString(), ":12345"));
+}
+ };
+ for (int i = 0; i < kNumThreads - 1; i++) {
+threads.emplace_back([&] {
+ for (int r = 0; r < kNumRes
[kudu] branch master updated (63a3293 -> 41ebabf)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 63a3293 [common] small re-factoring on PartitionPruner new 246505b [util] allow DnsResolver to refresh DNS addresses new 41ebabf [rpc] KUDU-75: refresh DNS entries if proxies hit a network error The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/rpc/CMakeLists.txt| 1 + src/kudu/rpc/connection_id.cc | 4 +- src/kudu/rpc/connection_id.h | 4 + src/kudu/rpc/mt-rpc-test.cc| 4 +- src/kudu/rpc/protoc-gen-krpc.cc| 19 ++- src/kudu/rpc/proxy-test.cc | 221 + src/kudu/rpc/proxy.cc | 156 +-- src/kudu/rpc/proxy.h | 67 +- src/kudu/rpc/rpc-test-base.h | 26 ++-- src/kudu/rpc/rpc-test.cc | 82 ++-- src/kudu/rpc/rpc_stub-test.cc | 2 +- src/kudu/util/net/dns_resolver-test.cc | 94 +- src/kudu/util/net/dns_resolver.cc | 20 +++ src/kudu/util/net/dns_resolver.h | 6 + src/kudu/util/net/net_util.cc | 23 src/kudu/util/net/sockaddr.cc | 3 + src/kudu/util/net/sockaddr.h | 2 +- src/kudu/util/ttl_cache.h | 4 + 18 files changed, 662 insertions(+), 76 deletions(-) create mode 100644 src/kudu/rpc/proxy-test.cc
[kudu] branch master updated: KUDU-3204: Add a metric for amount of available space
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new e60ac3c KUDU-3204: Add a metric for amount of available space
e60ac3c is described below
commit e60ac3ca93532fdffa71999e90661e42d7dcc0b0
Author: Abhishek Chennaka
AuthorDate: Fri Jul 23 14:03:31 2021 -0400
KUDU-3204: Add a metric for amount of available space
This commit adds two metrics to server entity.
- wal_dir_space_available_bytes - Exposes the free space available in the
wal directory
- data_dirs_space_available_bytes - Exposes the sum of free space in each
of the data directories.
We ensure the reserved space is being accounted for. This is specified by
the flags --fs_data_dirs_reserved_bytes and --fs_wal_dir_reserved_bytes in
data directories and wal directories respectively. For calculating the
total free space in data directories, we ensure we are not counting the
same device/filesystem twice by getting and checking for uniqueness
of the filesystemID of each of the data directories before summing it up.
Tested the uniqueness of the filesystemID by creating a new volume in my
local macOS machine. The space reported is based on the data directories in
different volumes (and hence unique filesystemID). Having multiple data
directories on the same volume does not increase the metric.
The free space calculated is cached for the period specified by the flags
--fs_data_dirs_available_space_cache_seconds and
--fs_wal_dir_available_space_cache_seconds.
Change-Id: I58a7419847d080498aaf431d1dab098e1af71ad0
Reviewed-on: http://gerrit.cloudera.org:8080/17725
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
---
src/kudu/consensus/log.cc| 10 +--
src/kudu/fs/data_dirs.cc | 11 ++-
src/kudu/fs/fs_manager.cc| 10 +++
src/kudu/integration-tests/disk_failure-itest.cc | 48
src/kudu/server/server_base.cc | 96
src/kudu/server/server_base.h| 24 ++
src/kudu/util/env.h | 1 +
src/kudu/util/env_posix.cc | 1 +
8 files changed, 191 insertions(+), 10 deletions(-)
diff --git a/src/kudu/consensus/log.cc b/src/kudu/consensus/log.cc
index ea24555..716e7e5 100644
--- a/src/kudu/consensus/log.cc
+++ b/src/kudu/consensus/log.cc
@@ -151,15 +151,7 @@ TAG_FLAG(log_segment_size_bytes_for_tests, unsafe);
// Other flags.
// -
-DEFINE_int64(fs_wal_dir_reserved_bytes, -1,
- "Number of bytes to reserve on the log directory filesystem for "
- "non-Kudu usage. The default, which is represented by -1, is that
"
- "1% of the disk space on each disk will be reserved. Any other "
- "value specified represents the number of bytes reserved and must
"
- "be greater than or equal to 0. Explicit percentages to reserve "
- "are not currently supported");
-DEFINE_validator(fs_wal_dir_reserved_bytes, [](const char* /*n*/, int64_t v) {
return v >= -1; });
-TAG_FLAG(fs_wal_dir_reserved_bytes, runtime);
+DECLARE_int64(fs_wal_dir_reserved_bytes);
DEFINE_bool(fs_wal_use_file_cache, true,
"Whether to use the server-wide file cache for WAL segments and "
diff --git a/src/kudu/fs/data_dirs.cc b/src/kudu/fs/data_dirs.cc
index f2f36be..b5dc54c 100644
--- a/src/kudu/fs/data_dirs.cc
+++ b/src/kudu/fs/data_dirs.cc
@@ -77,10 +77,19 @@ TAG_FLAG(fs_data_dirs_reserved_bytes, runtime);
TAG_FLAG(fs_data_dirs_reserved_bytes, evolving);
DEFINE_int32(fs_data_dirs_available_space_cache_seconds, 10,
- "Number of seconds we cache the available disk space in the block
manager. ");
+ "Number of seconds we cache the available disk space in the block
manager.");
+DEFINE_validator(fs_data_dirs_available_space_cache_seconds,
+ [](const char* /*n*/, int32_t v) { return v >= 0; });
TAG_FLAG(fs_data_dirs_available_space_cache_seconds, advanced);
TAG_FLAG(fs_data_dirs_available_space_cache_seconds, evolving);
+DEFINE_int32(fs_wal_dir_available_space_cache_seconds, 10,
+ "Number of seconds we cache the available disk space the WAL
directory.");
+DEFINE_validator(fs_wal_dir_available_space_cache_seconds,
+ [](const char* /*n*/, int32_t v) { return v >= 0; });
+TAG_FLAG(fs_wal_dir_available_space_cache_seconds, advanced);
+TAG_FLAG(fs_wal_dir_available_space_cache_seconds, evolving);
+
DEFINE_bool(fs_lock_data_dirs, true,
"Lock the data directories to prevent concurrent usage. "
[kudu] branch master updated: [fuzz-itest] add WAL GCs
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 25d7173 [fuzz-itest] add WAL GCs
25d7173 is described below
commit 25d71738c05972eba1383a9b93883a7f5617b7c9
Author: Andrew Wong
AuthorDate: Tue Aug 17 22:07:07 2021 -0700
[fuzz-itest] add WAL GCs
In the past we've had bugs that only surface when WALs were GCed.
Additionally, implementing certain changes to the tablet key-value store
typically must also account for being able to correctly bootstrap from
the WALs. To facilitate testing such changes, this patch adds a WAL GC
op to the fuzz test.
This also adds a bytes equivalent of the --log_segment_size_mb flag, to
speed up the generation of WAL segments.
I had to add some error handling to allow writes to collide with the
destruction of the server.
Change-Id: Iabd9fe33f9fa21f62b513d7607bf62b6fce6bb59
Reviewed-on: http://gerrit.cloudera.org:8080/17787
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/consensus/consensus_queue.cc| 5 +
src/kudu/consensus/log.cc| 12 +++-
src/kudu/integration-tests/fuzz-itest.cc | 21 -
3 files changed, 36 insertions(+), 2 deletions(-)
diff --git a/src/kudu/consensus/consensus_queue.cc
b/src/kudu/consensus/consensus_queue.cc
index eecaecb..1ce9da0 100644
--- a/src/kudu/consensus/consensus_queue.cc
+++ b/src/kudu/consensus/consensus_queue.cc
@@ -366,6 +366,11 @@ void
PeerMessageQueue::CheckPeersInActiveConfigIfLeaderUnlocked() const {
void PeerMessageQueue::LocalPeerAppendFinished(const OpId& id,
const StatusCallback& callback,
const Status& status) {
+ // If the server is being destructed, the log may be unavailable.
+ if (PREDICT_FALSE(status.IsServiceUnavailable())) {
+callback(Status::OK());
+return;
+ }
CHECK_OK(status);
// Fake an RPC response from the local peer.
diff --git a/src/kudu/consensus/log.cc b/src/kudu/consensus/log.cc
index e94b36c..ea24555 100644
--- a/src/kudu/consensus/log.cc
+++ b/src/kudu/consensus/log.cc
@@ -23,6 +23,7 @@
#include
#include
#include
+#include
#include
#include
@@ -142,6 +143,12 @@ DEFINE_double(log_inject_io_error_on_preallocate_fraction,
0.0,
TAG_FLAG(log_inject_io_error_on_preallocate_fraction, unsafe);
TAG_FLAG(log_inject_io_error_on_preallocate_fraction, runtime);
+DEFINE_int32(log_segment_size_bytes_for_tests, 0,
+ "The size for log segments, in bytes. This takes precedence over "
+ "--log_segment_size_mb in cases where significantly smaller
segments are desired. "
+ "If non-positive, --log_segment_size_mb is honored.");
+TAG_FLAG(log_segment_size_bytes_for_tests, unsafe);
+
// Other flags.
// -
DEFINE_int64(fs_wal_dir_reserved_bytes, -1,
@@ -435,7 +442,10 @@ SegmentAllocator::SegmentAllocator(const LogOptions* opts,
uint32_t schema_version)
: opts_(opts),
ctx_(ctx),
- max_segment_size_(opts_->segment_size_mb * 1024 * 1024),
+ max_segment_size_(
+ FLAGS_log_segment_size_bytes_for_tests > 0
+ ? FLAGS_log_segment_size_bytes_for_tests
+ : opts_->segment_size_mb * 1024 * 1024),
schema_(std::move(schema)),
schema_version_(schema_version),
sync_disabled_(false) {}
diff --git a/src/kudu/integration-tests/fuzz-itest.cc
b/src/kudu/integration-tests/fuzz-itest.cc
index 225c0c0..9eaeee4 100644
--- a/src/kudu/integration-tests/fuzz-itest.cc
+++ b/src/kudu/integration-tests/fuzz-itest.cc
@@ -78,6 +78,7 @@
DEFINE_int32(keyspace_size, 5, "number of distinct primary keys to test
with");
DEFINE_int32(max_open_txns, 5, "maximum number of open transactions to test
with");
+DECLARE_int32(log_segment_size_bytes_for_tests);
DECLARE_bool(enable_txn_partition_lock);
DECLARE_bool(enable_maintenance_manager);
DECLARE_bool(scanner_allow_snapshot_scans_with_logical_timestamps);
@@ -142,6 +143,7 @@ enum TestOpType {
TEST_BEGIN_TXN,
TEST_COMMIT_TXN,
TEST_ABORT_TXN,
+ TEST_GC_LOG,
TEST_NUM_OP_TYPES // max value for enum
};
@@ -169,12 +171,20 @@ const char* TestOpType_names[] = {
"TEST_BEGIN_TXN",
"TEST_COMMIT_TXN",
"TEST_ABORT_TXN",
+ "TEST_GC_LOG",
};
constexpr const int kNoTxnId = -1;
// Identical to kNoTxnId but more generic-sounding for ops that don't use
// transaction IDs.
constexpr const int kNoVal = -1;
+// To facilitate WAL GC, we'll make our segment size very small.
+//
+// TODO(awong): implement a rollover op. It's a bit tricky today, given
[kudu] branch master updated: [tools] Add a tool to recover master data from tablet servers
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new ab0421b [tools] Add a tool to recover master data from tablet servers
ab0421b is described below
commit ab0421b13b81798548b7dc2f7acd8f7362d459c5
Author: Will Berkeley
AuthorDate: Thu Feb 22 23:53:32 2018 -0800
[tools] Add a tool to recover master data from tablet servers
This adds a tool that attempts to recover master metadata from
tablet servers using ListTablets, writing the metadata to a
syscatalog table that a new master process can use to make the
cluster operational.
It has several limitations. See the help description for the new tool.
Note that it should be possible to fix some limitations by
giving more master metadata to tablet servers. The advantage
of the tool as-is is that it should work on all versions of
Kudu since 1.0.
Change-Id: If29e421d466a531ebad72e281ae27e74e458f8c6
Reviewed-on: http://gerrit.cloudera.org:8080/9490
Tested-by: Kudu Jenkins
Reviewed-by: Bankim Bhavsar
---
src/kudu/common/partition.cc | 4 -
src/kudu/common/partition.h | 7 +-
src/kudu/common/schema.h | 8 +
src/kudu/master/catalog_manager.h| 4 +-
src/kudu/tools/CMakeLists.txt| 1 +
src/kudu/tools/kudu-admin-test.cc| 247 ++-
src/kudu/tools/master_rebuilder.cc | 350 +++
src/kudu/tools/master_rebuilder.h| 131 ++
src/kudu/tools/tool_action_common.cc | 19 ++
src/kudu/tools/tool_action_common.h | 9 +
src/kudu/tools/tool_action_master.cc | 85 +++
src/kudu/tools/tool_action_remote_replica.cc | 1 +
12 files changed, 852 insertions(+), 14 deletions(-)
diff --git a/src/kudu/common/partition.cc b/src/kudu/common/partition.cc
index 9cb06af..1a47db0 100644
--- a/src/kudu/common/partition.cc
+++ b/src/kudu/common/partition.cc
@@ -1174,10 +1174,6 @@ bool PartitionSchema::operator==(const PartitionSchema&
rhs) const {
return true;
}
-bool PartitionSchema::operator!=(const PartitionSchema& rhs) const {
- return !(*this == rhs);
-}
-
// Encodes the specified primary key columns of the supplied row into the
buffer.
void PartitionSchema::EncodeColumns(const ConstContiguousRow& row,
const vector& column_ids,
diff --git a/src/kudu/common/partition.h b/src/kudu/common/partition.h
index fd72b69..ab03729 100644
--- a/src/kudu/common/partition.h
+++ b/src/kudu/common/partition.h
@@ -75,6 +75,9 @@ class Partition {
// Returns true iff the given partition 'rhs' is equivalent to this one.
bool operator==(const Partition& rhs) const;
+ bool operator!=(const Partition& rhs) const {
+return !(*this == rhs);
+ }
// Serializes a partition into a protobuf message.
void ToPB(PartitionPB* pb) const;
@@ -300,7 +303,9 @@ class PartitionSchema {
bool operator==(const PartitionSchema& rhs) const;
// Returns 'true' iff the partition schema 'rhs' is not equivalent to this
one.
- bool operator!=(const PartitionSchema& rhs) const;
+ bool operator!=(const PartitionSchema& rhs) const {
+return !(*this == rhs);
+ }
// Transforms an exclusive lower bound range partition key into an inclusive
// lower bound range partition key.
diff --git a/src/kudu/common/schema.h b/src/kudu/common/schema.h
index 3371ba4..c00e66d 100644
--- a/src/kudu/common/schema.h
+++ b/src/kudu/common/schema.h
@@ -819,6 +819,14 @@ class Schema {
return true;
}
+ bool operator==(const Schema& other) const {
+return this->Equals(other);
+ }
+
+ bool operator!=(const Schema& other) const {
+return !(*this == other);
+ }
+
// Return true if the key projection schemas have exactly the same set of
// columns and respective types.
bool KeyEquals(const Schema& other,
diff --git a/src/kudu/master/catalog_manager.h
b/src/kudu/master/catalog_manager.h
index 0ffc79d..f6c177c 100644
--- a/src/kudu/master/catalog_manager.h
+++ b/src/kudu/master/catalog_manager.h
@@ -305,7 +305,7 @@ class TableInfo : public RefCountedThreadSafe {
void GetTabletsInRange(const GetTableLocationsRequestPB* req,
std::vector>* ret) const;
- // Adds all tablets to the vector in partition key sorted order.
+ // Fills the vector with all tablets in partition key sorted order.
void GetAllTablets(std::vector>* ret) const;
// Access the persistent metadata. Typically you should use
@@ -992,7 +992,7 @@ class CatalogManager : public
tserver::TabletReplicaLookupIf {
const
boost::optional& dimension_label);
// Builds the Ta
[kudu] branch master updated: [pruning] KUDU-2671: Pruning compatible with custom hash schemas.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new bca9d73 [pruning] KUDU-2671: Pruning compatible with custom hash
schemas.
bca9d73 is described below
commit bca9d73f90d02209e67615c140cd9c5311a6d8fb
Author: Mahesh Reddy
AuthorDate: Wed Mar 24 17:47:14 2021 -0700
[pruning] KUDU-2671: Pruning compatible with custom hash schemas.
This patch introduces changes to the PartitionPruner class
to be compatible with custom hash bucket schemas per range.
There are three ways to set bounds on a scan.
- Adding predicates (e.g. range/equality)
- Setting lower and upper bound primary keys
- Setting lower and upper bound partition keys
This patch introduces changes that make the first two methods
of setting bounds on a scan compatible with custom hash bucket
schemas per range. The last way using partition keys is unstable
and for internal use only. While it's not necessary for the last
way to be compatible with per range hash bucket schemas, the
entire pruning functionality will not be complete until
PartitionPruner::RemovePartitionKeyRange() is modified.
That work will be done in a follow up patch.
Change-Id: I05c37495430f61a2c6f6012c72251138aee465b7
Reviewed-on: http://gerrit.cloudera.org:8080/17643
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/common/partial_row.h|1 +
src/kudu/common/partition_pruner-test.cc | 1008 +++---
src/kudu/common/partition_pruner.cc | 377 ++-
src/kudu/common/partition_pruner.h | 54 +-
4 files changed, 1051 insertions(+), 389 deletions(-)
diff --git a/src/kudu/common/partial_row.h b/src/kudu/common/partial_row.h
index 9983137..47033b1 100644
--- a/src/kudu/common/partial_row.h
+++ b/src/kudu/common/partial_row.h
@@ -658,6 +658,7 @@ class KUDU_EXPORT KuduPartialRow {
FRIEND_TEST(PartitionPrunerTest, TestIntPartialPrimaryKeyRangePruning);
FRIEND_TEST(PartitionPrunerTest, TestPartialPrimaryKeyRangePruning);
FRIEND_TEST(PartitionPrunerTest, TestPrimaryKeyRangePruning);
+ FRIEND_TEST(PartitionPrunerTest,
TestHashSchemasPerRangeWithPartialPrimaryKeyRangePruning);
FRIEND_TEST(RowOperationsTest, ProjectionTestWholeSchemaSpecified);
FRIEND_TEST(RowOperationsTest, TestProjectUpdates);
FRIEND_TEST(RowOperationsTest, TestProjectDeletes);
diff --git a/src/kudu/common/partition_pruner-test.cc
b/src/kudu/common/partition_pruner-test.cc
index 45c2f34..1789154 100644
--- a/src/kudu/common/partition_pruner-test.cc
+++ b/src/kudu/common/partition_pruner-test.cc
@@ -20,13 +20,12 @@
#include
#include
#include
-#include
#include
+#include
#include
#include
#include
-#include
#include
#include "kudu/common/column_predicate.h"
@@ -35,12 +34,13 @@
#include "kudu/common/partial_row.h"
#include "kudu/common/partition.h"
#include "kudu/common/row.h"
+#include "kudu/common/row_operations.h"
+#include "kudu/common/row_operations.pb.h"
#include "kudu/common/scan_spec.h"
#include "kudu/common/schema.h"
#include "kudu/gutil/strings/substitute.h"
#include "kudu/util/memory/arena.h"
#include "kudu/util/slice.h"
-#include "kudu/util/status.h"
#include "kudu/util/test_macros.h"
#include "kudu/util/test_util.h"
@@ -48,15 +48,34 @@ using boost::optional;
using std::count_if;
using std::get;
using std::make_tuple;
-using std::move;
+using std::pair;
using std::string;
using std::tuple;
-using std::unique_ptr;
using std::vector;
namespace kudu {
class PartitionPrunerTest : public KuduTest {
+ public:
+ typedef tuple, int32_t, uint32_t>
ColumnNamesNumBucketsAndSeed;
+ typedef pair ColumnNameAndStringValue;
+ typedef pair ColumnNameAndIntValue;
+
+ static void CreatePartitionSchemaPB(
+ const vector& range_columns,
+ const vector& table_hash_schema,
+ PartitionSchemaPB* partition_schema_pb);
+
+ static void AddRangePartitionWithSchema(
+ const Schema& schema,
+ const vector& lower_string_cols,
+ const vector& upper_string_cols,
+ const vector& lower_int_cols,
+ const vector& upper_int_cols,
+ const vector& hash_schemas,
+ vector>* bounds,
+ PartitionSchema::PerRangeHashBucketSchemas* range_hash_schemas,
+ PartitionSchemaPB* pb);
};
void CheckPrunedPartitions(const Schema& schema,
@@ -75,8 +94,12 @@ void CheckPrunedPartitions(const Schema& schema,
SCOPED_TRACE(strings::Substitute("schema: $0", schema.ToString()));
SCOPED_TRACE(strings::Substitute("partition schema: $0",
[kudu] branch master updated: [test] deflake ToolTest.TableScanCustomBatchSize
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new e5e2242 [test] deflake ToolTest.TableScanCustomBatchSize
e5e2242 is described below
commit e5e22422356dc9ab2d3af8631e6646e0c2dc003b
Author: Andrew Wong
AuthorDate: Mon Aug 2 14:48:09 2021 -0700
[test] deflake ToolTest.TableScanCustomBatchSize
There seems to be a bug when running loadgen with AUTO_FLUSH_BACKGROUND
mode, wherein background write flushes aren't waited for before the
process is terminated, resulting in a leak report. I'll address the
underlying issue separately, but for now, this patch adjusts the test to
use fewer rows so no leaks are reported.
The flaky test dashboard reports the test as ~10% flaky. With this
patch, I ran it in ASAN and it passed 100/100 times.
Change-Id: I0740de18a3425cfd47aefe354b6a9fe05aa291cf
Reviewed-on: http://gerrit.cloudera.org:8080/17747
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/tools/kudu-tool-test.cc | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/kudu/tools/kudu-tool-test.cc b/src/kudu/tools/kudu-tool-test.cc
index aa70e15..c8015df 100644
--- a/src/kudu/tools/kudu-tool-test.cc
+++ b/src/kudu/tools/kudu-tool-test.cc
@@ -4065,7 +4065,7 @@ TEST_F(ToolTest, TableScanCustomBatchSize) {
NO_FATALS(RunLoadgen(1,
{
"--num_threads=5",
- "--num_rows_per_thread=2",
+ "--num_rows_per_thread=2000",
},
kTableName));
// Use 256 KiByte scan batch.
@@ -4074,11 +4074,11 @@ TEST_F(ToolTest, TableScanCustomBatchSize) {
string err;
vector out_lines;
const auto s = RunTool(
-Substitute("table scan $0 $1 --scan_batch_size=262144",
+Substitute("table scan $0 $1 --scan_batch_size=26214",
cluster_->master()->bound_rpc_addr().ToString(),
kTableName),
, , _lines);
ASSERT_TRUE(s.ok()) << s.ToString() << ": " << err;
-ASSERT_STR_CONTAINS(out, "Total count 10 ");
+ASSERT_STR_CONTAINS(out, "Total count 1 ");
}
}
[kudu] branch master updated (1ec8b20 -> 2d62c42)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 1ec8b20 [util] introduce comparison operators for Slice new e7cd1f6 [docker] KUDU-3307: allow using multiple directories new 2d62c42 [helm] fix usage of multiple directories The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docker/kudu-entrypoint.sh| 49 ++-- kubernetes/helm/README.adoc | 94 +++- kubernetes/helm/kudu/templates/_helmutils.tpl| 26 +-- kubernetes/helm/kudu/templates/kudu-service.yaml | 8 ++ kubernetes/helm/kudu/values.yaml | 6 ++ 5 files changed, 153 insertions(+), 30 deletions(-)
[kudu] 02/02: [helm] fix usage of multiple directories
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git commit 2d62c427bf0e4d62df2f0cb355ebfc03705070a1 Author: Andrew Wong AuthorDate: Thu Jul 29 18:43:57 2021 -0700 [helm] fix usage of multiple directories This patch updates the /helm implementation to use the new FS_WAL_DIR and FS_DATA_DIRS environment variables. This patch also updates the README with steps I used to test the changes. I verified that I could update the number of directories in kudu/values.yaml, confirming that the servers used multiple directories. Below is a snippet of the logs from one tserver with multiple directories (5, set via kudu/values.yaml). Previously, even with multiple storage devices set, each server would use a single directory /var/lib/kudu for storage. $ kubectl logs kudu-tserver-1 I0730 00:36:07.404302 1 tablet_server_runner.cc:78] Tablet server non-default flags: --use_hybrid_clock=false --fs_data_dirs=/mnt/disk1,/mnt/disk2,/mnt/disk3,/mnt/disk4 --fs_wal_dir=/mnt/disk0 --webserver_doc_root=/opt/kudu/www --tserver_master_addrs=kudu-master-0.kudu-masters.default.svc.cluster.local,kudu-master-1.kudu-masters.default.svc.cluster.local,kudu-master-2.kudu-masters.default.svc.cluster.local --heap_profile_path=/tmp/kudu.1 --stderrthreshold=0 Tablet server version: kudu 1.16.0-SNAPSHOT revision 2ceec7749 build type RELEASE ... A rendered README can be found here: https://github.com/andrwng/kudu/blob/helm_multidir/kubernetes/helm/README.adoc Change-Id: I583dff69c809e0851052c98267740a3e70c60efa Reviewed-on: http://gerrit.cloudera.org:8080/17739 Tested-by: Kudu Jenkins Reviewed-by: Bankim Bhavsar --- kubernetes/helm/README.adoc | 94 +++- kubernetes/helm/kudu/templates/_helmutils.tpl| 26 +-- kubernetes/helm/kudu/templates/kudu-service.yaml | 8 ++ kubernetes/helm/kudu/values.yaml | 6 ++ 4 files changed, 110 insertions(+), 24 deletions(-) diff --git a/kubernetes/helm/README.adoc b/kubernetes/helm/README.adoc index 0bc9019..3d7acdc 100644 --- a/kubernetes/helm/README.adoc +++ b/kubernetes/helm/README.adoc @@ -21,10 +21,13 @@ NOTE: All of this work is experimental and subject to change or removal. == Getting Started -Helm — The package manager for Kubernetes. Helps to define, install, and upgrade Kubernetes applications +Helm — The package manager for Kubernetes. Helps to define, install, and +upgrade Kubernetes applications. NOTE: Read more about Helm here https://helm.sh/docs/using_helm/#quickstart +The below instructions rely on having Kudu Docker images built. + System Requirements kubectl @@ -32,31 +35,86 @@ NOTE: Read more about Helm here https://helm.sh/docs/using_helm/#quickstart docker helm - Build Kudu Docker Image - -../../docker/docker-build.py +=== Using Helm v3 + +. Deploy a Kubernetes cluster with a cluster manager choice. For the sake of + local development, `minikube` is a fine choice: ++ + +$ minikube start + + +. If you have made changes to your Docker images, or would otherwise like to + use local images rather than those found on DockerHub, ensure your Kubernetes + cluster can access your desired Docker images. ++ + +$ minikube cache add apache/kudu:latest + + +. Install the `apache-kudu` Helm Chart. Optionally supply an edited + `values.yaml` file. This deploys a Kudu cluster using the `minikube` + container. ++ + +$ helm install -f kudu/values.yaml apache-kudu ./kudu + + +. Verify the cluster is running and view its logs. ++ + +$ kubectl get pods +NAME READY STATUSRESTARTS AGE +kudu-master-01/1 Running 0 108s +kudu-master-11/1 Running 0 108s +kudu-master-21/1 Running 0 108s +kudu-tserver-0 1/1 Running 0 108s +kudu-tserver-1 1/1 Running 0 108s +kudu-tserver-2 1/1 Running 0 108s + +$ kubectl logs kudu-tserver-1 +I0730 00:36:07.404302 1 tablet_server_runner.cc:78] Tablet server non-default flags: +--use_hybrid_clock=false +--fs_data_dirs=/mnt/disk1,/mnt/disk2,/mnt/disk3,/mnt/disk4 +--fs_wal_dir=/mnt/disk0 +--webserver_doc_root=/opt/kudu/www +--tserver_master_addrs=kudu-master-0.kudu-masters.default.svc.cluster.local,kudu-master-1.kudu-masters.default.svc.cluster.local,kudu-master-2.kudu-masters.default.svc.cluster.local +--heap_profile_path=/tmp/kudu.1 +--stderrthreshold=0 + +Tablet server version: +... + + +. To stop Kudu, uninstall the Helm Chart. ++ + +$ helm uninstall apache-kudu + + +=== Using Helm v2 Creating Namespace -kubectl create -f ../namespace.yaml +$ kubectl create -f ../namespace.yaml Creating ServiceAccount And Role
[kudu] 01/02: [docker] KUDU-3307: allow using multiple directories
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e7cd1f6cfab4d8448ad4ebd834727c83f23f4899
Author: Andrew Wong
AuthorDate: Wed Jul 28 15:54:29 2021 -0700
[docker] KUDU-3307: allow using multiple directories
Previously, the Docker entrypoint would by default use a single root
directory, suffixed with either 'master' or 'tserver'. This makes it a
poor fit for production systems that may want to use Docker, since most
reasonable deployments use multple data directories, and don't colocate
WALs and data.
This patch adds a couple of environment variables to the entrypoint,
FS_WAL_DIR and FS_DATA_DIRS, that map directly to the --fs_wal_dir and
--fs_data_dirs gflags that Kudu masters and tservers know and love.
To maintain compatibility with existing guides, the existing single-root
DATA_DIR remains, but the new environment variables, if set, take
precedence.
Change-Id: I20e5f75c1f1f8280dca60de75255a4a948e44be1
Reviewed-on: http://gerrit.cloudera.org:8080/17738
Tested-by: Kudu Jenkins
Reviewed-by: Bankim Bhavsar
---
docker/kudu-entrypoint.sh | 49 +--
1 file changed, 43 insertions(+), 6 deletions(-)
diff --git a/docker/kudu-entrypoint.sh b/docker/kudu-entrypoint.sh
index 814bf0d..21485db 100755
--- a/docker/kudu-entrypoint.sh
+++ b/docker/kudu-entrypoint.sh
@@ -39,8 +39,18 @@ function print_help {
echo " Defines the kudu-master and kudu-tserver configured master
addresses."
echo " Defaults to localhost."
echo "DATA_DIR:"
- echo " Defines the root data directory to use."
+ echo " Defines the root directory to use. Subdirectories are added
depending on whether a "
+ echo " Kudu master or a Kudu tablet server is being deployed. Ignored if
the FS_WAL_DIR "
+ echo " environment variable is set."
+ echo " NOTE: this variable is deprecated. FS_WAL_DIR should be used
instead."
echo " Defaults to /var/lib/kudu."
+ echo "FS_WAL_DIR:"
+ echo " Defines the WAL directory to use. Takes precedence over the DATA_DIR
environment "
+ echo " variable."
+ echo "FS_DATA_DIRS:"
+ echo " Defines the data directories to use. If set, the FS_WAL_DIR
environment variable "
+ echo " must also be set."
+ echo " Defaults to the value of the FS_WAL_DIR environment variable."
echo "MASTER_ARGS:"
echo " Defines the arguments passed to kudu-master."
echo " Defaults to the value of the DEFAULT_ARGS environment variable."
@@ -51,12 +61,29 @@ function print_help {
echo " Defines a recommended base set of arguments."
}
+if [[ -z "$FS_WAL_DIR" && -n "$FS_DATA_DIRS" ]]; then
+ echo "If FS_DATA_DIRS is set, FS_WAL_DIR must also be set"
+ echo "FS_WAL_DIR: $FS_WAL_DIR"
+ echo "FS_DATA_DIRS: $FS_DATA_DIRS"
+ exit 1
+fi
+
+DATA_DIR=${DATA_DIR:="/var/lib/kudu"}
+if [[ -n "$FS_WAL_DIR" ]]; then
+ # Use the WAL directory for data if a data directory is not specified.
+ WAL_DIR="$FS_WAL_DIR"
+ DATA_DIRS=${FS_DATA_DIRS:="$FS_WAL_DIR"}
+else
+ # If no WAL directory is specified, use a subdirectory in the root directory.
+ WAL_DIR="$DATA_DIR/$1"
+ DATA_DIRS="$DATA_DIR/$1"
+fi
+
# Define the defaults environment variables.
KUDU_MASTERS=${KUDU_MASTERS:=""}
-DATA_DIR=${DATA_DIR:="/var/lib/kudu"}
-SERVICE_DIR="$DATA_DIR/$1"
# TODO: Remove use_hybrid_clock=false when ntpd is setup.
-DEFAULT_ARGS="--fs_wal_dir=$SERVICE_DIR \
+DEFAULT_ARGS="--fs_wal_dir=$WAL_DIR \
+ --fs_data_dirs=$DATA_DIRS \
--webserver_doc_root=/opt/kudu/www \
--stderrthreshold=0 \
--use_hybrid_clock=false"
@@ -85,6 +112,16 @@ function wait_for_master_hosts() {
unset IFS
}
+function make_directories() {
+ IFS=","
+ mkdir -p $WAL_DIR
+ for DIR in $DATA_DIRS
+ do
+mkdir -p $DIR
+ done
+ unset IFS
+}
+
# If no arguments are passed, print the help.
if [[ -z "$1" ]]; then
print_help
@@ -94,7 +131,7 @@ fi
# Note: we use "master" and "tserver" here so the kudu-master and kudu-tserver
# binaries can be manually invoked if needed.
if [[ "$1" == "master" ]]; then
- mkdir -p "$SERVICE_DIR"
+ make_directories
wait_for_master_hosts
# Supply --master_addresses even if a single master address is specified.
if [[ -n "$KUDU_MASTERS" ]]; then
@@ -102,7 +139,7 @@ if [[ "$1" == "master" ]]; then
fi
exec kudu master run ${DEFAULT_ARGS} ${MASTER_ARGS}
elif [[ "$1" == "tserver" ]]; then
- mkdir -p "$SERVICE_DIR"
+ make_directories
wait_for_master_hosts
if [[ -n "$KUDU_MASTERS" ]]; then
TSERVER_ARGS="--tserver_master_addrs=$KUDU_MASTERS $TSERVER_ARGS"
[kudu-site] branch asf-site updated: Add .asf.yaml file
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/kudu-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 5ff5ada Add .asf.yaml file 5ff5ada is described below commit 5ff5ada388cb8980f402a863981470cfd8ecffb9 Author: Andrew Wong AuthorDate: Tue Jul 27 16:30:25 2021 -0700 Add .asf.yaml file This is a follow-up from the "[NOTICE] Git web site publishing to be done via .asf.yaml only as of July 1st" email sent to us...@infra.apache.org. --- .asf.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.asf.yaml b/.asf.yaml new file mode 100644 index 000..6e33a03 --- /dev/null +++ b/.asf.yaml @@ -0,0 +1,2 @@ +publish: +whoami: asf-site
[kudu] branch master updated: [master] AddMaster and copy sys catalog automatically
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new 7e66534 [master] AddMaster and copy sys catalog automatically 7e66534 is described below commit 7e66534d0e62fb850bf300d52da4a0a76889f4b8 Author: Andrew Wong AuthorDate: Tue Jun 22 18:52:21 2021 -0700 [master] AddMaster and copy sys catalog automatically This patch updates the masters to ping all other masters, collect their UUIDs and Raft configs, and detect whether the master being started meant to join the cluster via AddMaster RPC. If so, it sends the RPC, performs a tablet copy, and bootstraps the master from the copy. This patch is fairly conservative about when it will perform the automatic operations -- the existing master Raft config must be in good health, with all masters agreeing on members and the current term. Otherwise, the new master will fail to come up, as it's not a part of the existing cluster. This patch includes a gflag with which to disable this behavior. It is disabled in the following cases: - when running the `kudu master add` tooling - tests for the existing tooling Some test utility updates come along with this patch as well, used in newly introduced tests: - the ExternalMiniCluster can now add a master - ExternalMiniCluster::WaitForCatalogManager() is made more robust to the shutting down and starting up of a master In order to get this to work in a secured environment, this patch also updates the coarse grained authz requirements of the AddMaster endpoint to allow the service user to send RPCs. Sample usage (from build/latest): $ ../../src/kudu/scripts/start_kudu.sh $ mkdir -p master-1/log $ ./bin/kudu-master --master_addresses=127.0.0.1:8764,127.0.0.1:8766 --fs_data_dirs=$PWD/master-1/data --fs_wal_dir=$PWD/master-1/wal --log_dir=$PWD/master-1/log --rpc_bind_addresses=127.0.0.1:8766 --time_source=system_unsync --unlock_unsafe_flags --webserver_port=8767 --webserver_interface=127.0.0.1 --webserver_doc_root=$PWD/../../www --logtostderr ... $ ../../src/kudu/scripts/stop_kudu.sh $ ../../src/kudu/scripts/start_kudu.sh --num-masters 2 Change-Id: Ic7635723f30317a45799ad7b9c9b725eafbd735b Reviewed-on: http://gerrit.cloudera.org:8080/17528 Reviewed-by: Alexey Serbin Tested-by: Alexey Serbin --- src/kudu/master/CMakeLists.txt | 2 +- src/kudu/master/dynamic_multi_master-test.cc | 631 - src/kudu/master/master.proto | 5 +- src/kudu/master/master_options.h | 7 +- src/kudu/master/master_runner.cc | 338 ++- .../mini-cluster/external_mini_cluster-test.cc | 40 +- src/kudu/mini-cluster/external_mini_cluster.cc | 240 +--- src/kudu/mini-cluster/external_mini_cluster.h | 22 +- src/kudu/tools/tool_action_master.cc | 1 + 9 files changed, 1043 insertions(+), 243 deletions(-) diff --git a/src/kudu/master/CMakeLists.txt b/src/kudu/master/CMakeLists.txt index dafeea6..5dd8bae 100644 --- a/src/kudu/master/CMakeLists.txt +++ b/src/kudu/master/CMakeLists.txt @@ -108,7 +108,7 @@ SET_KUDU_TEST_LINK_LIBS( ADD_KUDU_TEST(auto_rebalancer-test) ADD_KUDU_TEST(catalog_manager-test) -ADD_KUDU_TEST(dynamic_multi_master-test NUM_SHARDS 4) +ADD_KUDU_TEST(dynamic_multi_master-test NUM_SHARDS 6) ADD_KUDU_TEST(hms_notification_log_listener-test) ADD_KUDU_TEST(location_cache-test DATA_FILES ../scripts/first_argument.sh) ADD_KUDU_TEST(master_options-test) diff --git a/src/kudu/master/dynamic_multi_master-test.cc b/src/kudu/master/dynamic_multi_master-test.cc index e473a93..e6d345d 100644 --- a/src/kudu/master/dynamic_multi_master-test.cc +++ b/src/kudu/master/dynamic_multi_master-test.cc @@ -16,13 +16,16 @@ // under the License. #include +#include #include #include #include #include #include +#include #include #include +#include #include #include #include @@ -49,6 +52,7 @@ #include "kudu/gutil/strings/substitute.h" #include "kudu/integration-tests/cluster_itest_util.h" #include "kudu/integration-tests/cluster_verifier.h" +#include "kudu/integration-tests/test_workload.h" #include "kudu/master/master.h" #include "kudu/master/master.pb.h" #include "kudu/master/master.proxy.h" @@ -60,12 +64,15 @@ #include "kudu/tserver/tserver.pb.h" #include "kudu/util/env.h" #include "kudu/util/env_util.h" +#include "kudu/util/locks.h" #include "kudu/util/metrics.h" #include "kudu/util/monotime.h" #include "kudu/util/net/net_util.h" #include "kudu/util/net/so
[kudu] 01/02: [util] flag to specify hosts to fail DNS resolution
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit c2f4cf7b8b3240feef6ec0060c99f78d3744d76d
Author: Andrew Wong
AuthorDate: Wed Jun 9 12:42:31 2021 -0700
[util] flag to specify hosts to fail DNS resolution
This patch adds a flag to specify the specific hostports to which
failures are injected by --fail_dns_resolution. I intend on using this
in a subsequent patch.
Change-Id: I25f2d3073fa0fa0a430afacf7c8cd1402e6bd6ab
Reviewed-on: http://gerrit.cloudera.org:8080/17584
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/util/net/net_util-test.cc | 39 ++
src/kudu/util/net/net_util.cc | 24 +--
2 files changed, 61 insertions(+), 2 deletions(-)
diff --git a/src/kudu/util/net/net_util-test.cc
b/src/kudu/util/net/net_util-test.cc
index ae85220..c409b2b 100644
--- a/src/kudu/util/net/net_util-test.cc
+++ b/src/kudu/util/net/net_util-test.cc
@@ -25,10 +25,12 @@
#include
#include
+#include
#include
#include
#include "kudu/gutil/strings/join.h"
+#include "kudu/gutil/strings/substitute.h"
#include "kudu/gutil/strings/util.h"
#include "kudu/util/net/sockaddr.h"
#include "kudu/util/net/socket.h"
@@ -36,8 +38,12 @@
#include "kudu/util/test_macros.h"
#include "kudu/util/test_util.h"
+DECLARE_bool(fail_dns_resolution);
+DECLARE_string(fail_dns_resolution_hostports);
+
using std::string;
using std::vector;
+using strings::Substitute;
namespace kudu {
@@ -128,6 +134,39 @@ TEST_F(NetUtilTest, TestParseAddressesWithScheme) {
ASSERT_STR_CONTAINS(s.ToString(), "invalid scheme format");
}
+TEST_F(NetUtilTest, TestInjectFailureToResolveAddresses) {
+ HostPort hp1("localhost", 12345);
+ HostPort hp2("localhost", 12346);
+ FLAGS_fail_dns_resolution_hostports = hp1.ToString();
+ FLAGS_fail_dns_resolution = true;
+
+ // With a list of bad hostports specified, check that resolution fails as
+ // expected.
+ vector addrs;
+ Status s = hp1.ResolveAddresses();
+ ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+
+ // Addresses not in the list should resolve fine.
+ ASSERT_TRUE(addrs.empty());
+ ASSERT_OK(hp2.ResolveAddresses());
+ ASSERT_FALSE(addrs.empty());
+
+ // With both in the list, resolution should fail.
+ FLAGS_fail_dns_resolution_hostports = Substitute("$0,$1", hp1.ToString(),
hp2.ToString());
+ addrs.clear();
+ s = hp1.ResolveAddresses();
+ ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+ s = hp2.ResolveAddresses();
+ ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+
+ // When a list isn't specified, all resolution fails.
+ FLAGS_fail_dns_resolution_hostports = "";
+ s = hp1.ResolveAddresses();
+ ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+ s = hp2.ResolveAddresses();
+ ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+}
+
TEST_F(NetUtilTest, TestResolveAddresses) {
HostPort hp("localhost", 12345);
vector addrs;
diff --git a/src/kudu/util/net/net_util.cc b/src/kudu/util/net/net_util.cc
index edfa8e6..ce5e156 100644
--- a/src/kudu/util/net/net_util.cc
+++ b/src/kudu/util/net/net_util.cc
@@ -64,14 +64,19 @@
#define HOST_NAME_MAX 64
#endif
-DEFINE_bool(fail_dns_resolution, false, "Whether to fail all dns resolution,
for tests.");
+DEFINE_bool(fail_dns_resolution, false, "Whether to fail dns resolution, for
tests.");
TAG_FLAG(fail_dns_resolution, hidden);
+DEFINE_string(fail_dns_resolution_hostports, "",
+ "Comma-separated list of hostports that fail dns resolution. If
empty, fails all "
+ "dns resolution attempts. Only takes effect if
--fail_dns_resolution is 'true'.");
+TAG_FLAG(fail_dns_resolution_hostports, hidden);
using std::function;
using std::string;
using std::unordered_set;
using std::unique_ptr;
using std::vector;
+using strings::Split;
using strings::Substitute;
namespace kudu {
@@ -215,7 +220,22 @@ Status HostPort::ResolveAddresses(vector*
addresses) const {
}
}
if (PREDICT_FALSE(FLAGS_fail_dns_resolution)) {
-return Status::NetworkError("injected DNS resolution failure");
+if (FLAGS_fail_dns_resolution_hostports.empty()) {
+ return Status::NetworkError("injected DNS resolution failure");
+}
+unordered_set failed_hostports =
+Split(FLAGS_fail_dns_resolution_hostports, ",");
+for (const auto& hp_str : failed_hostports) {
+ HostPort hp;
+ Status s = hp.ParseString(hp_str, /*default_port=*/0);
+ if (!s.ok()) {
+LOG(WARNING) << "Could not parse: " << hp_str;
+continue;
+ }
+ if (hp == *this) {
+return Status::NetworkError("injected DNS resolution failure", hp_str);
+ }
+}
}
if (addresses) {
*addresses = std::move(result_addresses);
[kudu] 02/02: [consensus] KUDU-2302: don't crash if new leader can't resolve peer
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f9647149a49ddb87ea0ecf069eab3b5ec0217136
Author: Andrew Wong
AuthorDate: Wed Jun 9 11:55:36 2021 -0700
[consensus] KUDU-2302: don't crash if new leader can't resolve peer
When a tablet replica is elected leader, it constructs Peer objects for
each replica in the Raft config for the sake of sending RPCs to each.
If, during this construction, any remote peer cannot be reached for
whatever reason, this would result in a crash.
Rather than crashing, this patch allows us to start Peers without a
proxy, and retries constructing the proxy the next time a proxy is
required.
Change-Id: I22d870ecc526fa47b97f6856c3b023bc1ec029c7
Reviewed-on: http://gerrit.cloudera.org:8080/17585
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/consensus/consensus-test-util.h | 22
src/kudu/consensus/consensus_peers-test.cc | 26 +++--
src/kudu/consensus/consensus_peers.cc | 42 ++--
src/kudu/consensus/consensus_peers.h | 23 +++-
src/kudu/consensus/peer_manager.cc | 11 +-
src/kudu/consensus/peer_manager.h | 10 +-
src/kudu/consensus/raft_consensus.cc | 8 +-
src/kudu/consensus/raft_consensus.h| 2 +-
src/kudu/integration-tests/cluster_itest_util.cc | 7 +-
src/kudu/integration-tests/cluster_itest_util.h| 6 +-
.../raft_consensus_election-itest.cc | 119 +
11 files changed, 233 insertions(+), 43 deletions(-)
diff --git a/src/kudu/consensus/consensus-test-util.h
b/src/kudu/consensus/consensus-test-util.h
index 7f2fbbd..5e2d3ee 100644
--- a/src/kudu/consensus/consensus-test-util.h
+++ b/src/kudu/consensus/consensus-test-util.h
@@ -254,6 +254,28 @@ class DelayablePeerProxy : public TestPeerProxy {
CountDownLatch latch_;
};
+// Factory that takes ownership of an input pointer to an already constructed
+// proxy that is expected to only be used once. It is up to users to make sure
+// this is safe.
+template
+class OneTimeUsePeerProxyFactory : public PeerProxyFactory {
+ public:
+ OneTimeUsePeerProxyFactory(std::shared_ptr messenger,
+ ProxyT* proxy)
+ : messenger_(std::move(messenger)),
+proxy_(proxy) {}
+ Status NewProxy(const RaftPeerPB& /*peer_pb*/, std::unique_ptr*
proxy) override {
+*proxy = std::unique_ptr(proxy_);
+return Status::OK();
+ }
+ const std::shared_ptr& messenger() const override {
+return messenger_;
+ }
+ protected:
+ std::shared_ptr messenger_;
+ ProxyT* proxy_;
+};
+
// Allows complete mocking of a peer's responses.
// You set the response, it will respond with that.
class MockedPeerProxy : public TestPeerProxy {
diff --git a/src/kudu/consensus/consensus_peers-test.cc
b/src/kudu/consensus/consensus_peers-test.cc
index db547b2..ff1a3fa 100644
--- a/src/kudu/consensus/consensus_peers-test.cc
+++ b/src/kudu/consensus/consensus_peers-test.cc
@@ -70,6 +70,9 @@ const char* kTabletId = "test-peers-tablet";
const char* kLeaderUuid = "peer-0";
const char* kFollowerUuid = "peer-1";
+template class
OneTimeUsePeerProxyFactory>;
+template class OneTimeUsePeerProxyFactory;
+
class ConsensusPeersTest : public KuduTest {
public:
ConsensusPeersTest()
@@ -135,16 +138,18 @@ class ConsensusPeersTest : public KuduTest {
RaftPeerPB peer_pb;
peer_pb.set_permanent_uuid(peer_name);
peer_pb.set_member_type(RaftPeerPB::VOTER);
-auto proxy_ptr = new DelayablePeerProxy(
-raft_pool_.get(), new NoOpTestPeerProxy(raft_pool_.get(), peer_pb));
-unique_ptr proxy(proxy_ptr);
+std::unique_ptr> proxy(
+new DelayablePeerProxy(
+raft_pool_.get(), new NoOpTestPeerProxy(raft_pool_.get(),
peer_pb)));
+auto* proxy_ptr = proxy.get();
+OneTimeUsePeerProxyFactory> factory(
+messenger_, proxy.release());
Peer::NewRemotePeer(std::move(peer_pb),
kTabletId,
kLeaderUuid,
message_queue_.get(),
raft_pool_token_.get(),
-std::move(proxy),
-messenger_,
+,
peer);
return proxy_ptr;
}
@@ -179,6 +184,7 @@ class ConsensusPeersTest : public KuduTest {
unique_ptr raft_pool_token_;
unique_ptr clock_;
shared_ptr messenger_;
+
};
@@ -276,14 +282,14 @@ TEST_F(ConsensusPeersTest,
TestCloseWhenRemotePeerDoesntMakeProgress) {
BuildRaftConfigPBForTests(3));
auto mock_proxy = new MockedPeerProxy(raft_pool_.get());
+ OneTimeUsePeerProxyFactory factory(messenger_, mock_proxy);
shared_ptr peer;
P
[kudu] branch master updated (7673e0a -> f964714)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 7673e0a [clock] use GTEST_SKIP in BuiltinNtpWithMiniChronydTest new c2f4cf7 [util] flag to specify hosts to fail DNS resolution new f964714 [consensus] KUDU-2302: don't crash if new leader can't resolve peer The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/consensus/consensus-test-util.h | 22 src/kudu/consensus/consensus_peers-test.cc | 26 +++-- src/kudu/consensus/consensus_peers.cc | 42 ++-- src/kudu/consensus/consensus_peers.h | 23 +++- src/kudu/consensus/peer_manager.cc | 11 +- src/kudu/consensus/peer_manager.h | 10 +- src/kudu/consensus/raft_consensus.cc | 8 +- src/kudu/consensus/raft_consensus.h| 2 +- src/kudu/integration-tests/cluster_itest_util.cc | 7 +- src/kudu/integration-tests/cluster_itest_util.h| 6 +- .../raft_consensus_election-itest.cc | 119 + src/kudu/util/net/net_util-test.cc | 39 +++ src/kudu/util/net/net_util.cc | 24 - 13 files changed, 294 insertions(+), 45 deletions(-)
[kudu] branch branch-1.15.x updated: KUDU-3291: properly disambiguate between deltas of a row with the same timestamp
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.15.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/branch-1.15.x by this push:
new d98c398 KUDU-3291: properly disambiguate between deltas of a row with
the same timestamp
d98c398 is described below
commit d98c398d819f57ba4065f0ba29f4ea7cc45fba46
Author: Andrew Wong
AuthorDate: Sun Jun 6 21:02:38 2021 -0700
KUDU-3291: properly disambiguate between deltas of a row with the same
timestamp
In performing a diff scan, Kudu iterates in small batches of rows,
selecting deltas associated with each row that are relevant to the
scan's timestamp bounds. Once all selected deltas are collected for a
given row, the oldest and newest deltas are found by a sorting criteria
meant to sort deltas by their application order:
1. Deltas of lower timestamps are less than deltas of higher timestmaps.
2. UNDO deltas are less than REDO deltas.
3. Within each delta store's iterator, a counter of selected deltas is
used to disambiguate between rows that have the same timestamp. A
critical assumption here is that this disambiguator is only used for
deltas of the same delta store. For REDOs, a lower counter implies
a lower application order -- the opposite is true for UNDOs.
What the above criteria don't account for is the fact that certain
iterators can iterate over separate delta stores that have deltas of the
same timestamp and type. If Kudu delta flushes while applying a large
batch of updates to the same row, the result is that some of the batch's
updates can land in the newly flushed REDO delta file, while the rest
land in the new DMS. In iterating over these stores with a
DeltaIteratorMerger, which combines the deltas of several delta stores,
this breaks the assumption described above, resulting in the crash
reported in KUDU-3291.
To remediate this, in iterators that merge multiple delta stores,
namely, the DeltaIteratorMerger, a single top-level counter is used to
guide the disambiguators generated by each sub-iterator. Before
iterating over a new batch of deltas in a given sub-iterator, this
counter is propagated to the sub-iterators as the new starting point of
its counter. The result is that the disambiguators generated by the
DeltaIteratorMerger can be used to define a total ordering of the deltas
selected.
Change-Id: Iccfc518999d36679f85ed901ba65cf7b4894cd55
Reviewed-on: http://gerrit.cloudera.org:8080/17547
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
Reviewed-by: Grant Henke
(cherry picked from commit 9ecee7ba065f1f7c844f1afd7136ab0565ce2340)
Reviewed-on: http://gerrit.cloudera.org:8080/17561
Reviewed-by: Bankim Bhavsar
Tested-by: Kudu Jenkins
---
src/kudu/tablet/delta_iterator_merger.cc | 5 ++-
src/kudu/tablet/delta_iterator_merger.h | 27 +++---
src/kudu/tablet/delta_store.cc | 5 ---
src/kudu/tablet/delta_store.h| 43 +-
src/kudu/tablet/deltafile.h | 8 +
src/kudu/tablet/deltamemstore.h | 8 +
src/kudu/tablet/diff_scan-test.cc| 62
src/kudu/tablet/tablet-test-base.h | 33 ++---
8 files changed, 158 insertions(+), 33 deletions(-)
diff --git a/src/kudu/tablet/delta_iterator_merger.cc
b/src/kudu/tablet/delta_iterator_merger.cc
index 24ef27b..e3d0af0 100644
--- a/src/kudu/tablet/delta_iterator_merger.cc
+++ b/src/kudu/tablet/delta_iterator_merger.cc
@@ -41,7 +41,8 @@ using strings::Substitute;
DeltaIteratorMerger::DeltaIteratorMerger(
vector > iters)
-: iters_(std::move(iters)) {}
+: total_deltas_selected_in_prepare_(0),
+ iters_(std::move(iters)) {}
Status DeltaIteratorMerger::Init(ScanSpec* spec) {
for (const unique_ptr : iters_) {
@@ -59,7 +60,9 @@ Status DeltaIteratorMerger::SeekToOrdinal(rowid_t idx) {
Status DeltaIteratorMerger::PrepareBatch(size_t nrows, int prepare_flags) {
for (const unique_ptr : iters_) {
+iter->set_deltas_selected(total_deltas_selected_in_prepare_);
RETURN_NOT_OK(iter->PrepareBatch(nrows, prepare_flags));
+total_deltas_selected_in_prepare_ = iter->deltas_selected();
}
return Status::OK();
}
diff --git a/src/kudu/tablet/delta_iterator_merger.h
b/src/kudu/tablet/delta_iterator_merger.h
index bedf552..4c87e4d 100644
--- a/src/kudu/tablet/delta_iterator_merger.h
+++ b/src/kudu/tablet/delta_iterator_merger.h
@@ -14,14 +14,17 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
-#ifndef KUDU_TABLET_DELTA_ITERATOR_MERGER_H
-#define KUDU_TABLET_DELTA_ITERATOR_MERGER_H
+#pragma once
#include
+#include
#includ
[kudu] branch master updated: KUDU-3291: properly disambiguate between deltas of a row with the same timestamp
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 3a1ce30 KUDU-3291: properly disambiguate between deltas of a row with
the same timestamp
3a1ce30 is described below
commit 3a1ce304b3762166ba069767b842a42ea9af1009
Author: Andrew Wong
AuthorDate: Sun Jun 6 21:02:38 2021 -0700
KUDU-3291: properly disambiguate between deltas of a row with the same
timestamp
In performing a diff scan, Kudu iterates in small batches of rows,
selecting deltas associated with each row that are relevant to the
scan's timestamp bounds. Once all selected deltas are collected for a
given row, the oldest and newest deltas are found by a sorting criteria
meant to sort deltas by their application order:
1. Deltas of lower timestamps are less than deltas of higher timestmaps.
2. UNDO deltas are less than REDO deltas.
3. Within each delta store's iterator, a counter of selected deltas is
used to disambiguate between rows that have the same timestamp. A
critical assumption here is that this disambiguator is only used for
deltas of the same delta store. For REDOs, a lower counter implies
a lower application order -- the opposite is true for UNDOs.
What the above criteria don't account for is the fact that certain
iterators can iterate over separate delta stores that have deltas of the
same timestamp and type. If Kudu delta flushes while applying a large
batch of updates to the same row, the result is that some of the batch's
updates can land in the newly flushed REDO delta file, while the rest
land in the new DMS. In iterating over these stores with a
DeltaIteratorMerger, which combines the deltas of several delta stores,
this breaks the assumption described above, resulting in the crash
reported in KUDU-3291.
To remediate this, in iterators that merge multiple delta stores,
namely, the DeltaIteratorMerger, a single top-level counter is used to
guide the disambiguators generated by each sub-iterator. Before
iterating over a new batch of deltas in a given sub-iterator, this
counter is propagated to the sub-iterators as the new starting point of
its counter. The result is that the disambiguators generated by the
DeltaIteratorMerger can be used to define a total ordering of the deltas
selected.
Change-Id: Iccfc518999d36679f85ed901ba65cf7b4894cd55
Reviewed-on: http://gerrit.cloudera.org:8080/17547
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
Reviewed-by: Grant Henke
---
src/kudu/tablet/delta_iterator_merger.cc | 5 ++-
src/kudu/tablet/delta_iterator_merger.h | 27 +++---
src/kudu/tablet/delta_store.cc | 5 ---
src/kudu/tablet/delta_store.h| 43 +-
src/kudu/tablet/deltafile.h | 8 +
src/kudu/tablet/deltamemstore.h | 8 +
src/kudu/tablet/diff_scan-test.cc| 62
src/kudu/tablet/tablet-test-base.h | 33 ++---
8 files changed, 158 insertions(+), 33 deletions(-)
diff --git a/src/kudu/tablet/delta_iterator_merger.cc
b/src/kudu/tablet/delta_iterator_merger.cc
index 24ef27b..e3d0af0 100644
--- a/src/kudu/tablet/delta_iterator_merger.cc
+++ b/src/kudu/tablet/delta_iterator_merger.cc
@@ -41,7 +41,8 @@ using strings::Substitute;
DeltaIteratorMerger::DeltaIteratorMerger(
vector > iters)
-: iters_(std::move(iters)) {}
+: total_deltas_selected_in_prepare_(0),
+ iters_(std::move(iters)) {}
Status DeltaIteratorMerger::Init(ScanSpec* spec) {
for (const unique_ptr : iters_) {
@@ -59,7 +60,9 @@ Status DeltaIteratorMerger::SeekToOrdinal(rowid_t idx) {
Status DeltaIteratorMerger::PrepareBatch(size_t nrows, int prepare_flags) {
for (const unique_ptr : iters_) {
+iter->set_deltas_selected(total_deltas_selected_in_prepare_);
RETURN_NOT_OK(iter->PrepareBatch(nrows, prepare_flags));
+total_deltas_selected_in_prepare_ = iter->deltas_selected();
}
return Status::OK();
}
diff --git a/src/kudu/tablet/delta_iterator_merger.h
b/src/kudu/tablet/delta_iterator_merger.h
index bedf552..4c87e4d 100644
--- a/src/kudu/tablet/delta_iterator_merger.h
+++ b/src/kudu/tablet/delta_iterator_merger.h
@@ -14,14 +14,17 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
-#ifndef KUDU_TABLET_DELTA_ITERATOR_MERGER_H
-#define KUDU_TABLET_DELTA_ITERATOR_MERGER_H
+#pragma once
#include
+#include
#include
+#include
#include
#include
+#include
+
#include "kudu/common/rowid.h"
#include "kudu/tablet/delta_store.h"
#include "kudu/util/status.h"
@@ -44,7 +
[kudu] branch master updated: KUDU-3223: management of per-table limit by kudu CLI
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 323d8ea KUDU-3223: management of per-table limit by kudu CLI
323d8ea is described below
commit 323d8ea7312678b7a910e55ed4f74acaf3813ffc
Author: Hongjiang Zhang
AuthorDate: Fri May 14 18:26:14 2021 +0800
KUDU-3223: management of per-table limit by kudu CLI
Use kudu CLI to set disk_size_limit and row_count_limit on table level.
If '--enable_table_write_limit' is not set 'true' for kudu-master, kudu
CLI fails to set the limit. The table's disk_size_limit and
row_count_limit are 'N/A' when calling 'kudu table statistics'.
Only when '--enable_table_write_limit' is set 'true', can the kudu CLI
change the table limit. Use 'kudu table statistics' to check the
updated result.
Change-Id: I59bb64cc85d5fca48ae8ec980f5cfc62e4b3a1d3
Reviewed-on: http://gerrit.cloudera.org:8080/17444
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/master/catalog_manager.cc | 30 ++-
src/kudu/tools/kudu-tool-test.cc| 157
src/kudu/tools/tool_action_table.cc | 57 +
3 files changed, 240 insertions(+), 4 deletions(-)
diff --git a/src/kudu/master/catalog_manager.cc
b/src/kudu/master/catalog_manager.cc
index 12666d9..447126b 100644
--- a/src/kudu/master/catalog_manager.cc
+++ b/src/kudu/master/catalog_manager.cc
@@ -2894,9 +2894,15 @@ Status CatalogManager::AlterTable(const
AlterTableRequestPB& req,
!req.new_extra_configs().empty() ||
!alter_schema_steps.empty() ||
!alter_partitioning_steps.empty();
+ if (table_limit_change && !FLAGS_enable_table_write_limit) {
+return SetupError(Status::NotSupported(
+ "altering table limit is not supported because "
+ "--enable_table_write_limit is not enabled"),
+ resp, MasterErrorPB::UNKNOWN_ERROR);
+ }
if (table_limit_change && other_schema_change) {
return SetupError(Status::ConfigurationError(
- "Alter table limit cannot be combined with other
alterations"),
+ "altering table limit cannot be combined with other
alterations"),
resp, MasterErrorPB::UNKNOWN_ERROR);
}
@@ -2944,13 +2950,17 @@ Status CatalogManager::AlterTable(const
AlterTableRequestPB& req,
string normalized_table_name = NormalizeTableName(l.data().name());
*resp->mutable_table_id() = table->id();
- // Modify the table limit.
+ // Set the table limit.
if (table_limit_change) {
if (req.has_disk_size_limit()) {
if (req.disk_size_limit() == TableInfo::TABLE_WRITE_DEFAULT_LIMIT) {
l.mutable_data()->pb.clear_table_disk_size_limit();
+LOG(INFO) << Substitute("Resetting table $0 disk_size_limit to the
default setting",
+normalized_table_name);
} else if (req.disk_size_limit() >= 0) {
l.mutable_data()->pb.set_table_disk_size_limit(req.disk_size_limit());
+LOG(INFO) << Substitute("Setting table $0 disk_size_limit to $1",
+ normalized_table_name, req.disk_size_limit());
} else {
return SetupError(Status::InvalidArgument("disk size limit must "
"be greater than or equal to -1"),
@@ -2960,8 +2970,12 @@ Status CatalogManager::AlterTable(const
AlterTableRequestPB& req,
if (req.has_row_count_limit()) {
if (req.row_count_limit() == TableInfo::TABLE_WRITE_DEFAULT_LIMIT) {
l.mutable_data()->pb.clear_table_row_count_limit();
+LOG(INFO) << Substitute("Resetting table $0 row_count_limit to the
default setting",
+normalized_table_name);
} else if (req.row_count_limit() >= 0) {
l.mutable_data()->pb.set_table_row_count_limit(req.row_count_limit());
+LOG(INFO) << Substitute("Setting table $0 row_count_limit to $1",
+ normalized_table_name, req.row_count_limit());
} else {
return SetupError(Status::InvalidArgument("row count limit must "
"be greater than or equal to -1"),
@@ -3463,8 +3477,16 @@ Status CatalogManager::GetTableStatistics(const
GetTableStatisticsRequestPB* req
}
}
if (FLAGS_enable_table_write_limit) {
-resp->set_disk_size_limit(l.data().pb.table_disk_size_limit());
-resp->set_row_count_limit(l.data().pb.table_row_count_limit());
+if (l.data().pb.has_tabl
[kudu] branch master updated: [txns] add fields for create and update times
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 1a8e8c3 [txns] add fields for create and update times
1a8e8c3 is described below
commit 1a8e8c39a7bb378da6d10e88aae80bf4eeb2a057
Author: Andrew Wong
AuthorDate: Sun May 16 17:16:12 2021 -0700
[txns] add fields for create and update times
This patch adds fields to the transaction status entry for the start
time and the last update time. This is reported in both the `txn list`
and `txn show` tools.
$ ./bin/kudu txn list 0.0.0.0:8764 --included_states="*"
txn_id | user | state |commit_datetime|
start_datetime | last_transition_datetime
+---+---+---+---+---
0 | awong | COMMITTED | Thu, 29 Apr 2021 18:54:56 GMT |
|
1 | awong | COMMITTED | Thu, 29 Apr 2021 22:12:53 GMT |
|
2 | awong | ABORTED | |
|
3 | awong | ABORTED | |
|
4 | awong | COMMITTED | Tue, 18 May 2021 07:41:34 GMT | Tue, 18 May
2021 07:41:34 GMT | Tue, 18 May 2021 07:41:34 GMT
5 | awong | ABORTED | | Tue, 18 May
2021 07:42:22 GMT | Tue, 18 May 2021 07:42:56 GMT
6 | awong | COMMITTED | Tue, 18 May 2021 07:42:33 GMT | Tue, 18 May
2021 07:42:32 GMT | Tue, 18 May 2021 07:42:33 GMT
7 | awong | ABORTED | | Tue, 18 May
2021 07:47:16 GMT | Tue, 18 May 2021 07:47:46 GMT
Change-Id: I4d4f77b4c9b7fcc3f7f749b7f79a18a24be3ce4d
Reviewed-on: http://gerrit.cloudera.org:8080/17475
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
---
src/kudu/tools/kudu-txn-cli-test.cc | 47 +++--
src/kudu/tools/tool_action_txn.cc | 30
src/kudu/transactions/transactions.proto| 8 +
src/kudu/transactions/txn_status_manager.cc | 11 +-
src/kudu/transactions/txn_status_tablet-test.cc | 19 ++
src/kudu/transactions/txn_status_tablet.cc | 6 ++--
src/kudu/transactions/txn_status_tablet.h | 2 +-
7 files changed, 87 insertions(+), 36 deletions(-)
diff --git a/src/kudu/tools/kudu-txn-cli-test.cc
b/src/kudu/tools/kudu-txn-cli-test.cc
index 502565b..e4ea01f 100644
--- a/src/kudu/tools/kudu-txn-cli-test.cc
+++ b/src/kudu/tools/kudu-txn-cli-test.cc
@@ -87,27 +87,31 @@ TEST_F(KuduTxnsCliTest, TestBasicTxnsList) {
string out;
ASSERT_OK(RunKuduTool({ "txn", "list",
cluster_->master_rpc_addrs()[0].ToString(),
"--included_states=*" }, ));
- ASSERT_STR_MATCHES(out, R"( txn_id \| *user *\| state \| *commit_datetime
-\+-*\+---\+-*
- 0 \| *[a-z]* *\| COMMITTED \| .* GMT
- 1 \| *[a-z]* *\| ABORTED \|
- 2 \| *[a-z]* *\| OPEN \| )");
+ ASSERT_STR_MATCHES(out,
+R"(txn_id \| *user *\| *state *\| *commit_datetime *\| *start_datetime *\|
*last_transition_datetime
+\+-*\+---\+-*\+-*\+-*
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT *\| .* GMT *\| .* GMT
+ 1 \| *[a-z]* *\| ABORTED \| *\| .* GMT *\| .* GMT
+ 2 \| *[a-z]* *\| OPEN \| *\| .* GMT *\| .* GMT)");
ASSERT_OK(RunKuduTool({ "txn", "list",
cluster_->master_rpc_addrs()[0].ToString(),
"--included_states=aborted,open" }, ));
- ASSERT_STR_MATCHES(out, R"( txn_id \| *user *\| state \| *commit_datetime
-\+-*\+-\+-*
- 1 \| *[a-z]* *\| ABORTED \|
- 2 \| *[a-z]* *\| OPEN\| )");
+ ASSERT_STR_MATCHES(out,
+R"(txn_id \| *user *\| *state *\| *commit_datetime *\| *start_datetime *\|
*last_transition_datetime
+\+-*\+-\+-*\+-*\+-*
+ 1 \| *[a-z]* *\| ABORTED \| *\| .* GMT *\| .* GMT
+ 2 \| *[a-z]* *\| OPEN\| *\| .* GMT *\| .* GMT)");
ASSERT_OK(RunKuduTool({ "txn", "list",
cluster_->master_rpc_addrs()[0].ToString(),
"--included_states=open,committed" }, ));
- ASSERT_STR_MATCHES(out, R"( txn_id \| *user *\| state \| *commit_datetime
-\+-*\+---\+-*
- 0 \| *[a-z]* *\| COMMITTED \| .* GMT
- 2 \| *[a-z]* *\| OPEN \| )");
+ ASSERT_STR_MATCHES(out,
+R"(txn_id \| *user *\| *state *\| *commit_datetime *\| *start_datetime *\|
*last_transition_datetime
+\+-*\+---\+-*\+-*\+-*
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT *\| .* GMT *\| .* GMT
+ 2 \| *[a-z]* *\| OPEN \| *\| .* GMT
[kudu] branch branch-1.15.x updated (a6de161 -> bfcd94e)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch branch-1.15.x in repository https://gitbox.apache.org/repos/asf/kudu.git. from a6de161 KUDU-2612 automatically flush sessions on txn commit (Java client) new b031fd1 [txns] add a means to get transaction metadata from participant new bfcd94e [txns][tools] show details of a transaction The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../integration-tests/txn_participant-itest.cc | 62 - src/kudu/tablet/tablet_metadata.cc | 30 +- src/kudu/tablet/tablet_metadata.h | 4 + src/kudu/tablet/txn_metadata.h | 28 +- src/kudu/tablet/txn_participant-test.cc| 54 src/kudu/tools/kudu-txn-cli-test.cc| 140 +- src/kudu/tools/tool_action_txn.cc | 308 + src/kudu/transactions/participant_rpc.cc | 23 +- src/kudu/transactions/participant_rpc.h| 11 +- src/kudu/transactions/txn_system_client.cc | 15 +- src/kudu/transactions/txn_system_client.h | 10 +- src/kudu/tserver/tablet_service.cc | 27 ++ src/kudu/tserver/tserver_admin.proto | 7 + 13 files changed, 624 insertions(+), 95 deletions(-)
[kudu] 02/02: [txns][tools] show details of a transaction
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.15.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit bfcd94e7a899a98e2dce15a377186b849fe4ce94
Author: Andrew Wong
AuthorDate: Wed May 12 16:16:21 2021 -0700
[txns][tools] show details of a transaction
This adds a tool to show information related to a single transaction,
including metadata from all of its participants. The goal of this tool
is to be useful in debugging slow commits or inconsistencies with a
transaction.
Currently this outputs info stored in participant's metadata for the
transaction.
$ kudu txn show 0.0.0.0:7051 0
txn_id | user | state |commit_datetime
+---+---+---
0 | awong | COMMITTED | Fri, 14 May 2021 19:15:18 GMT
tablet_id | begin_commit_datetime |
commit_datetime
--+---+---
9204d93c0af14292843b6d432d281fb5 | Fri, 14 May 2021 19:15:18 GMT | Fri, 14
May 2021 19:15:18 GMT
The new tool abides by the --columns flag, as done with the
`kudu txn list` tool, with some additional options for
participant-specific columns.
Change-Id: I9cc5c23b6b46ee75e38aaffe4773881a1ece7294
Reviewed-on: http://gerrit.cloudera.org:8080/17391
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
Reviewed-on: http://gerrit.cloudera.org:8080/17468
Reviewed-by: Bankim Bhavsar
---
src/kudu/tools/kudu-txn-cli-test.cc | 140 +++-
src/kudu/tools/tool_action_txn.cc | 308 +---
2 files changed, 392 insertions(+), 56 deletions(-)
diff --git a/src/kudu/tools/kudu-txn-cli-test.cc
b/src/kudu/tools/kudu-txn-cli-test.cc
index bf03e92..502565b 100644
--- a/src/kudu/tools/kudu-txn-cli-test.cc
+++ b/src/kudu/tools/kudu-txn-cli-test.cc
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+#include
#include
#include
#include
@@ -31,6 +32,7 @@
#include "kudu/util/monotime.h"
#include "kudu/util/net/net_util.h"
#include "kudu/util/test_macros.h"
+#include "kudu/util/test_util.h"
using kudu::itest::TabletServerMap;
using kudu::itest::TServerDetails;
@@ -45,7 +47,10 @@ namespace tools {
class KuduTxnsCliTest : public ExternalMiniClusterITestBase {
void SetUp() override {
NO_FATALS(ExternalMiniClusterITestBase::SetUp());
-NO_FATALS(StartCluster({}, {"--txn_manager_enabled=true"}));
+// Some tests will depend on flushing MRSs quickly, so ensure flushes
+// happen quickly.
+NO_FATALS(StartCluster({ "--flush_threshold_mb=1",
"--flush_threshold_secs=1" },
+{"--txn_manager_enabled=true"}));
}
};
@@ -167,7 +172,138 @@ TEST_F(KuduTxnsCliTest, TestTxnsListHybridTimestamps) {
ASSERT_STR_MATCHES(out,
R"( txn_id \| *user *\| state \| *commit_datetime *|
*commit_hybridtime
\+-*\+---\+---
- 0 \| *[a-z]* *\| COMMITTED \| .* GMT | P: .* usec, L: .*)");
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT \| P: .* usec, L: .*)");
+}
+
+TEST_F(KuduTxnsCliTest, TestBasicShowTxn) {
+ TestWorkload w(cluster_.get());
+ w.set_begin_txn();
+ w.set_commit_txn();
+ w.Setup();
+ w.Start();
+ while (w.rows_inserted() < 10) {
+SleepFor(MonoDelta::FromMilliseconds(10));
+ }
+ w.StopAndJoin();
+
+ // Check the output of the tool with no arguments.
+ string out;
+ ASSERT_OK(RunKuduTool({ "txn", "show",
cluster_->master_rpc_addrs()[0].ToString(), "0" }, ));
+ ASSERT_STR_MATCHES(out, R"( txn_id \| *user *\| state \| *commit_datetime
+\+-*\+---\+-*
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT
+
+tablet_id \| *begin_commit_datetime *\|
*commit_datetime
+--\+-*\+-*
+ [a-z0-9]* \| .*GMT \| .*GMT)");
+
+ // Check the output specifying none of the transaction status columns. We
+ // shouldn't display the status table at all.
+ ASSERT_OK(RunKuduTool({ "txn", "show",
cluster_->master_rpc_addrs()[0].ToString(), "0",
+
"--columns=participant_tablet_id,participant_begin_commit_datetime,"
+"participant_commit_datetime" }, ));
+ ASSERT_STR_NOT_CONTAINS(out, "txn_id");
+ ASSERT_STR_MATCHES(out,
+ R"(tablet_id \| *begin_commit_datetime *\|
*commit_datetime
+--\+-*\+-*
+ [a-z0-9]* \| .*GMT \| .*GMT)");
+
+ // Check the output specifying none of the participant columns. We sho
[kudu] 01/02: [txns] add a means to get transaction metadata from participant
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.15.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit b031fd11783d370663c8bcb220c2da3ab91d99cd
Author: Andrew Wong
AuthorDate: Wed May 12 18:40:59 2021 -0700
[txns] add a means to get transaction metadata from participant
This patch adds a GET_METADATA participant RPC and a method to call it
through the TxnSystemClient. The RPC currently returns the TxnMetadataPB
for the given transaction, giving a bit of insight into the state of the
given participant.
I will use this in an upcoming tool that drills into a transaction and
its participants.
Change-Id: I4b03f13f174bd9a83609fb7ed6106746777b4207
Reviewed-on: http://gerrit.cloudera.org:8080/17446
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
Reviewed-on: http://gerrit.cloudera.org:8080/17467
Reviewed-by: Bankim Bhavsar
Tested-by: Andrew Wong
---
.../integration-tests/txn_participant-itest.cc | 62 +-
src/kudu/tablet/tablet_metadata.cc | 30 +--
src/kudu/tablet/tablet_metadata.h | 4 ++
src/kudu/tablet/txn_metadata.h | 28 --
src/kudu/tablet/txn_participant-test.cc| 54 +++
src/kudu/transactions/participant_rpc.cc | 23 +---
src/kudu/transactions/participant_rpc.h| 11 +++-
src/kudu/transactions/txn_system_client.cc | 15 --
src/kudu/transactions/txn_system_client.h | 10 +++-
src/kudu/tserver/tablet_service.cc | 27 ++
src/kudu/tserver/tserver_admin.proto | 7 +++
11 files changed, 232 insertions(+), 39 deletions(-)
diff --git a/src/kudu/integration-tests/txn_participant-itest.cc
b/src/kudu/integration-tests/txn_participant-itest.cc
index 46f6bc9..a8e7267 100644
--- a/src/kudu/integration-tests/txn_participant-itest.cc
+++ b/src/kudu/integration-tests/txn_participant-itest.cc
@@ -34,10 +34,10 @@
#include "kudu/clock/clock.h"
#include "kudu/common/partial_row.h"
#include "kudu/common/row_operations.h"
+#include "kudu/common/row_operations.pb.h"
#include "kudu/common/timestamp.h"
#include "kudu/common/wire_protocol-test-util.h"
#include "kudu/common/wire_protocol.h"
-#include "kudu/common/wire_protocol.pb.h"
#include "kudu/consensus/raft_consensus.h"
#include "kudu/consensus/time_manager.h"
#include "kudu/gutil/ref_counted.h"
@@ -98,7 +98,7 @@ using kudu::tablet::kInitializing;
using kudu::tablet::kOpen;
using kudu::tablet::MakeParticipantOp;
using kudu::tablet::TabletReplica;
-using kudu::tablet::TxnState;
+using kudu::tablet::TxnMetadataPB;
using kudu::tablet::TxnParticipant;
using kudu::tablet::TxnState;
using kudu::transactions::TxnSystemClient;
@@ -831,6 +831,64 @@ TEST_F(TxnParticipantITest, TestProxyTabletNotFound) {
}
}
+TEST_F(TxnParticipantITest, TestTxnSystemClientGetMetadata) {
+ unique_ptr txn_client;
+ ASSERT_OK(TxnSystemClient::Create(cluster_->master_rpc_addrs(),
_client));
+ constexpr const auto kTxnId = 0;
+ constexpr const int kLeaderIdx = 0;
+ vector replicas = SetUpLeaderGetReplicas(kLeaderIdx);
+ auto* leader_replica = replicas[kLeaderIdx];
+ const auto tablet_id = leader_replica->tablet_id();
+
ASSERT_OK(leader_replica->consensus()->WaitUntilLeaderForTests(kDefaultTimeout));
+
+ // Get commit-related metadata.
+ TxnMetadataPB meta_pb;
+ Status s = txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb);
+ ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::BEGIN_TXN),
kDefaultTimeout));
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb));
+ ASSERT_FALSE(meta_pb.has_aborted());
+ ASSERT_FALSE(meta_pb.has_commit_mvcc_op_timestamp());
+ ASSERT_FALSE(meta_pb.has_commit_timestamp());
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::BEGIN_COMMIT),
kDefaultTimeout));
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb));
+ ASSERT_FALSE(meta_pb.has_aborted());
+ ASSERT_TRUE(meta_pb.has_commit_mvcc_op_timestamp());
+ ASSERT_FALSE(meta_pb.has_commit_timestamp());
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::FI
[kudu] 02/02: [txns][tools] show details of a transaction
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit b2633cac57e171e1ee0d98e1ecc3e41b8a2f8cd2
Author: Andrew Wong
AuthorDate: Wed May 12 16:16:21 2021 -0700
[txns][tools] show details of a transaction
This adds a tool to show information related to a single transaction,
including metadata from all of its participants. The goal of this tool
is to be useful in debugging slow commits or inconsistencies with a
transaction.
Currently this outputs info stored in participant's metadata for the
transaction.
$ kudu txn show 0.0.0.0:7051 0
txn_id | user | state |commit_datetime
+---+---+---
0 | awong | COMMITTED | Fri, 14 May 2021 19:15:18 GMT
tablet_id | begin_commit_datetime |
commit_datetime
--+---+---
9204d93c0af14292843b6d432d281fb5 | Fri, 14 May 2021 19:15:18 GMT | Fri, 14
May 2021 19:15:18 GMT
The new tool abides by the --columns flag, as done with the
`kudu txn list` tool, with some additional options for
participant-specific columns.
Change-Id: I9cc5c23b6b46ee75e38aaffe4773881a1ece7294
Reviewed-on: http://gerrit.cloudera.org:8080/17391
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/tools/kudu-txn-cli-test.cc | 140 +++-
src/kudu/tools/tool_action_txn.cc | 308 +---
2 files changed, 392 insertions(+), 56 deletions(-)
diff --git a/src/kudu/tools/kudu-txn-cli-test.cc
b/src/kudu/tools/kudu-txn-cli-test.cc
index bf03e92..502565b 100644
--- a/src/kudu/tools/kudu-txn-cli-test.cc
+++ b/src/kudu/tools/kudu-txn-cli-test.cc
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+#include
#include
#include
#include
@@ -31,6 +32,7 @@
#include "kudu/util/monotime.h"
#include "kudu/util/net/net_util.h"
#include "kudu/util/test_macros.h"
+#include "kudu/util/test_util.h"
using kudu::itest::TabletServerMap;
using kudu::itest::TServerDetails;
@@ -45,7 +47,10 @@ namespace tools {
class KuduTxnsCliTest : public ExternalMiniClusterITestBase {
void SetUp() override {
NO_FATALS(ExternalMiniClusterITestBase::SetUp());
-NO_FATALS(StartCluster({}, {"--txn_manager_enabled=true"}));
+// Some tests will depend on flushing MRSs quickly, so ensure flushes
+// happen quickly.
+NO_FATALS(StartCluster({ "--flush_threshold_mb=1",
"--flush_threshold_secs=1" },
+{"--txn_manager_enabled=true"}));
}
};
@@ -167,7 +172,138 @@ TEST_F(KuduTxnsCliTest, TestTxnsListHybridTimestamps) {
ASSERT_STR_MATCHES(out,
R"( txn_id \| *user *\| state \| *commit_datetime *|
*commit_hybridtime
\+-*\+---\+---
- 0 \| *[a-z]* *\| COMMITTED \| .* GMT | P: .* usec, L: .*)");
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT \| P: .* usec, L: .*)");
+}
+
+TEST_F(KuduTxnsCliTest, TestBasicShowTxn) {
+ TestWorkload w(cluster_.get());
+ w.set_begin_txn();
+ w.set_commit_txn();
+ w.Setup();
+ w.Start();
+ while (w.rows_inserted() < 10) {
+SleepFor(MonoDelta::FromMilliseconds(10));
+ }
+ w.StopAndJoin();
+
+ // Check the output of the tool with no arguments.
+ string out;
+ ASSERT_OK(RunKuduTool({ "txn", "show",
cluster_->master_rpc_addrs()[0].ToString(), "0" }, ));
+ ASSERT_STR_MATCHES(out, R"( txn_id \| *user *\| state \| *commit_datetime
+\+-*\+---\+-*
+ 0 \| *[a-z]* *\| COMMITTED \| .* GMT
+
+tablet_id \| *begin_commit_datetime *\|
*commit_datetime
+--\+-*\+-*
+ [a-z0-9]* \| .*GMT \| .*GMT)");
+
+ // Check the output specifying none of the transaction status columns. We
+ // shouldn't display the status table at all.
+ ASSERT_OK(RunKuduTool({ "txn", "show",
cluster_->master_rpc_addrs()[0].ToString(), "0",
+
"--columns=participant_tablet_id,participant_begin_commit_datetime,"
+"participant_commit_datetime" }, ));
+ ASSERT_STR_NOT_CONTAINS(out, "txn_id");
+ ASSERT_STR_MATCHES(out,
+ R"(tablet_id \| *begin_commit_datetime *\|
*commit_datetime
+--\+-*\+-*
+ [a-z0-9]* \| .*GMT \| .*GMT)");
+
+ // Check the output specifying none of the participant columns. We shouldn't
+ // display the participant table at all.
+ ASSERT_OK(RunKuduTool({ "txn", "s
[kudu] 01/02: [txns] add a means to get transaction metadata from participant
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e5202c0fc04f97d22ea3eb88625d0bd9d86d07f0
Author: Andrew Wong
AuthorDate: Wed May 12 18:40:59 2021 -0700
[txns] add a means to get transaction metadata from participant
This patch adds a GET_METADATA participant RPC and a method to call it
through the TxnSystemClient. The RPC currently returns the TxnMetadataPB
for the given transaction, giving a bit of insight into the state of the
given participant.
I will use this in an upcoming tool that drills into a transaction and
its participants.
Change-Id: I4b03f13f174bd9a83609fb7ed6106746777b4207
Reviewed-on: http://gerrit.cloudera.org:8080/17446
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
.../integration-tests/txn_participant-itest.cc | 62 +-
src/kudu/tablet/tablet_metadata.cc | 30 +--
src/kudu/tablet/tablet_metadata.h | 4 ++
src/kudu/tablet/txn_metadata.h | 28 --
src/kudu/tablet/txn_participant-test.cc| 54 +++
src/kudu/transactions/participant_rpc.cc | 23 +---
src/kudu/transactions/participant_rpc.h| 11 +++-
src/kudu/transactions/txn_system_client.cc | 15 --
src/kudu/transactions/txn_system_client.h | 10 +++-
src/kudu/tserver/tablet_service.cc | 27 ++
src/kudu/tserver/tserver_admin.proto | 7 +++
11 files changed, 232 insertions(+), 39 deletions(-)
diff --git a/src/kudu/integration-tests/txn_participant-itest.cc
b/src/kudu/integration-tests/txn_participant-itest.cc
index 46f6bc9..a8e7267 100644
--- a/src/kudu/integration-tests/txn_participant-itest.cc
+++ b/src/kudu/integration-tests/txn_participant-itest.cc
@@ -34,10 +34,10 @@
#include "kudu/clock/clock.h"
#include "kudu/common/partial_row.h"
#include "kudu/common/row_operations.h"
+#include "kudu/common/row_operations.pb.h"
#include "kudu/common/timestamp.h"
#include "kudu/common/wire_protocol-test-util.h"
#include "kudu/common/wire_protocol.h"
-#include "kudu/common/wire_protocol.pb.h"
#include "kudu/consensus/raft_consensus.h"
#include "kudu/consensus/time_manager.h"
#include "kudu/gutil/ref_counted.h"
@@ -98,7 +98,7 @@ using kudu::tablet::kInitializing;
using kudu::tablet::kOpen;
using kudu::tablet::MakeParticipantOp;
using kudu::tablet::TabletReplica;
-using kudu::tablet::TxnState;
+using kudu::tablet::TxnMetadataPB;
using kudu::tablet::TxnParticipant;
using kudu::tablet::TxnState;
using kudu::transactions::TxnSystemClient;
@@ -831,6 +831,64 @@ TEST_F(TxnParticipantITest, TestProxyTabletNotFound) {
}
}
+TEST_F(TxnParticipantITest, TestTxnSystemClientGetMetadata) {
+ unique_ptr txn_client;
+ ASSERT_OK(TxnSystemClient::Create(cluster_->master_rpc_addrs(),
_client));
+ constexpr const auto kTxnId = 0;
+ constexpr const int kLeaderIdx = 0;
+ vector replicas = SetUpLeaderGetReplicas(kLeaderIdx);
+ auto* leader_replica = replicas[kLeaderIdx];
+ const auto tablet_id = leader_replica->tablet_id();
+
ASSERT_OK(leader_replica->consensus()->WaitUntilLeaderForTests(kDefaultTimeout));
+
+ // Get commit-related metadata.
+ TxnMetadataPB meta_pb;
+ Status s = txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb);
+ ASSERT_TRUE(s.IsInvalidArgument()) << s.ToString();
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::BEGIN_TXN),
kDefaultTimeout));
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb));
+ ASSERT_FALSE(meta_pb.has_aborted());
+ ASSERT_FALSE(meta_pb.has_commit_mvcc_op_timestamp());
+ ASSERT_FALSE(meta_pb.has_commit_timestamp());
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::BEGIN_COMMIT),
kDefaultTimeout));
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::GET_METADATA),
kDefaultTimeout,
+ /*begin_commit_timestamp*/nullptr, _pb));
+ ASSERT_FALSE(meta_pb.has_aborted());
+ ASSERT_TRUE(meta_pb.has_commit_mvcc_op_timestamp());
+ ASSERT_FALSE(meta_pb.has_commit_timestamp());
+
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakeParticipantOp(kTxnId, ParticipantOpPB::FINALIZE_COMMIT),
kDefaultTimeout));
+ ASSERT_OK(txn_client->ParticipateInTransaction(
+ tablet_id, MakePartici
[kudu] branch master updated (f418ce2 -> b2633ca)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from f418ce2 Update version to 1.16.0-SNAPSHOT new e5202c0 [txns] add a means to get transaction metadata from participant new b2633ca [txns][tools] show details of a transaction The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../integration-tests/txn_participant-itest.cc | 62 - src/kudu/tablet/tablet_metadata.cc | 30 +- src/kudu/tablet/tablet_metadata.h | 4 + src/kudu/tablet/txn_metadata.h | 28 +- src/kudu/tablet/txn_participant-test.cc| 54 src/kudu/tools/kudu-txn-cli-test.cc| 140 +- src/kudu/tools/tool_action_txn.cc | 308 + src/kudu/transactions/participant_rpc.cc | 23 +- src/kudu/transactions/participant_rpc.h| 11 +- src/kudu/transactions/txn_system_client.cc | 15 +- src/kudu/transactions/txn_system_client.h | 10 +- src/kudu/tserver/tablet_service.cc | 27 ++ src/kudu/tserver/tserver_admin.proto | 7 + 13 files changed, 624 insertions(+), 95 deletions(-)
[kudu] branch master updated: KUDU-2612: retry abort task on timeout
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 73310d9 KUDU-2612: retry abort task on timeout
73310d9 is described below
commit 73310d955b4d5d7ec413217d99517cc4b7794975
Author: Andrew Wong
AuthorDate: Wed Apr 28 13:59:58 2021 -0700
KUDU-2612: retry abort task on timeout
It's possible that in the event of a node failure, the abort that
happens automatically to avoid a deadlock will timeout. Instead of
leaving the failure as is, this patch updates the task to be retried.
To expedite testing, this also introduces a new flag to the
TxnSystemClient to reduce the default timeout.
This is a follow-up to d21a0d3.
Change-Id: I303a9a8c85a2191594a22d907770a82da5060f19
Reviewed-on: http://gerrit.cloudera.org:8080/17357
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
---
src/kudu/integration-tests/txn_write_ops-itest.cc | 77 ++-
src/kudu/transactions/txn_system_client.cc| 24 +++
src/kudu/transactions/txn_system_client.h | 12 ++--
src/kudu/tserver/ts_tablet_manager.cc | 14 -
4 files changed, 116 insertions(+), 11 deletions(-)
diff --git a/src/kudu/integration-tests/txn_write_ops-itest.cc
b/src/kudu/integration-tests/txn_write_ops-itest.cc
index 7554b78..b53c796 100644
--- a/src/kudu/integration-tests/txn_write_ops-itest.cc
+++ b/src/kudu/integration-tests/txn_write_ops-itest.cc
@@ -140,8 +140,10 @@ DECLARE_bool(txn_manager_enabled);
DECLARE_bool(txn_manager_lazily_initialized);
DECLARE_int32(txn_participant_begin_op_inject_latency_ms);
DECLARE_int32(txn_participant_registration_inject_latency_ms);
+DECLARE_int64(txn_system_client_op_timeout_ms);
DECLARE_uint32(tablet_max_pending_txn_write_ops);
DECLARE_uint32(txn_manager_status_table_num_replicas);
+DECLARE_uint32(txn_staleness_tracker_interval_ms);
namespace kudu {
@@ -930,10 +932,13 @@ class TxnOpDispatcherITest : public KuduTest {
CHECK_OK(BuildSchema(_));
}
- void Prepare(int num_tservers) {
+ void Prepare(int num_tservers, bool create_table = true, int num_replicas =
0) {
+if (num_replicas == 0) {
+ num_replicas = num_tservers;
+}
FLAGS_txn_manager_enabled = true;
FLAGS_txn_manager_lazily_initialized = false;
-FLAGS_txn_manager_status_table_num_replicas = num_tservers;
+FLAGS_txn_manager_status_table_num_replicas = num_replicas;
InternalMiniClusterOptions opts;
opts.num_tablet_servers = num_tservers;
@@ -943,7 +948,9 @@ class TxnOpDispatcherITest : public KuduTest {
KuduClientBuilder builder;
builder.default_admin_operation_timeout(kTimeout);
ASSERT_OK(cluster_->CreateClient(, _));
-ASSERT_OK(CreateTable(num_tservers));
+if (create_table) {
+ ASSERT_OK(CreateTable(num_replicas));
+}
for (auto i = 0; i < cluster_->num_tablet_servers(); ++i) {
auto* ts = cluster_->mini_tablet_server(i);
ASSERT_OK(ts->WaitStarted());
@@ -1163,6 +1170,70 @@ TEST_F(TxnOpDispatcherITest, LifecycleBasic) {
}
}
+// Test that the automatic abort to avoid deadlock gets retried if the op times
+// out.
+TEST_F(TxnOpDispatcherITest, TestRetryWaitDieAbortsWhenTServerUnavailable) {
+ SKIP_IF_SLOW_NOT_ALLOWED();
+
+ // Disable the staleness tracker so we know any aborts were done by the
+ // wait-die deadlock prevention.
+ FLAGS_txn_staleness_tracker_interval_ms = 0;
+ // Set a low system client timeout to make sure our abort task retries.
+ FLAGS_txn_system_client_op_timeout_ms = 1000;
+
+ NO_FATALS(Prepare(/*num_tservers*/2, /*create_table*/false,
/*num_replicas*/1));
+ // First, figure out which tablet server hosts the TxnStatusManager.
+ tserver::MiniTabletServer* tsm_server = nullptr;
+ ASSERT_EVENTUALLY([&] {
+for (int i = 0; i < cluster_->num_tablet_servers() && tsm_server ==
nullptr; i++) {
+ auto* mts = cluster_->mini_tablet_server(i);
+ auto* tablet_manager = mts->server()->tablet_manager();
+ vector> replicas;
+ tablet_manager->GetTabletReplicas();
+ if (!replicas.empty()) {
+tsm_server = mts;
+ }
+}
+ASSERT_FALSE(tsm_server == nullptr);
+ });
+
+ // Create a single-tablet table so shutting down the TxnStatusManager doesn't
+ // affect writes.
+ unique_ptr table_creator(client_->NewTableCreator());
+ ASSERT_OK(table_creator->table_name(kTableName)
+.schema(_)
+.set_range_partition_columns({ "key" })
+.num_replicas(1)
+.Create());
+ ASSERT_OK(client_->OpenTable(kTableName, _));
+ shared_ptr first_txn;
+ shared_ptr second_txn;
+ ASSERT_OK(client_->NewTransaction(_txn));
+ ASSERT_OK(client_->NewTransaction(_txn));
+
+ int64_t key = 0;
+ ASSERT_OK
[kudu] branch master updated: [client] retry master RPCs on network errors
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 96047a8 [client] retry master RPCs on network errors
96047a8 is described below
commit 96047a8d861d61673b9e2589930eccd78a16e483
Author: Andrew Wong
AuthorDate: Fri Apr 23 14:03:40 2021 -0700
[client] retry master RPCs on network errors
AsyncLeaderMasterRpc and AsyncRandomTxnManagerRpc fail fast when it runs
into a network error. We expect such errors in a couple of scenarios:
- the client is misconfigured and is pointing at the wrong masters
- the masters are down or otherwise unavailable
The current handling makes it convenient in the former case, since users
can immediately adjust their application or tooling with the correct
masters list. However, this seems to be a far less expected scenario
than the latter. So, this patch updates the behavior to retry on such
errors, as we would an RPC timeout. The end result is that
leader-master-bound requests should be more resilient to master
failures.
The primary motivation for this is to make TxnManager calls more robust
to master failures -- a later patch will leverage this behavior.
Change-Id: Iae2febb9e890acf6f7efd5cce3cb7e4f7b5f683d
Reviewed-on: http://gerrit.cloudera.org:8080/17343
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/client/client-test.cc | 73 ++
src/kudu/client/master_proxy_rpc.cc| 14 ++---
src/kudu/client/transaction-internal.cc| 12 ++--
src/kudu/client/txn_manager_proxy_rpc.cc | 17 ++---
src/kudu/integration-tests/master_hms-itest.cc | 6 +-
src/kudu/integration-tests/txn_commit-itest.cc | 2 +-
.../integration-tests/txn_status_table-itest.cc| 45 -
7 files changed, 97 insertions(+), 72 deletions(-)
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index 76bf022..7ab3e85 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -1010,7 +1010,7 @@ TEST_F(ClientTest, TestMasterDown) {
shared_ptr t;
client_->data_->default_admin_operation_timeout_ = MonoDelta::FromSeconds(1);
Status s = client_->OpenTable("other-tablet", );
- ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
+ ASSERT_TRUE(s.IsTimedOut()) << s.ToString();
}
// Test that we get errors when we try incorrectly configuring the scanner, and
@@ -3564,19 +3564,13 @@ void
ClientTest::DoTestWriteWithDeadServer(WhichServerToKill which) {
ASSERT_TRUE(s.IsIOError()) << s.ToString();
unique_ptr error = GetSingleErrorFromSession(session.get());
- switch (which) {
-case DEAD_MASTER:
- // Only one master, so no retry for finding the new leader master.
- ASSERT_TRUE(error->status().IsNetworkError());
- break;
-case DEAD_TSERVER:
- ASSERT_TRUE(error->status().IsTimedOut());
- // TODO(KUDU-1466) Re-enable this assertion once the jira gets solved.
We can't actually
- // make an assertion on the reason for the timeout since sometimes
tablet server connection
- // errors get reported as GetTabletLocations timeouts.
- // ASSERT_STR_CONTAINS(error->status().ToString(), "Connection refused");
- break;
- }
+ ASSERT_TRUE(error->status().IsTimedOut());
+ // TODO(KUDU-1466) Re-enable this assertion once the jira gets solved. We
can't actually
+ // make an assertion on the reason for the timeout since sometimes tablet
server connection
+ // errors get reported as GetTabletLocations timeouts.
+ // if (which == DEAD_TSERVER) {
+ // ASSERT_STR_CONTAINS(error->status().ToString(), "Connection refused");
+ // }
ASSERT_EQ(error->failed_op().ToString(),
R"(INSERT int32 key=1, int32 int_val=1, string string_val="x")");
@@ -7456,6 +7450,10 @@ TEST_F(ClientTest, AttemptToControlTxnByOtherUser) {
TEST_F(ClientTest, NoTxnManager) {
shared_ptr txn;
+ KuduClientBuilder builder;
+ builder.default_admin_operation_timeout(MonoDelta::FromSeconds(1));
+ builder.default_rpc_timeout(MonoDelta::FromMilliseconds(100));
+ ASSERT_OK(cluster_->CreateClient(, _));
ASSERT_OK(client_->NewTransaction());
// Shutdown all masters: a TxnManager is a part of master, so after shutting
@@ -7465,8 +7463,7 @@ TEST_F(ClientTest, NoTxnManager) {
{
shared_ptr txn;
auto s = client_->NewTransaction();
-ASSERT_TRUE(s.IsNetworkError()) << s.ToString();
-ASSERT_STR_CONTAINS(s.ToString(), "Client connection negotiation failed");
+ASSERT_TRUE(s.IsTimedOut()) << s.ToString();
}
const vector> txn_ctl_results = {
@@ -7476,8 +7473,7 @@ TEST_F(ClientTest,
[kudu] branch master updated: KUDU-2612: rollback txn on TXN_LOCKED_ABORT
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new d21a0d3 KUDU-2612: rollback txn on TXN_LOCKED_ABORT
d21a0d3 is described below
commit d21a0d38373b84f66fa2d37732b7ec7b8fee5f16
Author: Andrew Wong
AuthorDate: Thu Apr 1 17:51:21 2021 -0700
KUDU-2612: rollback txn on TXN_LOCKED_ABORT
Previously it was required that clients manually rollback transactions
when their ops failed with TXN_LOCKED_ABORTED (seen as an Aborted
error). Rather than pushing the burden onto application users, this
patch attempts to alleviate this by hooking in a callback to rollback
transactions automatically if any write op fails with TXN_LOCKED_ABORT.
Of course, users are still free to Rollback() on error -- this patch is
simply meant to automate it in case the user doesn't, giving us a bit
more control over ensuring our transactions don't lead to a deadlock.
Change-Id: I25415cad0cfb08d260e23bd8b368852a5006c1fb
Reviewed-on: http://gerrit.cloudera.org:8080/17263
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/integration-tests/txn_write_ops-itest.cc | 39 ++-
src/kudu/tserver/tablet_service.cc| 32 ---
src/kudu/tserver/ts_tablet_manager.cc | 21
src/kudu/tserver/ts_tablet_manager.h | 12 ++-
4 files changed, 84 insertions(+), 20 deletions(-)
diff --git a/src/kudu/integration-tests/txn_write_ops-itest.cc
b/src/kudu/integration-tests/txn_write_ops-itest.cc
index c597acb..2bab38d 100644
--- a/src/kudu/integration-tests/txn_write_ops-itest.cc
+++ b/src/kudu/integration-tests/txn_write_ops-itest.cc
@@ -50,8 +50,8 @@
#include "kudu/client/write_op.h"
#include "kudu/common/partial_row.h"
#include "kudu/common/row_operations.h"
+#include "kudu/common/row_operations.pb.h"
#include "kudu/common/schema.h"
-#include "kudu/common/wire_protocol.pb.h"
#include "kudu/consensus/consensus.pb.h"
#include "kudu/consensus/consensus.proxy.h"
#include "kudu/gutil/map-util.h"
@@ -297,9 +297,7 @@ class TxnWriteOpsITest : public
ExternalMiniClusterITestBase {
// Test that our deadlock prevention mechanisms work by writing across
// different tablets concurrently from multiple transactions.
-// TODO(awong): it'd be much more convenient to take control of aborting the
-// transactions ourselves, rather than relying on the application user.
-TEST_F(TxnWriteOpsITest, TestClientSideDeadlockPrevention) {
+TEST_F(TxnWriteOpsITest, TestDeadlockPrevention) {
constexpr const int kNumTxns = 8;
const vector master_flags = {
"--txn_manager_enabled=true",
@@ -338,8 +336,8 @@ TEST_F(TxnWriteOpsITest, TestClientSideDeadlockPrevention) {
Status s = session->Apply(insert.release());
LOG(INFO) << Substitute("Txn $0 wrote row $1: $2",
token.txn_id(), row_key, s.ToString());
- // If the write op failed because of a locking error, roll the
- // transaction back and retry the transaction after waiting a bit.
+ // If the write op failed because of a locking error, retry the
+ // transaction after waiting a bit.
if (!s.ok()) {
vector errors;
ElementDeleter d();
@@ -354,7 +352,6 @@ TEST_F(TxnWriteOpsITest, TestClientSideDeadlockPrevention) {
// in wait-die, that get retried) will still time out, after
// contending a bit with other ops.
ASSERT_TRUE(error.IsAborted() || error.IsTimedOut()) <<
error.ToString();
-ASSERT_OK(txn->Rollback());
needs_retry = true;
// Wait a bit before retrying the entire transaction to allow for
@@ -1063,6 +1060,8 @@ TEST_F(TxnOpDispatcherITest, LifecycleBasic) {
}
}
+// Test that when attempting to lock a transaction that is locked by an earlier
+// transaction, we abort the newer transaction.
TEST_F(TxnOpDispatcherITest, BeginTxnLockAbort) {
NO_FATALS(Prepare(1));
@@ -1118,21 +1117,31 @@ TEST_F(TxnOpDispatcherITest, BeginTxnLockAbort) {
ASSERT_TRUE(row_status.IsAborted()) << row_status.ToString();
ASSERT_STR_CONTAINS(row_status.ToString(), "should be aborted");
-// We should have an extra dispatcher for the new transactional write.
-ASSERT_EQ(1 + kNumPartitions, GetTxnOpDispatchersTotalCount());
+// The dispatcher for the new transactional write should eventually
+// disappear because the transaction is automatically aborted.
+ASSERT_EVENTUALLY([&] {
+ ASSERT_EQ(kNumPartitions, GetTxnOpDispatchersTotalCount());
+});
}
{
// Now, commit the first t
[kudu] branch master updated: [ksck] KUDU-3258: allow ksck and rebalancer to work on txn status table
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new bdb6d06 [ksck] KUDU-3258: allow ksck and rebalancer to work on txn status table bdb6d06 is described below commit bdb6d0643ec7b68f11cefc038dc256824ce96f5e Author: Andrew Wong AuthorDate: Fri Apr 9 18:22:29 2021 -0700 [ksck] KUDU-3258: allow ksck and rebalancer to work on txn status table This patch adds the transaction system table to the ksck output in its own section for system tables. Here's a sample snippet of an output that has the system table: Summary by system table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---++--+---+-++--+- kudu_system.kudu_transactions | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1| 0 Summary by table Name | RF | Status| Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---++-+---+-++--+- default.loadgen_auto_05cf5be513ea4a84a052e8044f641c1a | 1 | UNAVAILABLE | 8 | 6 | 0 | 0| 2 default.loadgen_auto_0c7ea48d5f6948408694b176f70e69ec | 1 | UNAVAILABLE | 8 | 5 | 0 | 0| 3 default.loadgen_auto_241be343981c46d081ab2b3d2e3b6e6a | 1 | UNAVAILABLE | 8 | 5 | 0 | 0| 3 default.loadgen_auto_385476d5d3b6493f8cbf659c8a4cf7cc | 1 | UNAVAILABLE | 8 | 6 | 0 | 0| 2 default.loadgen_auto_430e280e8aa7450591da67ae15ff0f37 | 1 | UNAVAILABLE | 8 | 6 | 0 | 0| 2 The section can be included/excluded via the --sections flag of ksck. Since ksck and the rebalancer use the same cluster-examining code, this patch also updates the rebalancer cluster_status class to account for system tables -- the tool would have crashed upon trying to find the replication factor of the system table otherwise. Change-Id: I8162f6eb046d98791c6bdeb5c15a0af72487300d Reviewed-on: http://gerrit.cloudera.org:8080/17315 Tested-by: Andrew Wong Reviewed-by: Alexey Serbin --- .../integration-tests/txn_status_table-itest.cc| 52 src/kudu/rebalance/cluster_status.h| 1 + src/kudu/tools/ksck-test.cc| 133 - src/kudu/tools/ksck.cc | 20 +++- src/kudu/tools/ksck.h | 17 +-- src/kudu/tools/ksck_remote.cc | 45 +-- src/kudu/tools/ksck_results.cc | 33 +++-- src/kudu/tools/ksck_results.h | 9 +- src/kudu/tools/rebalancer_tool.cc | 28 +++-- src/kudu/tools/tool.proto | 1 + src/kudu/tools/tool_action_cluster.cc | 2 +- 11 files changed, 247 insertions(+), 94 deletions(-) diff --git a/src/kudu/integration-tests/txn_status_table-itest.cc b/src/kudu/integration-tests/txn_status_table-itest.cc index d018beb..a766dda 100644 --- a/src/kudu/integration-tests/txn_status_table-itest.cc +++ b/src/kudu/integration-tests/txn_status_table-itest.cc @@ -40,6 +40,7 @@ #include "kudu/gutil/port.h" #include "kudu/gutil/ref_counted.h" #include "kudu/gutil/stl_util.h" +#include "kudu/gutil/strings/join.h" #include "kudu/gutil/strings/substitute.h" #include "kudu/integration-tests/cluster_itest_util.h" #include "kudu/integration-tests/test_workload.h" @@ -51,6 +52,7 @@ #include "kudu/tablet/metadata.pb.h" #include "kudu/tablet/tablet_metadata.h" #include "kudu/tablet/tablet_replica.h" +#include "kudu/tools/tool_test_util.h" #include "kudu/transactions/transactions.pb.h" #include "kudu/transactions/txn_status_tablet.h" #include "kudu/transactions/txn_system_client.h" @@ -58,6 +60,7 @@ #include "kudu/tserver/tablet_server.h" #include "kudu/tserver/ts_tablet_manager.h" #include "kudu/util/monotime.h" +#include "kudu/util/net/net_util.h" #include "kudu/util/scoped_cleanup.h" #include "kudu/util/status.h" #include "kudu/util/test_macros.h" @@ -85,6 +88,7 @@ using kudu::cluster::InternalMiniClusterOptions; using kudu::itest::TServerDetails; usin
[kudu] 01/02: KUDU-2612: acquire and release partition lock
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 8093f95e6b8eb496043562f69df3a66e078ad731
Author: hahao
AuthorDate: Sun Mar 7 23:59:59 2021 -0800
KUDU-2612: acquire and release partition lock
This patch plumbs partition locks into write ops for transactional and
non-transactional operations. Upon attempting to write, we try to
acquire the partition lock in the WriteOp prepare phase, and upon
successfully applying the op, we transfer ownership of the lock to the
Txn that the write was a part of (or just release the partition lock if
the write was non-transactional). Txns release the lock when
FINALIZE_COMMIT or ABORT_TXN is applied.
We take the partition lock for non-transactional write ops as well to
ensure we don’t duplicate keys (e.g. if a transaction inserts key=1 and
then a non-transactional write inserts key=1 before the transaction
commits). If the partition lock cannot be acquired, the write op is
aborted or retried, based on the wait-die mechanics already in the lock
manager.
A flag is also introduced to disable this locking for tests that
currently expect support for concurrent transactions.
Change-Id: If26733cae16810f3b3afd1fd05dcb984e6366939
Reviewed-on: http://gerrit.cloudera.org:8080/17159
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/client/batcher.cc | 12 +-
src/kudu/integration-tests/fuzz-itest.cc | 10 +
src/kudu/integration-tests/txn_commit-itest.cc | 13 ++
.../integration-tests/txn_participant-itest.cc | 62 ++
src/kudu/integration-tests/txn_write_ops-itest.cc | 216 +
src/kudu/tablet/lock_manager.h | 5 +
src/kudu/tablet/ops/participant_op.cc | 10 +
src/kudu/tablet/ops/write_op.cc| 62 +-
src/kudu/tablet/ops/write_op.h | 20 +-
src/kudu/tablet/tablet.cc | 10 +-
src/kudu/tablet/tablet.h | 14 +-
src/kudu/tablet/tablet_bootstrap.cc| 15 +-
src/kudu/tablet/txn_participant-test.cc| 126 +++-
src/kudu/tablet/txn_participant.cc | 24 ++-
src/kudu/tablet/txn_participant.h | 15 ++
src/kudu/transactions/participant_rpc.cc | 2 +
16 files changed, 600 insertions(+), 16 deletions(-)
diff --git a/src/kudu/client/batcher.cc b/src/kudu/client/batcher.cc
index dc26aed..f48a5c8 100644
--- a/src/kudu/client/batcher.cc
+++ b/src/kudu/client/batcher.cc
@@ -470,7 +470,9 @@ RetriableRpcStatus WriteRpc::AnalyzeResponse(const Status&
rpc_cb_status) {
}
}
- if (result.status.IsServiceUnavailable()) {
+ if (result.status.IsServiceUnavailable() ||
+ (resp_.has_error() &&
+ resp_.error().code() ==
tserver::TabletServerErrorPB::TXN_LOCKED_RETRY_OP)) {
result.result = RetriableRpcStatus::SERVICE_UNAVAILABLE;
return result;
}
@@ -507,7 +509,8 @@ RetriableRpcStatus WriteRpc::AnalyzeResponse(const Status&
rpc_cb_status) {
}
if (resp_.has_error() &&
- resp_.error().code() == tserver::TabletServerErrorPB::TXN_ILLEGAL_STATE)
{
+ (resp_.error().code() == tserver::TabletServerErrorPB::TXN_ILLEGAL_STATE
||
+ resp_.error().code() ==
tserver::TabletServerErrorPB::TXN_LOCKED_ABORT)) {
result.result = RetriableRpcStatus::NON_RETRIABLE_ERROR;
return result;
}
@@ -526,8 +529,9 @@ RetriableRpcStatus WriteRpc::AnalyzeResponse(const Status&
rpc_cb_status) {
//becomes a real issue when handling responses to write
//operations in the context of multi-row transactions.
//For example, Status::IllegalState() originated from
-//TabletServerErrorPB::TXN_ILLEGAL_STATE responses are
-//needlessly retried.
+//TabletServerErrorPB::TXN_ILLEGAL_STATE response and
+//Status::Abort() originated from
TabletServerErrorPB::TXN_LOCKED_ABORT
+//response are needlessly retried.
result.result = RetriableRpcStatus::REPLICA_NOT_LEADER;
return result;
}
diff --git a/src/kudu/integration-tests/fuzz-itest.cc
b/src/kudu/integration-tests/fuzz-itest.cc
index 9a5b0c1..579c660 100644
--- a/src/kudu/integration-tests/fuzz-itest.cc
+++ b/src/kudu/integration-tests/fuzz-itest.cc
@@ -78,6 +78,7 @@
DEFINE_int32(keyspace_size, 5, "number of distinct primary keys to test
with");
DEFINE_int32(max_open_txns, 5, "maximum number of open transactions to test
with");
+DECLARE_bool(enable_txn_partition_lock);
DECLARE_bool(enable_maintenance_manager);
DECLARE_bool(scanner_allow_snapshot_scans_with_logical_timestamps);
DE
[kudu] branch master updated (7fa9beb -> 2dda869)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 7fa9beb KUDU-2612: send next txn keepalive sooner in case of timeout new 8093f95 KUDU-2612: acquire and release partition lock new 2dda869 KUDU-2671: Adds new field to PartitionSchema. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/kudu/client/Operation.java | 2 +- .../java/org/apache/kudu/client/TestOperation.java | 2 +- src/kudu/client/batcher.cc | 12 +- src/kudu/client/scan_token-internal.cc | 7 +- src/kudu/common/CMakeLists.txt | 13 +- src/kudu/common/common.proto | 14 ++ src/kudu/common/partition-test.cc | 193 +- src/kudu/common/partition.cc | 106 +- src/kudu/common/partition.h| 22 ++- src/kudu/common/partition_pruner.cc| 5 +- src/kudu/common/row_operations-test.cc | 2 +- src/kudu/common/row_operations.h | 2 +- src/kudu/common/row_operations.proto | 83 src/kudu/common/wire_protocol.proto| 57 -- src/kudu/integration-tests/alter_table-test.cc | 2 +- src/kudu/integration-tests/fuzz-itest.cc | 10 + src/kudu/integration-tests/txn_commit-itest.cc | 13 ++ .../integration-tests/txn_participant-itest.cc | 62 ++ src/kudu/integration-tests/txn_write_ops-itest.cc | 216 + src/kudu/master/catalog_manager.cc | 16 +- src/kudu/master/master.proto | 6 +- src/kudu/master/sys_catalog.cc | 2 +- src/kudu/tablet/lock_manager.h | 5 + src/kudu/tablet/ops/participant_op.cc | 10 + src/kudu/tablet/ops/write_op.cc| 62 +- src/kudu/tablet/ops/write_op.h | 20 +- src/kudu/tablet/tablet.cc | 10 +- src/kudu/tablet/tablet.h | 14 +- src/kudu/tablet/tablet_bootstrap.cc| 15 +- src/kudu/tablet/tablet_metadata.cc | 2 +- src/kudu/tablet/txn_participant-test.cc| 126 +++- src/kudu/tablet/txn_participant.cc | 24 ++- src/kudu/tablet/txn_participant.h | 15 ++ src/kudu/transactions/participant_rpc.cc | 2 + src/kudu/tserver/tablet_copy_client.cc | 1 - src/kudu/tserver/tablet_server-test.cc | 1 + src/kudu/tserver/tablet_service.cc | 4 +- src/kudu/tserver/tserver.proto | 1 + 38 files changed, 1053 insertions(+), 106 deletions(-) create mode 100644 src/kudu/common/row_operations.proto
[kudu] 02/02: KUDU-2671: Adds new field to PartitionSchema.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 2dda869456659a36247eb89f5b9e5e3837e5f8a3
Author: Mahesh Reddy
AuthorDate: Mon Feb 1 14:53:30 2021 -0800
KUDU-2671: Adds new field to PartitionSchema.
This patch introduces a new field to PartitionSchema that combines range
bounds
and their respective hash bucket schemas. Any instance that assumes
the same hash schemas are used for each range will need this new field.
Some of the more important instances include partition pruning and many
of the internal PartitionSchema functions.
I moved RowOperationsPB to a separate .proto file due to some circular
dependency issues between common.proto and wire_protocol.proto. Most of
the proto changes in this patch revolve around this change.
Change-Id: Ic5d8615ab9967fdb40292b9c77eb68a19baeca1d
Reviewed-on: http://gerrit.cloudera.org:8080/17025
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
.../java/org/apache/kudu/client/Operation.java | 2 +-
.../java/org/apache/kudu/client/TestOperation.java | 2 +-
src/kudu/client/scan_token-internal.cc | 7 +-
src/kudu/common/CMakeLists.txt | 13 +-
src/kudu/common/common.proto | 14 ++
src/kudu/common/partition-test.cc | 193 -
src/kudu/common/partition.cc | 106 ++-
src/kudu/common/partition.h| 22 ++-
src/kudu/common/partition_pruner.cc| 5 +-
src/kudu/common/row_operations-test.cc | 2 +-
src/kudu/common/row_operations.h | 2 +-
src/kudu/common/row_operations.proto | 83 +
src/kudu/common/wire_protocol.proto| 57 --
src/kudu/integration-tests/alter_table-test.cc | 2 +-
src/kudu/master/catalog_manager.cc | 16 +-
src/kudu/master/master.proto | 6 +-
src/kudu/master/sys_catalog.cc | 2 +-
src/kudu/tablet/tablet_metadata.cc | 2 +-
src/kudu/tserver/tablet_copy_client.cc | 1 -
src/kudu/tserver/tablet_server-test.cc | 1 +
src/kudu/tserver/tablet_service.cc | 4 +-
src/kudu/tserver/tserver.proto | 1 +
22 files changed, 453 insertions(+), 90 deletions(-)
diff --git
a/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
b/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
index 4981d5e..4d871c9 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
@@ -34,10 +34,10 @@ import org.apache.yetus.audience.InterfaceAudience;
import org.apache.yetus.audience.InterfaceStability;
import org.apache.kudu.ColumnSchema;
+import org.apache.kudu.RowOperations.RowOperationsPB;
import org.apache.kudu.Schema;
import org.apache.kudu.Type;
import org.apache.kudu.WireProtocol.AppStatusPB.ErrorCode;
-import org.apache.kudu.WireProtocol.RowOperationsPB;
import org.apache.kudu.client.ProtobufHelper.SchemaPBConversionFlags;
import org.apache.kudu.client.Statistics.Statistic;
import org.apache.kudu.client.Statistics.TabletStatistics;
diff --git
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestOperation.java
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestOperation.java
index 20e7640..42c747c 100644
--- a/java/kudu-client/src/test/java/org/apache/kudu/client/TestOperation.java
+++ b/java/kudu-client/src/test/java/org/apache/kudu/client/TestOperation.java
@@ -29,9 +29,9 @@ import org.junit.Test;
import org.mockito.Mockito;
import org.apache.kudu.ColumnSchema;
+import org.apache.kudu.RowOperations.RowOperationsPB;
import org.apache.kudu.Schema;
import org.apache.kudu.Type;
-import org.apache.kudu.WireProtocol.RowOperationsPB;
import org.apache.kudu.client.Operation.ChangeType;
import org.apache.kudu.test.junit.RetryRule;
import org.apache.kudu.tserver.Tserver.WriteRequestPBOrBuilder;
diff --git a/src/kudu/client/scan_token-internal.cc
b/src/kudu/client/scan_token-internal.cc
index 132e772..00afa2b 100644
--- a/src/kudu/client/scan_token-internal.cc
+++ b/src/kudu/client/scan_token-internal.cc
@@ -40,7 +40,6 @@
#include "kudu/client/shared_ptr.h" // IWYU pragma: keep
#include "kudu/client/tablet-internal.h"
#include "kudu/client/tablet_server-internal.h"
-#include "kudu/common/column_predicate.h"
#include "kudu/common/common.pb.h"
#include "kudu/common/encoded_key.h"
#include "kudu/common/partition.h"
@@ -69,6 +68,7 @@ using strings::Substitute;
namespace kudu {
+class ColumnPredicate;
using master::GetTableLocationsResponsePB;
using master:
[kudu] branch master updated: [hms] allow hive.metastore.disallow.incompatible.col.type.changes to be true
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new ea370f1 [hms] allow
hive.metastore.disallow.incompatible.col.type.changes to be true
ea370f1 is described below
commit ea370f12d6c3f9fe8de9d3e7e7e8fda1d3cad7ca
Author: Andrew Wong
AuthorDate: Fri Apr 9 12:20:18 2021 -0700
[hms] allow hive.metastore.disallow.incompatible.col.type.changes to be true
A major distribution of Hive and Kudu (CDP) has changed its default
behavior to set hive.metastore.disallow.incompatible.col.type.changes to
true. This is a safer behavior for Hive, but breaks the Kudu-HMS
integration, since we require the configuration for DROP COLUMN
operations, and check the Hive configuration as such at startup.
HIVE-24987 tracks the work on the Hive-side to permit Kudu operations to
alter tables in a seemingly type-incompatible way that is actually safe
for Kudu.
Until then, this adds an option to disable this Kudu-side check (kept
the current behavior by default, but added a gflag to make it more
easily configurable).
Change-Id: Idc9777e134e243abfe768d631592541720e101a8
Reviewed-on: http://gerrit.cloudera.org:8080/17296
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/hms/hms_client.cc | 36 +---
1 file changed, 25 insertions(+), 11 deletions(-)
diff --git a/src/kudu/hms/hms_client.cc b/src/kudu/hms/hms_client.cc
index 440dde4..63499d3 100644
--- a/src/kudu/hms/hms_client.cc
+++ b/src/kudu/hms/hms_client.cc
@@ -25,6 +25,7 @@
#include
#include
+#include
#include
#include
#include
@@ -34,6 +35,7 @@
#include
#include
+#include "kudu/gutil/macros.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/strings/split.h"
#include "kudu/gutil/strings/strip.h"
@@ -42,10 +44,20 @@
#include "kudu/hms/hive_metastore_types.h"
#include "kudu/thrift/client.h"
#include "kudu/thrift/sasl_client_transport.h"
+#include "kudu/util/flag_tags.h"
#include "kudu/util/status.h"
#include "kudu/util/stopwatch.h"
#include "kudu/util/string_case.h"
+DEFINE_bool(disable_hms_incompatible_column_type_check, false,
+"Whether or not to disable the check for the "
+"'hive.metastore.disallow.incompatible.col.type.changes' "
+"configuration in the Hive Metastore. This check is important to "
+"ensure Hive will allow dropping columns, but may not be necessary
"
+"in some versions of Hive. See HIVE-24987 for more details.");
+TAG_FLAG(disable_hms_incompatible_column_type_check, advanced);
+TAG_FLAG(disable_hms_incompatible_column_type_check, hidden);
+
using apache::thrift::TApplicationException;
using apache::thrift::TException;
using apache::thrift::protocol::TJSONProtocol;
@@ -190,17 +202,19 @@ Status HmsClient::Start() {
// operations properly.
//
// See
org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange.
- string disallow_incompatible_column_type_changes;
-
HMS_RET_NOT_OK(client_.get_config_value(disallow_incompatible_column_type_changes,
- kDisallowIncompatibleColTypeChanges,
- "false"),
- Substitute("failed to get Hive Metastore $0 configuration",
-kDisallowIncompatibleColTypeChanges));
-
- if (iequals(disallow_incompatible_column_type_changes, "true")) {
-return Status::IllegalState(Substitute(
-"Hive Metastore configuration is invalid: $0 must be set to false",
-kDisallowIncompatibleColTypeChanges));
+ if (!FLAGS_disable_hms_incompatible_column_type_check) {
+string disallow_incompatible_column_type_changes;
+
HMS_RET_NOT_OK(client_.get_config_value(disallow_incompatible_column_type_changes,
+
kDisallowIncompatibleColTypeChanges,
+"false"),
+ Substitute("failed to get Hive Metastore $0 configuration",
+ kDisallowIncompatibleColTypeChanges));
+
+if (iequals(disallow_incompatible_column_type_changes, "true")) {
+ return Status::IllegalState(Substitute(
+ "Hive Metastore configuration is invalid: $0 must be set to false",
+ kDisallowIncompatibleColTypeChanges));
+}
}
// Check that the HMS is configured to add the entire thrift Table/Partition
[kudu] branch master updated: KUDU-3268: fix race between MM scheduling and unregistration
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 6aeb466 KUDU-3268: fix race between MM scheduling and unregistration
6aeb466 is described below
commit 6aeb466c538d8cc566031c4a82200f18f088e18a
Author: Andrew Wong
AuthorDate: Thu Apr 8 14:39:54 2021 -0700
KUDU-3268: fix race between MM scheduling and unregistration
Following commit 9e4664d the following race was possible (S: scheduler
thread,
T: another thread):
S: finds best op is op1
T: unregisters op1 because the tablet is being shut down
T: takes running_instances_lock_
- waits for running count to go to 0
- destructs op1
S: takes running_instances_lock_
- increments running count to 1
S: attempts to run op1 but segfaults
This resulted in a crash like the following, when shutting down the tserver
(though it seems possible this could manifest itself when deleting a
TabletReplica):
PC: @ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp()
*** SIGSEGV (@0x30) received by PID 21067 (TID 0x7ff96343b700) from PID 48;
stack trace: ***
@ 0x7ff976846980 (unknown) at ??:0
@ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7ff97596b538
_ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7ff97596f124
_ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data
at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff975a05e6e kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff975a06757 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv
at ??:0
@ 0x7ff975a07e7b
_ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data
at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff9759f8913 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff97683b6db start_thread at ??:0
@ 0x7ff97388e71f clone at ??:0
This patch fixes this by checking whether the op has been cancelled, while
holding running_instanes_lock_, before proceeding with the scheduling. I
added
a new test that failed consistently within 100 runs, that passed 5000 times
locally.
Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Reviewed-on: http://gerrit.cloudera.org:8080/17291
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/util/maintenance_manager-test.cc | 18 --
src/kudu/util/maintenance_manager.cc | 9 +
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/src/kudu/util/maintenance_manager-test.cc
b/src/kudu/util/maintenance_manager-test.cc
index 0fc5d80..de78f96 100644
--- a/src/kudu/util/maintenance_manager-test.cc
+++ b/src/kudu/util/maintenance_manager-test.cc
@@ -389,8 +389,8 @@ TEST_F(MaintenanceManagerTest,
TestNewOpsDontGetScheduledDuringUnregister) {
// Wait until two instances of the ops start running, since we have two MM
threads.
ASSERT_EVENTUALLY([&]() {
- ASSERT_EQ(op1.RunningGauge()->value(), 2);
-});
+ASSERT_EQ(op1.RunningGauge()->value(), 2);
+ });
// Trigger Unregister while they are running. This should wait for the
currently-
// running operations to complete, but no new operations should be scheduled.
@@ -885,4 +885,18 @@ TEST_F(MaintenanceManagerTest,
ManyOperationsHeavyUpdateStats) {
ops.front()->update_stats_count());
}
+// Regression test for KUDU-3268, where the unregistering and destruction of an
+// op may race with the scheduling of that op, resulting in a segfault.
+TEST_F(MaintenanceManagerTest, TestUnregisterWhileScheduling) {
+ TestMaintenanceOp op1("1", MaintenanceOp::HIGH_IO_USAGE);
+ op1.set_perf_improvement(10);
+ // Set a bunch of runs so we continually schedule the op.
+ op1.set_remaining_runs(100);
+ manager_->RegisterOp();
+ ASSERT_EVENTUALLY([&]() {
+ASSERT_GE(op1.DurationHistogram()->TotalCount(), 1);
+ });
+ op1.Unregister();
+}
+
} // namespace kudu
diff --git a/src/kudu/util/maintenance_manager.cc
b/src/kudu/util/maintenance_manager.cc
index 64a361a..deeaf2f 100644
--- a/src/kudu/util/maintenance_manager.cc
+++ b/src/kudu/util/maintenance_manager.cc
@@ -348,7 +348,16 @@ void MaintenanceManager::RunSchedulerThread() {
op_note = std::move(best_op_and_why.second);
}
if (op) {
+// While 'running_instances_lock_' is held, check one more time for
+// whether the op is cancelled. This ensures that we don't attempt to
+// launch an op that has been destructed in UnregisterOp(). See
+// KUDU-3268 for
[kudu] branch master updated: [txns] fix race between tasks and destructor
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new c2e5373 [txns] fix race between tasks and destructor
c2e5373 is described below
commit c2e5373493afe43766b0d5b6f04dfbcd340d633d
Author: Andrew Wong
AuthorDate: Wed Apr 7 16:13:50 2021 -0700
[txns] fix race between tasks and destructor
It previously possible for a CommitTask to be destructed before
completing the loop of scheduling all asynchronous tasks. This led to a
race as seen below:
WARNING: ThreadSanitizer: data race (pid=32435)
Write of size 8 at 0x7b1c000ce2d8 by thread T105 (mutexes: write
M424881254664896540):
#0 std::__1::__vector_base, std::__1::allocator >,
std::__1::allocator,
std::__1::allocator > > >::__destruct_at_end(std::__1::basic_string, std::__1::allocator >*)
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:427:12
(txn_commit-itest+0x576cb1)
#1 std::__1::__vector_base, std::__1::allocator >,
std::__1::allocator,
std::__1::allocator > > >::clear()
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:369:29
(txn_commit-itest+0x5770d1)
#2 std::__1::__vector_base, std::__1::allocator >,
std::__1::allocator,
std::__1::allocator > > >::~__vector_base()
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:463:9
(txn_commit-itest+0x59caf9)
#3 std::__1::vector, std::__1::allocator >,
std::__1::allocator,
std::__1::allocator > > >::~vector()
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/vector:555:5
(libtransactions.so+0x8c2a0)
#4 kudu::transactions::CommitTasks::~CommitTasks()
../src/kudu/transactions/txn_status_manager.h:177:26
(libtransactions.so+0xcce8b)
#5 kudu::RefCountedThreadSafe
>::DeleteInternal(kudu::transactions::CommitTasks const*)
../src/kudu/gutil/ref_counted.h:153:44 (libtransactions.so+0xcce1a)
#6
kudu::DefaultRefCountedThreadSafeTraits::Destruct(kudu::transactions::CommitTasks
const*) ../src/kudu/gutil/ref_counted.h:116:5 (libtransactions.so+0xccdc8)
#7 kudu::RefCountedThreadSafe
>::Release() const ../src/kudu/gutil/ref_counted.h:144:7
(libtransactions.so+0xccd70)
#8 scoped_refptr::~scoped_refptr()
../src/kudu/gutil/ref_counted.h:266:13 (libtransactions.so+0xbf785)
#9 std::__1::pair >::~pair()
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/utility:315:29
(libtransactions.so+0xc7652)
#10 void
std::__1::allocator_traits >, void*> >
>::__destroy >
>(std::__1::integral_constant,
std::__1::allocator >, void*> >&,
std::__1::pair >, void*> >
>::destroy >
>(std::__1::allocator >, void*> >&,
std::__1::pair >*)
/data/3/aw [...]
#12
std::__1::__hash_node_destructor >, void*> >
>::operator()(std::__1::__hash_node >, void*>*)
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/__hash_table:844:13
(libtransactions.so+0xc740d)
#13
std::__1::unique_ptr >, void*>,
std::__1::__hash_node_destructor >, void*> > >
>::reset(std::__1::__hash_node >, void*>*)
/data/3/awong/Repositories/kudu/thirdparty/installed [...]
#14
std::__1::unique_ptr >, void*>,
std::__1::__hash_node_destructor >, void*> > > >::~unique_ptr()
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2547:19
(libtransactions.so+0xc6cbc)
#15 std::__1::__hash_table >,
std::__1::__unordered_map_hasher >, std::__1::hash, true>,
std::__1::__unordered_map_equal >, std::__1::equal_to,
true>, std::__1::allocator, std::__1::hash,
std::__1::equal_to, std::__1::allocator > >
>::erase(std::__1::__hash_map_iterator >, void*>*> >)
/data/3/awong/Repositories/kudu/thirdparty/installed/tsan/include/c++/v1/u [...]
#17 kudu::transactions::TxnStatusManager::RemoveCommitTask(long,
kudu::transactions::CommitTasks const*)
../src/kudu/transactions/txn_status_manager.h:433:26
(libtransactions.so+0xbefc6)
#18 kudu::transactions::CommitTasks::IsShuttingDownCleanupIfLastOp()
../src/kudu/transactions/txn_status_manager.cc:181:28
(libtransactions.so+0x97dea)
#19
kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2::operator()(kudu::Status
const&) const ../src/kudu/transactions/txn_status_manager.cc:319:9
(libtransactions.so+0xaefd6)
#20
decltype(std::__1::forward(fp)(std::__1::forward(fp0)))
std::__1::__invoke(kudu::transactions::CommitTasks::AbortTxnAsyncTask(int)::$_2&,
kudu::Status const&)
/data/3/awong/Repositories/kudu/thirdparty/inst
[kudu] 01/02: KUDU-2612: allow ABORT_TXN txn ops to succeed if txn hasn't started
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 6be9794f91a2eebb291e4db2daf63a0f12c3ae2a
Author: Andrew Wong
AuthorDate: Tue Mar 23 23:42:36 2021 -0700
KUDU-2612: allow ABORT_TXN txn ops to succeed if txn hasn't started
This patch allows ABORT_TXN ops to proceed even if the participant
hasn't yet initialized the transaction. This is desirable in the case
where a participant has successfully registered with the
TxnStatusManager, but hasn't yet replicated its BEGIN_TXN op. In these
cases, the ops will unregister any pending TxnOpDispatchers, as well as
mark the existence of the transaction on the participant, preventing
further participant registrations for the transaction.
Take the following as an example:
1. a transactional write op is sent to a participant
2. the participant successfully registers with the TxnStatusManager
3. before the BEGIN_TXN op replicates, the node crashes
4. no further retry of the write is sent
5. the user attempt to commit the transaction
6. the BEGIN_COMMIT op fails because the transaction hasn't started on
the participant
7. an ABORT_TXN op is scheduled but that fails too because the the
transaction hasn't started on the participant
With this patch, step 7 should succeed, and successfully replicate the
ABORT_TXN op, even though the transaction doesn't exist. The test[1] for
this isn't enabled yet -- some additional plumbing is required to ensure
we return the correct error codes.
Along the way, I hardened up an edge case for BEGIN_COMMIT so
TXN_ILLEGAL_STATE is returned if the transaction doesn't exist, instead
of returning a plain IllegalState, which would have resulted in the
ParticipantRpc being retried.
[1] see TxnOpDispatcherITest.CommitWithWriteOpPendingParticipantRegistered
Change-Id: Ia4c864b4f14e42008d3aa8f4454c8b2abf9bb766
Reviewed-on: http://gerrit.cloudera.org:8080/17233
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
.../integration-tests/txn_participant-itest.cc | 44 ++
src/kudu/tablet/tablet.cc | 3 +-
src/kudu/tablet/tablet_metadata.cc | 5 +-
src/kudu/tablet/txn_metadata.h | 2 +-
src/kudu/tablet/txn_participant-test.cc| 49 ---
src/kudu/tablet/txn_participant.h | 70 +-
6 files changed, 119 insertions(+), 54 deletions(-)
diff --git a/src/kudu/integration-tests/txn_participant-itest.cc
b/src/kudu/integration-tests/txn_participant-itest.cc
index 6a7e21e..241b5e9 100644
--- a/src/kudu/integration-tests/txn_participant-itest.cc
+++ b/src/kudu/integration-tests/txn_participant-itest.cc
@@ -592,6 +592,27 @@ TEST_F(TxnParticipantITest, TestBeginCommitAfterFinalize) {
}
}
+TEST_F(TxnParticipantITest, TestProxyErrorWhenNotBegun) {
+ constexpr const int kLeaderIdx = 0;
+ auto txn_id = 0;
+ vector replicas = SetUpLeaderGetReplicas(kLeaderIdx);
+
ASSERT_OK(replicas[kLeaderIdx]->consensus()->WaitUntilLeaderForTests(kDefaultTimeout));
+ auto admin_proxy = cluster_->tserver_admin_proxy(kLeaderIdx);
+ const auto tablet_id = replicas[kLeaderIdx]->tablet_id();
+ for (auto type : { ParticipantOpPB::BEGIN_COMMIT,
+ ParticipantOpPB::FINALIZE_COMMIT }) {
+TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+Status s = ParticipateInTransactionCheckResp(
+admin_proxy.get(), tablet_id, txn_id++, type, );
+ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
+ASSERT_EQ(TabletServerErrorPB::TXN_ILLEGAL_STATE, code);
+ }
+ TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ ASSERT_OK(ParticipateInTransactionCheckResp(
+ admin_proxy.get(), tablet_id, txn_id++, ParticipantOpPB::ABORT_TXN,
));
+ ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+}
+
TEST_F(TxnParticipantITest, TestProxyIllegalStatesInCommitSequence) {
constexpr const int kLeaderIdx = 0;
constexpr const int kTxnId = 0;
@@ -932,6 +953,29 @@ TEST_F(TxnParticipantITest,
TestTxnSystemClientAbortSequence) {
{ { kTxnOne, kAborted, -1 }, { kTxnTwo, kAborted, -1 } }));
}
+TEST_F(TxnParticipantITest, TestTxnSystemClientErrorWhenNotBegun) {
+ constexpr const int kLeaderIdx = 0;
+ int txn_id = 0;
+ vector replicas = SetUpLeaderGetReplicas(kLeaderIdx);
+ auto* leader_replica = replicas[kLeaderIdx];
+ const auto tablet_id = leader_replica->tablet_id();
+
ASSERT_OK(leader_replica->consensus()->WaitUntilLeaderForTests(kDefaultTimeout));
+ unique_ptr txn_client;
+ ASSERT_OK(TxnSystemClient::Create(cluster_->master_rpc_addrs(),
_client));
+
+ for (auto type : { ParticipantOpPB::BEGIN_COMMIT,
+ ParticipantOpPB::FINALIZE_COMMIT }
[kudu] 02/02: KUDU-2612: route txn op dispatching errors to write ops
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 78fb2047da74a1ebd051dbfa29c4d188056b47bc
Author: Andrew Wong
AuthorDate: Wed Mar 24 00:57:14 2021 -0700
KUDU-2612: route txn op dispatching errors to write ops
This patch routes bad statuses from the TxnOpDispatcher to the write ops
that initiated the registration, and adjusts the batcher code to handle
such errors.
This also enables a couple of test cases that were written with this
behavior in mind; and adjusts some expected errors, addressing some
TODOs that expected this behavior.
A follow-up patch will make a similar change to the Java client.
Change-Id: Ibf85e0724ee579cb20dac642b82e3228faf90935
Reviewed-on: http://gerrit.cloudera.org:8080/17217
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/client/batcher.cc| 6 +++
src/kudu/integration-tests/txn_commit-itest.cc| 2 +-
src/kudu/integration-tests/txn_write_ops-itest.cc | 60 ++-
src/kudu/tablet/tablet_metadata.cc| 6 +--
src/kudu/tablet/tablet_replica.cc | 44 +
src/kudu/tablet/tablet_replica.h | 16 --
src/kudu/transactions/txn_status_manager.cc | 24 +++--
src/kudu/tserver/tablet_service.cc| 5 +-
src/kudu/tserver/ts_tablet_manager.cc | 21
src/kudu/tserver/ts_tablet_manager.h | 6 +--
10 files changed, 94 insertions(+), 96 deletions(-)
diff --git a/src/kudu/client/batcher.cc b/src/kudu/client/batcher.cc
index 93036b2..dc26aed 100644
--- a/src/kudu/client/batcher.cc
+++ b/src/kudu/client/batcher.cc
@@ -506,6 +506,12 @@ RetriableRpcStatus WriteRpc::AnalyzeResponse(const Status&
rpc_cb_status) {
return result;
}
+ if (resp_.has_error() &&
+ resp_.error().code() == tserver::TabletServerErrorPB::TXN_ILLEGAL_STATE)
{
+result.result = RetriableRpcStatus::NON_RETRIABLE_ERROR;
+return result;
+ }
+
// Alternatively, when we get a status code of IllegalState or Aborted, we
// assume this means that the replica we attempted to write to is not the
// current leader (maybe it got partitioned or slow and another node took
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index f125f54..4ddc4c0 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -679,7 +679,7 @@ TEST_F(TxnCommitITest, TestCommitAfterParticipantAbort) {
Status completion_status;
bool is_complete;
ASSERT_OK(txn->IsCommitComplete(_complete, _status));
- ASSERT_TRUE(completion_status.IsIncomplete()) <<
completion_status.ToString();
+ ASSERT_TRUE(completion_status.IsAborted()) << completion_status.ToString();
}
// Try concurrently beginning to commit a bunch of different transactions.
diff --git a/src/kudu/integration-tests/txn_write_ops-itest.cc
b/src/kudu/integration-tests/txn_write_ops-itest.cc
index cb7a8f0..2a37aa4 100644
--- a/src/kudu/integration-tests/txn_write_ops-itest.cc
+++ b/src/kudu/integration-tests/txn_write_ops-itest.cc
@@ -659,12 +659,14 @@ TEST_F(TxnWriteOpsITest, WriteOpForNonExistentTxn) {
// Try to write an extra row in the context of a transaction which has already
// been committed.
-//
-// TODO(aserbin): due to conversion of Status::IllegalState() into
-//RetriableRpcStatus::REPLICA_NOT_LEADER result code,
-//these sub-scenarios fail with Status::TimedOut() because
-//they retry in vain until RPC timeout elapses
-TEST_F(TxnWriteOpsITest, DISABLED_TxnWriteAfterCommit) {
+TEST_F(TxnWriteOpsITest, TxnWriteAfterCommit) {
+ const vector master_flags = {
+// Enable TxnManager in Kudu masters.
+// TODO(aserbin): remove this customization once the flag is 'on' by
default
+"--txn_manager_enabled=true",
+ };
+ NO_FATALS(StartCluster({}, master_flags, kNumTabletServers));
+ NO_FATALS(Prepare());
int idx = 0;
{
shared_ptr txn;
@@ -685,10 +687,10 @@ TEST_F(TxnWriteOpsITest, DISABLED_TxnWriteAfterCommit) {
ASSERT_TRUE(s.IsIOError()) << s.ToString();
ASSERT_STR_CONTAINS(s.ToString(), "Some errors occurred");
const auto err_status = GetSingleRowError(session.get());
- ASSERT_TRUE(err_status.IsInvalidArgument()) << err_status.ToString();
+ ASSERT_TRUE(err_status.IsIllegalState()) << err_status.ToString();
ASSERT_STR_CONTAINS(err_status.ToString(),
"Failed to write batch of 1 ops to tablet");
- ASSERT_STR_MATCHES(err_status.ToString(), "txn .* is not open");
+ ASSERT_STR_MATCHES(err_status.ToString(), "transaction .* not open");
[kudu] branch master updated (ed3d916 -> 78fb204)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from ed3d916 [master] Check for 'last_known_addr' field when adding master new 6be9794 KUDU-2612: allow ABORT_TXN txn ops to succeed if txn hasn't started new 78fb204 KUDU-2612: route txn op dispatching errors to write ops The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/client/batcher.cc | 6 ++ src/kudu/integration-tests/txn_commit-itest.cc | 2 +- .../integration-tests/txn_participant-itest.cc | 44 ++ src/kudu/integration-tests/txn_write_ops-itest.cc | 60 --- src/kudu/tablet/tablet.cc | 3 +- src/kudu/tablet/tablet_metadata.cc | 11 ++-- src/kudu/tablet/tablet_replica.cc | 44 +++--- src/kudu/tablet/tablet_replica.h | 16 +++-- src/kudu/tablet/txn_metadata.h | 2 +- src/kudu/tablet/txn_participant-test.cc| 49 --- src/kudu/tablet/txn_participant.h | 70 +- src/kudu/transactions/txn_status_manager.cc| 24 +++- src/kudu/tserver/tablet_service.cc | 5 +- src/kudu/tserver/ts_tablet_manager.cc | 21 --- src/kudu/tserver/ts_tablet_manager.h | 6 +- 15 files changed, 213 insertions(+), 150 deletions(-)
[kudu] branch master updated: KUDU-2612: add ScopedPartitionLock
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 6983d25 KUDU-2612: add ScopedPartitionLock
6983d25 is described below
commit 6983d25c35df8569c1851bf27e2c75e98036c10d
Author: hahao
AuthorDate: Sat Feb 20 13:03:58 2021 -0800
KUDU-2612: add ScopedPartitionLock
This patch introduces a coarse-grained partition-level lock
ScopedPartitionLock to prevent dirty writes for multi-row transactions,
similar to the ScopedRowLock, but for locking the entire LockManager
instead of individual rows.
A partition lock can only be held by a single transaction at a time. A
given transaction can acquire the lock multiple times. To prevent
deadlocks, a wait-die scheme is used -- if a transaction requires a lock
held by another transaction:
1. Retry the op if the requesting transaction has a lower txn ID than
the current holder ("wait"),
2. Otherwise abort the requesting transaction immediately ("die").
A later patch will plumb this locking into participant ops and
transactional write ops.
Change-Id: I158115739ce3e7cfb77bbcb854e834336c1256b1
Reviewed-on: http://gerrit.cloudera.org:8080/17097
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/tablet/lock_manager-test.cc | 233 ++-
src/kudu/tablet/lock_manager.cc | 179 ++-
src/kudu/tablet/lock_manager.h | 137 ++--
src/kudu/tserver/tserver.proto | 7 ++
4 files changed, 540 insertions(+), 16 deletions(-)
diff --git a/src/kudu/tablet/lock_manager-test.cc
b/src/kudu/tablet/lock_manager-test.cc
index d2d9005..7ff727a 100644
--- a/src/kudu/tablet/lock_manager-test.cc
+++ b/src/kudu/tablet/lock_manager-test.cc
@@ -19,6 +19,7 @@
#include
#include
+#include
#include
#include
#include
@@ -30,14 +31,20 @@
#include
#include
+#include "kudu/common/txn_id.h"
#include "kudu/gutil/macros.h"
#include "kudu/gutil/stringprintf.h"
+#include "kudu/tserver/tserver.pb.h"
#include "kudu/util/array_view.h"
+#include "kudu/util/countdown_latch.h"
#include "kudu/util/env.h"
+#include "kudu/util/monotime.h"
+#include "kudu/util/scoped_cleanup.h"
#include "kudu/util/slice.h"
#include "kudu/util/stopwatch.h"
#include "kudu/util/test_util.h"
+using kudu::tserver::TabletServerErrorPB;
using std::shared_ptr;
using std::string;
using std::thread;
@@ -53,7 +60,7 @@ class LockEntry;
class OpState;
static const OpState* kFakeTransaction =
- reinterpret_cast(0xdeadbeef);
+reinterpret_cast(0xdeadbeef);
class LockManagerTest : public KuduTest {
public:
@@ -116,7 +123,7 @@ TEST_F(LockManagerTest, TestRelockSameRow) {
VerifyAlreadyLocked(key_a[0]);
}
-TEST_F(LockManagerTest, TestMoveLock) {
+TEST_F(LockManagerTest, TestMoveRowLock) {
// Acquire a lock.
Slice key_a[] = {"a"};
ScopedRowLock row_lock(_manager_, kFakeTransaction, key_a,
LockManager::LOCK_EXCLUSIVE);
@@ -128,6 +135,228 @@ TEST_F(LockManagerTest, TestMoveLock) {
ASSERT_FALSE(row_lock.acquired()); // NOLINT(bugprone-use-after-move)
}
+TEST_F(LockManagerTest, TestLockUnlockPartitionSingleTxn) {
+ TxnId id(0);
+ TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ {
+ScopedPartitionLock l(_manager_, id);
+ASSERT_TRUE(l.IsAcquired());
+ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ASSERT_EQ(1, lock_manager_.partition_lock_refs());
+ }
+
+ // The same transaction should be able to acquire the partition lock multiple
+ // times.
+ ScopedPartitionLock first_lock(_manager_, id);
+ ASSERT_TRUE(first_lock.IsAcquired());
+ ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ {
+ScopedPartitionLock second_lock(_manager_, id);
+ASSERT_TRUE(second_lock.IsAcquired());
+ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ASSERT_EQ(2, lock_manager_.partition_lock_refs());
+ }
+
+ // 'partition_lock_refs' should decrease automatically when a
+ // ScopedPartitionLock goes out of scope.
+ ASSERT_EQ(1, lock_manager_.partition_lock_refs());
+ ASSERT_TRUE(first_lock.IsAcquired());
+ ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+}
+
+TEST_F(LockManagerTest, TestLockUnlockPartitionAbort) {
+ // Acquiring a lock that is held by another transaction with a lower txn ID
+ // will get a TXN_LOCKED_ABORT server error code.
+ TxnId txn1(1);
+ ScopedPartitionLock first_lock(_manager_, txn1);
+ TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ ASSERT_TRUE(first_lock.IsAcquired());
+ ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ ASSERT_EQ(1, lock_manager_.pa
[kudu] branch master updated: KUDU-2612: allow aborting after beginning to commit
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 058ec9a KUDU-2612: allow aborting after beginning to commit
058ec9a is described below
commit 058ec9a2baa9edb75904e8aa83716dd2734f3bbd
Author: Andrew Wong
AuthorDate: Tue Mar 2 17:07:39 2021 -0800
KUDU-2612: allow aborting after beginning to commit
This patch adjusts the TxnStatusManager state machine to include a new
FINALIZE_IN_PROGRESS state to serve as an intermediate step between
COMMIT_IN_PROGRESS and COMMITTED.
The goal is to allow for aborts to occur in the event that we've begun
committing, but not yet called FINALIZE_COMMIT on any of the
transaction's participants, which may be desirable in cases where
anything surprising has happened on the participants (e.g. if they're
deleted).
Change-Id: If1b6596df2db5601f7e17e528ad6dc68057b67f8
Reviewed-on: http://gerrit.cloudera.org:8080/17022
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
.../org/apache/kudu/client/KuduTransaction.java| 1 +
src/kudu/client/transaction-internal.cc| 1 +
src/kudu/integration-tests/txn_commit-itest.cc | 219 ---
src/kudu/master/txn_manager-test.cc| 2 +-
src/kudu/tablet/txn_participant-test.cc| 4 +-
src/kudu/transactions/transactions.proto | 33 +++
src/kudu/transactions/txn_status_entry.cc | 6 +
src/kudu/transactions/txn_status_entry.h | 6 +
src/kudu/transactions/txn_status_manager-test.cc | 160 ++-
src/kudu/transactions/txn_status_manager.cc| 305 +
src/kudu/transactions/txn_status_manager.h | 110 +++-
11 files changed, 681 insertions(+), 166 deletions(-)
diff --git
a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
index 19286f2..05a5616 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
@@ -317,6 +317,7 @@ public class KuduTransaction implements AutoCloseable {
throw new NonRecoverableException(Status.IllegalState("transaction is
still open"));
case COMMITTED:
return true;
+ case FINALIZE_IN_PROGRESS:
case COMMIT_IN_PROGRESS:
return false;
default:
diff --git a/src/kudu/client/transaction-internal.cc
b/src/kudu/client/transaction-internal.cc
index ada717a..1436e24 100644
--- a/src/kudu/client/transaction-internal.cc
+++ b/src/kudu/client/transaction-internal.cc
@@ -314,6 +314,7 @@ Status KuduTransaction::Data::IsCommitCompleteImpl(
*is_complete = true;
*completion_status = Status::Aborted("transaction has been aborted");
break;
+case TxnStatePB::FINALIZE_IN_PROGRESS:
case TxnStatePB::COMMIT_IN_PROGRESS:
*is_complete = false;
*completion_status = Status::Incomplete("commit is still in progress");
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index 4f2ea2b..f125f54 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -17,9 +17,11 @@
#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
@@ -40,8 +42,10 @@
#include "kudu/common/partial_row.h"
#include "kudu/common/txn_id.h"
#include "kudu/common/wire_protocol-test-util.h"
+#include "kudu/gutil/basictypes.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/ref_counted.h"
+#include "kudu/gutil/strings/substitute.h"
#include "kudu/integration-tests/test_workload.h"
#include "kudu/master/mini_master.h"
#include "kudu/mini-cluster/internal_mini_cluster.h"
@@ -94,6 +98,7 @@ using std::unique_ptr;
using std::unordered_map;
using std::unordered_set;
using std::vector;
+using strings::Substitute;
namespace kudu {
namespace itest {
@@ -190,18 +195,25 @@ class TxnCommitITest : public KuduTest {
return Status::OK();
}
- // Insert 'num_rows' rows to the given session, starting with 'start_row'.
+ // Insert 'num_rows' rows to the given session, starting with 'start_row', to
+ // every table in 'table_names' or 'table_name_' if not set.
Status InsertToSession(
- const shared_ptr& txn_session, int start_row, int num_rows)
{
-shared_ptr table;
-RETURN_NOT_OK(client_->OpenTable(table_name_, ));
-const int target_row_id = start_row + num_rows;
-for (int i = start_row; i < target_row_id; i++) {
- auto* insert
[kudu] branch master updated: [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new a0db990 [backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
a0db990 is described below
commit a0db990e08173293e42a7490322f08681abaa5d3
Author: Andrew Wong
AuthorDate: Sat Mar 20 21:04:43 2021 -0700
[backup] set spark.sql.legacy.parquet.int96RebaseModeInWrite
After the bump to Spark 3.1.1, TestKuduBackup.testRandomBackupAndRestore
started failing with errors like the following:
02:04:37.919 [ERROR - Executor task launch worker for task 0.0 in stage 0.0
(TID 0)] (Logging.scala:94) Aborting task
org.apache.spark.SparkUpgradeException: You may get a different result due
to the upgrading of Spark 3.0: writing dates before 1582-10-15 or timestamps
before 1900-01-01T00:00:00Z into Parquet INT96 files can be dangerous, as the
files may be read by Spark 2.x or legacy versions of Hive later, which uses a
legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian
calendar. See more details in SPARK-31404. You can set
spark.sql.legacy.parquet.int96RebaseModeInWrite [...]
at
org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInWrite(DataSourceUtils.scala:165)
~[spark-sql_2.12-3.1.1.jar:3.1.1]
...
Per their instructions, this sets the int96RebaseModeInWrite option.
Change-Id: Ib9ca4d9e69785dd9d056fa8e62c944d56cf219ed
Reviewed-on: http://gerrit.cloudera.org:8080/17213
Reviewed-by: Grant Henke
Tested-by: Andrew Wong
---
java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala | 1 +
1 file changed, 1 insertion(+)
diff --git
a/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
b/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
index c02f5de..13dcc5f 100644
--- a/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
+++ b/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
@@ -86,6 +86,7 @@ object KuduBackup {
// 1900-01-01T00:00:00Z in Parquet. Otherwise incorrect values may be read
by
// Spark 2 or legacy version of Hive. See more details in SPARK-31404.
session.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite",
"LEGACY")
+session.conf.set("spark.sql.legacy.parquet.int96RebaseModeInWrite",
"LEGACY")
// Write the data to the backup path.
// The backup path contains the timestampMs and should not already exist.
[kudu] branch master updated: [tool] KUDU-3226 Validate List of Masters kudu ksck
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new fae223e [tool] KUDU-3226 Validate List of Masters kudu ksck
fae223e is described below
commit fae223ecb139e0f54161279ee8970098067629c9
Author: Abhishek Chennaka
AuthorDate: Sun Feb 28 19:40:45 2021 -0800
[tool] KUDU-3226 Validate List of Masters kudu ksck
This patch checks for duplicate kudu masters provided in all of the
kudu command line tools which call ParseMasterAddresses(). Currently
ksck and rebalance (as rebalance calls ksck internally) are the only
commands which throw the reported stack trace as ksck constructs a
map internally for the kudu masters and bombs once it sees duplicate
UUIDs. Other commands do not report any issue currently with duplicate
masters.
The patch only does a string comparison check to detect and report all
the duplicate master(/s). If a user provides the IP address and hostname
of the same master, it won't be able to catch the duplicates. That
seems more of a deliberate attempt rather than a common mistake a user
could make and hence not considering that case.
Also wrote a simple test case to test this functionality. Decided not
to spin up a test cluster for this as this is more of client side check
and a master is not actually contacted.
This change got conflicted with another test i.e.
AdminCliTest.TestAuthzResetCacheIncorrectMasterAddressList. Updated this
test as well to by creating a new external mini cluster with three
masters and running the command in the test with just one master
Change-Id: I2f3b2b7dcf2ac78cb95cf43242651e3ce8fddf6f
Reviewed-on: http://gerrit.cloudera.org:8080/17141
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/tools/kudu-admin-test.cc| 20 +++-
src/kudu/tools/kudu-tool-test.cc | 14 ++
src/kudu/tools/tool_action_common.cc | 19 ++-
3 files changed, 43 insertions(+), 10 deletions(-)
diff --git a/src/kudu/tools/kudu-admin-test.cc
b/src/kudu/tools/kudu-admin-test.cc
index bf30864..208c64b 100644
--- a/src/kudu/tools/kudu-admin-test.cc
+++ b/src/kudu/tools/kudu-admin-test.cc
@@ -53,7 +53,6 @@
#include "kudu/gutil/basictypes.h"
#include "kudu/gutil/map-util.h"
#include "kudu/gutil/stl_util.h"
-#include "kudu/gutil/strings/join.h"
#include "kudu/gutil/strings/split.h"
#include "kudu/gutil/strings/strip.h"
#include "kudu/gutil/strings/substitute.h"
@@ -97,6 +96,8 @@ using kudu::client::KuduTableCreator;
using kudu::client::KuduValue;
using kudu::client::sp::shared_ptr;
using kudu::cluster::ExternalTabletServer;
+using kudu::cluster::ExternalMiniCluster;
+using kudu::cluster::ExternalMiniClusterOptions;
using kudu::consensus::COMMITTED_OPID;
using kudu::consensus::ConsensusStatePB;
using kudu::consensus::EXCLUDE_HEALTH_REPORT;
@@ -2236,11 +2237,12 @@ TEST_F(AdminCliTest, TestDumpMemTrackers) {
}
TEST_F(AdminCliTest, TestAuthzResetCacheIncorrectMasterAddressList) {
- NO_FATALS(BuildAndStart());
-
- const auto& master_addr =
cluster_->master()->bound_rpc_hostport().ToString();
- const vector dup_master_addresses = { master_addr, master_addr, };
- const auto& dup_master_addresses_str = JoinStrings(dup_master_addresses,
",");
+ ExternalMiniClusterOptions opts;
+ opts.num_masters = 3;
+ cluster_.reset(new ExternalMiniCluster(std::move(opts)));
+ ASSERT_OK(cluster_->Start());
+ auto master_addrs_str =
HostPort::ToCommaSeparatedString(cluster_->master_rpc_addrs());
+ auto incorrect_master_addrs_str =
cluster_->master(0)->bound_rpc_hostport().ToString();
string out;
string err;
Status s;
@@ -2249,14 +2251,14 @@ TEST_F(AdminCliTest,
TestAuthzResetCacheIncorrectMasterAddressList) {
"master",
"authz_cache",
"refresh",
-dup_master_addresses_str,
+incorrect_master_addrs_str,
}, , );
ASSERT_TRUE(s.IsRuntimeError()) << ToolRunInfo(s, out, err);
const auto ref_err_msg = Substitute(
"Invalid argument: list of master addresses provided ($0) "
"does not match the actual cluster configuration ($1)",
- dup_master_addresses_str, master_addr);
+ incorrect_master_addrs_str, master_addrs_str);
ASSERT_STR_CONTAINS(err, ref_err_msg);
// However, the '--force' option makes it possible to run the tool even
@@ -2269,7 +2271,7 @@ TEST_F(AdminCliTest,
TestAuthzResetCacheIncorrectMasterAddressList) {
"authz_cache",
"refresh",
"--force",
-dup_master_addresses_str,
+incorrect_master_addrs_str,
},
[kudu] branch master updated: KUDU-2671: No-op ClientServerMapping if only one schema exists.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 8ebe032 KUDU-2671: No-op ClientServerMapping if only one schema
exists.
8ebe032 is described below
commit 8ebe032847f618d76459b59c84b150c997016ee9
Author: Mahesh Reddy
AuthorDate: Tue Mar 2 00:45:12 2021 -0800
KUDU-2671: No-op ClientServerMapping if only one schema exists.
For the new field of PartitionSchema, a client schema is necessary
for RowOperationsPBDecoder to populate the new field. Not all call
sites for PartitionSchema::FromPB() have access to an explicit client
schema.
This patch allows the client schema to have column IDs if it's equal
to the tablet schema and no-ops the ClientServerMapping in this case.
If we don't have access to a client schema, we don't need a mapping
since we only have one schema.
Change-Id: I836f157201509a9b38a3ad42d653236f240dda5e
Reviewed-on: http://gerrit.cloudera.org:8080/17161
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/common/row_operations-test.cc | 19 +++
src/kudu/common/row_operations.cc | 32 ++--
src/kudu/common/row_operations.h | 6 ++
3 files changed, 43 insertions(+), 14 deletions(-)
diff --git a/src/kudu/common/row_operations-test.cc
b/src/kudu/common/row_operations-test.cc
index 34dccb7..d56b77b 100644
--- a/src/kudu/common/row_operations-test.cc
+++ b/src/kudu/common/row_operations-test.cc
@@ -33,8 +33,8 @@
#include "kudu/common/partial_row.h"
#include "kudu/common/row.h"
#include "kudu/common/schema.h"
-#include "kudu/common/types.h"
#include "kudu/common/wire_protocol.pb.h"
+#include "kudu/common/types.h"
#include "kudu/gutil/basictypes.h"
#include "kudu/gutil/dynamic_annotations.h"
#include "kudu/gutil/macros.h"
@@ -757,7 +757,10 @@ TEST_F(RowOperationsTest, TestProjectDeletes) {
}
}
-TEST_F(RowOperationsTest, SplitKeyRoundTrip) {
+namespace {
+// If 'use_tablet_schema_as_client_schema' is true, the tablet schema with its
+// column IDs is passed in as the client_schema for the RowOperationsPBDecoder.
+void TestSplitKeys(bool use_tablet_schema_as_client_schema) {
Schema client_schema = Schema({ ColumnSchema("int8", INT8),
ColumnSchema("int16", INT16),
ColumnSchema("int32", INT32),
@@ -787,7 +790,8 @@ TEST_F(RowOperationsTest, SplitKeyRoundTrip) {
RowOperationsPBEncoder().Add(RowOperationsPB::SPLIT_ROW, row);
Schema schema = client_schema.CopyWithColumnIds();
- RowOperationsPBDecoder decoder(, _schema, , nullptr);
+ RowOperationsPBDecoder decoder(
+ , use_tablet_schema_as_client_schema ? : _schema,
, nullptr);
vector ops;
ASSERT_OK(decoder.DecodeOperations());
ASSERT_EQ(1, ops.size());
@@ -821,7 +825,6 @@ TEST_F(RowOperationsTest, SplitKeyRoundTrip) {
CHECK(!row2->IsColumnSet("missing"));
}
-namespace {
void CheckExceedCellLimit(const Schema& client_schema,
const vector& col_values,
RowOperationsPB::Type op_type,
@@ -931,6 +934,14 @@ void CheckSplitExceedCellLimit(const Schema& client_schema,
}
} // anonymous namespace
+// Decodes a split row using RowOperationsPBDecoder with
DecoderMode::SPLIT_ROWS under
+// two conditions, one where the client schema doesn't have column IDs as
expected and
+// another where both the client schema and the tablet schema are the same
object.
+TEST_F(RowOperationsTest, SplitKeysRoundTrip) {
+ TestSplitKeys(false);
+ TestSplitKeys(true);
+}
+
TEST_F(RowOperationsTest, ExceedCellLimit) {
Schema client_schema = Schema({ ColumnSchema("key_string", STRING),
ColumnSchema("key_binary", BINARY),
diff --git a/src/kudu/common/row_operations.cc
b/src/kudu/common/row_operations.cc
index 363a424..84cfbce 100644
--- a/src/kudu/common/row_operations.cc
+++ b/src/kudu/common/row_operations.cc
@@ -408,6 +408,14 @@ class ClientServerMapping {
DISALLOW_COPY_AND_ASSIGN(ClientServerMapping);
};
+size_t RowOperationsPBDecoder::GetTabletColIdx(const ClientServerMapping&
mapping,
+ size_t client_col_idx) {
+ if (client_schema_ != tablet_schema_) {
+return mapping.client_to_tablet_idx(client_col_idx);
+ }
+ return client_col_idx;
+}
+
Status RowOperationsPBDecoder::DecodeInsertOrUpsert(const uint8_t*
prototype_row_storage,
const ClientServerMapping&
mapping,
DecodedRowOperation* op) {
@@ -446,7
[kudu] branch master updated: tablet: allow interleaving of row liveness between compaction input rows
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new e9ea984 tablet: allow interleaving of row liveness between compaction
input rows
e9ea984 is described below
commit e9ea9840d30cc223e3a8f746169c11353e23ffa8
Author: Andrew Wong
AuthorDate: Wed Nov 11 17:16:16 2020 -0800
tablet: allow interleaving of row liveness between compaction input rows
It was previously true that when merging multiple rowsets, a row's
liveness always moved to the same or newer rowsets. I.e., if a row was
deleted and inserted again, the new insert would either land in the same
rowset from which it was deleted (e.g. if deleting from the MRS and
inserting back into the same MRS), or a rowset whose rows are newer than
the rowset being deleted from (e.g. if deleting from a DRS and inserting
into the MRS).
This invariant was upheld by various checks in compaction code. The
invariant is no longer true when considering transactional inserts.
Take the following example:
- ts=10: insert 'a' to the tablet's MRS
- ts=11: delete 'a' from the tablet's MRS
- begin txn
- insert 'a' to a transactional MRS
- ts=12: commit the transactional MRS
- ts=13: delete 'a' from the transactional MRS
- ts=14: insert 'a' to the tablet's MRS
In considering the row history for 'a' in the context of defining the
row's history, we're left with the following row histories in each
rowset, which serve as an input to our history-merging code:
tablet MRS: UNDO(del@10) <- 'a' -> REDO(del@11) -> REDO(reins@14)
txn MRS:UNDO(del@12) <- 'a' -> REDO(del@13)
Previously, our invariant meant that one of these rows entire history
(both UNDOs and REDOs) was entirely ahead of the other. This meant that
merging was a matter of determining which input is older (which must be
a deleted "ghost" row, since there can only be a single live row at a
time), converting the older row and its history into UNDOs, and merging
that UNDO history with the newer row's UNDOs. Given the above sequence,
defining an older row isn't as straightforward, given the overlapping of
the histories. This led to broken invariants in the form of CHECK
failures or incorrect results.
However, a notable obvservation is that there _is_ a discernable history
that does abide by our previously held invariant, if we transfer the
newer REDOs from the tablet's MRS input to the txn's MRS input:
UNDO(del@10) <- 'a' -> REDO(del@11)
UNDO(del@12) <- 'a' -> REDO(del@13) -> REDO(reins@14)
This transferral is safe because we still expect that only a single
instance of a row can be "live" at a time. I.e., if there is such
overlap, it is caused by the deletion of a row from the older rowset,
and in transferring the history, there is no risk of ending up with two
histories for the same row that both end with a live row.
This patch implements this transferral of history onto the newer input,
and adds some fuzz test cases that helped snuff out the aforementioned
CHECK failures and incorrect results.
It also enables transactional ops in fuzz-itest, which helped find this
issue. Without this patch, fuzz-itest with transactional support fails
2/10 times in DEBUG mode. With it, it has passed 1000/1000 times.
Change-Id: I042a7d70d32edf9d2a3a077790821893f162880a
Reviewed-on: http://gerrit.cloudera.org:8080/16752
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/integration-tests/fuzz-itest.cc | 235 +++--
src/kudu/tablet/compaction-test.cc | 156 -
src/kudu/tablet/compaction.cc| 290 ---
src/kudu/tablet/compaction.h | 4 +-
src/kudu/tablet/mutation.h | 4 +
5 files changed, 608 insertions(+), 81 deletions(-)
diff --git a/src/kudu/integration-tests/fuzz-itest.cc
b/src/kudu/integration-tests/fuzz-itest.cc
index 01c0b40..499454f 100644
--- a/src/kudu/integration-tests/fuzz-itest.cc
+++ b/src/kudu/integration-tests/fuzz-itest.cc
@@ -259,8 +259,6 @@ struct Redo {
optional val;
};
-// TODO(awong): Merging multiple transactional MRSs together can sometimes lead
-// to a crash. Uncomment the transactional ops once fixed.
const vector kAllOps {TEST_INSERT,
TEST_INSERT_PK_ONLY,
TEST_INSERT_IGNORE,
@@ -279,15 +277,13 @@ const vector kAllOps {TEST_INSERT,
TEST_COMPACT_TABLET,
TEST_RESTART_TS,
TEST_SCAN_AT_TIMESTAMP,
[kudu] 01/03: test: add more natural test for KUDU-2233
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 930fe5bf8f02f97061bddcfa4199ab5a25f15419
Author: Andrew Wong
AuthorDate: Tue Feb 23 17:28:31 2021 -0800
test: add more natural test for KUDU-2233
I have a patch in-flight that touches an area of the code around where
we expect the infamous KUDU-2233 crash. Before merging it, I'd like to
ensure the graceful handling of this corruption is unaffected,
especially in cases where data has previously been corrupted and we've
just upgraded to a newer version of Kudu.
This patch adds such a test case, where data is corrupted but does not
result in a crash, and the tserver is restarted with bits that handle
corruption gracefully. In doing so, this patch also adds an off switch
to all of the guardrails we installed for KUDU-2233.
Change-Id: Icac3ad0a1b6bb9b17d5b6a901dc5bba79c0840fa
Reviewed-on: http://gerrit.cloudera.org:8080/17114
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/integration-tests/test_workload.cc| 6 +-
src/kudu/integration-tests/test_workload.h | 6 +-
.../timestamp_advancement-itest.cc | 131 +++--
src/kudu/tablet/compaction.cc | 35 --
src/kudu/tablet/tablet.cc | 7 +-
src/kudu/tablet/tablet_bootstrap.cc| 4 +-
src/kudu/tablet/tablet_replica.cc | 6 +-
7 files changed, 173 insertions(+), 22 deletions(-)
diff --git a/src/kudu/integration-tests/test_workload.cc
b/src/kudu/integration-tests/test_workload.cc
index 5cc6bdd..f646a70 100644
--- a/src/kudu/integration-tests/test_workload.cc
+++ b/src/kudu/integration-tests/test_workload.cc
@@ -161,7 +161,8 @@ void TestWorkload::WriteThread() {
unique_ptr insert(table->NewInsert());
KuduPartialRow* row = insert->mutable_row();
int32_t key;
- if (write_pattern_ == INSERT_SEQUENTIAL_ROWS) {
+ if (write_pattern_ == INSERT_SEQUENTIAL_ROWS ||
+ write_pattern_ == INSERT_SEQUENTIAL_ROWS_WITH_DELETE) {
key = sequential_key_gen_.Increment();
} else {
key = rng_.Next();
@@ -195,7 +196,8 @@ void TestWorkload::WriteThread() {
SleepFor(MonoDelta::FromMilliseconds(write_interval_millis_));
}
// Write delete row to cluster.
-if (write_pattern_ == INSERT_RANDOM_ROWS_WITH_DELETE) {
+if (write_pattern_ == INSERT_RANDOM_ROWS_WITH_DELETE ||
+write_pattern_ == INSERT_SEQUENTIAL_ROWS_WITH_DELETE) {
for (auto key : keys) {
unique_ptr op(table->NewDelete());
KuduPartialRow* row = op->mutable_row();
diff --git a/src/kudu/integration-tests/test_workload.h
b/src/kudu/integration-tests/test_workload.h
index f93cb58..fb86b76 100644
--- a/src/kudu/integration-tests/test_workload.h
+++ b/src/kudu/integration-tests/test_workload.h
@@ -212,7 +212,10 @@ class TestWorkload {
// Insert sequential rows.
// This causes flushes but no compactions.
-INSERT_SEQUENTIAL_ROWS
+INSERT_SEQUENTIAL_ROWS,
+
+// Insert sequential rows, then delete them.
+INSERT_SEQUENTIAL_ROWS_WITH_DELETE,
};
void set_write_pattern(WritePattern pattern) {
@@ -225,6 +228,7 @@ class TestWorkload {
case INSERT_RANDOM_ROWS_WITH_DELETE:
case UPDATE_ONE_ROW:
case INSERT_SEQUENTIAL_ROWS:
+ case INSERT_SEQUENTIAL_ROWS_WITH_DELETE:
set_already_present_allowed(false);
break;
default: LOG(FATAL) << "Unsupported WritePattern.";
diff --git a/src/kudu/integration-tests/timestamp_advancement-itest.cc
b/src/kudu/integration-tests/timestamp_advancement-itest.cc
index cc161cc..3e782d9 100644
--- a/src/kudu/integration-tests/timestamp_advancement-itest.cc
+++ b/src/kudu/integration-tests/timestamp_advancement-itest.cc
@@ -52,6 +52,7 @@
#include "kudu/mini-cluster/internal_mini_cluster.h"
#include "kudu/mini-cluster/mini_cluster.h"
#include "kudu/rpc/rpc_controller.h"
+#include "kudu/tablet/metadata.pb.h"
#include "kudu/tablet/mvcc.h"
#include "kudu/tablet/tablet.h"
#include "kudu/tablet/tablet_replica.h"
@@ -68,13 +69,22 @@
#include "kudu/util/test_util.h"
DECLARE_bool(enable_maintenance_manager);
+DECLARE_bool(compaction_force_small_rowset_tradeoff);
+DECLARE_bool(prevent_kudu_2233_corruption);
DECLARE_bool(log_preallocate_segments);
DECLARE_bool(log_async_preallocate_segments);
DECLARE_bool(raft_enable_pre_election);
+DECLARE_double(compaction_minimum_improvement);
DECLARE_double(leader_failure_max_missed_heartbeat_periods);
DECLARE_int32(consensus_inject_latency_ms_in_notifications);
DECLARE_int32(heartbeat_interval_ms);
DECLARE_int32(raft_heartbeat_interval_ms)
[kudu] 03/03: KUDU-2612: don't return NOT_FOUND when BEGIN_TXN has not yet run
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3faeb3d9c6c87e4d6815e06945f331a4a14402ba
Author: Andrew Wong
AuthorDate: Thu Feb 25 16:18:52 2021 -0800
KUDU-2612: don't return NOT_FOUND when BEGIN_TXN has not yet run
In testing an in-flight patch[1], it was found that our current handling of
errors when participant registration hasn't completed can lead to
incorrect behavior: the path in which we handle NOT_FOUND errors while
committing expects that such errors only occur if the tablet has been
deleted, and we currently proceed with the commit. The incorrect
assumption here was that NOT_FOUND errors are only produced when the
tablet has been deleted, which is not the case.
This behavior will be amended in an upcoming patch[2], and we'll
actually abort the transaction on NOT_FOUND errors. Until then, this
patch adjust the behavior to return ILLEGAL_STATE instead of NOT_FOUND,
so at least the error type is consistent with the rest of the
participant op lifecycle errors.
[1] https://gerrit.cloudera.org/c/17037/
[2] https://gerrit.cloudera.org/c/17022
Change-Id: I8fa8caba384ee30536114a3e6466ad90b6d8e45d
Reviewed-on: http://gerrit.cloudera.org:8080/17127
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
Reviewed-by: Hao Hao
---
src/kudu/tablet/txn_participant-test.cc | 2 +-
src/kudu/tablet/txn_participant.h | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/kudu/tablet/txn_participant-test.cc
b/src/kudu/tablet/txn_participant-test.cc
index 7cdbc39..ca8af95 100644
--- a/src/kudu/tablet/txn_participant-test.cc
+++ b/src/kudu/tablet/txn_participant-test.cc
@@ -256,7 +256,7 @@ TEST_F(TxnParticipantTest, TestTransactionNotFound) {
ASSERT_TRUE(resp.has_error());
ASSERT_TRUE(resp.error().has_status());
ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, resp.error().code());
- ASSERT_EQ(AppStatusPB::NOT_FOUND, resp.error().status().code());
+ ASSERT_EQ(AppStatusPB::ILLEGAL_STATE, resp.error().status().code());
ASSERT_FALSE(resp.has_timestamp());
}
};
diff --git a/src/kudu/tablet/txn_participant.h
b/src/kudu/tablet/txn_participant.h
index d276840..c497f04 100644
--- a/src/kudu/tablet/txn_participant.h
+++ b/src/kudu/tablet/txn_participant.h
@@ -140,7 +140,7 @@ class Txn : public RefCountedThreadSafe {
Timestamp timestamp;
TxnState meta_state;
if (!tablet_metadata_->HasTxnMetadata(txn_id_, _state, ))
{
-return Status::NotFound("Transaction hasn't been successfully
started");
+return Status::IllegalState("Transaction hasn't been successfully
started");
}
if (PREDICT_FALSE(meta_state != kCommitted && meta_state !=
kCommitInProgress)) {
*code = tserver::TabletServerErrorPB::TXN_ILLEGAL_STATE;
@@ -293,7 +293,7 @@ class Txn : public RefCountedThreadSafe {
txn_id_));
}
// TODO(awong): add another code for this?
- return Status::NotFound("Transaction hasn't been successfully started");
+ return Status::IllegalState("Transaction hasn't been successfully
started");
}
return Status::OK();
}
[kudu] 02/03: k8s: minor updates to the StatefulSet
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git commit 70e0998fb1003763e596595c1b3ebb57c3d32bf3 Author: Andrew Wong AuthorDate: Sat Feb 20 17:06:33 2021 -0800 k8s: minor updates to the StatefulSet I walked through some of the Kubernetes README and noticed the ports were off. This updates to use the typical defaults we use for masters and tservers. It also updates to use 4 tservers, which is closer to what would be used in production, given 3-4-3 replication. Change-Id: I50b0084f70d30187bcca4e356e38c35b2486c611 Reviewed-on: http://gerrit.cloudera.org:8080/17098 Tested-by: Kudu Jenkins Reviewed-by: Hao Hao Reviewed-by: Alexey Serbin --- kubernetes/README.adoc | 39 --- kubernetes/kudu-services.yaml| 11 --- kubernetes/kudu-statefulset.yaml | 10 +- 3 files changed, 33 insertions(+), 27 deletions(-) diff --git a/kubernetes/README.adoc b/kubernetes/README.adoc index c06f509..dafb89d 100644 --- a/kubernetes/README.adoc +++ b/kubernetes/README.adoc @@ -27,9 +27,10 @@ NOTE: Read more about Kubernetes here https://kubernetes.io/docs/tutorials/kuber System Requirements -kubectl -minikube or kubeadm -docker +- `minikube` or `kubeadm` to deploy your Kubernetes cluster. Start a Kubernetes + cluster before running through the next steps +- `kubectl` to run commands against the Kubernetes cluster +- `docker` to serve containers Build Kudu Docker Image @@ -50,11 +51,10 @@ Check status of kudu services: You should see below output on stdout ``` -NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE -kudu-master-ui NodePort10.110.150.101 8051:30082/TCP 8m2s -kudu-masters ClusterIP None 8051/TCP,8050/TCP 8m2s -kudu-tserversClusterIP None 7050/TCP,7051/TCP 8m2s -tiller-deployClusterIP 10.96.104.11 44134/TCP 47h +NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE +kudu-master-ui NodePort10.108.52.243 8051:30239/TCP 87m +kudu-masters ClusterIP None8051/TCP,7051/TCP 87m +kudu-tserversClusterIP None8050/TCP,7050/TCP 87m ``` === Create StatefulSet for Kudu Masters and Kudu TServers @@ -71,17 +71,18 @@ You should see below output on stdout ``` NAME READY AGE -kudu-master2/3 16s -kudu-tserver 3/3 16s - - -NAME READY STATUSRESTARTS AGE -kudu-master-01/1 Running 0 9m9s -kudu-master-11/1 Running 0 9m9s -kudu-master-21/1 Running 0 9m9s -kudu-tserver-0 1/1 Running 0 9m19s -kudu-tserver-1 1/1 Running 0 9m5s -kudu-tserver-2 1/1 Running 0 9m5s +kudu-master3/3 89m +kudu-tserver 4/4 89m + + +NAME READY STATUSRESTARTS AGE +kudu-master-01/1 Running 1 90m +kudu-master-11/1 Running 1 90m +kudu-master-21/1 Running 0 89m +kudu-tserver-0 1/1 Running 1 90m +kudu-tserver-1 1/1 Running 1 90m +kudu-tserver-2 1/1 Running 1 90m +kudu-tserver-3 1/1 Running 1 90m ``` Port Forward The Kudu Master UI diff --git a/kubernetes/kudu-services.yaml b/kubernetes/kudu-services.yaml index a5d0dfc..998e38c 100644 --- a/kubernetes/kudu-services.yaml +++ b/kubernetes/kudu-services.yaml @@ -15,6 +15,11 @@ # specific language governing permissions and limitations # under the License. + +# From https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ +# > StatefulSets currently require a Headless Service to be responsible for the +# network identity of the Pods. You are responsible for creating this Service. + --- # headless service for kudu masters @@ -31,7 +36,7 @@ spec: - name: ui port: 8051 - name: rpc-port - port: 8050 + port: 7051 selector: app: kudu-master @@ -68,8 +73,8 @@ spec: clusterIP: None ports: - name: ui - port: 7050 + port: 8050 - name: rpc-port - port: 7051 + port: 7050 selector: app: kudu-tserver diff --git a/kubernetes/kudu-statefulset.yaml b/kubernetes/kudu-statefulset.yaml index eddae07..f95d0d9 100644 --- a/kubernetes/kudu-statefulset.yaml +++ b/kubernetes/kudu-statefulset.yaml @@ -53,9 +53,9 @@ spec: value: "kudu-master-0.kudu-masters.apache-kudu.svc.cluster.local,kudu-master-1.kudu-masters.apache-kudu.svc.cluster.local,kudu-master-2.kudu-master
[kudu] branch master updated (1ba4a4b -> 3faeb3d)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 1ba4a4b [client-test] added WriteWhileRestartingMultipleTabletServers new 930fe5b test: add more natural test for KUDU-2233 new 70e0998 k8s: minor updates to the StatefulSet new 3faeb3d KUDU-2612: don't return NOT_FOUND when BEGIN_TXN has not yet run The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: kubernetes/README.adoc | 39 +++--- kubernetes/kudu-services.yaml | 11 +- kubernetes/kudu-statefulset.yaml | 10 +- src/kudu/integration-tests/test_workload.cc| 6 +- src/kudu/integration-tests/test_workload.h | 6 +- .../timestamp_advancement-itest.cc | 131 +++-- src/kudu/tablet/compaction.cc | 35 -- src/kudu/tablet/tablet.cc | 7 +- src/kudu/tablet/tablet_bootstrap.cc| 4 +- src/kudu/tablet/tablet_replica.cc | 6 +- src/kudu/tablet/txn_participant-test.cc| 2 +- src/kudu/tablet/txn_participant.h | 4 +- 12 files changed, 209 insertions(+), 52 deletions(-)
[kudu] branch master updated: KUDU-2671: Adds compatibility for per range hash schemas with unbounded ranges.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new d7b5abc KUDU-2671: Adds compatibility for per range hash schemas with
unbounded ranges.
d7b5abc is described below
commit d7b5abc027d492a60ebf5059b27541fc04cfaab3
Author: Mahesh Reddy
AuthorDate: Fri Feb 19 17:14:29 2021 -0800
KUDU-2671: Adds compatibility for per range hash schemas with unbounded
ranges.
This patch updates the logic at the end of
PartitionSchema::CreatePartiitons()
to allow per range hash schemas to be compatible with unbounded ranges. Some
additional context about the block of code is given below.
For the start partition key, it iterates in reverse order through the
partition's
hash buckets. It breaks out of the loop at the first instance of a bucket
not
equal to 0; If the bucket is equal to 0 it erases that part of the
partition key.
Essentially, if all hash buckets are equal to 0, then it erases the entire
key.
For the end partition key, it also iterates in reverse order through the
partition's hash buckets. It first erases the index portition of the
partition key. It then checks if the current hash bucket is the max
bucket of the current hash schema. If it is not the max, it encodes the
current hash bucket + 1 at the index portion of the key and breaks the loop.
If it is the max, it continues within the loop. Essentially, if all the
hash buckets are the max then it erases the entire key.
Prior to this change, this block of code assumed the same hash bucket
schema for each partition. With per range hash schemas, that may not
necessarily be the case. The vector 'partition_idx_to_hash_schemas_idx'
maps each partition to the index of 'bounds_with_hash_schemas' to ensure
the correct hash bucket schema is used. '-1' is used to signify the use
of the table wide hash schema.
Change-Id: I5f6c709e211359b04f7597af5f670c787bda7481
Reviewed-on: http://gerrit.cloudera.org:8080/17090
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/common/partition-test.cc | 139 ++
src/kudu/common/partition.cc | 38 +--
2 files changed, 172 insertions(+), 5 deletions(-)
diff --git a/src/kudu/common/partition-test.cc
b/src/kudu/common/partition-test.cc
index 03670a4..70b63f8 100644
--- a/src/kudu/common/partition-test.cc
+++ b/src/kudu/common/partition-test.cc
@@ -1089,4 +1089,143 @@ TEST_F(PartitionTest, TestVaryingHashSchemasPerRange) {
"range hash schemas.", s1.ToString());
}
+
+TEST_F(PartitionTest, TestVaryingHashSchemasPerUnboundedRanges) {
+ // CREATE TABLE t (a VARCHAR, b VARCHAR, c VARCHAR, PRIMARY KEY (a, b, c))
+ // PARTITION BY [HASH BUCKET (a, c), HASH BUCKET (b), RANGE (a, b, c)];
+ Schema schema({ ColumnSchema("a", STRING),
+ ColumnSchema("b", STRING),
+ ColumnSchema("c", STRING) },
+{ ColumnId(0), ColumnId(1), ColumnId(2) }, 3);
+
+ PartitionSchemaPB schema_builder;
+ // Table-wide hash schema defined below.
+ AddHashBucketComponent(_builder, { "b" }, 2, 0);
+ PartitionSchema partition_schema;
+ ASSERT_OK(PartitionSchema::FromPB(schema_builder, schema,
_schema));
+
+ ASSERT_EQ("HASH (b) PARTITIONS 2, RANGE (a, b, c)",
+partition_schema.DebugString(schema));
+
+ vector> bounds;
+ PartitionSchema::RangeHashSchema range_hash_schemas;
+
+ { // [(_, _, _), (a1, _, c1))
+KuduPartialRow lower();
+KuduPartialRow upper();
+ASSERT_OK(upper.SetStringCopy("a", "a1"));
+ASSERT_OK(upper.SetStringCopy("c", "c1"));
+PartitionSchema::HashBucketSchemas hash_schema_4_buckets =
{{{ColumnId(0)}, 4, 0}};
+bounds.emplace_back(lower, upper);
+range_hash_schemas.emplace_back(hash_schema_4_buckets);
+ }
+
+ { // [(a2, b2, _), (a3, b3, _))
+KuduPartialRow lower();
+KuduPartialRow upper();
+ASSERT_OK(lower.SetStringCopy("a", "a2"));
+ASSERT_OK(lower.SetStringCopy("b", "b2"));
+ASSERT_OK(upper.SetStringCopy("a", "a3"));
+ASSERT_OK(upper.SetStringCopy("b", "b3"));
+bounds.emplace_back(lower, upper);
+range_hash_schemas.emplace_back(PartitionSchema::HashBucketSchemas());
+ }
+
+ { // [(a4, b4, _), (_, _, _))
+KuduPartialRow lower();
+KuduPartialRow upper();
+ASSERT_OK(lower.SetStringCopy("a", "a4"));
+ASSERT_OK(lower.SetStringCopy("b", "b4"));
+PartitionSchema::HashBucketSchemas hash_schema_2_buckets_by_3 = {
+{{ColumnId(0)}, 2, 0},
+{{ColumnId(2)}, 3, 0}
+};
+bounds.empl
[kudu] branch master updated: python: pin pytest-runner to <5.3.0
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/master by this push: new f24ec6a python: pin pytest-runner to <5.3.0 f24ec6a is described below commit f24ec6a7757af421f6da3e5f4b93f3a6761d23bb Author: Andrew Wong AuthorDate: Tue Feb 16 17:17:59 2021 -0800 python: pin pytest-runner to <5.3.0 Per https://github.com/pytest-dev/pytest-runner/blob/v5.3.0/CHANGES.rst, version 5.3.0 started requiring python 3.6 or higher, which is above the minimum supported for Kudu. So, this pins the version to below 5.3.0. Change-Id: I54fe40b0a6e56f74c4a3d7ce2d336d25ce6a33e2 Reviewed-on: http://gerrit.cloudera.org:8080/17074 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- python/setup.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/python/setup.py b/python/setup.py index a09f7a9..86fe70b 100644 --- a/python/setup.py +++ b/python/setup.py @@ -191,7 +191,10 @@ setup( 'clean': clean, 'build_ext': build_ext }, -setup_requires=['pytest-runner'], +# pytest-runner 5.3.0 [1] started requiring python 3.6 or later. +# +# 1. https://github.com/pytest-dev/pytest-runner/blob/v5.3.0/CHANGES.rst +setup_requires=['pytest-runner <5.3.0'], # Note: dependencies in tests_require should also be listed in # requirements.txt so that dependencies aren't downloaded at test-time
[kudu] branch master updated: KUDU-2612: make BeginCommit return OK if already committed
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new b73f759 KUDU-2612: make BeginCommit return OK if already committed
b73f759 is described below
commit b73f75927f703eb7a5fbaaed525fd9c6b8c140f1
Author: Andrew Wong
AuthorDate: Tue Feb 16 15:42:58 2021 -0800
KUDU-2612: make BeginCommit return OK if already committed
This patch makes repeat calls to BeginCommit by the TxnStatusManager
return OK rather than an IllegalState error. This has previously
surfaced in flakiness in some tests, wherein repeated calls to
TxnManager::Commit() yields an IllegalState error since the commit has
already completed.
This also reverts 67018be8ba27480b050c11504df8a732f6a52daf that
addressed this by increasing commit latency.
Change-Id: I420ba1c5460d59103984ea9a1f16177b82b8d0e1
Reviewed-on: http://gerrit.cloudera.org:8080/17073
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/integration-tests/txn_commit-itest.cc | 1 -
src/kudu/transactions/txn_status_manager-test.cc | 8 ++--
src/kudu/transactions/txn_status_manager.cc | 3 ++-
3 files changed, 4 insertions(+), 8 deletions(-)
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index c43fc1d..17f2078 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -682,7 +682,6 @@ TEST_F(TxnCommitITest, TestConcurrentCommitCalls) {
// Test that committing the same transaction concurrently doesn't lead to any
// issues.
TEST_F(TxnCommitITest, TestConcurrentRepeatedCommitCalls) {
- FLAGS_txn_status_manager_inject_latency_finalize_commit_ms = 1000;
shared_ptr txn;
shared_ptr txn_session;
ASSERT_OK(BeginTransaction(participant_ids_, , _session));
diff --git a/src/kudu/transactions/txn_status_manager-test.cc
b/src/kudu/transactions/txn_status_manager-test.cc
index 35fa2da..61f38c6 100644
--- a/src/kudu/transactions/txn_status_manager-test.cc
+++ b/src/kudu/transactions/txn_status_manager-test.cc
@@ -806,14 +806,10 @@ TEST_F(TxnStatusManagerTest, TestUpdateTransactionState) {
s = txn_manager_->AbortTransaction(kTxnId2, kOwner, _error);
ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
- // Redundant finalize calls are also benign.
+ // Redundant begin commit or finalize calls are also benign.
+ ASSERT_OK(txn_manager_->BeginCommitTransaction(kTxnId2, kOwner, _error));
ASSERT_OK(txn_manager_->FinalizeCommitTransaction(
kTxnId2, Timestamp::kInitialTimestamp, _error));
-
- // Calls to begin committing should return an error if we've already
- // finalized the commit.
- s = txn_manager_->BeginCommitTransaction(kTxnId2, kOwner, _error);
- ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
}
// Test that we can only add participants to a transaction when it's in an
diff --git a/src/kudu/transactions/txn_status_manager.cc
b/src/kudu/transactions/txn_status_manager.cc
index 33f2df8..e6ec4da 100644
--- a/src/kudu/transactions/txn_status_manager.cc
+++ b/src/kudu/transactions/txn_status_manager.cc
@@ -923,7 +923,8 @@ Status TxnStatusManager::BeginCommitTransaction(int64_t
txn_id, const string& us
TransactionEntryLock txn_lock(txn.get(), LockMode::WRITE);
const auto& pb = txn_lock.data().pb;
const auto& state = pb.state();
- if (state == TxnStatePB::COMMIT_IN_PROGRESS) {
+ if (state == TxnStatePB::COMMIT_IN_PROGRESS ||
+ state == TxnStatePB::COMMITTED) {
return Status::OK();
}
if (PREDICT_FALSE(state != TxnStatePB::OPEN)) {
[kudu] branch master updated: KUDU-2612: add background task to abort transaction participants
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 451a4c6 KUDU-2612: add background task to abort transaction
participants
451a4c6 is described below
commit 451a4c61eed132e00b6dc50dcaf4cfeca348be2a
Author: Andrew Wong
AuthorDate: Sun Jan 31 02:30:59 2021 -0800
KUDU-2612: add background task to abort transaction participants
This patch implements background tasks that abort a given transaction
when TxnStatusManager::AbortTransaction() is called. Similar to the
commit tasks, aborts have the following life cycle:
1. AbortTransaction() is called. A new state, ABORT_IN_PROGRESS, is
written to the TxnStatusManager.
2. ABORT_TXN ops are sent to all participants in the transaction.
3. Once all participants have responded, the ABORTED state is written
to the TxnStatusManager.
This patch doesn't test races between commits and aborts. Some reworking
of the commit tasks will be required to account for such races, and will
be done in a follow-up.
Change-Id: I484c315c6f7331c5ec12cb06370fbaae9c7c343e
Reviewed-on: http://gerrit.cloudera.org:8080/17017
Tested-by: Andrew Wong
Reviewed-by: Hao Hao
---
.../org/apache/kudu/client/KuduTransaction.java| 2 +
.../apache/kudu/client/TestKuduTransaction.java| 9 +-
src/kudu/client/client-test.cc | 18 +-
src/kudu/client/transaction-internal.cc| 4 +
src/kudu/integration-tests/txn_commit-itest.cc | 229 +++--
.../integration-tests/txn_status_manager-itest.cc | 6 +-
.../integration-tests/txn_status_table-itest.cc| 4 +-
src/kudu/master/txn_manager-test.cc| 4 +-
src/kudu/transactions/transactions.proto | 1 +
src/kudu/transactions/txn_status_manager-test.cc | 10 +-
src/kudu/transactions/txn_status_manager.cc| 169 +--
src/kudu/transactions/txn_status_manager.h | 23 ++-
12 files changed, 420 insertions(+), 59 deletions(-)
diff --git
a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
index 292ccdb..19286f2 100644
--- a/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
+++ b/java/kudu-client/src/main/java/org/apache/kudu/client/KuduTransaction.java
@@ -309,6 +309,8 @@ public class KuduTransaction implements AutoCloseable {
GetTransactionStateResponse resp = KuduClient.joinAndHandleException(d);
final Transactions.TxnStatePB txnState = resp.txnState();
switch (txnState) {
+ case ABORT_IN_PROGRESS:
+throw new NonRecoverableException(Status.Aborted("transaction is being
aborted"));
case ABORTED:
throw new NonRecoverableException(Status.Aborted("transaction was
aborted"));
case OPEN:
diff --git
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
index 37c7772..5b8b035 100644
---
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
+++
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
@@ -249,6 +249,9 @@ public class TestKuduTransaction {
@MasterServerConfig(flags = {
"--txn_manager_enabled",
})
+ @TabletServerConfig(flags = {
+ "--txn_schedule_background_tasks=false"
+ })
public void testIsCommitCompleteSpecialCases() throws Exception {
KuduTransaction txn = client.newTransaction();
@@ -276,7 +279,7 @@ public class TestKuduTransaction {
}
});
assertTrue(ex.getStatus().isAborted());
- assertEquals("transaction was aborted", ex.getMessage());
+ assertEquals("transaction is being aborted", ex.getMessage());
}
// Try to call isCommitComplete() on a handle that isn't backed by any
@@ -606,7 +609,7 @@ public class TestKuduTransaction {
});
final String errmsg = ex.getMessage();
assertTrue(errmsg, errmsg.matches(
- ".* transaction ID .* is not open: state: ABORTED .*"));
+ ".* transaction ID .* is not open: state: ABORT.*"));
// Verify that KuduTransaction.rollback() successfully runs on a
transaction
// handle if the underlying transaction is already aborted automatically
@@ -666,7 +669,7 @@ public class TestKuduTransaction {
});
final String errmsg = ex.getMessage();
assertTrue(errmsg, errmsg.matches(
- ".* transaction ID .* is not open: state: ABORTED .*"));
+ ".* transaction ID .* is not open: state: ABORT.*"));
// Verify that
[kudu] branch master updated: txns: syntactically sweeten some tuple-like variables
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 7afdac0 txns: syntactically sweeten some tuple-like variables
7afdac0 is described below
commit 7afdac0155f1c0730242594b6d13a549c3084d5d
Author: Andrew Wong
AuthorDate: Mon Feb 8 16:07:18 2021 -0800
txns: syntactically sweeten some tuple-like variables
This uses the C++17 feature to use structured bindings[1] in a few
places.
To pass lint, this patch also pulls in a patch[2] merged into the
cpplint project.
Note that the Google Style Guide[3] does mention structured bindings, so
they seem like fair game in Kudu's codebase.
[1] https://en.cppreference.com/w/cpp/language/structured_binding
[2] https://github.com/cpplint/cpplint/pull/126
[3] https://google.github.io/styleguide/cppguide.html#Type_deduction
Change-Id: Ice0add1d0c8487f97772732cffb463292efe8910
Reviewed-on: http://gerrit.cloudera.org:8080/17046
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/transactions/txn_status_manager.cc| 34 ++
thirdparty/download-thirdparty.sh | 2 +-
thirdparty/patches/google-styleguide-cpplint.patch | 11 +++
3 files changed, 27 insertions(+), 20 deletions(-)
diff --git a/src/kudu/transactions/txn_status_manager.cc
b/src/kudu/transactions/txn_status_manager.cc
index b2d7a1f..9111451 100644
--- a/src/kudu/transactions/txn_status_manager.cc
+++ b/src/kudu/transactions/txn_status_manager.cc
@@ -22,6 +22,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -362,10 +363,7 @@ void
TxnStatusManagerBuildingVisitor::VisitTransactionEntries(
{
// Lock the transaction while we build the participants.
TransactionEntryLock txn_lock(txn.get(), LockMode::READ);
-for (auto& participant_and_state : participants) {
- const auto& prt_id = participant_and_state.first;
- auto& prt_entry_pb = participant_and_state.second;
-
+for (auto& [prt_id, prt_entry_pb] : participants) {
// Register a participant entry for this transaction.
auto prt = txn->GetOrCreateParticipant(prt_id);
ParticipantEntryLock l(prt.get(), LockMode::WRITE);
@@ -493,10 +491,9 @@ Status TxnStatusManager::LoadFromTabletUnlocked() {
unordered_map> commits_in_flight;
unordered_map> new_tasks;
if (txn_client) {
-for (const auto& txn_id_and_entry : txns_by_id) {
- const auto& txn_entry = txn_id_and_entry.second;
- if (txn_id_and_entry.second->state() == TxnStatePB::COMMIT_IN_PROGRESS) {
-const auto& txn_id = txn_id_and_entry.first;
+for (const auto& [txn_id, txn_entry] : txns_by_id) {
+ const auto& state = txn_entry->state();
+ if (state == TxnStatePB::COMMIT_IN_PROGRESS) {
new_tasks.emplace(txn_id,
new CommitTasks(txn_id, txn_entry->GetParticipantIds(),
txn_client, commit_pool_, this));
@@ -509,15 +506,15 @@ Status TxnStatusManager::LoadFromTabletUnlocked() {
txns_by_id_ = std::move(txns_by_id);
commits_in_flight = std::move(commits_in_flight_);
// Stop any previously on-going tasks.
-for (const auto& txn_id_and_tasks : commits_in_flight) {
- txn_id_and_tasks.second->stop();
+for (const auto& [_, tasks] : commits_in_flight) {
+ tasks->stop();
}
commits_in_flight_ = std::move(new_tasks);
if (!commits_in_flight_.empty()) {
LOG(INFO) << Substitute("Starting $0 commit tasks",
commits_in_flight_.size());
-}
-for (const auto& id_and_commit_task : commits_in_flight_) {
- id_and_commit_task.second->BeginCommitAsync();
+ for (const auto& [_, tasks] : commits_in_flight_) {
+tasks->BeginCommitAsync();
+ }
}
}
return Status::OK();
@@ -843,12 +840,12 @@ Status TxnStatusManager::BeginCommitTransaction(int64_t
txn_id, const string& us
if (PREDICT_TRUE(FLAGS_txn_schedule_background_tasks)) {
auto participant_ids = txn->GetParticipantIds();
std::unique_lock l(lock_);
-auto iter_and_emplaced = commits_in_flight_.emplace(txn_id,
+auto [map_iter, emplaced] = commits_in_flight_.emplace(txn_id,
new CommitTasks(txn_id, std::move(participant_ids),
txn_client, commit_pool_, this));
l.unlock();
-if (iter_and_emplaced.second) {
- iter_and_emplaced.first->second->BeginCommitAsync();
+if (emplaced) {
+ map_iter->second->BeginCommitAsync();
}
}
txn_lock.Commit();
@@ -1074,11 +1071,10 @@ void TxnStatusManager::AbortStaleTransactions() {
ParticipantIdsByTxnId TxnStatusManager::GetParticipantsByTxnIdForTests() const
{
Partici
[kudu] 03/04: txn_commit-itest: reduce runtime of TestCommitWhileDeletingTxnStatusManager
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 70b69cc3c168e9f49a9aa62890fb3c095adeb78b
Author: Andrew Wong
AuthorDate: Fri Feb 5 20:28:10 2021 -0800
txn_commit-itest: reduce runtime of TestCommitWhileDeletingTxnStatusManager
The test waits for a client timeout, which by default is 30 seconds.
This reduces it to 10 seconds, which seems like a reasonable timeout for
the minimal amount of DDL done in the rest of the test cases.
Change-Id: I2f4f3868bf7d88b0fe34cc63d6139ec867eba58e
Reviewed-on: http://gerrit.cloudera.org:8080/17033
Reviewed-by: Hao Hao
Tested-by: Andrew Wong
---
src/kudu/integration-tests/txn_commit-itest.cc | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index 23fb461..722c4b0 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -136,7 +136,9 @@ class TxnCommitITest : public KuduTest {
TxnSystemClient::Create(cluster_->master_rpc_addrs(), _client_);
ASSERT_OK(txn_client_->OpenTxnStatusTable());
-ASSERT_OK(cluster_->CreateClient(nullptr, _));
+client::KuduClientBuilder builder;
+builder.default_admin_operation_timeout(kTimeout);
+ASSERT_OK(cluster_->CreateClient(, _));
string authn_creds;
ASSERT_OK(client_->ExportAuthenticationCredentials(_creds));
client::AuthenticationCredentialsPB pb;
[kudu] 01/04: txn_commit-itest: deflake TestCommitTasksReloadOnLeadershipChange
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 546e68cfd39e2a6f1b16bd1ddb580d1ebc97c9a4
Author: Andrew Wong
AuthorDate: Fri Feb 5 18:04:23 2021 -0800
txn_commit-itest: deflake TestCommitTasksReloadOnLeadershipChange
The test shows up on the flaky test dashboard as failing around 20% of
the time. As it turns out, transferring leadership by quiescing multiple
replicas can lead to flakiness if we happen to pick a lagging replica as
the new leader.
Instead of targeting a specific tablet server as the host of the new
leaders, we'll now just quiesce the old leader tablet server and stop
quiescing the other tablet servers.
I ran the test in DEBUG mode 100 times. Before this patch, it failed 16
times; with it, it passed 100/100 times.
Change-Id: I2b27864e72888367eb0af7de59e044a9e018c31b
Reviewed-on: http://gerrit.cloudera.org:8080/17031
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao
---
src/kudu/integration-tests/txn_commit-itest.cc | 15 ---
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/src/kudu/integration-tests/txn_commit-itest.cc
b/src/kudu/integration-tests/txn_commit-itest.cc
index 084435e..23fb461 100644
--- a/src/kudu/integration-tests/txn_commit-itest.cc
+++ b/src/kudu/integration-tests/txn_commit-itest.cc
@@ -749,7 +749,6 @@ class ThreeNodeTxnCommitITest : public TxnCommitITest {
*cluster_->mini_tablet_server(i)->server()->mutable_quiescing() = i !=
leader_idx;
}
leader_ts_ = cluster_->mini_tablet_server(leader_idx);
-non_leader_ts_ = cluster_->mini_tablet_server(leader_idx + 1);
// We should have two leaders for our table, and one for the
// TxnStatusManager.
ASSERT_EVENTUALLY([&] {
@@ -758,7 +757,6 @@ class ThreeNodeTxnCommitITest : public TxnCommitITest {
}
protected:
MiniTabletServer* leader_ts_;
- MiniTabletServer* non_leader_ts_;
};
TEST_F(ThreeNodeTxnCommitITest, TestCommitTasksReloadOnLeadershipChange) {
@@ -776,13 +774,16 @@ TEST_F(ThreeNodeTxnCommitITest,
TestCommitTasksReloadOnLeadershipChange) {
ASSERT_FALSE(is_complete);
FLAGS_txn_schedule_background_tasks = true;
- // Change our quiescing state and bring the previous leader down so a new
- // leader can be elected.
- auto* new_leader_ts = non_leader_ts_;
- *new_leader_ts->server()->mutable_quiescing() = false;
+ // Change our quiescing states so a new leader can be elected.
*leader_ts_->server()->mutable_quiescing() = true;
+ for (int i = 0; i < cluster_->num_tablet_servers(); i++) {
+auto* mts = cluster_->mini_tablet_server(i);
+if (leader_ts_ != mts) {
+ *mts->server()->mutable_quiescing() = false;
+}
+ }
ASSERT_EVENTUALLY([&] {
-ASSERT_EQ(3, new_leader_ts->server()->num_raft_leaders()->value());
+ASSERT_EQ(0, leader_ts_->server()->num_raft_leaders()->value());
});
// Upon becoming leader, we should have started our commit task and completed
// the commit.
[kudu] 02/04: rpc: reduce logging when server is shutting down
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 4b0524f0bf8e81c004dd3f03a2467e9b0824b617
Author: Andrew Wong
AuthorDate: Fri Feb 5 19:45:00 2021 -0800
rpc: reduce logging when server is shutting down
TxnCommitITest.TestRestartingWhileCommittingAndDeleting exercises the
case where the client RPCs are sent to the tserver as the tserver is
shutting down. This results in several log messages like:
I0205 05:01:20.190737 1310 messenger.cc:384] service
kudu.tserver.TabletServerAdminService not registered on TabletServer
I0205 05:01:20.190891 1310 messenger.cc:384] service
kudu.tserver.TabletServerAdminService not registered on TabletServer
I0205 05:01:20.194700 1310 messenger.cc:384] service
kudu.tserver.TabletServerAdminService not registered on TabletServer
until the RPC timeout is reached. This message only seems interesting to
clients in cases outside the normal operation of a cluster, e.g. if the
endpoint actually doesn't exist, versus if we're just starting up or
shutting down. This patch adjusts the logging to reflect that.
Change-Id: I943a192bc12e3bae534b9129ac4ea8d24404adc6
Reviewed-on: http://gerrit.cloudera.org:8080/17032
Reviewed-by: Hao Hao
Tested-by: Andrew Wong
---
src/kudu/rpc/messenger.cc | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/kudu/rpc/messenger.cc b/src/kudu/rpc/messenger.cc
index f161faf..08716b7 100644
--- a/src/kudu/rpc/messenger.cc
+++ b/src/kudu/rpc/messenger.cc
@@ -381,8 +381,9 @@ void Messenger::QueueInboundCall(unique_ptr
call) {
if (PREDICT_FALSE(!service)) {
const auto msg = Substitute("service $0 not registered on $1",
call->remote_method().service_name(), name_);
-LOG(INFO) << msg;
if (state_ == kServicesRegistered) {
+ // NOTE: this message is only actually interesting if it's not transient.
+ LOG(INFO) << msg;
call.release()->RespondFailure(ErrorStatusPB::ERROR_NO_SUCH_SERVICE,
Status::NotFound(msg));
} else {
call.release()->RespondFailure(
[kudu] branch master updated (b9a8f2e -> d490850)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from b9a8f2e KUDU-2612: background task to commit transaction new 546e68c txn_commit-itest: deflake TestCommitTasksReloadOnLeadershipChange new 4b0524f rpc: reduce logging when server is shutting down new 70b69cc txn_commit-itest: reduce runtime of TestCommitWhileDeletingTxnStatusManager new d490850 [tablet_service] removed namespace prefixes The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: src/kudu/integration-tests/txn_commit-itest.cc | 19 ++-- src/kudu/rpc/messenger.cc | 3 +- src/kudu/tserver/tablet_service.cc | 142 + 3 files changed, 85 insertions(+), 79 deletions(-)
[kudu] 04/04: [tablet_service] removed namespace prefixes
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit d49085093f3b01a9ee1da6c37bdce0924ee536f2
Author: Alexey Serbin
AuthorDate: Fri Feb 5 15:28:58 2021 -0800
[tablet_service] removed namespace prefixes
This patch doesn't contain any functional modifications.
Change-Id: Id467e2a70764d39a02d0ed1dfcf10a65ab855e46
Reviewed-on: http://gerrit.cloudera.org:8080/17029
Tested-by: Kudu Jenkins
Reviewed-by: Bankim Bhavsar
---
src/kudu/tserver/tablet_service.cc | 142 +++--
1 file changed, 72 insertions(+), 70 deletions(-)
diff --git a/src/kudu/tserver/tablet_service.cc
b/src/kudu/tserver/tablet_service.cc
index 611340d..0a1834f 100644
--- a/src/kudu/tserver/tablet_service.cc
+++ b/src/kudu/tserver/tablet_service.cc
@@ -211,26 +211,27 @@ using kudu::consensus::VoteResponsePB;
using kudu::fault_injection::MaybeTrue;
using kudu::pb_util::SecureDebugString;
using kudu::pb_util::SecureShortDebugString;
-using kudu::rpc::ParseVerificationResult;
using kudu::rpc::ErrorStatusPB;
+using kudu::rpc::ParseVerificationResult;
using kudu::rpc::RpcContext;
using kudu::rpc::RpcSidecar;
-using kudu::security::TokenVerifier;
using kudu::security::TokenPB;
+using kudu::security::TokenVerifier;
using kudu::server::ServerBase;
using kudu::tablet::AlterSchemaOpState;
using kudu::tablet::MvccSnapshot;
+using kudu::tablet::OpCompletionCallback;
using kudu::tablet::ParticipantOpState;
using kudu::tablet::TABLET_DATA_COPYING;
using kudu::tablet::TABLET_DATA_DELETED;
using kudu::tablet::TABLET_DATA_TOMBSTONED;
using kudu::tablet::Tablet;
using kudu::tablet::TabletReplica;
-using kudu::tablet::OpCompletionCallback;
+using kudu::tablet::TabletStatePB;
using kudu::tablet::WriteAuthorizationContext;
-using kudu::tablet::WritePrivileges;
-using kudu::tablet::WritePrivilegeType;
using kudu::tablet::WriteOpState;
+using kudu::tablet::WritePrivilegeType;
+using kudu::tablet::WritePrivileges;
using std::shared_ptr;
using std::string;
using std::unique_ptr;
@@ -260,7 +261,7 @@ template
bool LookupTabletReplicaOrRespond(TabletReplicaLookupIf* tablet_manager,
const string& tablet_id,
RespClass* resp,
- rpc::RpcContext* context,
+ RpcContext* context,
scoped_refptr* replica) {
Status s = tablet_manager->GetTabletReplica(tablet_id, replica);
if (PREDICT_FALSE(!s.ok())) {
@@ -280,9 +281,9 @@ bool LookupTabletReplicaOrRespond(TabletReplicaLookupIf*
tablet_manager,
template
void RespondTabletNotRunning(const scoped_refptr& replica,
- tablet::TabletStatePB tablet_state,
+ TabletStatePB tablet_state,
RespClass* resp,
- rpc::RpcContext* context) {
+ RpcContext* context) {
Status s = Status::IllegalState("Tablet not RUNNING",
tablet::TabletStatePB_Name(tablet_state));
auto error_code = TabletServerErrorPB::TABLET_NOT_RUNNING;
@@ -303,9 +304,9 @@ void RespondTabletNotRunning(const
scoped_refptr& replica,
template
bool CheckTabletReplicaRunningOrRespond(const scoped_refptr&
replica,
RespClass* resp,
-rpc::RpcContext* context) {
+RpcContext* context) {
// Check RUNNING state.
- tablet::TabletStatePB state = replica->state();
+ TabletStatePB state = replica->state();
if (PREDICT_FALSE(state != tablet::RUNNING)) {
RespondTabletNotRunning(replica, state, resp, context);
return false;
@@ -322,7 +323,7 @@ template
bool LookupRunningTabletReplicaOrRespond(TabletReplicaLookupIf* tablet_manager,
const string& tablet_id,
RespClass* resp,
- rpc::RpcContext* context,
+ RpcContext* context,
scoped_refptr*
replica) {
if (!LookupTabletReplicaOrRespond(tablet_manager, tablet_id, resp, context,
replica)) {
return false;
@@ -338,7 +339,7 @@ bool CheckUuidMatchOrRespond(TabletReplicaLookupIf*
tablet_manager,
const char* method_name,
const ReqClass* req,
RespClass* resp,
- rpc::RpcContext* context) {
+ RpcContext* context) {
const string& local_uuid = tablet_manager->NodeInstance().permanent_uuid();
if (PREDICT_FALSE(!req->has_dest_uuid())) {
// Maintai
[kudu] branch master updated: KUDU-2612: background task to commit transaction
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new b9a8f2e KUDU-2612: background task to commit transaction
b9a8f2e is described below
commit b9a8f2e633af891b6e2b268a71e18a8e7e1ff34d
Author: Andrew Wong
AuthorDate: Sun Jan 31 02:30:25 2021 -0800
KUDU-2612: background task to commit transaction
This patch introduces background tasks that get run when
KuduTransaction::Commit() is called. The typical workflow is as follows:
1. Commit() is called, resulting in a BeginCommitTransaction() call on
the TxnStatusManager.
2. An update is made to the transaction status table, marking the
transaction's state as COMMIT_IN_PROGRESS.
3. The commit tasks are initiated -- BEGIN_COMMIT ops are sent
asynchronously to every participant of the transaction.
4. Once all responses are received from the participants, a commit
timestamp is determined, and FINALIZE_COMMIT ops are sent
asynchronously to every participant.
5. Once all responses are received from the participants, an update is
made to the transaction status table, marking the transaction's state
as COMMITTED.
There are some nuances here around error handling. Namely, what do we do
if there are errors in sending the above requests? Well, it depends on
the error. Transient errors (i.e. timeouts) are simply retried. More
permanent errors need a bit more thought though:
- If a participant has been deleted, what do we do? This patch makes a
best effort attempt to abort the transaction if so.
- Any other kinds of errors (e.g. illegal state errors from a
participant) aren't expected in normal operation of a cluster. For
this, we stop the commit task and log a warning. Hopefully an operator
can intervene.
Some follow-ups to expect:
- This isn't as robust to failures as an approach that writes an
intermediate state to the TxnStatusManager in between steps 3 and 4. A
follow-up patch will implement that.
- A separate patch will implement aborting transactions.
- I disabled the background tasks in some tests that assume state
changes are entirely controlled by clients. A follow-up change will
address these to account for the state changes more organically.
Change-Id: Ie2258dded3ab3d527cb5d0abdc7d5e7deb4da15e
Reviewed-on: http://gerrit.cloudera.org:8080/16952
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
.../apache/kudu/client/TestKuduTransaction.java| 15 +-
src/kudu/client/client-test.cc | 69 +-
src/kudu/integration-tests/CMakeLists.txt | 1 +
src/kudu/integration-tests/txn_commit-itest.cc | 796 +
.../integration-tests/txn_status_manager-itest.cc | 5 +
.../integration-tests/txn_status_table-itest.cc| 4 +
src/kudu/master/txn_manager-test.cc| 5 +
src/kudu/tablet/tablet_replica.cc | 4 +
src/kudu/tablet/txn_coordinator.h | 9 +-
src/kudu/transactions/transactions.proto | 7 +-
src/kudu/transactions/txn_status_entry.cc | 1 -
src/kudu/transactions/txn_status_manager-test.cc | 32 +-
src/kudu/transactions/txn_status_manager.cc| 311 +++-
src/kudu/transactions/txn_status_manager.h | 163 -
src/kudu/transactions/txn_system_client.cc | 19 +
src/kudu/transactions/txn_system_client.h | 16 +-
src/kudu/tserver/ts_tablet_manager.cc | 19 +-
src/kudu/tserver/ts_tablet_manager.h | 3 +
18 files changed, 1366 insertions(+), 113 deletions(-)
diff --git
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
index 7e18dfa..37c7772 100644
---
a/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
+++
b/java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduTransaction.java
@@ -155,6 +155,9 @@ public class TestKuduTransaction {
@MasterServerConfig(flags = {
"--txn_manager_enabled",
})
+ @TabletServerConfig(flags = {
+ "--txn_schedule_background_tasks=false"
+ })
public void testCommitAnEmptyTransaction() throws Exception {
KuduTransaction txn = client.newTransaction();
txn.commit(false);
@@ -226,6 +229,9 @@ public class TestKuduTransaction {
@MasterServerConfig(flags = {
"--txn_manager_enabled",
})
+ @TabletServerConfig(flags = {
+ "--txn_schedule_background_tasks=false"
+ })
public void testIsCommitComplete() throws Exception {
KuduTransaction txn = client.newTransaction();
@@ -319,9 +325,6 @@ public class TestKuduTr
[kudu] branch master updated: KUDU-2612: loosen restrictions for BEGIN_COMMIT ops
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 8c5e655 KUDU-2612: loosen restrictions for BEGIN_COMMIT ops
8c5e655 is described below
commit 8c5e655d2d29e7758f26a373ea240631722ea1cf
Author: Andrew Wong
AuthorDate: Mon Jan 25 18:28:20 2021 -0800
KUDU-2612: loosen restrictions for BEGIN_COMMIT ops
This patch updates the state validations for the BEGIN_COMMIT. Rather
than only requiring an in-flight transaction to be kOpen (the common
case) or kCommitInProgress (if re-attempting a BEGIN_COMMIT call), it's
more robust to allow the call to be made even after the transaction has
already been finalized. This will allow a to-be-merged implementation of
a commit task to be retried with much less fuss.
Here's some pseudo-code for such a commit task:
1. persist COMMIT_IN_PROGRESS record on TxnStatusManager
2. foreach participant as p: BEGIN_COMMIT(p)
3. commit_ts = max timestamp used across BEGIN_COMMIT ops
4. foreach participant as p: FINALIZE_COMMIT(p, commit_ts)
5. persist COMMITTED record on TxnStatusManager
If the commit task is interrupted (e.g. by some crash) between steps 3
and 5, we may be left with some participants with fully finalized
commits. In such cases, all other participants _must_ also finalize, and
they must finalize with the same timestamp. To ensure this, it must be
possible to re-run a commit task. However, re-running it without this
patch may lead to issues because the BEGIN_COMMIT ops would yield
errors, complaining about an illegal state on participants that were
finalized.
This patch allows for a BEGIN_COMMIT op to succeed and return
immediately if a FINALIZE_COMMIT op has already completed. If so, the
finalized commit timestamp is sent back, allowing for the above commit
task to be repeatable.
Change-Id: Ifa4c5314190c84648c1b1edea7aab776b4882f97
Reviewed-on: http://gerrit.cloudera.org:8080/16992
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
.../integration-tests/txn_participant-itest.cc | 65 ++
src/kudu/tablet/tablet_metadata.cc | 10 +++-
src/kudu/tablet/tablet_metadata.h | 7 ++-
src/kudu/tablet/txn_participant-test.cc| 27 -
src/kudu/tablet/txn_participant.h | 33 ++-
5 files changed, 123 insertions(+), 19 deletions(-)
diff --git a/src/kudu/integration-tests/txn_participant-itest.cc
b/src/kudu/integration-tests/txn_participant-itest.cc
index fcbd083..b270bea 100644
--- a/src/kudu/integration-tests/txn_participant-itest.cc
+++ b/src/kudu/integration-tests/txn_participant-itest.cc
@@ -549,6 +549,49 @@ TEST_F(TxnParticipantITest, TestProxyBasicCalls) {
}
}
+TEST_F(TxnParticipantITest, TestBeginCommitAfterFinalize) {
+ constexpr const int kLeaderIdx = 0;
+ constexpr const int kTxnId = 0;
+ vector replicas = SetUpLeaderGetReplicas(kLeaderIdx);
+
ASSERT_OK(replicas[kLeaderIdx]->consensus()->WaitUntilLeaderForTests(kDefaultTimeout));
+ auto admin_proxy = cluster_->tserver_admin_proxy(kLeaderIdx);
+ const auto tablet_id = replicas[kLeaderIdx]->tablet_id();
+ {
+TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ASSERT_OK(ParticipateInTransactionCheckResp(
+admin_proxy.get(), tablet_id, kTxnId, ParticipantOpPB::BEGIN_TXN,
));
+ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ }
+ // Commit the transaction.
+ Timestamp begin_commit_ts;
+ {
+TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ASSERT_OK(ParticipateInTransactionCheckResp(
+admin_proxy.get(), tablet_id, kTxnId, ParticipantOpPB::BEGIN_COMMIT,
+, _commit_ts));
+ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ASSERT_NE(Timestamp::kInvalidTimestamp, begin_commit_ts);
+ }
+ {
+TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+ASSERT_OK(ParticipateInTransactionCheckResp(
+admin_proxy.get(), tablet_id, kTxnId,
ParticipantOpPB::FINALIZE_COMMIT, ));
+ASSERT_EQ(TabletServerErrorPB::UNKNOWN_ERROR, code);
+ }
+ // A call to BEGIN_COMMIT should yield the finalized commit timestamp.
+ Timestamp refetched_begin_commit_ts;
+ {
+TabletServerErrorPB::Code code = TabletServerErrorPB::UNKNOWN_ERROR;
+Status s = ParticipateInTransactionCheckResp(
+admin_proxy.get(), tablet_id, kTxnId, ParticipantOpPB::BEGIN_COMMIT,
,
+_begin_commit_ts);
+ASSERT_TRUE(s.IsIllegalState()) << s.ToString();
+ASSERT_EQ(TabletServerErrorPB::TXN_OP_ALREADY_APPLIED, code);
+ASSERT_NE(Timestamp::kInvalidTimestamp, refetched_begin_commit_ts);
+ASSERT_EQ(Timestamp(kDummyCommitTimestamp), refetch
[kudu] branch master updated: KUDU-2612: restrict TxnStatusManager calls to be made by the leader only
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new c033487 KUDU-2612: restrict TxnStatusManager calls to be made by the
leader only
c033487 is described below
commit c033487fd1765f9b3c5045d7c51719ed7e1c4418
Author: hahao
AuthorDate: Thu Jan 28 18:31:36 2021 -0800
KUDU-2612: restrict TxnStatusManager calls to be made by the leader only
Currently, even though a non-leader TxnStatusManager will be unable to
write to the underlying table (in the Raft subsystem the write would be
aborted), we may want to restrict calls to be made by the leader
TxnStatusManagers only. The motivation is to provide a more robust system,
which avoids cases when the request was sent to a laggy follower, we may
end up failing the request with an error.
This patch introduces ScopedLeaderSharedLock (similar to the one in Catalog
Manager) to be used to ensure the requests were sent to leaders only and
to block all other operations while reloading the persistent transaction
status metadata upon leadership changes. Note that during failover the
leader replica will wait until in-flight ops in the previous consensus
term to be applied before reloading the metadata.
Change-Id: I42c1ad095dcb4bdffcbe0ecf9631a60bac208c2a
Reviewed-on: http://gerrit.cloudera.org:8080/16648
Tested-by: Hao Hao
Reviewed-by: Andrew Wong
---
src/kudu/client/client-test.cc | 3 +
.../integration-tests/ts_tablet_manager-itest.cc | 15 ++
.../integration-tests/txn_status_manager-itest.cc | 23 +-
.../integration-tests/txn_status_table-itest.cc| 143 ++-
src/kudu/master/sys_catalog.cc | 1 +
src/kudu/tablet/tablet_replica-test-base.cc| 1 +
src/kudu/tablet/tablet_replica.cc | 122 +-
src/kudu/tablet/tablet_replica.h | 45
src/kudu/tablet/txn_coordinator.h | 6 +-
src/kudu/transactions/txn_status_manager-test.cc | 135 ++
src/kudu/transactions/txn_status_manager.cc| 271 +++--
src/kudu/transactions/txn_status_manager.h | 102 +++-
src/kudu/transactions/txn_status_tablet.h | 1 -
.../tserver/tablet_copy_source_session-test.cc | 3 +-
src/kudu/tserver/tablet_service.cc | 53 ++--
src/kudu/tserver/ts_tablet_manager.cc | 55 -
src/kudu/tserver/ts_tablet_manager.h | 3 +
17 files changed, 846 insertions(+), 136 deletions(-)
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index ca09c40..a09538a 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -106,6 +106,7 @@
#include "kudu/tablet/txn_coordinator.h"
#include "kudu/tablet/txn_participant-test-util.h"
#include "kudu/transactions/transactions.pb.h"
+#include "kudu/transactions/txn_status_manager.h"
#include "kudu/tserver/mini_tablet_server.h"
#include "kudu/tserver/scanners.h"
#include "kudu/tserver/tablet_server.h"
@@ -203,6 +204,7 @@ using kudu::rpc::MessengerBuilder;
using kudu::security::SignedTokenPB;
using kudu::client::sp::shared_ptr;
using kudu::tablet::TabletReplica;
+using kudu::transactions::TxnStatusManager;
using kudu::transactions::TxnTokenPB;
using kudu::tserver::MiniTabletServer;
using kudu::tserver::ParticipantOpPB;
@@ -421,6 +423,7 @@ class ClientTest : public KuduTest {
if (!c) {
continue;
}
+TxnStatusManager::ScopedLeaderSharedLock l(c);
auto highest_txn_id = c->highest_txn_id();
if (txn_id > highest_txn_id) {
continue;
diff --git a/src/kudu/integration-tests/ts_tablet_manager-itest.cc
b/src/kudu/integration-tests/ts_tablet_manager-itest.cc
index 2ffefe7..a6a0d50 100644
--- a/src/kudu/integration-tests/ts_tablet_manager-itest.cc
+++ b/src/kudu/integration-tests/ts_tablet_manager-itest.cc
@@ -68,6 +68,7 @@
#include "kudu/tablet/metadata.pb.h"
#include "kudu/tablet/tablet_replica.h"
#include "kudu/tablet/txn_coordinator.h"
+#include "kudu/transactions/txn_status_manager.h"
#include "kudu/transactions/txn_status_tablet.h"
#include "kudu/tserver/heartbeater.h"
#include "kudu/tserver/mini_tablet_server.h"
@@ -140,6 +141,7 @@ using kudu::rpc::RpcController;
using kudu::tablet::TabletReplica;
using kudu::tablet::ParticipantIdsByTxnId;
using kudu::tablet::TxnCoordinator;
+using kudu::transactions::TxnStatusManager;
using kudu::transactions::TxnStatusTablet;
using kudu::tserver::MiniTabletServer;
using kudu::ClusterVerifier;
@@ -1110,6 +1112,9 @@ class TxnStatusTabletManagem
[kudu] 02/02: [master] KUDU-2671: Range specific hashing at table creation time.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 23ab89db1a1546d93fc3848052d076a9026b3068
Author: Mahesh Reddy
AuthorDate: Wed Dec 9 12:23:26 2020 -0800
[master] KUDU-2671: Range specific hashing at table creation time.
This patch updates CreateTableRequestPB to allow different
hash schemas to be defined per range at table creation time.
This new field is appropriately decoded in catalog_manager.cc.
While this patch handles the logic for creating the correct
partitions, it does not update the metadata for either the
table or tablets. The new per-range schemas will need to be
added to the table metadata in a following patch.
The changes to kudu/common include some refactoring and
putting functions back into an anonymous namespace.
Change-Id: I8f0dcbc3324f8f2d6e99b4d169fdf5c7f7dff95d
Reviewed-on: http://gerrit.cloudera.org:8080/16859
Reviewed-by: Andrew Wong
Tested-by: Andrew Wong
---
src/kudu/common/partition-test.cc | 10 +-
src/kudu/common/partition.cc | 68 +++--
src/kudu/common/partition.h| 11 ++-
.../integration-tests/table_locations-itest.cc | 110 -
src/kudu/master/catalog_manager.cc | 17 +++-
src/kudu/master/master-test.cc | 73 --
src/kudu/master/master.proto | 12 +++
7 files changed, 254 insertions(+), 47 deletions(-)
diff --git a/src/kudu/common/partition-test.cc
b/src/kudu/common/partition-test.cc
index c5e0873..03670a4 100644
--- a/src/kudu/common/partition-test.cc
+++ b/src/kudu/common/partition-test.cc
@@ -985,7 +985,7 @@ TEST_F(PartitionTest, TestVaryingHashSchemasPerRange) {
{ ColumnId(0), ColumnId(1), ColumnId(2) }, 3);
PartitionSchemaPB schema_builder;
- // Table-wide HashSchema defined below, 3 by 2 buckets so 6 total.
+ // Table-wide hash schema defined below, 3 by 2 buckets so 6 total.
AddHashBucketComponent(_builder, { "a", "c" }, 3, 0);
AddHashBucketComponent(_builder, { "b" }, 2, 0);
PartitionSchema partition_schema;
@@ -1080,5 +1080,13 @@ TEST_F(PartitionTest, TestVaryingHashSchemasPerRange) {
ASSERT_EQ("Invalid argument: Both 'split_rows' and 'range_hash_schemas' "
"cannot be populated at the same time.", s.ToString());
+ // adding another schema to range_hash_schemas to trigger
Status::InvalidArgument due to
+ // 'bounds and 'range_hash_schema' not being the same size.
+ range_hash_schemas.emplace_back(PartitionSchema::HashBucketSchemas());
+ Status s1 = partition_schema.CreatePartitions({}, bounds, range_hash_schemas,
+ schema, );
+ ASSERT_EQ("Invalid argument: The number of range bounds does not match the
number of per "
+"range hash schemas.", s1.ToString());
+
}
} // namespace kudu
diff --git a/src/kudu/common/partition.cc b/src/kudu/common/partition.cc
index 04c6f93..6414e0c 100644
--- a/src/kudu/common/partition.cc
+++ b/src/kudu/common/partition.cc
@@ -111,21 +111,10 @@ void Partition::FromPB(const PartitionPB& pb, Partition*
partition) {
}
namespace {
-// Sets a repeated field of column identifiers to the provided column IDs.
-void SetColumnIdentifiers(const vector& column_ids,
-
RepeatedPtrField* identifiers) {
-identifiers->Reserve(column_ids.size());
-for (const ColumnId& column_id : column_ids) {
- identifiers->Add()->set_id(column_id);
-}
-}
-} // namespace
-
// Extracts the column IDs from a protobuf repeated field of column
identifiers.
-Status PartitionSchema::ExtractColumnIds(
-const RepeatedPtrField& identifiers,
-const Schema& schema,
-vector* column_ids) {
+Status ExtractColumnIds(const
RepeatedPtrField& identifiers,
+const Schema& schema,
+vector* column_ids) {
vector new_column_ids;
new_column_ids.reserve(identifiers.size());
for (const auto& identifier : identifiers) {
@@ -136,7 +125,7 @@ Status PartitionSchema::ExtractColumnIds(
return Status::InvalidArgument("unknown column id",
SecureDebugString(identifier));
}
new_column_ids.emplace_back(std::move(column_id));
- continue;
+continue;
}
case PartitionSchemaPB_ColumnIdentifierPB::kName: {
int32_t column_idx = schema.find_column(identifier.name());
@@ -144,7 +133,7 @@ Status PartitionSchema::ExtractColumnIds(
return Status::InvalidArgument("unknown column",
SecureDebugString(identifier));
}
new_column_ids.emplace_back(schema.column_id(column_idx));
- c
[kudu] 01/02: [docs] RPC cancellation is now implemented
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git commit 5b2287aa3c1a2858f918aec25054eb9ce523572e Author: Alexey Serbin AuthorDate: Sun Jan 3 18:27:30 2021 -0800 [docs] RPC cancellation is now implemented This patch updates the documentation on the RPC design in rpc.md, refreshing the note on the RPC cancellation. Since KUDU-2065 has been addressed, it's now possible to cancel a remote call using the RpcController::Cancel() method (follow up JIRAs KUDU-2334 and KUDU-2011 are still open, but the functionality for RPC cancellation is there). Change-Id: I335b3c0fd965896aff7e1625fd01ae564c050b1a Reviewed-on: http://gerrit.cloudera.org:8080/16918 Tested-by: Kudu Jenkins Reviewed-by: Grant Henke --- docs/design-docs/rpc.md | 16 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/design-docs/rpc.md b/docs/design-docs/rpc.md index 474962d..988b065 100644 --- a/docs/design-docs/rpc.md +++ b/docs/design-docs/rpc.md @@ -93,12 +93,12 @@ currently this is just timeout functionality, but in the future may include other call properties such as tracing information, priority classes, deadline propagation, etc. -Upon issuing the asynchronous request, the RPC layer enqueues the call to be sent -to the server and immediately returns. During this period, the caller thread -may continue to send other RPCs or perform other processing while waiting for -the callback to be triggered. In the future, we will provide an RPC cancellation -function on the RpcController object in case the user determines that the call -is no longer required. +Upon issuing the asynchronous request, the RPC layer enqueues the call to be +sent to the server and immediately returns. During this period, the caller +thread may continue to send other RPCs or perform other processing while +waiting for the callback to be triggered. In addition, it's possible to cancel +the RPC using the RpcController::Cancel() method if the user determines that +the call is no longer required. When the call completes, the RPC layer will invoke the provided ResponseCallback function from within the context of the reactor thread. Given this, @@ -130,8 +130,8 @@ The generated ServiceIf class contains pure virtual methods for each of the RPCs in the service. Each method to be implemented has an API like: ``` - void MethodName(const RequestPB *req, - ResponsePB *resp, ::kudu::rpc::RpcContext *context); + void MethodName( + const RequestPB* req, ResponsePB* resp, ::kudu::rpc::RpcContext* context); ``` The request PB is the user-provided request, and the response PB is a cleared
[kudu] branch master updated (2884df1 -> 23ab89d)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git. from 2884df1 [thirpdarty] Upgrade to gperftools 2.8.1 new 5b2287a [docs] RPC cancellation is now implemented new 23ab89d [master] KUDU-2671: Range specific hashing at table creation time. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/design-docs/rpc.md| 16 +-- src/kudu/common/partition-test.cc | 10 +- src/kudu/common/partition.cc | 68 +++-- src/kudu/common/partition.h| 11 ++- .../integration-tests/table_locations-itest.cc | 110 - src/kudu/master/catalog_manager.cc | 17 +++- src/kudu/master/master-test.cc | 73 -- src/kudu/master/master.proto | 12 +++ 8 files changed, 262 insertions(+), 55 deletions(-)
[kudu] branch master updated: KUDU-2612: add RPC to send participant ops
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new cece57b KUDU-2612: add RPC to send participant ops
cece57b is described below
commit cece57bcdf5d8d8ce596a3c1da4c2c26472784bc
Author: Andrew Wong
AuthorDate: Mon Nov 30 22:03:40 2020 -0800
KUDU-2612: add RPC to send participant ops
This adds methods to the TxnSystemClient to send participant ops to
participants by their tablet ID. This will be used in steps 13 and
18 of the transactions write path[1].
The new ParticipantRpc abstraction borrows a lot from CoordinatorRpc
with regards to lookups and error handling, with the following
differences:
- Rather than doing the lookup by table and partition key, it performs a
lookup by tablet ID, using the functionality recently added to the
MetaCache.
- Since TxnParticipants don't return success on repeated participant op
requests calls, some additional handling is done for the
TXN_OP_ALREADY_APPLIED error code.
[1]
https://docs.google.com/document/d/1qv7Zejpfzg-HvF5azRL49g5lRLQ4437EmJ53GiupcWQ/edit#heading=h.4lm41o75ev1x
Change-Id: Ibb9ba09104761772f9aaffe582776ad34d8dbf57
Reviewed-on: http://gerrit.cloudera.org:8080/16879
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/client/client.h | 2 +
src/kudu/client/meta_cache.cc | 33 +-
.../integration-tests/txn_participant-itest.cc | 384 -
src/kudu/tablet/txn_participant-test-util.h| 29 +-
src/kudu/transactions/CMakeLists.txt | 1 +
src/kudu/transactions/participant_rpc.cc | 239 +
src/kudu/transactions/participant_rpc.h| 91 +
src/kudu/transactions/txn_system_client.cc | 46 +++
src/kudu/transactions/txn_system_client.h | 17 +
9 files changed, 806 insertions(+), 36 deletions(-)
diff --git a/src/kudu/client/client.h b/src/kudu/client/client.h
index b1ab428..504c5c3 100644
--- a/src/kudu/client/client.h
+++ b/src/kudu/client/client.h
@@ -73,6 +73,7 @@ class FuzzTest;
namespace transactions {
class CoordinatorRpc;
+class ParticipantRpc;
class TxnSystemClient;
} // namespace transactions
@@ -892,6 +893,7 @@ class KUDU_EXPORT KuduClient : public
sp::enable_shared_from_this {
friend class kudu::AuthzTokenTest;
friend class kudu::SecurityUnknownTskTest;
friend class transactions::CoordinatorRpc;
+ friend class transactions::ParticipantRpc;
friend class transactions::TxnSystemClient;
friend class tools::LeaderMasterProxy;
friend class tools::RemoteKsckCluster;
diff --git a/src/kudu/client/meta_cache.cc b/src/kudu/client/meta_cache.cc
index 1ab1a11..f296b4c 100644
--- a/src/kudu/client/meta_cache.cc
+++ b/src/kudu/client/meta_cache.cc
@@ -502,18 +502,29 @@ void MetaCacheServerPicker::PickLeader(const
ServerPickedCallback& callback,
// Put another way, we don't care about the lookup results at all; we're
// just using it to fetch the latest consensus configuration information.
//
- // TODO: When we support tablet splits, we should let the lookup shift
- // the write to another tablet (i.e. if it's since been split).
+ // TODO(dralves): When we support tablet splits, we should let the lookup
+ // shift the write to another tablet (i.e. if it's since been split).
if (!leader) {
-meta_cache_->LookupTabletByKey(
-table_,
-tablet_->partition().partition_key_start(),
-deadline,
-MetaCache::LookupType::kPoint,
-nullptr,
-[this, callback, deadline](const Status& s) {
- this->LookUpTabletCb(callback, deadline, s);
-});
+if (table_) {
+ meta_cache_->LookupTabletByKey(
+ table_,
+ tablet_->partition().partition_key_start(),
+ deadline,
+ MetaCache::LookupType::kPoint,
+ /*remote_tablet*/nullptr,
+ [this, callback, deadline](const Status& s) {
+this->LookUpTabletCb(callback, deadline, s);
+ });
+} else {
+ meta_cache_->LookupTabletById(
+ client_,
+ tablet_->tablet_id(),
+ deadline,
+ /*remote_tablet*/nullptr,
+ [this, callback, deadline](const Status& s) {
+this->LookUpTabletCb(callback, deadline, s);
+ });
+}
return;
}
diff --git a/src/kudu/integration-tests/txn_participant-itest.cc
b/src/kudu/integration-tests/txn_participant-itest.cc
index 63f884a..fcbd083 100644
--- a/src/kudu/integration-tests/txn_participant-itest.cc
+++ b/src/kudu/integration-tests/txn_participant-itest.cc
@@ -57,6 +57,7 @@
#include "kudu/tablet/tablet_replica.h"
#include "kudu/tablet/txn_partici
[kudu] branch master updated: rpc: fix non-retriable error when starting up or shutting down
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 7c2b003 rpc: fix non-retriable error when starting up or shutting down
7c2b003 is described below
commit 7c2b003b1cf2a812be5d4de8d469ece27b2f3fc9
Author: Andrew Wong
AuthorDate: Sun Dec 13 23:35:57 2020 -0800
rpc: fix non-retriable error when starting up or shutting down
I found in testing a later patch that when a server is starting up or
being destructed, we may run into non-retriable error messages like:
Bad status: Remote error: Failed to write to server:
ff6a8ff4d4bd456c99cc60b4ab35b6cc (127.0.0.1:60603): Service unavailable:
service kudu.tserver.TabletServerAdminService not registered on TabletServer
This is because during startup or destruction, there are points at which
RPC services may not be registered, during which time RPCs will run into
non-retriable errors like that posted.
This patch addresses this by replacing the 'closing_' flag in
rpc::Messenger with a state enum, allowing us to express windows during
which the Messenger may not have all services registered.
Change-Id: I04e2379de4cf632d257b93cd701d7c73cb2bbaed
Reviewed-on: http://gerrit.cloudera.org:8080/16876
Tested-by: Kudu Jenkins
Reviewed-by: Hao Hao
---
src/kudu/client/client-test.cc | 44 ++
src/kudu/rpc/messenger.cc | 24 +++
src/kudu/rpc/messenger.h | 29 +---
src/kudu/rpc/rpc-test.cc | 9 +
src/kudu/server/rpc_server.cc | 1 +
5 files changed, 96 insertions(+), 11 deletions(-)
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index ac49571..d1dca6d 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -3145,6 +3145,50 @@ TEST_F(ClientTest, TestWriteTimeout) {
}
}
+// Test that Write RPCs get properly retried through the duration of a restart.
+// This tests the narrow windows during startup and shutdown that RPC services
+// are unregistered, checking that any resuling "missing service" errors don't
+// get propogated to end users.
+TEST_F(ClientTest, TestWriteWhileRestarting) {
+ shared_ptr session = client_->NewSession();
+ ASSERT_OK(session->SetFlushMode(KuduSession::MANUAL_FLUSH));
+ Status writer_error;
+ int row_id = 1;
+
+ // Writes a row and checks for errors.
+ const auto& write_and_check_error = [&] {
+RETURN_NOT_OK(ApplyInsertToSession(session.get(), client_table_, row_id++,
1, "row"));
+RETURN_NOT_OK(session->Flush());
+// If we successfully flush, check for errors.
+vector errors;
+bool overflow;
+session->GetPendingErrors(, );
+CHECK(!overflow);
+if (PREDICT_FALSE(!errors.empty())) {
+ return errors[0]->status();
+}
+return Status::OK();
+ };
+
+ // Until we finish restarting, hit the tablet server with write requests.
+ CountDownLatch stop(1);
+ thread t([&] {
+while (writer_error.ok() && stop.count() == 1) {
+ writer_error = write_and_check_error();
+}
+ });
+ auto thread_joiner = MakeScopedCleanup([&] { t.join(); });
+ auto* ts = cluster_->mini_tablet_server(0);
+ ts->Shutdown();
+ ASSERT_OK(ts->Restart());
+ stop.CountDown();
+ thread_joiner.cancel();
+ t.join();
+
+ // The writer thread should have hit no issues.
+ ASSERT_OK(writer_error);
+}
+
TEST_F(ClientTest, TestFailedDnsResolution) {
shared_ptr session = client_->NewSession();
ASSERT_OK(session->SetFlushMode(KuduSession::MANUAL_FLUSH));
diff --git a/src/kudu/rpc/messenger.cc b/src/kudu/rpc/messenger.cc
index 784188f..f161faf 100644
--- a/src/kudu/rpc/messenger.cc
+++ b/src/kudu/rpc/messenger.cc
@@ -271,11 +271,11 @@ void Messenger::ShutdownInternal(ShutdownMode mode) {
RpcServicesMap services_to_release;
{
std::lock_guard guard(lock_);
-if (closing_) {
+if (state_ == kClosing) {
return;
}
VLOG(1) << "shutting down messenger " << name_;
-closing_ = true;
+state_ = kClosing;
services_to_release = std::move(rpc_services_);
pools_to_shutdown = std::move(acceptor_pools_);
@@ -330,6 +330,8 @@ Status Messenger::RegisterService(const string&
service_name,
const scoped_refptr& service) {
DCHECK(service);
std::lock_guard guard(lock_);
+ DCHECK_NE(kServicesUnregistered, state_);
+ DCHECK_NE(kClosing, state_);
if (InsertIfNotPresent(_services_, service_name, service)) {
return Status::OK();
} else {
@@ -342,6 +344,7 @@ void Messenger::UnregisterAllServices() {
{
std::lock_guard guard(lock_);
to_release = std::move(rpc_services_);
+state_ = kServi
[kudu] branch master updated: client: retry id-based lookups if there is no leader
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 59b3836 client: retry id-based lookups if there is no leader
59b3836 is described below
commit 59b3836a9a5b16fc68819a377cf8a5021a43b176
Author: Andrew Wong
AuthorDate: Sun Dec 13 23:33:54 2020 -0800
client: retry id-based lookups if there is no leader
In testing out a later patch, I stumbled on the fact that we may end
up never retrying a tablet lookup in cases where all cached replicas are
marked as followers. The correct behavior is to attempt another lookup
in hopes that the master has more up-to-date information about
leadership than the client.
This is only an issue with the new id-based MetaCache lookups. This
patch extends the id-based path with logic that was already in the
key-based lookup path that prevented this.
Along the way, I did some renaming for clarity.
Change-Id: If311f7e94a4fc2efa067476a4cef9ef5b378d981
Reviewed-on: http://gerrit.cloudera.org:8080/16875
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/client/client-test.cc | 42 +-
src/kudu/client/client.h | 1 +
src/kudu/client/meta_cache.cc | 12 +++-
src/kudu/client/meta_cache.h | 6 +++---
4 files changed, 52 insertions(+), 9 deletions(-)
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index c1087ce..ac49571 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -75,6 +75,7 @@
#include "kudu/common/timestamp.h"
#include "kudu/common/txn_id.h"
#include "kudu/common/wire_protocol.pb.h"
+#include "kudu/consensus/metadata.pb.h"
#include "kudu/gutil/atomicops.h"
#include "kudu/gutil/casts.h"
#include "kudu/gutil/integral_types.h"
@@ -123,9 +124,9 @@
#include "kudu/util/path_util.h"
#include "kudu/util/random.h"
#include "kudu/util/random_util.h"
+#include "kudu/util/scoped_cleanup.h"
#include "kudu/util/semaphore.h"
#include "kudu/util/slice.h"
-#include "kudu/util/scoped_cleanup.h"
#include "kudu/util/status.h"
#include "kudu/util/stopwatch.h"
#include "kudu/util/test_macros.h"
@@ -2469,6 +2470,45 @@ TEST_F(ClientTest, TestMetaCacheExpiryWithKeysAndIds) {
ASSERT_FALSE(entry.stale());
}
+// Test that if our cache entry indicates there is no leader, we will perform a
+// lookup and refresh our cache entry.
+TEST_F(ClientTest, TestMetaCacheLookupNoLeaders) {
+ auto& meta_cache = client_->data_->meta_cache_;
+ auto tablet_ids = cluster_->mini_tablet_server(0)->ListTablets();
+ ASSERT_FALSE(tablet_ids.empty());
+ const auto& tablet_id = tablet_ids[0];
+
+ // Populate the cache.
+ scoped_refptr rt;
+ ASSERT_OK(MetaCacheLookupById(tablet_id, ));
+ ASSERT_NE(nullptr, rt);
+
+ // Mark the cache entry's replicas as followers.
+ internal::MetaCacheEntry entry;
+ ASSERT_TRUE(meta_cache->LookupEntryByIdFastPath(tablet_id, ));
+ ASSERT_FALSE(entry.stale());
+ vector replicas;
+ auto remote_tablet = entry.tablet();
+ remote_tablet->GetRemoteReplicas();
+ for (auto& r : replicas) {
+remote_tablet->MarkTServerAsFollower(r.ts);
+ }
+ const auto has_leader = [&] {
+remote_tablet->GetRemoteReplicas();
+for (auto& r : replicas) {
+ if (r.role == consensus::RaftPeerPB::LEADER) {
+return true;
+ }
+}
+return false;
+ };
+ ASSERT_FALSE(has_leader());
+ // This should force a lookup RPC, rather than leaving our cache entries
+ // leaderless.
+ ASSERT_OK(MetaCacheLookupById(tablet_id, ));
+ ASSERT_TRUE(has_leader());
+}
+
TEST_F(ClientTest, TestGetTabletServerBlacklist) {
shared_ptr table;
NO_FATALS(CreateTable("blacklist",
diff --git a/src/kudu/client/client.h b/src/kudu/client/client.h
index 22d63c5..b1ab428 100644
--- a/src/kudu/client/client.h
+++ b/src/kudu/client/client.h
@@ -908,6 +908,7 @@ class KUDU_EXPORT KuduClient : public
sp::enable_shared_from_this {
FRIEND_TEST(ClientTest, TestMetaCacheExpiry);
FRIEND_TEST(ClientTest, TestMetaCacheExpiryById);
FRIEND_TEST(ClientTest, TestMetaCacheExpiryWithKeysAndIds);
+ FRIEND_TEST(ClientTest, TestMetaCacheLookupNoLeaders);
FRIEND_TEST(ClientTest, TestMetaCacheWithKeysAndIds);
FRIEND_TEST(ClientTest, TestNonCoveringRangePartitions);
FRIEND_TEST(ClientTest, TestRetrieveAuthzTokenInParallel);
diff --git a/src/kudu/client/meta_cache.cc b/src/kudu/client/meta_cache.cc
index adc2989..1ab1a11 100644
--- a/src/kudu/client/meta_cache.cc
+++ b/src/kudu/client/meta_cache.cc
@@ -453,7 +453,6 @@ void MetaCacheServerPicker::PickLeader(const
[kudu] branch master updated: client: allow tablet ID lookups from the MetaCache
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 2a55876 client: allow tablet ID lookups from the MetaCache
2a55876 is described below
commit 2a558768f8aa00068e72ccd1327081f07ba46b03
Author: Andrew Wong
AuthorDate: Sun Nov 29 22:41:36 2020 -0800
client: allow tablet ID lookups from the MetaCache
Having a client interface for getting a RemoteTablet asynchronously will
be useful when it comes time to send participant ops to specific tablet
IDs from the TxnSystemClient.
In addition to the existing key-indexed cache entries, this patch
extends the MetaCache to track a separate set of cache entries by tablet
ID. There is no relationship between the two sets of cache entries -- a
tablet may exist in one but not the other.
The new id-based cache entries are only used if performing an id-based
lookup in the MetaCache via its new internal LookupTabletById() call
that will be used in a later patch.
Change-Id: Ib2333add5c3ab8403c48e69d29c90f3aec0914b6
Reviewed-on: http://gerrit.cloudera.org:8080/16794
Tested-by: Andrew Wong
Reviewed-by: Alexey Serbin
---
src/kudu/client/client-test.cc | 199 ++-
src/kudu/client/client.h | 4 +
src/kudu/client/master_proxy_rpc.h | 10 +-
src/kudu/client/meta_cache.cc | 270 -
src/kudu/client/meta_cache.h | 76 +--
5 files changed, 508 insertions(+), 51 deletions(-)
diff --git a/src/kudu/client/client-test.cc b/src/kudu/client/client-test.cc
index 64bb791..57da237 100644
--- a/src/kudu/client/client-test.cc
+++ b/src/kudu/client/client-test.cc
@@ -145,6 +145,7 @@ DECLARE_bool(rpc_trace_negotiation);
DECLARE_bool(scanner_inject_service_unavailable_on_continue_scan);
DECLARE_bool(txn_manager_enabled);
DECLARE_bool(txn_manager_lazily_initialized);
+DECLARE_int32(client_tablet_locations_by_id_ttl_ms);
DECLARE_int32(flush_threshold_mb);
DECLARE_int32(flush_threshold_secs);
DECLARE_int32(heartbeat_interval_ms);
@@ -276,6 +277,19 @@ class ClientTest : public KuduTest {
return rt;
}
+ Status MetaCacheLookupById(
+ const string& tablet_id, scoped_refptr*
remote_tablet) {
+remote_tablet->reset();
+scoped_refptr rt;
+Synchronizer sync;
+client_->data_->meta_cache_->LookupTabletById(
+client_.get(), tablet_id, MonoTime::Max(), ,
+sync.AsStatusCallback());
+RETURN_NOT_OK(sync.Wait());
+*remote_tablet = std::move(rt);
+return Status::OK();
+ }
+
// Generate a set of split rows for tablets used in this test.
vector> GenerateSplitRows() {
vector> rows;
@@ -2208,7 +,6 @@ TEST_F(ClientTest, TestMinMaxRangeBounds) {
}
TEST_F(ClientTest, TestMetaCacheExpiry) {
- google::FlagSaver saver;
FLAGS_table_locations_ttl_ms = 25;
auto& meta_cache = client_->data_->meta_cache_;
@@ -2228,9 +2241,191 @@ TEST_F(ClientTest, TestMetaCacheExpiry) {
ASSERT_FALSE(meta_cache->LookupEntryByKeyFastPath(client_table_.get(), "",
));
// Force a lookup and ensure the entry is refreshed.
- CHECK_NOTNULL(MetaCacheLookup(client_table_.get(), "").get());
+ ASSERT_NE(nullptr, MetaCacheLookup(client_table_.get(), ""));
+ ASSERT_TRUE(entry.stale());
+ ASSERT_TRUE(meta_cache->LookupEntryByKeyFastPath(client_table_.get(), "",
));
+ ASSERT_FALSE(entry.stale());
+}
+
+TEST_F(ClientTest, TestBasicIdBasedLookup) {
+ auto tablet_ids = cluster_->mini_tablet_server(0)->ListTablets();
+ ASSERT_FALSE(tablet_ids.empty());
+ scoped_refptr rt;
+ for (const auto& tablet_id : tablet_ids) {
+ASSERT_OK(MetaCacheLookupById(tablet_id, ));
+ASSERT_TRUE(rt != nullptr);
+ASSERT_EQ(tablet_id, rt->tablet_id());
+ }
+ const auto& kDummyId = "dummy-tablet-id";
+ Status s = MetaCacheLookupById(kDummyId, );
+ ASSERT_TRUE(s.IsNotFound());
+ ASSERT_EQ(nullptr, rt);
+
+ auto& meta_cache = client_->data_->meta_cache_;
+ internal::MetaCacheEntry entry;
+ ASSERT_FALSE(meta_cache->LookupEntryByIdFastPath(kDummyId, ));
+}
+
+TEST_F(ClientTest, TestMetaCacheExpiryById) {
+ FLAGS_client_tablet_locations_by_id_ttl_ms = 25;
+ auto tablet_ids = cluster_->mini_tablet_server(0)->ListTablets();
+ ASSERT_FALSE(tablet_ids.empty());
+ const auto& tablet_id = tablet_ids[0];
+
+ auto& meta_cache = client_->data_->meta_cache_;
+ meta_cache->ClearCache();
+ {
+internal::MetaCacheEntry entry;
+ASSERT_FALSE(meta_cache->LookupEntryByIdFastPath(tablet_id, ));
+ASSERT_FALSE(entry.Initialized());
+ }
+ {
+scoped_refptr rt;
+ASSERT_OK(MetaCacheLookupById(tablet_id, ));
+ASSERT_NE(nullptr, rt);
+internal::Me
[kudu] branch master updated: mini-cluster: add a tserver admin proxy
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new 420b07e mini-cluster: add a tserver admin proxy
420b07e is described below
commit 420b07e6490e14f26107088bc1b09866b6d43bba
Author: Andrew Wong
AuthorDate: Wed Dec 2 12:14:17 2020 -0800
mini-cluster: add a tserver admin proxy
It can be more convenient to use cluster APIs rather than creating the
proxy via constructor.
Change-Id: Ie230ec40165ad5eea6b0ca19141707389b6175d0
Reviewed-on: http://gerrit.cloudera.org:8080/16813
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
---
src/kudu/integration-tests/ts_tablet_manager-itest.cc | 11 ---
src/kudu/mini-cluster/external_mini_cluster.cc| 9 +
src/kudu/mini-cluster/external_mini_cluster.h | 3 +++
src/kudu/mini-cluster/internal_mini_cluster.cc| 8
src/kudu/mini-cluster/internal_mini_cluster.h | 3 +++
src/kudu/mini-cluster/mini_cluster.h | 6 ++
6 files changed, 33 insertions(+), 7 deletions(-)
diff --git a/src/kudu/integration-tests/ts_tablet_manager-itest.cc
b/src/kudu/integration-tests/ts_tablet_manager-itest.cc
index aa28d48..ae1d6e9 100644
--- a/src/kudu/integration-tests/ts_tablet_manager-itest.cc
+++ b/src/kudu/integration-tests/ts_tablet_manager-itest.cc
@@ -1226,7 +1226,8 @@ TEST_F(TxnStatusTabletManagementTest,
TestTabletServerProxyCalls) {
FLAGS_superuser_acl = kSuperUser;
FLAGS_trusted_user_acl = kTrustedUser;
NO_FATALS(StartCluster({}));
- auto* ts = cluster_->mini_tablet_server(0);
+ constexpr const int kTServerIdx = 0;
+ auto* ts = cluster_->mini_tablet_server(kTServerIdx);
ASSERT_OK(CreateTxnStatusTablet(ts));
// Put together a sequence of ops that should succeed.
const vector kOpSequence = {
@@ -1241,9 +1242,7 @@ TEST_F(TxnStatusTabletManagementTest,
TestTabletServerProxyCalls) {
// the logged-in user (this user is the service user, since we just restarted
// the server).
const auto perform_ops = [&] (int64_t txn_id, const string& user, bool
expect_success) {
-unique_ptr admin_proxy(
-new TabletServerAdminServiceProxy(client_messenger_,
ts->bound_rpc_addr(),
- ts->bound_rpc_addr().host()));
+auto admin_proxy = cluster_->tserver_admin_proxy(kTServerIdx);
if (!user.empty()) {
rpc::UserCredentials user_creds;
user_creds.set_real_user(user);
@@ -1283,10 +1282,8 @@ TEST_F(TxnStatusTabletManagementTest,
TestTabletServerProxyCalls) {
TEST_F(TxnStatusTabletManagementTest, TestTabletServerProxyCallErrors) {
NO_FATALS(StartCluster({}));
auto* ts = cluster_->mini_tablet_server(0);
+ auto admin_proxy = cluster_->tserver_admin_proxy(0);
ASSERT_OK(CreateTxnStatusTablet(ts));
- unique_ptr admin_proxy(
- new TabletServerAdminServiceProxy(client_messenger_,
ts->bound_rpc_addr(),
-ts->bound_rpc_addr().host()));
// If the request is missing fields, it should be rejected.
{
CoordinateTransactionRequestPB req;
diff --git a/src/kudu/mini-cluster/external_mini_cluster.cc
b/src/kudu/mini-cluster/external_mini_cluster.cc
index d6bc748..7e09fd6 100644
--- a/src/kudu/mini-cluster/external_mini_cluster.cc
+++ b/src/kudu/mini-cluster/external_mini_cluster.cc
@@ -58,6 +58,7 @@
#include "kudu/tablet/metadata.pb.h"
#include "kudu/tablet/tablet.pb.h"
#include "kudu/tserver/tserver.pb.h"
+#include "kudu/tserver/tserver_admin.proxy.h"
#include "kudu/tserver/tserver_service.proxy.h"
#include "kudu/util/async_util.h"
#include "kudu/util/env.h"
@@ -86,6 +87,7 @@ using kudu::rpc::RpcController;
using kudu::server::ServerStatusPB;
using kudu::tserver::ListTabletsRequestPB;
using kudu::tserver::ListTabletsResponsePB;
+using kudu::tserver::TabletServerAdminServiceProxy;
using kudu::tserver::TabletServerServiceProxy;
using std::copy;
using std::pair;
@@ -892,6 +894,13 @@ std::shared_ptr
ExternalMiniCluster::tserver_proxy(int
return std::make_shared(messenger_, addr,
addr.host());
}
+std::shared_ptr
ExternalMiniCluster::tserver_admin_proxy(
+int idx) const {
+ CHECK_LT(idx, tablet_servers_.size());
+ const auto& addr = CHECK_NOTNULL(tablet_server(idx))->bound_rpc_addr();
+ return std::make_shared(messenger_, addr,
addr.host());
+}
+
Status ExternalMiniCluster::CreateClient(client::KuduClientBuilder* builder,
client::sp::shared_ptr* client) const {
client::KuduClientBuilder defaults;
diff --git a/src/kudu/mini-cluster/external_mini_cluster.h
b/src/kudu/mini-cluster/external_mini_cluster.h
index 1553f2f..4d1d776 100644
--- a/src/kudu/mini-cluster/ex
[kudu] branch branch-1.13.x updated: KUDU-3108: fix invalid memory accesses in merge iterator
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch branch-1.13.x in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/branch-1.13.x by this push: new 4502d36 KUDU-3108: fix invalid memory accesses in merge iterator 4502d36 is described below commit 4502d3651d40770bee51fd4704cf8e0a2fb1e0ea Author: Andrew Wong AuthorDate: Mon Nov 23 01:14:00 2020 -0800 KUDU-3108: fix invalid memory accesses in merge iterator The 'hotmaxes_' heap and 'hot_' heap in the MergeIterator are meant to represent the same set of iterator states currently deemed "hot" in the optimal merge algorithm[1]. They used different ordering constraints to allow constant time access to the smallest last row and the smallest next row across all hot states respectively. However, we were using pop() on both heaps when iterator states were no longer deemed hot, incorrectly expecting that the call would remove the same iterator state from both heaps. In the case that this pop() was followed by the destruction of the sub-iterator (e.g. if the iterator was fully exhausted), this would leave an iterator state in the heaps that pointed at destructed state: F1030 21:16:58.411253 40800 schema.h:706] Check failed: KeyEquals(*lhs.schema()) && KeyEquals(*rhs.schema()) *** Check failure stack trace: *** *** Aborted at 1604117818 (unix time) try "date -d @1604117818" if you are using GNU date *** PC: @ 0x7f701fcf11d7 __GI_raise *** SIGABRT (@0x11179efd) received by PID 40701 (TID 0x7f6ff0f47700) from PID 40701; stack trace: *** @ 0x7f7026a70370 (unknown) @ 0x7f701fcf11d7 __GI_raise @ 0x7f701fcf28c8 __GI_abort @ 0x7f70224377b9 google::logging_fail() @ 0x7f7022438f8d google::LogMessage::Fail() @ 0x7f702243aee3 google::LogMessage::SendToLog() @ 0x7f7022438ae9 google::LogMessage::Flush() @ 0x7f702243b86f google::LogMessageFatal::~LogMessageFatal() @ 0x7f702cc99fbc kudu::Schema::Compare<>() @ 0x7f7026167cfd kudu::MergeIterator::RefillHotHeap() @ 0x7f7026167357 kudu::MergeIterator::AdvanceAndReheap() @ 0x7f7026169617 kudu::MergeIterator::MaterializeOneRow() @ 0x7f70261688e9 kudu::MergeIterator::NextBlock() @ 0x7f702cbddd9b kudu::tablet::Tablet::Iterator::NextBlock() @ 0x7f70317bcab3 kudu::tserver::TabletServiceImpl::HandleContinueScanRequest() @ 0x7f70317bb857 kudu::tserver::TabletServiceImpl::HandleNewScanRequest() @ 0x7f70317b464e kudu::tserver::TabletServiceImpl::Scan() @ 0x7f702ddfd762 _ZZN4kudu7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS_12MetricEntityEERKS2_INS_3rpc13ResultTrackerEEENKUlPKN6google8protobuf7MessageEPSE_PNS7_10RpcContextEE4_clESG_SH_SJ_ @ 0x7f702de0064d _ZNSt17_Function_handlerIFvPKN6google8protobuf7MessageEPS2_PN4kudu3rpc10RpcContextEEZNS6_7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS6_12MetricEntityEERKSD_INS7_13ResultTrackerEEEUlS4_S5_S9_E4_E9_M_invokeERKSt9_Any_dataS4_S5_S9_ @ 0x7f702b4ddcc2 std::function<>::operator()() @ 0x7f702b4dd6ed kudu::rpc::GeneratedServiceIf::Handle() @ 0x7f702b4dfff8 kudu::rpc::ServicePool::RunThread() @ 0x7f702b4de8c5 _ZZN4kudu3rpc11ServicePool4InitEiENKUlvE_clEv @ 0x7f702b4e0337 _ZNSt17_Function_handlerIFvvEZN4kudu3rpc11ServicePool4InitEiEUlvE_E9_M_invokeERKSt9_Any_data @ 0x7f7033524b9c std::function<>::operator()() @ 0x7f70248227e0 kudu::Thread::SuperviseThread() @ 0x7f7026a68dc5 start_thread @ 0x7f701fdb376d __clone Aborted This patch removes the 'hotmaxes_' min heap in favor of a two-heap variant of the merge algorithm that Adar described to me that uses the last value in the top iterator in the hot heap. This is not as optimal in terms of merge window size, but is still correct and avoids this bug. I experimented with some other approaches, described below. I ran the same test used in 1567dec086 (generic_iterators-test TestMerge and TestMergeNonOverlapping with 1 rows per list), averaged over five runs: a: An iteration of this patch that used std::set for hotmaxes, a call to find() before calling Advance(), and a position-based erase() after. The find() allowed for an erase() call that did not rely at all on the comparator, which may have pointed at destructed state following the call to Advance(). b: An iteration of this patch that used std::set for hotmaxes, a call to a value-based erase() before calling Advance(). The call to erase() before Advance() ensured Advance() calls never interfered
[kudu] branch branch-1.12.x updated: KUDU-3108: fix invalid memory accesses in merge iterator
This is an automated email from the ASF dual-hosted git repository. awong pushed a commit to branch branch-1.12.x in repository https://gitbox.apache.org/repos/asf/kudu.git The following commit(s) were added to refs/heads/branch-1.12.x by this push: new 76855e4 KUDU-3108: fix invalid memory accesses in merge iterator 76855e4 is described below commit 76855e4cfab249de7eb492a6bce2658f277b9844 Author: Andrew Wong AuthorDate: Mon Nov 23 01:14:00 2020 -0800 KUDU-3108: fix invalid memory accesses in merge iterator CONFLICT: updated diff_scan-test to use MVCC's WaitForApplyingTransactionsToCommit(), which was renamed in later versions. The 'hotmaxes_' heap and 'hot_' heap in the MergeIterator are meant to represent the same set of iterator states currently deemed "hot" in the optimal merge algorithm[1]. They used different ordering constraints to allow constant time access to the smallest last row and the smallest next row across all hot states respectively. However, we were using pop() on both heaps when iterator states were no longer deemed hot, incorrectly expecting that the call would remove the same iterator state from both heaps. In the case that this pop() was followed by the destruction of the sub-iterator (e.g. if the iterator was fully exhausted), this would leave an iterator state in the heaps that pointed at destructed state: F1030 21:16:58.411253 40800 schema.h:706] Check failed: KeyEquals(*lhs.schema()) && KeyEquals(*rhs.schema()) *** Check failure stack trace: *** *** Aborted at 1604117818 (unix time) try "date -d @1604117818" if you are using GNU date *** PC: @ 0x7f701fcf11d7 __GI_raise *** SIGABRT (@0x11179efd) received by PID 40701 (TID 0x7f6ff0f47700) from PID 40701; stack trace: *** @ 0x7f7026a70370 (unknown) @ 0x7f701fcf11d7 __GI_raise @ 0x7f701fcf28c8 __GI_abort @ 0x7f70224377b9 google::logging_fail() @ 0x7f7022438f8d google::LogMessage::Fail() @ 0x7f702243aee3 google::LogMessage::SendToLog() @ 0x7f7022438ae9 google::LogMessage::Flush() @ 0x7f702243b86f google::LogMessageFatal::~LogMessageFatal() @ 0x7f702cc99fbc kudu::Schema::Compare<>() @ 0x7f7026167cfd kudu::MergeIterator::RefillHotHeap() @ 0x7f7026167357 kudu::MergeIterator::AdvanceAndReheap() @ 0x7f7026169617 kudu::MergeIterator::MaterializeOneRow() @ 0x7f70261688e9 kudu::MergeIterator::NextBlock() @ 0x7f702cbddd9b kudu::tablet::Tablet::Iterator::NextBlock() @ 0x7f70317bcab3 kudu::tserver::TabletServiceImpl::HandleContinueScanRequest() @ 0x7f70317bb857 kudu::tserver::TabletServiceImpl::HandleNewScanRequest() @ 0x7f70317b464e kudu::tserver::TabletServiceImpl::Scan() @ 0x7f702ddfd762 _ZZN4kudu7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS_12MetricEntityEERKS2_INS_3rpc13ResultTrackerEEENKUlPKN6google8protobuf7MessageEPSE_PNS7_10RpcContextEE4_clESG_SH_SJ_ @ 0x7f702de0064d _ZNSt17_Function_handlerIFvPKN6google8protobuf7MessageEPS2_PN4kudu3rpc10RpcContextEEZNS6_7tserver21TabletServerServiceIfC1ERK13scoped_refptrINS6_12MetricEntityEERKSD_INS7_13ResultTrackerEEEUlS4_S5_S9_E4_E9_M_invokeERKSt9_Any_dataS4_S5_S9_ @ 0x7f702b4ddcc2 std::function<>::operator()() @ 0x7f702b4dd6ed kudu::rpc::GeneratedServiceIf::Handle() @ 0x7f702b4dfff8 kudu::rpc::ServicePool::RunThread() @ 0x7f702b4de8c5 _ZZN4kudu3rpc11ServicePool4InitEiENKUlvE_clEv @ 0x7f702b4e0337 _ZNSt17_Function_handlerIFvvEZN4kudu3rpc11ServicePool4InitEiEUlvE_E9_M_invokeERKSt9_Any_data @ 0x7f7033524b9c std::function<>::operator()() @ 0x7f70248227e0 kudu::Thread::SuperviseThread() @ 0x7f7026a68dc5 start_thread @ 0x7f701fdb376d __clone Aborted This patch removes the 'hotmaxes_' min heap in favor of a two-heap variant of the merge algorithm that Adar described to me that uses the last value in the top iterator in the hot heap. This is not as optimal in terms of merge window size, but is still correct and avoids this bug. I experimented with some other approaches, described below. I ran the same test used in 1567dec086 (generic_iterators-test TestMerge and TestMergeNonOverlapping with 1 rows per list), averaged over five runs: a: An iteration of this patch that used std::set for hotmaxes, a call to find() before calling Advance(), and a position-based erase() after. The find() allowed for an erase() call that did not rely at all on the comparator, which may have pointed at destructed state following the call to Advance(). b: An iteration of this patch that used std::set for hotmaxes, a call t
[kudu] branch master updated: [test] Use /tmp for mini-chronyd sockets
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
The following commit(s) were added to refs/heads/master by this push:
new ef5b5ab [test] Use /tmp for mini-chronyd sockets
ef5b5ab is described below
commit ef5b5abdcefbc5b7d7edb7170d55b82843926b25
Author: Grant Henke
AuthorDate: Wed Dec 2 10:55:43 2020 -0600
[test] Use /tmp for mini-chronyd sockets
To ensure the path is less than 100 bytes this patch uses /tmp instead
of the test directory for chronyd sockets.
Change-Id: I567ff640fca5c56fef3f4400216fc8b23532713e
Reviewed-on: http://gerrit.cloudera.org:8080/16808
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
---
src/kudu/clock/test/mini_chronyd.cc | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/src/kudu/clock/test/mini_chronyd.cc
b/src/kudu/clock/test/mini_chronyd.cc
index 0d2cf78..5f4cf6d 100644
--- a/src/kudu/clock/test/mini_chronyd.cc
+++ b/src/kudu/clock/test/mini_chronyd.cc
@@ -422,10 +422,8 @@ $0
// The path to Unix domain socket file cannot be longer than ~100 bytes,
// so it's necessary to create a directory with shorter absolute path.
// TODO(aserbin): use some synthetic mount point instead?
-string dir;
-RETURN_NOT_OK(Env::Default()->GetTestDirectory());
-dir = JoinPathSegments(dir, Substitute("$0.$1",
Env::Default()->NowMicros(),
- getpid()));
+string dir = JoinPathSegments("/tmp", Substitute("$0.$1",
+Env::Default()->NowMicros(), getpid()));
const auto s = Env::Default()->CreateDir(dir);
if (!s.ok() && !s.IsAlreadyPresent()) {
return s;
@@ -433,6 +431,7 @@ $0
RETURN_NOT_OK(CorrectOwnership(dir));
options_.bindcmdaddress = Substitute("$0/chronyd.$1.sock",
dir, options_.index);
+// Set cmd_socket_dir_ so we can cleanup the unix domain sockets.
cmd_socket_dir_ = std::move(dir);
}
string username;
[kudu] 03/03: [test] Use /tmp for mini-chronyd sockets
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 2acc4d66d5fefff42f532b0e10899c66920c46f0
Author: Grant Henke
AuthorDate: Wed Dec 2 10:55:43 2020 -0600
[test] Use /tmp for mini-chronyd sockets
To ensure the path is less than 100 bytes this patch uses /tmp instead
of the test directory for chronyd sockets.
Change-Id: I567ff640fca5c56fef3f4400216fc8b23532713e
Reviewed-on: http://gerrit.cloudera.org:8080/16809
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/clock/test/mini_chronyd.cc | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/src/kudu/clock/test/mini_chronyd.cc
b/src/kudu/clock/test/mini_chronyd.cc
index 0d2cf78..5f4cf6d 100644
--- a/src/kudu/clock/test/mini_chronyd.cc
+++ b/src/kudu/clock/test/mini_chronyd.cc
@@ -422,10 +422,8 @@ $0
// The path to Unix domain socket file cannot be longer than ~100 bytes,
// so it's necessary to create a directory with shorter absolute path.
// TODO(aserbin): use some synthetic mount point instead?
-string dir;
-RETURN_NOT_OK(Env::Default()->GetTestDirectory());
-dir = JoinPathSegments(dir, Substitute("$0.$1",
Env::Default()->NowMicros(),
- getpid()));
+string dir = JoinPathSegments("/tmp", Substitute("$0.$1",
+Env::Default()->NowMicros(), getpid()));
const auto s = Env::Default()->CreateDir(dir);
if (!s.ok() && !s.IsAlreadyPresent()) {
return s;
@@ -433,6 +431,7 @@ $0
RETURN_NOT_OK(CorrectOwnership(dir));
options_.bindcmdaddress = Substitute("$0/chronyd.$1.sock",
dir, options_.index);
+// Set cmd_socket_dir_ so we can cleanup the unix domain sockets.
cmd_socket_dir_ = std::move(dir);
}
string username;
[kudu] branch branch-1.13.x updated (07eb5a9 -> 2acc4d6)
This is an automated email from the ASF dual-hosted git repository. awong pushed a change to branch branch-1.13.x in repository https://gitbox.apache.org/repos/asf/kudu.git. from 07eb5a9 [kudu-tool-test] fix ClusterNameResolverFileCorrupt with glibc 2.31 new 5a66fff [dist_test] Ship security libraries to dist_test new 1e630db KUDU-2700: Add a LSAN suppression new 2acc4d6 [test] Use /tmp for mini-chronyd sockets The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build-support/dist_test.py | 63 +++-- build-support/run_dist_test.py | 6 cmake_modules/FindCyrusSASL.cmake | 2 +- src/kudu/clock/test/mini_chronyd.cc | 7 ++--- src/kudu/util/sanitizer_options.cc | 4 +++ 5 files changed, 75 insertions(+), 7 deletions(-)
[kudu] 02/03: KUDU-2700: Add a LSAN suppression
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 1e630db0348c44793f55513eddd23e8f4ab9b2c4
Author: Grant Henke
AuthorDate: Tue Nov 24 11:18:56 2020 -0600
KUDU-2700: Add a LSAN suppression
OpenSSL 1.1 has a leak in libcrypto.so when running tests that use the CLI
to connect to a remote server. Previously this didn’t impact CI runs
because Jenkins and dist_test were using Ubuntu 14 and therefore
OpenSSL 1.0. Now that we are using Ubuntu 18 and OpenSSL 1.1
this leak is reproducable 100% of the time.
This patch suppresses the leak to unbreak the ASAN builds.
Change-Id: I5d1378a820997d3eabfa7e5cda53c07dd4df917d
Reviewed-on: http://gerrit.cloudera.org:8080/16781
Reviewed-by: Andrew Wong
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/16804
Reviewed-by: Grant Henke
Tested-by: Andrew Wong
---
src/kudu/util/sanitizer_options.cc | 4
1 file changed, 4 insertions(+)
diff --git a/src/kudu/util/sanitizer_options.cc
b/src/kudu/util/sanitizer_options.cc
index 8c8c0d3..bd77459 100644
--- a/src/kudu/util/sanitizer_options.cc
+++ b/src/kudu/util/sanitizer_options.cc
@@ -194,6 +194,10 @@ SANITIZER_HOOK_ATTRIBUTE const char
*__lsan_default_suppressions() {
// Fixed by upstream commit 379d39c17b8930718e98185a5b32a0f7f3e3b4b6
"leak:krb5_authdata_import_attributes\n"
+ // KUDU-2700: OpenSSL 1.1 has a leak in libcrypto.so when running tests
+ // that use the CLI to connect to a remote server.
+ "leak:libcrypto.so.1.1\n"
+
// KUDU-2653: Memory leak in libgssapi_krb5 [1]. Exists in certain patched
// versions of krb5-1.12 (such as krb5 in Debian 8).
//
[kudu] 01/03: [dist_test] Ship security libraries to dist_test
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 5a66fffcdcfbb97ea471d6db05d61dcf4b1b7c22
Author: Grant Henke
AuthorDate: Thu Nov 12 19:05:11 2020 -0600
[dist_test] Ship security libraries to dist_test
This patch adjusts the lib whitelist to allow shipping the security
libraries
to dist_test and allow more flexibility when versions do not match the
dist_test images versions.
This was already happening for rhel6 installs due to the rhel6 workaround
script
linked below. With this change the libraries will be shipped even when not
in thirdparty.
https://github.com/apache/kudu/blob/master/thirdparty/install-openssl-el6-workaround.sh
I also needed to adjust run_dist_test.py in order to set the SASL_PATH
environment variable if SASL modules are present. Otherwise the
system modules were still used.
Change-Id: Id10afab6e9c48b9ffcf0da905993c7f2a1e606a6
Reviewed-on: http://gerrit.cloudera.org:8080/16716
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
Reviewed-by: Alexey Serbin
(cherry picked from commit fab3a38d1da0d5d13d8e9c91fd306e03b65da4e6)
Reviewed-on: http://gerrit.cloudera.org:8080/16802
Reviewed-by: Grant Henke
Tested-by: Andrew Wong
---
build-support/dist_test.py| 63 +--
build-support/run_dist_test.py| 6
cmake_modules/FindCyrusSASL.cmake | 2 +-
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/build-support/dist_test.py b/build-support/dist_test.py
index 1696ea8..89e53bf 100755
--- a/build-support/dist_test.py
+++ b/build-support/dist_test.py
@@ -240,6 +240,11 @@ def get_test_executions(tests_regex, extra_args=None):
def is_lib_whitelisted(lib):
# No need to ship things like libc, libstdcxx, etc.
if lib.startswith("/lib") or lib.startswith("/usr"):
+# Ship the dynamically linked security libraries from
+# OpenSSL and Cyrus SASL to better support submitting
+# installed versions different from the dist_test image.
+if "libcrypto" in lib or "libsasl2" in lib or "libssl" in lib:
+ return True
return False
return True
@@ -262,8 +267,41 @@ def get_base_deps(dep_extractor):
# of the test executable. We must include those dependencies in the archive
# for the binaries to be usable.
deps.extend(dep_extractor.extract_deps(d))
+
+ add_sasl_module_deps(deps)
return deps
+def add_sasl_module_deps(deps):
+ """
+ The SASL module dependencies are used at runtime but are not discovered
+ via ldd in the dep_extractor. This method finds the sasl2 directory
+ relative to the libsasl2 library and adds all the libraries in that
+ directory.
+ """
+ # Find the libsasl2 module in the dependencies.
+ sasl_lib = None
+ for dep in deps:
+if "libsasl2" in dep:
+ sasl_lib = dep
+ break
+
+ # Look for libplain in potential sasl2 module paths, which is required for
+ # Kudu's basic operation.
+ sasl_path = None
+ if sasl_lib:
+path = os.path.join(os.path.dirname(sasl_lib), "sasl2")
+if os.path.exists(path):
+ children = os.listdir(path)
+ for child in children:
+if "libplain" in child:
+ sasl_path = path
+ break
+
+ if sasl_path:
+for dirpath, subdirs, files in os.walk(sasl_path):
+ for f in files:
+dep = os.path.join(dirpath, f)
+deps.append(dep)
def is_outside_of_tree(path):
repo_dir = rel_to_abs("./")
@@ -282,7 +320,18 @@ def copy_system_library(lib):
sys_lib_dir = rel_to_abs("build/dist-test-system-libs")
if not os.path.exists(sys_lib_dir):
os.makedirs(sys_lib_dir)
- dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
+ sasl_dir = os.path.join(sys_lib_dir, "sasl2")
+ if not os.path.exists(sasl_dir):
+os.makedirs(sasl_dir)
+
+ # If the library is a SASL module keep it in its own directory so
+ # we can set the SASL_PATH environment variable in run_dist_test.py.
+ if "/sasl2/" in lib:
+dst = os.path.join(sasl_dir, os.path.basename(lib))
+ else:
+dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
# Copy if it doesn't exist, or the mtimes don't match.
# Using shutil.copy2 preserves the mtime after the copy (like cp -p)
if not os.path.exists(dst) or os.stat(dst).st_mtime != os.stat(lib).st_mtime:
@@ -664,7 +713,17 @@ def add_java_subparser(subparsers):
loop.set_defaults(func=loop_java_test)
def dump_base_deps(parser, options):
- print(json.dumps(get_base_deps(create_dependency_extractor(
+ deps = get_base_deps(create_dependency_extractor())
+ relocated_deps = []
+ # Deduplicate dependencies included via DEPS_FOR_ALL.
+ for d in set(de
[kudu] 01/03: [dist_test] Ship security libraries to dist_test
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.12.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e011d26fcb161d449ecb075f997138c314ffe0fe
Author: Grant Henke
AuthorDate: Thu Nov 12 19:05:11 2020 -0600
[dist_test] Ship security libraries to dist_test
This patch adjusts the lib whitelist to allow shipping the security
libraries
to dist_test and allow more flexibility when versions do not match the
dist_test images versions.
This was already happening for rhel6 installs due to the rhel6 workaround
script
linked below. With this change the libraries will be shipped even when not
in thirdparty.
https://github.com/apache/kudu/blob/master/thirdparty/install-openssl-el6-workaround.sh
I also needed to adjust run_dist_test.py in order to set the SASL_PATH
environment variable if SASL modules are present. Otherwise the
system modules were still used.
Change-Id: Id10afab6e9c48b9ffcf0da905993c7f2a1e606a6
Reviewed-on: http://gerrit.cloudera.org:8080/16716
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
Reviewed-by: Alexey Serbin
(cherry picked from commit fab3a38d1da0d5d13d8e9c91fd306e03b65da4e6)
Reviewed-on: http://gerrit.cloudera.org:8080/16801
Reviewed-by: Grant Henke
Tested-by: Andrew Wong
---
build-support/dist_test.py| 63 +--
build-support/run_dist_test.py| 6
cmake_modules/FindCyrusSASL.cmake | 2 +-
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/build-support/dist_test.py b/build-support/dist_test.py
index aa20d70..43cc537 100755
--- a/build-support/dist_test.py
+++ b/build-support/dist_test.py
@@ -241,6 +241,11 @@ def get_test_executions(tests_regex, extra_args=None):
def is_lib_whitelisted(lib):
# No need to ship things like libc, libstdcxx, etc.
if lib.startswith("/lib") or lib.startswith("/usr"):
+# Ship the dynamically linked security libraries from
+# OpenSSL and Cyrus SASL to better support submitting
+# installed versions different from the dist_test image.
+if "libcrypto" in lib or "libsasl2" in lib or "libssl" in lib:
+ return True
return False
return True
@@ -263,8 +268,41 @@ def get_base_deps(dep_extractor):
# of the test executable. We must include those dependencies in the archive
# for the binaries to be usable.
deps.extend(dep_extractor.extract_deps(d))
+
+ add_sasl_module_deps(deps)
return deps
+def add_sasl_module_deps(deps):
+ """
+ The SASL module dependencies are used at runtime but are not discovered
+ via ldd in the dep_extractor. This method finds the sasl2 directory
+ relative to the libsasl2 library and adds all the libraries in that
+ directory.
+ """
+ # Find the libsasl2 module in the dependencies.
+ sasl_lib = None
+ for dep in deps:
+if "libsasl2" in dep:
+ sasl_lib = dep
+ break
+
+ # Look for libplain in potential sasl2 module paths, which is required for
+ # Kudu's basic operation.
+ sasl_path = None
+ if sasl_lib:
+path = os.path.join(os.path.dirname(sasl_lib), "sasl2")
+if os.path.exists(path):
+ children = os.listdir(path)
+ for child in children:
+if "libplain" in child:
+ sasl_path = path
+ break
+
+ if sasl_path:
+for dirpath, subdirs, files in os.walk(sasl_path):
+ for f in files:
+dep = os.path.join(dirpath, f)
+deps.append(dep)
def is_outside_of_tree(path):
repo_dir = rel_to_abs("./")
@@ -283,7 +321,18 @@ def copy_system_library(lib):
sys_lib_dir = rel_to_abs("build/dist-test-system-libs")
if not os.path.exists(sys_lib_dir):
os.makedirs(sys_lib_dir)
- dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
+ sasl_dir = os.path.join(sys_lib_dir, "sasl2")
+ if not os.path.exists(sasl_dir):
+os.makedirs(sasl_dir)
+
+ # If the library is a SASL module keep it in its own directory so
+ # we can set the SASL_PATH environment variable in run_dist_test.py.
+ if "/sasl2/" in lib:
+dst = os.path.join(sasl_dir, os.path.basename(lib))
+ else:
+dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
# Copy if it doesn't exist, or the mtimes don't match.
# Using shutil.copy2 preserves the mtime after the copy (like cp -p)
if not os.path.exists(dst) or os.stat(dst).st_mtime != os.stat(lib).st_mtime:
@@ -665,7 +714,17 @@ def add_java_subparser(subparsers):
loop.set_defaults(func=loop_java_test)
def dump_base_deps(parser, options):
- print(json.dumps(get_base_deps(create_dependency_extractor(
+ deps = get_base_deps(create_dependency_extractor())
+ relocated_deps = []
+ # Deduplicate dependencies included via DEPS_FOR_ALL.
+ for d in set(de
[kudu] 03/03: [test] Use /tmp for mini-chronyd sockets
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.12.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit ec44e4d3e205eb385f22a56b04a891ffd0770131
Author: Grant Henke
AuthorDate: Wed Dec 2 10:55:43 2020 -0600
[test] Use /tmp for mini-chronyd sockets
To ensure the path is less than 100 bytes this patch uses /tmp instead
of the test directory for chronyd sockets.
Change-Id: I567ff640fca5c56fef3f4400216fc8b23532713e
Reviewed-on: http://gerrit.cloudera.org:8080/16808
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong
(cherry picked from commit ef5b5abdcefbc5b7d7edb7170d55b82843926b25)
Reviewed-on: http://gerrit.cloudera.org:8080/16812
Reviewed-by: Alexey Serbin
Tested-by: Andrew Wong
---
src/kudu/clock/test/mini_chronyd.cc | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/src/kudu/clock/test/mini_chronyd.cc
b/src/kudu/clock/test/mini_chronyd.cc
index 0d2cf78..5f4cf6d 100644
--- a/src/kudu/clock/test/mini_chronyd.cc
+++ b/src/kudu/clock/test/mini_chronyd.cc
@@ -422,10 +422,8 @@ $0
// The path to Unix domain socket file cannot be longer than ~100 bytes,
// so it's necessary to create a directory with shorter absolute path.
// TODO(aserbin): use some synthetic mount point instead?
-string dir;
-RETURN_NOT_OK(Env::Default()->GetTestDirectory());
-dir = JoinPathSegments(dir, Substitute("$0.$1",
Env::Default()->NowMicros(),
- getpid()));
+string dir = JoinPathSegments("/tmp", Substitute("$0.$1",
+Env::Default()->NowMicros(), getpid()));
const auto s = Env::Default()->CreateDir(dir);
if (!s.ok() && !s.IsAlreadyPresent()) {
return s;
@@ -433,6 +431,7 @@ $0
RETURN_NOT_OK(CorrectOwnership(dir));
options_.bindcmdaddress = Substitute("$0/chronyd.$1.sock",
dir, options_.index);
+// Set cmd_socket_dir_ so we can cleanup the unix domain sockets.
cmd_socket_dir_ = std::move(dir);
}
string username;
[kudu] 02/03: KUDU-2700: Add a LSAN suppression
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.12.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 8c70df0ff70a962ece6be83ea09d712deff1ee8f
Author: Grant Henke
AuthorDate: Tue Nov 24 11:18:56 2020 -0600
KUDU-2700: Add a LSAN suppression
OpenSSL 1.1 has a leak in libcrypto.so when running tests that use the CLI
to connect to a remote server. Previously this didn’t impact CI runs
because Jenkins and dist_test were using Ubuntu 14 and therefore
OpenSSL 1.0. Now that we are using Ubuntu 18 and OpenSSL 1.1
this leak is reproducable 100% of the time.
This patch suppresses the leak to unbreak the ASAN builds.
Change-Id: I5d1378a820997d3eabfa7e5cda53c07dd4df917d
Reviewed-on: http://gerrit.cloudera.org:8080/16781
Reviewed-by: Andrew Wong
Reviewed-by: Alexey Serbin
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/16803
Reviewed-by: Grant Henke
Tested-by: Andrew Wong
---
src/kudu/util/sanitizer_options.cc | 4
1 file changed, 4 insertions(+)
diff --git a/src/kudu/util/sanitizer_options.cc
b/src/kudu/util/sanitizer_options.cc
index 8c8c0d3..bd77459 100644
--- a/src/kudu/util/sanitizer_options.cc
+++ b/src/kudu/util/sanitizer_options.cc
@@ -194,6 +194,10 @@ SANITIZER_HOOK_ATTRIBUTE const char
*__lsan_default_suppressions() {
// Fixed by upstream commit 379d39c17b8930718e98185a5b32a0f7f3e3b4b6
"leak:krb5_authdata_import_attributes\n"
+ // KUDU-2700: OpenSSL 1.1 has a leak in libcrypto.so when running tests
+ // that use the CLI to connect to a remote server.
+ "leak:libcrypto.so.1.1\n"
+
// KUDU-2653: Memory leak in libgssapi_krb5 [1]. Exists in certain patched
// versions of krb5-1.12 (such as krb5 in Debian 8).
//
