[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add variable uds_address in NetworkAddressPB and TNetworkAddress. Replace TNetworkAddress with NetworkAddressPB for KRPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. - BackendId of admissiond is not available. Use admissiond's IP address as unique ID for UDS. TODO: Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Reviewed-on: http://gerrit.cloudera.org:8080/18369 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto M common/thrift/Types.thrift A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 39 files changed, 597 insertions(+), 161 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 12 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 May 2022 06:27:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8157/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 May 2022 01:49:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 11 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 May 2022 01:49:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 10: Code-Review+2 Thanks, this looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 May 2022 00:48:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10635/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 24 May 2022 19:29:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add variable uds_address in NetworkAddressPB and TNetworkAddress. Replace TNetworkAddress with NetworkAddressPB for KRPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. - BackendId of admissiond is not available. Use admissiond's IP address as unique ID for UDS. TODO: Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto M common/thrift/Types.thrift A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 39 files changed, 597 insertions(+), 161 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/10 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 10 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
..
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else if (strcasecmp("ip_address",
FLAGS_uds_address_unique_id.c_str()) == 0) {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> I see that you disable UDS if the value is invalid.
fixed.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in
"Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:". This
field is optional.
: // It is only used for KRPC bind/listen/connect when
FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> My thought is that sometimes we put KRPC addresses in a Thrift structure to
will add uds_address in TNetworkAddress.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 24 May 2022 18:36:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
..
Patch Set 9:
(2 comments)
This is looking good to me. I had a couple thoughts, but I'm basically ready to
approve this.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else if (strcasecmp("ip_address",
FLAGS_uds_address_unique_id.c_str()) == 0) {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> Should we stop running if usd_address_unique_id is not one of allowed value
I see that you disable UDS if the value is invalid.
I think it is also ok to fail startup if uds_address_unique_id has been set to
an invalid value. My guess is that the caller of RpcMgr::Init() would handle a
not-OK status by halting startup. It can be confusing if the system ignores an
invalid value. Admins or developers may think that they have UDS configured
when it actually isn't enabled.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in
"Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:". This
field is optional.
: // It is only used for KRPC bind/listen/connect when
FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> We only support UDS for KPRC, not for thrift RPC. I think TNetworkAddress d
My thought is that sometimes we put KRPC addresses in a Thrift structure to
send to other daemons. The information is used on the other side to initialize
a NetworkAddressPB to use with KRPC, but it is inconvenient to embed a
NetworkAddressPB in a Thrift struct.
For example, the code in query-driver.cc and query-state.cc uses a second field
to communicate the backend id. That's fine, and this is a rare case. If we
wanted TNetworkAddress to have the UDS information, we could, and that case
would be a little easier.
I'm also fine with the current way. We should add a sentence to the comment
"Refer to Types.thrift for documentation" to say that UDS is limited to KRPC
and is only needed for NetworkAddressPB.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Tue, 24 May 2022 00:38:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10617/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 23 May 2022 14:47:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 618 insertions(+), 159 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/9 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has posted comments on this change. (
http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
..
Patch Set 8:
(5 comments)
Joe, thanks for your feedback.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc@380
PS8, Line 380: // Run tests with Unix domain socket.
: FLAGS_rpc_use_unix_domain_socket = true;
: discard_result(RUN_ALL_TESTS());
: // Run tests with TCP socket.
: FLAGS_rpc_use_unix_domain_socket = false;
: return RUN_ALL_TESTS();
> Couple thoughts:
Will fix it by using INSTANTIATE_TEST_CASE_P in gtest.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> It would be good to match "ip_address" and produce an error if usd_address_
Should we stop running if usd_address_unique_id is not one of allowed values?
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc@273
PS8, Line 273: // TODO: get BackendId of admissiond. Use a fixed UUID now.
: admissiond_backend_id.set_hi(12345678);
: admissiond_backend_id.set_lo(87654321);
> Will this be addressed in a separate JIRA?
I am not sure if we should support to use backend_id as unique id for UDS. But
I saw Kudu to use UUID as unique id for UDS. If we decide to support it, I will
address it in a separate JIRA.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc
File be/src/util/network-util.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc@198
PS8, Line 198: NetworkAddressPB MakeNetworkAddressPB(const std::string&
hostname, int port,
: const UniqueIdPB& backend_id, const UdsAddressUniqueIdPB&
uds_addr_unique_id) {
> Would it be useful to have a wrapper function that generates its own random
Good suggestion. Will do.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in
"Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:". This
field is optional.
: // It is only used for KRPC bind/listen/connect when
FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> Should we have a symmetric change in TNetworkAddress? The locations doing F
We only support UDS for KPRC, not for thrift RPC. I think TNetworkAddress don't
need to know about UDS.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 20 May 2022 01:18:32 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
..
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc@380
PS8, Line 380: // Run tests with Unix domain socket.
: FLAGS_rpc_use_unix_domain_socket = true;
: discard_result(RUN_ALL_TESTS());
: // Run tests with TCP socket.
: FLAGS_rpc_use_unix_domain_socket = false;
: return RUN_ALL_TESTS();
Couple thoughts:
1. We want to notice failures in the UDS case, so we want to care about the
return code.
2. One way to handle this is to use the INSTANTIATE_TEST_CASE_P functionality
in gtest. This lets you run the same set of tests in different configurations.
Then, there is a single RUN_ALL_TESTS() call. See thrift-server-test.cc for
some examples (and there are a few others in the codebase).
3. The other way to do this is to exit if the UDS RUN_ALL_TESTS() fails. I
think the way it works is that each RUN_ALL_TESTS() invocation would write out
a JUnitXML file and the second invocation would overwrite the first. Exiting on
failure keeps the JUnitXML state consistent (i.e. it shows the failures), but
the downside is that in the success case, the JUnitXML file doesn't contain the
first round of tests.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
It would be good to match "ip_address" and produce an error if
usd_address_unique_id is not one of the allowed values.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc@273
PS8, Line 273: // TODO: get BackendId of admissiond. Use a fixed UUID now.
: admissiond_backend_id.set_hi(12345678);
: admissiond_backend_id.set_lo(87654321);
Will this be addressed in a separate JIRA?
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc
File be/src/util/network-util.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc@198
PS8, Line 198: NetworkAddressPB MakeNetworkAddressPB(const std::string&
hostname, int port,
: const UniqueIdPB& backend_id, const UdsAddressUniqueIdPB&
uds_addr_unique_id) {
Would it be useful to have a wrapper function that generates its own random
backend_id and calls MakeNetworkAddressPB with it? There are several spots that
do something like this:
UniqueIdPB backend_id;
UUIDToUniqueIdPB(boost::uuids::random_generator()(), _id);
tls_krpc_address = MakeNetworkAddressPB(
ip, tls_service_port, backend_id,
tls_rpc_mgr.GetUdsAddressUniqueId());
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in
"Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:". This
field is optional.
: // It is only used for KRPC bind/listen/connect when
FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
Should we have a symmetric change in TNetworkAddress? The locations doing
FromTNetworkAddress() would be simpler if we didn't need the extra args.
More generally, does TNetworkAddress need to know about UDS?
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Thu, 19 May 2022 23:47:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 8: We can use UDS address as file path or address in the UNIX domain "abstract namespace" (a leading '@' indicates "abstract namespace"). I found that UDS addresses are not released immediately by system when impala daemons are killed if we use UDS address in the form of file path. This cause socket binding failures when restarting impala daemons. Don't see this issue with UDS address in the form of address in abstract namespace. So we use UDS address as address in abstract namespace. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 13 May 2022 21:41:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 13 May 2022 18:55:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 593 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/8 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 02 May 2022 17:09:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 593 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/7 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10432/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 13 Apr 2022 03:22:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization with unique_id specified by starting flag FLAGS_uds_address_unique_id. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. Show remote UDS address for KRPC inbound connections on rpcz Web page as '*' when using UDS since the remote UDS addresses are not available. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 593 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/6 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10400/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 05 Apr 2022 17:02:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS address. It could be 'ip_address', 'backend_id', or 'none'. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. - Show remote UDS address for KRPC inbound connections on rpcz Web page. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 582 insertions(+), 139 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/5 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10374/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 00:30:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10373/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 01 Apr 2022 00:20:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over UDS. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. - Show remote address for KRPC inbound connections on rpcz Web page. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 478 insertions(+), 137 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/4 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. Patch Set 3: (6 comments) http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py File tests/custom_cluster/test_krpc_socket.py: http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@24 PS3, Line 24: class TestKrpcSocket(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@48 PS3, Line 48: + flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@49 PS3, Line 49: + flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@80 PS3, Line 80: + flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@81 PS3, Line 81: + flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@91 PS3, Line 91: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Fri, 01 Apr 2022 00:01:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18369 ) Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket .. IMPALA-11129: Support running KRPC over Unix Domain Socket This patch make following changes to support running KRPC over UDS. - Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over UDS. - Add field uds_address in NetworkAddressPB. Replace TNetworkAddress with NetworkAddressPB for RPC related class variables and APIs. - Set UDS address for each daemon as @impala-kprc: during initialization. - When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC server will be binded to the UDS address of the daemon. KRPC Client will connect to KRPC server with the UDS address of the server when creating proxy service, which in turn call kudu::Socket::Connect() function to connect KRPC server. - rpcz Web page show TCP related stats as 'N/A' when using UDS. - Add new unit-tests for UDS. TODO: - Advertise BackendId of admissiond in global admission control mode. - Show remote address for KRPC inbound connections on rpcz Web page. Testing: - Passed core test with FLAG_rpc_use_unix_domain_socket as fault value false. - Passed core test with FLAG_rpc_use_unix_domain_socket as true. Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 --- M be/src/benchmarks/expr-benchmark.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-driver.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/runtime/tmp-file-mgr.cc M be/src/scheduling/admissiond-env.cc M be/src/scheduling/admissiond-env.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M common/protobuf/common.proto A tests/custom_cluster/test_krpc_socket.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py M www/rpcz.tmpl 38 files changed, 477 insertions(+), 137 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/3 -- To view, visit http://gerrit.cloudera.org:8080/18369 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7 Gerrit-Change-Number: 18369 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
