[jira] [Resolved] (KUDU-3491) MiniDumpExceptionHandler assert randomly fails
[
https://issues.apache.org/jira/browse/KUDU-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3491.
Fix Version/s: 1.18.0
Resolution: Fixed
> MiniDumpExceptionHandler assert randomly fails
> ---
>
> Key: KUDU-3491
> URL: https://issues.apache.org/jira/browse/KUDU-3491
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Bakai Ádám
>Assignee: Bakai Ádám
>Priority: Major
> Fix For: 1.18.0
>
>
> When starting master, this error randomly happens on my system:
> {code:java}
> + exec
> /opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p0.43161468/lib/kudu/sbin/kudu-master
>
> --master_addresses=abakai-1.abakai.root.hwx.site,abakai-2.abakai.root.hwx.site,abakai-3.abakai.root.hwx.site
>
> --location_mapping_cmd=/var/run/cloudera-scm-agent/process/126-kudu-KUDU_MASTER/topology.py
> --flagfile=/var/run/cloudera-scm-agent/process/126-kudu-KUDU_MASTER/gflagfile
> F20230717 10:37:23.626719 101405 minidump.cc:273] Check failed: 0 ==
> MinidumpExceptionHandler::current_num_instances_.fetch_add(1) (0 vs. 1)
> *** Check failure stack trace: ***
> Wrote minidump to
> /var/log/kudu/minidumps/kudu-master/1664cde8-f41a-4b7f-121578b0-55ba68a6.dmp
> *** Aborted at 1689590243 (unix time) try "date -d @1689590243" if you are
> using GNU date ***
> PC: @ 0x0 (unknown)
> *** SIGABRT (@0x9c2900018c1d) received by PID 101405 (TID 0x7f204a7cda00)
> from PID 101405; stack trace: ***
> @ 0xe4df76 google::(anonymous namespace)::FailureSignalHandler()
> @ 0x7f204a19e6d0 (unknown)
> @ 0x7f20483a6277 __GI_raise
> @ 0x7f20483a7968 __GI_abort
> @ 0xcb50af kudu::AbortFailureFunction()
> @ 0xe4296d google::LogMessage::Fail()
> @ 0xe4584a google::LogMessage::SendToLog()
> @ 0xe4249e google::LogMessage::Flush()
> @ 0xe439d9 google::LogMessageFatal::~LogMessageFatal()
> @ 0x319a0a1
> kudu::MinidumpExceptionHandler::RegisterMinidumpExceptionHandler()
> @ 0x319a107
> kudu::MinidumpExceptionHandler::MinidumpExceptionHandler()
> @ 0x12821f0 kudu::server::ServerBase::ServerBase()
> @ 0x124577e kudu::kserver::KuduServer::KuduServer()
> @ 0xddc9ce kudu::master::Master::Master()
> @ 0xd5732b kudu::master::RunMasterServer()
> @ 0xd51d8a kudu::master::MasterMain()
> @ 0x7f2048392445 __libc_start_main
> @ 0xd51b64 (unknown) {code}
> Version information:
> {code:java}
> [root@abakai-1 ~]#
> /opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p0.43161468/lib/kudu/sbin-release/kudu-master
> -version
> kudu 1.15.0.7.2.18.0-205
> revision 7e222133b1a13ce6c212ffb32d8ceaa0c6a8545a
> build type RELEASE
> built by None at 14 Jul 2023 06:51:13 UTC on re-centos-slave-large-2wczs
> build id 1178567 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KUDU-3491) MiniDumpExceptionHandler assert randomly fails
[
https://issues.apache.org/jira/browse/KUDU-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3491:
--
Assignee: Bakai Ádám
> MiniDumpExceptionHandler assert randomly fails
> ---
>
> Key: KUDU-3491
> URL: https://issues.apache.org/jira/browse/KUDU-3491
> Project: Kudu
> Issue Type: Bug
>Affects Versions: 1.15.0
>Reporter: Bakai Ádám
>Assignee: Bakai Ádám
>Priority: Major
>
> When starting master, this error randomly happens on my system:
> {code:java}
> + exec
> /opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p0.43161468/lib/kudu/sbin/kudu-master
>
> --master_addresses=abakai-1.abakai.root.hwx.site,abakai-2.abakai.root.hwx.site,abakai-3.abakai.root.hwx.site
>
> --location_mapping_cmd=/var/run/cloudera-scm-agent/process/126-kudu-KUDU_MASTER/topology.py
> --flagfile=/var/run/cloudera-scm-agent/process/126-kudu-KUDU_MASTER/gflagfile
> F20230717 10:37:23.626719 101405 minidump.cc:273] Check failed: 0 ==
> MinidumpExceptionHandler::current_num_instances_.fetch_add(1) (0 vs. 1)
> *** Check failure stack trace: ***
> Wrote minidump to
> /var/log/kudu/minidumps/kudu-master/1664cde8-f41a-4b7f-121578b0-55ba68a6.dmp
> *** Aborted at 1689590243 (unix time) try "date -d @1689590243" if you are
> using GNU date ***
> PC: @ 0x0 (unknown)
> *** SIGABRT (@0x9c2900018c1d) received by PID 101405 (TID 0x7f204a7cda00)
> from PID 101405; stack trace: ***
> @ 0xe4df76 google::(anonymous namespace)::FailureSignalHandler()
> @ 0x7f204a19e6d0 (unknown)
> @ 0x7f20483a6277 __GI_raise
> @ 0x7f20483a7968 __GI_abort
> @ 0xcb50af kudu::AbortFailureFunction()
> @ 0xe4296d google::LogMessage::Fail()
> @ 0xe4584a google::LogMessage::SendToLog()
> @ 0xe4249e google::LogMessage::Flush()
> @ 0xe439d9 google::LogMessageFatal::~LogMessageFatal()
> @ 0x319a0a1
> kudu::MinidumpExceptionHandler::RegisterMinidumpExceptionHandler()
> @ 0x319a107
> kudu::MinidumpExceptionHandler::MinidumpExceptionHandler()
> @ 0x12821f0 kudu::server::ServerBase::ServerBase()
> @ 0x124577e kudu::kserver::KuduServer::KuduServer()
> @ 0xddc9ce kudu::master::Master::Master()
> @ 0xd5732b kudu::master::RunMasterServer()
> @ 0xd51d8a kudu::master::MasterMain()
> @ 0x7f2048392445 __libc_start_main
> @ 0xd51b64 (unknown) {code}
> Version information:
> {code:java}
> [root@abakai-1 ~]#
> /opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p0.43161468/lib/kudu/sbin-release/kudu-master
> -version
> kudu 1.15.0.7.2.18.0-205
> revision 7e222133b1a13ce6c212ffb32d8ceaa0c6a8545a
> build type RELEASE
> built by None at 14 Jul 2023 06:51:13 UTC on re-centos-slave-large-2wczs
> build id 1178567 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KUDU-3531) Limit the amount of resources used by tombstoned tablet replicas
[ https://issues.apache.org/jira/browse/KUDU-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3531: --- Labels: scalability (was: ) > Limit the amount of resources used by tombstoned tablet replicas > > > Key: KUDU-3531 > URL: https://issues.apache.org/jira/browse/KUDU-3531 > Project: Kudu > Issue Type: Improvement >Reporter: Alexey Serbin >Priority: Major > Labels: scalability > > I came across a case where a tablet server had just about 2K live tablet > replicas, but it opened about 24K files in its WAL and data directories. The > issue stems from the fact that tombstoned tablet replica's files are opened > by the FS manager the same as for a live replica, and those are kept open > even if they are never about to change. It would be prudent to avoid keeping > tombstoned tablet replicas' files open, if possible: maybe, just read the > required information (last voted term and opId index?) and keep it in runtime > structures, but close corresponding files right after bootstrapping? > Otherwise, this doesn't seem to scale well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3520) File descriptor leak in Env::NewRWFile() when ecryption-at-rest is enabled
[
https://issues.apache.org/jira/browse/KUDU-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3520.
Fix Version/s: 1.18.0
Resolution: Fixed
> File descriptor leak in Env::NewRWFile() when ecryption-at-rest is enabled
> --
>
> Key: KUDU-3520
> URL: https://issues.apache.org/jira/browse/KUDU-3520
> Project: Kudu
> Issue Type: Bug
> Components: fs, security, tserver
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Alexey Serbin
>Assignee: Attila Bukor
>Priority: Major
> Fix For: 1.18.0
>
>
> There is a file descriptor leak in {{Env::NewRWFile()}} on an error path when
> encryption-at-rest is enabled.
> In the code below, if {{ReadEncryptionHeader()}} or
> {{WriteEncryptionHeader()}} failed, the descriptor of the file opened by
> {{DoOpen()}} would be leaked.
> {noformat}
> RETURN_NOT_OK(DoOpen(fname, opts.mode, ));
> EncryptionHeader eh;
> if (encrypt) {
> DCHECK(encryption_key_);
> if (size >= kEncryptionHeaderSize) {
> RETURN_NOT_OK(ReadEncryptionHeader(fd, fname, *encryption_key_, ));
> } else {
> RETURN_NOT_OK(GenerateHeader());
> RETURN_NOT_OK(WriteEncryptionHeader(fd, fname, *encryption_key_, eh));
> }
> }
> result->reset(new PosixRWFile(fname, fd, opts.sync_on_close, encrypt,
> eh));
> {noformat}
> It's been evidenced in the wild when creating the metadata file for a tablet
> during tablet copying failed with the error like below:
> {noformat}
> Runtime error: Couldn't create tablet metadata: Failed to write tablet
> metadata d199a872b03848d695f067ed5c694835: Failed to initialize encryption:
> error:0607B083:digital envelope routines:EVP_CipherInit_ex:no cipher
> set:crypto/evp/evp_enc.c:170
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KUDU-3519) list of masters in "/dump-entites" endpoint
[ https://issues.apache.org/jira/browse/KUDU-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3519: --- Fix Version/s: 1.18.0 Resolution: Fixed Status: Resolved (was: In Review) > list of masters in "/dump-entites" endpoint > --- > > Key: KUDU-3519 > URL: https://issues.apache.org/jira/browse/KUDU-3519 > Project: Kudu > Issue Type: Improvement >Reporter: Gabor Zele >Assignee: Attila Bukor >Priority: Major > Fix For: 1.18.0 > > > I'm gathering all kudu metrics for analytics by using the "/metrics" endpoins > on all master/tablet servers. I'm getting the list of tablet-servers by > querying "/dump-entites" on a master server. > "/dump-entites" has top level key for "tablet_servers", but it doesn't > contain any information about the other masters. For this, I have to maintain > a list of master http adresses and server UUIDs in my solution, and update it > there are changes to the masters. > If the information would be available here, knowing the http address of any > working master would give me the list of configured masters, which I could > contact for metrics and save for future use. > (https://github.com/apache/kudu/blob/master/src/kudu/master/master_path_handlers.cc#L777) > This Jira is a request to add a key like "master_servers" to the > "/dump-entites" endpoint with the information about the other masters, like > we have now in "tablet_servers" for the tablet servers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (KUDU-3337) Tool to manually create cmeta files
[ https://issues.apache.org/jira/browse/KUDU-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor closed KUDU-3337. -- Fix Version/s: 1.18.0 Resolution: Fixed > Tool to manually create cmeta files > --- > > Key: KUDU-3337 > URL: https://issues.apache.org/jira/browse/KUDU-3337 > Project: Kudu > Issue Type: New Feature >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.18.0 > > > Power outages can lead to empty cmeta files on XFS (KUDU-2195), and sometimes > all replicas are affected. By checking the ksck output and the WAL dumps it's > possible to reconstruct how the cmeta should look like, except for the > voted_for part, but that isn't required to be able to bootstrap a tablet, so > a tool to manually create a cmeta file would be useful to recover such > tablets. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (KUDU-3522) A tablet server starts in non-functional state when enabling data-at-rest encryption
[
https://issues.apache.org/jira/browse/KUDU-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor closed KUDU-3522.
--
Fix Version/s: 1.18.0
Resolution: Fixed
> A tablet server starts in non-functional state when enabling data-at-rest
> encryption
>
>
> Key: KUDU-3522
> URL: https://issues.apache.org/jira/browse/KUDU-3522
> Project: Kudu
> Issue Type: Bug
> Components: security, tserver
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Alexey Serbin
>Assignee: Attila Bukor
>Priority: Major
> Fix For: 1.18.0
>
>
> It's possible to configure a Kudu tablet server by enabling the data-at-rest
> encryption feature in such a way that the server runs in a non-functional
> state: {{kudu-tserver}} process starts and runs with no visible issues, but
> it's not able to host any tablet replicas.
> It's easy to fix/address the issue by adding an extra sanity check: when
> opening an already existing FS data directory structure, make sure the server
> encryption key isn't empty if Kudu server is run with the
> {{\-\-encrypt_data_at_rest}} flag. There might be more alternatives around.
> The reproduction scenario for the issue is below.
> # Start a tablet server without encryption-at-rest, making sure the tablet
> server starts and creates the directory structure on the file system.
> # Don't create any tables/ranges yet. Essentially, it's necessary to make
> sure not a single tablet replica is placed at the server yet.
> # Shut down the tablet server.
> # Update the configuration for the tablet server, enabling
> encryption-at-rest and specifying the key provider. For test purposes, it's
> enough to use the "default" key provider:
> {noformat}
> --encrypt_data_at_rest=true
> --encryption_key_provider=default
> {noformat}
> # Start the tablet server.
> # Try to create a new tablet replica that would be placed at the tablet
> server. That could be creation of a new table, or try to move a tablet
> replica from some other tablet server by using the {{kudu tablet
> change_config move_replica}} CLI tool.
> # Check logs of Kudu master or the {{kudu}} CLI tool: there should be error
> messages like {{Failed to initialize encryption: error:0607B083:digital
> envelope routines:EVP_CipherInit_ex:no cipher set}}
> # No tablet replica can now be placed at the tablet server, while nothing
> suspicious can be found in the tablet server's log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KUDU-3522) A tablet server starts in non-functional state when enabling data-at-rest encryption
[
https://issues.apache.org/jira/browse/KUDU-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3522:
--
Assignee: Attila Bukor
> A tablet server starts in non-functional state when enabling data-at-rest
> encryption
>
>
> Key: KUDU-3522
> URL: https://issues.apache.org/jira/browse/KUDU-3522
> Project: Kudu
> Issue Type: Bug
> Components: security, tserver
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Alexey Serbin
>Assignee: Attila Bukor
>Priority: Major
>
> It's possible to configure a Kudu tablet server by enabling the data-at-rest
> encryption feature in such a way that the server runs in a non-functional
> state: {{kudu-tserver}} process starts and runs with no visible issues, but
> it's not able to host any tablet replicas.
> It's easy to fix/address the issue by adding an extra sanity check: when
> opening an already existing FS data directory structure, make sure the server
> encryption key isn't empty if Kudu server is run with the
> {{\-\-encrypt_data_at_rest}} flag. There might be more alternatives around.
> The reproduction scenario for the issue is below.
> # Start a tablet server without encryption-at-rest, making sure the tablet
> server starts and creates the directory structure on the file system.
> # Don't create any tables/ranges yet. Essentially, it's necessary to make
> sure not a single tablet replica is placed at the server yet.
> # Shut down the tablet server.
> # Update the configuration for the tablet server, enabling
> encryption-at-rest and specifying the key provider. For test purposes, it's
> enough to use the "default" key provider:
> {noformat}
> --encrypt_data_at_rest=true
> --encryption_key_provider=default
> {noformat}
> # Start the tablet server.
> # Try to create a new tablet replica that would be placed at the tablet
> server. That could be creation of a new table, or try to move a tablet
> replica from some other tablet server by using the {{kudu tablet
> change_config move_replica}} CLI tool.
> # Check logs of Kudu master or the {{kudu}} CLI tool: there should be error
> messages like {{Failed to initialize encryption: error:0607B083:digital
> envelope routines:EVP_CipherInit_ex:no cipher set}}
> # No tablet replica can now be placed at the tablet server, while nothing
> suspicious can be found in the tablet server's log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KUDU-3520) File descriptor leak in Env::NewRWFile() when ecryption-at-rest is enabled
[
https://issues.apache.org/jira/browse/KUDU-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3520:
--
Assignee: Attila Bukor
> File descriptor leak in Env::NewRWFile() when ecryption-at-rest is enabled
> --
>
> Key: KUDU-3520
> URL: https://issues.apache.org/jira/browse/KUDU-3520
> Project: Kudu
> Issue Type: Bug
> Components: fs, security, tserver
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Alexey Serbin
>Assignee: Attila Bukor
>Priority: Major
>
> There is a file descriptor leak in {{Env::NewRWFile()}} on an error path when
> encryption-at-rest is enabled.
> In the code below, if {{ReadEncryptionHeader()}} or
> {{WriteEncryptionHeader()}} failed, the descriptor of the file opened by
> {{DoOpen()}} would be leaked.
> {noformat}
> RETURN_NOT_OK(DoOpen(fname, opts.mode, ));
> EncryptionHeader eh;
> if (encrypt) {
> DCHECK(encryption_key_);
> if (size >= kEncryptionHeaderSize) {
> RETURN_NOT_OK(ReadEncryptionHeader(fd, fname, *encryption_key_, ));
> } else {
> RETURN_NOT_OK(GenerateHeader());
> RETURN_NOT_OK(WriteEncryptionHeader(fd, fname, *encryption_key_, eh));
> }
> }
> result->reset(new PosixRWFile(fname, fd, opts.sync_on_close, encrypt,
> eh));
> {noformat}
> It's been evidenced in the wild when creating the metadata file for a tablet
> during tablet copying failed with the error like below:
> {noformat}
> Runtime error: Couldn't create tablet metadata: Failed to write tablet
> metadata d199a872b03848d695f067ed5c694835: Failed to initialize encryption:
> error:0607B083:digital envelope routines:EVP_CipherInit_ex:no cipher
> set:crypto/evp/evp_enc.c:170
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KUDU-3519) list of masters in "/dump-entites" endpoint
[ https://issues.apache.org/jira/browse/KUDU-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3519: --- Status: In Review (was: In Progress) > list of masters in "/dump-entites" endpoint > --- > > Key: KUDU-3519 > URL: https://issues.apache.org/jira/browse/KUDU-3519 > Project: Kudu > Issue Type: Improvement >Reporter: Gabor Zele >Assignee: Attila Bukor >Priority: Major > > I'm gathering all kudu metrics for analytics by using the "/metrics" endpoins > on all master/tablet servers. I'm getting the list of tablet-servers by > querying "/dump-entites" on a master server. > "/dump-entites" has top level key for "tablet_servers", but it doesn't > contain any information about the other masters. For this, I have to maintain > a list of master http adresses and server UUIDs in my solution, and update it > there are changes to the masters. > If the information would be available here, knowing the http address of any > working master would give me the list of configured masters, which I could > contact for metrics and save for future use. > (https://github.com/apache/kudu/blob/master/src/kudu/master/master_path_handlers.cc#L777) > This Jira is a request to add a key like "master_servers" to the > "/dump-entites" endpoint with the information about the other masters, like > we have now in "tablet_servers" for the tablet servers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KUDU-3519) list of masters in "/dump-entites" endpoint
[ https://issues.apache.org/jira/browse/KUDU-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3519: -- Assignee: Attila Bukor > list of masters in "/dump-entites" endpoint > --- > > Key: KUDU-3519 > URL: https://issues.apache.org/jira/browse/KUDU-3519 > Project: Kudu > Issue Type: Improvement >Reporter: Gabor Zele >Assignee: Attila Bukor >Priority: Major > > I'm gathering all kudu metrics for analytics by using the "/metrics" endpoins > on all master/tablet servers. I'm getting the list of tablet-servers by > querying "/dump-entites" on a master server. > "/dump-entites" has top level key for "tablet_servers", but it doesn't > contain any information about the other masters. For this, I have to maintain > a list of master http adresses and server UUIDs in my solution, and update it > there are changes to the masters. > If the information would be available here, knowing the http address of any > working master would give me the list of configured masters, which I could > contact for metrics and save for future use. > (https://github.com/apache/kudu/blob/master/src/kudu/master/master_path_handlers.cc#L777) > This Jira is a request to add a key like "master_servers" to the > "/dump-entites" endpoint with the information about the other masters, like > we have now in "tablet_servers" for the tablet servers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3448) Store IPKI and TSK key material encrypted
[
https://issues.apache.org/jira/browse/KUDU-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3448.
Fix Version/s: 1.18.0
Resolution: Fixed
> Store IPKI and TSK key material encrypted
> -
>
> Key: KUDU-3448
> URL: https://issues.apache.org/jira/browse/KUDU-3448
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Critical
> Labels: security
> Fix For: 1.18.0
>
>
> Key material for IPKI TLS and TSK should be stored on disk securely, even
> when user data is not encrypted. The symmetric encryption key should be
> derived from a password using PBKDF2 which is a FIPS-approved KDF. The
> masters should have a flag that expects a command which outputs the password
> (similar to {{{}--webserver_private_key_password_cmd{}}}), that way the users
> can integrate with a HSM or choose another way to provide the password
> securely without storing it on a disk.
> Generating new keys or encrypting existing key material is outside the scope
> of this ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KUDU-3322) Handle to situation of last seen event ID in Kudu master being more than the latest event ID from HMS
[
https://issues.apache.org/jira/browse/KUDU-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3322:
--
Assignee: Abhishek Chennaka
> Handle to situation of last seen event ID in Kudu master being more than the
> latest event ID from HMS
> -
>
> Key: KUDU-3322
> URL: https://issues.apache.org/jira/browse/KUDU-3322
> Project: Kudu
> Issue Type: Improvement
>Reporter: Abhishek Chennaka
>Assignee: Abhishek Chennaka
>Priority: Minor
> Fix For: 1.17.0
>
>
> Came across a scenario where HMS database was lost and restored from an old
> backup which reset the latest event ID (monotonically increasing integer) in
> HMS to a lower value than what it should be.
> Since Kudu master has a last seen event ID greater than the one in HMS
> currently, it could not process any new events generated. For example, Kudu
> table deletion was not happening as the Kudu master expects an event ID which
> is higher than the one it has last seen but the event ID in HMS for the table
> deletion is less than the one in the Kudu master.
> This also causes discrepancy between the metadata in HMS and Kudu masters. It
> would be better if the Kudu master upon startup does the comparison of the
> last seen event ID and latest event ID in HMS and crash if the one in HMS is
> lower with a helpful message/clarifying question like:
> {code:java}
> Found the last seen event ID in the local Kudu master to be greater than the
> latest event ID in HMS. Was there any backup or restore done on HMS
> recently?{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KUDU-3322) Handle to situation of last seen event ID in Kudu master being more than the latest event ID from HMS
[
https://issues.apache.org/jira/browse/KUDU-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3322.
Fix Version/s: 1.17.0
Resolution: Fixed
> Handle to situation of last seen event ID in Kudu master being more than the
> latest event ID from HMS
> -
>
> Key: KUDU-3322
> URL: https://issues.apache.org/jira/browse/KUDU-3322
> Project: Kudu
> Issue Type: Improvement
>Reporter: Abhishek Chennaka
>Priority: Minor
> Fix For: 1.17.0
>
>
> Came across a scenario where HMS database was lost and restored from an old
> backup which reset the latest event ID (monotonically increasing integer) in
> HMS to a lower value than what it should be.
> Since Kudu master has a last seen event ID greater than the one in HMS
> currently, it could not process any new events generated. For example, Kudu
> table deletion was not happening as the Kudu master expects an event ID which
> is higher than the one it has last seen but the event ID in HMS for the table
> deletion is less than the one in the Kudu master.
> This also causes discrepancy between the metadata in HMS and Kudu masters. It
> would be better if the Kudu master upon startup does the comparison of the
> last seen event ID and latest event ID in HMS and crash if the one in HMS is
> lower with a helpful message/clarifying question like:
> {code:java}
> Found the last seen event ID in the local Kudu master to be greater than the
> latest event ID in HMS. Was there any backup or restore done on HMS
> recently?{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KUDU-3319) Log Kudu notification/event ID during master startup
[
https://issues.apache.org/jira/browse/KUDU-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3319.
Fix Version/s: 1.17.0
Resolution: Fixed
> Log Kudu notification/event ID during master startup
>
>
> Key: KUDU-3319
> URL: https://issues.apache.org/jira/browse/KUDU-3319
> Project: Kudu
> Issue Type: Improvement
> Components: hms, master
>Reporter: Abhishek Chennaka
>Priority: Minor
> Fix For: 1.17.0
>
>
> Currently during startup Kudu masters log the below during startup
> {code:java}
> I0601 18:24:05.250339 90422 catalog_manager.cc:1095] Loading latest processed
> Hive Metastore notification log event ID...{code}
> Would be helpful while debugging if we can log the actual ID retrieved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KUDU-3504) Crash master on subprocess failure
[ https://issues.apache.org/jira/browse/KUDU-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3504: --- Status: In Review (was: Open) > Crash master on subprocess failure > -- > > Key: KUDU-3504 > URL: https://issues.apache.org/jira/browse/KUDU-3504 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > > If the Ranger subprocess crashes, authorization will fail, but there's no > other indication that the process has died. The master won't restart the > subprocess and there's no way to restart it manually without restarting Kudu > anyway. Furthermore, if the subprocess crashes, that's likely a symptom of > something that requires manual intervention to resolve to avoid crashing in > the future, so it's best if the master crashes upon the subprocess crashing > as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KUDU-3504) Crash master on subprocess failure
Attila Bukor created KUDU-3504: -- Summary: Crash master on subprocess failure Key: KUDU-3504 URL: https://issues.apache.org/jira/browse/KUDU-3504 Project: Kudu Issue Type: Improvement Reporter: Attila Bukor Assignee: Attila Bukor If the Ranger subprocess crashes, authorization will fail, but there's no other indication that the process has died. The master won't restart the subprocess and there's no way to restart it manually without restarting Kudu anyway. Furthermore, if the subprocess crashes, that's likely a symptom of something that requires manual intervention to resolve to avoid crashing in the future, so it's best if the master crashes upon the subprocess crashing as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KUDU-3503) Allow extra Java arguments to be passed to Ranger client subprocess
[ https://issues.apache.org/jira/browse/KUDU-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3503: -- Assignee: Attila Bukor > Allow extra Java arguments to be passed to Ranger client subprocess > --- > > Key: KUDU-3503 > URL: https://issues.apache.org/jira/browse/KUDU-3503 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KUDU-3503) Allow extra Java arguments to be passed to Ranger client subprocess
[ https://issues.apache.org/jira/browse/KUDU-3503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3503: --- Status: In Review (was: In Progress) > Allow extra Java arguments to be passed to Ranger client subprocess > --- > > Key: KUDU-3503 > URL: https://issues.apache.org/jira/browse/KUDU-3503 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KUDU-3359) Enable audit writing to HDFS in Ranger Kudu Client
[ https://issues.apache.org/jira/browse/KUDU-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3359: --- Status: In Review (was: In Progress) > Enable audit writing to HDFS in Ranger Kudu Client > -- > > Key: KUDU-3359 > URL: https://issues.apache.org/jira/browse/KUDU-3359 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Mahesh Reddy >Assignee: Attila Bukor >Priority: Major > > Ranger audits aren't written to HDFS because HDFS isn't included in the jar > classpath. A new flag will have to be created in the ranger client that will > consist of the hdfs classpath. In the time being, audits can be written to > Solr. Multiple customers have run into this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KUDU-3503) Allow extra Java arguments to be passed to Ranger client subprocess
Attila Bukor created KUDU-3503: -- Summary: Allow extra Java arguments to be passed to Ranger client subprocess Key: KUDU-3503 URL: https://issues.apache.org/jira/browse/KUDU-3503 Project: Kudu Issue Type: Improvement Reporter: Attila Bukor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KUDU-3359) Enable audit writing to HDFS in Ranger Kudu Client
[ https://issues.apache.org/jira/browse/KUDU-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3359: -- Assignee: Attila Bukor (was: Marton Greber) > Enable audit writing to HDFS in Ranger Kudu Client > -- > > Key: KUDU-3359 > URL: https://issues.apache.org/jira/browse/KUDU-3359 > Project: Kudu > Issue Type: Bug >Affects Versions: 1.15.0 >Reporter: Mahesh Reddy >Assignee: Attila Bukor >Priority: Major > > Ranger audits aren't written to HDFS because HDFS isn't included in the jar > classpath. A new flag will have to be created in the ranger client that will > consist of the hdfs classpath. In the time being, audits can be written to > Solr. Multiple customers have run into this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3392) Support custom certificate when Kudu acts as a client
[ https://issues.apache.org/jira/browse/KUDU-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3392. Fix Version/s: 1.17.0 Resolution: Fixed > Support custom certificate when Kudu acts as a client > - > > Key: KUDU-3392 > URL: https://issues.apache.org/jira/browse/KUDU-3392 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > > Kudu connects to Ranger KMS when encryption is enabled using libcurl, and if > the certificate is not trusted on the OS-level, it fails to connect. It > should be possible to trust a certificate file by providing it in the CLI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-2252) Design and implement native encryption at rest
[ https://issues.apache.org/jira/browse/KUDU-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-2252. Fix Version/s: 1.17.0 Resolution: Fixed > Design and implement native encryption at rest > -- > > Key: KUDU-2252 > URL: https://issues.apache.org/jira/browse/KUDU-2252 > Project: Kudu > Issue Type: New Feature > Components: fs, security >Reporter: Mike Percy >Assignee: Attila Bukor >Priority: Major > Labels: roadmap-candidate > Fix For: 1.17.0 > > > It would be beneficial for Kudu to support native encryption at rest. > While the underlying filesystem can be encrypted with Kudu on top, there are > benefits to native encryption at rest: > * With native encryption support, it may be possible to specify different > encryption keys for different objects or object trees (such as is possible > with HDFS encryption) > * It may be possible to share a single block device with other storage, > including HDFS (note that some linux setups allow for "splitting" a device) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3385) Ranger KMS key provider
[ https://issues.apache.org/jira/browse/KUDU-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3385. Fix Version/s: 1.17.0 Resolution: Fixed > Ranger KMS key provider > --- > > Key: KUDU-3385 > URL: https://issues.apache.org/jira/browse/KUDU-3385 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KUDU-3423) Kudu can't handle multiple Ranger KMS servers
[ https://issues.apache.org/jira/browse/KUDU-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3423: --- Fix Version/s: 1.17.0 Resolution: Fixed Status: Resolved (was: In Review) > Kudu can't handle multiple Ranger KMS servers > - > > Key: KUDU-3423 > URL: https://issues.apache.org/jira/browse/KUDU-3423 > Project: Kudu > Issue Type: Bug > Components: security >Affects Versions: 1.17.0 >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > > It's possible to set up multiple Ranger KMS servers in HA, but Kudu is only > able to handle one as a key provider for data at rest encryption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KUDU-1592) Documentation that mentions file block manager should sound more ominous
[ https://issues.apache.org/jira/browse/KUDU-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-1592: --- Fix Version/s: 1.8.0 Resolution: Fixed Status: Resolved (was: In Review) > Documentation that mentions file block manager should sound more ominous > > > Key: KUDU-1592 > URL: https://issues.apache.org/jira/browse/KUDU-1592 > Project: Kudu > Issue Type: Improvement > Components: documentation >Reporter: Todd Lipcon >Assignee: Attila Bukor >Priority: Major > Fix For: 1.8.0 > > > In troubleshooting.adoc, as well as in the error message when we fail to hole > punch, we suggest using the file block manager as a workaround. It says > something vague about "at the cost of some scalability and efficiency" but > should be something a lot more ominous -- users are quickly running out of > file descriptors if they try the FBM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3449) Kudu client should get auto redirected to the active master
[
https://issues.apache.org/jira/browse/KUDU-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686783#comment-17686783
]
Attila Bukor commented on KUDU-3449:
Thanks [~mylogi...@gmail.com] for bringing this up. I drafted a design doc:
https://s.apache.org/kudu-3349-design
> Kudu client should get auto redirected to the active master
> ---
>
> Key: KUDU-3449
> URL: https://issues.apache.org/jira/browse/KUDU-3449
> Project: Kudu
> Issue Type: Improvement
>Reporter: Manish Maheshwari
>Priority: Major
>
> When providing only 1 Kudu master host, clients get an error as below. We
> should ideally allow clients to be redirected to the active master cc
> [~abukor] [~arawat]
> {code:java}
> ERROR: AnalysisException: Cannot analyze Kudu table 'my_first_kudu_table':
> Error determining if Kudu's integration with the Hive Metastore is enabled:
> Could not connect to a leader master. Client configured with 1 master(s)
> (pm-kudu-master30.pm-sandb.a465-9q4k.cloudera.site:7051) but cluster
> indicates it expects 3 master(s)
> (pm-kudu-master10.pm-sandb.a465-9q4k.cloudera.site:7051,pm-kudu-master20.pm-sandb.a465-9q4k.cloudera.site:7051,pm-kudu-master30.pm-sandb.a465-9q4k.cloudera.site:7051)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KUDU-3448) Store IPKI and TSK key material encrypted
[
https://issues.apache.org/jira/browse/KUDU-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3448:
--
Assignee: Attila Bukor
> Store IPKI and TSK key material encrypted
> -
>
> Key: KUDU-3448
> URL: https://issues.apache.org/jira/browse/KUDU-3448
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Critical
> Labels: security
>
> Key material for IPKI TLS and TSK should be stored on disk securely, even
> when user data is not encrypted. The symmetric encryption key should be
> derived from a password using PBKDF2 which is a FIPS-approved KDF. The
> masters should have a flag that expects a command which outputs the password
> (similar to {{{}--webserver_private_key_password_cmd{}}}), that way the users
> can integrate with a HSM or choose another way to provide the password
> securely without storing it on a disk.
> Generating new keys or encrypting existing key material is outside the scope
> of this ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KUDU-3448) Store IPKI and TSK key material encrypted
Attila Bukor created KUDU-3448:
--
Summary: Store IPKI and TSK key material encrypted
Key: KUDU-3448
URL: https://issues.apache.org/jira/browse/KUDU-3448
Project: Kudu
Issue Type: Improvement
Reporter: Attila Bukor
Key material for IPKI TLS and TSK should be stored on disk securely, even when
user data is not encrypted. The symmetric encryption key should be derived from
a password using PBKDF2 which is a FIPS-approved KDF. The masters should have a
flag that expects a command which outputs the password (similar to
{{{}--webserver_private_key_password_cmd{}}}), that way the users can integrate
with a HSM or choose another way to provide the password securely without
storing it on a disk.
Generating new keys or encrypting existing key material is outside the scope of
this ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KUDU-3423) Kudu can't handle multiple Ranger KMS servers
[ https://issues.apache.org/jira/browse/KUDU-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3423: --- Status: In Review (was: In Progress) > Kudu can't handle multiple Ranger KMS servers > - > > Key: KUDU-3423 > URL: https://issues.apache.org/jira/browse/KUDU-3423 > Project: Kudu > Issue Type: Bug > Components: security >Affects Versions: 1.17.0 >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > > It's possible to set up multiple Ranger KMS servers in HA, but Kudu is only > able to handle one as a key provider for data at rest encryption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KUDU-3423) Kudu can't handle multiple Ranger KMS servers
[ https://issues.apache.org/jira/browse/KUDU-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3423: --- Code Review: https://gerrit.cloudera.org/c/19271/ > Kudu can't handle multiple Ranger KMS servers > - > > Key: KUDU-3423 > URL: https://issues.apache.org/jira/browse/KUDU-3423 > Project: Kudu > Issue Type: Bug > Components: security >Affects Versions: 1.17.0 >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > > It's possible to set up multiple Ranger KMS servers in HA, but Kudu is only > able to handle one as a key provider for data at rest encryption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KUDU-3423) Kudu can't handle multiple Ranger KMS servers
Attila Bukor created KUDU-3423: -- Summary: Kudu can't handle multiple Ranger KMS servers Key: KUDU-3423 URL: https://issues.apache.org/jira/browse/KUDU-3423 Project: Kudu Issue Type: Bug Components: security Affects Versions: 1.17.0 Reporter: Attila Bukor Assignee: Attila Bukor It's possible to set up multiple Ranger KMS servers in HA, but Kudu is only able to handle one as a key provider for data at rest encryption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3405) Tabelt Server become 'Zombie Process'
[
https://issues.apache.org/jira/browse/KUDU-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635877#comment-17635877
]
Attila Bukor commented on KUDU-3405:
If you need assistance with a problem you're facing, I suggest using the
[u...@kudu.apache.org|mailto:u...@kudu.apache.org] mailing list to ask for help
instead of JIRA. Tickets in this system are for tracking known bugs that need
to be fixed, planned/requested features, and improvements.
> Tabelt Server become 'Zombie Process'
> -
>
> Key: KUDU-3405
> URL: https://issues.apache.org/jira/browse/KUDU-3405
> Project: Kudu
> Issue Type: Bug
> Components: tserver
>Affects Versions: 1.13.0
> Environment: linux-4.18.0-147.8.1.el7.aarch64
> gcc-4.8.5
> docker-19.03.4
> kudu-1.13.0(master*3，tserver*11)
>Reporter: Jie Huang
>Priority: Major
>
> In ARM, running KUDU. Tablet server incidental 'Zombie Process':
> 1. Process is exist;
> 2.Port 7050/8050 exist;
> 3.Log print is stopped;
> 4.minidmp is printed;
> 5.other tserver failed to connect it;
> 6.master failed to connect it, and loss of heartbeat;
> 7.memory/cpu/disk is normal;
>
> example-1(2022-11-17 add):
> {code:java}
> root 7 1 4 02:44 ? 00:20:47
> /opt/java/kudu/release/bin/kudu-tserver
> --flagfile=/opt/java/kudu/conf/tserver.gflagfile
> root 25428 7 0 02:55 ? 00:00:01 [rpc worker-89]
> {code}
>
> example-2(2022-11-17 add):
>
> {code:java}
> root 7 1 72 Nov16 ? 13:52:49
> /opt/java/kudu/release/bin/kudu-tserver
> --flagfile=/opt/java/kudu/conf/tserver.gflagfile
> root 40977 7 0 Nov16 ? 00:00:01 [rpc reactor-23]
> {code}
>
> example-3(2022-11-17 add):
>
> {code:java}
> root 7 1 1 Nov16 ? 00:14:27
> /opt/java/kudu/release/bin/kudu-tserver
> --flagfile=/opt/java/kudu/conf/tserver.gflagfile
> root 13818 7 0 Nov16 ? 00:00:01 [rpc worker-162]
> {code}
>
>
> minidmp:
> {quote}${KUDU_HOME}/thirdparty/installed/uninstrumented/bin/minidump-2-core
> -o xxx.core xxx.dmp
> gdb bin/kudu-tserver xxx.core
> (gdb) bt
> {quote}
>
> {quote}GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <[http://gnu.org/licenses/gpl.html]>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "aarch64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <[http://www.gnu.org/software/gdb/bugs/]>...
> Reading symbols from
> /opt/kudu-1.13.0-aarch64/kudu-1.13.0-aarch64/release/bin/kudu-tserver...done.
> warning: core file may not match specified executable file.
> [New LWP 44946]
> [New LWP 7]
> [New LWP 9]
> [New LWP 10]
> [New LWP 11]
> [New LWP 12]
> [New LWP 13]
> [New LWP 14]
> [New LWP 85]
> [New LWP 86]
> [New LWP 87]
> [New LWP 88]
> [New LWP 89]
> [New LWP 90]
> [New LWP 91]
> [New LWP 92]
> [New LWP 93]
> [New LWP 148]
> [New LWP 149]
> [New LWP 150]
> [New LWP 151]
> [New LWP 152]
> [New LWP 153]
> [New LWP 154]
> [New LWP 155]
> [New LWP 156]
> [New LWP 157]
> [New LWP 158]
> [New LWP 159]
> [New LWP 160]
> [New LWP 161]
> [New LWP 162]
> [New LWP 163]
> [New LWP 164]
> [New LWP 165]
> [New LWP 166]
> [New LWP 167]
> [New LWP 168]
> [New LWP 169]
> [New LWP 170]
> [New LWP 171]
> [New LWP 172]
> [New LWP 173]
> [New LWP 174]
> [New LWP 175]
> [New LWP 176]
> [New LWP 177]
> [New LWP 178]
> [New LWP 179]
> [New LWP 180]
> [New LWP 181]
> [New LWP 182]
> [New LWP 183]
> [New LWP 184]
> [New LWP 185]
> [New LWP 186]
> [New LWP 187]
> [New LWP 188]
> [New LWP 189]
> [New LWP 190]
> [New LWP 191]
> [New LWP 192]
> [New LWP 193]
> [New LWP 194]
> [New LWP 195]
> [New LWP 196]
> [New LWP 197]
> [New LWP 198]
> [New LWP 199]
> [New LWP 200]
> [New LWP 201]
> [New LWP 202]
> [New LWP 203]
> [New LWP 204]
> [New LWP 205]
> [New LWP 206]
> [New LWP 207]
> [New LWP 208]
> [New LWP 209]
> [New LWP 210]
> [New LWP 211]
> [New LWP 212]
> [New LWP 213]
> [New LWP 214]
> [New LWP 215]
> [New LWP 216]
> [New LWP 217]
> [New LWP 218]
> [New LWP 219]
> [New LWP 220]
> [New LWP 221]
> [New LWP 222]
> [New LWP 223]
> [New LWP 224]
> [New LWP 225]
> [New LWP 226]
> [New LWP 227]
> [New LWP 228]
> [New LWP 229]
> [New LWP 230]
> [New LWP 231]
> [New LWP 232]
> [New LWP 233]
> [New LWP 234]
> [New LWP 235]
> [New LWP 236]
> [New LWP 237]
> [New LWP 238]
> [New LWP 239]
> [New LWP 240]
> [New LWP 241]
> [New LWP 242]
> [New LWP 243]
> [New LWP 244]
> [New LWP 245]
> [New LWP 246]
> [New LWP 247]
> [New LWP 248]
> [New LWP 249]
> [New LWP 250]
> [New LWP 251]
> [New LWP 252]
> [New LWP 253]
> [New LWP 254]
> [New LWP 255]
> [New LWP 884]
>
[jira] [Created] (KUDU-3392) Support custom certificate when Kudu acts as a client
Attila Bukor created KUDU-3392: -- Summary: Support custom certificate when Kudu acts as a client Key: KUDU-3392 URL: https://issues.apache.org/jira/browse/KUDU-3392 Project: Kudu Issue Type: Improvement Reporter: Attila Bukor Assignee: Attila Bukor Kudu connects to Ranger KMS when encryption is enabled using libcurl, and if the certificate is not trusted on the OS-level, it fails to connect. It should be possible to trust a certificate file by providing it in the CLI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KUDU-3316) Store encrypted encryption keys in encrypted files
[
https://issues.apache.org/jira/browse/KUDU-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569900#comment-17569900
]
Attila Bukor commented on KUDU-3316:
The server keys have a random IV, as returned by the KMS (Ranger KMS works the
same way). Encrypting and decrypting the file keys, on the other hand, take
place within Kudu, so the IV can be be controlled more closely.
{quote}My concern is non-randomized IV is easier to be exploited by plaintext
attack{quote}
Can you elaborate on this? IV+key is not reused, as each file has a separate
key.
> Store encrypted encryption keys in encrypted files
> --
>
> Key: KUDU-3316
> URL: https://issues.apache.org/jira/browse/KUDU-3316
> Project: Kudu
> Issue Type: Sub-task
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Major
> Fix For: 1.17.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KUDU-3385) Ranger KMS key provider
Attila Bukor created KUDU-3385: -- Summary: Ranger KMS key provider Key: KUDU-3385 URL: https://issues.apache.org/jira/browse/KUDU-3385 Project: Kudu Issue Type: Sub-task Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3316) Store encrypted encryption keys in encrypted files
[ https://issues.apache.org/jira/browse/KUDU-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3316. Fix Version/s: 1.17.0 Resolution: Fixed > Store encrypted encryption keys in encrypted files > -- > > Key: KUDU-3316 > URL: https://issues.apache.org/jira/browse/KUDU-3316 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KUDU-3373) Key provider interface with a default (test-only) implementation
[ https://issues.apache.org/jira/browse/KUDU-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3373. Fix Version/s: 1.17.0 Resolution: Fixed > Key provider interface with a default (test-only) implementation > > > Key: KUDU-3373 > URL: https://issues.apache.org/jira/browse/KUDU-3373 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KUDU-3373) Key provider interface with a default (test-only) implementation
Attila Bukor created KUDU-3373: -- Summary: Key provider interface with a default (test-only) implementation Key: KUDU-3373 URL: https://issues.apache.org/jira/browse/KUDU-3373 Project: Kudu Issue Type: Sub-task Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (KUDU-3368) Encrypt file keys with server keys
[ https://issues.apache.org/jira/browse/KUDU-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3368: --- Fix Version/s: 1.17.0 Resolution: Fixed Status: Resolved (was: In Review) > Encrypt file keys with server keys > -- > > Key: KUDU-3368 > URL: https://issues.apache.org/jira/browse/KUDU-3368 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (KUDU-3368) Encrypt file keys with server keys
[ https://issues.apache.org/jira/browse/KUDU-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3368: --- Status: In Review (was: In Progress) > Encrypt file keys with server keys > -- > > Key: KUDU-3368 > URL: https://issues.apache.org/jira/browse/KUDU-3368 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (KUDU-3368) Encrypt file keys with server keys
[ https://issues.apache.org/jira/browse/KUDU-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3368: --- Code Review: https://gerrit.cloudera.org/c/18192/ > Encrypt file keys with server keys > -- > > Key: KUDU-3368 > URL: https://issues.apache.org/jira/browse/KUDU-3368 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (KUDU-3368) Encrypt file keys with server keys
Attila Bukor created KUDU-3368: -- Summary: Encrypt file keys with server keys Key: KUDU-3368 URL: https://issues.apache.org/jira/browse/KUDU-3368 Project: Kudu Issue Type: Sub-task Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (KUDU-3331) Integrate encrypted Env
[ https://issues.apache.org/jira/browse/KUDU-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3331. Fix Version/s: 1.16.0 Resolution: Fixed > Integrate encrypted Env > --- > > Key: KUDU-3331 > URL: https://issues.apache.org/jira/browse/KUDU-3331 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.16.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (KUDU-3316) Store encrypted encryption keys in encrypted files
[ https://issues.apache.org/jira/browse/KUDU-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521751#comment-17521751 ] Attila Bukor commented on KUDU-3316: Thanks for your comments [~kirbyzhou] > In my opinion, EEK had better be of variable length. This actually represents an unencrypted/decrypted key, so all this metadata will be stripped/handled before the key is stored in an EncryptionHeader. > The IV of AES-CTR is hard-coded in DoEncryptV, CTR requires a nonce-encryption key pair to be unique, and each file has a unique file key, so as long as the nonce doesn't repeat, it is secure. Adding a randomized IV requires more entropy, with no real benefit in this case. > Store encrypted encryption keys in encrypted files > -- > > Key: KUDU-3316 > URL: https://issues.apache.org/jira/browse/KUDU-3316 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (KUDU-3309) Adding repartitioning logic along with coalesce logic to backup output
[ https://issues.apache.org/jira/browse/KUDU-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3309: -- Assignee: Minakshi Korad > Adding repartitioning logic along with coalesce logic to backup output > -- > > Key: KUDU-3309 > URL: https://issues.apache.org/jira/browse/KUDU-3309 > Project: Kudu > Issue Type: Task > Components: backup >Affects Versions: 1.16.0 >Reporter: Minakshi Korad >Assignee: Minakshi Korad >Priority: Minor > Fix For: 1.16.0 > > > * Adding repartition logic along with coalesce to output files > * Both the above params are optional. > * Coalesce takes precedence over repartition if both of them are defined. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (KUDU-3337) Tool to manually create cmeta files
Attila Bukor created KUDU-3337: -- Summary: Tool to manually create cmeta files Key: KUDU-3337 URL: https://issues.apache.org/jira/browse/KUDU-3337 Project: Kudu Issue Type: New Feature Reporter: Attila Bukor Assignee: Attila Bukor Power outages can lead to empty cmeta files on XFS (KUDU-2195), and sometimes all replicas are affected. By checking the ksck output and the WAL dumps it's possible to reconstruct how the cmeta should look like, except for the voted_for part, but that isn't required to be able to bootstrap a tablet, so a tool to manually create a cmeta file would be useful to recover such tablets. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (KUDU-1921) Add ability for clients to require authentication/encryption
[ https://issues.apache.org/jira/browse/KUDU-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-1921. Fix Version/s: 1.16.0 Resolution: Fixed > Add ability for clients to require authentication/encryption > > > Key: KUDU-1921 > URL: https://issues.apache.org/jira/browse/KUDU-1921 > Project: Kudu > Issue Type: Improvement > Components: client, security >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Assignee: Attila Bukor >Priority: Critical > Labels: roadmap-candidate > Fix For: 1.16.0 > > > Currently, the clients always operate in "optional" mode for authentication > and encryption. This means that they are vulnerable to downgrade attacks by a > MITM. We should provide APIs so that clients can be configured to prohibit > downgrade when connecting to clusters they know to be secure. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (KUDU-3315) Support encryption in Env layer
[ https://issues.apache.org/jira/browse/KUDU-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3315. Fix Version/s: n/a Resolution: Fixed > Support encryption in Env layer > --- > > Key: KUDU-3315 > URL: https://issues.apache.org/jira/browse/KUDU-3315 > Project: Kudu > Issue Type: Sub-task > Components: fs, security >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: n/a > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3331) Integrate encrypted Env
Attila Bukor created KUDU-3331: -- Summary: Integrate encrypted Env Key: KUDU-3331 URL: https://issues.apache.org/jira/browse/KUDU-3331 Project: Kudu Issue Type: Sub-task Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3316) Store encrypted encryption keys in encrypted files
Attila Bukor created KUDU-3316: -- Summary: Store encrypted encryption keys in encrypted files Key: KUDU-3316 URL: https://issues.apache.org/jira/browse/KUDU-3316 Project: Kudu Issue Type: Sub-task Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3315) Support encryption in Env layer
Attila Bukor created KUDU-3315: -- Summary: Support encryption in Env layer Key: KUDU-3315 URL: https://issues.apache.org/jira/browse/KUDU-3315 Project: Kudu Issue Type: Sub-task Components: fs, security Reporter: Attila Bukor Assignee: Attila Bukor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3293) Confusing Ranger audit logs
Attila Bukor created KUDU-3293: -- Summary: Confusing Ranger audit logs Key: KUDU-3293 URL: https://issues.apache.org/jira/browse/KUDU-3293 Project: Kudu Issue Type: Bug Components: authz, ranger Reporter: Attila Bukor When a client opens a table, the master authorizes DML actions on it and returns a list of allowed actions to the client which is then forwarded to the tablet servers so that it doesn't have to talk to Ranger. This significantly reduces the number of requests to Ranger, but it messes with the audit logs, as it will show ALL, and if it's denied, then also SELECT, UPDATE, INSERT and DELETE for each open table request, even if the client is only doing one of these things. which can be confusing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-1884) Support customizing Kerberos principal
[ https://issues.apache.org/jira/browse/KUDU-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-1884. Fix Version/s: 1.15.0 Resolution: Fixed > Support customizing Kerberos principal > -- > > Key: KUDU-1884 > URL: https://issues.apache.org/jira/browse/KUDU-1884 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Assignee: Attila Bukor >Priority: Major > Labels: config > Fix For: 1.15.0 > > > Currently (aiming for Kudu 1.3) the Kerberos principal is hardcoded to > 'kudu/_HOST'. While we have a flag to change it on the server, it isn't > effective because we have no way to configure it on clients. > The difficulty here is that there is currently no concept of Kudu client > configuration files, so while we could theoretically add a new API to > KuduClientBuilder, this would then mean that everyone embedding the API would > have to add a new configuration, etc. We should consider a more generic > key/value "connection properties" (as used by JDBC URLs) or some concept of a > client configuration file (with an API to specify where to find it, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3282) TestScanToken.testBuildTokensWithDownTabletServer is flaky
Attila Bukor created KUDU-3282:
--
Summary: TestScanToken.testBuildTokensWithDownTabletServer is flaky
Key: KUDU-3282
URL: https://issues.apache.org/jira/browse/KUDU-3282
Project: Kudu
Issue Type: Bug
Reporter: Attila Bukor
Doesn't show up on flaky dashboard, but it happened at least once:
{code}
1) testBuildTokensWithDownTabletServer(org.apache.kudu.client.TestScanToken)
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at
org.apache.kudu.client.TestScanToken.testBuildTokensWithDownTabletServer(TestScanToken.java:734)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KUDU-3281) ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps is flaky
Attila Bukor created KUDU-3281:
--
Summary: ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps
is flaky
Key: KUDU-3281
URL: https://issues.apache.org/jira/browse/KUDU-3281
Project: Kudu
Issue Type: Bug
Reporter: Attila Bukor
Flaky dashboard currently shows 2.25% flaky rate on txn_participant-itest
because of this failure:
{code}
ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/0:
test_workload.cc:298] Already present: key already present
@ 0x7f928a0be1f1 google::(anonymous namespace)::FailureSignalHandler()
at ??:0
@ 0x7f9287bf2fb7 gsignal at ??:0
@ 0x7f9287bf4921 abort at ??:0
@ 0x7f9290443bb8 kudu::TestWorkload::GetNumberOfErrors() at ??:0
@ 0x7f9290445599 kudu::TestWorkload::WriteThread() at ??:0
@ 0x7f928b3bb8df execute_native_thread_routine at ??:0
@ 0x7f928bb4b6db start_thread at ??:0
@ 0x7f9287cd571f clone at ??:0
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3274) Buffer overflow in SASL
[
https://issues.apache.org/jira/browse/KUDU-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326361#comment-17326361
]
Attila Bukor commented on KUDU-3274:
This might be the cause: https://github.com/cyrusimap/cyrus-sasl/issues/587
According to a comment, CVE-2019-19906 was assigned to that issue, which
unfortunately [won't be fixed in
RHEL7|https://access.redhat.com/security/cve/cve-2019-19906], so this
suppression would need to stay until we can upgrade our test infra to RHEL8.
> Buffer overflow in SASL
> ---
>
> Key: KUDU-3274
> URL: https://issues.apache.org/jira/browse/KUDU-3274
> Project: Kudu
> Issue Type: Bug
>Reporter: Attila Bukor
>Priority: Major
>
> There seems to be a buffer overflow in SASL under certain conditions ("Server
> not found in Kerberos database" error):
> {code}
> ==9298==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x60e3e2d6 at pc 0x00530bf4 bp 0x7f8eb50ad0f0 sp 0x7f8eb50ac8a0
> READ of size 151 at 0x60e3e2d6 thread T88 (client-negotiat)
> #0 0x530bf3 in __interceptor_strlen.part.35
> sanitizer_common/sanitizer_common_interceptors.inc:365:5
> #1 0x7f8ee6ad9ee8 in std::basic_ostream >&
> std::operator<< >(std::basic_ostream std::char_traits >&, char const*)
> (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x113ee8)
> #2 0x7f8eeb7c9c9b in kudu::rpc::SaslLogCallback(void*, int, char const*)
> ../src/kudu/rpc/sasl_common.cc:102:29
> #3 0x7f8eeb30241c in sasl_seterror
> (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x1441c)
> #4 0x7f8edd8f143d in _init
> (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x243d)
> #5 0x7f8edd8f2452 in _init
> (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x3452)
> #6 0x7f8eeb2f7844 in sasl_client_step
> (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9844)
> #7 0x7f8eeb2f7bc5 in sasl_client_start
> (/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9bc5)
> #8 0x7f8eeb678679 in
> kudu::rpc::ClientNegotiation::SendSaslInitiate()::$_1::operator()() const
> ../src/kudu/rpc/client_negotiation.cc:594:14
> #9 0x7f8eeb67831c in std::_Function_handler kudu::rpc::ClientNegotiation::SendSaslInitiate()::$_1>::_M_invoke(std::_Any_data
> const&) ../../../include/c++/8/bits/std_function.h:282:9
> #10 0x7f8ef3b28220 in std::function::operator()() const
> ../../../include/c++/8/bits/std_function.h:687:14
> #11 0x7f8eeb7c5840 in kudu::rpc::WrapSaslCall(sasl_conn*,
> std::function const&, char const*)
> ../src/kudu/rpc/sasl_common.cc:341:12
> #12 0x7f8eeb67363b in kudu::rpc::ClientNegotiation::SendSaslInitiate()
> ../src/kudu/rpc/client_negotiation.cc:593:20
> #13 0x7f8eeb66e0c7 in
> kudu::rpc::ClientNegotiation::AuthenticateBySasl(kudu::faststring*,
> std::unique_ptr std::default_delete >*)
> ../src/kudu/rpc/client_negotiation.cc:523:14
> #14 0x7f8eeb667b99 in
> kudu::rpc::ClientNegotiation::Negotiate(std::unique_ptr std::default_delete >*)
> ../src/kudu/rpc/client_negotiation.cc:220:7
> #15 0x7f8eeb715027 in
> kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag,
> kudu::TriStateFlag, kudu::MonoTime, std::unique_ptr std::default_delete >*)
> ../src/kudu/rpc/negotiation.cc:218:3
> #16 0x7f8eeb712095 in
> kudu::rpc::Negotiation::RunNegotiation(scoped_refptr
> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)
> ../src/kudu/rpc/negotiation.cc:295:9
> #17 0x7f8eeb74d4ad in
> kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr
> const&)::$_1::operator()() const ../src/kudu/rpc/reactor.cc:614:3
> #18 0x7f8eeb74d06c in std::_Function_handler kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr
> const&)::$_1>::_M_invoke(std::_Any_data const&)
> ../../../include/c++/8/bits/std_function.h:297:2
> #19 0x71b760 in std::function::operator()() const
> ../../../include/c++/8/bits/std_function.h:687:14
> #20 0x7f8ee917d03d in kudu::ThreadPool::DispatchThread()
> ../src/kudu/util/threadpool.cc:669:7
> #21 0x7f8ee91817dc in kudu::ThreadPool::CreateThread()::$_1::operator()()
> const ../src/kudu/util/threadpool.cc:742:48
> #22 0x7f8ee918162c in std::_Function_handler kudu::ThreadPool::CreateThread()::$_1>::_M_invoke(std::_Any_data const&)
> ../../../include/c++/8/bits/std_function.h:297:2
> #23 0x71b760 in std::function::operator()() const
> ../../../include/c++/8/bits/std_function.h:687:14
> #24 0x7f8ee915660a in kudu::Thread::SuperviseThread(void*)
> ../src/kudu/util/thread.cc:674:3
> #25 0x7f8eec6106da in start_thread
> (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
> #26 0x7f8ee64de71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
> 0x60e3e2d6 is
[jira] [Created] (KUDU-3274) Buffer overflow in SASL
Attila Bukor created KUDU-3274:
--
Summary: Buffer overflow in SASL
Key: KUDU-3274
URL: https://issues.apache.org/jira/browse/KUDU-3274
Project: Kudu
Issue Type: Bug
Reporter: Attila Bukor
There seems to be a buffer overflow in SASL under certain conditions ("Server
not found in Kerberos database" error):
{code}
==9298==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e3e2d6
at pc 0x00530bf4 bp 0x7f8eb50ad0f0 sp 0x7f8eb50ac8a0
READ of size 151 at 0x60e3e2d6 thread T88 (client-negotiat)
#0 0x530bf3 in __interceptor_strlen.part.35
sanitizer_common/sanitizer_common_interceptors.inc:365:5
#1 0x7f8ee6ad9ee8 in std::basic_ostream >&
std::operator<< >(std::basic_ostream >&, char const*)
(/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x113ee8)
#2 0x7f8eeb7c9c9b in kudu::rpc::SaslLogCallback(void*, int, char const*)
../src/kudu/rpc/sasl_common.cc:102:29
#3 0x7f8eeb30241c in sasl_seterror
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x1441c)
#4 0x7f8edd8f143d in _init
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x243d)
#5 0x7f8edd8f2452 in _init
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/sasl2/libgssapiv2.so+0x3452)
#6 0x7f8eeb2f7844 in sasl_client_step
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9844)
#7 0x7f8eeb2f7bc5 in sasl_client_start
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0x9bc5)
#8 0x7f8eeb678679 in
kudu::rpc::ClientNegotiation::SendSaslInitiate()::$_1::operator()() const
../src/kudu/rpc/client_negotiation.cc:594:14
#9 0x7f8eeb67831c in std::_Function_handler::_M_invoke(std::_Any_data
const&) ../../../include/c++/8/bits/std_function.h:282:9
#10 0x7f8ef3b28220 in std::function::operator()() const
../../../include/c++/8/bits/std_function.h:687:14
#11 0x7f8eeb7c5840 in kudu::rpc::WrapSaslCall(sasl_conn*, std::function const&, char const*) ../src/kudu/rpc/sasl_common.cc:341:12
#12 0x7f8eeb67363b in kudu::rpc::ClientNegotiation::SendSaslInitiate()
../src/kudu/rpc/client_negotiation.cc:593:20
#13 0x7f8eeb66e0c7 in
kudu::rpc::ClientNegotiation::AuthenticateBySasl(kudu::faststring*,
std::unique_ptr >*)
../src/kudu/rpc/client_negotiation.cc:523:14
#14 0x7f8eeb667b99 in
kudu::rpc::ClientNegotiation::Negotiate(std::unique_ptr >*)
../src/kudu/rpc/client_negotiation.cc:220:7
#15 0x7f8eeb715027 in
kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag,
kudu::TriStateFlag, kudu::MonoTime, std::unique_ptr >*)
../src/kudu/rpc/negotiation.cc:218:3
#16 0x7f8eeb712095 in
kudu::rpc::Negotiation::RunNegotiation(scoped_refptr
const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)
../src/kudu/rpc/negotiation.cc:295:9
#17 0x7f8eeb74d4ad in
kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr
const&)::$_1::operator()() const ../src/kudu/rpc/reactor.cc:614:3
#18 0x7f8eeb74d06c in std::_Function_handler
const&)::$_1>::_M_invoke(std::_Any_data const&)
../../../include/c++/8/bits/std_function.h:297:2
#19 0x71b760 in std::function::operator()() const
../../../include/c++/8/bits/std_function.h:687:14
#20 0x7f8ee917d03d in kudu::ThreadPool::DispatchThread()
../src/kudu/util/threadpool.cc:669:7
#21 0x7f8ee91817dc in kudu::ThreadPool::CreateThread()::$_1::operator()()
const ../src/kudu/util/threadpool.cc:742:48
#22 0x7f8ee918162c in std::_Function_handler::_M_invoke(std::_Any_data const&)
../../../include/c++/8/bits/std_function.h:297:2
#23 0x71b760 in std::function::operator()() const
../../../include/c++/8/bits/std_function.h:687:14
#24 0x7f8ee915660a in kudu::Thread::SuperviseThread(void*)
../src/kudu/util/thread.cc:674:3
#25 0x7f8eec6106da in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#26 0x7f8ee64de71e in clone (/lib/x86_64-linux-gnu/libc.so.6+0x12171e)
0x60e3e2d6 is located 0 bytes to the right of 150-byte region
[0x60e3e240,0x60e3e2d6)
allocated by thread T88 (client-negotiat) here:
#0 0x5a4bb8 in malloc
/home/abukor/src/kudu/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145:3
#1 0x7f8eeb2fa1df in _buf_alloc
(/tmp/dist-test-taskexUtyr/build/dist-test-system-libs/libsasl2.so.3+0xc1df)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-1884) Support customizing Kerberos principal
[ https://issues.apache.org/jira/browse/KUDU-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-1884: -- Assignee: Attila Bukor > Support customizing Kerberos principal > -- > > Key: KUDU-1884 > URL: https://issues.apache.org/jira/browse/KUDU-1884 > Project: Kudu > Issue Type: Improvement > Components: security >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Assignee: Attila Bukor >Priority: Major > Labels: config > > Currently (aiming for Kudu 1.3) the Kerberos principal is hardcoded to > 'kudu/_HOST'. While we have a flag to change it on the server, it isn't > effective because we have no way to configure it on clients. > The difficulty here is that there is currently no concept of Kudu client > configuration files, so while we could theoretically add a new API to > KuduClientBuilder, this would then mean that everyone embedding the API would > have to add a new configuration, etc. We should consider a more generic > key/value "connection properties" (as used by JDBC URLs) or some concept of a > client configuration file (with an API to specify where to find it, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KUDU-3230) SASL PROTO name is hardcode to "kudu"
[ https://issues.apache.org/jira/browse/KUDU-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor closed KUDU-3230. -- Resolution: Duplicate > SASL PROTO name is hardcode to "kudu" > - > > Key: KUDU-3230 > URL: https://issues.apache.org/jira/browse/KUDU-3230 > Project: Kudu > Issue Type: Task >Reporter: Redriver >Assignee: Attila Bukor >Priority: Major > > Kudu expected users to create "kudu" user to deploy kudu-master and > kudu-tserver, and > hard code "kudu" for sasl proto name, as a result, if I used other user > account to deploy kudu, and my kerberos keytab does not use kudu as > principal, it failed. > It should allow users to customize this name. If no customization, "kudu" is > the default name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3230) SASL PROTO name is hardcode to "kudu"
[ https://issues.apache.org/jira/browse/KUDU-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3230: -- Assignee: Attila Bukor > SASL PROTO name is hardcode to "kudu" > - > > Key: KUDU-3230 > URL: https://issues.apache.org/jira/browse/KUDU-3230 > Project: Kudu > Issue Type: Task >Reporter: Redriver >Assignee: Attila Bukor >Priority: Major > > Kudu expected users to create "kudu" user to deploy kudu-master and > kudu-tserver, and > hard code "kudu" for sasl proto name, as a result, if I used other user > account to deploy kudu, and my kerberos keytab does not use kudu as > principal, it failed. > It should allow users to customize this name. If no customization, "kudu" is > the default name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3207) Standardize RSA private key format
[
https://issues.apache.org/jira/browse/KUDU-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3207.
Fix Version/s: 1.15.0
Resolution: Fixed
> Standardize RSA private key format
> --
>
> Key: KUDU-3207
> URL: https://issues.apache.org/jira/browse/KUDU-3207
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Minor
> Labels: OpenSSL, security
> Fix For: 1.15.0
>
>
> Currently, Kudu stores RSA private keys in PEM format using
> PEM_write_bio_RSAPrivateKey(), which doesn't specify the format in which the
> key is stored. It expects it to be PKCS #1 (BEGIN/END RSA PRIVATE KEY), but
> it seems there are some OpenSSL versions (CryptoComply) that use PKCS #8
> instead (BEGIN/END PRIVATE KEY). {{CryptoTest.RsaPrivateKeyInputOutputPEM}}
> fails due to this, as it compares the private key to an expected string,
> which is in PKCS #1 format. The read functions are explicitly said to handle
> any known format, so this shouldn't cause any issues, but it would still be
> nice to standardize on a single format (probably PKCS #8).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3207) Standardize RSA private key format
[
https://issues.apache.org/jira/browse/KUDU-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3207:
--
Assignee: Attila Bukor
> Standardize RSA private key format
> --
>
> Key: KUDU-3207
> URL: https://issues.apache.org/jira/browse/KUDU-3207
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Minor
> Labels: OpenSSL, security
>
> Currently, Kudu stores RSA private keys in PEM format using
> PEM_write_bio_RSAPrivateKey(), which doesn't specify the format in which the
> key is stored. It expects it to be PKCS #1 (BEGIN/END RSA PRIVATE KEY), but
> it seems there are some OpenSSL versions (CryptoComply) that use PKCS #8
> instead (BEGIN/END PRIVATE KEY). {{CryptoTest.RsaPrivateKeyInputOutputPEM}}
> fails due to this, as it compares the private key to an expected string,
> which is in PKCS #1 format. The read functions are explicitly said to handle
> any known format, so this shouldn't cause any issues, but it would still be
> nice to standardize on a single format (probably PKCS #8).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-3236) Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in Kerberos database
[
https://issues.apache.org/jira/browse/KUDU-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3236.
Fix Version/s: n/a
Resolution: Not A Bug
> Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in
> Kerberos database
> -
>
> Key: KUDU-3236
> URL: https://issues.apache.org/jira/browse/KUDU-3236
> Project: Kudu
> Issue Type: Task
> Components: authz
>Affects Versions: 1.10.0
> Environment: Centos7.7 kudu-1.10.0-cdh6.3.0
>Reporter: sun
>Priority: Major
> Fix For: n/a
>
>
> hi everybody,
> When I started Kerberos for kudu according to the official documents, I found
> that the result was not satisfactory:(:(. The kudu is containerized and
> installed on the big data platform. After I configured Kerberos according to
> the official documents, I found that tserver could not be registered in the
> master。What I expect is krbtgt/bigdata.xueliang@bigdata.xueliang.com ，but
> got krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com.:( . could anybody
> give me some tips? thanks in advance.
> The kudu master.gflagfile:
> --log_dir=/opt/java/kudu/master/logs
> --fs_wal_dir=/opt/java/kudu/master/wal
> --fs_data_dirs=/opt/java/kudu/master/data/1,/opt/java/kudu/master/data/2,/opt/java/kudu/master/data/3
> --raft_get_node_instance_timeout_ms=30
> --webserver_port=8051
> --master_addresses=
> service-kudu-xueliang-master-0:7051,service-kudu-xueliang-master-1:7051,service-kudu-xueliang-master-2:7051
> --block_cache_capacity_mb=512
> --memory_limit_hard_bytes=0
> --rpc_service_queue_length=50
> --max_clock_sync_error_usec=1000
> --maintenance_manager_num_threads=1
> --webserver_doc_root=/opt/java/kudu/www
> --rpc_encryption=required
> --rpc_authentication=required
> --trusted_subnets=0.0.0.0/0
> --keytab_file=/opt/java/kudu/conf/kuduxueliang.keytab
> The kudu tserver.gflagfile:
> --log_dir=/opt/java/kudu/tserver/logs
> --fs_wal_dir=/opt/java/kudu/tserver/wal
> --fs_data_dirs=/opt/java/kudu/tserver/data/1
> --webserver_port=8050
> --tserver_master_addrs=
> service-kudu-xueliang-master-0:7051,service-kudu-xueliang-master-1:7051,service-kudu-xueliang-master-2:7051
> --block_cache_capacity_mb=512
> --memory_limit_hard_bytes=26843545600
> --rpc_service_queue_length=50
> --max_clock_sync_error_usec=1000
> --maintenance_manager_num_threads=1
> --webserver_doc_root=/opt/java/kudu/www
> --rpc_encryption=required
> --rpc_authentication=required
> --trusted_subnets=0.0.0.0/0
> --keytab_file=/opt/java/kudu-1.10.0-cdh6.3.0/conf/kuduxueliang.keytab
> the krb5.conf:
> [logging]
> default = FILE:/var/log/krb5libs.log
> kdc = FILE:/var/log/krb5kdc.log
> admin_server = FILE:/var/log/kadmind.log
> [libdefaults]
> default_realm = BIGDATA.XUELIANG.COM
> dns_lookup_realm = true
> dns_lookup_kdc = true
> rdns = true
> ticket_lifetime = 24h
> forwardable = true
> udp_preference_limit = 0
> [realms]
> BIGDATA.XUELIANG.COM = {
> kdc = hdh136.bigdata.xueliang.com:88
> master_kdc = hdh136.bigdata.xueliang.com:88
> admin_server = hdh136.bigdata.xueliang.com:749
> default_domain = bigdata.xueliang.com
> pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
> pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
> }
> [domain_realm]
> .bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> hdh136.bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> [dbmodules]
> BIGDATA.XUELIANG.COM = {
> db_library = ipadb.so
> }
> the kudu tserver log:
> heartbeater.cc:566] Failed to heartbeat to
> service-kudu-xueliang-master-1:7051 (7471 consecutive failures): Not
> authorized: Failed to ping master at service-kudu-xueliang-master-1:7051:
> Client connection negotiation failed: client connection to 10.103.68.4:7051:
> Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in
> Kerberos database .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3236) Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in Kerberos database
[
https://issues.apache.org/jira/browse/KUDU-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269223#comment-17269223
]
Attila Bukor commented on KUDU-3236:
It seems this is a misconfiguration on the Kerberos side rather than a Kudu
bug. This JIRA serves to track bugs and features and not as a helpdesk. If
you're sure that Kerberos itself is set up correctly, I suggest reaching out to
the u...@kudu.apache.org mailing list instead of this JIRA.
> Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in
> Kerberos database
> -
>
> Key: KUDU-3236
> URL: https://issues.apache.org/jira/browse/KUDU-3236
> Project: Kudu
> Issue Type: Task
> Components: authz
>Affects Versions: 1.10.0
> Environment: Centos7.7 kudu-1.10.0-cdh6.3.0
>Reporter: sun
>Priority: Major
>
> hi everybody,
> When I started Kerberos for kudu according to the official documents, I found
> that the result was not satisfactory:(:(. The kudu is containerized and
> installed on the big data platform. After I configured Kerberos according to
> the official documents, I found that tserver could not be registered in the
> master。What I expect is krbtgt/bigdata.xueliang@bigdata.xueliang.com ，but
> got krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com.:( . could anybody
> give me some tips? thanks in advance.
> The kudu master.gflagfile:
> --log_dir=/opt/java/kudu/master/logs
> --fs_wal_dir=/opt/java/kudu/master/wal
> --fs_data_dirs=/opt/java/kudu/master/data/1,/opt/java/kudu/master/data/2,/opt/java/kudu/master/data/3
> --raft_get_node_instance_timeout_ms=30
> --webserver_port=8051
> --master_addresses=
> service-kudu-xueliang-master-0:7051,service-kudu-xueliang-master-1:7051,service-kudu-xueliang-master-2:7051
> --block_cache_capacity_mb=512
> --memory_limit_hard_bytes=0
> --rpc_service_queue_length=50
> --max_clock_sync_error_usec=1000
> --maintenance_manager_num_threads=1
> --webserver_doc_root=/opt/java/kudu/www
> --rpc_encryption=required
> --rpc_authentication=required
> --trusted_subnets=0.0.0.0/0
> --keytab_file=/opt/java/kudu/conf/kuduxueliang.keytab
> The kudu tserver.gflagfile:
> --log_dir=/opt/java/kudu/tserver/logs
> --fs_wal_dir=/opt/java/kudu/tserver/wal
> --fs_data_dirs=/opt/java/kudu/tserver/data/1
> --webserver_port=8050
> --tserver_master_addrs=
> service-kudu-xueliang-master-0:7051,service-kudu-xueliang-master-1:7051,service-kudu-xueliang-master-2:7051
> --block_cache_capacity_mb=512
> --memory_limit_hard_bytes=26843545600
> --rpc_service_queue_length=50
> --max_clock_sync_error_usec=1000
> --maintenance_manager_num_threads=1
> --webserver_doc_root=/opt/java/kudu/www
> --rpc_encryption=required
> --rpc_authentication=required
> --trusted_subnets=0.0.0.0/0
> --keytab_file=/opt/java/kudu-1.10.0-cdh6.3.0/conf/kuduxueliang.keytab
> the krb5.conf:
> [logging]
> default = FILE:/var/log/krb5libs.log
> kdc = FILE:/var/log/krb5kdc.log
> admin_server = FILE:/var/log/kadmind.log
> [libdefaults]
> default_realm = BIGDATA.XUELIANG.COM
> dns_lookup_realm = true
> dns_lookup_kdc = true
> rdns = true
> ticket_lifetime = 24h
> forwardable = true
> udp_preference_limit = 0
> [realms]
> BIGDATA.XUELIANG.COM = {
> kdc = hdh136.bigdata.xueliang.com:88
> master_kdc = hdh136.bigdata.xueliang.com:88
> admin_server = hdh136.bigdata.xueliang.com:749
> default_domain = bigdata.xueliang.com
> pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
> pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
> }
> [domain_realm]
> .bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> hdh136.bigdata.xueliang.com = BIGDATA.XUELIANG.COM
> [dbmodules]
> BIGDATA.XUELIANG.COM = {
> db_library = ipadb.so
> }
> the kudu tserver log:
> heartbeater.cc:566] Failed to heartbeat to
> service-kudu-xueliang-master-1:7051 (7471 consecutive failures): Not
> authorized: Failed to ping master at service-kudu-xueliang-master-1:7051:
> Client connection negotiation failed: client connection to 10.103.68.4:7051:
> Server krbtgt/xueliang.svc.cluster.lo...@bigdata.xueliang.com not found in
> Kerberos database .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3133) Poor TLS cypher performance on Java 8
[
https://issues.apache.org/jira/browse/KUDU-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3133:
--
Assignee: wangningito
> Poor TLS cypher performance on Java 8
> -
>
> Key: KUDU-3133
> URL: https://issues.apache.org/jira/browse/KUDU-3133
> Project: Kudu
> Issue Type: Bug
> Components: perf, security
>Reporter: Grant Henke
>Assignee: wangningito
>Priority: Major
>
> It was reported a while back that Kudu TLS doesn't perform well on Java 8 due
> to a potential GCM cypher bug or bad selection via `PREFERRED_CIPHER_SUITES`.
> It would be good to get to the bottom of this and fix it or document the
> recommendation to use Java 11.
> Here was the observed impact:
> {code}
> ./bin/ycsb run kudu -P workloads/workloadc -p operationcount=1
> -threads 64 -p kudu_num_clients=16 -s -p fieldlength=1 -p
> kudu_table_num_replicas=1
> java 7u75 with master:
> 0205 11:18:48.647920 (+28us) server_negotiation.cc:581] Negotiated
> TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~12k rows/sec
> java 8_141 with master:
> 0205 11:17:45.977107 (+31us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> 6k rows/sec
> java 8_141 with GCM-based codecs removed from Negotiator.java
> 0205 11:25:33.268689 (+40us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AES(256) Mac=SHA384
> ~6k rows/sec
> java 8_141 with only AES256-SHA remaining in Negotiator.java:
> "TLS_RSA_WITH_AES_256_CBC_SHA" )
> 0205 11:32:07.674860 (+44us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~9.5k rows/sec
> java 11 with master:
> 0205 11:17:01.416066 (+41us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> ~19k rows/sec
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3226) Validate List of Masters for kudu ksck
[ https://issues.apache.org/jira/browse/KUDU-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3226: --- Labels: beginner newbie trivial (was: newbie) > Validate List of Masters for kudu ksck > -- > > Key: KUDU-3226 > URL: https://issues.apache.org/jira/browse/KUDU-3226 > Project: Kudu > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: beginner, newbie, trivial > > I recently accidentally included a list of masters where I fat-fingered the > host names and included the same node twice. > I got some stack trace and an error message about duplicate keys inserted > into a map. I eventually figured out what I did wrong, but the error > condition was not super helpful. > Please add an early validation step that explicitly captures this conditions > and provides a helpful error message that includes the name of the host name > which was duplicated on the command line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3226) Validate List of Masters for kudu ksck
[ https://issues.apache.org/jira/browse/KUDU-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3226: --- Labels: newbie (was: newb) > Validate List of Masters for kudu ksck > -- > > Key: KUDU-3226 > URL: https://issues.apache.org/jira/browse/KUDU-3226 > Project: Kudu > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: newbie > > I recently accidentally included a list of masters where I fat-fingered the > host names and included the same node twice. > I got some stack trace and an error message about duplicate keys inserted > into a map. I eventually figured out what I did wrong, but the error > condition was not super helpful. > Please add an early validation step that explicitly captures this conditions > and provides a helpful error message that includes the name of the host name > which was duplicated on the command line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3226) Validate List of Masters for kudu ksck
[ https://issues.apache.org/jira/browse/KUDU-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3226: --- Labels: newb (was: ) > Validate List of Masters for kudu ksck > -- > > Key: KUDU-3226 > URL: https://issues.apache.org/jira/browse/KUDU-3226 > Project: Kudu > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: newb > > I recently accidentally included a list of masters where I fat-fingered the > host names and included the same node twice. > I got some stack trace and an error message about duplicate keys inserted > into a map. I eventually figured out what I did wrong, but the error > condition was not super helpful. > Please add an early validation step that explicitly captures this conditions > and provides a helpful error message that includes the name of the host name > which was duplicated on the command line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3210) Support FIPS approved mode
Attila Bukor created KUDU-3210: -- Summary: Support FIPS approved mode Key: KUDU-3210 URL: https://issues.apache.org/jira/browse/KUDU-3210 Project: Kudu Issue Type: Improvement Reporter: Attila Bukor Assignee: Attila Bukor FIPS 140-2 is a standard used to approve cryptographic modules. Some versions of OpenSSL support a "FIPS mode" where only approved algorithms and key sizes are enabled. Kudu should be able to run when FIPS mode is enabled and should provide a way for admins to require that FIPS mode is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3207) Standardize RSA private key format
Attila Bukor created KUDU-3207:
--
Summary: Standardize RSA private key format
Key: KUDU-3207
URL: https://issues.apache.org/jira/browse/KUDU-3207
Project: Kudu
Issue Type: Improvement
Reporter: Attila Bukor
Currently, Kudu stores RSA private keys in PEM format using
PEM_write_bio_RSAPrivateKey(), which doesn't specify the format in which the
key is stored. It expects it to be PKCS #1 (BEGIN/END RSA PRIVATE KEY), but it
seems there are some OpenSSL versions (CryptoComply) that use PKCS #8 instead
(BEGIN/END PRIVATE KEY). {{CryptoTest.RsaPrivateKeyInputOutputPEM}} fails due
to this, as it compares the private key to an expected string, which is in PKCS
#1 format. The read functions are explicitly said to handle any known format,
so this shouldn't cause any issues, but it would still be nice to standardize
on a single format (probably PKCS #8).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-3091) Support ownership privilege with Ranger
[ https://issues.apache.org/jira/browse/KUDU-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3091. Fix Version/s: 1.13.0 Resolution: Fixed > Support ownership privilege with Ranger > --- > > Key: KUDU-3091 > URL: https://issues.apache.org/jira/browse/KUDU-3091 > Project: Kudu > Issue Type: Task > Components: authz, ranger, security >Reporter: Hao Hao >Assignee: Attila Bukor >Priority: Major > Fix For: 1.13.0 > > > Currently, ownership privilege in Ranger is not available as Kudu has no > concept of owner, and does not store owner information internally. It would > be nice to enable it once Kudu introduces owner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3185) Add table owner to web UI and CLI outputs
[
https://issues.apache.org/jira/browse/KUDU-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor updated KUDU-3185:
---
Fix Version/s: 1.13.0
Resolution: Fixed
Status: Resolved (was: In Review)
> Add table owner to web UI and CLI outputs
> -
>
> Key: KUDU-3185
> URL: https://issues.apache.org/jira/browse/KUDU-3185
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Minor
> Fix For: 1.13.0
>
>
> Checking the owner of a table is impossible with our tooling, it can only be
> done from code using the client API. It would make sense to list it in {{kudu
> table describe}} and in the {{/tables}} page of the web UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-1422) java client: Error buffer size is hard-coded
[ https://issues.apache.org/jira/browse/KUDU-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-1422. Fix Version/s: 1.13.0 Resolution: Fixed > java client: Error buffer size is hard-coded > > > Key: KUDU-1422 > URL: https://issues.apache.org/jira/browse/KUDU-1422 > Project: Kudu > Issue Type: Bug > Components: client >Affects Versions: 0.8.0 >Reporter: Mike Percy >Priority: Major > Labels: trivial > Fix For: 1.13.0 > > > The java client has an error buffer for auto flush. The error buffer size is > hard coded to 1000 errors; this should be tunable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3116) Enhance operation accumulators in KuduContext to track operations per table
[ https://issues.apache.org/jira/browse/KUDU-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3116. Fix Version/s: 1.13.0 Resolution: Fixed > Enhance operation accumulators in KuduContext to track operations per table > --- > > Key: KUDU-3116 > URL: https://issues.apache.org/jira/browse/KUDU-3116 > Project: Kudu > Issue Type: Improvement > Components: java, spark >Reporter: Brian McDevitt >Assignee: Brian McDevitt >Priority: Minor > Fix For: 1.13.0 > > > The KuduContext tracks row count metrics in LongAccumulators, but often the > same KuduContext is used for multiple tables causing the LongAccumulators to > report incorrect row count values. Having those metrics tracked per table > would improve the results/logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3116) Enhance operation accumulators in KuduContext to track operations per table
[ https://issues.apache.org/jira/browse/KUDU-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3116: -- Assignee: Brian McDevitt > Enhance operation accumulators in KuduContext to track operations per table > --- > > Key: KUDU-3116 > URL: https://issues.apache.org/jira/browse/KUDU-3116 > Project: Kudu > Issue Type: Improvement > Components: java, spark >Reporter: Brian McDevitt >Assignee: Brian McDevitt >Priority: Minor > > The KuduContext tracks row count metrics in LongAccumulators, but often the > same KuduContext is used for multiple tables causing the LongAccumulators to > report incorrect row count values. Having those metrics tracked per table > would improve the results/logging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3185) Add table owner to web UI and CLI outputs
[
https://issues.apache.org/jira/browse/KUDU-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor updated KUDU-3185:
---
Code Review: https://gerrit.cloudera.org/c/16394/
> Add table owner to web UI and CLI outputs
> -
>
> Key: KUDU-3185
> URL: https://issues.apache.org/jira/browse/KUDU-3185
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Minor
>
> Checking the owner of a table is impossible with our tooling, it can only be
> done from code using the client API. It would make sense to list it in {{kudu
> table describe}} and in the {{/tables}} page of the web UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3185) Add table owner to web UI and CLI outputs
[
https://issues.apache.org/jira/browse/KUDU-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor updated KUDU-3185:
---
Status: In Review (was: In Progress)
> Add table owner to web UI and CLI outputs
> -
>
> Key: KUDU-3185
> URL: https://issues.apache.org/jira/browse/KUDU-3185
> Project: Kudu
> Issue Type: Improvement
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Minor
>
> Checking the owner of a table is impossible with our tooling, it can only be
> done from code using the client API. It would make sense to list it in {{kudu
> table describe}} and in the {{/tables}} page of the web UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-2252) Design and implement native encryption at rest
[ https://issues.apache.org/jira/browse/KUDU-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-2252: -- Assignee: Attila Bukor > Design and implement native encryption at rest > -- > > Key: KUDU-2252 > URL: https://issues.apache.org/jira/browse/KUDU-2252 > Project: Kudu > Issue Type: New Feature > Components: fs, security >Reporter: Mike Percy >Assignee: Attila Bukor >Priority: Major > Labels: roadmap-candidate > > It would be beneficial for Kudu to support native encryption at rest. > While the underlying filesystem can be encrypted with Kudu on top, there are > benefits to native encryption at rest: > * With native encryption support, it may be possible to specify different > encryption keys for different objects or object trees (such as is possible > with HDFS encryption) > * It may be possible to share a single block device with other storage, > including HDFS (note that some linux setups allow for "splitting" a device) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3185) Add table owner to web UI and CLI outputs
Attila Bukor created KUDU-3185:
--
Summary: Add table owner to web UI and CLI outputs
Key: KUDU-3185
URL: https://issues.apache.org/jira/browse/KUDU-3185
Project: Kudu
Issue Type: Improvement
Reporter: Attila Bukor
Assignee: Attila Bukor
Checking the owner of a table is impossible with our tooling, it can only be
done from code using the client API. It would make sense to list it in {{kudu
table describe}} and in the {{/tables}} page of the web UI.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-3176) Backup & restore incompatibility
[ https://issues.apache.org/jira/browse/KUDU-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3176. Fix Version/s: n/a Resolution: Cannot Reproduce > Backup & restore incompatibility > > > Key: KUDU-3176 > URL: https://issues.apache.org/jira/browse/KUDU-3176 > Project: Kudu > Issue Type: Bug >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Blocker > Fix For: n/a > > > The ownership in the backup metadata introduced in KUDU-3090 seems to have > backward/forward compatibility issues as restoring a backup with > post-ownership backup tool that was created on a pre-ownership cluster with > the matching backup tool fails. Other combinations might also fail but I > haven't reproduced them so far. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3176) Backup & restore incompatibility
[
https://issues.apache.org/jira/browse/KUDU-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177155#comment-17177155
]
Attila Bukor commented on KUDU-3176:
I've been trying to reproduce this in the past couple of hours, but couldn't.
Every single combination I could think of, except for one, works fine. I tried
on different cluster versions with different kudu-backup2 versions. The only
failure I could get was when I tried to restore a backup that didn't have table
owner on a cluster that did. This failed due to trying to set the owner to an
empty string, but this is a different issue and it's solved by 4bed27234, you
just need to add {{--restoreOwner false}}. Whatever it was, I likely messed up
the backup somehow, or it was some other environmental issue. I think it is
safe to say this bug doesn't exist.
> Backup & restore incompatibility
>
>
> Key: KUDU-3176
> URL: https://issues.apache.org/jira/browse/KUDU-3176
> Project: Kudu
> Issue Type: Bug
>Reporter: Attila Bukor
>Assignee: Attila Bukor
>Priority: Blocker
>
> The ownership in the backup metadata introduced in KUDU-3090 seems to have
> backward/forward compatibility issues as restoring a backup with
> post-ownership backup tool that was created on a pre-ownership cluster with
> the matching backup tool fails. Other combinations might also fail but I
> haven't reproduced them so far.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3176) Backup & restore incompatibility
[ https://issues.apache.org/jira/browse/KUDU-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3176: --- Priority: Blocker (was: Critical) > Backup & restore incompatibility > > > Key: KUDU-3176 > URL: https://issues.apache.org/jira/browse/KUDU-3176 > Project: Kudu > Issue Type: Bug >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Blocker > > The ownership in the backup metadata introduced in KUDU-3090 seems to have > backward/forward compatibility issues as restoring a backup with > post-ownership backup tool that was created on a pre-ownership cluster with > the matching backup tool fails. Other combinations might also fail but I > haven't reproduced them so far. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-2345) Add developer docs for the python client
[
https://issues.apache.org/jira/browse/KUDU-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-2345:
--
Assignee: Attila Bukor (was: Jordan Birdsell)
> Add developer docs for the python client
>
>
> Key: KUDU-2345
> URL: https://issues.apache.org/jira/browse/KUDU-2345
> Project: Kudu
> Issue Type: Improvement
> Components: documentation, python
>Reporter: Grant Henke
>Assignee: Attila Bukor
>Priority: Minor
>
> I am far from a Python expert. Especially with Cython in the mix, so it took
> me a bit just to get started working on the kudu python client.
> We should document basic steps for how to develop and test the kudu python
> client. Including environment setup, building, and testing (running a single
> test too).
> For now I essentially boiled my work down to this:
>
> {code:java}
> cd /path/to/kudu
> cd build/debug
> make -j4
> make install
> cd /path/to/kudu/python
> git clean -fdx
> export KUDU_HOME=/path/to/kudu
> pip install -r requirements.txt
> python setup.py build_ext
> python setup.py test
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KUDU-3090) Add owner concept in Kudu
[
https://issues.apache.org/jira/browse/KUDU-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor resolved KUDU-3090.
Fix Version/s: 1.13.0
Resolution: Fixed
> Add owner concept in Kudu
> -
>
> Key: KUDU-3090
> URL: https://issues.apache.org/jira/browse/KUDU-3090
> Project: Kudu
> Issue Type: New Feature
> Components: authz, security
>Reporter: Hao Hao
>Assignee: Attila Bukor
>Priority: Major
> Labels: roadmap-candidate
> Fix For: 1.13.0
>
>
> As mentioned in the Ranger integration design doc, Ranger supports ownership
> privilege by creating a default policy that allows \{OWNER} of a resource to
> access it without creating additional policy manually. Unless Kudu actually
> has a full support for owner, ownership privilege is not possible with Ranger
> integration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KUDU-3176) Backup & restore incompatibility
Attila Bukor created KUDU-3176: -- Summary: Backup & restore incompatibility Key: KUDU-3176 URL: https://issues.apache.org/jira/browse/KUDU-3176 Project: Kudu Issue Type: Bug Reporter: Attila Bukor Assignee: Attila Bukor The ownership in the backup metadata introduced in KUDU-3090 seems to have backward/forward compatibility issues as restoring a backup with post-ownership backup tool that was created on a pre-ownership cluster with the matching backup tool fails. Other combinations might also fail but I haven't reproduced them so far. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-2345) Add developer docs for the python client
[
https://issues.apache.org/jira/browse/KUDU-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161268#comment-17161268
]
Attila Bukor commented on KUDU-2345:
[~jtbirdsell] are you still interested in documenting it? If not, I can take a
stab at it.
> Add developer docs for the python client
>
>
> Key: KUDU-2345
> URL: https://issues.apache.org/jira/browse/KUDU-2345
> Project: Kudu
> Issue Type: Improvement
> Components: documentation, python
>Reporter: Grant Henke
>Assignee: Jordan Birdsell
>Priority: Minor
>
> I am far from a Python expert. Especially with Cython in the mix, so it took
> me a bit just to get started working on the kudu python client.
> We should document basic steps for how to develop and test the kudu python
> client. Including environment setup, building, and testing (running a single
> test too).
> For now I essentially boiled my work down to this:
>
> {code:java}
> cd /path/to/kudu
> cd build/debug
> make -j4
> make install
> cd /path/to/kudu/python
> git clean -fdx
> export KUDU_HOME=/path/to/kudu
> pip install -r requirements.txt
> python setup.py build_ext
> python setup.py test
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KUDU-3167) Can not connect to all kudu-master started by docker via kudu-python
[
https://issues.apache.org/jira/browse/KUDU-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor closed KUDU-3167.
--
Resolution: Not A Bug
> Can not connect to all kudu-master started by docker via kudu-python
>
>
> Key: KUDU-3167
> URL: https://issues.apache.org/jira/browse/KUDU-3167
> Project: Kudu
> Issue Type: Bug
> Components: client, docker
>Reporter: Wang
>Priority: Major
>
> I start the docker with the reference
> [https://kudu.apache.org/docs/quickstart.html#_start_the_quickstart_cluster]
> $KUDU_QUICKSTART_IP is 10.44.192.145
> $KUDU_QUICKSTART_VERSION is 1.9.0
> kudu-python version is 1.2.0
>
> I success to start the docker with 3 masters and 5 tablets.
> Master port are 7051, 7151,7251 and 7051 is the leader.
> Via kudu-python, I can only connect the leader of kudu-master, but can't
> connect to the follower of the kudu-masters.
>
> {code:java}
> //代码占位符
> In [1]: import kudu
> In [2]: client = kudu.connect(host='10.44.192.145', port=7051)
> W0714 15:27:28.267447 5576 openssl_util.cc:102] It appears that OpenSSL has
> been previously initialized by code outside of Kudu. Please use
> kudu::client::DisableOpenSSLInitialization() to avoid potential crashes due
> to conflicting initialization.
> In [3]: client = kudu.connect(host='10.44.192.145', port=7151)
> ---
> KuduBadStatus Traceback (most recent call last)
> in ()
> > 1 client = kudu.connect(host='10.44.192.145', port=7151)
> /usr/lib64/python2.7/site-packages/kudu/__init__.pyc in connect(host, port,
> admin_timeout_ms, rpc_timeout_ms)
> 90
> 91 return Client(addresses, admin_timeout_ms=admin_timeout_ms,
> ---> 92 rpc_timeout_ms=rpc_timeout_ms)
> 93
> 94
> /usr/lib64/python2.7/site-packages/kudu/client.pyx in
> kudu.client.Client.__cinit__()
> 269 builder.default_rpc_timeout(timeout.delta)
> 270
> --> 271 check_status(builder.Build())
> 272
> 273 # A convenience
> /usr/lib64/python2.7/site-packages/kudu/errors.pyx in
> kudu.errors.check_status()
> 60 raise KuduInvalidArgument(c_message)
> 61 else:
> ---> 62 raise KuduBadStatus(status.ToString())
> KuduBadStatus: Timed out: Could not connect to the cluster:
> ConnectToClusterRpc(addrs: 10.44.192.145:7151, num_attempts: 242) passed its
> deadline: Not found: no leader found: ConnectToClusterRpc(addrs:
> 10.44.192.145:7151, num_attempts: 1)
> {code}
>
>
> ksck the cluster
> {code:java}
> //代码占位符
> [root@data-21-dev work]# docker exec -it $(docker ps -aqf
> "name=kudu-master-1") /bin/bash
> kudu@2b8a6a7d6000:/opt/kudu$ kudu cluster ksck
> kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251Master Summary
>UUID | Address | Status
> --++-
> 0fb04588705440cfae885fdf71f8fbf6 | kudu-master-2:7151 | HEALTHY
> e201d47a50624b02a8c0d200acc88e57 | kudu-master-1:7051 | HEALTHY
> fce36c217e6f4589948954cff3bf7e81 | kudu-master-3:7251 | HEALTHY Flag
> | Value | Tags | Master
> --+---++-
> use_hybrid_clock | false | hidden | all 3 server(s) checkedTablet Server
> Summary
>UUID | Address | Status | Location
> --++-+--
> 09b3f4efc69a42348ace15e80189d68c | 10.44.192.145:7150 | HEALTHY |
> 1d3057335b4149198513c71b75ee6fe8 | 10.44.192.145:7050 | HEALTHY |
> 7415842c9d5945ceaf227fdc83c26c83 | 10.44.192.145:7450 | HEALTHY |
> 901baca56619415cb74ec7ee24fd2110 | 10.44.192.145:7350 | HEALTHY |
> 95f16c59c05f45b2b86fb10b157b1826 | 10.44.192.145:7250 | HEALTHY |
> Tablet Server Location Summary
> Location | Count
> --+-
> | 5
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3167) Can not connect to all kudu-master started by docker via kudu-python
[
https://issues.apache.org/jira/browse/KUDU-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157987#comment-17157987
]
Attila Bukor commented on KUDU-3167:
This is not a bug, but the expected behavior. You need to list all masters when
connecting to a cluster instead of just one, and then the client can
automatically select the leader at all times. Also, Kudu 1.2.0 is very old, you
should use a more recent client version. 1.9.0 is also fairly old and you might
want to use a newer quickstart as well.
> Can not connect to all kudu-master started by docker via kudu-python
>
>
> Key: KUDU-3167
> URL: https://issues.apache.org/jira/browse/KUDU-3167
> Project: Kudu
> Issue Type: Bug
> Components: client, docker
>Reporter: Wang
>Priority: Major
>
> I start the docker with the reference
> [https://kudu.apache.org/docs/quickstart.html#_start_the_quickstart_cluster]
> $KUDU_QUICKSTART_IP is 10.44.192.145
> $KUDU_QUICKSTART_VERSION is 1.9.0
> kudu-python version is 1.2.0
>
> I success to start the docker with 3 masters and 5 tablets.
> Master port are 7051, 7151,7251 and 7051 is the leader.
> Via kudu-python, I can only connect the leader of kudu-master, but can't
> connect to the follower of the kudu-masters.
>
> {code:java}
> //代码占位符
> In [1]: import kudu
> In [2]: client = kudu.connect(host='10.44.192.145', port=7051)
> W0714 15:27:28.267447 5576 openssl_util.cc:102] It appears that OpenSSL has
> been previously initialized by code outside of Kudu. Please use
> kudu::client::DisableOpenSSLInitialization() to avoid potential crashes due
> to conflicting initialization.
> In [3]: client = kudu.connect(host='10.44.192.145', port=7151)
> ---
> KuduBadStatus Traceback (most recent call last)
> in ()
> > 1 client = kudu.connect(host='10.44.192.145', port=7151)
> /usr/lib64/python2.7/site-packages/kudu/__init__.pyc in connect(host, port,
> admin_timeout_ms, rpc_timeout_ms)
> 90
> 91 return Client(addresses, admin_timeout_ms=admin_timeout_ms,
> ---> 92 rpc_timeout_ms=rpc_timeout_ms)
> 93
> 94
> /usr/lib64/python2.7/site-packages/kudu/client.pyx in
> kudu.client.Client.__cinit__()
> 269 builder.default_rpc_timeout(timeout.delta)
> 270
> --> 271 check_status(builder.Build())
> 272
> 273 # A convenience
> /usr/lib64/python2.7/site-packages/kudu/errors.pyx in
> kudu.errors.check_status()
> 60 raise KuduInvalidArgument(c_message)
> 61 else:
> ---> 62 raise KuduBadStatus(status.ToString())
> KuduBadStatus: Timed out: Could not connect to the cluster:
> ConnectToClusterRpc(addrs: 10.44.192.145:7151, num_attempts: 242) passed its
> deadline: Not found: no leader found: ConnectToClusterRpc(addrs:
> 10.44.192.145:7151, num_attempts: 1)
> {code}
>
>
> ksck the cluster
> {code:java}
> //代码占位符
> [root@data-21-dev work]# docker exec -it $(docker ps -aqf
> "name=kudu-master-1") /bin/bash
> kudu@2b8a6a7d6000:/opt/kudu$ kudu cluster ksck
> kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251Master Summary
>UUID | Address | Status
> --++-
> 0fb04588705440cfae885fdf71f8fbf6 | kudu-master-2:7151 | HEALTHY
> e201d47a50624b02a8c0d200acc88e57 | kudu-master-1:7051 | HEALTHY
> fce36c217e6f4589948954cff3bf7e81 | kudu-master-3:7251 | HEALTHY Flag
> | Value | Tags | Master
> --+---++-
> use_hybrid_clock | false | hidden | all 3 server(s) checkedTablet Server
> Summary
>UUID | Address | Status | Location
> --++-+--
> 09b3f4efc69a42348ace15e80189d68c | 10.44.192.145:7150 | HEALTHY |
> 1d3057335b4149198513c71b75ee6fe8 | 10.44.192.145:7050 | HEALTHY |
> 7415842c9d5945ceaf227fdc83c26c83 | 10.44.192.145:7450 | HEALTHY |
> 901baca56619415cb74ec7ee24fd2110 | 10.44.192.145:7350 | HEALTHY |
> 95f16c59c05f45b2b86fb10b157b1826 | 10.44.192.145:7250 | HEALTHY |
> Tablet Server Location Summary
> Location | Count
> --+-
> | 5
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3154) RangerClientTestBase.TestLogging sometimes fails
[
https://issues.apache.org/jira/browse/KUDU-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151966#comment-17151966
]
Attila Bukor commented on KUDU-3154:
Yea I know the ordering can be explained by having multiple threads, but it's
still weird that both instances of this failure we've seen the order was
swapped, while it's correct in my my manual tests.
I tried reproducing it on dist-test, but all 500 runs succeeded.
> RangerClientTestBase.TestLogging sometimes fails
>
>
> Key: KUDU-3154
> URL: https://issues.apache.org/jira/browse/KUDU-3154
> Project: Kudu
> Issue Type: Bug
> Components: ranger, test
>Affects Versions: 1.13.0
>Reporter: Alexey Serbin
>Priority: Major
> Attachments: ranger_client-test.txt.xz
>
>
> The {{RangerClientTestBase.TestLogging}} scenario of the
> {{ranger_client-test}} sometimes fails (all types of builds) with error
> message like below:
> {noformat}
> src/kudu/ranger/ranger_client-test.cc:398: Failure
> Failed
>
> Bad status: Timed out: timed out while in flight
>
> I0620 07:06:02.907177 1140 server.cc:247] Received an EOF from the
> subprocess
> I0620 07:06:02.910923 1137 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> I0620 07:06:02.910964 1141 server.cc:380] outbound queue shut down: Aborted:
>
> I0620 07:06:02.910995 1138 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> I0620 07:06:02.910984 1139 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> {noformat}
> The log is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KUDU-3154) RangerClientTestBase.TestLogging sometimes fails
[
https://issues.apache.org/jira/browse/KUDU-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149740#comment-17149740
]
Attila Bukor commented on KUDU-3154:
Hm this is very interesting. I've seen this happen again in a test. A response
is never constructed, which [would be
logged|[https://github.com/apache/kudu/blob/master/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ranger/RangerProtocolHandler.java#L62|https://github.com/apache/kudu/blob/d23ee5d38ddc4317f431dd65df0c825c00cc968a/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ranger/RangerProtocolHandler.java#L62]],
so I guess it is possible that it never gets evaluated, but I think it's
unlikely that this happens due to the high frequency of the requests.
It's configured to refresh policies every
[200ms|[https://github.com/apache/kudu/blob/master/src/kudu/ranger/mini_ranger.h#L193]]
and from the logs we can see that it does indeed get refreshed every 200ms +
epsilon and takes ~30ms so even if there's a lock shared by the authorizer and
the refresher the authorizer should have plenty of time to acquire.
{code:java}
2020-06-20 07:05:49.672 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:489) ==> PolicyRefresher(serviceName=kudu).loadRoles()
2020-06-20 07:05:49.713 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:287) <== PolicyRefresher(serviceName=kudu).loadPolicy()
2020-06-20 07:05:49.872 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:489) ==> PolicyRefresher(serviceName=kudu).loadRoles()
2020-06-20 07:05:49.896 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:287) <== PolicyRefresher(serviceName=kudu).loadPolicy()
2020-06-20 07:05:50.072 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:489) ==> PolicyRefresher(serviceName=kudu).loadRoles()
2020-06-20 07:05:50.292 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:287) <== PolicyRefresher(serviceName=kudu).loadPolicy()
2020-06-20 07:05:50.473 [DEBUG - PolicyRefresher(serviceName=kudu)-12]
(PolicyRefresher.java:489) ==> PolicyRefresher(serviceName=kudu).loadRoles()
{code}
Tbh I'm not sure how that InboundRequest is logged to be taken before it's put
but I find it very weird. It was also logged the same way in the failing test
that I've seen.
If I run this test on my Mac it succeeds and the QueueUtil messages are in the
expected/correct order:
{code}
2020-07-02 01:17:19.459 [DEBUG - pool-2-thread-1] (QueueUtil.java:56) Message:
org.apache.kudu.subprocess.InboundRequest@1cf099ad has been put on the queue
2020-07-02 01:17:19.459 [DEBUG - pool-3-thread-3] (QueueUtil.java:41) Message:
org.apache.kudu.subprocess.InboundRequest@1cf099ad has been taken from the queue
{code}
I tried to search for some blocking queue bugs but couldn't find any so far.
> RangerClientTestBase.TestLogging sometimes fails
>
>
> Key: KUDU-3154
> URL: https://issues.apache.org/jira/browse/KUDU-3154
> Project: Kudu
> Issue Type: Bug
> Components: ranger, test
>Affects Versions: 1.13.0
>Reporter: Alexey Serbin
>Priority: Major
> Attachments: ranger_client-test.txt.xz
>
>
> The {{RangerClientTestBase.TestLogging}} scenario of the
> {{ranger_client-test}} sometimes fails (all types of builds) with error
> message like below:
> {noformat}
> src/kudu/ranger/ranger_client-test.cc:398: Failure
> Failed
>
> Bad status: Timed out: timed out while in flight
>
> I0620 07:06:02.907177 1140 server.cc:247] Received an EOF from the
> subprocess
> I0620 07:06:02.910923 1137 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> I0620 07:06:02.910964 1141 server.cc:380] outbound queue shut down: Aborted:
>
> I0620 07:06:02.910995 1138 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> I0620 07:06:02.910984 1139 server.cc:317] get failed, inbound queue shut
> down: Aborted:
> {noformat}
> The log is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-3090) Add owner concept in Kudu
[
https://issues.apache.org/jira/browse/KUDU-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Bukor reassigned KUDU-3090:
--
Assignee: Attila Bukor
> Add owner concept in Kudu
> -
>
> Key: KUDU-3090
> URL: https://issues.apache.org/jira/browse/KUDU-3090
> Project: Kudu
> Issue Type: New Feature
>Reporter: Hao Hao
>Assignee: Attila Bukor
>Priority: Major
> Labels: roadmap-candidate
>
> As mentioned in the Ranger integration design doc, Ranger supports ownership
> privilege by creating a default policy that allows \{OWNER} of a resource to
> access it without creating additional policy manually. Unless Kudu actually
> has a full support for owner, ownership privilege is not possible with Ranger
> integration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KUDU-1921) Add ability for clients to require authentication/encryption
[ https://issues.apache.org/jira/browse/KUDU-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-1921: -- Assignee: Attila Bukor > Add ability for clients to require authentication/encryption > > > Key: KUDU-1921 > URL: https://issues.apache.org/jira/browse/KUDU-1921 > Project: Kudu > Issue Type: Improvement > Components: client, security >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Assignee: Attila Bukor >Priority: Critical > > Currently, the clients always operate in "optional" mode for authentication > and encryption. This means that they are vulnerable to downgrade attacks by a > MITM. We should provide APIs so that clients can be configured to prohibit > downgrade when connecting to clusters they know to be secure. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3109) Log administrative operations
Attila Bukor created KUDU-3109: -- Summary: Log administrative operations Key: KUDU-3109 URL: https://issues.apache.org/jira/browse/KUDU-3109 Project: Kudu Issue Type: Task Reporter: Attila Bukor Sometimes it's impossible to determine what caused an issue when administrators run unsafe commands on the cluster. Logging these in an audit log would help. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3081) Add Kerberos support to MiniRanger
[ https://issues.apache.org/jira/browse/KUDU-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor resolved KUDU-3081. Fix Version/s: 1.12.0 Resolution: Fixed > Add Kerberos support to MiniRanger > -- > > Key: KUDU-3081 > URL: https://issues.apache.org/jira/browse/KUDU-3081 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3078) Ranger integration testing
[ https://issues.apache.org/jira/browse/KUDU-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3078: -- Assignee: Attila Bukor > Ranger integration testing > -- > > Key: KUDU-3078 > URL: https://issues.apache.org/jira/browse/KUDU-3078 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > > The Ranger integration should be properly tested before we can remove the > experimental flag. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KUDU-3081) Add Kerberos support to MiniRanger
[ https://issues.apache.org/jira/browse/KUDU-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor reassigned KUDU-3081: -- Assignee: Attila Bukor > Add Kerberos support to MiniRanger > -- > > Key: KUDU-3081 > URL: https://issues.apache.org/jira/browse/KUDU-3081 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3079) Add MiniRanger for integration tests
[ https://issues.apache.org/jira/browse/KUDU-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3079: --- Fix Version/s: 1.12.0 Resolution: Fixed Status: Resolved (was: In Review) > Add MiniRanger for integration tests > > > Key: KUDU-3079 > URL: https://issues.apache.org/jira/browse/KUDU-3079 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.12.0 > > > To write full integration tests we need a bundled Ranger service -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3079) Add MiniRanger for integration tests
[ https://issues.apache.org/jira/browse/KUDU-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Bukor updated KUDU-3079: --- Code Review: https://gerrit.cloudera.org/c/15483/ (was: https://gerrit.cloudera.org/c/15483/1) > Add MiniRanger for integration tests > > > Key: KUDU-3079 > URL: https://issues.apache.org/jira/browse/KUDU-3079 > Project: Kudu > Issue Type: Sub-task >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > > To write full integration tests we need a bundled Ranger service -- This message was sent by Atlassian Jira (v8.3.4#803005)
