[jira] [Created] (KUDU-3231) Kudu HMS tool should check user perms
Hunter Logan created KUDU-3231: -- Summary: Kudu HMS tool should check user perms Key: KUDU-3231 URL: https://issues.apache.org/jira/browse/KUDU-3231 Project: Kudu Issue Type: Improvement Components: CLI Reporter: Hunter Logan If a user runs "kudu hms fix -drop_orphan_hms_tables" from a profile that has perms to read the tables in HMS but not in Kudu, the script will consider all tables missing the Kudu information to be orphaned and drop them. This can occur even as the Kudu user if the environment has Sentry. It would be nice if the script warned about this, and/or required a "–force" argument if the user wishes to proceed. At the very least the script should probably be sure that it is being run as the Kudu user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-1563) Add support for INSERT/UPDATE/DELETE IGNORE
[ https://issues.apache.org/jira/browse/KUDU-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-1563. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add support for INSERT/UPDATE/DELETE IGNORE > --- > > Key: KUDU-1563 > URL: https://issues.apache.org/jira/browse/KUDU-1563 > Project: Kudu > Issue Type: New Feature >Reporter: Dan Burkert >Assignee: Grant Henke >Priority: Major > Labels: backup, impala, roadmap-candidate > Fix For: 1.14.0 > > > The Java client currently has an [option to ignore duplicate row key errors| > https://kudu.apache.org/apidocs/org/kududb/client/AsyncKuduSession.html#setIgnoreAllDuplicateRows-boolean-], > which is implemented by filtering the errors on the client side. If we are > going to continue to support this feature (and the consensus seems to be that > we probably should), we should promote it to a first class operation type > that is handled on the server side. This would have a modest perf. > improvement since less errors are returned, and it would allow INSERT IGNORE > ops to be mixed in the same batch as other INSERT, DELETE, UPSERT, etc. ops. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3202) Add Spark 3 Support
[ https://issues.apache.org/jira/browse/KUDU-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3202. --- Fix Version/s: 1.14.0 Resolution: Fixed > Add Spark 3 Support > --- > > Key: KUDU-3202 > URL: https://issues.apache.org/jira/browse/KUDU-3202 > Project: Kudu > Issue Type: Improvement > Components: spark >Affects Versions: 1.13.0 >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > Now that Spark 3 is released we should build and publish the Spark related > jars using Spark 3 while maintaining cross-build support for Spark 2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3222) std::bad_alloc in full_stack-insert-scan-test.cc
[
https://issues.apache.org/jira/browse/KUDU-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-3222.
---
Fix Version/s: 1.14.0
Resolution: Fixed
I think this has been resolved by the follow on gperftools upgrade.
> std::bad_alloc in full_stack-insert-scan-test.cc
>
>
> Key: KUDU-3222
> URL: https://issues.apache.org/jira/browse/KUDU-3222
> Project: Kudu
> Issue Type: Bug
> Components: test
>Reporter: Andrew Wong
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: FullStackScanInsertMRSOnly3.log
>
>
> Recently we've been starting to see the following in our runs of
> full_stack-insert-scan-test:
> {code:java}
> I1214 13:30:32.995853 39072 full_stack-insert-scan-test.cc:271] Insertion
> thread 7 of 50 is 69% done.
> terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc
> *** Aborted at 1607981433 (unix time) try "date -d @1607981433" if you are
> using GNU date ***
> PC: @ 0x3f85032625 __GI_raise
> *** SIGABRT (@0x11569802) received by PID 38914 (TID 0x7f81b4a02700) from
> PID 38914; stack trace: ***
> @ 0xcf8a21 google::(anonymous namespace)::FailureSignalHandler()
> @ 0x3f8540f710 (unknown)
> @ 0x3f85032625 __GI_raise
> @ 0x3f85033e05 __GI_abort
> @ 0x3f884bea7d __gnu_cxx::__verbose_terminate_handler()
> @ 0x3f884bcbd6 (unknown)
> @ 0x3f884bcc03 std::terminate()
> @ 0x3f884bcd22 __cxa_throw
> @ 0xd14bd5 (anonymous namespace)::handle_oom()
> @ 0x2ff3872 tcmalloc::allocate_full_cpp_throw_oom()
> @ 0x2cd4c1a
> _ZNSt6vectorIN4kudu19DecodedRowOperationESaIS1_EE17_M_realloc_insertIJRKS1_EEEvN9__gnu_cxx17__normal_iteratorIPS1_S3_EEDpOT_
> @ 0x2cd535a kudu::RowOperationsPBDecoder::DecodeOperations<>()
> @ 0x131c2a6 kudu::tablet::Tablet::DecodeWriteOperations()
> I1214 13:30:33.075912 39094 full_stack-insert-scan-test.cc:271] Insertion
> thread 29 of 50 is 69% done.
> @ 0x135bcb6 kudu::tablet::WriteOp::Prepare()
> @ 0x13514ac kudu::tablet::OpDriver::Prepare()
> @ 0x13520ed
> _ZNSt17_Function_handlerIFvvEZN4kudu6tablet8OpDriver12ExecuteAsyncEvEUlvE_E9_M_invokeERKSt9_Any_data
> @ 0x2e2409e kudu::ThreadPool::DispatchThread()
> @ 0x2e1b2c5 kudu::Thread::SuperviseThread()
> @ 0x3f854079d1 start_thread
> @ 0x3f850e88fd clone {code}
> This runs as a part of a suite of several tests in {{scripts/benchmarks.sh}}.
> This started happening fairly consistently starting around commit
> 2943aa701ee092158c2084c614a91f92513ef7c4, when we bumped glog and gperftools,
> though I'm not sure they are directly related here.
> The attached logs are a run on CentOS 6.6, with around 100GB of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3221) Add Bloom filter predicate push down support in Kudu Java client
[ https://issues.apache.org/jira/browse/KUDU-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3221: -- Labels: roadmap-candidate (was: ) > Add Bloom filter predicate push down support in Kudu Java client > > > Key: KUDU-3221 > URL: https://issues.apache.org/jira/browse/KUDU-3221 > Project: Kudu > Issue Type: Task > Components: client, java >Affects Versions: 1.13.0 >Reporter: Bankim Bhavsar >Priority: Major > Labels: roadmap-candidate > > We added support for predicate pushdown with Bloom filter predicate in 1.13. > See KUDU-2483. > However instead of using the existing Bloom filter in Kudu which is used > internally for index look ups, we decided to import the Cache, Hash and Space > efficient Bloom filter from Impala which was written in C++. > We need to create a binary compatible Bloom filter in Java to be push down > the Bloom filter predicate to Kudu servers from Java clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-613) Scan-resistant cache replacement algorithm for the block cache
[ https://issues.apache.org/jira/browse/KUDU-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-613: - Labels: performance roadmap-candidate (was: roadmap-candidate) > Scan-resistant cache replacement algorithm for the block cache > -- > > Key: KUDU-613 > URL: https://issues.apache.org/jira/browse/KUDU-613 > Project: Kudu > Issue Type: Improvement > Components: perf >Affects Versions: M4.5 >Reporter: Andrew Wang >Priority: Major > Labels: performance, roadmap-candidate > > The block cache currently uses LRU, which is vulnerable to large scan > workloads. It'd be good to implement something like 2Q. > ARC (patent encumbered, but good for ideas): > https://www.usenix.org/conference/fast-03/arc-self-tuning-low-overhead-replacement-cache > HBase (2Q like): > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3133) Poor TLS cypher performance on Java 8
[
https://issues.apache.org/jira/browse/KUDU-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3133:
--
Affects Version/s: 1.7.0
> Poor TLS cypher performance on Java 8
> -
>
> Key: KUDU-3133
> URL: https://issues.apache.org/jira/browse/KUDU-3133
> Project: Kudu
> Issue Type: Bug
> Components: perf, security
>Affects Versions: 1.7.0
>Reporter: Grant Henke
>Assignee: wangningito
>Priority: Major
>
> It was reported a while back that Kudu TLS doesn't perform well on Java 8 due
> to a potential GCM cypher bug or bad selection via `PREFERRED_CIPHER_SUITES`.
> It would be good to get to the bottom of this and fix it or document the
> recommendation to use Java 11.
> Here was the observed impact:
> {code}
> ./bin/ycsb run kudu -P workloads/workloadc -p operationcount=1
> -threads 64 -p kudu_num_clients=16 -s -p fieldlength=1 -p
> kudu_table_num_replicas=1
> java 7u75 with master:
> 0205 11:18:48.647920 (+28us) server_negotiation.cc:581] Negotiated
> TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~12k rows/sec
> java 8_141 with master:
> 0205 11:17:45.977107 (+31us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> 6k rows/sec
> java 8_141 with GCM-based codecs removed from Negotiator.java
> 0205 11:25:33.268689 (+40us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AES(256) Mac=SHA384
> ~6k rows/sec
> java 8_141 with only AES256-SHA remaining in Negotiator.java:
> "TLS_RSA_WITH_AES_256_CBC_SHA" )
> 0205 11:32:07.674860 (+44us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~9.5k rows/sec
> java 11 with master:
> 0205 11:17:01.416066 (+41us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> ~19k rows/sec
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3133) Poor TLS cypher performance on Java 8
[
https://issues.apache.org/jira/browse/KUDU-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-3133:
--
Labels: performance (was: )
> Poor TLS cypher performance on Java 8
> -
>
> Key: KUDU-3133
> URL: https://issues.apache.org/jira/browse/KUDU-3133
> Project: Kudu
> Issue Type: Bug
> Components: perf, security
>Affects Versions: 1.7.0
>Reporter: Grant Henke
>Assignee: wangningito
>Priority: Major
> Labels: performance
>
> It was reported a while back that Kudu TLS doesn't perform well on Java 8 due
> to a potential GCM cypher bug or bad selection via `PREFERRED_CIPHER_SUITES`.
> It would be good to get to the bottom of this and fix it or document the
> recommendation to use Java 11.
> Here was the observed impact:
> {code}
> ./bin/ycsb run kudu -P workloads/workloadc -p operationcount=1
> -threads 64 -p kudu_num_clients=16 -s -p fieldlength=1 -p
> kudu_table_num_replicas=1
> java 7u75 with master:
> 0205 11:18:48.647920 (+28us) server_negotiation.cc:581] Negotiated
> TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~12k rows/sec
> java 8_141 with master:
> 0205 11:17:45.977107 (+31us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> 6k rows/sec
> java 8_141 with GCM-based codecs removed from Negotiator.java
> 0205 11:25:33.268689 (+40us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AES(256) Mac=SHA384
> ~6k rows/sec
> java 8_141 with only AES256-SHA remaining in Negotiator.java:
> "TLS_RSA_WITH_AES_256_CBC_SHA" )
> 0205 11:32:07.674860 (+44us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
> ~9.5k rows/sec
> java 11 with master:
> 0205 11:17:01.416066 (+41us) server_negotiation.cc:581] Negotiated
> TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
> Enc=AESGCM(256) Mac=AEAD
> ~19k rows/sec
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-3147) Balance tablets based on range hash buckets
[ https://issues.apache.org/jira/browse/KUDU-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3147: -- Labels: balancer performance roadmap-candidate (was: balancer perf roadmap-candidate) > Balance tablets based on range hash buckets > --- > > Key: KUDU-3147 > URL: https://issues.apache.org/jira/browse/KUDU-3147 > Project: Kudu > Issue Type: Improvement > Components: master, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Assignee: Ravi Bhanot >Priority: Major > Labels: balancer, performance, roadmap-candidate > > When a user defines a schema that uses range + hash partitioning its is often > the case that the tablets in the latest range, based on time or any > semi-sequential data, are the only tablets that receive writes. Or even if > not the latest, it is common for a single range to receive a burst of writes > if backloading. > This is so common, that the default Kudu balancing scheme should consider > placing/rebalancing the tablets for the hash buckets within each range on as > many servers as possible in order to support the maximum write throughput. In > that case, `min(#buckets, #total-cluster-tservers)` tservers will be used to > handle the writes if the cluster is perfectly balanced. Today, even if > perfectly balanced, it is possible for all the hash buckets to be on a single > tserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2038) Add b-tree or inverted index on value field
[ https://issues.apache.org/jira/browse/KUDU-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2038: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Add b-tree or inverted index on value field > --- > > Key: KUDU-2038 > URL: https://issues.apache.org/jira/browse/KUDU-2038 > Project: Kudu > Issue Type: Task >Reporter: Yi Guolei >Priority: Major > Labels: performance, roadmap-candidate > > Do we have a plan to add index on any column [not primary column] ? Currently > kudu does not have btree or inverted index on columns. In this case if a > query wants to filter a column then kudu has to scan all datas in all > rowsets. > For example, select * from table where salary > 1 and age < 40, the bloom > filter or min max index will have no effect, kudu has to scan all datas in > all row sets. But if kudu has inverted index, then it will be much faster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-2817. --- Fix Version/s: 1.14.0 Resolution: Fixed Resolving as many of these upgrades have occurred across the 1.12, 1.13, and 1.14 releases. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * gperftools: 2.6.90 -> 2.8 > * protobuf: 3.12.3 -> 3.14.0 > * cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * lz4: 1.9.2 -> 1.9.3 > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squeasel: current- > * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> > cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) > ** Consider using official mustach c++ support? > * -curl: 7.59.0 (Mar 2018) -> 7.68.0- > * -crcutil: current- > * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 > * llvm: 9.0.0 -> 11.0.0 > * iwyu: 0.13 -> 0.14 > * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- > ** -Patch to replace with memkind is posted- > * -boost: 1.61.0 (patched, 2016) -> 1.74.0- > * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> > 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) > * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> > 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) > * thrift: 0.11 (Dec 2017) -> 0.13 > * -bison: 3.0.4 (patched, 2015) -> 3.5.4- > * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> > 3.1.1 (Oct 2018)- > * hadoop: 2.8.5 (Sept 2018) -> 3.2.0 > * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- > * ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 > * python: 2.7.13 -> (a lot of choices here) > * chrony: 3.5 -> 4.0 > A quick risk/reward review should be done and we should upgrade the > dependencies that are expected to be beneficial. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8- * -protobuf: 3.12.3 -> 3.14.0- * -cmake: 3.16.4 -> 3.19.0- * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3- * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0- * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0- * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8 * -protobuf: 3.12.3 -> 3.14.0 * -cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * -gperftools: 2.6.90 -> 2.8- > * -protobuf: 3.12.3 -> 3.14.0- > * -cmake: 3.16.4 -> 3.19.0- > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * -lz4: 1.9.2 -> 1.9.3- > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squease
[jira] [Updated] (KUDU-2817) C++ Upgrades for before Kudu 1.13 release
[ https://issues.apache.org/jira/browse/KUDU-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2817: -- Description: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * -gperftools: 2.6.90 -> 2.8 * -protobuf: 3.12.3 -> 3.14.0 * -cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * -lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * -hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * -ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. was: We should consider reviewing and upgrading our dependencies before the next release. Below is a list of current dependencies and their latest release. * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) * gmock: 1.8.0 -> 1.8.1 * gperftools: 2.6.90 -> 2.8 * protobuf: 3.12.3 -> 3.14.0 * cmake: 3.16.4 -> 3.19.0 * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- * lz4: 1.9.2 -> 1.9.3 * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- * libev: 4.20 -> 4.22 * -rapidjson: 1.1.0 (current)- * -squeasel: current- * mustache: 87a592e8aa04497764c533acd6e887618ca7b8a8 (Feb 2017) -> cf5c3dd499ea2bc9eb5c2072fb551dc7af75aa57 (Jun 2017) ** Consider using official mustach c++ support? * -curl: 7.59.0 (Mar 2018) -> 7.68.0- * -crcutil: current- * libunwind: 1.3-rc1 (patched, Nov 2017) -> 1.5.0 * llvm: 9.0.0 -> 11.0.0 * iwyu: 0.13 -> 0.14 * -nvml: 1.1 (2016) -> 1.6 (now called pmdk, Mar 2019)- ** -Patch to replace with memkind is posted- * -boost: 1.61.0 (patched, 2016) -> 1.74.0- * breakpad: 9eac2058b70615519b2c4d8c6bdbfca1bd079e39 (Apr 2013) -> 21b48a72aa50dde84149267f6b7402522b846b24 (Apr 2019) * sparsepp: 47a55825ca3b35eab1ca22b7ab82b9544e32a9af (Nov 2016) -> 5ca6de766db32b3fb08a040636423cd3988d2d4f (Jun 2018) * thrift: 0.11 (Dec 2017) -> 0.13 * -bison: 3.0.4 (patched, 2015) -> 3.5.4- * -hive: 498021fa15186aee8b282d3c032fbd2cede6bec4 (commit in Hive 2) -> 3.1.1 (Oct 2018)- * hadoop: 2.8.5 (Sept 2018) -> 3.2.0 * -sentry: 505b42e81a9d85c4ebe8db3f48ad7a6e824a5db5 (commit in Master)- * ranger: f37f5407eee8d2627a4306a25938b151f8e2ba31 -> 2.1.0 * python: 2.7.13 -> (a lot of choices here) * chrony: 3.5 -> 4.0 A quick risk/reward review should be done and we should upgrade the dependencies that are expected to be beneficial. > C++ Upgrades for before Kudu 1.13 release > - > > Key: KUDU-2817 > URL: https://issues.apache.org/jira/browse/KUDU-2817 > Project: Kudu > Issue Type: Improvement > Components: build >Affects Versions: 1.10.0 >Reporter: Grant Henke >Priority: Major > Fix For: 1.14.0 > > > We should consider reviewing and upgrading our dependencies before the next > release. Below is a list of current dependencies and their latest release. > * -gflags: 2.2.0 (Nov 2016) -> 2.2.2 (Nov 2018)- > * glog: 0.3.5 (May 2017) -> 0.4.0 (Mar 2019) > * gmock: 1.8.0 -> 1.8.1 > * -gperftools: 2.6.90 -> 2.8 > * -protobuf: 3.12.3 -> 3.14.0 > * -cmake: 3.16.4 -> 3.19.0 > * -snappy: 1.1.4 (Jan 2017) -> 1.1.8- > * -lz4: 1.9.2 -> 1.9.3 > * -bitshuffle: 55f9b4c (patched, 2016) -> 0.3.5 (Nov 2018)- > * -zlib: 1.2.8 (Apr 2013) -> 1.2.11 (Jan 2017)- > * libev: 4.20 -> 4.22 > * -rapidjson: 1.1.0 (current)- > * -squeasel: current- > *
[jira] [Resolved] (KUDU-3210) Support FIPS approved mode
[ https://issues.apache.org/jira/browse/KUDU-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3210. --- Fix Version/s: 1.14.0 Resolution: Fixed > Support FIPS approved mode > -- > > Key: KUDU-3210 > URL: https://issues.apache.org/jira/browse/KUDU-3210 > Project: Kudu > Issue Type: Improvement >Reporter: Attila Bukor >Assignee: Attila Bukor >Priority: Major > Fix For: 1.14.0 > > > FIPS 140-2 is a standard used to approve cryptographic modules. Some versions > of OpenSSL support a "FIPS mode" where only approved algorithms and key sizes > are enabled. Kudu should be able to run when FIPS mode is enabled and should > provide a way for admins to require that FIPS mode is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3186) Translate columnar RPC to row based API in C++ client
[ https://issues.apache.org/jira/browse/KUDU-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3186: -- Labels: impala performance roadmap-candidate (was: impala perf roadmap-candidate) > Translate columnar RPC to row based API in C++ client > -- > > Key: KUDU-3186 > URL: https://issues.apache.org/jira/browse/KUDU-3186 > Project: Kudu > Issue Type: Improvement > Components: client >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: impala, performance, roadmap-candidate > > The Java client is capable of receiving the columnar based RPC row format > response and then translating it for use in the row based API. This is useful > because it allows the more efficient (wire size and server CPU usage) format > to be use without code changes in client applications. > This would be especially beneficial for Apache Impala in cases where an > external cluster is accessing Kudu data remotely. > The Java client commit for reference: > https://github.com/apache/kudu/commit/34e85622f37f88b0436d9ba51df7592bf0e159de -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3186) Translate columnar RPC to row based API in C++ client
[ https://issues.apache.org/jira/browse/KUDU-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3186: -- Component/s: perf > Translate columnar RPC to row based API in C++ client > -- > > Key: KUDU-3186 > URL: https://issues.apache.org/jira/browse/KUDU-3186 > Project: Kudu > Issue Type: Improvement > Components: client, perf >Affects Versions: 1.12.0 >Reporter: Grant Henke >Priority: Major > Labels: impala, performance, roadmap-candidate > > The Java client is capable of receiving the columnar based RPC row format > response and then translating it for use in the row based API. This is useful > because it allows the more efficient (wire size and server CPU usage) format > to be use without code changes in client applications. > This would be especially beneficial for Apache Impala in cases where an > external cluster is accessing Kudu data remotely. > The Java client commit for reference: > https://github.com/apache/kudu/commit/34e85622f37f88b0436d9ba51df7592bf0e159de -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-636) optimization: we spend a lot of time in alloc/free
[ https://issues.apache.org/jira/browse/KUDU-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-636: - Labels: performance (was: ) > optimization: we spend a lot of time in alloc/free > -- > > Key: KUDU-636 > URL: https://issues.apache.org/jira/browse/KUDU-636 > Project: Kudu > Issue Type: Improvement > Components: perf >Affects Versions: Public beta >Reporter: Todd Lipcon >Priority: Major > Labels: performance > > Looking at a workload in the cluster, several of the top 10 lines of perf > report are tcmalloc-related. It seems like we don't do a good job of making > use of the per-thread free-lists, and we end up in a lot of contention on the > central free list. There are a few low-hanging fruit things we could do to > improve this for a likely perf boost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-3160) Upgrade devtoolset version for el6
[ https://issues.apache.org/jira/browse/KUDU-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke resolved KUDU-3160. --- Fix Version/s: 1.14.0 Resolution: Fixed This was completed in https://github.com/apache/kudu/commit/5aacd6c64ddd2669005d0b3ca57801a7a7124033 > Upgrade devtoolset version for el6 > -- > > Key: KUDU-3160 > URL: https://issues.apache.org/jira/browse/KUDU-3160 > Project: Kudu > Issue Type: Task > Components: build >Affects Versions: 1.12.0 >Reporter: Bankim Bhavsar >Priority: Major > Fix For: 1.14.0 > > Attachments: client_symbol-test.txt > > > Currently for building RHEL6 based distributions, kudu uses devtoolset3. > However devtoolset3 is EOL as of Nov 2016 and it's repo location has been > removed from the official software collections repository. See related > KUDU-3159. > So upgrade devtoolset to a newer version for building kudu on RHEL6 based > distributions. > Apart from the fact that devtoolset3 is EOL and not available officially, > newer version of devtoolset will also lead to generation of better code. > See comment from [~tlipcon] > https://gerrit.cloudera.org/c/16114/#message-86ea8228_e84eeb60 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1291) Efficiently support predicates on non-prefix key components
[ https://issues.apache.org/jira/browse/KUDU-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1291: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Efficiently support predicates on non-prefix key components > --- > > Key: KUDU-1291 > URL: https://issues.apache.org/jira/browse/KUDU-1291 > Project: Kudu > Issue Type: Sub-task > Components: perf, tablet >Reporter: Todd Lipcon >Priority: Major > Labels: performance, roadmap-candidate > > In a lot of workloads, users have a compound primary key where the first > component (or few components) is low cardinality. For example, a time series > workload may have (year, month, day, entity_id, timestamp) as a primary key. > A metrics or log storage workload might have (hostname, timestamp). > It's common to want to do cross-user or cross-date analytics like 'WHERE > timestamp BETWEEN and ' without specifying any predicate for the first > column(s) of the PK. Currently, we do not execute this efficiently, but > rather scan the whole table evaluating the predicate. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2019) Expose table/column statistics
[ https://issues.apache.org/jira/browse/KUDU-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2019: -- Labels: impala performance roadmap-candidate (was: impala roadmap-candidate) > Expose table/column statistics > -- > > Key: KUDU-2019 > URL: https://issues.apache.org/jira/browse/KUDU-2019 > Project: Kudu > Issue Type: New Feature > Components: client >Affects Versions: 1.0.0 >Reporter: Matthew Jacobs >Priority: Major > Labels: impala, performance, roadmap-candidate > > It would be helpful for query engines such as Impala to get table/column > statistics from Kudu. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2975) Spread WAL across multiple data directories
[ https://issues.apache.org/jira/browse/KUDU-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2975: -- Labels: performance roadmap-candidate scalability (was: roadmap-candidate scalability) > Spread WAL across multiple data directories > --- > > Key: KUDU-2975 > URL: https://issues.apache.org/jira/browse/KUDU-2975 > Project: Kudu > Issue Type: New Feature > Components: fs, perf, tablet, tserver >Reporter: LiFu He >Assignee: YangSong >Priority: Major > Labels: performance, roadmap-candidate, scalability > Attachments: network.png, tserver-WARNING.png, util.png > > > Recently, we deployed a new kudu cluster and every node has 12 SSD. Then, we > created a big table and loaded data to it through flink. We noticed that the > util of one SSD which is used to store WAL is 100% but others are free. So, > we suggest to spread WAL across multiple data directories. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3033) Add min/max values for the non-primary key columns in the metadata of rowsets/datablocks
[ https://issues.apache.org/jira/browse/KUDU-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3033: -- Labels: performance roadmap-candidate (was: ) > Add min/max values for the non-primary key columns in the metadata of > rowsets/datablocks > > > Key: KUDU-3033 > URL: https://issues.apache.org/jira/browse/KUDU-3033 > Project: Kudu > Issue Type: New Feature > Components: cfile, perf, tablet >Reporter: LiFu He >Priority: Major > Labels: performance, roadmap-candidate > > It's possible to add min/max values for the non-primary key columns in the > metadata of diskrowset/datablock, and then we can skip decoding/evaluating > the unnecessary diskrowset/datablock while scanning. Just like the "compute > stats" feature on impala, and the only difference is that kudu supports > updates. So, the min/max values should be invalid if the columns that have > deltas while scanning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-523) Add comprehensive cfile read/write benchmarks to dashboard
[ https://issues.apache.org/jira/browse/KUDU-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-523: - Labels: benchmarks performance (was: benchmarks) > Add comprehensive cfile read/write benchmarks to dashboard > -- > > Key: KUDU-523 > URL: https://issues.apache.org/jira/browse/KUDU-523 > Project: Kudu > Issue Type: Improvement > Components: perf, test >Affects Versions: Backlog >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: benchmarks, performance > > We have a couple of benchmarks in cfile-test, but we don't expose them as > part of the benchmarks dashboard. > We should set up a templated test that runs through all valid pairs of > type/encoding and times sequential write, sequential read, and random read of > a cfile with a few million rows. We should also test each including some > NULLs (eg 50/50 alternating nulls). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2887) Expose the tablet statistics in Client API
[ https://issues.apache.org/jira/browse/KUDU-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2887: -- Labels: impala performance roadmap-candidate (was: impala roadmap-candidate) > Expose the tablet statistics in Client API > -- > > Key: KUDU-2887 > URL: https://issues.apache.org/jira/browse/KUDU-2887 > Project: Kudu > Issue Type: Improvement > Components: client >Reporter: LiFu He >Priority: Minor > Labels: impala, performance, roadmap-candidate > > The patch about aggregating tablet statistics on the kudu-master is on the > way. And I think it's important to expose these statistics in client api by > which the query engine can optimize their query plan. For example: (1) adjust > the order of scanning tables, (2) Split a big tablet into multiple range > pieces(KUDU-2437) to improve concurrency automatically, (3) speed up the > query like "select count( *) from table". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2482) Add "delete ignore" capability to spark
[
https://issues.apache.org/jira/browse/KUDU-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke resolved KUDU-2482.
---
Fix Version/s: 1.14.0
Resolution: Fixed
This is resolved in KUDU-1563
> Add "delete ignore" capability to spark
> ---
>
> Key: KUDU-2482
> URL: https://issues.apache.org/jira/browse/KUDU-2482
> Project: Kudu
> Issue Type: Improvement
> Components: spark
>Affects Versions: 1.7.1
>Reporter: William Berkeley
>Priority: Major
> Fix For: 1.14.0
>
>
> If a user tries to delete a row from a Kudu table using kudu-spark but the
> row is not present, they receive an exception like
> {{java.lang.RuntimeException: failed to write N rows from DataFrame to Kudu;
> sample errors: Not found: key not found (error 0)}}
> This is the expected behavior, but it might also be nice to allow users to
> ignore PK errors so they can achieve clean semantics of "delete if present,
> else ignore".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-1954) Improve maintenance manager behavior in heavy write workload
[ https://issues.apache.org/jira/browse/KUDU-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1954: -- Labels: performance roadmap-candidate scalability (was: roadmap-candidate scalability) > Improve maintenance manager behavior in heavy write workload > > > Key: KUDU-1954 > URL: https://issues.apache.org/jira/browse/KUDU-1954 > Project: Kudu > Issue Type: Improvement > Components: compaction, perf, tserver >Affects Versions: 1.3.0 >Reporter: Todd Lipcon >Priority: Major > Labels: performance, roadmap-candidate, scalability > Attachments: mm-trace.png > > > During the investigation in [this > doc|https://docs.google.com/document/d/1U1IXS1XD2erZyq8_qG81A1gZaCeHcq2i0unea_eEf5c/edit] > I found a few maintenance-manager-related issues during heavy writes: > - we don't schedule flushes until we are already in "backpressure" realm, so > we spent most of our time doing backpressure > - even if we configure N maintenance threads, we typically are only using > ~50% of those threads due to the scheduling granularity > - when we do hit the "memory-pressure flush" threshold, all threads quickly > switch to flushing, which then brings us far beneath the threshold > - long running compactions can temporarily starve flushes > - high volume of writes can starve compactions -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2076) Deleting/updating is slow on single numeric row key tables
[ https://issues.apache.org/jira/browse/KUDU-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2076: -- Labels: performance (was: ) > Deleting/updating is slow on single numeric row key tables > -- > > Key: KUDU-2076 > URL: https://issues.apache.org/jira/browse/KUDU-2076 > Project: Kudu > Issue Type: Bug > Components: perf, tablet >Affects Versions: 1.4.0 >Reporter: Jean-Daniel Cryans >Priority: Major > Labels: performance > > A user reported that deleting 50M rows on the simplest of tables, (id INT, > text STRING, PRIMARY KEY (id)), doesn't complete. > It reproduces locally and I was able to see that we're deleting 1.4 rows / ms > which is awful considering that everything is cached. > Todd found that we're spending most of our time decoding big blocks of > bit-shuffled keys. Intuitively he though that having a composite row key > would perform better and indeed adding a column set to 0 in front makes it > 10x faster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2244) spinlock contention in raft_consensus
[
https://issues.apache.org/jira/browse/KUDU-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-2244:
--
Labels: performance (was: )
> spinlock contention in raft_consensus
> -
>
> Key: KUDU-2244
> URL: https://issues.apache.org/jira/browse/KUDU-2244
> Project: Kudu
> Issue Type: Improvement
> Components: consensus
>Reporter: Andrew Wong
>Priority: Major
> Labels: performance
>
> I was going through the logs of a cluster that was seeing a bunch of
> kernel_stack_watchdog traces, and the slowness seemed to be caused by a lot
> of activity in consensus requests. E.g.
> W1214 18:57:29.514219 36138 kernel_stack_watchdog.cc:145] Thread 36317 stuck
> at
> /data/jenkins/workspace/generic-package-centos64-7-0-impala/topdir/BUILD/kudu-1.3.0-cdh5.11.0/src/kudu/rpc/outbound_call.cc:192
> for 123ms:
> Kernel stack:
> [] sys_sched_yield+0x65/0xd0
> [] system_call_fastpath+0x16/0x1b
> [] 0x
> User stack:
> @ 0x7f72fab92057 __GI___sched_yield
> @ 0x19498bf kudu::Thread::StartThread()
> @ 0x1952e7d kudu::ThreadPool::CreateThreadUnlocked()
> @ 0x19534d3 kudu::ThreadPool::Submit()
> @ 0x1953a27 kudu::ThreadPool::SubmitFunc()
> @ 0x1953ecb kudu::ThreadPool::SubmitClosure()
> @ 0x9c94ec kudu::consensus::RaftConsensus::ElectionCallback()
> @ 0x9e6032 kudu::consensus::LeaderElection::CheckForDecision()
> @ 0x9e78c3
> kudu::consensus::LeaderElection::VoteResponseRpcCallback()
> @ 0xa8b137 kudu::rpc::OutboundCall::CallCallback()
> @ 0xa8c2bc kudu::rpc::OutboundCall::SetResponse()
> @ 0xa822c0 kudu::rpc::Connection::HandleCallResponse()
> @ 0xa83ffc ev::base<>::method_thunk<>()
> @ 0x198a07f ev_invoke_pending
> @ 0x198af71 ev_run
> @ 0xa5e049 kudu::rpc::ReactorThread::RunThread()
> So it seemed to be cause by some slowness in getting threads. Upon perusing
> the logs a bit more, there were a sizable number of spinlock profiling traces:
> W1214 18:54:27.897955 36379 rpcz_store.cc:238] Trace:
> 1214 18:54:26.766922 (+ 0us) service_pool.cc:143] Inserting onto call
> queue
> 1214 18:54:26.771135 (+ 4213us) service_pool.cc:202] Handling call
> 1214 18:54:26.771138 (+ 3us) raft_consensus.cc:1126] Updating replica for
> 0 ops
> 1214 18:54:27.897699 (+1126561us) raft_consensus.cc:1165] Early marking
> committed up to index 0
> 1214 18:54:27.897700 (+ 1us) raft_consensus.cc:1170] Triggering prepare
> for 0 ops
> 1214 18:54:27.897701 (+ 1us) raft_consensus.cc:1282] Marking committed up
> to 1766
> 1214 18:54:27.897702 (+ 1us) raft_consensus.cc:1332] Filling consensus
> response to leader.
> 1214 18:54:27.897736 (+34us) spinlock_profiling.cc:255] Waited 991 ms on
> lock 0x120b3540. stack: 019406c5 009c60d7 009c75f7
> 007dc628 00a7adfc 00a7b9cd 0194d059
> 7f72fbcc2dc4 7f72fabad1cc
> 1214 18:54:27.897737 (+ 1us) raft_consensus.cc:1327] UpdateReplicas()
> finished
> 1214 18:54:27.897741 (+ 4us) inbound_call.cc:130] Queueing success
> response
> Metrics: {"spinlock_wait_cycles":2478395136}
> Each of the traces noted on the order of 500-1000ms of waiting on spinlocks.
> Upon looking at raft_consensus.cc, it seems we're holding a spinlock
> (update_lock_) while we call RaftConsensus::UpdateReplica(), which according
> to its header, "won't return until all operations have been stored in the log
> and all Prepares() have been completed". While locking may be necessary, it's
> worth considering using a different kind of lock here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-1865) Create fast path for RespondSuccess() in KRPC
[
https://issues.apache.org/jira/browse/KUDU-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1865:
--
Component/s: perf
> Create fast path for RespondSuccess() in KRPC
> -
>
> Key: KUDU-1865
> URL: https://issues.apache.org/jira/browse/KUDU-1865
> Project: Kudu
> Issue Type: Improvement
> Components: perf, rpc
>Reporter: Sailesh Mukil
>Priority: Major
> Labels: perfomance, rpc, scalability
> Attachments: alloc-pattern.py, cross-thread.txt
>
>
> A lot of RPCs just respond with RespondSuccess() which returns the exact
> payload every time. This takes the same path as any other response by
> ultimately calling Connection::QueueResponseForCall() which has a few small
> allocations. These small allocations (and their corresponding deallocations)
> are called quite frequently (once for every IncomingCall) and end up taking
> quite some time in the kernel (traversing the free list, spin locks etc.)
> This was found when [~mmokhtar] ran some profiles on Impala over KRPC on a 20
> node cluster and found the following:
> The exact % of time spent is hard to quantify from the profiles, but these
> were the among the top 5 of the slowest stacks:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! __gnu_cxx::new_allocator::deallocate + 0x4 -
> new_allocator.h:110
> impalad ! std::_Vector_base std::allocator>::_M_deallocate + 0x5 - stl_vector.h:178
> impalad ! ~_Vector_base + 0x4 - stl_vector.h:160
> impalad ! ~vector - stl_vector.h:425 'slices' vector
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0xac -
> connection.cc:433
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ..
> {code}
> {code:java}
> impalad ! tcmalloc::CentralFreeList::FetchFromOneSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::RemoveRange + 0xc0 - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown
> source file]
> impalad ! operator new + 0x297 - [unknown source file]<--- Creating
> new 'OutboundTransferTask' object.
> impalad ! kudu::rpc::Connection::QueueResponseForCall + 0x76 -
> connection.cc:432
> impalad ! kudu::rpc::InboundCall::Respond + 0xfa - inbound_call.cc:133
> impalad ! kudu::rpc::InboundCall::RespondSuccess + 0x43 - inbound_call.cc:77
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1f7 - rpc_context.cc:66
> ...
> {code}
> Even creating and deleting the 'RpcContext' takes a lot of time:
> {code:java}
> impalad ! tcmalloc::CentralFreeList::ReleaseToSpans - [unknown source file]
> impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown
> source file]
> impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source
> file]
> impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown
> source file]
> impalad ! tcmalloc::ThreadCache::Scavenge + 0x3e - [unknown source file]
> impalad ! operator delete + 0x329 - [unknown source file]
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x16 -
> impala_internal_service.pb.cc:1221
> impalad ! impala::TransmitDataResponsePb::~TransmitDataResponsePb + 0x8 -
> impala_internal_service.pb.cc:1222
> impalad ! kudu::DefaultDeleter::operator() + 0x5 -
> gscoped_ptr.h:145
> impalad ! ~gscoped_ptr_impl + 0x9 - gscoped_ptr.h:228
> impalad ! ~gscoped_ptr - gscoped_ptr.h:318
> impalad ! kudu::rpc::RpcContext::~RpcContext + 0x1e - rpc_context.cc:53
> <-
> impalad ! kudu::rpc::RpcContext::RespondSuccess + 0x1ff - rpc_context.cc:67
> {code}
> The above show that creating these small objects under moderately heavy load
> results in heavy contention in the kernel. We will benefit a lot if we create
> a fast path for 'RespondSuccess'.
> My suggestion is to create all these small objects at once along with the
> 'InboundCall' object while it is being created, in a 'RespondSuccess'
> structure, and just use that structure when we want to send 'success' back to
> the sender. This would already contain the 'OutboundTransferTask', a 'Slice'
> with 'success', etc. We would expect that most RPCs respond with 'success' a
> majorit
[jira] [Updated] (KUDU-1439) Optimization for batch inserts into empty key ranges
[ https://issues.apache.org/jira/browse/KUDU-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1439: -- Labels: performance (was: ) > Optimization for batch inserts into empty key ranges > > > Key: KUDU-1439 > URL: https://issues.apache.org/jira/browse/KUDU-1439 > Project: Kudu > Issue Type: Improvement > Components: perf, tablet >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Labels: performance > > Got this idea from a CockroachDB optimization: > https://github.com/cockroachdb/cockroach/pull/6375 > The short version is that if we have a moderately large batch of inserts > which are sorted, we can do the following pseudocode: > - sort the inserts by primary key > - instead of using bloom filter checks, use SeekAtOrAfter on the first > primary key in the batch. This yields the next higher primary key that might > exist in the table (_nextKey_). > - for each of the keys in the sorted batch, if it's less than _nextKey_, we > don't need to do an existence check for it. > In the common case where clients are writing non-overlapping batches of rows > (eg importing from parquet) this should reduce the number of seeks and bloom > checks dramatically (order of batch size). Plus, it doesn't require much new > code to be written, so worth a shot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1639) Improve predicate pushdown
[ https://issues.apache.org/jira/browse/KUDU-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1639: -- Labels: performance roadmap-candidate (was: perf roadmap-candidate) > Improve predicate pushdown > -- > > Key: KUDU-1639 > URL: https://issues.apache.org/jira/browse/KUDU-1639 > Project: Kudu > Issue Type: Improvement > Components: client, perf, tablet >Reporter: Dan Burkert >Priority: Major > Labels: performance, roadmap-candidate > > Umbrella ticket for proposed improvements to predicates, scan optimization > based on predicates, and partition pruning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-1639) Improve predicate pushdown
[ https://issues.apache.org/jira/browse/KUDU-1639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-1639: -- Component/s: perf > Improve predicate pushdown > -- > > Key: KUDU-1639 > URL: https://issues.apache.org/jira/browse/KUDU-1639 > Project: Kudu > Issue Type: Improvement > Components: client, perf, tablet >Reporter: Dan Burkert >Priority: Major > Labels: perf, roadmap-candidate > > Umbrella ticket for proposed improvements to predicates, scan optimization > based on predicates, and partition pruning. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3132: -- Component/s: perf > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: perf, rpc >Reporter: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3132) Support RPC compression
[ https://issues.apache.org/jira/browse/KUDU-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-3132: -- Labels: performance roadmap-candidate (was: roadmap-candidate) > Support RPC compression > --- > > Key: KUDU-3132 > URL: https://issues.apache.org/jira/browse/KUDU-3132 > Project: Kudu > Issue Type: New Feature > Components: rpc >Reporter: Grant Henke >Priority: Major > Labels: performance, roadmap-candidate > > I have seen more and more deployments of Kudu where the tablet servers are > not co-located with the compute resources such as Impala or Spark. In > deployments like this, there could be significant network savings by > compressing the RPC messages (especially those that write or scan data). > Adding simple LZ4 or Snappy compression support to the RPC messages when not > on a loopback/local connection should be a great improvement for network > bound applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-2935) Implement built-in NTP client
[
https://issues.apache.org/jira/browse/KUDU-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-2935:
--
Labels: clock roadmap-candidate (was: clock)
> Implement built-in NTP client
> -
>
> Key: KUDU-2935
> URL: https://issues.apache.org/jira/browse/KUDU-2935
> Project: Kudu
> Issue Type: New Feature
> Components: clock, master, tserver
>Affects Versions: 1.11.0
>Reporter: Alexey Serbin
>Priority: Major
> Labels: clock, roadmap-candidate
>
> It would be nice to add a stripped-down implementation of built-in NTP client
> without any reliance on the kernel NTP discipline. The built-in client
> should maintain wall clock synchronized with NTP servers, and calling
> {{WalltimeWithError()}} should return wall clock timestamp with the
> estimation of error/offset from true time. Having built-in NTP client would
> provide more control over acceptable clock error and jitter acceptable for
> HybridTime timestamp generation.
> From the operability perspective, it would make it easier to run Kudu in
> containerized environments and overall make it easier for users to configure
> NTP even if they don't have superuser privileges at a node.
> The very first implementation should be good enough to work with properly
> configured and well behaving NTP servers, not necessarily being full-featured
> and 100% RFC-compliant NTP client. Later on, we can add more features and
> constraints to protect against misbehaving and rogue NTP servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-1235) Add Get API
[
https://issues.apache.org/jira/browse/KUDU-1235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grant Henke updated KUDU-1235:
--
Labels: performance roadmap-candidate (was: roadmap-candidate)
> Add Get API
> ---
>
> Key: KUDU-1235
> URL: https://issues.apache.org/jira/browse/KUDU-1235
> Project: Kudu
> Issue Type: New Feature
> Components: client, perf, tablet, tserver
>Reporter: Binglin Chang
>Priority: Major
> Labels: performance, roadmap-candidate
> Attachments: perf-get.svg, perf-scan-opt.svg, perf-scan.svg
>
>
> Get API is more user friendly and efficient if use just want primary key
> lookup.
> I setup a cluster and test get/scan single row using ycsb, initial test shows
> better performance for get.
> {noformat}
> kudu_workload:
> recordcount=100
> operationcount=100
> workload=com.yahoo.ycsb.workloads.CoreWorkload
> readallfields=false
> readproportion=1
> updateproportion=0
> scanproportion=0
> insertproportion=0
> requestdistribution=uniform
> use_get_api=false
> load:
> ./bin/ycsb load kudu -P workloads/kudu_workload -p sync_ops=false -p
> pre_split_num_tablets=1 -p table_name=ycsb_wiki_example -p
> masterQuorum='c3-kudu-tst-st01.bj:32600' -threads 100
> read test:
> ./bin/ycsb run kudu -P workloads/kudu_workload -p
> masterQuorum='c3-kudu-tst-st01.bj:32600' -threads 100
> {noformat}
> Get API:
> [OVERALL], RunTime(ms), 21304.0
> [OVERALL], Throughput(ops/sec), 46939.54187007135
> [CLEANUP], Operations, 100.0
> [CLEANUP], AverageLatency(us), 423.57
> [CLEANUP], MinLatency(us), 24.0
> [CLEANUP], MaxLatency(us), 19327.0
> [CLEANUP], 95thPercentileLatency(us), 52.0
> [CLEANUP], 99thPercentileLatency(us), 18815.0
> [READ], Operations, 100.0
> [READ], AverageLatency(us), 2065.185152
> [READ], MinLatency(us), 134.0
> [READ], MaxLatency(us), 92159.0
> [READ], 95thPercentileLatency(us), 2391.0
> [READ], 99thPercentileLatency(us), 6359.0
> [READ], Return=0, 100
> Scan API:
> [OVERALL], RunTime(ms), 38259.0
> [OVERALL], Throughput(ops/sec), 26137.6408165399
> [CLEANUP], Operations, 100.0
> [CLEANUP], AverageLatency(us), 47.32
> [CLEANUP], MinLatency(us), 16.0
> [CLEANUP], MaxLatency(us), 1837.0
> [CLEANUP], 95thPercentileLatency(us), 41.0
> [CLEANUP], 99thPercentileLatency(us), 158.0
> [READ], Operations, 100.0
> [READ], AverageLatency(us), 3595.825249
> [READ], MinLatency(us), 139.0
> [READ], MaxLatency(us), 3139583.0
> [READ], 95thPercentileLatency(us), 3775.0
> [READ], 99thPercentileLatency(us), 7659.0
> [READ], Return=0, 100
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KUDU-2981) Push predicate evaluation into more CFile decoders
[ https://issues.apache.org/jira/browse/KUDU-2981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-2981: -- Labels: newbie performance (was: newbie) > Push predicate evaluation into more CFile decoders > -- > > Key: KUDU-2981 > URL: https://issues.apache.org/jira/browse/KUDU-2981 > Project: Kudu > Issue Type: Improvement > Components: cfile, perf >Reporter: Bankim Bhavsar >Priority: Major > Labels: newbie, performance > > Commit c0f3727 added an optimization to push predicate evaluation into the > CFile decoders without fully materializing the contents of each cblock. It > did this with dictionary-encoded blocks, but the optimization can be applied > to any other encoding types too. > KUDU-736 also notes that we may be able to apply some predicates on > bitshuffled data. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-331) Allow evicting already-timed-out RPCs from inbound queue when new RPC arrives
[ https://issues.apache.org/jira/browse/KUDU-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KUDU-331: - Labels: performance scalability (was: ) > Allow evicting already-timed-out RPCs from inbound queue when new RPC arrives > - > > Key: KUDU-331 > URL: https://issues.apache.org/jira/browse/KUDU-331 > Project: Kudu > Issue Type: Improvement > Components: rpc >Affects Versions: M4 >Reporter: Todd Lipcon >Priority: Major > Labels: performance, scalability > > If the queue is really backed up, there might be RPCs in the queue which are > already timed out. Currently, we skip those when we get to processing them. > We could do a little better, though, by checking for timed-out calls when a > new call arrives. If a call arrives and the queue is full, just iterate > through the queue and clean out any that have already timed out to make room > for the new call. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KUDU-3232) GetTables in HMS Client should batch large requests
Hunter Logan created KUDU-3232: -- Summary: GetTables in HMS Client should batch large requests Key: KUDU-3232 URL: https://issues.apache.org/jira/browse/KUDU-3232 Project: Kudu Issue Type: Improvement Reporter: Hunter Logan [https://github.com/apache/kudu/blob/master/src/kudu/hms/hms_client.cc#L323] Because of how the thrift "get_table_objects_by_name" message works on the Hive side, Kudu should batch (or offer a configuration to batch) calls that have a large number of names (~100 per batch is likely an appropriate default). The performance issues with this call are particularly apparent when using the "kudu hms list" tool. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-3205) NPE in KuduScanTokenBuilder#build after a tablet server goes down
[
https://issues.apache.org/jira/browse/KUDU-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264326#comment-17264326
]
ASF subversion and git services commented on KUDU-3205:
---
Commit 929cb67da52f9bd30827527bb440b83d2260f608 in kudu's branch
refs/heads/master from Grant Henke
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=929cb67 ]
KUDU-3205: Fix building scan tokens when tablet not found errors occur
When tablet not found errors occur AsyncKuduClient.invalidateTabletCache
is called which then calls RemoteTablet.removeTabletClient to remove
a server (by uuid) from the RemoteTablet tabletServers list. This means that
the RemoteTablet can have a replica returned from RemoteTablet.getReplicas
which has a server that is not returned by RemoteTablet.getTabletServersCopy.
This results in a NPE when building a scan token because there is no matching
server for the replica in the serverIndexMap.
To fix this I simply check if the serverIndex found via the serverIndexMap is
null and ignore that replica if it is. A better long term fix is likely to build
the server info from the replicas themself, or to change the RemoteTablet
behavior
to blacklist servers instead of removing them from the actual tabletServers
list, or
to also remove the replica itself when RemoteTablet.removeTabletClient is
called.
I didn’t do any of these options now because they are all larger changes with a
wider potential impact.
Change-Id: I68679dee1ad7ebca405dd6e086770f3e034e310c
Reviewed-on: http://gerrit.cloudera.org:8080/16941
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin
> NPE in KuduScanTokenBuilder#build after a tablet server goes down
> -
>
> Key: KUDU-3205
> URL: https://issues.apache.org/jira/browse/KUDU-3205
> Project: Kudu
> Issue Type: Bug
> Components: spark
>Affects Versions: 1.13.0
>Reporter: Junegunn Choi
>Assignee: Grant Henke
>Priority: Major
>
> When a tablet server goes down while running a query on Spark, the connection
> becomes unusable due to the cached tablet locations that have become stale.
> h2. Steps to reproduce
> h3. Start spark-shell with kudu-spark2 1.13.0
> The problem is not reproducible with kudu-spark2 1.12.0 or below, because it
> was introduced in [KUDU-1802
> |https://github.com/apache/kudu/commit/d23ee5d38ddc4317f431dd65df0c825c00cc968a].
> h3. Run a scan query
> {code:scala}
> import org.apache.kudu.spark.kudu._
> val dummy = spark.read.options(Map("kudu.master" -> kuduMasters, "kudu.table"
> -> "dummy")).kudu
> dummy.createOrReplaceTempView("dummy")
> spark.sql("select sum(id), min(val2), max(val2), count(*) from dummy").show
> {code}
> h3. Kill a tablet server
> Kill one of the tablet servers that are serving data for the query. The query
> should fail immediately.
> {noformat}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in
> stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0
> (TID 2, localhost, executor driver): java.lang.RuntimeException:
> org.apache.kudu.client.NonRecoverableException: Scanner *** not found (it may
> have expired)
> {noformat}
> h3. Re-run the query
> {code:scala}
> spark.sql("select sum(id), min(val2), max(val2), count(*) from dummy").show
> {code}
> Doesn't work, fails with an NPE.
> {noformat}
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.kudu.client.KuduScanToken$KuduScanTokenBuilder.build(KuduScanToken.java:697)
> at org.apache.kudu.spark.kudu.KuduRDD.getPartitions(KuduRDD.scala:95)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrElse(Option.scala:121)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:273)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:269)
> at scala.Option.getOrEl
[jira] [Created] (KUDU-3233) Kudu CLI timeout option
Hunter Logan created KUDU-3233: -- Summary: Kudu CLI timeout option Key: KUDU-3233 URL: https://issues.apache.org/jira/browse/KUDU-3233 Project: Kudu Issue Type: Improvement Components: CLI, hms Reporter: Hunter Logan When running commands from the kudu CLI, particularly the HMS tools, the user should be able to configure a timeout option in the case where these commands would run for a long amount of time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
