[jira] [Resolved] (IMPALA-12024) For CTAS, include time to create the table in the query profile timeline
[
https://issues.apache.org/jira/browse/IMPALA-12024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang resolved IMPALA-12024.
-
Fix Version/s: Impala 4.3.0
Resolution: Fixed
> For CTAS, include time to create the table in the query profile timeline
>
>
> Key: IMPALA-12024
> URL: https://issues.apache.org/jira/browse/IMPALA-12024
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 4.3.0
>Reporter: Joe McDonnell
>Assignee: Quanlong Huang
>Priority: Major
> Fix For: Impala 4.3.0
>
> Attachments: profile_7347b5216e8840b9_a19c3b08.txt
>
>
> If the metastore is slow, catalog operations to create / drop tables can take
> a long time. For a CTAS statement, we don't have a clear indication in the
> profile that this is happening. Instead, it shows up as a gap between
> planning finished and submitting for admission:
> {noformat}
> Query Timeline: 12s696ms
> - Query submitted: 64.303us (64.303us)
>- Planning finished: 117.816ms (117.752ms)
>- Submit for admission: 12s360ms (12s243ms)
> ...{noformat}
> This may apply for other statement types.
> To reproduce with CTAS on an Impala devenv:
> {noformat}
> 1. Start Impala
> 2. In impala-shell, run "select * from functional.alltypes"
> 3. Find the metastore pid (jps -v | grep metastore)
> 4. sudo gdb attach ${metastore_pid}
> 5. In impala-shell, run "create table foo as select * from
> functional.alltypes"
> .. This will hang because it can't create the table, wait a bit
> 6. Back in the gdb session, detach from metastore and let the CTAS
> finish{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12024) For CTAS, include time to create the table in the query profile timeline
[
https://issues.apache.org/jira/browse/IMPALA-12024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-12024:
Component/s: Catalog
(was: Backend)
> For CTAS, include time to create the table in the query profile timeline
>
>
> Key: IMPALA-12024
> URL: https://issues.apache.org/jira/browse/IMPALA-12024
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
>Reporter: Joe McDonnell
>Assignee: Quanlong Huang
>Priority: Major
> Fix For: Impala 4.3.0
>
> Attachments: profile_7347b5216e8840b9_a19c3b08.txt
>
>
> If the metastore is slow, catalog operations to create / drop tables can take
> a long time. For a CTAS statement, we don't have a clear indication in the
> profile that this is happening. Instead, it shows up as a gap between
> planning finished and submitting for admission:
> {noformat}
> Query Timeline: 12s696ms
> - Query submitted: 64.303us (64.303us)
>- Planning finished: 117.816ms (117.752ms)
>- Submit for admission: 12s360ms (12s243ms)
> ...{noformat}
> This may apply for other statement types.
> To reproduce with CTAS on an Impala devenv:
> {noformat}
> 1. Start Impala
> 2. In impala-shell, run "select * from functional.alltypes"
> 3. Find the metastore pid (jps -v | grep metastore)
> 4. sudo gdb attach ${metastore_pid}
> 5. In impala-shell, run "create table foo as select * from
> functional.alltypes"
> .. This will hang because it can't create the table, wait a bit
> 6. Back in the gdb session, detach from metastore and let the CTAS
> finish{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12024) For CTAS, include time to create the table in the query profile timeline
[
https://issues.apache.org/jira/browse/IMPALA-12024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-12024:
Affects Version/s: (was: Impala 4.3.0)
> For CTAS, include time to create the table in the query profile timeline
>
>
> Key: IMPALA-12024
> URL: https://issues.apache.org/jira/browse/IMPALA-12024
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Joe McDonnell
>Assignee: Quanlong Huang
>Priority: Major
> Fix For: Impala 4.3.0
>
> Attachments: profile_7347b5216e8840b9_a19c3b08.txt
>
>
> If the metastore is slow, catalog operations to create / drop tables can take
> a long time. For a CTAS statement, we don't have a clear indication in the
> profile that this is happening. Instead, it shows up as a gap between
> planning finished and submitting for admission:
> {noformat}
> Query Timeline: 12s696ms
> - Query submitted: 64.303us (64.303us)
>- Planning finished: 117.816ms (117.752ms)
>- Submit for admission: 12s360ms (12s243ms)
> ...{noformat}
> This may apply for other statement types.
> To reproduce with CTAS on an Impala devenv:
> {noformat}
> 1. Start Impala
> 2. In impala-shell, run "select * from functional.alltypes"
> 3. Find the metastore pid (jps -v | grep metastore)
> 4. sudo gdb attach ${metastore_pid}
> 5. In impala-shell, run "create table foo as select * from
> functional.alltypes"
> .. This will hang because it can't create the table, wait a bit
> 6. Back in the gdb session, detach from metastore and let the CTAS
> finish{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-11535) Skip events happen before manual REFRESH
[ https://issues.apache.org/jira/browse/IMPALA-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760244#comment-17760244 ] ASF subversion and git services commented on IMPALA-11535: -- Commit b718d63860356a04814e07d91711c3c748b3e769 in impala's branch refs/heads/master from Sai Hemanth Gantasala [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b718d6386 ] IMPALA-11535: Skip older events in the event processor based on the latestRefreshEventID Summary: If the table has been manually refreshed, all its events happen before the manual REFRESH can be skipped. This happens when catalogd is lagging behind in processing events. When processing an event, we can check whether there are manual REFRESH executed after its eventTime. In such case, we don't need to process the event to refresh anything. This helps catalogd to catch up HMS events quickly. Implementation details: Updated the lastRefreshEventId on the table or partition whenever there is table or partition level refresh/load. By comparing the lastRefreshEventId to current event id in the event processor the older events can be skipped. set enable_skipping_older_events to true to enable this optimization Testing: - Unit end-to-end test and unit test to test the functionality. Change-Id: Ic0dc5c7396d80616680d8a5805ce80db293b72e1 Reviewed-on: http://gerrit.cloudera.org:8080/20022 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Skip events happen before manual REFRESH > > > Key: IMPALA-11535 > URL: https://issues.apache.org/jira/browse/IMPALA-11535 > Project: IMPALA > Issue Type: Improvement >Reporter: Quanlong Huang >Assignee: Sai Hemanth Gantasala >Priority: Critical > > If the table has been manually refreshed, all its events happen before the > manual REFRESH can be skipped. > > This happens when catalogd is lagging behind in processing events. When > processing an event, we can check whether there are manual REFRESH executed > after its eventTime. In such case, we don't need to process the event to > refresh anything. This helps catalogd to catch up HMS events quickly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12024) For CTAS, include time to create the table in the query profile timeline
[ https://issues.apache.org/jira/browse/IMPALA-12024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760243#comment-17760243 ] ASF subversion and git services commented on IMPALA-12024: -- Commit 27d37a60c954d7734cfaea35e73939d5c7681d55 in impala's branch refs/heads/master from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=27d37a60c ] IMPALA-12024: Add catalog profile for createTable DDL/DMLs might spend a long time in the execution in catalogd. Currently, the profiles just have a counter of CatalogOpExecTimer. We need more items to show the details of how they are executed in catalogd. As the first step, this patch adds the profile timeline for how the createTable DDL is executed in Catalog Server. Also added more events in the existing "Query Timeline" for when the CatalogDdlRequest finished and when the catalog updates are applied. To implement this, a 'profile' field is added in TDdlExecResponse to carry the execution counters and timeline in catalogd. Currently, we just uses the timeline of it. We can add more counters in the future. Several methods add a timeline parameter to mark the progress in them. Timeline events are added after each RPC finished. Here is an example when HMS is hanging for 26s in a CTAS. I used gdb to attach to HMS as the JIRA description mentioned. In the timeline, we can see the time is spent in the first HMS RPC that fetching the current HMS event id: Catalog Server Operation: 26s560ms - Got metastoreDdlLock: 163.977us (163.977us) - Got Metastore client: 166.339us (2.362us) - Got current Metastore event id 8355270: 26s494ms (26s494ms) - Created table in Metastore: 26s558ms (63.507ms) - Fetched event batch from Metastore: 26s559ms (1.155ms) - Created table in catalog cache: 26s560ms (1.164ms) - DDL finished: 26s560ms (65.538us) Query Compilation: 164.257ms - Metadata of all 1 tables cached: 10.535ms (10.535ms) - Analysis finished: 118.324ms (107.788ms) - Authorization finished (noop): 118.489ms (164.626us) - Value transfer graph computed: 118.830ms (341.792us) - Single node plan created: 150.150ms (31.319ms) - Runtime filters computed: 150.254ms (103.529us) - Distributed plan created: 151.832ms (1.578ms) - Planning finished: 164.257ms (12.425ms) Query Timeline: 27s304ms - Query submitted: 129.658us (129.658us) - Planning finished: 170.224ms (170.095ms) - CatalogDdlRequest finished: 26s731ms (26s561ms) - Applied catalog updates from DDL: 26s740ms (8.752ms) - Submit for admission: 26s740ms (22.233us) - Completed admission: 26s740ms (286.295us) - Ready to start on 3 backends: 26s740ms (155.916us) - All 3 execution backends (3 fragment instances) started: 26s751ms (10.864ms) - Last row fetched: 26s920ms (168.226ms) - Released admission control resources: 26s920ms (27.635us) - DML data written: 26s920ms (126.369us) - Applied catalog updates from DDL: 26s985ms (65.354ms) - DML Metastore update finished: 26s985ms (30.343us) - Rows available: 26s985ms (27.050us) - Unregister query: 27s304ms (318.661ms) An example of creating a Kudu table: Catalog Server Operation: 1s730ms - Got Metastore client: 113.403us (113.403us) - Got current Metastore event id 8355276: 974.500us (861.097us) - Got Kudu client: 212.123ms (211.148ms) - Got kuduDdlLock: 212.128ms (4.680us) - Checked table existence in Kudu: 850.786ms (638.658ms) - Created table in Kudu: 1s623ms (772.379ms) - Got metastoreDdlLock: 1s623ms (397.305us) - Got Metastore client: 1s623ms (7.813us) - Checked table existence in Metastore: 1s648ms (25.154ms) - Created table in Metastore: 1s725ms (76.348ms) - Fetched event batch from Metastore: 1s728ms (3.004ms) - Created table in catalog cache: 1s730ms (2.024ms) - DDL finished: 1s730ms (84.448us) An example of creating an Iceberg table: Catalog Server Operation: 1s573ms - Got Metastore client: 141.799us (141.799us) - Checked table existence in Metastore: 2.957ms (2.815ms) - Got current Metastore event id 8355277: 3.669ms (712.475us) - Created table using Iceberg Catalog HIVE_CATALOG: 1s379ms (1s375ms) - Fetched event batch from Metastore: 1s381ms (2.188ms) - Created table in catalog cache: 1s382ms (1.556ms) - Set Iceberg table owner in Metastore: 1s573ms (190.262ms) - DDL finished: 1s573ms (59.176us) Tests: - Add e2e tests to verify the DDL timeline events exist in profile Change-Id: I3ebf591625e71391a5b23f56ddca8f0ae97b1efa Reviewed-on: http://gerrit.cloudera.org:8080/20368 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > For CTAS, include time to create the table in the query
[jira] [Comment Edited] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760202#comment-17760202 ] Maxwell Guo edited comment on IMPALA-12402 at 8/30/23 2:30 AM: --- [gerrit link|https://gerrit.cloudera.org/#/c/20435/] and it is ready for review agagin now. was (Author: maxwellguo): [gerrit link|https://gerrit.cloudera.org/#/c/20435/] > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12402: - Language: java C++ (was: java) > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12402: - [gerrit link|https://gerrit.cloudera.org/#/c/20435/] > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760189#comment-17760189 ] Maxwell Guo commented on IMPALA-12402: -- Thanks [~stigahuang] > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12402 started by Maxwell Guo. > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12411) TSAN ThreadSanitizer: data race during expr-test teardown
[
https://issues.apache.org/jira/browse/IMPALA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760175#comment-17760175
]
Andrew Sherman commented on IMPALA-12411:
-
[~MikaelSmith] I am picking on you randomly to take a look. This has only
happened once.
> TSAN ThreadSanitizer: data race during expr-test teardown
> -
>
> Key: IMPALA-12411
> URL: https://issues.apache.org/jira/browse/IMPALA-12411
> Project: IMPALA
> Issue Type: Bug
> Components: be
>Affects Versions: Impala 4.3.0
>Reporter: Andrew Sherman
>Assignee: Michael Smith
>Priority: Critical
> Attachments: expr-test-tsan-failure.log
>
>
> The racing threads are
> {code:java}
> 20:14:05 Read of size 8 at 0x0a8d3348 by main thread:
> 20:14:05 #0 std::vector std::allocator >::~vector()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
> (unifiedbetests+0x3fcd9b9)
> 20:14:05 #1
> impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
> (unifiedbetests+0x3fc1397)
> 20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
> (unifiedbetests+0x4b2989d)
> 20:14:05 #3 at_exit_wrapper(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
> (unifiedbetests+0x21b3554)
> {code}
> and
> {code:java}
> 20:14:05 Previous write of size 8 at 0x0a8d3348 by thread T586:
> 20:14:05 #0 std::vector std::allocator
> >::_M_erase_at_end(impala::TJvmMemoryPool*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
> (unifiedbetests+0x4afabcc)
> 20:14:05 #1 std::vector std::allocator >::clear()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
> (unifiedbetests+0x4afa4b4)
> 20:14:05 #2 unsigned int
> impala::TGetJvmMemoryMetricsResponse::read(apache::thrift::protocol::TProtocol*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
> (unifiedbetests+0x4afa21d)
> 20:14:05 #3 impala::Status
> impala::DeserializeThriftMsg(unsigned
> char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
> (unifiedbetests+0x4af9da6)
> 20:14:05 #4 impala::Status
> impala::DeserializeThriftMsg(JNIEnv_*,
> _jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
> (unifiedbetests+0x4af9c62)
> 20:14:05 #5 impala::Status
> impala::JniCall::ObjectToResult(_jobject*,
> impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
> (unifiedbetests+0x4af9b24)
> 20:14:05 #6 impala::Status
> impala::JniCall::Call(impala::TGetJvmMemoryMetricsResponse*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
> (unifiedbetests+0x4af92a6)
> 20:14:05 #7
> impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
> (unifiedbetests+0x4af89a3)
> 20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
> (unifiedbetests+0x4b2780d)
> 20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
> (*)(impala::TGetJvmMemoryMetricsResponse const&))
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
> (unifiedbetests+0x4b27711)
> 20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
> (unifiedbetests+0x4b276bf)
> 20:14:05 #11
> impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB const*,
> impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
>
[jira] [Assigned] (IMPALA-12411) TSAN ThreadSanitizer: data race during expr-test teardown
[
https://issues.apache.org/jira/browse/IMPALA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned IMPALA-12411:
---
Assignee: Michael Smith
> TSAN ThreadSanitizer: data race during expr-test teardown
> -
>
> Key: IMPALA-12411
> URL: https://issues.apache.org/jira/browse/IMPALA-12411
> Project: IMPALA
> Issue Type: Bug
> Components: be
>Affects Versions: Impala 4.3.0
>Reporter: Andrew Sherman
>Assignee: Michael Smith
>Priority: Critical
> Attachments: expr-test-tsan-failure.log
>
>
> The racing threads are
> {code:java}
> 20:14:05 Read of size 8 at 0x0a8d3348 by main thread:
> 20:14:05 #0 std::vector std::allocator >::~vector()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
> (unifiedbetests+0x3fcd9b9)
> 20:14:05 #1
> impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
> (unifiedbetests+0x3fc1397)
> 20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
> (unifiedbetests+0x4b2989d)
> 20:14:05 #3 at_exit_wrapper(void*)
> /mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
> (unifiedbetests+0x21b3554)
> {code}
> and
> {code:java}
> 20:14:05 Previous write of size 8 at 0x0a8d3348 by thread T586:
> 20:14:05 #0 std::vector std::allocator
> >::_M_erase_at_end(impala::TJvmMemoryPool*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
> (unifiedbetests+0x4afabcc)
> 20:14:05 #1 std::vector std::allocator >::clear()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
> (unifiedbetests+0x4afa4b4)
> 20:14:05 #2 unsigned int
> impala::TGetJvmMemoryMetricsResponse::read(apache::thrift::protocol::TProtocol*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
> (unifiedbetests+0x4afa21d)
> 20:14:05 #3 impala::Status
> impala::DeserializeThriftMsg(unsigned
> char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
> (unifiedbetests+0x4af9da6)
> 20:14:05 #4 impala::Status
> impala::DeserializeThriftMsg(JNIEnv_*,
> _jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
> (unifiedbetests+0x4af9c62)
> 20:14:05 #5 impala::Status
> impala::JniCall::ObjectToResult(_jobject*,
> impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
> (unifiedbetests+0x4af9b24)
> 20:14:05 #6 impala::Status
> impala::JniCall::Call(impala::TGetJvmMemoryMetricsResponse*)
>
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
> (unifiedbetests+0x4af92a6)
> 20:14:05 #7
> impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
> (unifiedbetests+0x4af89a3)
> 20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
> (unifiedbetests+0x4b2780d)
> 20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
> (*)(impala::TGetJvmMemoryMetricsResponse const&))
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
> (unifiedbetests+0x4b27711)
> 20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
> (unifiedbetests+0x4b276bf)
> 20:14:05 #11
> impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB const*,
> impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
> /data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
> (unifiedbetests+0x455bc15)
> 20:14:05 #12
> boost::detail::function::function_obj_invoker0
[jira] [Updated] (IMPALA-12411) TSAN ThreadSanitizer: data race during expr-test teardown
[
https://issues.apache.org/jira/browse/IMPALA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated IMPALA-12411:
Description:
The racing threads are
{code:java}
20:14:05 Read of size 8 at 0x0a8d3348 by main thread:
20:14:05 #0 std::vector >::~vector()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
(unifiedbetests+0x3fcd9b9)
20:14:05 #1
impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
(unifiedbetests+0x3fc1397)
20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
(unifiedbetests+0x4b2989d)
20:14:05 #3 at_exit_wrapper(void*)
/mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
(unifiedbetests+0x21b3554)
{code}
and
{code:java}
20:14:05 Previous write of size 8 at 0x0a8d3348 by thread T586:
20:14:05 #0 std::vector
>::_M_erase_at_end(impala::TJvmMemoryPool*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
(unifiedbetests+0x4afabcc)
20:14:05 #1 std::vector >::clear()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
(unifiedbetests+0x4afa4b4)
20:14:05 #2 unsigned int
impala::TGetJvmMemoryMetricsResponse::read(apache::thrift::protocol::TProtocol*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
(unifiedbetests+0x4afa21d)
20:14:05 #3 impala::Status
impala::DeserializeThriftMsg(unsigned
char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
(unifiedbetests+0x4af9da6)
20:14:05 #4 impala::Status
impala::DeserializeThriftMsg(JNIEnv_*,
_jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
(unifiedbetests+0x4af9c62)
20:14:05 #5 impala::Status
impala::JniCall::ObjectToResult(_jobject*,
impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
(unifiedbetests+0x4af9b24)
20:14:05 #6 impala::Status
impala::JniCall::Call(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
(unifiedbetests+0x4af92a6)
20:14:05 #7
impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
(unifiedbetests+0x4af89a3)
20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
(unifiedbetests+0x4b2780d)
20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
(*)(impala::TGetJvmMemoryMetricsResponse const&))
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
(unifiedbetests+0x4b27711)
20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
(unifiedbetests+0x4b276bf)
20:14:05 #11 impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB
const*, impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
(unifiedbetests+0x455bc15)
20:14:05 #12
boost::detail::function::function_obj_invoker0::invoke(boost::detail::function::function_buffer&)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:137:18
(unifiedbetests+0x455b9d9)
20:14:05 #13 boost::function0::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
(unifiedbetests+0x4b57bf1)
20:14:05 #14 impala::RuntimeProfile::DerivedCounter::value() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile-counters.h:384:12
(unifiedbetests+0x4b8b38d)
20:14:05 #15
[jira] [Updated] (IMPALA-12411) TSAN ThreadSanitizer: data race during expr-test teardown
[
https://issues.apache.org/jira/browse/IMPALA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated IMPALA-12411:
Description:
The racing threads are
{code:java}
20:14:05 Read of size 8 at 0x0a8d3348 by main thread:
20:14:05 #0 std::vector >::~vector()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
(unifiedbetests+0x3fcd9b9)
20:14:05 #1
impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
(unifiedbetests+0x3fc1397)
20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
(unifiedbetests+0x4b2989d)
20:14:05 #3 at_exit_wrapper(void*)
/mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
(unifiedbetests+0x21b3554)
{code}
and
{code:java}
20:14:05 Previous write of size 8 at 0x0a8d3348 by thread T586:
20:14:05 #0 std::vector
>::_M_erase_at_end(impala::TJvmMemoryPool*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
(unifiedbetests+0x4afabcc)
20:14:05 #1 std::vector >::clear()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
(unifiedbetests+0x4afa4b4)
20:14:05 #2 unsigned int
impala::TGetJvmMemoryMetricsResponse::read(apache::thrift::protocol::TProtocol*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
(unifiedbetests+0x4afa21d)
20:14:05 #3 impala::Status
impala::DeserializeThriftMsg(unsigned
char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
(unifiedbetests+0x4af9da6)
20:14:05 #4 impala::Status
impala::DeserializeThriftMsg(JNIEnv_*,
_jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
(unifiedbetests+0x4af9c62)
20:14:05 #5 impala::Status
impala::JniCall::ObjectToResult(_jobject*,
impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
(unifiedbetests+0x4af9b24)
20:14:05 #6 impala::Status
impala::JniCall::Call(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
(unifiedbetests+0x4af92a6)
20:14:05 #7
impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
(unifiedbetests+0x4af89a3)
20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
(unifiedbetests+0x4b2780d)
20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
(*)(impala::TGetJvmMemoryMetricsResponse const&))
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
(unifiedbetests+0x4b27711)
20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
(unifiedbetests+0x4b276bf)
20:14:05 #11 impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB
const*, impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
(unifiedbetests+0x455bc15)
20:14:05 #12
boost::detail::function::function_obj_invoker0::invoke(boost::detail::function::function_buffer&)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:137:18
(unifiedbetests+0x455b9d9)
20:14:05 #13 boost::function0::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
(unifiedbetests+0x4b57bf1)
20:14:05 #14 impala::RuntimeProfile::DerivedCounter::value() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/runtime-profile-counters.h:384:12
(unifiedbetests+0x4b8b38d)
20:14:05 #15
[jira] [Created] (IMPALA-12411) TSAN ThreadSanitizer: data race during expr-test teardown
Andrew Sherman created IMPALA-12411:
---
Summary: TSAN ThreadSanitizer: data race during expr-test teardown
Key: IMPALA-12411
URL: https://issues.apache.org/jira/browse/IMPALA-12411
Project: IMPALA
Issue Type: Bug
Components: be
Affects Versions: Impala 4.3.0
Reporter: Andrew Sherman
Attachments: expr-test-tsan-failure.log
The racing threads are
{code:java}
20:14:05 Read of size 8 at 0x0a8d3348 by main thread:
20:14:05 #0 std::vector >::~vector()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:680:54
(unifiedbetests+0x3fcd9b9)
20:14:05 #1
impala::TGetJvmMemoryMetricsResponse::~TGetJvmMemoryMetricsResponse()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.cpp:4158:1
(unifiedbetests+0x3fc1397)
20:14:05 #2 impala::JvmMetricCache::~JvmMetricCache()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.h:170:7
(unifiedbetests+0x4b2989d)
20:14:05 #3 at_exit_wrapper(void*)
/mnt/source/llvm/llvm-5.0.1.src-p7/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:361:31
(unifiedbetests+0x21b3554)
{code}
and
{code:java}
20:14:05 Previous write of size 8 at 0x0a8d3348 by thread T586:
20:14:05 #0 std::vector
>::_M_erase_at_end(impala::TJvmMemoryPool*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1798:30
(unifiedbetests+0x4afabcc)
20:14:05 #1 std::vector >::clear()
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/stl_vector.h:1499:9
(unifiedbetests+0x4afa4b4)
20:14:05 #2 unsigned int
impala::TGetJvmMemoryMetricsResponse::read(apache::thrift::protocol::TProtocol*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/Frontend_types.tcc:5673:32
(unifiedbetests+0x4afa21d)
20:14:05 #3 impala::Status
impala::DeserializeThriftMsg(unsigned
char const*, unsigned int*, bool, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/thrift-util.h:136:23
(unifiedbetests+0x4af9da6)
20:14:05 #4 impala::Status
impala::DeserializeThriftMsg(JNIEnv_*,
_jbyteArray*, impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/rpc/jni-thrift-util.h:61:3
(unifiedbetests+0x4af9c62)
20:14:05 #5 impala::Status
impala::JniCall::ObjectToResult(_jobject*,
impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:493:3
(unifiedbetests+0x4af9b24)
20:14:05 #6 impala::Status
impala::JniCall::Call(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.h:486:3
(unifiedbetests+0x4af92a6)
20:14:05 #7
impala::JniUtil::GetJvmMemoryMetrics(impala::TGetJvmMemoryMetricsResponse*)
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/jni-util.cc:299:72
(unifiedbetests+0x4af89a3)
20:14:05 #8 impala::JvmMetricCache::GrabMetricsIfNecessary()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:294:19
(unifiedbetests+0x4b2780d)
20:14:05 #9 impala::JvmMetricCache::GetCounterMetric(long
(*)(impala::TGetJvmMemoryMetricsResponse const&))
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:305:3
(unifiedbetests+0x4b27711)
20:14:05 #10 impala::JvmMemoryCounterMetric::GetValue()
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/util/memory-metrics.cc:270:41
(unifiedbetests+0x4b276bf)
20:14:05 #11 impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB
const*, impala::TExecPlanFragmentInfo const&)::$_3::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/repos/Impala/be/src/runtime/query-state.cc:185:50
(unifiedbetests+0x455bc15)
20:14:05 #12
boost::detail::function::function_obj_invoker0::invoke(boost::detail::function::function_buffer&)
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:137:18
(unifiedbetests+0x455b9d9)
20:14:05 #13 boost::function0::operator()() const
/data/jenkins/workspace/impala-cdw-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/function/function_template.hpp:763:14
[jira] [Commented] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760172#comment-17760172 ] Quanlong Huang commented on IMPALA-12402: - I just added you into the Contributors group. You should be able to assign JIRAs to yourself in the future. Welcome to the Impala community! > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reassigned IMPALA-12402: --- Assignee: Maxwell Guo > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-11772) Server crashes with SIGSEGV in TExprNode::operator= when running UBSAN tests
[
https://issues.apache.org/jira/browse/IMPALA-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760167#comment-17760167
]
Joe McDonnell commented on IMPALA-11772:
The latest occurrence also is crashing with SIGSEGV via
ImpalaHttpHandler::QueryStateHandler's call to QueryStateRecord::Init().
{noformat}
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 565 (crashed)
0 libc.so.6 + 0x15bac4
rax = 0x rdx = 0xf8924df0
rcx = 0x076db200 rbx = 0x13f83638
rsi = 0x076db210 rdi = 0x0010
rbp = 0x7f36e0cfa200 rsp = 0x7f36e0cf9ea8
r8 = 0xr9 = 0x00fff88ae03d
r10 = 0x r11 = 0x7f3822d74750
r12 = 0x r13 = 0x
r14 = 0xf8924e00 r15 = 0x13f83c70
rip = 0x7f3822d41ac4
Found by: given as instruction pointer in context
1 impalad!impala::TPlanNode::TPlanNode(impala::TPlanNode const&)
[PlanNodes_types.cpp : 3049 + 0x10]
rbp = 0x7f36e0cfa2d0 rsp = 0x7f36e0cfa210
rip = 0x0122ae13
Found by: previous frame's frame pointer
2 impalad!std::vector
>::operator=(std::vector >
const&) [stl_construct.h : 109 + 0x8]
rbp = 0x7f36e0cfa320 rsp = 0x7f36e0cfa2e0
rip = 0x01234834
Found by: previous frame's frame pointer
3 impalad!impala::TPlan::operator=(impala::TPlan const&) [PlanNodes_types.cpp
: 3220 + 0x5]
rbp = 0x7f36e0cfa340 rsp = 0x7f36e0cfa330
rip = 0x0122b2c7
Found by: previous frame's frame pointer
4 impalad!impala::TPlanFragment::TPlanFragment(impala::TPlanFragment const&)
[Planner_types.cpp : 110 + 0xc]
rbp = 0x7f36e0cfa390 rsp = 0x7f36e0cfa350
rip = 0x01235590
Found by: previous frame's frame pointer
5 impalad!void std::vector
>::_M_range_insert<__gnu_cxx::__normal_iterator > >
>(__gnu_cxx::__normal_iterator > >,
__gnu_cxx::__normal_iterator > >,
__gnu_cxx::__normal_iterator > >,
std::forward_iterator_tag) [stl_construct.h : 109 + 0x8]
rbp = 0x7f36e0cfa420 rsp = 0x7f36e0cfa3a0
rip = 0x01650eb0
Found by: previous frame's frame pointer
6
impalad!impala::ImpalaServer::QueryStateRecord::Init(impala::ClientRequestState
const&) [stl_vector.h : 1665 + 0x5]
rbp = 0x7f36e0cfa520 rsp = 0x7f36e0cfa430
rip = 0x015f6c22
Found by: previous frame's frame pointer
7
impalad!impala::ImpalaServer::QueryStateRecord::QueryStateRecord(impala::ClientRequestState
const&) [impala-server.cc : 2482 + 0x5]
rbp = 0x7f36e0cfa5b0 rsp = 0x7f36e0cfa530
rip = 0x015f73d6
Found by: previous frame's frame pointer
8
impalad!impala::ImpalaHttpHandler::QueryStateHandler(kudu::WebCallbackRegistry::WebRequest
const&, rapidjson::GenericDocument,
rapidjson::MemoryPoolAllocator,
rapidjson::CrtAllocator>*)::{lambda(std::shared_ptr
const&)#1}::operator()(std::shared_ptr const&) const + 0x2b
rbp = 0x7f36e0cfa8f0 rsp = 0x7f36e0cfa5c0
rip = 0x015d3eff
Found by: previous frame's frame pointer
9 impalad!std::_Function_handler&),
impala::ImpalaHttpHandler::QueryStateHandler(const WebRequest&,
rapidjson::Document*)::&)>
>::_M_invoke [invoke.h : 60 + 0x5]
rbp = 0x7f36e0cfa900 rsp = 0x7f36e0cfa900
rip = 0x015d3f3e
Found by: previous frame's frame pointer
10 impalad!std::function
const&)>::operator()(std::shared_ptr const&) const
[std_function.h : 622 + 0x3]
rbp = 0x7f36e0cfa910 rsp = 0x7f36e0cfa910
rip = 0x015d93bc
Found by: previous frame's frame pointer
11
impalad!impala::ImpalaHttpHandler::QueryStateHandler(kudu::WebCallbackRegistry::WebRequest
const&, rapidjson::GenericDocument,
rapidjson::MemoryPoolAllocator,
rapidjson::CrtAllocator>*) [sharded-query-map-util.h : 53 + 0x5]
rbp = 0x7f36e0cfad70 rsp = 0x7f36e0cfa920
rip = 0x015d3fe7
Found by: previous frame's frame pointer
12 impalad!boost::detail::function::void_function_obj_invoker2<(anonymous
namespace)::MakeCallback >*)>::, void, const kudu::WebCallbackRegistry::WebRequest&,
rapidjson::GenericDocument,
rapidjson::MemoryPoolAllocator,
rapidjson::CrtAllocator>*>::invoke [impala-http-handler.cc : 77 + 0x5]
rbp = 0x7f36e0cfad80 rsp = 0x7f36e0cfad80
rip = 0x015bd33d
Found by: previous frame's frame pointer
13 impalad!impala::Webserver::RenderUrlWithTemplate(sq_connection const*,
kudu::WebCallbackRegistry::WebRequest const&, impala::Webserver::UrlHandler
const&, std::__cxx11::basic_stringstream,
std::allocator >*, impala::ContentType*, std::__cxx11::basic_string, std::allocator > const&) [function_template.hpp :
763 + 0x6]
rbp = 0x7f36e0cfb330 rsp = 0x7f36e0cfad90
rip =
[jira] [Created] (IMPALA-12410) Impala's CONVERT TO ICEBERG statement does not retain table properties
Zoltán Borók-Nagy created IMPALA-12410: -- Summary: Impala's CONVERT TO ICEBERG statement does not retain table properties Key: IMPALA-12410 URL: https://issues.apache.org/jira/browse/IMPALA-12410 Project: IMPALA Issue Type: Bug Components: Frontend Reporter: Zoltán Borók-Nagy Impala's CONVERT TO ICEBERG statement does not retain table properties. Table properties should be retained except the ones used by Iceberg, e.g.: * metadata_location * iceberg.table_identifier * name * iceberg.mr.table.identifier -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12406) OPTIMIZE statement as an alias for INSERT OVERWRITE
[
https://issues.apache.org/jira/browse/IMPALA-12406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12406 started by Noemi Pap-Takacs.
-
> OPTIMIZE statement as an alias for INSERT OVERWRITE
> ---
>
> Key: IMPALA-12406
> URL: https://issues.apache.org/jira/browse/IMPALA-12406
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Reporter: Noemi Pap-Takacs
>Assignee: Noemi Pap-Takacs
>Priority: Major
> Labels: impala-iceberg
>
> If an Iceberg table is frequently updated/written to in small batches, a lot
> of small files are created. This decreases read performance. Similarly,
> frequent row-level deletes contribute to this problem by creating delete
> files which have to be merged on read.
> Currently INSERT OVERWRITE is used as a workaround to rewrite and compact
> Iceberg tables.
> OPTIMIZE statement offers a new syntax and an Iceberg specific solution to
> this problem.
> This patch introduces the new syntax as an alias for INSERT OVERWRITE.
> {code:java}
> Syntax: OPTIMIZE [TABLE] ;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12293) Optimize statement for compacting Iceberg tables
[
https://issues.apache.org/jira/browse/IMPALA-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12293 started by Noemi Pap-Takacs.
-
> Optimize statement for compacting Iceberg tables
>
>
> Key: IMPALA-12293
> URL: https://issues.apache.org/jira/browse/IMPALA-12293
> Project: IMPALA
> Issue Type: New Feature
> Components: Frontend
>Reporter: Noemi Pap-Takacs
>Assignee: Noemi Pap-Takacs
>Priority: Major
> Labels: impala-iceberg
>
> A simple syntax to compact Iceberg tables. It executes the following tasks:
> * compact small files
> * rewrite partitions according to latest spec
> * merge delete deltas
> {code:java}
> Syntax:
> OPTIMIZE TABLE
> [ REWRITE DATA ]
> [ ( { FILE_SIZE_THRESHOLD | MIN_INPUT_FILES } = [, ... ] ) ]
> [ WHERE ];{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6557) Show details of recent topic delta update
[ https://issues.apache.org/jira/browse/IMPALA-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reassigned IMPALA-6557: -- Assignee: Quanlong Huang > Show details of recent topic delta update > - > > Key: IMPALA-6557 > URL: https://issues.apache.org/jira/browse/IMPALA-6557 > Project: IMPALA > Issue Type: New Feature > Components: Catalog >Reporter: Juan Yu >Assignee: Quanlong Huang >Priority: Major > > Details of metadata topic delta updates are very useful for troubleshooting. > E.g. Num of tables and list of tables in recent topic updates help us know if > there are many tables being updated concurrently. > Are several large tables often updated together? > Are catalog cache version much higher than coordinator cache version? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6556) Show in-flight DDLs and what tables have been loading on Catalog WebUI
[ https://issues.apache.org/jira/browse/IMPALA-6556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reassigned IMPALA-6556: -- Assignee: Quanlong Huang > Show in-flight DDLs and what tables have been loading on Catalog WebUI > -- > > Key: IMPALA-6556 > URL: https://issues.apache.org/jira/browse/IMPALA-6556 > Project: IMPALA > Issue Type: New Feature > Components: Catalog >Reporter: Juan Yu >Assignee: Quanlong Huang >Priority: Major > > This helps users to know how many DDLs are running. How many tables have been > loading. > So users could know if a query is hung or just waiting for metadata. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-9118) Add debug page for in-flight DDLs in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759860#comment-17759860 ] Quanlong Huang commented on IMPALA-9118: Uploaded a patch for review: [https://gerrit.cloudera.org/c/20428/] Do you have time to take a look since you first added this page? [~tamas.mate] Here is a screenshot of the new /operations page: !Selection_082.png|width=681,height=395! > Add debug page for in-flight DDLs in catalogd > - > > Key: IMPALA-9118 > URL: https://issues.apache.org/jira/browse/IMPALA-9118 > Project: IMPALA > Issue Type: New Feature > Components: Catalog >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Major > Labels: observability, supportability > Attachments: Selection_082.png > > > In a busy cluster, it's possible that many DDL/DML queries keep in the > CREATED state for several minutes. Especially when using with sync_ddl=true, > tens of minutes are also possible. They may be waiting for the ExecDdl RPC to > catalogd to finish. > It'd be helpful for debugging DDL/DML hangs if we can show the in-flight DDLs > in catalogd. I think the following fields are important: > * thread id > * coordinator > * db name / table name > * ddl type, e.g. AddPartition, DropTable, CreateTable, etc. More types > [here|https://github.com/apache/impala/blob/3.3.0/common/thrift/JniCatalog.thrift#L31]. > * last event, e.g. waiting for table lock, got table lock, loading file > metadata, waiting for sync ddl version etc. > * start time > * time elapsed > * (optional) params link to show the TDdlExecRequest in json format > It'd be better to also include running REFRESH/INVALIDATE METADATA commands -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9118) Add debug page for in-flight DDLs in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang updated IMPALA-9118: --- Attachment: Selection_082.png > Add debug page for in-flight DDLs in catalogd > - > > Key: IMPALA-9118 > URL: https://issues.apache.org/jira/browse/IMPALA-9118 > Project: IMPALA > Issue Type: New Feature > Components: Catalog >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Major > Labels: observability, supportability > Attachments: Selection_082.png > > > In a busy cluster, it's possible that many DDL/DML queries keep in the > CREATED state for several minutes. Especially when using with sync_ddl=true, > tens of minutes are also possible. They may be waiting for the ExecDdl RPC to > catalogd to finish. > It'd be helpful for debugging DDL/DML hangs if we can show the in-flight DDLs > in catalogd. I think the following fields are important: > * thread id > * coordinator > * db name / table name > * ddl type, e.g. AddPartition, DropTable, CreateTable, etc. More types > [here|https://github.com/apache/impala/blob/3.3.0/common/thrift/JniCatalog.thrift#L31]. > * last event, e.g. waiting for table lock, got table lock, loading file > metadata, waiting for sync ddl version etc. > * start time > * time elapsed > * (optional) params link to show the TDdlExecRequest in json format > It'd be better to also include running REFRESH/INVALIDATE METADATA commands -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-9118) Add debug page for in-flight DDLs in catalogd
[ https://issues.apache.org/jira/browse/IMPALA-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-9118 started by Quanlong Huang. -- > Add debug page for in-flight DDLs in catalogd > - > > Key: IMPALA-9118 > URL: https://issues.apache.org/jira/browse/IMPALA-9118 > Project: IMPALA > Issue Type: New Feature > Components: Catalog >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Major > Labels: observability, supportability > Attachments: Selection_082.png > > > In a busy cluster, it's possible that many DDL/DML queries keep in the > CREATED state for several minutes. Especially when using with sync_ddl=true, > tens of minutes are also possible. They may be waiting for the ExecDdl RPC to > catalogd to finish. > It'd be helpful for debugging DDL/DML hangs if we can show the in-flight DDLs > in catalogd. I think the following fields are important: > * thread id > * coordinator > * db name / table name > * ddl type, e.g. AddPartition, DropTable, CreateTable, etc. More types > [here|https://github.com/apache/impala/blob/3.3.0/common/thrift/JniCatalog.thrift#L31]. > * last event, e.g. waiting for table lock, got table lock, loading file > metadata, waiting for sync ddl version etc. > * start time > * time elapsed > * (optional) params link to show the TDdlExecRequest in json format > It'd be better to also include running REFRESH/INVALIDATE METADATA commands -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Mate reassigned IMPALA-12402: --- Assignee: (was: Tamas Mate) > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Mate reassigned IMPALA-12402: --- Assignee: Tamas Mate > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Assignee: Tamas Mate >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12402: - Attachment: (was: 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch) > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12402: - Attachment: 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12402) Add some configurations for CatalogdMetaProvider's cache_
[ https://issues.apache.org/jira/browse/IMPALA-12402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12402: - Attachment: 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > Add some configurations for CatalogdMetaProvider's cache_ > - > > Key: IMPALA-12402 > URL: https://issues.apache.org/jira/browse/IMPALA-12402 > Project: IMPALA > Issue Type: Improvement > Components: fe >Reporter: Maxwell Guo >Priority: Minor > Labels: pull-request-available > Attachments: > 0001-IMPALA-12402-Add-some-configurations-for-CatalogdMet.patch > > > when the cluster contains many db and tables such as if there are more than > 10 tables, and if we restart the impalad , the local cache_ > CatalogMetaProvider's need to doing some loading process. > As we know that the goole's guava cache 's concurrencyLevel os set to 4 by > default. > but if there is many tables the loading process will need more time and > increase the probability of lock contention, see > [here|https://github.com/google/guava/blob/master/guava/src/com/google/common/cache/CacheBuilder.java#L437]. > > So we propose to add some configurations here, the first is the concurrency > of cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
