[jira] [Commented] (IMPALA-8173) run-workload.py KeyError on 'query_id'
[ https://issues.apache.org/jira/browse/IMPALA-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765658#comment-16765658 ] ASF subversion and git services commented on IMPALA-8173: - Commit 8b5cb576d23d771456ed571ee0d362d96af6ae13 in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=8b5cb57 ] IMPALA-8173: Fix KeyError in run-workload.py A recent change (IMPALA-7694) causes run-workload.py to fail with a KeyError due to trying to construct an ImpalaBeeswaxResult without a query id. This patch fixes that issue and two related issues: - This problem was not caught by automated testing even though we run run-workload.py in run-all-tests.sh because of an issue where queries that fail to produce results are silently ignored. - Fixes an issue where queries that fail to produce results would confusingly produce the error: 'NoneType' object has no attribute 'time_taken' Testing: - Ran run-workload.py locally and demonstrated that it works now. - Ran run-all-tests.sh locally and demonstrated that it fails now when the KeyError issue isn't fixed. Change-Id: I5b8a3c3dd7499335b9290d5667c194e8c0eabd12 Reviewed-on: http://gerrit.cloudera.org:8080/12397 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins > run-workload.py KeyError on 'query_id' > -- > > Key: IMPALA-8173 > URL: https://issues.apache.org/jira/browse/IMPALA-8173 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > > A recent commit (IMPALA-7694) broke bin/run-workload.py by requiring that an > ImpalaBeeswaxResult is constructed with a query_id available, which is > violated in query_exec_functions.py > We should fix this, and probably also add an automated test that runs > run-workload.py to prevent regressions like this in the future > [~lv] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7694) Add CPU resource utilization (user, system, iowait) timelines to profiles
[ https://issues.apache.org/jira/browse/IMPALA-7694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765659#comment-16765659 ] ASF subversion and git services commented on IMPALA-7694: - Commit 8b5cb576d23d771456ed571ee0d362d96af6ae13 in impala's branch refs/heads/master from Thomas Tauber-Marshall [ https://gitbox.apache.org/repos/asf?p=impala.git;h=8b5cb57 ] IMPALA-8173: Fix KeyError in run-workload.py A recent change (IMPALA-7694) causes run-workload.py to fail with a KeyError due to trying to construct an ImpalaBeeswaxResult without a query id. This patch fixes that issue and two related issues: - This problem was not caught by automated testing even though we run run-workload.py in run-all-tests.sh because of an issue where queries that fail to produce results are silently ignored. - Fixes an issue where queries that fail to produce results would confusingly produce the error: 'NoneType' object has no attribute 'time_taken' Testing: - Ran run-workload.py locally and demonstrated that it works now. - Ran run-all-tests.sh locally and demonstrated that it fails now when the KeyError issue isn't fixed. Change-Id: I5b8a3c3dd7499335b9290d5667c194e8c0eabd12 Reviewed-on: http://gerrit.cloudera.org:8080/12397 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins > Add CPU resource utilization (user, system, iowait) timelines to profiles > - > > Key: IMPALA-7694 > URL: https://issues.apache.org/jira/browse/IMPALA-7694 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Major > Labels: observability, supportability > Fix For: Impala 3.2.0 > > > We often struggle to determine why a query was slow, in particular if it was > caused by other tasks on the same machine using resources. To help with this > we should include timelines for system resource utilization to the profiles. > These should eventually include CPU and disk and network I/O. If it is too > expensive to include these in all queries we should add a flag to add these > to a percentage of queries, and a query option to force-enable them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8173) run-workload.py KeyError on 'query_id'
[ https://issues.apache.org/jira/browse/IMPALA-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-8173. Resolution: Fixed Fix Version/s: Impala 3.3.0 > run-workload.py KeyError on 'query_id' > -- > > Key: IMPALA-8173 > URL: https://issues.apache.org/jira/browse/IMPALA-8173 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.2.0 >Reporter: Thomas Tauber-Marshall >Assignee: Thomas Tauber-Marshall >Priority: Major > Fix For: Impala 3.3.0 > > > A recent commit (IMPALA-7694) broke bin/run-workload.py by requiring that an > ImpalaBeeswaxResult is constructed with a query_id available, which is > violated in query_exec_functions.py > We should fix this, and probably also add an automated test that runs > run-workload.py to prevent regressions like this in the future > [~lv] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5031) UBSAN clean and method for testing UBSAN cleanliness
[ https://issues.apache.org/jira/browse/IMPALA-5031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765660#comment-16765660 ] ASF subversion and git services commented on IMPALA-5031: - Commit 6fc05a6f7385bf21de1c01a3071b9a91efc85b07 in impala's branch refs/heads/master from Jim Apple [ https://gitbox.apache.org/repos/asf?p=impala.git;h=6fc05a6 ] IMPALA-5031: signed overflow is undefined behavior Tis patch fixes a signed overflow in the test StringToDecimal.LargeDecimals. The interesting part of the backtrace is: util/string-parser.h:397:14: runtime error: signed integer overflow: 0x4b3b4ca85a86c47a098a223f * 10 cannot be represented in type '__int128' #0 void StringParser::ApplyExponent<__int128>(int, int, signed char, __int128*, int*, int*) util/string-parser.h:397:14 #1 DecimalValue<__int128> StringParser::StringToDecimal<__int128> (char const*, int, int, int, bool, StringParser::ParseResult*) util/string-parser.h:221:5 #2 void VerifyParse<__int128>(string const&, int, int, bool, DecimalValue<__int128> const&, StringParser::ParseResult) runtime/decimal-test.cc:53:25 #3 void VerifyParse<__int128>(string const&, int, int, DecimalValue<__int128> const&, StringParser::ParseResult) runtime/decimal-test.cc:65:3 #4 StringToDecimal_LargeDecimals_Test::TestBody() runtime/decimal-test.cc:443:3 Change-Id: Ifb4effa291e1e4dac62b84251c74c483d99b06e7 Reviewed-on: http://gerrit.cloudera.org:8080/12426 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins > UBSAN clean and method for testing UBSAN cleanliness > > > Key: IMPALA-5031 > URL: https://issues.apache.org/jira/browse/IMPALA-5031 > Project: IMPALA > Issue Type: Task > Components: Backend, Infrastructure >Affects Versions: Impala 2.9.0 >Reporter: Jim Apple >Assignee: Jim Apple >Priority: Minor > > http://releases.llvm.org/3.8.0/tools/clang/docs/UndefinedBehaviorSanitizer.html > builds are supported after https://gerrit.cloudera.org/#/c/6186/, but > Impala's test suite triggers many errors under UBSAN. Those errors should be > fixed and then there should be a way to run the test suite under UBSAN and > fail if there were any errors detected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8177) Log DDL exceptions in the coordinator log [supportability]
[ https://issues.apache.org/jira/browse/IMPALA-8177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765656#comment-16765656 ] ASF subversion and git services commented on IMPALA-8177: - Commit fa604fb5059307321a174db02469b4f5fb78410f in impala's branch refs/heads/master from Bharath Vissapragada [ https://gitbox.apache.org/repos/asf?p=impala.git;h=fa604fb ] IMPALA-8177: Log DDL failures in coordinator logs If a DDL fails for some reason, it helps to log the failure message in the coordinator logs so that we can differentiate between failed and successful DDL queries. For ex: [0d66cd6004b4:21000] default> drop database foo; Query: drop database foo ERROR: ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: CAUSED BY: NoSuchObjectException: foo - coordinator logs --- I0208 12:13:26.251695 25474 Frontend.java:1242] 704e86fff482c0b5:f0501f94] Analyzing query: drop database foo I0208 12:13:26.253773 25474 Frontend.java:1282] 704e86fff482c0b5:f0501f94] Analysis finished. I0208 12:13:26.419946 25474 client-request-state.cc:176] 704e86fff482c0b5:f0501f94] ImpalaRuntimeException: Error making 'dropDatabase' RPC to Hive Metastore: CAUSED BY: NoSuchObjectException: foo I0208 12:13:26.419992 25474 impala-server.cc:1142] 704e86fff482c0b5:f0501f94] UnregisterQuery(): query_id=704e86fff482c0b5:f0501f94 I0208 12:13:26.419997 25474 impala-server.cc:1249] 704e86fff482c0b5:f0501f94] Cancel(): query_id=704e86fff482c0b5:f0501f94 --- Testing: Verified manually by running a few DDLs that fail and then inspecting the coordinator log file. Change-Id: Ie89291ee27156c701e07cea44ad3ee07ec54ab42 Reviewed-on: http://gerrit.cloudera.org:8080/12414 Reviewed-by: Bharath Vissapragada Tested-by: Impala Public Jenkins > Log DDL exceptions in the coordinator log [supportability] > -- > > Key: IMPALA-8177 > URL: https://issues.apache.org/jira/browse/IMPALA-8177 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Catalog >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: bharath v >Assignee: bharath v >Priority: Major > Labels: supportability > Fix For: Impala 3.2.0 > > > I ran into this issue while debugging DDL failures. > If a DDL fails for some reason, we propagate the error back to the shell and > update the runtime profile, but we do not log the failure message to the > coordinator logs. This means that the coordinator side logging remains the > same for both successful and failed DDL queries. > We need to log the failure status in the coordinator logs so that it is easy > to differentiate successful vs failed queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7214) Update Impala docs to reflect coordinator/executor separation and decoupling from DataNodes.
[
https://issues.apache.org/jira/browse/IMPALA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765657#comment-16765657
]
ASF subversion and git services commented on IMPALA-7214:
-
Commit 697a15b341186046d8fae3a2139f1ad13d304734 in impala's branch
refs/heads/master from Alex Rodoni
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=697a15b ]
IMPALA-7214: [DOCS] Update Impala docs to decouple Impala and DataNodes
- Take 1: Let's review these docs before we go clean up many more.
Change-Id: I1c91f7975c09dae9908591eeeac0d55e5355b2d4
Reviewed-on: http://gerrit.cloudera.org:8080/12400
Reviewed-by: Alex Rodoni
Tested-by: Impala Public Jenkins
> Update Impala docs to reflect coordinator/executor separation and decoupling
> from DataNodes.
>
>
> Key: IMPALA-7214
> URL: https://issues.apache.org/jira/browse/IMPALA-7214
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 2.12.0
>Reporter: Tim Armstrong
>Assignee: Alex Rodoni
>Priority: Major
>
> The docs tend to conflate DataNodes (a HDFS service) and Impala daemons. I
> think this stems from the original deployment practice of always colocating
> Impala daemons with HDFS datanodes so that HDFS data could always be read
> from a local DataNode.
> I'm a bit pedantic so the conflation feels wrong to me regardless, but I
> think this will become increasingly confusing as alternative deployments
> without colocated HDFS DataNodes become more common (e.g. running against S3,
> running with a separate HDFS service).
> E.g. picking an example at random:
> {noformat}
> In Impala 1.4.0 and higher, the LIMIT clause is now
> optional (rather than required) for
> queries that use the ORDER BY clause. Impala
> automatically uses a temporary disk work area
> to perform the sort if the sort operation would otherwise exceed the
> Impala memory limit for a particular
> DataNode.
> {noformat}
> This is wrong because the memory limit is for an Impala daemon, which is the
> process that does the actual sorting. So here I think it should be "Impala
> daemon" instead of "DataNode".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-8155) Switch to Impala-lzo/2.x for Impala-2.x
[ https://issues.apache.org/jira/browse/IMPALA-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang reassigned IMPALA-8155: -- Assignee: Quanlong Huang > Switch to Impala-lzo/2.x for Impala-2.x > --- > > Key: IMPALA-8155 > URL: https://issues.apache.org/jira/browse/IMPALA-8155 > Project: IMPALA > Issue Type: Task >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Major > Fix For: Impala 2.13.0 > > > Impala-2.x is currently based on Cloudera/Impala-lzo master branch. As it > updated, builds of Impala-2.x will fail. We need to switch to another branch > that points to the original commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8155) Switch to Impala-lzo/2.x for Impala-2.x
[ https://issues.apache.org/jira/browse/IMPALA-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang updated IMPALA-8155: --- Fix Version/s: Impala 2.13.0 > Switch to Impala-lzo/2.x for Impala-2.x > --- > > Key: IMPALA-8155 > URL: https://issues.apache.org/jira/browse/IMPALA-8155 > Project: IMPALA > Issue Type: Task >Reporter: Quanlong Huang >Priority: Major > Fix For: Impala 2.13.0 > > > Impala-2.x is currently based on Cloudera/Impala-lzo master branch. As it > updated, builds of Impala-2.x will fail. We need to switch to another branch > that points to the original commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8178) Tests failing with “Could not allocate memory while trying to increase reservation” on EC filesystem
[
https://issues.apache.org/jira/browse/IMPALA-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated IMPALA-8178:
---
Summary: Tests failing with “Could not allocate memory while trying to
increase reservation” on EC filesystem (was: Tests failing with “Memory is
likely oversubscribed” on EC filesystem)
> Tests failing with “Could not allocate memory while trying to increase
> reservation” on EC filesystem
>
>
> Key: IMPALA-8178
> URL: https://issues.apache.org/jira/browse/IMPALA-8178
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> In tests run against an Erasure Coding filesystem, multiple tests failed with
> memory allocation errors.
> In total 10 tests failed:
> * query_test.test_scanners.TestParquet.test_decimal_encodings
> * query_test.test_scanners.TestTpchScanRangeLengths.test_tpch_scan_ranges
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 0]
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 1]
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node
> * query_test.test_scanners.TestParquet.test_def_levels
> *
> query_test.test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiterquery_test.test_hbase_queries.TestHBaseQueries.test_hbase_filters
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_inline_views
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_top_n
> The first failure looked like this on the client side:
> {quote}
> F
> query_test/test_scanners.py::TestParquet::()::test_decimal_encodings[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_scanners.py:717: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:472: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:699: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:381: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:ExecQueryFInstances rpc
> query_id=6e44c3c949a31be2:f973c7ff failed: Failed to get minimum
> memory reservation of 8.00 KB on daemon xxx.com:22001 for query
> 6e44c3c949a31be2:f973c7ff due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(6e44c3c949a31be2:f973c7ff) could not allocate 8.00 KB
> without exceeding limit.
> E Error occurred on backend xxx.com:22001
> E Memory left in process limit: 1.19 GB
> E Query(6e44c3c949a31be2:f973c7ff): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {quote}
> On the server side log:
> {quote}
> I0207 18:25:19.329311 5562 impala-server.cc:1063]
> 6e44c3c949a31be2:f973c7ff] Registered query
> query_id=6e44c3c949a31be2:f973c7ff
> session_id=93497065f69e9d01:8a3bd06faff3da5
> I0207 18:25:19.329434 5562 Frontend.java:1242]
> 6e44c3c949a31be2:f973c7ff] Analyzing query: select score from
> decimal_stored_as_int32
> I0207 18:25:19.329583 5562 FeSupport.java:285]
> 6e44c3c949a31be2:f973c7ff] Requesting prioritized load of table(s):
> test_decimal_encodings_28d99c0e.decimal_stored_as_int32
> I0207 18:25:30.776041 5562 Frontend.java:1282]
> 6e44c3c949a31be2:f973c7ff] Analysis finished.
> I0207 18:25:35.919486 10418 admission-controller.cc:608]
> 6e44c3c949a31be2:f973c7ff] Schedule for
> id=6e44c3c949a31be2:f973c7ff in pool_name=default-pool
> per_host_mem_estimate=16.02 MB PoolConfig: max_requests=-1 max_queued=200
> max_mem=-1.00 B
> I0207 18:25:35.919528 10418 admission-controller.cc:613]
> 6e44c3c949a31be2:f973c7ff] Stats: agg_num_running=2,
> agg_num_queued=0, agg_mem_reserved=24.13
[jira] [Comment Edited] (IMPALA-8178) Tests failing with “Memory is likely oversubscribed” on EC filesystem
[
https://issues.apache.org/jira/browse/IMPALA-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765561#comment-16765561
]
Andrew Sherman edited comment on IMPALA-8178 at 2/12/19 12:17 AM:
--
Another failure:
* test_scanners.TestTextScanRangeLengths.test_text_scanner_with_header
* test_queries.TestQueries.test_analytic_fns
* test_scanners.TestParquet.test_rle_encoded_bools
* test_kudu.TestCreateExternalTable.test_column_name_case
* test_scanners.TestParquet.test_corrupt_rle_counts
* test_queries.TestQueries.test_limit
* test_scanners.TestTextSplitDelimiters.test_text_split_delimiters
* test_scanners.TestParquet.test_rle_encoded_bools
*
test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiter
* test_spilling.TestSpillingDebugActionDimensions.test_spilling
so this is probably not about a particular one of these tests
was (Author: asherman):
Another failure:
* test_scanners.TestTextScanRangeLengths.test_text_scanner_with_header
* test_queries.TestQueries.test_analytic_fns
* test_scanners.TestParquet.test_rle_encoded_bools
* test_kudu.TestCreateExternalTable.test_column_name_case
* test_scanners.TestParquet.test_corrupt_rle_counts
* test_queries.TestQueries.test_limit
* test_scanners.TestTextSplitDelimiters.test_text_split_delimiters
* test_scanners.TestParquet.test_rle_encoded_bools
*
test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiter
* test_spilling.TestSpillingDebugActionDimensions.test_spilling
so this is probably not about a particular one of these tests
> Tests failing with “Memory is likely oversubscribed” on EC filesystem
> -
>
> Key: IMPALA-8178
> URL: https://issues.apache.org/jira/browse/IMPALA-8178
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> In tests run against an Erasure Coding filesystem, multiple tests failed with
> memory allocation errors.
> In total 10 tests failed:
> * query_test.test_scanners.TestParquet.test_decimal_encodings
> * query_test.test_scanners.TestTpchScanRangeLengths.test_tpch_scan_ranges
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 0]
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 1]
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node
> * query_test.test_scanners.TestParquet.test_def_levels
> *
> query_test.test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiterquery_test.test_hbase_queries.TestHBaseQueries.test_hbase_filters
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_inline_views
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_top_n
> The first failure looked like this on the client side:
> {quote}
> F
> query_test/test_scanners.py::TestParquet::()::test_decimal_encodings[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_scanners.py:717: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:472: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:699: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:381: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:ExecQueryFInstances rpc
> query_id=6e44c3c949a31be2:f973c7ff failed: Failed to get minimum
> memory reservation of 8.00 KB on daemon xxx.com:22001 for query
> 6e44c3c949a31be2:f973c7ff due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(6e44c3c949a31be2:f973c7ff) could not allocate 8.00 KB
> without exceeding limit.
> E Error occurred on backend xxx.com:22001
> E Memory left in process limit: 1.19 GB
> E Query(6e44c3c949a31be2:f973c7ff): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency
[jira] [Commented] (IMPALA-8178) Tests failing with “Memory is likely oversubscribed” on EC filesystem
[
https://issues.apache.org/jira/browse/IMPALA-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765561#comment-16765561
]
Andrew Sherman commented on IMPALA-8178:
Another failure:
* test_scanners.TestTextScanRangeLengths.test_text_scanner_with_header
* test_queries.TestQueries.test_analytic_fns
* test_scanners.TestParquet.test_rle_encoded_bools
* test_kudu.TestCreateExternalTable.test_column_name_case
* test_scanners.TestParquet.test_corrupt_rle_counts
* test_queries.TestQueries.test_limit
* test_scanners.TestTextSplitDelimiters.test_text_split_delimiters
* test_scanners.TestParquet.test_rle_encoded_bools
*
test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiter
* test_spilling.TestSpillingDebugActionDimensions.test_spilling
so this is probably not about a particular one of these tests
> Tests failing with “Memory is likely oversubscribed” on EC filesystem
> -
>
> Key: IMPALA-8178
> URL: https://issues.apache.org/jira/browse/IMPALA-8178
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
>
> In tests run against an Erasure Coding filesystem, multiple tests failed with
> memory allocation errors.
> In total 10 tests failed:
> * query_test.test_scanners.TestParquet.test_decimal_encodings
> * query_test.test_scanners.TestTpchScanRangeLengths.test_tpch_scan_ranges
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 0]
> * query_test.test_exprs.TestExprs.test_exprs [enable_expr_rewrites: 1]
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node
> * query_test.test_scanners.TestParquet.test_def_levels
> *
> query_test.test_scanners.TestTextSplitDelimiters.test_text_split_across_buffers_delimiterquery_test.test_hbase_queries.TestHBaseQueries.test_hbase_filters
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_inline_views
> * query_test.test_hbase_queries.TestHBaseQueries.test_hbase_top_n
> The first failure looked like this on the client side:
> {quote}
> F
> query_test/test_scanners.py::TestParquet::()::test_decimal_encodings[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'debug_action':
> '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5',
> 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
> query_test/test_scanners.py:717: in test_decimal_encodings
> self.run_test_case('QueryTest/parquet-decimal-formats', vector,
> unique_database)
> common/impala_test_suite.py:472: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:699: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:381: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:ExecQueryFInstances rpc
> query_id=6e44c3c949a31be2:f973c7ff failed: Failed to get minimum
> memory reservation of 8.00 KB on daemon xxx.com:22001 for query
> 6e44c3c949a31be2:f973c7ff due to following error: Memory limit
> exceeded: Could not allocate memory while trying to increase reservation.
> E Query(6e44c3c949a31be2:f973c7ff) could not allocate 8.00 KB
> without exceeding limit.
> E Error occurred on backend xxx.com:22001
> E Memory left in process limit: 1.19 GB
> E Query(6e44c3c949a31be2:f973c7ff): Reservation=0
> ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or
> configuring admission control may help avoid this error.
> {quote}
> On the server side log:
> {quote}
> I0207 18:25:19.329311 5562 impala-server.cc:1063]
> 6e44c3c949a31be2:f973c7ff] Registered query
> query_id=6e44c3c949a31be2:f973c7ff
> session_id=93497065f69e9d01:8a3bd06faff3da5
> I0207 18:25:19.329434 5562 Frontend.java:1242]
> 6e44c3c949a31be2:f973c7ff] Analyzing query: select score from
> decimal_stored_as_int32
> I0207 18:25:19.329583 5562 FeSupport.java:285]
> 6e44c3c949a31be2:f973c7ff] Requesting prioritized load of table(s):
> test_decimal_encodings_28d99c0e.decimal_stored_as_int32
> I0207 18:25:30.776041 5562 Frontend.java:1282]
> 6e44c3c949a31be2:f973c7ff] Analysis
[jira] [Commented] (IMPALA-8188) Some SSDs are not properly detected as non-rotational
[
https://issues.apache.org/jira/browse/IMPALA-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765552#comment-16765552
]
Philip Zeyliger commented on IMPALA-8188:
-
https://unix.stackexchange.com/posts/308724/revisions seems to point to
following {{/sys/class/block/nvme0n1p1}} (or similar) to look at this. It's all
messy.
> Some SSDs are not properly detected as non-rotational
> -
>
> Key: IMPALA-8188
> URL: https://issues.apache.org/jira/browse/IMPALA-8188
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Joe McDonnell
>Priority: Critical
>
> Here is an example Impala log:
>
> {noformat}
> I0211 10:50:40.650727 18344 init.cc:288] Disk Info:
> Num disks 2:
> nvme0n (rotational=true)
> nvme0n1p (rotational=true){noformat}
> I logged into an equivalent machine, and the OS sees these as not rotational:
>
>
> {noformat}
> # cat /sys/block/nvme0n1/queue/rotational
> 0
> {noformat}
> Device names that end in a number get trimmed (i.e. /dev/sda2 becomes
> /dev/sda). See
> [https://github.com/apache/impala/blob/master/be/src/util/disk-info.cc#L73-L74]
> These devices don't follow that pattern, so we don't find the right files.
> Neither /sys/block/nvme0n nor /sys/block/nvme0n1p exist, so both fall back to
> being rotational.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8183) TestRPCTimeout.test_reportexecstatus_retry times out
[
https://issues.apache.org/jira/browse/IMPALA-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765520#comment-16765520
]
Andrew Sherman commented on IMPALA-8183:
Log says:
E0210 11:23:32.714677 21460 query-state.cc:424]
5a4a60d227d5669e:cfc4ed21] Cancelling fragment instances due to failure
to reach the coordinator. (ReportExecStatus() RPC failed: Remote error: Service
unavailable: ReportExecStatus request on impala.ControlService from
127.0.0.1:49775 dropped due to backpressure. The service queue is full; it has
2147483647 items.
). Query 5a4a60d227d5669e:cfc4ed21 may hang. See IMPALA-2990.
so is this just an instance of IMPALA-2990?
> TestRPCTimeout.test_reportexecstatus_retry times out
>
>
> Key: IMPALA-8183
> URL: https://issues.apache.org/jira/browse/IMPALA-8183
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Thomas Tauber-Marshall
>Priority: Major
>
> There are 2 forms of failure, where the test itself times out, and where the
> whole test run times out, suspiciously just after running
> test_reportexecstatus_retry
> {quote}
> Error Message
> Failed: Timeout >7200s
> Stacktrace
> custom_cluster/test_rpc_timeout.py:143: in test_reportexecstatus_retry
> self.execute_query_verify_metrics(self.TEST_QUERY, None, 10)
> custom_cluster/test_rpc_timeout.py:45: in execute_query_verify_metrics
> self.execute_query(query, query_options)
> common/impala_test_suite.py:601: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:632: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:699: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:384: in wait_for_finished
> time.sleep(0.05)
> E Failed: Timeout >7200s
> {quote}
> {quote}
> Test run timed out. This probably happened due to a hung thread which can be
> confirmed by looking at the stacktrace of running impalad processes at
> /data/jenkins/workspace/xxx/repos/Impala/logs/timeout_stacktrace
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8183) TestRPCTimeout.test_reportexecstatus_retry times out
[
https://issues.apache.org/jira/browse/IMPALA-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765502#comment-16765502
]
Andrew Sherman commented on IMPALA-8183:
Code that hangs is
[https://github.com/apache/impala/blob/master/tests/beeswax/impala_beeswax.py#L372-L384|https://github.com/apache/impala/blob/master/tests/beeswax/impala_beeswax.py#L372-L384]
which suggests query has some other state than FINISHED or EXCEPTION.
Meanwhile log files contain no trace of the query id
3d486150ff28c15c:118403c2
> TestRPCTimeout.test_reportexecstatus_retry times out
>
>
> Key: IMPALA-8183
> URL: https://issues.apache.org/jira/browse/IMPALA-8183
> Project: IMPALA
> Issue Type: Bug
>Reporter: Andrew Sherman
>Priority: Major
>
> There are 2 forms of failure, where the test itself times out, and where the
> whole test run times out, suspiciously just after running
> test_reportexecstatus_retry
> {quote}
> Error Message
> Failed: Timeout >7200s
> Stacktrace
> custom_cluster/test_rpc_timeout.py:143: in test_reportexecstatus_retry
> self.execute_query_verify_metrics(self.TEST_QUERY, None, 10)
> custom_cluster/test_rpc_timeout.py:45: in execute_query_verify_metrics
> self.execute_query(query, query_options)
> common/impala_test_suite.py:601: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:632: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:699: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:360: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:384: in wait_for_finished
> time.sleep(0.05)
> E Failed: Timeout >7200s
> {quote}
> {quote}
> Test run timed out. This probably happened due to a hung thread which can be
> confirmed by looking at the stacktrace of running impalad processes at
> /data/jenkins/workspace/xxx/repos/Impala/logs/timeout_stacktrace
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8187) UDF/UDA samples should explicitly export entry points
Tim Armstrong created IMPALA-8187: - Summary: UDF/UDA samples should explicitly export entry points Key: IMPALA-8187 URL: https://issues.apache.org/jira/browse/IMPALA-8187 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Tim Armstrong Assignee: Tim Armstrong It's possible for UDF authors to get into a pickle by exporting symbols from UDF/UDA shared objects unintentionally. E.g. if using boost headers, which result in weak symbols in the .so, which then get merged with the (incompatible) symbols in the impalad binary. We should really be keeping symbols hidden by default then exporting the entry points explicitly in the UDF/UDA code -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3873) Proxy class for QueryExecState and QueryStateRecord
[
https://issues.apache.org/jira/browse/IMPALA-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-3873.
---
Resolution: Later
Assignee: (was: Henry Robinson)
No sense in keeping a code cleanup JIRA open and untouched for years
> Proxy class for QueryExecState and QueryStateRecord
> ---
>
> Key: IMPALA-3873
> URL: https://issues.apache.org/jira/browse/IMPALA-3873
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 2.6.0
>Reporter: Henry Robinson
>Priority: Minor
> Labels: observability
>
> {{ImpalaServer}} keeps historical query information in two forms:
> # {{QueryExecState}} is for actively executing queries
> # {{QueryStateRecord}} is an archive of the final state of a query.
> In several places, most often in the debug page generators, we want to treat
> these as the same object because we're pulling the same data out from each of
> them. This leads to repeated code like:
> {code}
> if (query_id in active list) {
> // Pull out relevant data from QueryExecState
> } else if (query_id in archive log) {
> // Pull out relevant data from QueryStateRecord
> } else {
> // Handle not found case
> }
> {code}
> and it would obviously be better to write:
> {code}
> if (query_id in either active list or archive log) {
> // Pull out relevant data from common data structure
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5043) Admission control error messages don't hint that information is stale when disconnected from statestore
[ https://issues.apache.org/jira/browse/IMPALA-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-5043. --- Resolution: Fixed Fix Version/s: Impala 3.2.0 > Admission control error messages don't hint that information is stale when > disconnected from statestore > --- > > Key: IMPALA-5043 > URL: https://issues.apache.org/jira/browse/IMPALA-5043 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.6.0 >Reporter: Thomas Scott >Assignee: Tim Armstrong >Priority: Major > Labels: admission-control, resource-management, supportability > Fix For: Impala 3.2.0 > > > When (for whatever reason) one or more daemons are disconnected from the > statestore the admission control data held on the daemon goes stale. This can > lead to the daemon accepting queries when there is not capacity or rejecting > queries when there is capacity. > For example, a pool somepool has a limit of 10 concurrent queries and is at > that limit when a daemon is disconnected from the statestore. Even when other > queries in somepool finish and the pool is now empty the disconnected daemon > will report the following when new queries are executed: > ERROR: Admission for query exceeded timeout 6ms. Queued reason: number of > running queries 10 is over limit 10 > Could we have some warning to say that the admission control data is stale > here? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8186) Automate setup of docker bridge network for dockerised minicluster
Tim Armstrong created IMPALA-8186:
-
Summary: Automate setup of docker bridge network for dockerised
minicluster
Key: IMPALA-8186
URL: https://issues.apache.org/jira/browse/IMPALA-8186
Project: IMPALA
Issue Type: Sub-task
Components: Infrastructure
Reporter: Tim Armstrong
Assignee: Tim Armstrong
I had some manual setup that I did to create a docker network and set
INTERNAL_LISTEN_HOST as the gateway. I think we could automate it with some
logic like the following:
{code}
docker network create -d bridge impala-cluster
IMPALA_NETWORK_GATEWAY=$(docker network inspect impala-cluster | grep -o
'"Gateway": "[0-9\.]*"' | grep -o "[0-9.]*")
echo "export INTERNAL_LISTEN_HOST=${IMPALA_NETWORK_GATEWAY}" >>
bin/impala-config-local.sh
echo 'export DEFAULT_FS=hdfs://${INTERNAL_LISTEN_HOST}:20500' >>
bin/impala-config-local.sh
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8182) Extend PlanCtx to provide access to the entire internal plan
[
https://issues.apache.org/jira/browse/IMPALA-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8182 started by Paul Rogers.
---
> Extend PlanCtx to provide access to the entire internal plan
>
>
> Key: IMPALA-8182
> URL: https://issues.apache.org/jira/browse/IMPALA-8182
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Historically, the Impala planner is a function: AST in one end, Thrift plan
> out the other. The planner itself creates a rich intermediate plan
> representation, but none of those objects were available for testing, forcing
> all tests, no matter how detailed, to run as end-to-end tests: SQL in one
> end, verify a (very abbreviated) DESCRIBE output out the other.
> A recent change introduced the {{PlanCtx}} that provided access to the
> parallelized plan fragments. Doing so allowed adding a variety of cardinality
> tests.
> It turns out that additional testing requires access to the "single node
> plan" in order to verify things like join cardinality.
> This ticket asks to modify the {{PlanCtx}} to capture all the internal plan
> nodes: both single node and distributed, to enable full unit testing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8181) Show abbreviated row counts in DESCRIBE output
[ https://issues.apache.org/jira/browse/IMPALA-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8181 started by Paul Rogers. --- > Show abbreviated row counts in DESCRIBE output > -- > > Key: IMPALA-8181 > URL: https://issues.apache.org/jira/browse/IMPALA-8181 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > > A recent change added plan node cardinality to the DESCRIBE output using a > "metric" format: 123.46M instead of 123456789. This makes the output easier > to read. Also, since all numbers are estimate, having seven or eight digits > of precision is not very helpful. > This ticket requests that the same formatting be used for the other places > where the DESCRIBE output shows row counts: table stats, extrapolated row > count, partition row counts, and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8185) Factor out production/mock file system logic into a facade
[
https://issues.apache.org/jira/browse/IMPALA-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8185 started by Paul Rogers.
---
> Factor out production/mock file system logic into a facade
> --
>
> Key: IMPALA-8185
> URL: https://issues.apache.org/jira/browse/IMPALA-8185
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> The recent addition of the test case builder allows the planner to work
> either against a real system ("production" mode) or against metadata captured
> in a test case ("test case" mode.)
> This ticket asks to move code for the two cases into a facade class so that
> we pick the right facade for each mode. Move code currently in if-statements
> into the facade.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8185) Factor out production/mock file system logic into a facade
Paul Rogers created IMPALA-8185:
---
Summary: Factor out production/mock file system logic into a facade
Key: IMPALA-8185
URL: https://issues.apache.org/jira/browse/IMPALA-8185
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
The recent addition of the test case builder allows the planner to work either
against a real system ("production" mode) or against metadata captured in a
test case ("test case" mode.)
This ticket asks to move code for the two cases into a facade class so that we
pick the right facade for each mode. Move code currently in if-statements into
the facade.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8184) Add timestamp validation to Orc scanner
Csaba Ringhofer created IMPALA-8184:
---
Summary: Add timestamp validation to Orc scanner
Key: IMPALA-8184
URL: https://issues.apache.org/jira/browse/IMPALA-8184
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Csaba Ringhofer
Similarly to Parquet, Orc can also contain timestamps that are not valid in
Impala, e.g. Hive can insert timestamps before 1400 while these are invalid in
Impala. These invalid timestamps are often handled similarly to NULL, bur are
actually not "real" NULLs, which can lead to some some weird behavior:
Hive:
create table orcts (ts timestamp) stored as orc;
insert into orcts values ("1200-01-01");
Impala:
select * from orcts where ts is not null;
Returns 1 row:
NULL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8179) Backend tests are flaky: time-based srand
[
https://issues.apache.org/jira/browse/IMPALA-8179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765216#comment-16765216
]
Joe McDonnell edited comment on IMPALA-8179 at 2/11/19 5:59 PM:
I will provide a way to override the seed. That was something I had meant to do
and forgot.
We were seeding with time before this latest change. All of our tests call
InitCommonRuntime() (e.g.
[IMPALA_TEST_MAIN()|https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]),
and that calls srand(time(NULL)):
[https://github.com/apache/impala/blob/master/be/src/common/init.cc#L201]
was (Author: joemcdonnell):
I will provide a way to override the seed. That was something I had meant to do
and forgot.
We were seeding with time before this latest change. All of our tests call
InitCommonRuntime() (e.g. [[IMPALA_TEST_MAIN()
||https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]
[https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]
[]|https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]),
and that calls srand(time(NULL)):
[https://github.com/apache/impala/blob/master/be/src/common/init.cc#L201]
> Backend tests are flaky: time-based srand
> -
>
> Key: IMPALA-8179
> URL: https://issues.apache.org/jira/browse/IMPALA-8179
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Jim Apple
>Assignee: Joe McDonnell
>Priority: Major
>
> https://gerrit.cloudera.org/#/c/12124/9/be/src/service/unified-betest-main.cc
> added
> {noformat}
> uint32_t seed = time(NULL);
> cout << "seed = " << seed << endl;
> srand(seed);
> {noformat}
> There is no way to force the seed, and backend tests using {{rand()}} are
> then flaky by default. We should make them unflaky by default (using a
> specific random seed) and we should also provide a way to override this by
> setting it by time or to a specific seed for just one run, to help with
> debugging.
> If we want the coverage that comes from running with a different seed each
> time, we could run a test multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8179) Backend tests are flaky: time-based srand
[
https://issues.apache.org/jira/browse/IMPALA-8179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765216#comment-16765216
]
Joe McDonnell commented on IMPALA-8179:
---
I will provide a way to override the seed. That was something I had meant to do
and forgot.
We were seeding with time before this latest change. All of our tests call
InitCommonRuntime() (e.g. [[IMPALA_TEST_MAIN()
||https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]
[https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]
[]|https://github.com/apache/impala/blob/master/be/src/testutil/gtest-util.h#L55]),
and that calls srand(time(NULL)):
[https://github.com/apache/impala/blob/master/be/src/common/init.cc#L201]
> Backend tests are flaky: time-based srand
> -
>
> Key: IMPALA-8179
> URL: https://issues.apache.org/jira/browse/IMPALA-8179
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Jim Apple
>Assignee: Joe McDonnell
>Priority: Major
>
> https://gerrit.cloudera.org/#/c/12124/9/be/src/service/unified-betest-main.cc
> added
> {noformat}
> uint32_t seed = time(NULL);
> cout << "seed = " << seed << endl;
> srand(seed);
> {noformat}
> There is no way to force the seed, and backend tests using {{rand()}} are
> then flaky by default. We should make them unflaky by default (using a
> specific random seed) and we should also provide a way to override this by
> setting it by time or to a specific seed for just one run, to help with
> debugging.
> If we want the coverage that comes from running with a different seed each
> time, we could run a test multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8183) TestRPCTimeout.test_reportexecstatus_retry times out
Andrew Sherman created IMPALA-8183:
--
Summary: TestRPCTimeout.test_reportexecstatus_retry times out
Key: IMPALA-8183
URL: https://issues.apache.org/jira/browse/IMPALA-8183
Project: IMPALA
Issue Type: Bug
Reporter: Andrew Sherman
There are 2 forms of failure, where the test itself times out, and where the
whole test run times out, suspiciously just after running
test_reportexecstatus_retry
{quote}
Error Message
Failed: Timeout >7200s
Stacktrace
custom_cluster/test_rpc_timeout.py:143: in test_reportexecstatus_retry
self.execute_query_verify_metrics(self.TEST_QUERY, None, 10)
custom_cluster/test_rpc_timeout.py:45: in execute_query_verify_metrics
self.execute_query(query, query_options)
common/impala_test_suite.py:601: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:632: in execute_query
return self.__execute_query(self.client, query, query_options)
common/impala_test_suite.py:699: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:174: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:183: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:360: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:384: in wait_for_finished
time.sleep(0.05)
E Failed: Timeout >7200s
{quote}
{quote}
Test run timed out. This probably happened due to a hung thread which can be
confirmed by looking at the stacktrace of running impalad processes at
/data/jenkins/workspace/xxx/repos/Impala/logs/timeout_stacktrace
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8182) Extend PlanCtx to provide access to the entire internal plan
Paul Rogers created IMPALA-8182:
---
Summary: Extend PlanCtx to provide access to the entire internal
plan
Key: IMPALA-8182
URL: https://issues.apache.org/jira/browse/IMPALA-8182
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Historically, the Impala planner is a function: AST in one end, Thrift plan out
the other. The planner itself creates a rich intermediate plan representation,
but none of those objects were available for testing, forcing all tests, no
matter how detailed, to run as end-to-end tests: SQL in one end, verify a (very
abbreviated) DESCRIBE output out the other.
A recent change introduced the {{PlanCtx}} that provided access to the
parallelized plan fragments. Doing so allowed adding a variety of cardinality
tests.
It turns out that additional testing requires access to the "single node plan"
in order to verify things like join cardinality.
This ticket asks to modify the {{PlanCtx}} to capture all the internal plan
nodes: both single node and distributed, to enable full unit testing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8181) Show abbreviated row counts in DESCRIBE output
Paul Rogers created IMPALA-8181: --- Summary: Show abbreviated row counts in DESCRIBE output Key: IMPALA-8181 URL: https://issues.apache.org/jira/browse/IMPALA-8181 Project: IMPALA Issue Type: Improvement Components: Frontend Affects Versions: Impala 3.1.0 Reporter: Paul Rogers Assignee: Paul Rogers A recent change added plan node cardinality to the DESCRIBE output using a "metric" format: 123.46M instead of 123456789. This makes the output easier to read. Also, since all numbers are estimate, having seven or eight digits of precision is not very helpful. This ticket requests that the same formatting be used for the other places where the DESCRIBE output shows row counts: table stats, extrapolated row count, partition row counts, and so on. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8180) Change Kudu timestamp writer to round towards minus infinity
Csaba Ringhofer created IMPALA-8180: --- Summary: Change Kudu timestamp writer to round towards minus infinity Key: IMPALA-8180 URL: https://issues.apache.org/jira/browse/IMPALA-8180 Project: IMPALA Issue Type: New Feature Components: Backend Affects Versions: Impala 3.1.0 Reporter: Csaba Ringhofer Fix For: Impala 4.0 Kudu timestamps are microseconds since Unix epoch stored as int64, so Impala has to round its nanosecond timestamps before writing them to Kudu tables. Currently this is done by rounding to the nearest microsecond. Meanwhile Hive uses rounding towards minus infinity when reducing the precision of timestamps, which is a better way in my opinion, because it cannot move a timestamp into a different day, and should be also a bit faster. Changing the rounding method is breaking change, so I would only do this in the next major release. Example: create table tkudu (id int primary key, t timestamp) stored as kudu; insert into tkudu values (1,"1970-01-01 00:00:00.111"), -- all sub-second parts are 7 digit (2,"1970-01-01 23:59:59.999"), (3,"1969-12-31 23:59:59.999"); select * from tkudu; This currently returns: 1,1970-01-01 00:00:00.11000 2,1970-01-02 00:00:00 3,1970-01-01 00:00:00 1 was rounded down to microsec precision, while 2 and 3 were rounded up and also stepped to another day. If the table was written using rounding toward minus infinity, then the query would return this: 1,1970-01-01 00:00:00.11000 2,1970-01-01 23:59:59.99000 3,1969-12-31 23:59:59.99000 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7985) Port RemoteShutdown() to KRPC
[ https://issues.apache.org/jira/browse/IMPALA-7985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman resolved IMPALA-7985. Resolution: Fixed Fix Version/s: Impala 3.2.0 > Port RemoteShutdown() to KRPC > - > > Key: IMPALA-7985 > URL: https://issues.apache.org/jira/browse/IMPALA-7985 > Project: IMPALA > Issue Type: Sub-task > Components: Distributed Exec >Affects Versions: Impala 3.1.0 >Reporter: Michael Ho >Assignee: Andrew Sherman >Priority: Major > Labels: ramp-up > Fix For: Impala 3.2.0 > > > Port RemoteShutdown() to KRPC. It's currently implemented as Thrift RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8179) Backend tests are flaky: time-based srand
[
https://issues.apache.org/jira/browse/IMPALA-8179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765094#comment-16765094
]
Tim Armstrong commented on IMPALA-8179:
---
Providing a mechanism to seed them is a good idea. I disagree with seeding them
by default - I don't think a test is flaky just because its randomised. If
tests are randomised they need to be written carefully so that they're only
testing actual invariants of the code, but we have a bunch of tests that fit
that category. If we have a product bug or a poorly-written test we'd rather
know than masking it by derandomising the test.
> Backend tests are flaky: time-based srand
> -
>
> Key: IMPALA-8179
> URL: https://issues.apache.org/jira/browse/IMPALA-8179
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Jim Apple
>Assignee: Joe McDonnell
>Priority: Major
>
> https://gerrit.cloudera.org/#/c/12124/9/be/src/service/unified-betest-main.cc
> added
> {noformat}
> uint32_t seed = time(NULL);
> cout << "seed = " << seed << endl;
> srand(seed);
> {noformat}
> There is no way to force the seed, and backend tests using {{rand()}} are
> then flaky by default. We should make them unflaky by default (using a
> specific random seed) and we should also provide a way to override this by
> setting it by time or to a specific seed for just one run, to help with
> debugging.
> If we want the coverage that comes from running with a different seed each
> time, we could run a test multiple times.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
