[jira] [Resolved] (IMPALA-8336) TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
[
https://issues.apache.org/jira/browse/IMPALA-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8336.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> TestNestedTypes.test_tpch_mem_limit_single_node failed on centos6
> -
>
> Key: IMPALA-8336
> URL: https://issues.apache.org/jira/browse/IMPALA-8336
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.3.0
>Reporter: Joe McDonnell
>Assignee: Tim Armstrong
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> The test expects the memory limit to be exceeded, but it doesn't happen:
> {noformat}
> query_test.test_nested_types.TestNestedTypes.test_tpch_mem_limit_single_node[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> orc/def/block]
> query_test/test_nested_types.py:123: in test_tpch_mem_limit_single_node
> new_vector, use_db='tpch_nested' + db_suffix)
> common/impala_test_suite.py:487: in run_test_case
> assert False, "Expected exception: %s" % expected_str
> E AssertionError: Expected exception: row_regex: .*Memory limit exceeded:
> Failed to allocate [0-9]+ bytes for collection
> 'tpch_nested_.*.customer.c_orders.item.o_lineitems'.*{noformat}
> Seen once on Centos 6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8428) Add support for caching file handles on s3
Joe McDonnell created IMPALA-8428: - Summary: Add support for caching file handles on s3 Key: IMPALA-8428 URL: https://issues.apache.org/jira/browse/IMPALA-8428 Project: IMPALA Issue Type: Bug Components: Backend Affects Versions: Impala 3.3.0 Reporter: Joe McDonnell The file handle cache is currently disabled for S3, as the S3 connector needed to implement proper unbuffer support. Now that https://issues.apache.org/jira/browse/HADOOP-14747 is fixed, Impala should provide an option to cache S3 file handles. This is particularly important for data caching, as accessing the data cache happens after obtaining a file handle. If getting a file handle is slow, the caching will be less effective. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8415) test_corrupt_stats exhaustive test failed
[
https://issues.apache.org/jira/browse/IMPALA-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-8415.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> test_corrupt_stats exhaustive test failed
> -
>
> Key: IMPALA-8415
> URL: https://issues.apache.org/jira/browse/IMPALA-8415
> Project: IMPALA
> Issue Type: Test
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Attila Jeges
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> IMPALA-6050 broke test_corrupt_stats exhaustive test:
> {code:java}
> metadata/test_compute_stats.py:238: in test_corrupt_stats
> self.run_test_case('QueryTest/corrupt-stats', vector, unique_database)
> common/impala_test_suite.py:517: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:370: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:449: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:239: in verify_query_result_is_subset
> assert expected_literal_strings <= actual_literal_strings
> E assert Items in expected results not found in actual results:
> E ' partitions=1/2 files=1 size=24B'
> E Items in actual results:
> E '02:EXCHANGE [UNPARTITIONED]'
> E 'Max Per-Host Resource Reservation: Memory=8.00KB Threads=3'
> E '| output: count(*)'
> E '| row-size=8B cardinality=0'
> E 'test_corrupt_stats_bb50e9c7.corrupted'
> E 'WARNING: The following tables have potentially corrupt table
> statistics.'
> E 'Per-Host Resource Estimates: Memory=52MB'
> E '03:AGGREGATE [FINALIZE]'
> E ' partition predicates: org = 1'
> E 'PLAN-ROOT SINK'
> E ''
> E '| output: count:merge(*)'
> E ' HDFS partitions=1/2 files=1 size=24B'
> E ' row-size=0B cardinality=0'
> E '|'
> E '00:SCAN HDFS [test_corrupt_stats_bb50e9c7.corrupted]'
> E '01:AGGREGATE'
> E 'Drop and re-compute statistics to resolve this problem.'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8427) Document the behavior change in IMPALA-7800
Michael Ho created IMPALA-8427:
--
Summary: Document the behavior change in IMPALA-7800
Key: IMPALA-8427
URL: https://issues.apache.org/jira/browse/IMPALA-8427
Project: IMPALA
Issue Type: Documentation
Components: Clients
Affects Versions: Impala 3.3.0
Reporter: Michael Ho
Assignee: Alex Rodoni
IMPALA-7800 changes the default behavior of client connection timeout with HS2
and Beeswax Thrift servers. Quote from the commit message:
{noformat}
The current implementation of the FE thrift server waits
indefinitely to open the new session, if the maximum number of
FE service threads specified by --fe_service_threads has been
allocated.
This patch introduces a startup flag to control how the server
should treat new connection requests if we have run out of the
configured number of server threads.
If --accepted_client_cnxn_timeout > 0, new connection requests are
rejected by the server if we can't get a server thread within
the specified timeout.
We set the default timeout to be 5 minutes. The old behavior
can be restored by setting --accepted_client_cnxn_timeout=0,
i.e., no timeout. The timeout applies only to client facing thrift
servers, i.e., HS2 and Beeswax servers.
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8426) Logging error in DROP_TABLE event processing.
Bharathkrishna Guruvayoor Murali created IMPALA-8426:
Summary: Logging error in DROP_TABLE event processing.
Key: IMPALA-8426
URL: https://issues.apache.org/jira/browse/IMPALA-8426
Project: IMPALA
Issue Type: Bug
Reporter: Bharathkrishna Guruvayoor Murali
Assignee: Bharathkrishna Guruvayoor Murali
We have the following logging code in DROP_TABLE process() method, here always
the first condition is triggered and the second will never be matched. So we
should switch the order of the conditions.
{code:java}
if (!tblMatched.getRef()) {
LOG.warn(debugString("Table %s was not removed from "
+ "catalog since the creation time of the table did not match", tblName_));
} else if (!tblWasFound.getRef()) {
debugLog("Table {} was not removed since it did not exist in catalog.",
tblName_);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8425) Evaluate size of Impala docker containers
Tim Armstrong created IMPALA-8425: - Summary: Evaluate size of Impala docker containers Key: IMPALA-8425 URL: https://issues.apache.org/jira/browse/IMPALA-8425 Project: IMPALA Issue Type: Sub-task Components: Infrastructure Reporter: Tim Armstrong Assignee: Tim Armstrong We should take a look at the size of the containers produced by the build and see if it is worth optimising. I think the main contributor to size right now is likely the impala binary, which we could shrink by stripping debug symbols. We could also look at using a different base image from Ubuntu. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8424) Impala Doc: Doc the column level permission on views
Alex Rodoni created IMPALA-8424: --- Summary: Impala Doc: Doc the column level permission on views Key: IMPALA-8424 URL: https://issues.apache.org/jira/browse/IMPALA-8424 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8338) Check CREATION_TIME of databases in event processor to avoid incorrect/redundant invalidates
[ https://issues.apache.org/jira/browse/IMPALA-8338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali resolved IMPALA-8338. -- Resolution: Fixed > Check CREATION_TIME of databases in event processor to avoid > incorrect/redundant invalidates > > > Key: IMPALA-8338 > URL: https://issues.apache.org/jira/browse/IMPALA-8338 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > HIVE-21077 adds CREATION_TIME to Database, and it will be available in the > metastore notifications. Hence, > # For CREATE_DATABASE events, we should compare the creationTime of the > Database in catalog with the Database in the event to make sure we are > ignoring the event only if catalog has the latest Database object. > # For DROP_DATABASE events, we should check that creation time of database > in the notification and the catalog match to make sure that we are removing > the correct database object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8363) Disable column masking and row filtering
[ https://issues.apache.org/jira/browse/IMPALA-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-8363. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Disable column masking and row filtering > > > Key: IMPALA-8363 > URL: https://issues.apache.org/jira/browse/IMPALA-8363 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > Because of policy sharing between Hive and Ranger, Impala needs throw an > authorization exception when column masking and row filtering is enabled in > Hive. This is only temporary until Impala has proper support for column > masking and row filtering. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8421) get problem when start catalogd with the
[ https://issues.apache.org/jira/browse/IMPALA-8421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8421. --- Resolution: Invalid If you have questions about using or developing impala, please send a question to one of the mailing lists: https://impala.apache.org/community.html > get problem when start catalogd with the > > > Key: IMPALA-8421 > URL: https://issues.apache.org/jira/browse/IMPALA-8421 > Project: IMPALA > Issue Type: Bug >Reporter: Lycan >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8423) Add rule to remove useless SELECT node
Quanlong Huang created IMPALA-8423:
--
Summary: Add rule to remove useless SELECT node
Key: IMPALA-8423
URL: https://issues.apache.org/jira/browse/IMPALA-8423
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Reporter: Quanlong Huang
We can add some rules to optimize the plan after we chose a cheapest plan based
on cost. For example, one useful rule can be "removing useless SELECT nodes".
Impala will generated a useless SELECT for the following query:
{code:sql}
SELECT t.id, t.int_col
FROM functional.alltypestiny t
LEFT JOIN
(SELECT id, int_col
FROM functional.alltypestiny) t2
ON (t.id = t2.id)
WHERE t.int_col = t.id
UNION ALL
VALUES (NULL, NULL){code}
Its single node plan is
{code:java}
PLAN-ROOT SINK
|
00:UNION
| constant-operands=1
| row-size=8B cardinality=1
|
04:SELECT
| predicates: t.id = t.int_col
| row-size=12B cardinality=0
|
03:HASH JOIN [RIGHT OUTER JOIN]
| hash predicates: id = t.id
| runtime filters: RF000 <- t.id
| row-size=12B cardinality=1
|
|--01:SCAN HDFS [functional.alltypestiny t]
| HDFS partitions=4/4 files=4 size=460B
| predicates: t.int_col = t.id
| row-size=8B cardinality=1
|
02:SCAN HDFS [functional.alltypestiny]
HDFS partitions=4/4 files=4 size=460B
runtime filters: RF000 -> id
row-size=4B cardinality=8{code}
The SELECT node (id=04) is useless since its only predicate "t.id = t.int_col"
has been enforced in the SCAN node (id=01) which is the right hand side of the
RIGHT OUTER JOIN. The SELECT node won't filter out any more rows.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8422) get problem when start catalogd with the tarballs builded by myself
Lycan created IMPALA-8422: - Summary: get problem when start catalogd with the tarballs builded by myself Key: IMPALA-8422 URL: https://issues.apache.org/jira/browse/IMPALA-8422 Project: IMPALA Issue Type: Bug Environment: centos6.5 Reporter: Lycan I compiled impala under Ubuntu16.04，and run catalogd under centos6.5.Then I get the following problem: E0416 19:17:35.646098 153074 logging.cc:147] stderr will be logged to this file. java.lang.ClassFormatError: org.apache.impala.common.JniUtil (unrecognized class file version) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java.security.SecureClassLoader.defineClass(libgcj.so.10) at java.net.URLClassLoader.findClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) F0416 19:17:36.217576 153074 init.cc:313] Failed to find JniUtil class. . Impalad exiting. *** Check failure stack trace: *** @ 0x7fd12ac07dac @ 0x7fd12ac09651 @ 0x7fd12ac07786 @ 0x7fd12ac0ad4d @ 0x7fd12d8efef3 @ 0x7fd12f634d29 @ 0x697906 @ 0x7fd129c0a92f @ 0x6d6038 loadFileSystems error: ClassFormatError: org.apache.hadoop.fs.FileSystem (unrecognized class file version)java.lang.ClassFormatError: org.apache.hadoop.fs.FileSystem (unrecognized class file ver sion) at java.lang.VMClassLoader.defineClass(libgcj.so.10) at java.lang.ClassLoader.defineClass(libgcj.so.10) at java.security.SecureClassLoader.defineClass(libgcj.so.10) at java.net.URLClassLoader.findClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) at java.lang.ClassLoader.loadClass(libgcj.so.10) this is my java version: [root@impala3.2]# java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8421) get problem when start catalogd with the
Lycan created IMPALA-8421: - Summary: get problem when start catalogd with the Key: IMPALA-8421 URL: https://issues.apache.org/jira/browse/IMPALA-8421 Project: IMPALA Issue Type: Bug Reporter: Lycan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (IMPALA-8404) failed when i build Impala in centos7
[
https://issues.apache.org/jira/browse/IMPALA-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lycan closed IMPALA-8404.
-
Resolution: Fixed
> failed when i build Impala in centos7
> --
>
> Key: IMPALA-8404
> URL: https://issues.apache.org/jira/browse/IMPALA-8404
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.2.0
> Environment: centos7 Impala3.2
>Reporter: Lycan
>Priority: Minor
>
> when I builded impala under centos7 through the
> documents：https://cwiki.apache.org/confluence/display/IMPALA/Building+Impala，I
> got the flowing problem：
> {code:java}
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/row_batch.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_row_batch.proto.dir/all] Error
> 2
> make[2]: *** Waiting for unfinished jobs
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/generated-sources/gen-cpp/common.pb.cc] Error 1
> make[2]: *** [common/protobuf/CMakeFiles/PROTO_common.proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/histogram.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/histogram_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> [ 16%] Built target thrift-deps
> make[3]: *** [be/src/kudu/util/pb_util.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/pb_util_proto.dir/all] Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/maintenance_manager.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/maintenance_manager_proto.dir/all]
> Error 2
> --insertions_out: protoc-gen-insertions: Plugin killed by signal 11.
> make[3]: *** [be/src/kudu/util/version_info.pb.cc] Error 1
> make[2]: *** [be/src/kudu/util/CMakeFiles/version_info_proto.dir/all] Error 2
> make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2
> make: *** [impalad] Error 2
> {code}
> And then I check the CMakeErrorlog, I got the flowwing problem:
> {code:java}
> Run Build Command:"/usr/bin/gmake" "cmTC_2a4fc/fast"
> /usr/bin/gmake -f CMakeFiles/cmTC_2a4fc.dir/build.make
> CMakeFiles/cmTC_2a4fc.dir/build
> gmake[1]: Entering directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> Building C object CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -o
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -c
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c
> Linking C executable cmTC_2a4fc
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/cmake-3.8.2-p1/bin/cmake -E
> cmake_link_script CMakeFiles/cmTC_2a4fc.dir/link.txt --verbose=1
> /root/zqdou/impala3.2-hadoop3.0/impala/toolchain/gcc-4.9.2/bin/gcc -rdynamic
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o -o cmTC_2a4fc
> CMakeFiles/cmTC_2a4fc.dir/CheckSymbolExists.c.o: In function `main':
> CheckSymbolExists.c:(.text+0x16): undefined reference to `pthread_create'
> collect2: error: ld returned 1 exit status
> gmake[1]: *** [cmTC_2a4fc] Error 1
> gmake[1]: Leaving directory
> `/root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp'
> gmake: *** [cmTC_2a4fc/fast] Error 2
> File
> /root/zqdou/impala3.2-hadoop3.0/impala/CMakeFiles/CMakeTmp/CheckSymbolExists.c:
> /* */
> #include
> int main(int argc, char** argv)
> {
> (void)argv;
> #ifndef pthread_create
> return ((int*)(&pthread_create))[argc];
> #else
> (void)argc;
> return 0;
> #endif
> }
> {code}
> It seems the pthread lib is missed because uninstalled or unwirded.But,when I
> check the /usr/lib/, the pthread.so/pthread.a is available.
> {code:java}
> -rwxr-xr-x 1 root root 129956 Nov 20 2015 libpthread-2.17.so
> -rw-r--r-- 1 root root 153162 Apr 10 14:28 libpthread.a
> lrwxrwxrwx 1 root root 24 Apr 10 11:22 libpthread.so ->
> /usr/lib64/libpthread.so
> lrwxrwxrwx 1 root root 18 Apr 23 2018 libpthread.so.0 ->
> libpthread-2.17.so{code}
> This problem is really puzzled.Any sloution is appreciating
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
