[jira] [Commented] (IMPALA-7644) Hide Parquet page index writing with feature flag
[ https://issues.apache.org/jira/browse/IMPALA-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648721#comment-16648721 ] ASF subversion and git services commented on IMPALA-7644: - Commit af76186e013607cb64baf151c039e4f6aaab4350 in impala's branch refs/heads/master from [~joemcdonnell] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=af76186 ] IMPALA-7704: Revert "IMPALA-7644: Hide Parquet page index writing with feature flag" The fix for IMPALA-7644 introduced ASAN issues detailed in IMPALA-7704. Reverting for now. This reverts commit 843683ed6c2ef41c7c25e9fa4af68801dbdd1a78. Change-Id: Icf0a64d6ec747275e3ecd6e801e054f81095591a Reviewed-on: http://gerrit.cloudera.org:8080/11671 Tested-by: Impala Public Jenkins Reviewed-by: Michael Ho > Hide Parquet page index writing with feature flag > - > > Key: IMPALA-7644 > URL: https://issues.apache.org/jira/browse/IMPALA-7644 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Labels: parquet, performance > > Currently there is no released Impala version that can write the Parquet page > index: > [https://github.com/apache/parquet-format/blob/master/PageIndex.md] > However, the current Impala master writes the page index since IMPALA-5842, > but cannot read it. > I think we should hide the write path with a feature flag until Impala is > able to read it back and has better test coverage on it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter
[
https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648720#comment-16648720
]
ASF subversion and git services commented on IMPALA-7704:
-
Commit af76186e013607cb64baf151c039e4f6aaab4350 in impala's branch
refs/heads/master from [~joemcdonnell]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=af76186 ]
IMPALA-7704: Revert "IMPALA-7644: Hide Parquet page index writing with feature
flag"
The fix for IMPALA-7644 introduced ASAN issues detailed in
IMPALA-7704. Reverting for now.
This reverts commit 843683ed6c2ef41c7c25e9fa4af68801dbdd1a78.
Change-Id: Icf0a64d6ec747275e3ecd6e801e054f81095591a
Reviewed-on: http://gerrit.cloudera.org:8080/11671
Tested-by: Impala Public Jenkins
Reviewed-by: Michael Ho
> ASAN tests failing in HdfsParquetTableWriter
>
>
> Key: IMPALA-7704
> URL: https://issues.apache.org/jira/browse/IMPALA-7704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
> at pc 0x016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion
> (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite
> /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in
> impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in
> impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
> impala::OutputPartition*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t std::string const&, boost::function,
[jira] [Commented] (IMPALA-7701) Grant option always shows as NULL in SHOW GRANT ROLE/USER for any HS2 clients
[
https://issues.apache.org/jira/browse/IMPALA-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648718#comment-16648718
]
ASF subversion and git services commented on IMPALA-7701:
-
Commit a80ec4a6d987464352e8f7da89110151569a5a64 in impala's branch
refs/heads/master from [~fredyw]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=a80ec4a ]
IMPALA-7701: grant_option in SHOW GRANT always returns NULL from HS2 clients
Prior to this patch, SHOW GRANT ROLE/USER always showed NULL in
grant_option column because the grant_option column header was set
to use BOOLEAN type but the column value was set to use STRING.
This mismatch causes HS2 clients to interpret the column value as
not set (NULL). The patch fixes the issue by setting the grant_option
column value to use BOOLEAN value. The patch also renames
test_show_grant_user.py to test_show_grant.py for all tests related to
SHOW GRANT statements.
Testing:
- Ran all FE tests
- Added new E2E test running SHOW GRANT statements from HS2 client
- Ran all E2E authorization tests
Change-Id: I1e175544172b63d36dceedc61e1f47e0f910d7cf
Reviewed-on: http://gerrit.cloudera.org:8080/11663
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Grant option always shows as NULL in SHOW GRANT ROLE/USER for any HS2 clients
> -
>
> Key: IMPALA-7701
> URL: https://issues.apache.org/jira/browse/IMPALA-7701
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> In Impala shell:
> {noformat}
> [localhost:21000] default> show grant user test_user;
> Query: show grant user test_user
> +++--+--+---++-+---+--+---+
> | principal_type | principal_name | scope| database | table | column |
> uri | privilege | grant_option | create_time |
> +++--+--+---++-+---+--+---+
> | USER | test_user | database | foo | ||
> | owner | true | Thu, Oct 11 2018 11:48:57.186 |
> | ROLE | admin | server | | ||
> | all | false| Thu, Oct 11 2018 11:48:48.203 |
> +++--+--+---++-+---+--+---+
> Fetched 2 row(s) in 0.13s
> [localhost:21000] default> show grant role admin;
> Query: show grant role admin
> ++--+---++-+---+--+---+
> | scope | database | table | column | uri | privilege | grant_option |
> create_time |
> ++--+---++-+---+--+---+
> | server | | || | all | false| Thu,
> Oct 11 2018 11:48:48.203 |
> ++--+---++-+---+--+---+
> Fetched 1 row(s) in 0.01s
> {noformat}
> Using Impyla (HS2 client):
> {noformat}
> >>> from impala.dbapi import connect
> >>> conn = connect(host='localhost', port=21050)
> >>> cursor = conn.cursor()
> >>> cursor.execute('show grant user test_user')
> >>> print(cursor.fetchall())
> [('USER', 'test_user', 'database', 'foo', '', '', '', 'owner', None, 'Thu,
> Oct 11 2018 11:48:57.186'), ('ROLE', 'admin', 'server', '', '', '', '',
> 'all', None, 'Thu, Oct 11 2018 11:48:48.203')]
> >>> cursor.execute('show grant role admin')
> >>> print(cursor.fetchall())
> [('server', '', '', '', '', 'all', None, 'Thu, Oct 11 2018 11:48:48.203')]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7690) TestAdmissionController.test_pool_config_change_while_queued fails on centos6
[
https://issues.apache.org/jira/browse/IMPALA-7690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648717#comment-16648717
]
ASF subversion and git services commented on IMPALA-7690:
-
Commit e65ac1a4341acfce7c1afc08c0c0566ee0ca50ab in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=e65ac1a ]
IMPALA-7690: Make test_pool_config_change_while_queued compatible with
python 2.6
The ElementTree XML API used in test_pool_config_change_while_queued
used iter() which was added in python 2.7. Switching it to
getiterator() made it compatible with python 2.6.
Change-Id: Id2593609e5be288054d1361f0fe57580e17ea042
Reviewed-on: http://gerrit.cloudera.org:8080/11660
Reviewed-by: Pooja Nilangekar
Reviewed-by: Michael Brown
Tested-by: Impala Public Jenkins
> TestAdmissionController.test_pool_config_change_while_queued fails on centos6
> -
>
> Key: IMPALA-7690
> URL: https://issues.apache.org/jira/browse/IMPALA-7690
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Pooja Nilangekar
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> TestAdmissionController.test_pool_config_change_while_queued fails on Centos6
> because python 2.6 does not support iter() on {{xml.etree.ElementTree. }}
>
> Here are the logs from the test failure:
>
> {code:java}
> custom_cluster/test_admission_controller.py:767: in
> test_pool_config_change_while_queued
> config.set_config_value(pool_name, config_str, 1)
> common/resource_pool_config.py:43: in set_config_value
> node = self.__find_xml_node(self.root, pool_name, config_str)
> common/resource_pool_config.py:86: in __find_xml_node
> for property in xml_root.iter('property'):
> E AttributeError: _ElementInterface instance has no attribute 'iter'
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7644) Hide Parquet page index writing with feature flag
[ https://issues.apache.org/jira/browse/IMPALA-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648722#comment-16648722 ] ASF subversion and git services commented on IMPALA-7644: - Commit af76186e013607cb64baf151c039e4f6aaab4350 in impala's branch refs/heads/master from [~joemcdonnell] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=af76186 ] IMPALA-7704: Revert "IMPALA-7644: Hide Parquet page index writing with feature flag" The fix for IMPALA-7644 introduced ASAN issues detailed in IMPALA-7704. Reverting for now. This reverts commit 843683ed6c2ef41c7c25e9fa4af68801dbdd1a78. Change-Id: Icf0a64d6ec747275e3ecd6e801e054f81095591a Reviewed-on: http://gerrit.cloudera.org:8080/11671 Tested-by: Impala Public Jenkins Reviewed-by: Michael Ho > Hide Parquet page index writing with feature flag > - > > Key: IMPALA-7644 > URL: https://issues.apache.org/jira/browse/IMPALA-7644 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 3.1.0 >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Labels: parquet, performance > > Currently there is no released Impala version that can write the Parquet page > index: > [https://github.com/apache/parquet-format/blob/master/PageIndex.md] > However, the current Impala master writes the page index since IMPALA-5842, > but cannot read it. > I think we should hide the write path with a feature flag until Impala is > able to read it back and has better test coverage on it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter
[
https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648723#comment-16648723
]
ASF subversion and git services commented on IMPALA-7704:
-
Commit af76186e013607cb64baf151c039e4f6aaab4350 in impala's branch
refs/heads/master from [~joemcdonnell]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=af76186 ]
IMPALA-7704: Revert "IMPALA-7644: Hide Parquet page index writing with feature
flag"
The fix for IMPALA-7644 introduced ASAN issues detailed in
IMPALA-7704. Reverting for now.
This reverts commit 843683ed6c2ef41c7c25e9fa4af68801dbdd1a78.
Change-Id: Icf0a64d6ec747275e3ecd6e801e054f81095591a
Reviewed-on: http://gerrit.cloudera.org:8080/11671
Tested-by: Impala Public Jenkins
Reviewed-by: Michael Ho
> ASAN tests failing in HdfsParquetTableWriter
>
>
> Key: IMPALA-7704
> URL: https://issues.apache.org/jira/browse/IMPALA-7704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
> at pc 0x016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion
> (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite
> /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in
> impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in
> impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
> impala::OutputPartition*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t std::string const&, boost::function,
[jira] [Commented] (IMPALA-7554) Update custom cluster tests to have sentry create new log on each start
[ https://issues.apache.org/jira/browse/IMPALA-7554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648716#comment-16648716 ] ASF subversion and git services commented on IMPALA-7554: - Commit 21f521a7c280031e33cde7c61a979683c5abed00 in impala's branch refs/heads/master from [~aholley] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=21f521a ] IMPALA-7554: Update custom cluster tests to have new logs for sentry This patch adds the ability to create a new log for each spawn of the sentry service. This will enable better trouble shooting for the custom cluster tests that restart the sentry service. Testing: - Ran all custom cluster tests. Change-Id: I6e538af7fd6e6ea21dc3f4442bdebf3b31558516 Reviewed-on: http://gerrit.cloudera.org:8080/11624 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins > Update custom cluster tests to have sentry create new log on each start > --- > > Key: IMPALA-7554 > URL: https://issues.apache.org/jira/browse/IMPALA-7554 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Adam Holley >Assignee: Adam Holley >Priority: Trivial > Fix For: Impala 3.1.0 > > > Currently when the sentry service is restarted in various custom cluster > tests, the previous log file is overwritten. It should use a unique log file > name to preserve any errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7622) Add query profile metrics for RPC's used when pulling incremental stats.
[ https://issues.apache.org/jira/browse/IMPALA-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648719#comment-16648719 ] ASF subversion and git services commented on IMPALA-7622: - Commit 97f028299c9d9d7493bdbeaacbf0a288678f9371 in impala's branch refs/heads/master from [~vercego] [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=97f0282 ] IMPALA-7622: adds profile metrics for incremental stats Reapplies change after fixing where frontend profile is placed in runtime profile. When computing incremental statistics by fetching the stats directly from catalogd, a potentially expensive RPC is made from the impalad coordinator to catalogd. This change adds metrics to the frontend section of the profile to track how long the request takes, the size of the compressed bytes received, and the number of partitions received. The profile for a 'compute incremental ...' command on a table with no statistics looks like this: Frontend: - StatsFetch.CompressedBytes: 0 - StatsFetch.TotalPartitions: 24 - StatsFetch.NumPartitionsWithStats: 0 - StatsFetch.Time: 26ms And the profile looks as follows when the table has stats, so the stats are fetched: Frontend: - StatsFetch.CompressedBytes: 24622 - StatsFetch.TotalPartitions: 23 - StatsFetch.NumPartitionsWithStats: 23 - StatsFetch.Time: 14ms Testing: - manual inspection - e2e test to check the profile Change-Id: I94559a749500d44aa6aad564134d55c39e1d5273 Reviewed-on: http://gerrit.cloudera.org:8080/11670 Reviewed-by: Tianyi Wang Tested-by: Impala Public Jenkins > Add query profile metrics for RPC's used when pulling incremental stats. > > > Key: IMPALA-7622 > URL: https://issues.apache.org/jira/browse/IMPALA-7622 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Vuk Ercegovac >Assignee: Vuk Ercegovac >Priority: Major > Fix For: Impala 3.1.0 > > > When --pull_incremental_statistics is enabled, the frontend will fetch these > stats from catalogd. We should record metrics for this, such as number > partitions fetched, size of received bytes, and elapsed time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7633) count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant
[ https://issues.apache.org/jira/browse/IMPALA-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648704#comment-16648704 ] Adam Holley commented on IMPALA-7633: - Reviewed-on: http://gerrit.cloudera.org:8080/11595 Reviewed-by: Fredy Wijaya Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins > count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant > -- > > Key: IMPALA-7633 > URL: https://issues.apache.org/jira/browse/IMPALA-7633 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Alex Rodoni >Assignee: Adam Holley >Priority: Critical > Fix For: Impala 3.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7501) Slim down metastore Partition objects in LocalCatalog cache
[
https://issues.apache.org/jira/browse/IMPALA-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629621#comment-16629621
]
Paul Rogers edited comment on IMPALA-7501 at 10/13/18 1:37 AM:
---
The local cache holds onto HMS {{Partition}} objects via the
{{PartitionMetadataImpl}} class within the {{CatalogdMetaProvider}} class.
Adding a single line to the constructor of that class should remove the
unwanted column schemas:
{noformat}
msPartition_.getSd().unsetCols();
{noformat}
Rerunning the {{LocalCatalogTest}} cases showed no issues.
was (Author: paul.rogers):
The path to the HMS {{Partition}} objects appears to be:
* {{HdfsTable}} holds onto a set of {{FeFsPartition}} objects.
* In local catalog mode, the {{FeFsParition}} is an instance of
{{LocalFsPartition}}.
* {{LocalFsPartition}} holds onto the HMS {{Partition}} objects.
* {{Partition}} holds onto a {{StorageDescriptor}} which holds onto a list of
the {{FieldSchema}} objects that Todd noted.
However, there is no obvious path that causes code to hold onto the
{{LocalFsParition}} objects; in the local catalog implementation, they are
converted to Thrift format, then discarded. It is not clear how the
{{FeFsPartition}} objects are recreated for a query. The available tests don’t
exercise this path.
Perhaps code changed since this issues was reported?
No code in {{LocalFsParition}} accesses the columns stored in the
{{StorageDescriptor}} associated with the {{Partition}}, so it is probably safe
to nuke them. Added the following to the {{LocalFsPartition}} constructor:
{noformat}
msPartition_.getSd().unsetCols();
{noformat}
Rerunning the {{LocalCatalogTest}} cases showed no issues.
Need to talk to Todd to better understand the design, and to determine how to
run a test case similar to that in the ticket description.
> Slim down metastore Partition objects in LocalCatalog cache
> ---
>
> Key: IMPALA-7501
> URL: https://issues.apache.org/jira/browse/IMPALA-7501
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Todd Lipcon
>Priority: Minor
>
> I took a heap dump of an impalad running in LocalCatalog mode with a 2G limit
> after running a production workload simulation for a couple hours. It had
> 38.5M objects and 2.02GB heap (the vast majority of the heap is, as expected,
> in the LocalCatalog cache). Of this total footprint, 1.78GB and 34.6M objects
> are retained by 'Partition' objects. Drilling into those, 1.29GB and 33.6M
> objects are retained by FieldSchema, which, as far as I remember, are ignored
> on the partition level by the Impala planner. So, with a bit of slimming down
> of these objects, we could make a huge dent in effective cache capacity given
> a fixed budget. Reducing object count should also have the effect of improved
> GC performance (old gen GC is more closely tied to object count than size)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6955) Timeout when starting test_query_expiration custom cluster
[
https://issues.apache.org/jira/browse/IMPALA-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-6955.
Resolution: Cannot Reproduce
Fix Version/s: Not Applicable
Hasn't happened for a while.
> Timeout when starting test_query_expiration custom cluster
> --
>
> Key: IMPALA-6955
> URL: https://issues.apache.org/jira/browse/IMPALA-6955
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Vuk Ercegovac
>Priority: Blocker
> Fix For: Not Applicable
>
>
> Ran into the following crash on a rhel test recently:
> {noformat}
> Error starting cluster: num_known_live_backends did not reach expected value
> in time{noformat}
> Backtrace:
> {noformat}
> #0 0x7f92365185c9 in raise () from /lib64/libc.so.6
> #1 0x7f9236519cd8 in abort () from /lib64/libc.so.6
> #2 0x7f92393841a5 in os::abort(bool) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #3 0x7f9239514843 in VMError::report_and_die() () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #4 0x7f9239389562 in JVM_handle_linux_signal () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #5 0x7f92393804f3 in signalHandler(int, siginfo*, void*) () from
> /opt/toolchain/sun-jdk-64bit-1.8.0.05/jre/lib/amd64/server/libjvm.so
> #6
> #7 0x016fded0 in base::subtle::NoBarrier_CompareAndSwap (ptr=0x238,
> old_value=0, new_value=1) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/atomicops-internals-x86.h:85
> #8 0x016fdf50 in base::subtle::Acquire_CompareAndSwap (ptr=0x238,
> old_value=0, new_value=1) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/atomicops-internals-x86.h:138
> #9 0x016fe26c in base::SpinLock::Lock (this=0x238) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/gutil/spinlock.h:74
> #10 0x016fe2f6 in impala::SpinLock::lock (this=0x238) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/spinlock.h:34
> #11 0x01aa8c96 in
> impala::ScopedShardedMapRef
> >::ScopedShardedMapRef (this=0x7f91aa81eb90, query_id=..., sharded_map=0x1c0)
> at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/util/sharded-query-map-util.h:99
> #12 0x01a999e2 in impala::ImpalaServer::GetClientRequestState
> (this=0xa569000, query_id=...) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-server.cc:2123
> #13 0x01b3ace6 in impala::ImpalaHttpHandler::QuerySummaryHandler
> (this=0x6f057a0, include_json_plan=true, include_summary=true, args=...,
> document=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-http-handler.cc:755
> #14 0x01b3cc11 in impala::ImpalaHttpHandler:: auto:6*)>::operator(),
> std::basic_string >, rapidjson::GenericDocument >
> >(const std::map,
> std::allocator >, std::basic_string,
> std::allocator >, std::less std::char_traits, std::allocator > >,
> std::allocator,
> std::allocator > const, std::basic_string,
> std::allocator > > > > &,
> rapidjson::GenericDocument,
> rapidjson::MemoryPoolAllocator > *) const
> (__closure=0xd9884b8, args=..., doc=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/repos/Impala/be/src/service/impala-http-handler.cc:132
> #15 0x01b3cc46 in
> boost::detail::function::void_function_obj_invoker2 auto:5&, auto:6*)>, void, const std::map std::char_traits, std::allocator >, std::basic_string std::char_traits, std::allocator >,
> std::less,
> std::allocator > >, std::allocator std::basic_string, std::allocator >,
> std::basic_string, std::allocator > > >
> >&, rapidjson::GenericDocument,
> rapidjson::MemoryPoolAllocator
> >*>::invoke(boost::detail::function::function_buffer &, const
> std::map, std::allocator
> >, std::basic_string, std::allocator >,
> std::le\
> ss, std::allocator > >,
> std::allocator,
> std::allocator > const, std::basic_string,
> std::allocator > > > > &,
> rapidjson::GenericDocument,
> rapidjson::MemoryPoolAllocator > *)
> (function_obj_ptr=..., a0=..., a1=0x7f91aa81f230) at
> /data/jenkins/workspace/impala-cdh6.x-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
> #16 0x01c4f528 in boost::function2 std::string, std::less, std::allocator const, std::string> > > const&,
> rapidjson::GenericDocument,
> rapidjson::MemoryPoolAllocator >*>::operator()
> (this=0xd9884b0, a0=..., a1=0x7f91aa81f230) at
>
[jira] [Commented] (IMPALA-6890) split-hbase.sh: Can't get master address from ZooKeeper; znode data == null
[
https://issues.apache.org/jira/browse/IMPALA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648635#comment-16648635
]
Michael Ho commented on IMPALA-6890:
[~joemcdonnell], can you please help with this one ? Thanks !
> split-hbase.sh: Can't get master address from ZooKeeper; znode data == null
> ---
>
> Key: IMPALA-6890
> URL: https://issues.apache.org/jira/browse/IMPALA-6890
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.12.0
>Reporter: Vuk Ercegovac
>Assignee: Joe McDonnell
>Priority: Blocker
>
> {noformat}
> 20:57:13 FAILED (Took: 7 min 58 sec)
> 20:57:13
> '/data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/testdata/bin/split-hbase.sh'
> failed. Tail of log:
> 20:57:13 Wed Apr 18 20:49:43 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13 Wed Apr 18 20:49:43 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13 Wed Apr 18 20:49:44 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> ...
> 20:57:13 Wed Apr 18 20:57:13 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13
> 20:57:13 at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4329)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4321)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2952)
> 20:57:13 at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.(HBaseTestDataRegionAssigment.java:74)
> 20:57:13 at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.main(HBaseTestDataRegionAssigment.java:310)
> 20:57:13 Caused by: org.apache.hadoop.hbase.MasterNotRunningException:
> java.io.IOException: Can't get master address from ZooKeeper; znode data ==
> null
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1698)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1718)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1875)
> 20:57:13 at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> 20:57:13 at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> 20:57:13 ... 5 more
> 20:57:13 Caused by: java.io.IOException: Can't get master address from
> ZooKeeper; znode data == null
> 20:57:13 at
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:154)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1648)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1689)
> 20:57:13 ... 9 more
> 20:57:13 Error in
> /data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/testdata/bin/split-hbase.sh
> at line 41: "$JAVA" ${JAVA_KERBEROS_MAGIC} \
> 20:57:13 Error in
> /data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/bin/run-all-tests.sh
> at line 48: # Run End-to-end Tests{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-6890) split-hbase.sh: Can't get master address from ZooKeeper; znode data == null
[
https://issues.apache.org/jira/browse/IMPALA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-6890:
--
Assignee: Joe McDonnell
> split-hbase.sh: Can't get master address from ZooKeeper; znode data == null
> ---
>
> Key: IMPALA-6890
> URL: https://issues.apache.org/jira/browse/IMPALA-6890
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.12.0
>Reporter: Vuk Ercegovac
>Assignee: Joe McDonnell
>Priority: Blocker
>
> {noformat}
> 20:57:13 FAILED (Took: 7 min 58 sec)
> 20:57:13
> '/data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/testdata/bin/split-hbase.sh'
> failed. Tail of log:
> 20:57:13 Wed Apr 18 20:49:43 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13 Wed Apr 18 20:49:43 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13 Wed Apr 18 20:49:44 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> ...
> 20:57:13 Wed Apr 18 20:57:13 PDT 2018,
> RpcRetryingCaller{globalStartTime=1524109783051, pause=100, retries=31},
> org.apache.hadoop.hbase.MasterNotRunningException: java.io.IOException: Can't
> get master address from ZooKeeper; znode data == null
> 20:57:13
> 20:57:13 at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4329)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4321)
> 20:57:13 at
> org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2952)
> 20:57:13 at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.(HBaseTestDataRegionAssigment.java:74)
> 20:57:13 at
> org.apache.impala.datagenerator.HBaseTestDataRegionAssigment.main(HBaseTestDataRegionAssigment.java:310)
> 20:57:13 Caused by: org.apache.hadoop.hbase.MasterNotRunningException:
> java.io.IOException: Can't get master address from ZooKeeper; znode data ==
> null
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1698)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1718)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1875)
> 20:57:13 at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> 20:57:13 at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> 20:57:13 ... 5 more
> 20:57:13 Caused by: java.io.IOException: Can't get master address from
> ZooKeeper; znode data == null
> 20:57:13 at
> org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:154)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1648)
> 20:57:13 at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1689)
> 20:57:13 ... 9 more
> 20:57:13 Error in
> /data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/testdata/bin/split-hbase.sh
> at line 41: "$JAVA" ${JAVA_KERBEROS_MAGIC} \
> 20:57:13 Error in
> /data/jenkins/workspace/impala-cdh5-2.12.0_5.15.0-exhaustive-thrift/repos/Impala/bin/run-all-tests.sh
> at line 48: # Run End-to-end Tests{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7117) Lower debug level for HDFS S3 connector back to INFO
[ https://issues.apache.org/jira/browse/IMPALA-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Ho reassigned IMPALA-7117: -- Assignee: Lars Volker > Lower debug level for HDFS S3 connector back to INFO > > > Key: IMPALA-7117 > URL: https://issues.apache.org/jira/browse/IMPALA-7117 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Blocker > Labels: s3 > > This change will increase the log level for the HDFS S3 connector to DEBUG to > help with IMPALA-6910 and IMPALA-7070. Before the next release we need to > lower it again. > https://gerrit.cloudera.org/#/c/10596/ > I'm making this a P1 to remind us that we must do this before cutting a > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7117) Lower debug level for HDFS S3 connector back to INFO
[ https://issues.apache.org/jira/browse/IMPALA-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648626#comment-16648626 ] Michael Ho commented on IMPALA-7117: [~lv], can you please hold on to this for now ? > Lower debug level for HDFS S3 connector back to INFO > > > Key: IMPALA-7117 > URL: https://issues.apache.org/jira/browse/IMPALA-7117 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 2.13.0, Impala 3.1.0 >Reporter: Lars Volker >Assignee: Lars Volker >Priority: Blocker > Labels: s3 > > This change will increase the log level for the HDFS S3 connector to DEBUG to > help with IMPALA-6910 and IMPALA-7070. Before the next release we need to > lower it again. > https://gerrit.cloudera.org/#/c/10596/ > I'm making this a P1 to remind us that we must do this before cutting a > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7194) Impala crashed at FragmentInstanceState::Open()
[
https://issues.apache.org/jira/browse/IMPALA-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648623#comment-16648623
]
Michael Ho commented on IMPALA-7194:
[~boroknagyz], can you please drive this to completion ? Thanks.
> Impala crashed at FragmentInstanceState::Open()
> ---
>
> Key: IMPALA-7194
> URL: https://issues.apache.org/jira/browse/IMPALA-7194
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Xiaomin Zhang
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: crash
>
> Below is the crash stack resolved from minidump
> {code}
> Thread 3473 (crashed)
> 0 0x7fb0222e9000
> 1 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
> 255 + 0x11]
> 2 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
> 80 + 0xb]
> 3 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 382 + 0x10]
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::Promise*)
> [function_template.hpp : 767 + 0x7]
> 5 impalad!boost::detail::thread_data (std::string const&, std::string const&, boost::function,
> impala::Promise), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value> > > >::run() [bind.hpp : 457 + 0x6]
> 6 impalad!thread_proxy + 0xda
> {code}
> We do not have further details about this crash, and we only saw it happened
> once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7194) Impala crashed at FragmentInstanceState::Open()
[
https://issues.apache.org/jira/browse/IMPALA-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-7194:
--
Assignee: Zoltán Borók-Nagy
> Impala crashed at FragmentInstanceState::Open()
> ---
>
> Key: IMPALA-7194
> URL: https://issues.apache.org/jira/browse/IMPALA-7194
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 2.11.0
>Reporter: Xiaomin Zhang
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: crash
>
> Below is the crash stack resolved from minidump
> {code}
> Thread 3473 (crashed)
> 0 0x7fb0222e9000
> 1 impalad!impala::FragmentInstanceState::Open() [fragment-instance-state.cc :
> 255 + 0x11]
> 2 impalad!impala::FragmentInstanceState::Exec() [fragment-instance-state.cc :
> 80 + 0xb]
> 3 impalad!impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> [query-state.cc : 382 + 0x10]
> 4 impalad!impala::Thread::SuperviseThread(std::string const&, std::string
> const&, boost::function, impala::Promise*)
> [function_template.hpp : 767 + 0x7]
> 5 impalad!boost::detail::thread_data (std::string const&, std::string const&, boost::function,
> impala::Promise), boost::_bi::list4,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value> > > >::run() [bind.hpp : 457 + 0x6]
> 6 impalad!thread_proxy + 0xda
> {code}
> We do not have further details about this crash, and we only saw it happened
> once.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7471) Impala crashes or returns incorrect results when querying parquet nested types
[
https://issues.apache.org/jira/browse/IMPALA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho reassigned IMPALA-7471:
--
Assignee: Attila Jeges
> Impala crashes or returns incorrect results when querying parquet nested types
> --
>
> Key: IMPALA-7471
> URL: https://issues.apache.org/jira/browse/IMPALA-7471
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Attila Jeges
>Priority: Blocker
> Labels: correctness, crash, parquet
> Attachments: test_users_131786401297925138_0.parquet
>
>
> From
> http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-bug-with-nested-arrays-of-structures-where-some-of/m-p/78507/highlight/false#M4779
> {quote}We found a case where Impala returns incorrect values from simple
> query. Our data contains nested array of structures and structures contains
> other structures.
> We generated minimal sample data allowing to reproduce the issue.
>
> SQL to create a table:
> {quote}
> {code}
> CREATE TABLE plat_test.test_users (
> id INT,
> name STRING,
> devices ARRAY<
> STRUCT<
> id:STRING,
> device_info:STRUCT<
> model:STRING
> >
> >
> >
> )
> STORED AS PARQUET
> {code}
> {quote}
> Please put attached parquet file to the location of the table and refresh the
> table.
> In sample data we have 2 users, one with 2 devices, second one with 3. Some
> of the devices.device_info.model fields are NULL.
>
> When I issue a query:
> {quote}
> {code}
> SELECT u.name, d.device_info.model as model
> FROM test_users u,
> u.devices d;
> {code}
> {quote}
> I'm expecting to get 5 records in results, but getting only one1.png
> If I change query to:
> {quote}
> {code}
> SELECT u.name, d.device_info.model as model
> FROM test_users u
> LEFT OUTER JOIN u.devices d;
> {code}
> {quote}
> I'm getting two records in the results, but still not as it should be.
> We found some workaround to this problem. If we add to the result columns
> device.id we will get all records from parquet file:
> {quote}
> {code}
> SELECT u.name, d.id, d.device_info.model as model
> FROM test_users u
> , u.devices d
> {code}
> {quote}
> And result is 3.png
>
> But we can't rely on this workaround, because we don't need device.id in all
> queries and Impala optimizes it, and as a result we are getting unpredicted
> results.
>
> I tested Hive query on this table and it returns expected results:
> {quote}
> {code}
> SELECT u.name, d.device_info.model
> FROM test_users u
> lateral view outer inline (u.devices) d;
> {code}
> {quote}
> results:
> 4.png
> Please advice if it's a problem in Impala engine or we did some mistake in
> our query.
>
> Best regards,
> Come2Play team.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7471) Impala crashes or returns incorrect results when querying parquet nested types
[
https://issues.apache.org/jira/browse/IMPALA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648610#comment-16648610
]
Michael Ho commented on IMPALA-7471:
Hi [~attilaj], can you please look into this ?
> Impala crashes or returns incorrect results when querying parquet nested types
> --
>
> Key: IMPALA-7471
> URL: https://issues.apache.org/jira/browse/IMPALA-7471
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Tim Armstrong
>Assignee: Attila Jeges
>Priority: Blocker
> Labels: correctness, crash, parquet
> Attachments: test_users_131786401297925138_0.parquet
>
>
> From
> http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-bug-with-nested-arrays-of-structures-where-some-of/m-p/78507/highlight/false#M4779
> {quote}We found a case where Impala returns incorrect values from simple
> query. Our data contains nested array of structures and structures contains
> other structures.
> We generated minimal sample data allowing to reproduce the issue.
>
> SQL to create a table:
> {quote}
> {code}
> CREATE TABLE plat_test.test_users (
> id INT,
> name STRING,
> devices ARRAY<
> STRUCT<
> id:STRING,
> device_info:STRUCT<
> model:STRING
> >
> >
> >
> )
> STORED AS PARQUET
> {code}
> {quote}
> Please put attached parquet file to the location of the table and refresh the
> table.
> In sample data we have 2 users, one with 2 devices, second one with 3. Some
> of the devices.device_info.model fields are NULL.
>
> When I issue a query:
> {quote}
> {code}
> SELECT u.name, d.device_info.model as model
> FROM test_users u,
> u.devices d;
> {code}
> {quote}
> I'm expecting to get 5 records in results, but getting only one1.png
> If I change query to:
> {quote}
> {code}
> SELECT u.name, d.device_info.model as model
> FROM test_users u
> LEFT OUTER JOIN u.devices d;
> {code}
> {quote}
> I'm getting two records in the results, but still not as it should be.
> We found some workaround to this problem. If we add to the result columns
> device.id we will get all records from parquet file:
> {quote}
> {code}
> SELECT u.name, d.id, d.device_info.model as model
> FROM test_users u
> , u.devices d
> {code}
> {quote}
> And result is 3.png
>
> But we can't rely on this workaround, because we don't need device.id in all
> queries and Impala optimizes it, and as a result we are getting unpredicted
> results.
>
> I tested Hive query on this table and it returns expected results:
> {quote}
> {code}
> SELECT u.name, d.device_info.model
> FROM test_users u
> lateral view outer inline (u.devices) d;
> {code}
> {quote}
> results:
> 4.png
> Please advice if it's a problem in Impala engine or we did some mistake in
> our query.
>
> Best regards,
> Come2Play team.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7622) Add query profile metrics for RPC's used when pulling incremental stats.
[ https://issues.apache.org/jira/browse/IMPALA-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vuk Ercegovac resolved IMPALA-7622. --- Resolution: Fixed > Add query profile metrics for RPC's used when pulling incremental stats. > > > Key: IMPALA-7622 > URL: https://issues.apache.org/jira/browse/IMPALA-7622 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Vuk Ercegovac >Assignee: Vuk Ercegovac >Priority: Major > Fix For: Impala 3.1.0 > > > When --pull_incremental_statistics is enabled, the frontend will fetch these > stats from catalogd. We should record metrics for this, such as number > partitions fetched, size of received bytes, and elapsed time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7699) TestSpillingNoDebugActionDimensions fails earlier than expected
[
https://issues.apache.org/jira/browse/IMPALA-7699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pooja Nilangekar reassigned IMPALA-7699:
Assignee: Bikramjeet Vig (was: Tim Armstrong)
> TestSpillingNoDebugActionDimensions fails earlier than expected
>
>
> Key: IMPALA-7699
> URL: https://issues.apache.org/jira/browse/IMPALA-7699
> Project: IMPALA
> Issue Type: Bug
>Reporter: Pooja Nilangekar
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: broken-build
>
> In some of the recent runs, the query fails without sufficient memory,
> however it is in the HDFS scan node rather than the hash join node. Here are
> the corresponding logs:
> Stacktrace:
> {code:java}
> query_test/test_spilling.py:113: in test_spilling_no_debug_action
> self.run_test_case('QueryTest/spilling-no-debug-action', vector)
> common/impala_test_suite.py:466: in run_test_case
> self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> common/impala_test_suite.py:319: in __verify_exceptions (expected_str,
> actual_str) E AssertionError: Unexpected exception string. Expected:
> row_regex:.*Cannot perform hash join at node with id .*. Repartitioning did
> not reduce the size of a spilled partition.* E Not found in actual:
> ImpalaBeeswaxException: Query aborted:Memory limit exceeded: Failed to
> allocate tuple bufferHDFS_SCAN_NODE (id=1) could not allocate 190.00 KB
> without exceeding limit.Error occurred on backend localhost:22001 by fragment
> 2e4f0f944d373848:9ae1d7e20002
> {code}
>
> Here are the impalad logs:
> {code:java}
> I1010 18:31:30.721693 7270 coordinator.cc:498] ExecState: query
> id=2e4f0f944d373848:9ae1d7e2
> finstance=2e4f0f944d373848:9ae1d7e20002 on host=localhost:22001
> (EXECUTING -> ERROR) status=Memory limit exceeded: Failed to allocate tuple
> buffer
> HDFS_SCAN_NODE (id=1) could not allocate 190.00 KB without exceeding limit.
> Error occurred on backend localhost:22001 by fragment
> 2e4f0f944d373848:9ae1d7e20002
> Memory left in process limit: 9.19 GB
> Memory left in query limit: 157.62 KB
> Query(2e4f0f944d373848:9ae1d7e2): Limit=150.00 MB Reservation=117.25
> MB ReservationLimit=118.00 MB OtherMemory=32.60 MB Total=149.85 MB
> Peak=149.85 MB
> Unclaimed reservations: Reservation=5.75 MB OtherMemory=0 Total=5.75 MB
> Peak=55.75 MB
> Fragment 2e4f0f944d373848:9ae1d7e20003: Reservation=2.00 MB
> OtherMemory=22.20 MB Total=24.20 MB Peak=24.20 MB
> Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB
> OtherMemory=0 Total=2.00 MB Peak=2.00 MB
> SORT_NODE (id=3): Total=0 Peak=0
> HASH_JOIN_NODE (id=2): Total=42.25 KB Peak=42.25 KB
> Exprs: Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=2): Total=13.12 KB Peak=13.12 KB
> Hash Join Builder (join_node_id=2) Exprs: Total=13.12 KB Peak=13.12 KB
> HDFS_SCAN_NODE (id=0): Total=0 Peak=0
> EXCHANGE_NODE (id=4): Reservation=18.79 MB OtherMemory=235.89 KB
> Total=19.02 MB Peak=19.02 MB
> KrpcDeferredRpcs: Total=235.89 KB Peak=235.89 KB
> KrpcDataStreamSender (dst_id=5): Total=480.00 B Peak=480.00 B
> CodeGen: Total=3.13 MB Peak=3.13 MB
> Fragment 2e4f0f944d373848:9ae1d7e20002: Reservation=109.50 MB
> OtherMemory=10.39 MB Total=119.89 MB Peak=119.89 MB
> HDFS_SCAN_NODE (id=1): Reservation=109.50 MB OtherMemory=10.20 MB
> Total=119.70 MB Peak=119.70 MB
> Queued Batches: Total=6.12 MB Peak=6.12 MB
> KrpcDataStreamSender (dst_id=4): Total=688.00 B Peak=688.00 B
> CodeGen: Total=488.00 B Peak=51.00 KB
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7706) Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7706 started by Alex Rodoni. --- > Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala > --- > > Key: IMPALA-7706 > URL: https://issues.apache.org/jira/browse/IMPALA-7706 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > ALTER TABLE SET OWNER, which requires either ALL WITH GRANT, or OWNER WITH > GRANT for the TABLE, is not mentioned on the page. It's a special case so > may not fit in the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7706) Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648599#comment-16648599 ] Alex Rodoni commented on IMPALA-7706: - https://gerrit.cloudera.org/#/c/11675/ > Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala > --- > > Key: IMPALA-7706 > URL: https://issues.apache.org/jira/browse/IMPALA-7706 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > ALTER TABLE SET OWNER, which requires either ALL WITH GRANT, or OWNER WITH > GRANT for the TABLE, is not mentioned on the page. It's a special case so > may not fit in the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7706) Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7706: Description: ALTER TABLE SET OWNER, which requires either ALL WITH GRANT, or OWNER WITH GRANT for the TABLE, is not mentioned on the page. It's a special case so may not fit in the table. > Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala > --- > > Key: IMPALA-7706 > URL: https://issues.apache.org/jira/browse/IMPALA-7706 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > ALTER TABLE SET OWNER, which requires either ALL WITH GRANT, or OWNER WITH > GRANT for the TABLE, is not mentioned on the page. It's a special case so > may not fit in the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7706) Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala
Alex Rodoni created IMPALA-7706: --- Summary: Impala Doc: ALTER TABLE SET OWNER not on Sentry page for Impala Key: IMPALA-7706 URL: https://issues.apache.org/jira/browse/IMPALA-7706 Project: IMPALA Issue Type: Task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7705) Impala 2.13 & 3.1 Docs: ALTER DATABASE SET OWNER
[ https://issues.apache.org/jira/browse/IMPALA-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648564#comment-16648564 ] Alex Rodoni commented on IMPALA-7705: - https://gerrit.cloudera.org/#/c/11674/ > Impala 2.13 & 3.1 Docs: ALTER DATABASE SET OWNER > > > Key: IMPALA-7705 > URL: https://issues.apache.org/jira/browse/IMPALA-7705 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > > ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7076) Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648549#comment-16648549 ] Alex Rodoni commented on IMPALA-7076: - https://gerrit.cloudera.org/#/c/11673/ > Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER > > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: (was: ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 ) > Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER > > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 (was: The new statements are: ALTER DB/TABLE/VIEW SET OWNER SHOW GRANT USER SHOW GRANT USER ON Everything else is defects or implementation details. ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 -Fixed in IMPALA-7525 SHOW GRANT USER https://issues.apache.org/jira/browse/IMPALA-6989- -Fixed in IMPALA-7525 SHOW GRANT ROLE/USER ON https://issues.apache.org/jira/browse/IMPALA-6991- ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 ) > Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER > > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7705) CLONE - Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER
Alex Rodoni created IMPALA-7705: --- Summary: CLONE - Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER Key: IMPALA-7705 URL: https://issues.apache.org/jira/browse/IMPALA-7705 Project: IMPALA Issue Type: Sub-task Components: Docs Reporter: Alex Rodoni Assignee: Alex Rodoni The new statements are: ALTER DB/TABLE/VIEW SET OWNER SHOW GRANT USER SHOW GRANT USER ON Everything else is defects or implementation details. ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 -Fixed in IMPALA-7525 SHOW GRANT USER https://issues.apache.org/jira/browse/IMPALA-6989- -Fixed in IMPALA-7525 SHOW GRANT ROLE/USER ON https://issues.apache.org/jira/browse/IMPALA-6991- ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Summary: Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER (was: Impala 2.13 & 3.1 Docs: Object ownership for Impala) > Impala 2.13 & 3.1 Docs: ALTER TABLE / VIEW SET OWNER > > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > The new statements are: > ALTER DB/TABLE/VIEW SET OWNER > SHOW GRANT USER > SHOW GRANT USER ON > > Everything else is defects or implementation details. > ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 > -Fixed in IMPALA-7525 SHOW GRANT USER > https://issues.apache.org/jira/browse/IMPALA-6989- > -Fixed in IMPALA-7525 SHOW GRANT ROLE/USER ON > https://issues.apache.org/jira/browse/IMPALA-6991- > ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5605) document how to increase thread resource limits
[
https://issues.apache.org/jira/browse/IMPALA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648361#comment-16648361
]
Alex Rodoni commented on IMPALA-5605:
-
[~tarmstrong] I think we can add the info here:
https://impala.apache.org/docs/build3x/html/topics/impala_prereqs.html#prereqs
> document how to increase thread resource limits
> ---
>
> Key: IMPALA-5605
> URL: https://issues.apache.org/jira/browse/IMPALA-5605
> Project: IMPALA
> Issue Type: Task
> Components: Docs
>Affects Versions: Impala 2.9.0
>Reporter: Matthew Mulder
>Assignee: Alex Rodoni
>Priority: Major
> Fix For: Impala 2.10.0
>
>
> Depending on the workload, Impala may need to create a very large number of
> threads. If so, it is necessary to configure the system correctly to prevent
> Impala from crashing because of resource limitations. Such a crash would look
> like this:{code}F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit
> fatal error: Unable to allocate section memory!
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl
> >'{code}To prevent this, each Impala host should be configured like
> this:{code}echo 200 > /proc/sys/kernel/threads-max
> echo 200 > /proc/sys/kernel/pid_max
> echo 800 > /proc/sys/vm/max_map_count{code}In /etc/security/limits.conf
> add{code}impala soft nproc 262144
> impala hard nproc 262144{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Target Version: Impala 3.1.0, Impala 2.13.0 (was: Impala 2.13.0, Impala 3.1.0) Description: The new statements are: ALTER DB/TABLE/VIEW SET OWNER SHOW GRANT USER SHOW GRANT USER ON Everything else is defects or implementation details. ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 -Fixed in IMPALA-7525 SHOW GRANT USER https://issues.apache.org/jira/browse/IMPALA-6989- -Fixed in IMPALA-7525 SHOW GRANT ROLE/USER ON https://issues.apache.org/jira/browse/IMPALA-6991- ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 was: The new statements are: ALTER DB/TABLE/VIEW SET OWNER SHOW GRANT USER SHOW GRANT USER ON Everything else is defects or implementation details. https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 https://issues.apache.org/jira/browse/IMPALA-7016 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > The new statements are: > ALTER DB/TABLE/VIEW SET OWNER > SHOW GRANT USER > SHOW GRANT USER ON > > Everything else is defects or implementation details. > ALTER TABLE / VIEW https://issues.apache.org/jira/browse/IMPALA-6988 > -Fixed in IMPALA-7525 SHOW GRANT USER > https://issues.apache.org/jira/browse/IMPALA-6989- > -Fixed in IMPALA-7525 SHOW GRANT ROLE/USER ON > https://issues.apache.org/jira/browse/IMPALA-6991- > ALTER DATABASE SET OWNER https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter
[
https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648307#comment-16648307
]
Joe McDonnell commented on IMPALA-7704:
---
Reproduces locally on ASAN query_test/test_insert_parquet.py. Without
IMPALA-7644, the tests don't hit the ASAN issue. I will put a revert up for
review and start broader tests.
> ASAN tests failing in HdfsParquetTableWriter
>
>
> Key: IMPALA-7704
> URL: https://issues.apache.org/jira/browse/IMPALA-7704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
> at pc 0x016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion
> (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite
> /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in
> impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in
> impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
> impala::OutputPartition*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
> #16 0x39a11e9 in thread_proxy
>
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: The new statements are: ALTER DB/TABLE/VIEW SET OWNER SHOW GRANT USER SHOW GRANT USER ON Everything else is defects or implementation details. https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 https://issues.apache.org/jira/browse/IMPALA-7016 was: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 https://issues.apache.org/jira/browse/IMPALA-7016 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > The new statements are: > ALTER DB/TABLE/VIEW SET OWNER > SHOW GRANT USER > SHOW GRANT USER ON > > Everything else is defects or implementation details. > https://issues.apache.org/jira/browse/IMPALA-6988 > https://issues.apache.org/jira/browse/IMPALA-6989 > https://issues.apache.org/jira/browse/IMPALA-6991 > https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7076 started by Alex Rodoni. --- > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > The new statements are: > ALTER DB/TABLE/VIEW SET OWNER > SHOW GRANT USER > SHOW GRANT USER ON > > Everything else is defects or implementation details. > https://issues.apache.org/jira/browse/IMPALA-6988 > https://issues.apache.org/jira/browse/IMPALA-6989 > https://issues.apache.org/jira/browse/IMPALA-6991 > https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 https://issues.apache.org/jira/browse/IMPALA-7016 was: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > https://issues.apache.org/jira/browse/IMPALA-6988 > > https://issues.apache.org/jira/browse/IMPALA-6989 > > https://issues.apache.org/jira/browse/IMPALA-6991 > > https://issues.apache.org/jira/browse/IMPALA-7016 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 https://issues.apache.org/jira/browse/IMPALA-6991 was: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > https://issues.apache.org/jira/browse/IMPALA-6988 > > https://issues.apache.org/jira/browse/IMPALA-6989 > > https://issues.apache.org/jira/browse/IMPALA-6991 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: https://issues.apache.org/jira/browse/IMPALA-6988 https://issues.apache.org/jira/browse/IMPALA-6989 was: https://issues.apache.org/jira/browse/IMPALA-6988 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > https://issues.apache.org/jira/browse/IMPALA-6988 > > https://issues.apache.org/jira/browse/IMPALA-6989 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7076) Impala 2.13 & 3.1 Docs: Object ownership for Impala
[ https://issues.apache.org/jira/browse/IMPALA-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7076: Description: https://issues.apache.org/jira/browse/IMPALA-6988 > Impala 2.13 & 3.1 Docs: Object ownership for Impala > --- > > Key: IMPALA-7076 > URL: https://issues.apache.org/jira/browse/IMPALA-7076 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Fredy Wijaya >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc > > https://issues.apache.org/jira/browse/IMPALA-6988 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter
[
https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648228#comment-16648228
]
Lars Volker commented on IMPALA-7704:
-
Could be this change: https://gerrit.cloudera.org/#/c/11563/
CC [~boroknagyz]
> ASAN tests failing in HdfsParquetTableWriter
>
>
> Key: IMPALA-7704
> URL: https://issues.apache.org/jira/browse/IMPALA-7704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
> at pc 0x016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion
> (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite
> /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in
> impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in
> impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
> impala::OutputPartition*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
> #16 0x39a11e9 in thread_proxy
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
> #17 0x7f02e19f4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> #18 0x7f02e150b34c in __clone (/lib64/libc.so.6+0xf834c)
>
[jira] [Assigned] (IMPALA-7704) ASAN tests failing in HdfsParquetTableWriter
[
https://issues.apache.org/jira/browse/IMPALA-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker reassigned IMPALA-7704:
---
Assignee: Zoltán Borók-Nagy
> ASAN tests failing in HdfsParquetTableWriter
>
>
> Key: IMPALA-7704
> URL: https://issues.apache.org/jira/browse/IMPALA-7704
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Zoltán Borók-Nagy
>Priority: Blocker
> Labels: broken-build
>
> ASAN tests have been failing for the last few runs. Here is the output:
> {noformat}
> ==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0
> at pc 0x016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0
> READ of size 32681 at 0x7ef9312e0ec0 thread T82364
> #0 0x16a0c97 in __interceptor_memcpy
> /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738
> #1 0x7f02e43c2dca in jni_SetByteArrayRegion
> (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca)
> #2 0x4ae29b7 in hdfsWrite
> /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626
> #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13
> #4 0x25ac037 in
> impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5
> #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5
> #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3
> #7 0x251e424 in
> impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*,
> impala::OutputPartition*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5
> #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5
> #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3
> #10 0x1fbbd5c in impala::FragmentInstanceState::Exec()
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14
> #11 0x1fd5f94 in
> impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24
> #12 0x1cdef96 in boost::function0::operator()() const
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14
> #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*)
> /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3
> #14 0x23b7708 in void boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*>
> >::operator() boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list0>(boost::_bi::type, void (*&)(std::string const&,
> std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*), boost::_bi::list0&, int)
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9
> #15 0x23b755b in boost::_bi::bind_t std::string const&, boost::function, impala::ThreadDebugInfo const*,
> impala::Promise*),
> boost::_bi::list5,
> boost::_bi::value, boost::_bi::value >,
> boost::_bi::value,
> boost::_bi::value*> >
> >::operator()()
> /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27
> #16 0x39a11e9 in thread_proxy
> (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9)
> #17 0x7f02e19f4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
> #18 0x7f02e150b34c in __clone (/lib64/libc.so.6+0xf834c)
> 0x7ef9312e0ec0 is located 161472 bytes inside of
[jira] [Commented] (IMPALA-7623) Impala Doc: Disallow set kudu.table_name in CREATE and ALTER TABLE
[ https://issues.apache.org/jira/browse/IMPALA-7623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648211#comment-16648211 ] Alex Rodoni commented on IMPALA-7623: - [DOCS] Added a version notice for changing managed Kudu table names in Impala With [IMPALA-5654|http://issues.cloudera.org/browse/IMPALA-5654], users can no longer change kudu.table_name property for managed Kudu tables in Impala. Change-Id: If26f103d931dd7ed57b56cf34b0010d14c098928 Reviewed-on: [http://gerrit.cloudera.org:8080/11515] Reviewed-by: Alexey Serbin Tested-by: Kudu Jenkins > Impala Doc: Disallow set kudu.table_name in CREATE and ALTER TABLE > --- > > Key: IMPALA-7623 > URL: https://issues.apache.org/jira/browse/IMPALA-7623 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Fix For: Impala 3.1.0 > > > Update the following docs for managed(internal) Kudu tables: > impala_table > impala_create_table > impala_alter_table > impala_kudu > > https://gerrit.cloudera.org/#/c/11518/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7654) TRUNCATE docs appear to be inaccurate
[ https://issues.apache.org/jira/browse/IMPALA-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648206#comment-16648206 ] Alex Rodoni commented on IMPALA-7654: - |[IMPALA-7654|http://issues.cloudera.org/browse/IMPALA-7654]: [DOCS] TRUNCATE does not put HDFS data files to the trash Change-Id: I4930b2345515abf9b086575a476d20c87117e5c4 Reviewed-on: [http://gerrit.cloudera.org:8080/11583] Tested-by: Impala Public Jenkins Reviewed-by: Joe McDonnell | > TRUNCATE docs appear to be inaccurate > - > > Key: IMPALA-7654 > URL: https://issues.apache.org/jira/browse/IMPALA-7654 > Project: IMPALA > Issue Type: Bug > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Fix For: Impala 3.1.0 > > > Some users pointed out that this seems to be inaccurate: "Any HDFS data files > removed by this statement go into the HDFS trashcan, from which you can > recover them within a defined time interval if this operation turns out to be > a mistake." I didn't confirm this but filing JIRA to track. > http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-truncate-doesn-t-move-to-HDFS-trashcan/m-p/80580#M4955?eid=1=1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7607) Add a reference to EXEC_TIME_LIMIT_S to "Setting Timeouts" page
[ https://issues.apache.org/jira/browse/IMPALA-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648204#comment-16648204 ] Alex Rodoni commented on IMPALA-7607: - |[IMPALA-7607|http://issues.cloudera.org/browse/IMPALA-7607]: [DOCS] Add a reference to EXEC_TIME_LIMIT_S to TIMEOUT doc Change-Id: Iaf7bfd7535ba0f8f51fd0994ad6e9e3f2637b2c1 Reviewed-on: [http://gerrit.cloudera.org:8080/11621] Tested-by: Impala Public Jenkins Reviewed-by: Bikramjeet Vig | > Add a reference to EXEC_TIME_LIMIT_S to "Setting Timeouts" page > --- > > Key: IMPALA-7607 > URL: https://issues.apache.org/jira/browse/IMPALA-7607 > Project: IMPALA > Issue Type: Task > Components: Docs >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Vincent Tran >Assignee: Alex Rodoni >Priority: Major > Fix For: Impala 3.1.0 > > > The "Setting Timeouts" page in the doc should include a bit about the > recently added flag EXEC_TIME_LIMIT_S. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7680) Impala Doc: Document how to set default query options for a resource pool
[ https://issues.apache.org/jira/browse/IMPALA-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648198#comment-16648198 ] Alex Rodoni commented on IMPALA-7680: - |[IMPALA-7680|http://issues.cloudera.org/browse/IMPALA-7680]: [DOCS] Minor edits to the command line config section Change-Id: I3a3115feb2de1112ff0d2687a83a1307bd0791a5 Reviewed-on: [http://gerrit.cloudera.org:8080/11636] Tested-by: Impala Public Jenkins Reviewed-by: Bikramjeet Vig | > Impala Doc: Document how to set default query options for a resource pool > - > > Key: IMPALA-7680 > URL: https://issues.apache.org/jira/browse/IMPALA-7680 > Project: IMPALA > Issue Type: Improvement > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Fix For: Impala 3.1.0 > > > Describe how to set default query options for each resource pool. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7682) AuthorizationPolicy is not thread-safe
[ https://issues.apache.org/jira/browse/IMPALA-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648154#comment-16648154 ] Fredy Wijaya commented on IMPALA-7682: -- IMPALA-7682: Make AuthorizationPolicy thread-safe This patch makes AuthorizationPolicy thread-safe by making AuthorizationPolicy::listPrivileges(Set groups, Set users, ActiveRoleSet roleSet) synchronized. Change-Id: I28c516c0ab3e48ab4284084afd7149ed1a3c6cb2 Reviewed-on: http://gerrit.cloudera.org:8080/11644 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > AuthorizationPolicy is not thread-safe > -- > > Key: IMPALA-7682 > URL: https://issues.apache.org/jira/browse/IMPALA-7682 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Fix For: Impala 3.1.0 > > > The method public Set listPrivileges(Set groups, Set > users, ActiveRoleSet roleSet) in AuthorizationPolicy needs to be > synchronized. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7688) Spurious error messages when updating owner privileges
[ https://issues.apache.org/jira/browse/IMPALA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648155#comment-16648155 ] Fredy Wijaya commented on IMPALA-7688: -- IMPALA-7688: Fix spurious error messages when updating owner privileges Failure in updating owner privileges is not an issue since this could mean a Sentry refresh occurred while updating owner privileges and Sentry refresh itself will update all privileges including owner privileges. This patch changes the code to log the failure in updating owner privileges as WARN instead of ERROR. Testing: - Ran all FE tests - Ran all E2E authorization tests Change-Id: I4b46df01c5675ffed6528b7a73e68518664d8be0 Reviewed-on: http://gerrit.cloudera.org:8080/11649 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Spurious error messages when updating owner privileges > -- > > Key: IMPALA-7688 > URL: https://issues.apache.org/jira/browse/IMPALA-7688 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Fix For: Impala 3.1.0 > > > Failure in updating owner privileges is not an issue since this could mean a > Sentry refresh occurred while updating owner privileges and Sentry refresh > itself will update all privileges including owner privileges. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7676) DESCRIBE on table should require VIEW_METADATA privilege
[
https://issues.apache.org/jira/browse/IMPALA-7676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648153#comment-16648153
]
Fredy Wijaya commented on IMPALA-7676:
--
IMPALA-7676: DESCRIBE on table should require VIEW_METADATA privilege
IMPALA-6479 broke the DESCRIBE's privilege model by changing the
privilege from VIEW_METADATA to ANY in order to support column-level
privileges in DESCRIBE. This caused an issue where having non-
VIEW_METADATA privilege, such as CREATE privilege on a particular
database allows executing a DESCRIBE statement on all tables in the
database. This behavior is also inconsistent with Hive's DESCRIBE
and Impala's DESCRIBE DATABASE privilege models. Although there is not
any security risk for this particular issue since having non-
VIEW METADATA on a particular database always returns an empty result,
fixing this issue will make the behavior consistent with Hive and also
DESCRIBE DATABASE in Impala. This patch fixes the issue by changing the
privilege requirement back from ANY to VIEW_METADATA.
Testing:
- Ran all FE tests
Change-Id: I283e30ebff6d61e779a4cec8284cae0ccb90cc49
Reviewed-on: http://gerrit.cloudera.org:8080/11617
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> DESCRIBE on table should require VIEW_METADATA privilege
>
>
> Key: IMPALA-7676
> URL: https://issues.apache.org/jira/browse/IMPALA-7676
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.0, Impala 2.12.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
> Labels: security
> Fix For: Impala 3.1.0
>
>
> Assume there exists a table foo.bar.
> {noformat}
> [localhost:21000] default> create role foo_role;
> [localhost:21000] default> grant role to group test_user;
> [localhost:21000] default> grant create on database foo to role foo_role;
> [localhost:21000] default> describe foo.bar;
> Query: describe foo.bar
> Fetched 0 row(s) in 0.01s
> {noformat}
> Let's contrast this with "describe database".
> {noformat}
> [localhost:21000] default> grant create on server to role foo_role;
> [localhost:21000] default> describe database functional;
> Query: describe database functional
> ERROR: AuthorizationException: User 'test_user' does not have privileges to
> access: functional
> {noformat}
> The "describe table" behavior is inconsistent with "describe database" and
> it's also not a good idea having "create" privilege on a particular table to
> allow issuing "describe table" on another table although no information is
> shown on another table. This is also to make the behavior consistent with
> Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7671) SHOW GRANT USER ON is broken
[
https://issues.apache.org/jira/browse/IMPALA-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648151#comment-16648151
]
Fredy Wijaya commented on IMPALA-7671:
--
IMPALA-7671: Fix broken SHOW GRANT USER ON
This patch fixes the broken SHOW GRANT USER ON that always
shows an empty result due to incorrect comparison between TPrivilege for
the filter vs TPrivilege for the actual privilege that should not
consider the "grantoption".
Testing:
- Added new E2E tests
- Ran all FE tests
- Ran all authorization E2E tests
Change-Id: I7adc403caddd18e5a954cf6affd5d1d555b9f5f0
Reviewed-on: http://gerrit.cloudera.org:8080/11598
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> SHOW GRANT USER ON is broken
> -
>
> Key: IMPALA-7671
> URL: https://issues.apache.org/jira/browse/IMPALA-7671
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.1.0
>Reporter: Fredy Wijaya
>Assignee: Fredy Wijaya
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> {noformat}
> [localhost:21000] default> show grant user foobar;
> Query: show grant user foobar
> ++++--+---++-+---+--+---+
> | principal_type | principal_name | scope | database | table | column | uri
> | privilege | grant_option | create_time |
> ++++--+---++-+---+--+---+
> | USER | foobar| table | default | foo || |
> owner | true | Fri, Oct 05 2018 11:38:14.173 |
> ++++--+---++-+---+--+---+
> [localhost:21000] default> show grant user foobar on table foo;
> Query: show grant user foobar on table foo
> Fetched 0 row(s) in 0.01s
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7633) count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant
[ https://issues.apache.org/jira/browse/IMPALA-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648150#comment-16648150 ] Fredy Wijaya commented on IMPALA-7633: -- IMPALA-7633: count_user_privilege isn't 0 at the end of test_owner This patch adds a retry loop to validate the count of user privileges in a SHOW GRANT USER statement after a DDL operation. The core of the problem is cache consistency. When a DDL operation is executing, like drop database, HMS is updated with the correct metadata, and Sentry is updated to remove privileges from HMS. However, if a Sentry Refresh happens between when HMS is updated CatalogOpExecutor:1322, and when the local catalog privileges are updated CatalogOpExecutor:1341, then the remove privilege call will fail and a log entry with "User does not exist: foo_user" will be written to the log. The result is that the response back to impalad with catalog updates will not contain the user and privilege updates. Ultimately, when the "SHOW GRANT USER" statement is run, it uses the local Impalad catalog which still contains the privlege because it has not yet been updated from statestore. This is not a security problem because the privilege exists for a maximum of 2s by default, for an object that does not exist. This is the same result as if the database was dropped from Hive, except in that case it can be up to 62s by default that the privilege exists for no object. Testing: - After retry was added, ran tests until log entry appeared and validate test did not fail. Change-Id: Ifbba0fbd0e24a24b3f2af82ad5209f3fb7fb387b > count_user_privilege isn't 0 at the end of test_owner_privileges_without_grant > -- > > Key: IMPALA-7633 > URL: https://issues.apache.org/jira/browse/IMPALA-7633 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Alex Rodoni >Assignee: Adam Holley >Priority: Critical > Fix For: Impala 3.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7667) sentry.db.explicit.grants.permitted does not accept empty value
[ https://issues.apache.org/jira/browse/IMPALA-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648148#comment-16648148 ] Fredy Wijaya commented on IMPALA-7667: -- IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value Empty value in sentry.db.explicit.grants.permitted means allow all privileges to be explicitly granted. However a bug in SENTRY-2424 causes empty value to be ignored. The patch uses a workaround to add a space to mean the same thing as an empty value. The patch also upgrades CDH_BUILD_NUMBER to 618371 to enable sentry.db.explicit.grants.permitted configuration. Testing: - Ran all core tests Change-Id: Iddcb80da6ba37405ce59e2d5cd79916f8eb004ec Reviewed-on: http://gerrit.cloudera.org:8080/11592 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > sentry.db.explicit.grants.permitted does not accept empty value > --- > > Key: IMPALA-7667 > URL: https://issues.apache.org/jira/browse/IMPALA-7667 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.1.0 > > > sentry.db.explicit.grants.permitted supports putting an empty value to mean > allow all privileges. However, there's a bug with empty value as described in > https://issues.apache.org/jira/browse/SENTRY-2424. The workaround is to add a > space until the to mean allow all privileges. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-7703) Upgrade to Sentry 2.1.0
[ https://issues.apache.org/jira/browse/IMPALA-7703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7703 started by Fredy Wijaya. > Upgrade to Sentry 2.1.0 > --- > > Key: IMPALA-7703 > URL: https://issues.apache.org/jira/browse/IMPALA-7703 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > > Upgrade to Sentry 2.1.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7703) Upgrade to Sentry 2.1.0
Fredy Wijaya created IMPALA-7703: Summary: Upgrade to Sentry 2.1.0 Key: IMPALA-7703 URL: https://issues.apache.org/jira/browse/IMPALA-7703 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Fredy Wijaya Assignee: Fredy Wijaya Upgrade to Sentry 2.1.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
