[jira] [Resolved] (IMPALA-5843) Use page index in Parquet files to skip pages
[ https://issues.apache.org/jira/browse/IMPALA-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-5843. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Use page index in Parquet files to skip pages > - > > Key: IMPALA-5843 > URL: https://issues.apache.org/jira/browse/IMPALA-5843 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.10.0 >Reporter: Lars Volker >Assignee: Zoltán Borók-Nagy >Priority: Critical > Labels: parquet, performance > Fix For: Impala 3.3.0 > > > Once IMPALA-5842 has been resolved, we should skip pages based on the page > index in Parquet files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8524) Avoid calling "hive" via command line in EE tests
[ https://issues.apache.org/jira/browse/IMPALA-8524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer resolved IMPALA-8524. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Avoid calling "hive" via command line in EE tests > - > > Key: IMPALA-8524 > URL: https://issues.apache.org/jira/browse/IMPALA-8524 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Fix For: Impala 3.3.0 > > > "hive -e SQL..." without further parameters no longer works when > USE_CDP_HIVE=true (it doesn't establish a connection). Some tests used this > to load data. > These calls can be replaced with ImpalaTestSuite.run_stmt_in_hive() which > seems like a good idea regardless of the Hive 3 efforts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8541) Some of the JUnit tests in AuthorizationTest.java would fail if there are lingering roles already granted the privileges to access some databases
Fang-Yu Rao created IMPALA-8541: --- Summary: Some of the JUnit tests in AuthorizationTest.java would fail if there are lingering roles already granted the privileges to access some databases Key: IMPALA-8541 URL: https://issues.apache.org/jira/browse/IMPALA-8541 Project: IMPALA Issue Type: Improvement Reporter: Fang-Yu Rao Assignee: Fang-Yu Rao The test TestShortUsernameUsed() in AuthorizationTest.java would fail if there exists some role already granted the privilege to access some databases, e.g., the database "functional". Specifically, for the SQL statement "select * from alltypes", if the role is already granted the privilege to the database "functional", only AnalysisException will be thrown. To address this issue, we remove all the lingering roles before running each test using [https://github.com/apache/impala/blob/master/fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java#L172]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8542) Access trace collection for data cache
Todd Lipcon created IMPALA-8542: --- Summary: Access trace collection for data cache Key: IMPALA-8542 URL: https://issues.apache.org/jira/browse/IMPALA-8542 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Todd Lipcon Assignee: Todd Lipcon Now that we have a remote-read data cache, it would be useful to log an access trace. The trace can be then fed back into various cache policy simulators to compare the relative performance, and do "what if" analysis (how would hit rate react with larger/smaller capacities) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8543) Log more diagnostics information in TAcceptQueueServer
Michael Ho created IMPALA-8543:
--
Summary: Log more diagnostics information in TAcceptQueueServer
Key: IMPALA-8543
URL: https://issues.apache.org/jira/browse/IMPALA-8543
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 3.2.0, Impala 3.1.0, Impala 2.12.0
Reporter: Michael Ho
Assignee: Michael Ho
There is currently not much diagnostic information in
{{TAcceptQueueServer.cpp}}. It would be nice to add some additional logging to
identify clients which misbehaved or took longer than expected when setting up
connections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-2029) Impala in CDH 5.2.0 fails to compile with hadoop 2.7
[ https://issues.apache.org/jira/browse/IMPALA-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-2029. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Impala in CDH 5.2.0 fails to compile with hadoop 2.7 > > > Key: IMPALA-2029 > URL: https://issues.apache.org/jira/browse/IMPALA-2029 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.2 > Environment: Red Hat 6.4 and gcc 4.3.4 >Reporter: Varun Saxena >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.3.0 > > > Compilation fails with below error message : > ../../build/release/exec/libExec.a(hbase-table-scanner.cc.o): In function > `impala::HBaseTableScanner::Init()': > /usr1/code/Impala/code/current/impala/be/src/exec/hbase-table-scanner.cc:113: > undefined reference to `getJNIEnv' > ../../build/release/exprs/libExprs.a(hive-udf-call.cc.o):/usr1/code/Impala/code/current/impala/be/src/exprs/hive-udf-call.cc:227: > more undefined references to `getJNIEnv' follow > collect2: ld returned 1 exit status > make[3]: *** [be/build/release/service/impalad] Error 1 > make[2]: *** [be/src/service/CMakeFiles/impalad.dir/all] Error 2 > make[1]: *** [be/src/service/CMakeFiles/impalad.dir/rule] Error 2 > make: *** [impalad] Error 2 > Compiler Impala Failed, exit > This issue is coming because libhdfs in hadoop 2.7.0 has made visibility of > getJNIEnv() as hidden. > But shouldn't Impala create its own JNIEnv ? > These HBase related files seem to have no direct connection with libhdfs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8544) Expose additional S3A / S3Guard metrics
Sahil Takiar created IMPALA-8544:
Summary: Expose additional S3A / S3Guard metrics
Key: IMPALA-8544
URL: https://issues.apache.org/jira/browse/IMPALA-8544
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
S3A / S3Guard internally collects several useful metrics that we should
consider exposing to Impala users. The full list of statistics can be found in
{{o.a.h.fs.s3a.Statistic}}. The stats include: the number of S3 operations
performed (put, get, etc.), invocation counts for various {{FileSystem}}
methods, stream statistics (bytes read, written, etc.), etc.
Some interesting stats that stand out:
* "stream_aborted": "Count of times the TCP stream was aborted" - the number
of TCP connection aborts, a high value would indicate performance issues
* "stream_read_exceptions" : "Number of exceptions invoked on input streams" -
incremented whenever an {{IOException}} is caught while reading (these
exception don't always get propagated to Impala because they trigger a retry)
* "store_io_throttled": "Requests throttled and retried" - looks like it
tracks the number of times the fs retries an operation because the original
request hit a throttling exception
* "s3guard_metadatastore_retry": "S3Guard metadata store retry events" - looks
like it tracks the number of times the fs retries S3Guard operations
* "s3guard_metadatastore_throttled" : "S3Guard metadata store throttled
events" - similar to "store_io_throttled" but looks like it is specific to
S3Guard
We should consider how to expose these metrics via Impala logs / runtime
profiles.
There are a few options:
* {{S3AFileSystem}} exposes {{StorageStatistics}} specific to S3A / S3Guard
via the {{FileSystem#getStorageStatistics}} method; the
{{S3AStorageStatistics}} seems to include all the S3A / S3Guard metrics,
however, I think the stats might be aggregated globally, which would make it
hard to create per-query specific metrics
* {{S3AInstrumentation}} exposes all the metrics as well, and looks like it is
per-fs instance, so it is not aggregated globally; {{S3AInstrumentation}}
extends {{o.a.h.metrics2.MetricsSource}} so perhaps it is exposed via some API
(haven't looked into this yet)
* {{S3AInputStream#toString}} dumps the statistics from
{{o.a.h.fs.s3a.S3AInstrumentation.InputStreamStatistics}} and
{{S3AFileSystem#toString}} dumps them all as well
* {{S3AFileSystem}} updates the stats in
{{o.a.h.fs.Statistics.StatisticsData}} as well (e.g. bytesRead, bytesWritten,
etc.)
Impala has a {{hdfs-fs-cache}} as well, so {{hdfsFs}} objects get shared across
threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-1857) Consistent and human-friendly messages for HDFS permission errors
[
https://issues.apache.org/jira/browse/IMPALA-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-1857.
---
Resolution: Won't Fix
Seems fine to me. I think we can live with this
> Consistent and human-friendly messages for HDFS permission errors
> -
>
> Key: IMPALA-1857
> URL: https://issues.apache.org/jira/browse/IMPALA-1857
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 2.2, Impala 2.1.2
>Reporter: John Russell
>Priority: Minor
> Labels: ramp-up, supportability
>
> Attempting to do statements such as INSERT, LOAD DATA, SELECT, COMPUTE STATS,
> or even REFRESH, DESCRIBE, or SHOW COLUMN STATS can fail if the 'impala' user
> does not have permissions to read/write/execute the right files and
> directories.
> Currently the code just bubbles up whatever raw error was reported from the
> low-level library, including a stack trace or raw libhdfs error code. This
> means that equivalent problems in fe and be have different error messages. It
> also creates the impression that it's a scary internal error rather than a
> configuration problem.
> For example, a SELECT appears to fail if any file in the table does not have
> read permission. INSERT appears to need read and execute permissions on the
> table directory, but not any read/write permissions on existing files in the
> table (i.e. to remove them in the case of INSERT OVERWRITE).
> The stack traces look like so:
> {code}
> [localhost:21000] > show column stats dir_no_read;
> ERROR: AnalysisException: Failed to load metadata for table:
> hdfs_perms.dir_no_read
> CAUSED BY: TableLoadingException: Failed to load metadata for table:
> dir_no_read
> CAUSED BY: CatalogException: Failed to create partition:
> CAUSED BY: AccessControlException: Permission denied: user=impala,
> access=READ_EXECUTE,
> inode="/user/impala/warehouse/hdfs_perms.db/dir_no_read":impala:hive:d-wx-wx-wt
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6287)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6269)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:6194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4793)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4755)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:800)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getListing(AuthorizationProviderProxyClientProtocol.java:310)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:606)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> {code}
> and
> {code}
> [localhost:21000] > load data inpath '/user/impala/foo' into table
> dir_no_write;
> ERROR: AccessControlException: Permission denied: user=impala, access=WRITE,
> inode="/user/impala/warehouse/hdfs_perms.db/dir_no_write":impala:hive:dr-xr-xr-t
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:257)
> at
> org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:238)
> at
> org.apache.had
[jira] [Closed] (IMPALA-5105) Improve error message in the event catalogd crashes
[
https://issues.apache.org/jira/browse/IMPALA-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig closed IMPALA-5105.
--
Resolution: Won't Fix
Behavior has changed, impalad works off of local catalog cache. For a new query
that requires metadata not in the cache, the query fails with a descriptive
message "Error making an RPC call to Catalog server"
> Improve error message in the event catalogd crashes
> ---
>
> Key: IMPALA-5105
> URL: https://issues.apache.org/jira/browse/IMPALA-5105
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Reporter: David Knupp
>Priority: Major
>
> IMPALA-4704 documents a case whereby impalad opens up ports 21000 and 21050
> before the catalog update has been received, and this can lead to queries
> failing with the message:
> {noformat}
> AnalysisException: This Impala daemon is not ready to accept user requests.
> Status: Waiting for catalog update from the StateStore.
> {noformat}
> Also, we have seen cases where queries fail when the catalogd crashes for
> some reason or another, and the exact same error is produced:
> {noformat}
> [impalad-2.abc.xyz.com:21000] > show tables;
> Query: show tables
> ERROR: AnalysisException: This Impala daemon is not ready to accept user
> requests. Status: Waiting for catalog update from the StateStore.
> {noformat}
> One error is a recoverable timing issue, the other is a crashed processed. It
> would be nice if these two conditions were differentiated by unique error
> messages.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-6048) Queries make very slow progress and report WaitForRPC() stuck for too long
[
https://issues.apache.org/jira/browse/IMPALA-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-6048.
Resolution: Duplicate
Fix Version/s: Not Applicable
So, as explained, this could be triggered by heavy spilling under high
concurrency so some nodes are slow to consume row batches, leading to long RPC
wait time. This seems to be very similar to what's being tracked in IMPALA-6294.
> Queries make very slow progress and report WaitForRPC() stuck for too long
> ---
>
> Key: IMPALA-6048
> URL: https://issues.apache.org/jira/browse/IMPALA-6048
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Distributed Exec
>Affects Versions: Impala 2.11.0
>Reporter: Mostafa Mokhtar
>Assignee: Michael Ho
>Priority: Critical
> Fix For: Not Applicable
>
> Attachments: Archive 2.zip
>
>
> When running 32 concurrent queries from TPCDS a couple of instances from
> TPC-DS Q78 9 hours to finish and it appeared to be hung.
> On an idle cluster the query finished in under 5 minutes, profiles attached.
> When the query ran for long fragments reported +16 hours of network
> send/receive time
> The logs show there is a lot of messages like the one below, there are
> incidents for this log message where a node waited too long from an RPC from
> itself
> {code}
> W1012 00:47:57.633549 117475 krpc-data-stream-sender.cc:360] XXX:
> WaitForRPC() stuck for too long address=10.17.234.37:29000
> fragment_instace_id_=1e48ef897e797131:2f05789b05eb dest_node_id_=24
> sender_id_=81
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (IMPALA-8512) Data cache tests failing on older CentOS 6 versions
[
https://issues.apache.org/jira/browse/IMPALA-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Ho resolved IMPALA-8512.
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Data cache tests failing on older CentOS 6 versions
> ---
>
> Key: IMPALA-8512
> URL: https://issues.apache.org/jira/browse/IMPALA-8512
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Michael Ho
>Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.3.0
>
>
> They are failing with errors like:
> {noformat}
> Error: Data dir /tmp/data-cache-test.0 is on an ext4 filesystem vulnerable to
> KUDU-1508.
> {noformat}
> {noformat}
> custom_cluster.test_data_cache.TestDataCache.test_data_cache_deterministic[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> DataCacheTest.TestBasics
> DataCacheTest.RotateFiles
> DataCacheTest.RotateAndDeleteFiles
> DataCacheTest.Eviction
> DataCacheTest.MultiThreadedNoMisses
> DataCacheTest.MultiThreadedWithMisses
> DataCacheTest.MultiPartitions
> DataCacheTest.LargeFootprint
> FilesystemUtil.DirEntryTypes
> custom_cluster.test_data_cache.TestDataCache.test_data_cache[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> {noformat}
> Can we disable these tests on affected systems?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IMPALA-8545) Test ldap authentication
Thomas Tauber-Marshall created IMPALA-8545: -- Summary: Test ldap authentication Key: IMPALA-8545 URL: https://issues.apache.org/jira/browse/IMPALA-8545 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall Impala doesn't currently have any automated tests that exercise the ldap auth functionality. Some ideas to fix this: - Setting up a local ldap server for the minicluster to use during tests, eg. ApacheDS - Mocking out the openldap calls -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8530) Make some unsupported SQL error messages to be more user friendly
[ https://issues.apache.org/jira/browse/IMPALA-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-8530. -- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Make some unsupported SQL error messages to be more user friendly > - > > Key: IMPALA-8530 > URL: https://issues.apache.org/jira/browse/IMPALA-8530 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Critical > Fix For: Impala 3.3.0 > > > Some error messages, such as running "SHOW ROLES" with Ranger aren't very > user friendly. We need to fix those error messages for unsupported SQL > commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (IMPALA-8515) Test impala-shell distribution instead of special dev environment version
[ https://issues.apache.org/jira/browse/IMPALA-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8515. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Test impala-shell distribution instead of special dev environment version > - > > Key: IMPALA-8515 > URL: https://issues.apache.org/jira/browse/IMPALA-8515 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 3.3.0 > > > Impala shell tests use bin/impala-shell.sh, which uses impala-python and > various dev-environment specific infrastructure to run impala-shell. We also > build a shell tarball, which is meant to be a self-contained version of the > shell with all dependencies. > In principle it's better to test the build artifacts rather than the > development environment. Therefore for full builds, where we build the > tarball, we should test the contents of the tarball including the bundled > libraries. > For remote cluster tests, we can continue to use the dev environment (since > we don't necessarily build the shell tarball there). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IMPALA-8546) Collect logs from docker containers in tests
Tim Armstrong created IMPALA-8546: - Summary: Collect logs from docker containers in tests Key: IMPALA-8546 URL: https://issues.apache.org/jira/browse/IMPALA-8546 Project: IMPALA Issue Type: Sub-task Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Tim Armstrong Assignee: Tim Armstrong We should collect the logs from the cluster processes into the logs/ subdirectory for debugging purposes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
