[jira] [Created] (IMPALA-9130) Upgrade external non-ACID table to ACID from Impala
Gabor Kaszab created IMPALA-9130:
Summary: Upgrade external non-ACID table to ACID from Impala
Key: IMPALA-9130
URL: https://issues.apache.org/jira/browse/IMPALA-9130
Project: IMPALA
Issue Type: Bug
Components: Catalog, Frontend
Affects Versions: Impala 3.3.0
Reporter: Gabor Kaszab
If you have an external, non-ACID table and try to upgrade it to become an ACID
table you get an error message that an external table is not allowed to be
promoted to ACID. This is fine, however if in the very same step you set
'EXTERNAL' = 'FALSE' in table properties you still get the same error while
Hive is able to execute it.
Steps to repro:
1) Create a non-ACID external table. (or a single non-ACID table if you use
Hive that contains HIVE-22158)
2) Upgrade the table
{code:java}
alter table tbl set tblproperties ('transactional'='true',
'transactional_properties'='insert_only', 'EXTERNAL'='FALSE');
{code}
Step 2) fails in Impala but succeeds in Hive
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IMPALA-9131) Use single quotes when printing out FORMAT clause within CAST.
Gabor Kaszab created IMPALA-9131:
Summary: Use single quotes when printing out FORMAT clause within
CAST.
Key: IMPALA-9131
URL: https://issues.apache.org/jira/browse/IMPALA-9131
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.3.0
Reporter: Gabor Kaszab
Assignee: Gabor Kaszab
Here the content of the FORMAT clause is surrounded by double quotes.
{code:java}
select cast('2016/10/10' as date format '/MM/DD');
++
| cast('2016/10/10' as date format "/mm/dd") |
++
| 2016-10-10 |
++
{code}
In order to follow SQL standards this should be surrounded by single quotes
regardless of how the user gave the FORMAT clause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IMPALA-9132) Explain statements should not cause NPE in LogLineageRecord()
Anurag Mantripragada created IMPALA-9132: Summary: Explain statements should not cause NPE in LogLineageRecord() Key: IMPALA-9132 URL: https://issues.apache.org/jira/browse/IMPALA-9132 Project: IMPALA Issue Type: Bug Reporter: Anurag Mantripragada Assignee: Anurag Mantripragada For DDLs, LogLineageRecord() adds certain fields to the lineageGraph in the backend. However, explain statements do not have a catalogOpExecutor causing a NPE. We should, in general, avoid creating lineage records for Explain as Atlas currently does not use them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-8557) Impala on ABFS failed with error "IllegalArgumentException: ABFS does not allow files or directories to end with a dot."
[ https://issues.apache.org/jira/browse/IMPALA-8557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved IMPALA-8557. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed > Impala on ABFS failed with error "IllegalArgumentException: ABFS does not > allow files or directories to end with a dot." > > > Key: IMPALA-8557 > URL: https://issues.apache.org/jira/browse/IMPALA-8557 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Eric Lin >Assignee: Sahil Takiar >Priority: Major > Fix For: Impala 3.4.0 > > > HDFS introduced below feature to stop users from creating a file that ends > with "." on ABFS: > https://issues.apache.org/jira/browse/HADOOP-15860 > As a result of this change, Impala now writes to ABFS fails with such error. > I can see that it generates temp file using this format "$0.$1.$2": > https://github.com/cloudera/Impala/blob/cdh6.2.0/be/src/exec/hdfs-table-sink.cc#L329 > $2 is the file extension and will be empty if it is TEXT file format: > https://github.com/cloudera/Impala/blob/cdh6.2.0/be/src/exec/hdfs-text-table-writer.cc#L65 > Since HADOOP-15860 was backported into CDH6.2, it is currently only affecting > 6.2 and works in older versions. > There is no way to override this empty file extension so no workaround is > possible, unless user choose another file format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-9117) test_lineage.py and test_mt_dop.py are failing on ABFS
[
https://issues.apache.org/jira/browse/IMPALA-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar resolved IMPALA-9117.
--
Fix Version/s: Impala 3.4.0
Resolution: Fixed
> test_lineage.py and test_mt_dop.py are failing on ABFS
> --
>
> Key: IMPALA-9117
> URL: https://issues.apache.org/jira/browse/IMPALA-9117
> Project: IMPALA
> Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: Impala 3.4.0
>
>
> Both failures are known issues.
> {{TestLineage::test_lineage_output}} is failing because the test requires
> HBase to run (this test is already disabled for S3).
> {{TestMtDopFlags::test_mt_dop_all}} is failing because it runs
> {{QueryTest/insert}} which includes a query that writes a folder that ends in
> a dot. ABFS does not allow files or directories to end in a dot - IMPALA-7860
> / IMPALA-7681 / HADOOP-15860.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (IMPALA-7726) Drop with purge tests fail against ABFS due to trash misbehavior
[ https://issues.apache.org/jira/browse/IMPALA-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved IMPALA-7726. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed Closing as Fixed. I re-enabled the tests, looped them overnight, and didn't hit any failures. So it is likely whatever bug was causing these issues has been resolved. > Drop with purge tests fail against ABFS due to trash misbehavior > > > Key: IMPALA-7726 > URL: https://issues.apache.org/jira/browse/IMPALA-7726 > Project: IMPALA > Issue Type: Bug >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Labels: flaky > Fix For: Impala 3.4.0 > > > In testing IMPALA-7681, I've seen test_drop_partition_with_purge and > test_drop_table_with_purge fail because of files not found in the trash are a > drop without purge. I've traced that functionality through Hive, which uses > Hadoop's Trash API, and traced through a bunch of scenarios in that API with > ABFS and I can't see it misbehaving in any way. It also should be pretty > FS-agnostic. I also suspected a bug in abfs_utils.py's exists() function, but > have not been able to find one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-7860) Tests use partition name that isn't supported on ABFS
[ https://issues.apache.org/jira/browse/IMPALA-7860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved IMPALA-7860. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed Closing this as Fixed. HADOOP-15860 was done a while ago, and now Impala-on-ABFS cannot write files / directories that end with a period (which is expected). There was one bug that was introduced to Impala due to this change: IMPALA-8557 - but that has been fixed now as well. In IMPALA-9117 I created a new skip flag for ABFS tests for the "cannot write write trailing periods" behavior, and added it to any affected tests. > Tests use partition name that isn't supported on ABFS > - > > Key: IMPALA-7860 > URL: https://issues.apache.org/jira/browse/IMPALA-7860 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Reporter: Sean Mackrory >Priority: Major > Fix For: Impala 3.4.0 > > > IMPALA-7681 introduced support for the ADLS Gen2 service / ABFS client. As > mentioned in the code review for that > (https://gerrit.cloudera.org/#/c/11630/) a couple of tests were failing > because they use a partition name that ends with a period. If the tests are > modified to end with anything other than a period, they work just fine. > In HADOOP-15860, that's sounding like it's just a known limitation of the > blob storage that shares infrastructure with ADLS Gen2 that won't be changing > any time soon. I propose we modify the tests to just use a slightly different > partition name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IMPALA-9132) Explain statements should not cause NPE in LogLineageRecord()
[ https://issues.apache.org/jira/browse/IMPALA-9132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada resolved IMPALA-9132. -- Fix Version/s: Impala 3.4.0 Resolution: Fixed > Explain statements should not cause NPE in LogLineageRecord() > - > > Key: IMPALA-9132 > URL: https://issues.apache.org/jira/browse/IMPALA-9132 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Blocker > Labels: crash > Fix For: Impala 3.4.0 > > > For DDLs, LogLineageRecord() adds certain fields to the lineageGraph in the > backend. However, explain statements do not have a catalogOpExecutor causing > a NPE. > We should, in general, avoid creating lineage records for Explain as Atlas > currently does not use them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
