[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189735#comment-15189735 ] Alan Gates commented on HIVE-13201: --- The tests shouldn't be dropped, they should instead be changed so that they are properly transactional tables. The point of the tests is to cover that 'alter table compact' works from the command line. If you feel this is tested elsewhere in a better way then we can remove these tests. Other than that, +1 on the patch. > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15187624#comment-15187624 ] Wei Zheng commented on HIVE-13201: -- [~alangates] Can you confirm? I'm planning to remove those 3 tests. > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183336#comment-15183336 ] Eugene Koifman commented on HIVE-13201: --- Seems to me like a test bug... > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[ https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15182617#comment-15182617 ] Wei Zheng commented on HIVE-13201: -- [~alangates] Can you take a look? As you can see, dbtxnmgr_compact[1-3].q failed. I'm a little confused as to whether we should allow compaction to "pass thru" for non-ACID table. [~ekoifman] FYI > Compaction shouldn't be allowed on non-ACID table > - > > Key: HIVE-13201 > URL: https://issues.apache.org/jira/browse/HIVE-13201 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.0.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13201.1.patch > > > Looks like compaction is allowed on non-ACID table, although that's of no > sense and does nothing. Moreover the compaction request will be enqueued into > COMPACTION_QUEUE metastore table, which brings unnecessary overhead. > We should prevent compaction commands being allowed on non-ACID tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[
https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15181981#comment-15181981
]
Hive QA commented on HIVE-13201:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12791364/HIVE-13201.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9767 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-groupby3_map.q-sample2.q-auto_join14.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-groupby_map_ppr_multi_distinct.q-table_access_keys_stats.q-groupby4_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join_rc.q-insert1.q-vectorized_rcfile_columnar.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ppd_join4.q-join9.q-ppd_join3.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_compact1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_compact2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_compact3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7173/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7173/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7173/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12791364 - PreCommit-HIVE-TRUNK-Build
> Compaction shouldn't be allowed on non-ACID table
> -
>
> Key: HIVE-13201
> URL: https://issues.apache.org/jira/browse/HIVE-13201
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13201.1.patch
>
>
> Looks like compaction is allowed on non-ACID table, although that's of no
> sense and does nothing. Moreover the compaction request will be enqueued into
> COMPACTION_QUEUE metastore table, which brings unnecessary overhead.
> We should prevent compaction commands being allowed on non-ACID tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13201) Compaction shouldn't be allowed on non-ACID table
[
https://issues.apache.org/jira/browse/HIVE-13201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178435#comment-15178435
]
Wei Zheng commented on HIVE-13201:
--
Create a non-ACID table
{code}
hive> create table nonacid (a int);
OK
Time taken: 0.939 seconds
hive> desc formatted nonacid;
OK
# col_name data_type comment
a int
# Detailed Table Information
Database: default
Owner: hive
CreateTime: Thu Mar 03 19:22:56 UTC 2016
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location:
hdfs://jvaria-hive-1-4.novalocal:8020/apps/hive/warehouse/nonacid
Table Type: MANAGED_TABLE
Table Parameters:
transient_lastDdlTime 1457032976
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat:org.apache.hadoop.mapred.TextInputFormat
OutputFormat:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed: No
Num Buckets:-1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format1
Time taken: 0.26 seconds, Fetched: 26 row(s)
hive> alter table nonacid compact 'major';
Compaction enqueued.
OK
Time taken: 0.098 seconds
{code}
On the metastore side, the compaction request is enqueued
{code}
mysql> select * from COMPACTION_QUEUE;
+---+-+--+--+--+-+--+---+---+---+--+--+
| CQ_ID | CQ_DATABASE | CQ_TABLE | CQ_PARTITION | CQ_STATE | CQ_TYPE |
CQ_WORKER_ID | CQ_START | CQ_RUN_AS | CQ_HIGHEST_TXN_ID | CQ_META_INFO |
CQ_HADOOP_JOB_ID |
+---+-+--+--+--+-+--+---+---+---+--+--+
| 5 | default | nonacid | NULL | r| a | NULL
| 1457033021000 | hive | NULL | NULL | NULL
|
+---+-+--+--+--+-+--+---+---+---+--+--+
1 row in set (0.00 sec)
{code}
> Compaction shouldn't be allowed on non-ACID table
> -
>
> Key: HIVE-13201
> URL: https://issues.apache.org/jira/browse/HIVE-13201
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Looks like compaction is allowed on non-ACID table, although that's of no
> sense and does nothing. Moreover the compaction request will be enqueued into
> COMPACTION_QUEUE metastore table, which brings unnecessary overhead.
> We should prevent compaction commands being allowed on non-ACID tables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
