[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579091#comment-16579091
]
Sergey Shelukhin commented on HIVE-20343:
-
Probably default-to-acid thing is messing with this somehow.
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns: | []
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579135#comment-16579135
]
Sergey Shelukhin commented on HIVE-20343:
-
Confirmed after some testing... if neither of the default flags is set, the
incorrect table is created with txn props but no txn.
If mm is on by default, everything is fine.
If ACID is on by default, the conversion code treats the table as non-txn
(similar to the first case) and then replaces txn=false, props=insert_only with
full ACID.
I think the solution is to outlaw this syntax.. if properties are specified
explicitly, the flag should also be specified, it's not good for us to guess
what user wanted based on partial specification, given that there are 4
combinations of default options.
Location is a separate issue.
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Libr
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579147#comment-16579147
]
Sergey Shelukhin commented on HIVE-20343:
-
[~ekoifman] can you take a look? a small patch
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579709#comment-16579709
]
Hive QA commented on HIVE-20343:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-13210/dev-support/hive-personality.sh
|
| git revision | master / bfa69e1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-13210/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579751#comment-16579751
]
Hive QA commented on HIVE-20343:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12935465/HIVE-20343.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14878 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/13210/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13210/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12935465 - PreCommit-HIVE-Build
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580367#comment-16580367
]
Eugene Koifman commented on HIVE-20343:
---
+1
> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc TBLPROPERTIES
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected
> "insert_only"
> {code}
> describe formatted ctasexampleinsertonly
>
> +---++---+
> | col_name| data_type
>|comment|
> +---++---+
> | # col_name| data_type
>| comment |
> | name | varchar(8)
>| |
> | time | double
>| |
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| hive
>| NULL |
> | CreateTime: | Wed Aug 08 21:35:15 UTC 2018
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
> | NULL |
> | Table Type: | MANAGED_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {}|
> | | bucketing_version
>| 2 |
> | | numFiles
>| 1 |
> | | numRows
>| 1 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 754 |
> | | transactional
>| true |
> | | transactional_properties
>| default |
> | | transient_lastDdlTime
>| 1533764115|
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde
>| NULL |
> | InputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL |
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL |
> | Compressed: | No
>| NULL |
> | Num Buckets: | -1
>| NULL |
> | Bucket Columns:
