[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579091#comment-16579091 ] Sergey Shelukhin commented on HIVE-20343: - Probably default-to-acid thing is messing with this somehow. > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Priority: Major > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL >| NULL | > | Database: | default >| NULL | > | OwnerType:| USER >| NULL | > | Owner:| hive >| NULL | > | CreateTime: | Wed Aug 08 21:35:15 UTC 2018 >| NULL | > | LastAccessTime: | UNKNOWN >| NULL | > | Retention:| 0 >| NULL | > | Location: | > hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly > | NULL | > | Table Type: | MANAGED_TABLE >| NULL | > | Table Parameters: | NULL >| NULL | > | | COLUMN_STATS_ACCURATE >| {}| > | | bucketing_version >| 2 | > | | numFiles >| 1 | > | | numRows >| 1 | > | | rawDataSize >| 0 | > | | totalSize >| 754 | > | | transactional >| true | > | | transactional_properties >| default | > | | transient_lastDdlTime >| 1533764115| > | | NULL >| NULL | > | # Storage Information | NULL >| NULL | > | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde >| NULL | > | InputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | > | Compressed: | No >| NULL | > | Num Buckets: | -1 >| NULL | > | Bucket Columns: | []
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579135#comment-16579135 ] Sergey Shelukhin commented on HIVE-20343: - Confirmed after some testing... if neither of the default flags is set, the incorrect table is created with txn props but no txn. If mm is on by default, everything is fine. If ACID is on by default, the conversion code treats the table as non-txn (similar to the first case) and then replaces txn=false, props=insert_only with full ACID. I think the solution is to outlaw this syntax.. if properties are specified explicitly, the flag should also be specified, it's not good for us to guess what user wanted based on partial specification, given that there are 4 combinations of default options. Location is a separate issue. > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL >| NULL | > | Database: | default >| NULL | > | OwnerType:| USER >| NULL | > | Owner:| hive >| NULL | > | CreateTime: | Wed Aug 08 21:35:15 UTC 2018 >| NULL | > | LastAccessTime: | UNKNOWN >| NULL | > | Retention:| 0 >| NULL | > | Location: | > hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly > | NULL | > | Table Type: | MANAGED_TABLE >| NULL | > | Table Parameters: | NULL >| NULL | > | | COLUMN_STATS_ACCURATE >| {}| > | | bucketing_version >| 2 | > | | numFiles >| 1 | > | | numRows >| 1 | > | | rawDataSize >| 0 | > | | totalSize >| 754 | > | | transactional >| true | > | | transactional_properties >| default | > | | transient_lastDdlTime >| 1533764115| > | | NULL >| NULL | > | # Storage Information | NULL >| NULL | > | SerDe Libr
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579147#comment-16579147 ] Sergey Shelukhin commented on HIVE-20343: - [~ekoifman] can you take a look? a small patch > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20343.patch > > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL >| NULL | > | Database: | default >| NULL | > | OwnerType:| USER >| NULL | > | Owner:| hive >| NULL | > | CreateTime: | Wed Aug 08 21:35:15 UTC 2018 >| NULL | > | LastAccessTime: | UNKNOWN >| NULL | > | Retention:| 0 >| NULL | > | Location: | > hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly > | NULL | > | Table Type: | MANAGED_TABLE >| NULL | > | Table Parameters: | NULL >| NULL | > | | COLUMN_STATS_ACCURATE >| {}| > | | bucketing_version >| 2 | > | | numFiles >| 1 | > | | numRows >| 1 | > | | rawDataSize >| 0 | > | | totalSize >| 754 | > | | transactional >| true | > | | transactional_properties >| default | > | | transient_lastDdlTime >| 1533764115| > | | NULL >| NULL | > | # Storage Information | NULL >| NULL | > | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde >| NULL | > | InputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | > | Compressed: | No >| NULL | > | Num Buckets: | -1 >| NULL
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579709#comment-16579709 ] Hive QA commented on HIVE-20343: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 15s{color} | {color:blue} ql in master has 2306 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13210/dev-support/hive-personality.sh | | git revision | master / bfa69e1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13210/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20343.patch > > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579751#comment-16579751 ] Hive QA commented on HIVE-20343: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935465/HIVE-20343.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 14878 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13210/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13210/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13210/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12935465 - PreCommit-HIVE-Build > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20343.patch > > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL >| NULL | > | Database: | default >| NULL | > | OwnerType:| USER >| NULL | > | Owner:| hive >| NULL | > | CreateTime: | Wed Aug 08 21:35:15 UTC 2018 >| NULL | > | LastAccessTime: | UNKNOWN >| NULL | > | Retention:| 0 >| NULL | > | Location: | > hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly > | NULL | > | Table Type: | MANAGED_TABLE >| NULL | > | Table Parameters: | NULL >| NULL | > | | COLUMN_STATS_ACCURATE >| {}| > | | bucketing_version >| 2 | > | | numFiles >| 1 | > | | numRows >| 1 | > | | rawDataSize >| 0 | > | | totalSize >| 754 | > | | transactional >| true | > | | transactional_properties >| default | > | | transient_lastDdlTime
[jira] [Commented] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties
[ https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580367#comment-16580367 ] Eugene Koifman commented on HIVE-20343: --- +1 > Hive 3: CTAS does not respect transactional_properties > -- > > Key: HIVE-20343 > URL: https://issues.apache.org/jira/browse/HIVE-20343 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 > Environment: hive-3 >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20343.patch > > > Steps to reproduce: > {code} > create table ctasexampleinsertonly stored as orc TBLPROPERTIES > ("transactional_properties"="insert_only") as select * from testtable limit 1; > {code} > look for transactional_properties which is 'default' not the expected > "insert_only" > {code} > describe formatted ctasexampleinsertonly > > +---++---+ > | col_name| data_type >|comment| > +---++---+ > | # col_name| data_type >| comment | > | name | varchar(8) >| | > | time | double >| | > | | NULL >| NULL | > | # Detailed Table Information | NULL >| NULL | > | Database: | default >| NULL | > | OwnerType:| USER >| NULL | > | Owner:| hive >| NULL | > | CreateTime: | Wed Aug 08 21:35:15 UTC 2018 >| NULL | > | LastAccessTime: | UNKNOWN >| NULL | > | Retention:| 0 >| NULL | > | Location: | > hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly > | NULL | > | Table Type: | MANAGED_TABLE >| NULL | > | Table Parameters: | NULL >| NULL | > | | COLUMN_STATS_ACCURATE >| {}| > | | bucketing_version >| 2 | > | | numFiles >| 1 | > | | numRows >| 1 | > | | rawDataSize >| 0 | > | | totalSize >| 754 | > | | transactional >| true | > | | transactional_properties >| default | > | | transient_lastDdlTime >| 1533764115| > | | NULL >| NULL | > | # Storage Information | NULL >| NULL | > | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde >| NULL | > | InputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL | > | OutputFormat: | > org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat | NULL | > | Compressed: | No >| NULL | > | Num Buckets: | -1 >| NULL | > | Bucket Columns: