[jira] [Work logged] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?focusedWorklogId=618881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618881 ] ASF GitHub Bot logged work on HIVE-25306: - Author: ASF GitHub Bot Created on: 06/Jul/21 05:31 Start Date: 06/Jul/21 05:31 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2445: URL: https://github.com/apache/hive/pull/2445#issuecomment-874472306 @zabetak Could you please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 618881) Time Spent: 20m (was: 10m) > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618872
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 04:26
Start Date: 06/Jul/21 04:26
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664226219
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work started] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25306 started by Ashish Sharma. > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25307) Hive Server 2 crashes when Thrift library encounters particular security protocol issue
[ https://issues.apache.org/jira/browse/HIVE-25307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17375104#comment-17375104 ] Matt McCline commented on HIVE-25307: - Work-in-progress patch attached. Next step: create a pull request. FYI: [~ashish-kumar-sharma] [~sankarh] > Hive Server 2 crashes when Thrift library encounters particular security > protocol issue > --- > > Key: HIVE-25307 > URL: https://issues.apache.org/jira/browse/HIVE-25307 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: hive-thrift-fix-01-3_1.patch > > > A RuntimeException is thrown by the Thrift library that causes Hive Server 2 > to crash on our customer's machine. If you Google this the exception has been > reported a couple of times over the years but not fixed. A blog (see > references below) says it is an occasional security protocol issue between > Hive Server 2 and a proxy like a Gateway. > One challenge is the Thrift TTransportFactory getTransport method declaration > throws no Exceptions hence the likely choice of RuntimeException. But that > Exception is fatal to Hive Server 2. > The proposed fix is a work around that catches RuntimeException in Hive > Server 2, saves the Exception cause in a dummy TTransport object, and throws > the cause when TTransport's open method is called later. > > ExceptionClassName: > java.lang.RuntimeException > ExceptionStackTrace: > java.lang.RuntimeException: > org.apache.thrift.transport.TSaslTransportException: No data or no sasl data > in the stream > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no > sasl data in the stream > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 10 more > > References: > [Hive server 2 thrift error - Cloudera Community - > 34293|https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/34293] > Eric Lin blog "“NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 > LOG" > [HIVE-12754] AuthTypes.NONE cause exception after HS2 start - ASF JIRA > (apache.org) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25307) Hive Server 2 crashes when Thrift library encounters particular security protocol issue
[ https://issues.apache.org/jira/browse/HIVE-25307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-25307: Attachment: hive-thrift-fix-01-3_1.patch > Hive Server 2 crashes when Thrift library encounters particular security > protocol issue > --- > > Key: HIVE-25307 > URL: https://issues.apache.org/jira/browse/HIVE-25307 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: hive-thrift-fix-01-3_1.patch > > > A RuntimeException is thrown by the Thrift library that causes Hive Server 2 > to crash on our customer's machine. If you Google this the exception has been > reported a couple of times over the years but not fixed. A blog (see > references below) says it is an occasional security protocol issue between > Hive Server 2 and a proxy like a Gateway. > One challenge is the Thrift TTransportFactory getTransport method declaration > throws no Exceptions hence the likely choice of RuntimeException. But that > Exception is fatal to Hive Server 2. > The proposed fix is a work around that catches RuntimeException in Hive > Server 2, saves the Exception cause in a dummy TTransport object, and throws > the cause when TTransport's open method is called later. > > ExceptionClassName: > java.lang.RuntimeException > ExceptionStackTrace: > java.lang.RuntimeException: > org.apache.thrift.transport.TSaslTransportException: No data or no sasl data > in the stream > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no > sasl data in the stream > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 10 more > > References: > [Hive server 2 thrift error - Cloudera Community - > 34293|https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/34293] > Eric Lin blog "“NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 > LOG" > [HIVE-12754] AuthTypes.NONE cause exception after HS2 start - ASF JIRA > (apache.org) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25307) Hive Server 2 crashes when Thrift library encounters particular security protocol issue
[ https://issues.apache.org/jira/browse/HIVE-25307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-25307: --- > Hive Server 2 crashes when Thrift library encounters particular security > protocol issue > --- > > Key: HIVE-25307 > URL: https://issues.apache.org/jira/browse/HIVE-25307 > Project: Hive > Issue Type: Bug >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > > A RuntimeException is thrown by the Thrift library that causes Hive Server 2 > to crash on our customer's machine. If you Google this the exception has been > reported a couple of times over the years but not fixed. A blog (see > references below) says it is an occasional security protocol issue between > Hive Server 2 and a proxy like a Gateway. > One challenge is the Thrift TTransportFactory getTransport method declaration > throws no Exceptions hence the likely choice of RuntimeException. But that > Exception is fatal to Hive Server 2. > The proposed fix is a work around that catches RuntimeException in Hive > Server 2, saves the Exception cause in a dummy TTransport object, and throws > the cause when TTransport's open method is called later. > > ExceptionClassName: > java.lang.RuntimeException > ExceptionStackTrace: > java.lang.RuntimeException: > org.apache.thrift.transport.TSaslTransportException: No data or no sasl data > in the stream > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710) > at > org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no > sasl data in the stream > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326) > at > org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) > at > org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) > ... 10 more > > References: > [Hive server 2 thrift error - Cloudera Community - > 34293|https://community.cloudera.com/t5/Support-Questions/Hive-server-2-thrift-error/td-p/34293] > Eric Lin blog "“NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 > LOG" > [HIVE-12754] AuthTypes.NONE cause exception after HS2 start - ASF JIRA > (apache.org) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618820
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:59
Start Date: 06/Jul/21 00:59
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664172062
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -1090,6 +1172,26 @@ Long bootStrapDump(Path dumpRoot, DumpMetaData dmd, Path
cmRoot, Hive hiveDb)
}
}
+ private void unsetReplFailoverEnabled(Hive hiveDb, Database db) throws
HiveException {
Review comment:
How will this fxn signature know for which db it has to unset this prop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 618820)
Time Spent: 2h 50m (was: 2h 40m)
> Handle failover case during Repl Dump
> -
>
> Key: HIVE-24918
> URL: https://issues.apache.org/jira/browse/HIVE-24918
> Project: Hive
> Issue Type: New Feature
>Reporter: Haymant Mangla
>Assignee: Haymant Mangla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> To handle:
> a) Whenever user wants to go ahead with failover, during the next or
> subsequent repl dump operation upon confirming that there are no pending open
> transaction events, It should create a _failover_ready marker file in the
> dump dir. This marker file would contain scheduled query name
> that has generated this dump.
> b) Skip next repl dump instances once we have the marker file placed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618816=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618816
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:53
Start Date: 06/Jul/21 00:53
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664170761
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618812=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618812
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:51
Start Date: 06/Jul/21 00:51
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664170277
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618811
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:49
Start Date: 06/Jul/21 00:49
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664169753
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618810=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618810
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:41
Start Date: 06/Jul/21 00:41
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664168191
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618809
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 06/Jul/21 00:39
Start Date: 06/Jul/21 00:39
Worklog Time Spent: 10m
Work Description: hmangla98 commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664167672
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
+ Case 1) Txns that belong to different db or have not acquired HIVE LOCKS:
These txns would be caught in
+ _failovermetadata file.
+ Case 2) Txns that belong to db under replication: These txns would be
aborted as part of dump operation.
+ */
+// Open 3 txns for Database which is not under replication
+int numTxnsForSecDb = 3;
+List txnsForSecDb = openTxns(numTxnsForSecDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of secondary db for 3 txns
+// t1=5 and t2=5
+Map tablesInSecDb = new HashMap<>();
+tablesInSecDb.put("t1", (long) numTxnsForSecDb);
+tablesInSecDb.put("t2", (long) numTxnsForSecDb);
+List lockIdsForSecDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName + "_extra",
+tablesInSecDb, txnHandler, txnsForSecDb, primaryConf);
+
+//Open 2 txns for Primary Db
+int numTxnsForPrimaryDb = 2;
+List txnsForPrimaryDb = openTxns(numTxnsForPrimaryDb, txnHandler,
primaryConf);
+
+// Allocate write ids for both tables of primary db for 2 txns
+// t1=5 and t2=5
+Map tablesInPrimaryDb = new HashMap<>();
+tablesInPrimaryDb.put("t1", (long) numTxnsForPrimaryDb + 1);
+tablesInPrimaryDb.put("t2", (long) numTxnsForPrimaryDb + 2);
+List lockIdsForPrimaryDb =
allocateWriteIdsForTablesAndAquireLocks(primaryDbName,
+tablesInPrimaryDb, txnHandler, txnsForPrimaryDb, primaryConf);
+
+//Open 1 txn with no hive locks acquired
+List txnsWithNoLocks = openTxns(1, txnHandler, primaryConf);
+
+dumpData = primary.dump(primaryDbName, failoverConfigs);
+
+fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+dumpPath = new Path(dumpData.dumpLocation, ReplUtils.REPL_HIVE_BASE_DIR);
+assertTrue(fs.exists(new Path(dumpPath, DUMP_ACKNOWLEDGEMENT.toString(;
+assertTrue(fs.exists(new Path(dumpPath,
FailoverMetaData.FAILOVER_METADATA)));
+assertTrue(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertTrue(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+FailoverMetaData failoverMD = new FailoverMetaData(dumpPath, conf);
+
+List openTxns = failoverMD.getOpenTxns();
+List txnsAborted = failoverMD.getAbortedTxns();
+assertTrue(txnsAborted.size() == 2);
+
[jira] [Work logged] (HIVE-24918) Handle failover case during Repl Dump
[
https://issues.apache.org/jira/browse/HIVE-24918?focusedWorklogId=618795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618795
]
ASF GitHub Bot logged work on HIVE-24918:
-
Author: ASF GitHub Bot
Created on: 05/Jul/21 20:38
Start Date: 05/Jul/21 20:38
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2121:
URL: https://github.com/apache/hive/pull/2121#discussion_r664101074
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into t1 values(1)")
+.run("insert into t2 partition(name='Bob') values(11)")
+.run("insert into t2 partition(name='Carl') values(10)");
+
+/**Open transactions can be of two types:
Review comment:
nit:
/* Open transactions can be of two types:
Case 1) Txns that have not acquired HIVE LOCKS or they belong to
different db: These txns would be captured in
_failovermetadata file inside dump directory.
Case 2) Txns that have acquired HIVE LOCKS and belong to db under
replication: These txns would be aborted by hive
as a part of dump operation.
*/
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java
##
@@ -138,6 +141,226 @@ public void tearDown() throws Throwable {
primary.run("drop database if exists " + primaryDbName + "_extra cascade");
}
+ @Test
+ public void testFailoverDuringDump() throws Throwable {
+HiveConf primaryConf = primary.getConf();
+TxnStore txnHandler = TxnUtils.getTxnStore(primary.getConf());
+WarehouseInstance.Tuple dumpData = null;
+List failoverConfigs = Arrays.asList("'" +
HiveConf.ConfVars.HIVE_REPL_FAILOVER_START + "'='true'");
+dumpData = primary.run("use " + primaryDbName)
+.run("create table t1 (id int) clustered by(id) into 3 buckets
stored as orc " +
+"tblproperties (\"transactional\"=\"true\")")
+.run("create table t2 (rank int) partitioned by (name string)
tblproperties(\"transactional\"=\"true\", " +
+"\"transactional_properties\"=\"insert_only\")")
+.dump(primaryDbName, failoverConfigs);
+
+//This dump is not failover ready as target db can be used for replication
only after first incremental load.
+FileSystem fs = new Path(dumpData.dumpLocation).getFileSystem(conf);
+Path dumpPath = new Path(dumpData.dumpLocation,
ReplUtils.REPL_HIVE_BASE_DIR);
+assertFalse(fs.exists(new Path(dumpPath,
ReplAck.FAILOVER_READY_MARKER.toString(;
+
assertFalse(MetaStoreUtils.isDbBeingFailedOver(primary.getDatabase(primaryDbName)));
+
+replica.load(replicatedDbName, primaryDbName)
+.run("use " + replicatedDbName)
+.run("show tables")
+.verifyResults(new String[]{"t1", "t2"})
+.run("repl status " + replicatedDbName)
+.verifyResult(dumpData.lastReplicationId);
+
+primary.run("use " + primaryDbName)
+.run("insert into
[jira] [Work logged] (HIVE-25255) Support ALTER TABLE REPLACE COLUMNS for Iceberg
[
https://issues.apache.org/jira/browse/HIVE-25255?focusedWorklogId=618674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618674
]
ASF GitHub Bot logged work on HIVE-25255:
-
Author: ASF GitHub Bot
Created on: 05/Jul/21 14:32
Start Date: 05/Jul/21 14:32
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2418:
URL: https://github.com/apache/hive/pull/2418#discussion_r663975836
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+for (FieldSchema first : minuendCollection) {
+ boolean found = false;
+ for (FieldSchema second : subtrahendCollection) {
+if (first.getName().equals(second.getName())) {
Review comment:
I didn't use it because `name` should never be null, unlike comment (and
maybe type?). Anyways, I've changed it to use Objects.equals to be safe and for
symmetry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 618674)
Time Spent: 2.5h (was: 2h 20m)
> Support ALTER TABLE REPLACE COLUMNS for Iceberg
> ---
>
> Key: HIVE-25255
> URL: https://issues.apache.org/jira/browse/HIVE-25255
> Project: Hive
> Issue Type: New Feature
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-25255) Support ALTER TABLE REPLACE COLUMNS for Iceberg
[
https://issues.apache.org/jira/browse/HIVE-25255?focusedWorklogId=618673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618673
]
ASF GitHub Bot logged work on HIVE-25255:
-
Author: ASF GitHub Bot
Created on: 05/Jul/21 14:31
Start Date: 05/Jul/21 14:31
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2418:
URL: https://github.com/apache/hive/pull/2418#discussion_r663975175
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java
##
@@ -474,6 +480,43 @@ private static PartitionSpec spec(Configuration
configuration, Schema schema, Pr
}
}
+ private void handleReplaceColumns(org.apache.hadoop.hive.metastore.api.Table
hmsTable) throws MetaException {
+HiveSchemaUtil.SchemaDifference schemaDifference =
HiveSchemaUtil.getSchemaDiff(hmsTable.getSd().getCols(),
+HiveSchemaUtil.convert(icebergTable.schema()), true);
+if (!schemaDifference.isEmpty()) {
+ updateSchema = icebergTable.updateSchema();
+} else {
+ // we should get here if the user restated the exactly the existing
columns in the REPLACE COLUMNS command
+ LOG.info("Found no difference between new and old schema for ALTER TABLE
REPLACE COLUMNS for" +
+ " table: {}. There will be no Iceberg commit.",
hmsTable.getTableName());
Review comment:
Done
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+for (FieldSchema first : minuendCollection) {
+ boolean found = false;
+ for (FieldSchema second : subtrahendCollection) {
+if (first.getName().equals(second.getName())) {
+ found = true;
+ if (!Objects.equals(first.getType(), second.getType())) {
+difference.typeChanged(first);
+ }
+ if (!Objects.equals(first.getComment(), second.getComment())) {
+difference.commentChanged(first);
+ }
+}
+ }
+ if (!found) {
+difference.missingFromSecond(first);
+ }
+}
+
+if (bothDirections) {
+ SchemaDifference otherWay = getSchemaDiff(subtrahendCollection,
minuendCollection, false);
+ otherWay.missingFromSecond().forEach(difference::missingFromFirst);
+}
-return minuendCollection.stream()
-.filter(fs ->
!subtrahendWithoutComment.contains(unsetCommentFunc.apply(fs))).collect(Collectors.toList());
+return difference;
}
+ public static class SchemaDifference {
+private final List missingFromFirst = new ArrayList<>();
+private final List missingFromSecond = new ArrayList<>();
+private final List typeChanged = new ArrayList<>();
+private final List commentChanged = new ArrayList<>();
+
+public List missingFromFirst() {
Review comment:
Good point, renamed the methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 618673)
Time Spent: 2h 20m (was: 2h 10m)
> Support ALTER TABLE REPLACE COLUMNS for Iceberg
> ---
>
> Key: HIVE-25255
> URL:
[jira] [Work logged] (HIVE-25255) Support ALTER TABLE REPLACE COLUMNS for Iceberg
[
https://issues.apache.org/jira/browse/HIVE-25255?focusedWorklogId=618643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618643
]
ASF GitHub Bot logged work on HIVE-25255:
-
Author: ASF GitHub Bot
Created on: 05/Jul/21 13:20
Start Date: 05/Jul/21 13:20
Worklog Time Spent: 10m
Work Description: szlta commented on a change in pull request #2418:
URL: https://github.com/apache/hive/pull/2418#discussion_r663922844
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+for (FieldSchema first : minuendCollection) {
+ boolean found = false;
+ for (FieldSchema second : subtrahendCollection) {
+if (first.getName().equals(second.getName())) {
+ found = true;
+ if (!Objects.equals(first.getType(), second.getType())) {
+difference.typeChanged(first);
+ }
+ if (!Objects.equals(first.getComment(), second.getComment())) {
+difference.commentChanged(first);
+ }
+}
+ }
+ if (!found) {
+difference.missingFromSecond(first);
+ }
+}
+
+if (bothDirections) {
+ SchemaDifference otherWay = getSchemaDiff(subtrahendCollection,
minuendCollection, false);
+ otherWay.missingFromSecond().forEach(difference::missingFromFirst);
+}
-return minuendCollection.stream()
-.filter(fs ->
!subtrahendWithoutComment.contains(unsetCommentFunc.apply(fs))).collect(Collectors.toList());
+return difference;
}
+ public static class SchemaDifference {
+private final List missingFromFirst = new ArrayList<>();
+private final List missingFromSecond = new ArrayList<>();
+private final List typeChanged = new ArrayList<>();
+private final List commentChanged = new ArrayList<>();
+
+public List missingFromFirst() {
Review comment:
I'd rather avoid having an overload for this method that has a very
different behavior (get vs add), maybe you could rename either set of these
methods to be more descriptive.
##
File path:
iceberg/iceberg-catalog/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
##
@@ -138,22 +137,88 @@ public static Type convert(TypeInfo typeInfo) {
}
/**
- * Produces the difference of two FieldSchema lists by only taking into
account the field name and type.
+ * Returns a SchemaDifference containing those fields which are present in
only one of the collections, as well as
+ * those fields which are present in both (in terms of the name) but their
type or comment has changed.
* @param minuendCollection Collection of fields to subtract from
* @param subtrahendCollection Collection of fields to subtract
- * @return the result list of difference
+ * @param bothDirections Whether or not to compute the missing fields from
the minuendCollection as well
+ * @return the difference between the two schemas
*/
- public static Collection schemaDifference(
- Collection minuendCollection, Collection
subtrahendCollection) {
+ public static SchemaDifference getSchemaDiff(Collection
minuendCollection,
+ Collection
subtrahendCollection, boolean bothDirections) {
+SchemaDifference difference = new SchemaDifference();
-Function unsetCommentFunc = fs -> new
FieldSchema(fs.getName(), fs.getType(), null);
-Set subtrahendWithoutComment =
-
subtrahendCollection.stream().map(unsetCommentFunc).collect(Collectors.toSet());
+
[jira] [Updated] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25306: -- Labels: pull-request-available (was: ) > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?focusedWorklogId=618639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618639 ] ASF GitHub Bot logged work on HIVE-25306: - Author: ASF GitHub Bot Created on: 05/Jul/21 13:08 Start Date: 05/Jul/21 13:08 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma opened a new pull request #2445: URL: https://github.com/apache/hive/pull/2445 ### What changes were proposed in this pull request? Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to convert the date/timpstamp from int,string,char etc to Date or Timestamp. Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date like "1992-13-12" is converted to "2000-01-12", Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like "1992-13-12" is not be converted instead NULL is return. ### Why are the changes needed? ResolverStyle.LENIENT to ResolverStyle.STRICT ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? UTs and QTs added as part of PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 618639) Remaining Estimate: 0h Time Spent: 10m > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-25306: - Description: Description - Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to convert the date/timpstamp from int,string,char etc to Date or Timestamp. Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date like "1992-13-12" is converted to "2000-01-12", Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like "1992-13-12" is not be converted instead NULL is return. was: Description - Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to convert the date/timpstamp from int,string,char etc to Date or Timestamp. Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date like "1992-13-12" is converted to "2000-01-12", Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like "1992-13-12" is not be converted instead null is return. > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead NULL is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-25306) Move Date and Timestamp parsing from ResolverStyle.LENIENT to ResolverStyle.STRICT
[ https://issues.apache.org/jira/browse/HIVE-25306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma reassigned HIVE-25306: > Move Date and Timestamp parsing from ResolverStyle.LENIENT to > ResolverStyle.STRICT > -- > > Key: HIVE-25306 > URL: https://issues.apache.org/jira/browse/HIVE-25306 > Project: Hive > Issue Type: Bug > Components: Query Planning, UDF >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Major > > Description - > Currently Date.java and Timestamp.java use DateTimeFormatter for parsing to > convert the date/timpstamp from int,string,char etc to Date or Timestamp. > Default DateTimeFormatter which use ResolverStyle.LENIENT which mean date > like "1992-13-12" is converted to "2000-01-12", > Moving DateTimeFormatter which use ResolverStyle.STRICT which mean date like > "1992-13-12" is not be converted instead null is return. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-25056) cast ('000-00-00 00:00:00' as timestamp/datetime) results in wrong conversion
[
https://issues.apache.org/jira/browse/HIVE-25056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17374698#comment-17374698
]
Ashish Sharma commented on HIVE-25056:
--
[~anuragshekhar] are you actively working on the ticket. if not can I take it
over?
> cast ('000-00-00 00:00:00' as timestamp/datetime) results in wrong conversion
> --
>
> Key: HIVE-25056
> URL: https://issues.apache.org/jira/browse/HIVE-25056
> Project: Hive
> Issue Type: Bug
>Reporter: Anurag Shekhar
>Assignee: Anurag Shekhar
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> select cast ('-00-00' as date) , cast ('000-00-00 00:00:00' as timestamp)
> +--+---+
> |_c0|_c1|
> +--+---+
> |0002-11-30|0002-11-30 00:00:00.0|
> +--+---+
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-25252) All new compaction metrics should be lower case
[ https://issues.apache.org/jira/browse/HIVE-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage resolved HIVE-25252. -- Fix Version/s: 4.0.0 Resolution: Fixed Committed to master branch. Thanks for your contribution [~asinkovits]! > All new compaction metrics should be lower case > --- > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-25252) All new compaction metrics should be lower case
[ https://issues.apache.org/jira/browse/HIVE-25252?focusedWorklogId=618572=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-618572 ] ASF GitHub Bot logged work on HIVE-25252: - Author: ASF GitHub Bot Created on: 05/Jul/21 08:16 Start Date: 05/Jul/21 08:16 Worklog Time Spent: 10m Work Description: klcopp merged pull request #2422: URL: https://github.com/apache/hive/pull/2422 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 618572) Time Spent: 50m (was: 40m) > All new compaction metrics should be lower case > --- > > Key: HIVE-25252 > URL: https://issues.apache.org/jira/browse/HIVE-25252 > Project: Hive > Issue Type: Sub-task >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > E.g: > compaction_worker_cycle_MINOR -> compaction_worker_cycle_minor > compaction_worker_cycle_MAJOR -> compaction_worker_cycle_major > compaction_cleaner_cycle_MINOR -> compaction_cleaner_cycle_minor > compaction_cleaner_cycle_MAJOR -> compaction_cleaner_cycle_major -- This message was sent by Atlassian Jira (v8.3.4#803005)
