[jira] [Assigned] (HIVE-18741) Add support for Import into Acid table
[ https://issues.apache.org/jira/browse/HIVE-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18741: - > Add support for Import into Acid table > -- > > Key: HIVE-18741 > URL: https://issues.apache.org/jira/browse/HIVE-18741 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > This should follow Load Data approach (or use load data directly) > Note that import supports partition spec -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18740) Add support for Export from partitioned Acid table
[ https://issues.apache.org/jira/browse/HIVE-18740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18740: - > Add support for Export from partitioned Acid table > -- > > Key: HIVE-18740 > URL: https://issues.apache.org/jira/browse/HIVE-18740 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > figure out how to translate (partial) partition spec from Export command into > a "where" clause -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18739) Add support for Export from unpartitioned Acid table
[ https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18739: - > Add support for Export from unpartitioned Acid table > > > Key: HIVE-18739 > URL: https://issues.apache.org/jira/browse/HIVE-18739 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18735:
--
Description:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
Specifying props explicitly does work
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1 TBLPROPERTIES ('transactional'='true');
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
TBLPROPERTIES (
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1518814111')
was:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1518813564')
> {noformat}
> Specifying props explicitly does work
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1 TBLPROPERTIES ('transactional'='true');
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
> TBLPROPERTIES (
> 'transactional'='true',
> 'transactional_properties'='default',
> 'transient_lastDdlTime'='1518814111')
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18735:
--
Description:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
Specifying props explicitly does work
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1 TBLPROPERTIES ('transactional'='true');
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
TBLPROPERTIES (
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1518814111')
{noformat}
was:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
Specifying props explicitly does work
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1 TBLPROPERTIES ('transactional'='true');
show create table T ;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
TBLPROPERTIES (
'transactional'='true',
'transactional_properties'='default',
'transient_lastDdlTime'='1518814111')
> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1518813564')
> {noformat}
> Specifying props explicitly does work
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1 TBLPROPERTIES ('transactional'='true');
> show create table T ;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518814098564/warehouse/t'
> TBLPROPERTIES (
> 'transactional'='true',
>
[jira] [Updated] (HIVE-18735) Create table like loses transactional attribute
[
https://issues.apache.org/jira/browse/HIVE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18735:
--
Description:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
was:
{noformat}
create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
TBLPROPERTIES ('transactional'='true')";
create table T like T1;
show create table T;
CREATE TABLE `T`(
`a` int,
`b` int)
CLUSTERED BY (
a)
INTO 2 BUCKETS
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
TBLPROPERTIES (
'transient_lastDdlTime'='1518813564')
{noformat}
> Create table like loses transactional attribute
> ---
>
> Key: HIVE-18735
> URL: https://issues.apache.org/jira/browse/HIVE-18735
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> {noformat}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('transactional'='true')";
> create table T like T1;
> show create table T;
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
>
> 'file:/Users/ekoifman/IdeaProjects/hive/ql/target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518813536099/warehouse/t'
> TBLPROPERTIES (
> 'transient_lastDdlTime'='1518813564')
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18712) Design HMS Api v2
[ https://issues.apache.org/jira/browse/HIVE-18712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367679#comment-16367679 ] Eugene Koifman commented on HIVE-18712: --- +1 for consistent use of Request/Response objects Thirft is not very well documented - one argument in favor of more common REST > Design HMS Api v2 > - > > Key: HIVE-18712 > URL: https://issues.apache.org/jira/browse/HIVE-18712 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alexander Kolbasov >Priority: Major > > This is an umbrella Jira covering the design of Hive Metastore API v2. > It is supposed to be a placeholder for discussion and design documents. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18685) Add catalogs to metastore
[
https://issues.apache.org/jira/browse/HIVE-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367665#comment-16367665
]
Eugene Koifman commented on HIVE-18685:
---
{quote}9. It is not yet determined what types of cross catalog queries and DML
statements will be supported. The initial version(s) should not do anything
that prevents cross catalog {color:#33}queries or DML statements in the
future{color}
{quote}
Until this is determined, what will guard against a DML that does cross
catalogs? There is nothing in the above discussion about Lock Managers which
don't understand catalog concept.
> Add catalogs to metastore
> -
>
> Key: HIVE-18685
> URL: https://issues.apache.org/jira/browse/HIVE-18685
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Attachments: HMS Catalog Design Doc.pdf
>
>
> SQL supports two levels of namespaces, called in the spec catalogs and
> schemas (with schema being equivalent to Hive's database). I propose to add
> the upper level of catalog. The attached design doc covers the use cases,
> requirements, and brief discussion of how it will be implemented in a
> backwards compatible way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[ https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366513#comment-16366513 ] Eugene Koifman commented on HIVE-18693: --- this won't enqueue any compaction requests so CompactorMR will never run, i.e. it won't clean aborted dirs also, it will not clean relevant entries from txn_components or completed_txn_componentes. The test should look at the directories on the file system and make sure that they are actually deleted. > Snapshot Isolation does not work for Micromanaged table when a insert > transaction is aborted > > > Key: HIVE-18693 > URL: https://issues.apache.org/jira/browse/HIVE-18693 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch > > > TestTxnCommands2#writeBetweenWorkerAndCleaner with minor > changes (changing delete command to insert command) fails on MM table. > Specifically the last SELECT commands returns wrong results. > But this test works fine with full ACID table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted
[ https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366352#comment-16366352 ] Eugene Koifman commented on HIVE-18693: --- Suppose you have 200 txns all of which committed (with this patch). How will Initiator handle all the data in completed_txn_components? > Snapshot Isolation does not work for Micromanaged table when a insert > transaction is aborted > > > Key: HIVE-18693 > URL: https://issues.apache.org/jira/browse/HIVE-18693 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-18693.01.patch > > > TestTxnCommands2#writeBetweenWorkerAndCleaner with minor > changes (changing delete command to insert command) fails on MM table. > Specifically the last SELECT commands returns wrong results. > But this test works fine with full ACID table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366255#comment-16366255
]
Eugene Koifman commented on HIVE-15077:
---
no related failures
[~alangates] could you review please
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files/directories
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Attachment: HIVE-18659.06.patch > add acid version marker to acid files/directories > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch, > HIVE-18659.05.patch, HIVE-18659.06.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files/directories
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Attachment: HIVE-18659.05.patch > add acid version marker to acid files/directories > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch, > HIVE-18659.05.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15077:
--
Affects Version/s: 2.3.0
Status: Patch Available (was: Open)
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15077:
--
Attachment: HIVE-15077.02.patch
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15077:
--
Description:
HIVE-10242 made the acid LM unfair.
In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number is
extLockId).
Then
LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
may look like this (all explicitly listed locks are in Waiting state)
{, SR5 SW3 X4}
So the algorithm will find SR5 in the list and start looking backwards (to the
left).
According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
examined and so SR5 may be granted.
Theoretically, this could cause starvation.
The query that generates the list already has
query.append(" and hl_lock_ext_id <= ").append(extLockId);
but it should use "<" rather than "<=" to exclude the locks being checked from
"locks" list which will make the algorithm look at all locks "in front" of a
given lock.
Here is an example (add to TestDbTxnManager2)
{noformat}
@Test
public void testFairness2() throws Exception {
dropTable(new String[]{"T7"});
CommandProcessorResponse cpr = driver.run("create table if not exists T7 (a
int) partitioned by (p int) stored as orc TBLPROPERTIES
('transactional'='true')");
checkCmdOnDriver(cpr);
checkCmdOnDriver(driver.run("insert into T7 partition(p)
values(1,1),(1,2)"));//create 2 partitions
cpr = driver.compileAndRespond("select a from T7 ");
checkCmdOnDriver(cpr);
txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
HiveTxnManager txnMgr2 =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
swapTxnManager(txnMgr2);
cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
checkCmdOnDriver(cpr);
//tries to get X lock on T7.p=1 and gets Waiting state
LockState lockState = ((DbTxnManager)
txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
List locks = getLocks();
Assert.assertEquals("Unexpected lock count", 4, locks.size());
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", null,
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=1",
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=2",
locks);
checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
locks);
HiveTxnManager txnMgr3 =
TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
swapTxnManager(txnMgr3);
//this should block behind the X lock on T7.p=1
cpr = driver.compileAndRespond("select a from T7");
checkCmdOnDriver(cpr);
txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
locks = getLocks();
Assert.assertEquals("Unexpected lock count", 7, locks.size());
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", null,
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=1",
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=2",
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", null,
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=1",
locks);
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7", "p=2",
locks);
checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
locks);
}
{noformat}
The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
{{select * from T7}} are all acquired while they should block behind the X lock
to be fair.
was:
HIVE-10242 made the acid LM unfair.
In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number is
extLockId).
Then
LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
may look like this (all explicitly listed locks are in Waiting state)
{, SR5 SW3 X4}
So the algorithm will find SR5 in the list and start looking backwards (to the
left).
According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
examined and so SR5 may be granted.
Theoretically, this could cause starvation.
The query that generates the list already has
query.append(" and hl_lock_ext_id <= ").append(extLockId);
but it should use "<" rather than "<=" to exclude the locks being checked from
"locks" list which will make the algorithm look at all locks "in front" of a
given lock.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to
[jira] [Resolved] (HIVE-16964) _orc_acid_version file is missing
[ https://issues.apache.org/jira/browse/HIVE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-16964. --- Resolution: Fixed Fix Version/s: 3.0.0 included in HIVE-18659 > _orc_acid_version file is missing > - > > Key: HIVE-16964 > URL: https://issues.apache.org/jira/browse/HIVE-16964 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 3.0.0 > > > OrcRecordUpdater creates OrcRecordUpdater.ACID_FORMAT in the dir that it > creates - but there is nothing Hive.moveAcidFiles() that copies it final > location. > It doesn't look like CompactorMR even attempts to create it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files/directories
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Summary: add acid version marker to acid files/directories (was: add acid version marker to acid files) > add acid version marker to acid files/directories > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Attachment: HIVE-18659.04.patch > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch, HIVE-18659.04.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18709) Enable Compaction to work on more than one partition per job
[ https://issues.apache.org/jira/browse/HIVE-18709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18709: - > Enable Compaction to work on more than one partition per job > > > Key: HIVE-18709 > URL: https://issues.apache.org/jira/browse/HIVE-18709 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > currently compaction launches 1 MR job per partition that needs to be > compacted. > The number of tasks is equal to the number of buckets in the table (or number > or writers in the 'widest' write). > The number of AMs in a cluster is usually limited to a small percentage of > the nodes. This limits how much compaction can be done in parallel. > Investigate what it would take for a single job to be able to handle multiple > partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18689) restore inheritPerms functionality and extend it to ACID
[ https://issues.apache.org/jira/browse/HIVE-18689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18689: -- Component/s: Transactions > restore inheritPerms functionality and extend it to ACID > > > Key: HIVE-18689 > URL: https://issues.apache.org/jira/browse/HIVE-18689 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18689.patch > > > This functionality was removed for no clear reason (if it doesn't apply to > some use case it can just be disabled). > It's still in use; in fact, it should be extended to ACID table > subdirectories. > This patch restores the functionality with some cleanup (to not access config > everywhere, mostly), disables it by default, and extends it to ACID tables. > There's a coming HDFS feature that will automatically inherit permissions. > When that is shipped in a non-beta version and stabilized a bit, we can > remove this functionality... however I dunno if that is good for other > potential use cases, like non-HDFS file systems that do have a concept of a > directory (Isilon?) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18700) qtests: authorization_update_own_table.q breaks row__id.q
[
https://issues.apache.org/jira/browse/HIVE-18700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362691#comment-16362691
]
Eugene Koifman commented on HIVE-18700:
---
[~kgyrtkirk] can you add details please?
> qtests: authorization_update_own_table.q breaks row__id.q
> -
>
> Key: HIVE-18700
> URL: https://issues.apache.org/jira/browse/HIVE-18700
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Zoltan Haindrich
>Priority: Major
>
> {code}
> mvn install -Pitests install -pl itests/qtest -am -Dtest=TestCliDriver
> -Dqfile=authorization_update_own_table.q,row__id.q
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361500#comment-16361500 ] Eugene Koifman commented on HIVE-18659: --- it's already linked > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-16964) _orc_acid_version file is missing
[
https://issues.apache.org/jira/browse/HIVE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361477#comment-16361477
]
Eugene Koifman edited comment on HIVE-16964 at 2/12/18 9:34 PM:
This is the layout that {{Hive.moveAcidFiles()}} sees. It loses
{{_orc_acid_version}}
{noformat}
HW13943:ql ekoifman$ tree
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
├── 00_0
│ └── _orc_acid_version
└── 01_0
├── _orc_acid_version
└── delta_016_016_
└── bucket_1
3 directories, 3 files
HW13943:ql ekoifman$
{noformat}
was (Author: ekoifman):
This is the layout that {{Hive.moveAcidFiles()}} sees. It looses
{{_orc_acid_version}}
{noformat}
HW13943:ql ekoifman$ tree
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
├── 00_0
│ └── _orc_acid_version
└── 01_0
├── _orc_acid_version
└── delta_016_016_
└── bucket_1
3 directories, 3 files
HW13943:ql ekoifman$
{noformat}
> _orc_acid_version file is missing
> -
>
> Key: HIVE-16964
> URL: https://issues.apache.org/jira/browse/HIVE-16964
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> OrcRecordUpdater creates OrcRecordUpdater.ACID_FORMAT in the dir that it
> creates - but there is nothing Hive.moveAcidFiles() that copies it final
> location.
> It doesn't look like CompactorMR even attempts to create it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16964) _orc_acid_version file is missing
[
https://issues.apache.org/jira/browse/HIVE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361477#comment-16361477
]
Eugene Koifman commented on HIVE-16964:
---
This is the layout that {{Hive.moveAcidFiles()}} sees. It looses
{{_orc_acid_version}}
{noformat}
HW13943:ql ekoifman$ tree
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
target/tmp/org.apache.hadoop.hive.ql.TestTxnCommands-1518470371936/warehouse/nonacidorctbl/.hive-staging_hive_2018-02-12_13-26-08_544_7812499075009504959-1/-ext-1/
├── 00_0
│ └── _orc_acid_version
└── 01_0
├── _orc_acid_version
└── delta_016_016_
└── bucket_1
3 directories, 3 files
HW13943:ql ekoifman$
{noformat}
> _orc_acid_version file is missing
> -
>
> Key: HIVE-16964
> URL: https://issues.apache.org/jira/browse/HIVE-16964
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> OrcRecordUpdater creates OrcRecordUpdater.ACID_FORMAT in the dir that it
> creates - but there is nothing Hive.moveAcidFiles() that copies it final
> location.
> It doesn't look like CompactorMR even attempts to create it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HIVE-16964) _orc_acid_version file is missing
[ https://issues.apache.org/jira/browse/HIVE-16964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reopened HIVE-16964: --- Assignee: Eugene Koifman (was: Steve Yeom) > _orc_acid_version file is missing > - > > Key: HIVE-16964 > URL: https://issues.apache.org/jira/browse/HIVE-16964 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > OrcRecordUpdater creates OrcRecordUpdater.ACID_FORMAT in the dir that it > creates - but there is nothing Hive.moveAcidFiles() that copies it final > location. > It doesn't look like CompactorMR even attempts to create it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361463#comment-16361463 ] Eugene Koifman commented on HIVE-18659: --- _orc_acid_version is not written consistently. in fact is't only there from streaming ingest use case. Also, it's written into the partition dir, which can have base/deltas written by different versions of hive. I agree it's cheaper but putting into the file itself is more reliable especially since people tend to try to move acid files around. > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18678) fix exim for MM tables and reinstante the test
[ https://issues.apache.org/jira/browse/HIVE-18678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361212#comment-16361212 ] Eugene Koifman commented on HIVE-18678: --- +1 > fix exim for MM tables and reinstante the test > -- > > Key: HIVE-18678 > URL: https://issues.apache.org/jira/browse/HIVE-18678 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18678.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361182#comment-16361182 ] Eugene Koifman commented on HIVE-18575: --- +1 I left a couple nits on RB > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.01.patch, HIVE-18575.02.patch, > HIVE-18575.03.patch, HIVE-18575.04.patch, HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15077:
--
Description:
HIVE-10242 made the acid LM unfair.
In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number is
extLockId).
Then
LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
may look like this (all explicitly listed locks are in Waiting state)
{, SR5 SW3 X4}
So the algorithm will find SR5 in the list and start looking backwards (to the
left).
According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
examined and so SR5 may be granted.
Theoretically, this could cause starvation.
The query that generates the list already has
query.append(" and hl_lock_ext_id <= ").append(extLockId);
but it should use "<" rather than "<=" to exclude the locks being checked from
"locks" list which will make the algorithm look at all locks "in front" of a
given lock.
was:
HIVE-10242 made the acid LM unfair.
In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number is
extLockId).
Then
LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
may look like this (all explicitly listed locks are in Waiting state)
{, SR5 SW3 X4}
So the algorithm will find SR5 in the list and start looking backwards.
According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
examined and so SR5 may be granted.
Theoretically, this could cause starvation.
The query that generates the list already has
query.append(" and hl_lock_ext_id <= ").append(extLockId);
but it should use "<" rather than "<=" to exclude the locks being checked from
"locks" list which will make the algorithm look at all locks "in front" of a
given lock.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17284) remove OrcRecordUpdater.deleteEventIndexBuilder
[
https://issues.apache.org/jira/browse/HIVE-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358915#comment-16358915
]
Eugene Koifman commented on HIVE-17284:
---
this may not be the right thing to do. ORC flattens structs (ROW__ID) and will
maintain min/max for individual columns. To filter events we really need
min/max ROW__ID
> remove OrcRecordUpdater.deleteEventIndexBuilder
> ---
>
> Key: HIVE-17284
> URL: https://issues.apache.org/jira/browse/HIVE-17284
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
>
> There is no point in it. We know how many rows a delete_delta file has from
> ORC and they are all the same type - so no need for AcidStats.
> hive.acid.key.index has no value since delete_delta files are never split
> and are not likely to have more than 1 stripe since they are very small.
> Also can remove KeyIndexBuilder.acidStats - we only have 1 type of event per
> file
>
> if doing this, make sure to fix {{OrcInputFormat.isOriginal(Reader)}} and
> {{OrcInputFormat.isOriginal(Footer)}} etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-17284) remove OrcRecordUpdater.deleteEventIndexBuilder
[
https://issues.apache.org/jira/browse/HIVE-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358915#comment-16358915
]
Eugene Koifman edited comment on HIVE-17284 at 2/9/18 8:31 PM:
---
this may not be the right thing to do. ORC flattens structs ({{ROW__ID}}) and
will maintain min/max for individual columns. To filter events we really need
min/max {{ROW__ID}}
was (Author: ekoifman):
this may not be the right thing to do. ORC flattens structs (ROW__ID) and will
maintain min/max for individual columns. To filter events we really need
min/max ROW__ID
> remove OrcRecordUpdater.deleteEventIndexBuilder
> ---
>
> Key: HIVE-17284
> URL: https://issues.apache.org/jira/browse/HIVE-17284
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
>
> There is no point in it. We know how many rows a delete_delta file has from
> ORC and they are all the same type - so no need for AcidStats.
> hive.acid.key.index has no value since delete_delta files are never split
> and are not likely to have more than 1 stripe since they are very small.
> Also can remove KeyIndexBuilder.acidStats - we only have 1 type of event per
> file
>
> if doing this, make sure to fix {{OrcInputFormat.isOriginal(Reader)}} and
> {{OrcInputFormat.isOriginal(Footer)}} etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18662) hive.acid.key.index is missing entries
[
https://issues.apache.org/jira/browse/HIVE-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18662:
--
Description:
OrcRecordUpdater.KeyIndexBuilder stores an index in ORC footer where each entry
is the last ROW__ID of each stripe. In acid1 this is used to filter the events
from delta file when merging with part of the base.
as can be seen in {{TestTxnCommands.testVersioning()}} (added in HIVE-18659)
the {{hive.acid.key.index}} is empty.
This is because very little data is written and WriterImpl.flushStripe() is not
called except when {
{WriterImpl.close()}
is called. In the later, {{WriterCallback.preFooterWrite()}} is called before
{{preStripeWrite}} and so KeyIndexBuilder.preFooterWriter() records nothing in
\{{hive.acid.key.index}}
need to investigate if this is an issue, in particular acid 2
was:
OrcRecordUpdater.KeyIndexBuilder stores an index in ORC footer where each entry
is the last ROW__ID of each stripe. In acid1 this is used to filter the events
from delta file when merging with part of the base.
as can be seen in \{{TestTxnCommands.testVersioning()}} (added in HIVE-18659)
the \{{hive.acid.key.index}} is empty.
This is because very little data is written and WriterImpl.flushStripe() is not
called except when \{{WriterImpl.close()} is called. In the later,
\{{WriterCallback.preFooterWrite()}} is called before \{{preStripeWrite}} and
so KeyIndexBuilder.preFooterWriter() records nothing in \{{hive.acid.key.index}}
> hive.acid.key.index is missing entries
> --
>
> Key: HIVE-18662
> URL: https://issues.apache.org/jira/browse/HIVE-18662
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Priority: Major
>
> OrcRecordUpdater.KeyIndexBuilder stores an index in ORC footer where each
> entry is the last ROW__ID of each stripe. In acid1 this is used to filter
> the events from delta file when merging with part of the base.
>
> as can be seen in {{TestTxnCommands.testVersioning()}} (added in HIVE-18659)
> the {{hive.acid.key.index}} is empty.
>
> This is because very little data is written and WriterImpl.flushStripe() is
> not called except when {
> {WriterImpl.close()}
> is called. In the later, {{WriterCallback.preFooterWrite()}} is called
> before {{preStripeWrite}} and so KeyIndexBuilder.preFooterWriter() records
> nothing in \{{hive.acid.key.index}}
>
> need to investigate if this is an issue, in particular acid 2
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357756#comment-16357756 ] Eugene Koifman commented on HIVE-18659: --- [~prasanth_j] could you review please > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Attachment: HIVE-18659.01.patch > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18659: -- Status: Patch Available (was: Open) > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18659.01.patch > > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18659) add acid version marker to acid files
[ https://issues.apache.org/jira/browse/HIVE-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18659: - > add acid version marker to acid files > - > > Key: HIVE-18659 > URL: https://issues.apache.org/jira/browse/HIVE-18659 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > add acid version marker to acid files so that we know which version of acid > wrote the file -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17284) remove OrcRecordUpdater.deleteEventIndexBuilder
[
https://issues.apache.org/jira/browse/HIVE-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-17284:
--
Description:
There is no point in it. We know how many rows a delete_delta file has from ORC
and they are all the same type - so no need for AcidStats.
hive.acid.key.index has no value since delete_delta files are never split and
are not likely to have more than 1 stripe since they are very small.
Also can remove KeyIndexBuilder.acidStats - we only have 1 type of event per
file
if doing this, make sure to fix {{OrcInputFormat.isOriginal(Reader)}} and
{{OrcInputFormat.isOriginal(Footer)}} etc
was:
There is no point in it. We know how many rows a delete_delta file has from
ORC and they are all the same type - so no need for AcidStats.
hive.acid.key.index has no value since delete_delta files are never split and
are not likely to have more than 1 stripe since they are very small.
Also can remove KeyIndexBuilder.acidStats - we only have 1 type of event per
file
> remove OrcRecordUpdater.deleteEventIndexBuilder
> ---
>
> Key: HIVE-17284
> URL: https://issues.apache.org/jira/browse/HIVE-17284
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
>
> There is no point in it. We know how many rows a delete_delta file has from
> ORC and they are all the same type - so no need for AcidStats.
> hive.acid.key.index has no value since delete_delta files are never split
> and are not likely to have more than 1 stripe since they are very small.
> Also can remove KeyIndexBuilder.acidStats - we only have 1 type of event per
> file
>
> if doing this, make sure to fix {{OrcInputFormat.isOriginal(Reader)}} and
> {{OrcInputFormat.isOriginal(Footer)}} etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Resolution: Fixed Status: Resolved (was: Patch Available) committed to master thanks Sergey for the review > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18649) HiveInputFormat job conf object lifecycle is unclear or broken
[ https://issues.apache.org/jira/browse/HIVE-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18649: -- Component/s: Transactions > HiveInputFormat job conf object lifecycle is unclear or broken > -- > > Key: HIVE-18649 > URL: https://issues.apache.org/jira/browse/HIVE-18649 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > > Follow-up from HIVE-18575 > ACID properties may be added to the same job object for multiple tables, at > least by the looks of it. > There also exists a JobConf field "job" in HIF; and a separate JobConf input > argument to some methods. These methods apply some changes to one jobconf and > some to another, for no clear reason. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18619) Verification of temporary Micromanaged table atomicity is needed
[ https://issues.apache.org/jira/browse/HIVE-18619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355950#comment-16355950 ] Eugene Koifman commented on HIVE-18619: --- to test this in UT you can use HiveConf.HIVETESTMODEROLLBACKTXN after you set it, commit will fail - it's used simulate failures > Verification of temporary Micromanaged table atomicity is needed > - > > Key: HIVE-18619 > URL: https://issues.apache.org/jira/browse/HIVE-18619 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Minor > > Session based temporary table by HIVE-7090 had no consideration of > Micromanaged table > (MM) since there was no insert-only ACID table at its creation tije. > HIVE-18599 addressed the issue of no writes during CTTAS (Create Temporary > Table As Select) > on Micro-Managed table. But atomicity of temporary MM table is not verified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[
https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18575:
--
Comment: was deleted
(was: what was the point of renaming isAcid to isFullAcid? {{isAcid}} was used
everywhere to mean full acid table.
it now leads to code like
"compBuilder.setIsAcid(AcidUtils.isTransactionalTable(t));" which will cause
confusion?)
> ACID properties usage in jobconf is ambiguous for MM tables
> ---
>
> Key: HIVE-18575
> URL: https://issues.apache.org/jira/browse/HIVE-18575
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18575.01.patch, HIVE-18575.patch
>
>
> Vectorization checks for ACID table trigger for MM tables where they don't
> apply. Other places seem to set the setting for transactional case while most
> of the code seems to assume it implies full acid.
> Overall, many places in the code use the settings directly or set the ACID
> flag without setting the ACID properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355929#comment-16355929 ] Eugene Koifman commented on HIVE-18575: --- made some comments on RB > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.01.patch, HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Status: Patch Available (was: Open) [~sershe] could you review please > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Attachment: HIVE-18636.01.patch > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-18636.01.patch > > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18643) don't check for archived partitions for ACID ops
[ https://issues.apache.org/jira/browse/HIVE-18643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354840#comment-16354840 ] Eugene Koifman commented on HIVE-18643: --- +1 > don't check for archived partitions for ACID ops > > > Key: HIVE-18643 > URL: https://issues.apache.org/jira/browse/HIVE-18643 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18643.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18636: -- Priority: Blocker (was: Major) > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18636) fix TestTxnNoBuckets.testCTAS - keeps failing on ptest
[ https://issues.apache.org/jira/browse/HIVE-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18636: - Assignee: Eugene Koifman > fix TestTxnNoBuckets.testCTAS - keeps failing on ptest > -- > > Key: HIVE-18636 > URL: https://issues.apache.org/jira/browse/HIVE-18636 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > need to update expected result -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[
https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354201#comment-16354201
]
Eugene Koifman commented on HIVE-18575:
---
what was the point of renaming isAcid to isFullAcid? {{isAcid}} was used
everywhere to mean full acid table.
it now leads to code like
"compBuilder.setIsAcid(AcidUtils.isTransactionalTable(t));" which will cause
confusion?
> ACID properties usage in jobconf is ambiguous for MM tables
> ---
>
> Key: HIVE-18575
> URL: https://issues.apache.org/jira/browse/HIVE-18575
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-18575.01.patch, HIVE-18575.patch
>
>
> Vectorization checks for ACID table trigger for MM tables where they don't
> apply. Other places seem to set the setting for transactional case while most
> of the code seems to assume it implies full acid.
> Overall, many places in the code use the settings directly or set the ACID
> flag without setting the ACID properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352863#comment-16352863
]
Eugene Koifman commented on HIVE-16605:
---
merge_constraint_notnull.q sets hive.merge.cardinality.check=false
is that intentional? if so why?
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch, HIVE-16605.4.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352769#comment-16352769 ] Eugene Koifman commented on HIVE-18575: --- I left some comments on RB. Mostly, I think the change in TableScanDesc() doesn't do the right thing. Also, the test for full acid is sometimes called isAcidTable() and sometimes isFullAcidTable() - either is fine but I think it's confusing to have both. > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18128) Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes downstream errors
[ https://issues.apache.org/jira/browse/HIVE-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352746#comment-16352746 ] Eugene Koifman commented on HIVE-18128: --- fixed in HIVE-18575? > Setting AcidUtils.setTransactionalTableScan in HiveInputFormat causes > downstream errors > --- > > Key: HIVE-18128 > URL: https://issues.apache.org/jira/browse/HIVE-18128 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Priority: Major > Attachments: HIVE-18128.WIP.patch > > > This should really be set in addSplitsForGroup(). See attached patch for > details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18575: -- Component/s: Transactions > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18575.patch > > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) committed to master thanks Gopal for the review > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18125) Support arbitrary file names in input to Load Data
[ https://issues.apache.org/jira/browse/HIVE-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-18125. --- Resolution: Fixed Fix Version/s: 3.0.0 fixed via HIVE-18516 > Support arbitrary file names in input to Load Data > -- > > Key: HIVE-18125 > URL: https://issues.apache.org/jira/browse/HIVE-18125 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18125.01.patch, HIVE-18125.02.patch > > > HIVE-17361 only allows 0_0 and _0_copy_1. Should it support > arbitrary names? > If so, should it sort them and rename _0, 0001_0, etc? > This is probably a lot easier than changing the whole code base to assign > proper 'bucket' (writerId) everywhere Acid reads such file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18221) test acid default
[ https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18221: -- Status: Open (was: Patch Available) > test acid default > - > > Key: HIVE-18221 > URL: https://issues.apache.org/jira/browse/HIVE-18221 > Project: Hive > Issue Type: Test > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, > HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, > HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, > HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, > HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch, > HIVE-18221.19.patch, HIVE-18221.20.patch, HIVE-18221.21.patch, > HIVE-18221.22.patch, HIVE-18221.23.patch, HIVE-18221.24.patch, > HIVE-18221.26.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Release Note: N/A
Status: Resolved (was: Patch Available)
pushed to master
Thanks Sergey for the review
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch,
> HIVE-18606.03.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350945#comment-16350945 ] Eugene Koifman commented on HIVE-18516: --- +1 > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.10.patch, > HIVE-18516.2.patch, HIVE-18516.3.patch, HIVE-18516.4.patch, > HIVE-18516.5.patch, HIVE-18516.6.patch, HIVE-18516.7.patch, > HIVE-18516.8.patch, HIVE-18516.9.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Attachment: HIVE-18606.03.patch
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch,
> HIVE-18606.03.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349633#comment-16349633
]
Eugene Koifman commented on HIVE-18606:
---
yes, you are right. fixed in patch2
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Attachment: HIVE-18606.02.patch
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch, HIVE-18606.02.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349627#comment-16349627
]
Eugene Koifman commented on HIVE-18606:
---
[~sershe] could you review please
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Status: Patch Available (was: Open)
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Attachment: HIVE-18606.01.patch
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18606.01.patch
>
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18606:
--
Component/s: Transactions
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Priority: Major
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18606) CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
[
https://issues.apache.org/jira/browse/HIVE-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-18606:
-
Assignee: Eugene Koifman
> CTAS on empty table throws NPE from org.apache.hadoop.hive.ql.exec.MoveTask
> ---
>
> Key: HIVE-18606
> URL: https://issues.apache.org/jira/browse/HIVE-18606
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {noformat}
> @Test
> public void testCtasEmpty() throws Exception {
> MetastoreConf.setBoolVar(hiveConf,
> MetastoreConf.ConfVars.CREATE_TABLES_AS_ACID, true);
> runStatementOnDriver("create table myctas stored as ORC as" +
> " select a, b from " + Table.NONACIDORCTBL);
> List rs = runStatementOnDriver("select ROW__ID, a, b,
> INPUT__FILE__NAME" +
> " from myctas order by ROW__ID");
> }
> {noformat}
> {noformat}
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(822)) - 114: Done
> cleaning up thread local RawStore
> 2018-02-01T19:08:52,813 INFO [HiveServer2-Background-Pool: Thread-463]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(305)) - ugi=hive
> ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> exec.Task (SessionState.java:printError(1228)) - Failed with exception null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.moveAcidFiles(Hive.java:3816)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:298)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2267)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1919)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1651)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1388)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:253)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:358)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2018-02-01T19:08:52,815 ERROR [HiveServer2-Background-Pool: Thread-463]:
> ql.Driver (SessionState.java:printError(1228)) - FAILED: Execution Error,
> return code 1 from {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348899#comment-16348899 ] Eugene Koifman commented on HIVE-18589: --- no related failures [~gopalv] could you review please > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Attachment: HIVE-18589.02.patch > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Description: txnid:17 starts reading T2/P1 txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 still running txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. before Insert overwrite was supported, only the compactor could produce base_20 by running major compaction. Major compaction erases history and so a reader with txnid:17 open, can't use base_20. Normally, the Cleaner is smart enough to not clean pre-compaction files if it's possible that there is a reader that requires them. There is a safety check that creates "Not enough history.." error if it finds that the current reader can't properly execute based on the files available. with the introduction of IOW on acid tables, there is another way to produce a base. The difference is that here, the base has no history by definition and so the same check is not needed but is triggered in the scenario above. > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347618#comment-16347618 ] Eugene Koifman commented on HIVE-18536: --- +1 > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347577#comment-16347577 ] Eugene Koifman edited comment on HIVE-18536 at 1/31/18 8:39 PM: I was asking about "Boolean baseDir" param on RB. why is it needed? I don't understand your answer was (Author: ekoifman): I was asking about "Boolean baseDir" param on RB. why is it needed? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347577#comment-16347577 ] Eugene Koifman commented on HIVE-18536: --- I was asking about "Boolean baseDir" param on RB. why is it needed? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18592) DP insert on insert only table causes StatTask to fail
[
https://issues.apache.org/jira/browse/HIVE-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18592:
--
Component/s: Transactions
> DP insert on insert only table causes StatTask to fail
> --
>
> Key: HIVE-18592
> URL: https://issues.apache.org/jira/browse/HIVE-18592
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Zoltan Haindrich
>Priority: Major
>
> can be reproduced with:
> {code}
> set hive.mapred.mode=nonstrict;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.create.as.insert.only=true;
> set metastore.create.as.acid=true;
> drop table if exists student;
> create table student(
> name string,
> age int,
> gpa double);
> insert into student values
> ('asd',1,2),
> ('asdx',2,3),
> ('asdx',2,3),
> ('asdx',3,3),
> ('asdx',3,3),
> ('asdx',3,3);
> create table p1 (name STRING, GPA DOUBLE) PARTITIONED BY (age INT);
> SET hive.exec.dynamic.partition.mode=nonstrict;
> INSERT OVERWRITE TABLE p1 PARTITION (age) SELECT name, gpa, age FROM student;
> {code}
> causes exception
> {code}
> 2018-01-31T02:16:24,135 ERROR [22bd4065-6e2f-4f4c-8f29-8d6aad8edda8 main]
> exec.StatsTask: Failed to run stats task
> org.apache.hadoop.hive.ql.metadata.HiveException:
> NoSuchObjectException(message:Partition for which stats is gathered doesn't
> exist.)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4295)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:180)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:84)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> ...
> Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException:
> Partition for which stats is gathered doesn't exist.
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:7757)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_151]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy38.updatePartitionColumnStatistics(Unknown
> Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStats(HiveMetaStore.java:5394)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6907)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_151]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.set_aggr_stats_for(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1736)
>
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Attachment: HIVE-18589.01.patch > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Status: Patch Available (was: Open) > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18589: - > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18577) SemanticAnalyzer.validate has some pointless metastore calls
[ https://issues.apache.org/jira/browse/HIVE-18577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18577: -- Component/s: Transactions > SemanticAnalyzer.validate has some pointless metastore calls > > > Key: HIVE-18577 > URL: https://issues.apache.org/jira/browse/HIVE-18577 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18577.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344354#comment-16344354 ] Eugene Koifman commented on HIVE-18536: --- I imagine AcidUtils can just as well be in common > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344129#comment-16344129 ] Eugene Koifman edited comment on HIVE-18575 at 1/29/18 10:36 PM: - hive.acid.table.scan and TABLE_IS_TRANSACTIONAL = "transactional"; as in AcidUtils.isTablePropertyTransactional when set on Configuration were always meant for full acid and only full acid The code running on the Grid uses these to know it's doing acid I think is the part that got broken - it's now set where it shouldn't be was (Author: ekoifman): hive.acid.table.scan and TABLE_IS_TRANSACTIONAL = "transactional"; as in AcidUtils.isTablePropertyTransactional when set on Configuration were always meant for full acid and only full acid I think is the part that got broken - it's now set where it shouldn't be > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18575) ACID properties usage in jobconf is ambiguous for MM tables
[ https://issues.apache.org/jira/browse/HIVE-18575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344129#comment-16344129 ] Eugene Koifman commented on HIVE-18575: --- hive.acid.table.scan and TABLE_IS_TRANSACTIONAL = "transactional"; as in AcidUtils.isTablePropertyTransactional when set on Configuration were always meant for full acid and only full acid I think is the part that got broken - it's now set where it shouldn't be > ACID properties usage in jobconf is ambiguous for MM tables > --- > > Key: HIVE-18575 > URL: https://issues.apache.org/jira/browse/HIVE-18575 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > > Vectorization checks for ACID table trigger for MM tables where they don't > apply. Other places seem to set the setting for transactional case while most > of the code seems to assume it implies full acid. > Overall, many places in the code use the settings directly or set the ACID > flag without setting the ACID properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343649#comment-16343649 ] Eugene Koifman commented on HIVE-18536: --- I left 1 comment on RB. The code to create/parse base/delta dir names has spread to different places from being localized to AcidUtils. This will come back to haunt us but not introduced in this patch. there is a bunch of new checkstyle warning otherwise it looks ok. > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18126) IOW Mechanics of multiple commands with OVERWRITE in a singe transactions
[ https://issues.apache.org/jira/browse/HIVE-18126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18126: -- Summary: IOW Mechanics of multiple commands with OVERWRITE in a singe transactions (was: Mechanics of multiple commands with OVERWRITE in a singe transactions) > IOW Mechanics of multiple commands with OVERWRITE in a singe transactions > - > > Key: HIVE-18126 > URL: https://issues.apache.org/jira/browse/HIVE-18126 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > for Insert overwrite/load data overwrite we create base_x/ to hold the data > thus are able to make Overwrite command non-blocking. > What happens if multiple IOWs are run against the same table/partition in the > same transaction. > should base support a suffix base_x_000 like deltas? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17457) IOW Acid Insert Overwrite when the transaction fails
[ https://issues.apache.org/jira/browse/HIVE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17457: -- Summary: IOW Acid Insert Overwrite when the transaction fails (was: Acid Insert Overwrite when the transaction fails) > IOW Acid Insert Overwrite when the transaction fails > > > Key: HIVE-17457 > URL: https://issues.apache.org/jira/browse/HIVE-17457 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > HIVE-14988 adds support for Insert Overwrite for Acid tables. > once we have direct write to target dir (i.e. no move op) - how do we handle > the case where the txn running IOW aborts? See if getAcidState() does the > right thing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18154) IOW Acid Load Data/Insert with Overwrite in multi statement transactions
[
https://issues.apache.org/jira/browse/HIVE-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18154:
--
Summary: IOW Acid Load Data/Insert with Overwrite in multi statement
transactions (was: Acid Load Data/Insert with Overwrite in multi statement
transactions)
> IOW Acid Load Data/Insert with Overwrite in multi statement transactions
>
>
> Key: HIVE-18154
> URL: https://issues.apache.org/jira/browse/HIVE-18154
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> Consider:
> {noformat}
> START TRANSACTION
> insert into T values(1,2),(3,4)
> load data local inpath '" + getWarehouseDir() + "/1/data' overwrite into
> table T
> update T set a = 0 where a = 6
> COMMIT
> {noformat}
> So what we should have on disk is
> {noformat}
> ├── base_028
> │ ├── 00_0
> │ └── _metadata_acid
> ├── delete_delta_028_028_0002
> │ └── bucket_0
> ├── delta_028_028_
> │ └── bucket_0
> └── delta_028_028_0002
> └── bucket_0
> {noformat}
> where base_28 is from overwrite, delta_028_028_ from 1st insert
> nad delta_028_028_0002/delete_delta_028_028_0002 is from
> update.
> AcidUtils.getAcidState() only returns base_28 thinking that all other deltas
> are included in it - not what we want here.
> Same applies for Insert Overwrite.
> The simple way to get correct behavior is to disallow commands with Overwrite
> clause in multi-statement txns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18157) Vectorization : Insert in bucketed table is broken with vectorization
[ https://issues.apache.org/jira/browse/HIVE-18157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18157: -- Component/s: Vectorization Transactions > Vectorization : Insert in bucketed table is broken with vectorization > - > > Key: HIVE-18157 > URL: https://issues.apache.org/jira/browse/HIVE-18157 > Project: Hive > Issue Type: Bug > Components: Transactions, Vectorization >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18157.1.patch, HIVE-18157.3.patch > > > create temporary table foo (x int) clustered by (x) into 4 buckets; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 9 00_0 > 4 00_0 > 3 00_0 > 2 00_0 > 1 00_0 > set hive.vectorized.execution.enabled=false; > insert overwrite table foo values(1),(2),(3),(4),(9); > select *, regexp_extract(INPUT__FILE__NAME, '.*/(.*)', 1) from foo; > OK > 4 00_0 > 9 01_0 > 1 01_0 > 2 02_0 > 3 03_0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338208#comment-16338208 ] Eugene Koifman commented on HIVE-18192: --- Suppose A is the set of all ValidTxnList across all active readers. Each ValidTxnList has minOpenTxnId. MIN_HISTORY_LEVEL allows us to determine min(minOpenTxnId) across all currently active readers. It's not the same as COMPLETED_TXN_COMPONENTS. Entries from COMPLETED_TXN_COMPONENTS get removed once compaction has processed the relevant partition which can happen before all readers that think txn X is open have gone away. Suppose txn 17 starts at t1 and sees txnid 13 with writeID 13 open. 13 commits (via it's parent txn) at t2 > t1. (17 is still running). Compaction runs at t3 >t2 to produce base_14 (or delta_10_14 for example) on Table1/Part1 (17 is still running) COMPLETED_TXN_COMPONENTS may be cleaned at this point. at t4 > t3 17 may (multi stmt txn) needs to read Table1/Part1. It now needs to construct a ValidWriteIDList as it would have looked like at the time 17 started to maintain SI. But if TXNS2WRITE_ID no longer has txnid13 -> writeID13 mapping, there is no way to know that ValidWriteIDList should have writeID13 Open. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch, HIVE-18192.04.patch, HIVE-18192.05.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336844#comment-16336844 ] Eugene Koifman commented on HIVE-18519: --- +1 pending tests > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18519) do not create materialized CTEs with ACID/MM
[ https://issues.apache.org/jira/browse/HIVE-18519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18519: -- Component/s: Transactions > do not create materialized CTEs with ACID/MM > > > Key: HIVE-18519 > URL: https://issues.apache.org/jira/browse/HIVE-18519 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18519.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18520) add current txnid to ValidTxnList
[ https://issues.apache.org/jira/browse/HIVE-18520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18520: - > add current txnid to ValidTxnList > - > > Key: HIVE-18520 > URL: https://issues.apache.org/jira/browse/HIVE-18520 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > add the Id of the transaction that obtained this ValidTxnList > if nothing else, convenient for debugging > in particular include it in ErrorMsg.ACID_NOT_ENOUGH_HISTORY -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15658) hive.ql.session.SessionState start() is not atomic, SessionState thread local variable can get into inconsistent state
[
https://issues.apache.org/jira/browse/HIVE-15658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15658:
--
Component/s: Transactions
> hive.ql.session.SessionState start() is not atomic, SessionState thread local
> variable can get into inconsistent state
> --
>
> Key: HIVE-15658
> URL: https://issues.apache.org/jira/browse/HIVE-15658
> Project: Hive
> Issue Type: Bug
> Components: API, HCatalog, Transactions
>Affects Versions: 1.1.0, 1.2.1, 2.0.0, 2.0.1
> Environment: CDH5.8.0, Flume 1.6.0, Hive 1.1.0
>Reporter: Michal Klempa
>Priority: Major
> Attachments: HIVE-15658_branch-1.2_1.patch,
> HIVE-15658_branch-2.1_1.patch
>
>
> Method start() in hive.ql.session.SessionState is supposed to setup needed
> preconditions, like HDFS scratch directories for session.
> This happens to be not an atomic operation with setting thread local
> variable, which can later be obtained by calling SessionState.get().
> Therefore, even is the start() method itself fails, the SessionState.get()
> does not return null and further re-use of the thread which previously
> invoked start() may lead to obtaining SessionState object in inconsistent
> state.
> I have observed this using Flume Hive Sink, which uses Hive Streaming
> interface. When the directory /tmp/hive is not writable by session user, the
> start() method fails (throwing RuntimeException). If the thread is re-used
> (like it is in Flume), further executions work with wrongly initialized
> SessionState object (HDFS dirs are non-existent). In Flume, this happens to
> me when Flume should create partition if not exists (but the code doing this
> is in Hive Streaming).
> Steps to reproduce:
> 0. create test spooldir and allow flume to write to it, in my case
> /home/ubuntu/flume_test, 775, ubuntu:flume
> 1. create Flume config (see attachment)
> 2. create Hive table
> {code}
> create table default.flume_test (column1 string, column2 string) partitioned
> by (dt string) clustered by (column1) INTO 2 BUCKETS STORED AS ORC;
> {code}
> 3. start flume agent:
> {code}
> bin/flume-ng agent -n a1 -c conf -f conf/flume-config.txt
> {code}
> 4. hdfs dfs -chmod 600 /tmp/hive
> 5. put this file into spooldir:
> {code}
> echo value1,value2 > file1
> {code}
> Expected behavior:
> Exception regarding scratch dir permissions to be thrown repeatedly.
> example (note that the line numbers are wrong as Cloudera is cloning the
> source codes here https://github.com/cloudera/flume-ng/ and here
> https://github.com/cloudera/hive):
> {code}
> 2017-01-18 12:39:38,926 WARN org.apache.flume.sink.hive.HiveSink: sink_hive_1
> : Failed connecting to EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083',
> database='default', table='flume_test', partitionVals=[20170118] }
> org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to
> EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083', database='default',
> table='flume_test', partitionVals=[20170118] }
> at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:99)
> at
> org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
> at
> org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
> at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed
> connecting to EndPoint {metaStoreUri='thrift://n02.cdh.ideata:9083',
> database='default', table='flume_test', partitionVals=[20170118] }
> at
> org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:380)
> at org.apache.flume.sink.hive.HiveWriter.(HiveWriter.java:86)
> ... 6 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root
> scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:
> rw---
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:540)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:358)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.(HiveEndPoint.java:276)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.(HiveEndPoint.java:243)
> at
> org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
> at
>
[jira] [Commented] (HIVE-18504) Hive is throwing InvalidObjectException(message:Invalid column type name is too long.
[
https://issues.apache.org/jira/browse/HIVE-18504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334702#comment-16334702
]
Eugene Koifman commented on HIVE-18504:
---
2.6.3 looks like HDP version
> Hive is throwing InvalidObjectException(message:Invalid column type name is
> too long.
> -
>
> Key: HIVE-18504
> URL: https://issues.apache.org/jira/browse/HIVE-18504
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Jimson K James
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 2.3.0, 3.0.0
>
> Attachments: hive2.log, tweets.sql
>
>
> Hive 2.6.3 is still throwing InvalidObjectException(message:Invalid column
> type name is too long.
> Please find attached the create table query. For more details please refer to
> HIVE-15249
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> InvalidObjectException(message:Invalid column type name length 2980 exceeds
> max allowed length 2000, type
> struct>,symbols:array...
> {code}
>
> {code:java}
> [root@sandbox-hdp hive-json]# hive --version
> Hive 1.2.1000.2.6.3.0-235
> Subversion
> git://ctr-e134-1499953498516-254436-01-04.hwx.site/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
> -r 5f360bda08bb5489fbb3189b5aeaaf58029ed4b5
> Compiled by jenkins on Mon Oct 30 02:48:31 UTC 2017
> From source with checksum 94298cc1f5f5bf0f3470f3ea2e92d646
> [root@sandbox-hdp hive-json]# beeline
> Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive
> beeline> !connect
> jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> Connecting to
> jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> Enter username for
> jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
> hive
> Enter password for
> jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
>
> Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
> Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> 0: jdbc:hive2://sandbox-hdp.hortonworks.com:2>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18503) MM/ACID tables: make tests that will never be compatible with acid use non-txn tables explicitly
[ https://issues.apache.org/jira/browse/HIVE-18503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16333133#comment-16333133 ] Eugene Koifman commented on HIVE-18503: --- linking HIVE-18315 for completeness - it did a bunch of similar work > MM/ACID tables: make tests that will never be compatible with acid use > non-txn tables explicitly > > > Key: HIVE-18503 > URL: https://issues.apache.org/jira/browse/HIVE-18503 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18503.WIP.patch > > > Some tests do stuff that will simply never work with ACID tables, e.g. delete > table files. > They should be changed to use external table, or explicitly set > transactional=false -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18502) base_-9223372036854775808 is confusing
[ https://issues.apache.org/jira/browse/HIVE-18502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18502: -- Component/s: Transactions > base_-9223372036854775808 is confusing > -- > > Key: HIVE-18502 > URL: https://issues.apache.org/jira/browse/HIVE-18502 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 0.13.0 >Reporter: Eugene Koifman >Priority: Minor > > start with no acid table > write some data > Alter Table to acid table > run major compaction > It will create base_-9223372036854775808 (if there are no delta dirs) > Nothing wrong with it but confuses users. Should probably make it base_0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18192: -- Description: To support ACID replication, we will be introducing a per table write Id which will replace the transaction id in the primary key for each row in a ACID table. The current primary key is determined via which will move to For each table modified by the given transaction will have a table level write ID allocated and a persisted map of global txn id -> to table -> write id for that table has to be maintained to allow Snapshot isolation. Readers should use the combination of ValidTxnList and ValidWriteIdList(Table) for snapshot isolation. [Hive Replication - ACID Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] has a section "Per Table Sequences (Write-Id)" with more detials was: To support ACID replication, we will be introducing a per table write Id which will replace the transaction id in the primary key for each row in a ACID table. The current primary key is determined via which will move to For each table modified by the given transaction will have a table level write ID allocated and a persisted map of global txn id -> to table -> write id for that table has to be maintained to allow Snapshot isolation. Readers should use the combination of ValidTxnList and ValidWriteIdList(Table) for snapshot isolation. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18460:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch, HIVE-18460.04.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330801#comment-16330801
]
Eugene Koifman commented on HIVE-18460:
---
no related failures
committed to master
thanks Prasanth for the review
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch, HIVE-18460.04.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18460:
--
Status: Patch Available (was: Open)
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch, HIVE-18460.04.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18460:
--
Attachment: HIVE-18460.04.patch
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch, HIVE-18460.04.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18460:
--
Status: Open (was: Patch Available)
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18460) Compactor doesn't pass Table properties to the Orc writer
[
https://issues.apache.org/jira/browse/HIVE-18460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18460:
--
Attachment: HIVE-18460.03.patch
> Compactor doesn't pass Table properties to the Orc writer
> -
>
> Key: HIVE-18460
> URL: https://issues.apache.org/jira/browse/HIVE-18460
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 0.13
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18460.01.patch, HIVE-18460.02.patch,
> HIVE-18460.03.patch
>
>
>
> CompactorMap.getWrite()/getDeleteEventWriter() both do
> AcidOutputFormat.Options.tableProperties() but
> OrcOutputFormat.getRawRecordWriter() does
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getConfiguration());
> {noformat}
> which ignores tableProperties value.
> It should do
> {noformat}
> final OrcFile.WriterOptions opts =
> OrcFile.writerOptions(options.getTableProperties(),
> options.getConfiguration());
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
