Jenkins build is back to normal : Hive-trunk-h0.20 #623

[Apache Hudson Server]

See 

[jira] Commented: (HIVE-2065) RCFile issues

[He Yongqiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008730#comment-13008730
 ] 

He Yongqiang commented on HIVE-2065:


Good catch about point 2. It is for uncompressed length, and then is used as 
compressed length in the code when seeking to next block. Fortunately we have 
not triggered this bug in Hive so far :) .

> RCFile issues
> -
>
> Key: HIVE-2065
> URL: https://issues.apache.org/jira/browse/HIVE-2065
> Project: Hive
>  Issue Type: Bug
>Reporter: Krishna Kumar
>Assignee: Krishna Kumar
>Priority: Minor
> Attachments: Slide1.png
>
>
> Some potential issues with RCFile
> 1. Remove unwanted synchronized modifiers on the methods of RCFile. As per 
> yongqiang he, the class is not meant to be thread-safe (and it is not). Might 
> as well get rid of the confusing and performance-impacting lock acquisitions.
> 2. Record Length overstated for compressed files. IIUC, the key compression 
> happens after we have written the record length.
> {code}
>   int keyLength = key.getSize();
>   if (keyLength < 0) {
> throw new IOException("negative length keys not allowed: " + key);
>   }
>   out.writeInt(keyLength + valueLength); // total record length
>   out.writeInt(keyLength); // key portion length
>   if (!isCompressed()) {
> out.writeInt(keyLength);
> key.write(out); // key
>   } else {
> keyCompressionBuffer.reset();
> keyDeflateFilter.resetState();
> key.write(keyDeflateOut);
> keyDeflateOut.flush();
> keyDeflateFilter.finish();
> int compressedKeyLen = keyCompressionBuffer.getLength();
> out.writeInt(compressedKeyLen);
> out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
>   }
> {code}
> 3. For sequence file compatibility, the compressed key length should be the 
> next field to record length, not the uncompressed key length.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[John Sichi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-2011:
-

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Committed to branch and trunk.  Thanks Carl!

p.s. ship it!


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt, HIVE-2011.6.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Resolved: (HIVE-2010) Improve naming and README files for MetaStore upgrade scripts

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach resolved HIVE-2010.
--

Resolution: Duplicate

Fixed in HIVE-2011


> Improve naming and README files for MetaStore upgrade scripts
> -
>
> Key: HIVE-2010
> URL: https://issues.apache.org/jira/browse/HIVE-2010
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation, Metastore
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
>
> The Hive metastore upgrade scripts currently use the naming convention 
> upgrade-X.Y.Z..sql, with the intention of indicating that this file 
> handles the upgrade to version X.Y.Z from the previous release. This has 
> caused confusion with users who wonder if upgrade script actually handles the 
> upgrade to X.Y.Z from all earlier versions of Hive.
> I propose changing the naming convention to 
> upgrade-A.B.C-to-X.Y.Z..sql in order to clarify the intended usage of 
> these scripts.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008723#comment-13008723
 ] 

Carl Steinbach commented on HIVE-2011:
--

Review request: https://reviews.apache.org/r/471/


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt, HIVE-2011.6.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Status: Patch Available  (was: Open)

> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt, HIVE-2011.6.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Attachment: HIVE-2011.6.patch.txt

HIVE-2011.6.patch.txt:

* Fix identifier capitalization issues in MySQL schema scripts.


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt, HIVE-2011.6.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-2011: upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/471/
---

(Updated 2011-03-18 20:19:24.237325)


Review request for hive.


Summary
---

Review request for HIVE-2011 and HIVE-2010.


This addresses bugs HIVE-2010 and HIVE-2011.
https://issues.apache.org/jira/browse/HIVE-2010
https://issues.apache.org/jira/browse/HIVE-2011


Diffs (updated)
-

  metastore/scripts/upgrade/derby/001-HIVE-972.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/002-HIVE-1068.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/003-HIVE-675.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/004-HIVE-1364.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/005-HIVE-417.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/006-HIVE-1823.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/007-HIVE-78.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/README 506eb54 
  metastore/scripts/upgrade/derby/hive-schema-0.3.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.1.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.5.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.5.0-to-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0-to-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0.derby.sql 77a0f39 
  metastore/scripts/upgrade/derby/upgrade-0.7.0.derby.sql ccad4e7 
  metastore/scripts/upgrade/mysql/001-HIVE-972.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/002-HIVE-1068.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/003-HIVE-675.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/004-HIVE-1364.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/005-HIVE-417.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/006-HIVE-1823.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/007-HIVE-78.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/README 3d4360f 
  metastore/scripts/upgrade/mysql/hive-schema-0.3.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.1.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.5.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.5.0-to-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0-to-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0.mysql.sql 8cbd501 
  metastore/scripts/upgrade/mysql/upgrade-0.7.0.mysql.sql aab664e 

Diff: https://reviews.apache.org/r/471/diff


Testing
---


Thanks,

Carl

[jira] Commented: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008719#comment-13008719
 ] 

Carl Steinbach commented on HIVE-2011:
--

Ugh. HFS+'s case insensitivity bites me again: 
http://dev.mysql.com/doc/refman/5.0/en/identifier-case-sensitivity.html

I'm making the changes right now. Sorry for the inconvenience.


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008716#comment-13008716
 ] 

John Sichi commented on HIVE-2011:
--

The latest patch still has lowercase in all of the REFERENCES clauses, e.g.

{noformat}
  CONSTRAINT `SDS_FK1` FOREIGN KEY (`SERDE_ID`) REFERENCES `serdes` (`SERDE_ID`)
{noformat}

This causes inserts to fail.


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[John Sichi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-2011:
-

Status: Open  (was: Patch Available)

> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hive-trunk-h0.20 #622

[Apache Hudson Server]

See 

Changes:

[nzhang] HIVE-1976. Exception should be thrown when invalid jar,file,archive is 
given to add command (Chinna Rao Lalam via Ning Zhang)

--
[...truncated 27984 lines...]
[junit] POSTHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] POSTHOOK: type: LOAD
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable
[junit] PREHOOK: type: QUERY
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_19-34-42_823_8683092852421127168/-mr-1
[junit] Total MapReduce jobs = 1
[junit] Launching Job 1 out of 1
[junit] Number of reduce tasks determined at compile time: 1
[junit] In order to change the average load for a reducer (in bytes):
[junit]   set hive.exec.reducers.bytes.per.reducer=
[junit] In order to limit the maximum number of reducers:
[junit]   set hive.exec.reducers.max=
[junit] In order to set a constant number of reducers:
[junit]   set mapred.reduce.tasks=
[junit] Job running in-process (local Hadoop)
[junit] Hadoop job information for null: number of mappers: 0; number of 
reducers: 0
[junit] 2011-03-18 19:34:45,903 null map = 100%,  reduce = 100%
[junit] Ended Job = job_local_0001
[junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable
[junit] POSTHOOK: type: QUERY
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_19-34-42_823_8683092852421127168/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: CREATETABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: CREATETABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] PREHOOK: type: LOAD
[junit] Copying data from 

[junit] Loading data to table default.testhivedrivertable
[junit] POSTHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] POSTHOOK: type: LOAD
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select * from testhivedrivertable limit 10
[junit] PREHOOK: type: QUERY
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_19-34-47_436_3705758857571005678/-mr-1
[junit] POSTHOOK: query: select * from testhivedrivertable limit 10
[junit] POSTHOOK: type: QUERY
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_19-34-47_436_3705758857571005678/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table

Re: RCFile - some queries

[Krishna Kumar]

Hi yongqiang he,

Have created a bug https://issues.apache.org/jira/browse/HIVE-2065 to carry 
on the discussion. Have attached the picture there too: 
https://issues.apache.org/jira/secure/attachment/12474055/Slide1.png. (looks 
like attachments are stripped from posts here?)

Please comment there.

Cheers,
 Krishna


On 3/18/11 11:47 PM, "yongqiang he"  wrote:

>> but the recordLength is not the actual on-disk length of the record.
It is acutal on-disk length. It is compressed key length plus the
compressed value length

>>Similarly, the next field - key length - is not the on-disk length of the 
>>compressed key.

There are two keyLengths, one is compressed key length, the other is
uncompressed keyLength

For 2, it wo't be a problem. record length is compressed length

>>Thread-Safety.
It is not thread safe. Application should do it themselves.
 It is initially designed for Hive. Thread safety is there at first
time, and then removed because Hive does not need that, and
'synchronized' may need extra overhead

>>3.1
Reader.nextBlock() is later added for file merge. So the normal reader
should not use this method.
>>3.2.
True.

On Fri, Mar 18, 2011 at 8:30 AM, Krishna Kumar  wrote:
> Hello,
>
>I was looking into the RCFile format, esp when used with compression; a
> picture of the file layout as I understand it in this case is attached.
>
>Some queries/potential issues:
>
>1. RCFile makes a claim of being sequence file compatible; but the
> recordLength is not the actual on-disk length of the record. As shown in the
> picture, it is the uncompressed key length plus the compressed value length.
> Similarly, the next field - key length - is not the on-disk length of the
> compressed key.
>
>2. Record Length is also used for seeking on the inputstream. See
> Reader.seekToNextKeyBuffer(). Since record length is overstated for
> compressed records, this can result in incorrect positioning.
>
>3. Thread-Safety: Is the RCFile.Reader class meant to be thread-safe?
> Some public methods are marked synchronized which gives that appearance but
> there are a few thread-safety issues I think.
>
>3.1 Other public methods, such as Reader.nextBlock() are not
> synchronized which operate on the same data structures.
>
>3.2. Callbacks such as LazyDecompressionCallbackImpl.decompress
> operates on the valuebuffer currentValue, which can be simultaneously
> modified by the public methods on the Reader.
>
> Cheers,
>  Krishna
>
>

[jira] Assigned: (HIVE-2065) RCFile issues

[Krishna Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Kumar reassigned HIVE-2065:
---

Assignee: Krishna Kumar

> RCFile issues
> -
>
> Key: HIVE-2065
> URL: https://issues.apache.org/jira/browse/HIVE-2065
> Project: Hive
>  Issue Type: Bug
>Reporter: Krishna Kumar
>Assignee: Krishna Kumar
>Priority: Minor
> Attachments: Slide1.png
>
>
> Some potential issues with RCFile
> 1. Remove unwanted synchronized modifiers on the methods of RCFile. As per 
> yongqiang he, the class is not meant to be thread-safe (and it is not). Might 
> as well get rid of the confusing and performance-impacting lock acquisitions.
> 2. Record Length overstated for compressed files. IIUC, the key compression 
> happens after we have written the record length.
> {code}
>   int keyLength = key.getSize();
>   if (keyLength < 0) {
> throw new IOException("negative length keys not allowed: " + key);
>   }
>   out.writeInt(keyLength + valueLength); // total record length
>   out.writeInt(keyLength); // key portion length
>   if (!isCompressed()) {
> out.writeInt(keyLength);
> key.write(out); // key
>   } else {
> keyCompressionBuffer.reset();
> keyDeflateFilter.resetState();
> key.write(keyDeflateOut);
> keyDeflateOut.flush();
> keyDeflateFilter.finish();
> int compressedKeyLen = keyCompressionBuffer.getLength();
> out.writeInt(compressedKeyLen);
> out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
>   }
> {code}
> 3. For sequence file compatibility, the compressed key length should be the 
> next field to record length, not the uncompressed key length.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2065) RCFile issues

[Krishna Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Krishna Kumar updated HIVE-2065:


Attachment: Slide1.png

Compressed RCFile Layout

> RCFile issues
> -
>
> Key: HIVE-2065
> URL: https://issues.apache.org/jira/browse/HIVE-2065
> Project: Hive
>  Issue Type: Bug
>Reporter: Krishna Kumar
>Priority: Minor
> Attachments: Slide1.png
>
>
> Some potential issues with RCFile
> 1. Remove unwanted synchronized modifiers on the methods of RCFile. As per 
> yongqiang he, the class is not meant to be thread-safe (and it is not). Might 
> as well get rid of the confusing and performance-impacting lock acquisitions.
> 2. Record Length overstated for compressed files. IIUC, the key compression 
> happens after we have written the record length.
> {code}
>   int keyLength = key.getSize();
>   if (keyLength < 0) {
> throw new IOException("negative length keys not allowed: " + key);
>   }
>   out.writeInt(keyLength + valueLength); // total record length
>   out.writeInt(keyLength); // key portion length
>   if (!isCompressed()) {
> out.writeInt(keyLength);
> key.write(out); // key
>   } else {
> keyCompressionBuffer.reset();
> keyDeflateFilter.resetState();
> key.write(keyDeflateOut);
> keyDeflateOut.flush();
> keyDeflateFilter.finish();
> int compressedKeyLen = keyCompressionBuffer.getLength();
> out.writeInt(compressedKeyLen);
> out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
>   }
> {code}
> 3. For sequence file compatibility, the compressed key length should be the 
> next field to record length, not the uncompressed key length.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (HIVE-2065) RCFile issues

[Krishna Kumar (JIRA)]

RCFile issues
-

 Key: HIVE-2065
 URL: https://issues.apache.org/jira/browse/HIVE-2065
 Project: Hive
  Issue Type: Bug
Reporter: Krishna Kumar
Priority: Minor


Some potential issues with RCFile

1. Remove unwanted synchronized modifiers on the methods of RCFile. As per 
yongqiang he, the class is not meant to be thread-safe (and it is not). Might 
as well get rid of the confusing and performance-impacting lock acquisitions.

2. Record Length overstated for compressed files. IIUC, the key compression 
happens after we have written the record length.

{code}
  int keyLength = key.getSize();
  if (keyLength < 0) {
throw new IOException("negative length keys not allowed: " + key);
  }

  out.writeInt(keyLength + valueLength); // total record length
  out.writeInt(keyLength); // key portion length
  if (!isCompressed()) {
out.writeInt(keyLength);
key.write(out); // key
  } else {
keyCompressionBuffer.reset();
keyDeflateFilter.resetState();
key.write(keyDeflateOut);
keyDeflateOut.flush();
keyDeflateFilter.finish();
int compressedKeyLen = keyCompressionBuffer.getLength();
out.writeInt(compressedKeyLen);
out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
  }
{code}

3. For sequence file compatibility, the compressed key length should be the 
next field to record length, not the uncompressed key length.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2064:
-

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

Committed to trunk and branch-0.7. Thanks Eli!

> Make call to SecurityUtil.getServerPrincipal unambiguous
> 
>
> Key: HIVE-2064
> URL: https://issues.apache.org/jira/browse/HIVE-2064
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 0.7.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: hive-2064-1.patch
>
>
> HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes 
> null for the 2nd arg. When building against the hadoop security branch this 
> is a compilation error as it matches the signatures of both 
> getServerPrincipal methods (one takes a String for the 2nd arg, one an 
> InetAddress). This call needs to be made unambiguous eg by passing "0.0.0.0" 
> instead of null, which per the getServerPrincipal javadoc is equivalent:
> {quote}
> It replaces hostname pattern with hostname, which should be
>* fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
>* dynamically looked-up fqdn of the current host instead.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2062) HivePreparedStatement.executeImmediate always throw exception

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008700#comment-13008700
 ] 

Namit Jain commented on HIVE-2062:
--

+1

> HivePreparedStatement.executeImmediate always throw exception
> -
>
> Key: HIVE-2062
> URL: https://issues.apache.org/jira/browse/HIVE-2062
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Assignee: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2062.patch
>
>
> executeImmediate:
> try {
>   clearWarnings();
>   resultSet = null;
>   client.execute(sql);
> }
> but:
>   public void clearWarnings() throws SQLException {
> // TODO Auto-generated method stub
> throw new SQLException("Method not supported");
>   }
> in result all calls executeQuery() for prepared statement return exception

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Attachment: HIVE-2011.5.patch.txt

HIVE-2011.5.patch.txt:

* Backout whitespace changes to package.jdo (not sure how those slipped in...)


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt, 
> HIVE-2011.5.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-2011: upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/471/
---

(Updated 2011-03-18 17:30:36.904588)


Review request for hive.


Summary
---

Review request for HIVE-2011 and HIVE-2010.


This addresses bugs HIVE-2010 and HIVE-2011.
https://issues.apache.org/jira/browse/HIVE-2010
https://issues.apache.org/jira/browse/HIVE-2011


Diffs (updated)
-

  metastore/scripts/upgrade/derby/001-HIVE-972.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/002-HIVE-1068.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/003-HIVE-675.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/004-HIVE-1364.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/005-HIVE-417.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/006-HIVE-1823.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/007-HIVE-78.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/README 506eb54 
  metastore/scripts/upgrade/derby/hive-schema-0.3.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.1.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.5.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.5.0-to-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0-to-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0.derby.sql 77a0f39 
  metastore/scripts/upgrade/derby/upgrade-0.7.0.derby.sql ccad4e7 
  metastore/scripts/upgrade/mysql/001-HIVE-972.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/002-HIVE-1068.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/003-HIVE-675.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/004-HIVE-1364.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/005-HIVE-417.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/006-HIVE-1823.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/007-HIVE-78.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/README 3d4360f 
  metastore/scripts/upgrade/mysql/hive-schema-0.3.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.1.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.5.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.5.0-to-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0-to-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0.mysql.sql 8cbd501 
  metastore/scripts/upgrade/mysql/upgrade-0.7.0.mysql.sql aab664e 

Diff: https://reviews.apache.org/r/471/diff


Testing
---


Thanks,

Carl

Re: Review Request: HIVE-2011: upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/471/
---

(Updated 2011-03-18 17:27:05.515537)


Review request for hive.


Summary
---

Review request for HIVE-2011 and HIVE-2010.


This addresses bugs HIVE-2010 and HIVE-2011.
https://issues.apache.org/jira/browse/HIVE-2010
https://issues.apache.org/jira/browse/HIVE-2011


Diffs (updated)
-

  metastore/scripts/upgrade/derby/001-HIVE-972.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/002-HIVE-1068.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/003-HIVE-675.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/004-HIVE-1364.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/005-HIVE-417.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/006-HIVE-1823.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/007-HIVE-78.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/README 506eb54 
  metastore/scripts/upgrade/derby/hive-schema-0.3.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.4.1.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.5.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.5.0-to-0.6.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0-to-0.7.0.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/upgrade-0.6.0.derby.sql 77a0f39 
  metastore/scripts/upgrade/derby/upgrade-0.7.0.derby.sql ccad4e7 
  metastore/scripts/upgrade/mysql/001-HIVE-972.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/002-HIVE-1068.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/003-HIVE-675.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/004-HIVE-1364.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/005-HIVE-417.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/006-HIVE-1823.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/007-HIVE-78.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/README 3d4360f 
  metastore/scripts/upgrade/mysql/hive-schema-0.3.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.4.1.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.5.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.5.0-to-0.6.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0-to-0.7.0.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/upgrade-0.6.0.mysql.sql 8cbd501 
  metastore/scripts/upgrade/mysql/upgrade-0.7.0.mysql.sql aab664e 
  metastore/src/model/package.jdo 88ac402 

Diff: https://reviews.apache.org/r/471/diff


Testing
---


Thanks,

Carl

[jira] Commented: (HIVE-2003) LOAD compilation does not set the outputs during semantic analysis resulting in no authorization checks being done for it.

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008697#comment-13008697
 ] 

Namit Jain commented on HIVE-2003:
--

Can you regenerate the patch ?
I was having trouble applying it ?

Also, can you create a review-board request ?

> LOAD compilation does not set the outputs during semantic analysis resulting 
> in no authorization checks being done for it.
> --
>
> Key: HIVE-2003
> URL: https://issues.apache.org/jira/browse/HIVE-2003
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Krishna Kumar
>Assignee: Krishna Kumar
>Priority: Minor
> Attachments: HIVE-2003.patch.txt
>
>
> The table/partition being loaded is not being added to outputs in the 
> LoadSemanticAnalyzer.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2003) LOAD compilation does not set the outputs during semantic analysis resulting in no authorization checks being done for it.

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-2003:
-

Status: Open  (was: Patch Available)

> LOAD compilation does not set the outputs during semantic analysis resulting 
> in no authorization checks being done for it.
> --
>
> Key: HIVE-2003
> URL: https://issues.apache.org/jira/browse/HIVE-2003
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Krishna Kumar
>Assignee: Krishna Kumar
>Priority: Minor
> Attachments: HIVE-2003.patch.txt
>
>
> The table/partition being loaded is not being added to outputs in the 
> LoadSemanticAnalyzer.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Attachment: HIVE-2011.4.patch.txt

HIVE-2011.4.patch.txt:

* Fix capitalization of table names in MySQL schema dumps.

@John: I created these schema's using DataNucleus's schematool utility. I think 
this capitalization issue is actually a bug in DataNucleus.


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Issue Comment Edited: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008696#comment-13008696
 ] 

Carl Steinbach edited comment on HIVE-2011 at 3/19/11 12:24 AM:


HIVE-2011.4.patch.txt:

* Fix capitalization of table names in MySQL schema dumps.

@John: I created these schemas using DataNucleus's schematool utility. I think 
this capitalization issue is actually a bug in DataNucleus.


  was (Author: cwsteinbach):
HIVE-2011.4.patch.txt:

* Fix capitalization of table names in MySQL schema dumps.

@John: I created these schema's using DataNucleus's schematool utility. I think 
this capitalization issue is actually a bug in DataNucleus.

  
> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt, HIVE-2011.4.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008690#comment-13008690
 ] 

John Sichi commented on HIVE-2011:
--

@Carl, I ran hive-schema-0.6.0.mysql.sql and two of the tablenames (dbs and 
types) came out in lowercase; they should be ALLCAPS.

{noformat}
mysql> show tables;
+-+
| Tables_in_jvs_metastore |
+-+
| BUCKETING_COLS  | 
| COLUMNS | 
| PARTITIONS  | 
| PARTITION_KEYS  | 
| PARTITION_KEY_VALS  | 
| PARTITION_PARAMS| 
| SDS | 
| SD_PARAMS   | 
| SERDES  | 
| SERDE_PARAMS| 
| SORT_COLS   | 
| TABLE_PARAMS| 
| TBLS| 
| TYPE_FIELDS | 
| dbs | 
| types   | 
+-+
{noformat}


> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2051) getInputSummary() to call FileSystem.getContentSummary() in parallel

[Siying Dong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008676#comment-13008676
 ] 

Siying Dong commented on HIVE-2051:
---

I still feel that it's too dangerous to ignore InterruptedException : 
http://download.oracle.com/javase/1.5.0/docs/api/java/lang/Thread.html#interrupt().
 It sounds like command to shutdown the thread smoothly. In that case, we 
should have special reason if we don't follow the command.

> getInputSummary() to call FileSystem.getContentSummary() in parallel
> 
>
> Key: HIVE-2051
> URL: https://issues.apache.org/jira/browse/HIVE-2051
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.1.patch, HIVE-2051.2.patch, HIVE-2051.3.patch, 
> HIVE-2051.4.patch
>
>
> getInputSummary() now call FileSystem.getContentSummary() one by one, which 
> can be extremely slow when the number of input paths are huge. By calling 
> those functions in parallel, we can cut latency in most cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Jenkins build is back to normal : Hive-trunk-h0.20 #621

[Apache Hudson Server]

See 

[jira] Commented: (HIVE-2051) getInputSummary() to call FileSystem.getContentSummary() in parallel

[Siying Dong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008657#comment-13008657
 ] 

Siying Dong commented on HIVE-2051:
---

Joydeep, sorry you were talking about ExecutionException about 
InterruptedException. In that case, I'll just rethrow it.

> getInputSummary() to call FileSystem.getContentSummary() in parallel
> 
>
> Key: HIVE-2051
> URL: https://issues.apache.org/jira/browse/HIVE-2051
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.1.patch, HIVE-2051.2.patch, HIVE-2051.3.patch, 
> HIVE-2051.4.patch
>
>
> getInputSummary() now call FileSystem.getContentSummary() one by one, which 
> can be extremely slow when the number of input paths are huge. By calling 
> those functions in parallel, we can cut latency in most cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2051) getInputSummary() to call FileSystem.getContentSummary() in parallel

[Siying Dong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008654#comment-13008654
 ] 

Siying Dong commented on HIVE-2051:
---

Joydeep, I'm nervous about putting a synchronized object put in context and 
have it available everywhere. I'll made this static method synchronized, so 
that no parallel call to it.

I'm not sure whether I understand correctly but it seems that 
ExecutionException indicates that the waiting thread gets the signal, instead 
of the thread being waited. It does more sound like someone wants to kill the 
process. What we can do if we don't ignore nor throw ExecutionException? We 
only have 3 realistic choices: always throw, always ignore or continue to wait 
as a retry.. To me, always throwing sounds a better idea, as when we catch an 
exception that we don't know how to handle it, throwing it sounds the safest 
way to go. What's your suggestion to handle it?

I'll remove the awaitTermination()

> getInputSummary() to call FileSystem.getContentSummary() in parallel
> 
>
> Key: HIVE-2051
> URL: https://issues.apache.org/jira/browse/HIVE-2051
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.1.patch, HIVE-2051.2.patch, HIVE-2051.3.patch, 
> HIVE-2051.4.patch
>
>
> getInputSummary() now call FileSystem.getContentSummary() one by one, which 
> can be extremely slow when the number of input paths are huge. By calling 
> those functions in parallel, we can cut latency in most cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-1976) Exception should be thrown when invalid jar,file,archive is given to add command

[Ning Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ning Zhang updated HIVE-1976:
-

   Resolution: Fixed
Fix Version/s: 0.8.0
 Release Note: Unit tests passed and committed. Thanks Chinna!
 Hadoop Flags: [Reviewed]
   Status: Resolved  (was: Patch Available)

> Exception should be thrown when invalid jar,file,archive is given to add 
> command
> 
>
> Key: HIVE-1976
> URL: https://issues.apache.org/jira/browse/HIVE-1976
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.5.0, 0.7.0
> Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise 
> Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Fix For: 0.8.0
>
> Attachments: HIVE-1976.2.patch, HIVE-1976.3.patch, HIVE-1976.patch
>
>
> When executed add command with non existing jar it should throw exception 
> through   HiveStatement
> Ex:
> {noformat}
>   add jar /root/invalidpath/testjar.jar
> {noformat}
> Here testjar.jar is not exist so it should throw exception.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-1976) Exception should be thrown when invalid jar,file,archive is given to add command

[Ning Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008652#comment-13008652
 ] 

Ning Zhang commented on HIVE-1976:
--

+1.

Sorry about the delay. 

> Exception should be thrown when invalid jar,file,archive is given to add 
> command
> 
>
> Key: HIVE-1976
> URL: https://issues.apache.org/jira/browse/HIVE-1976
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.5.0, 0.7.0
> Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise 
> Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
>Reporter: Chinna Rao Lalam
>Assignee: Chinna Rao Lalam
> Attachments: HIVE-1976.2.patch, HIVE-1976.3.patch, HIVE-1976.patch
>
>
> When executed add command with non existing jar it should throw exception 
> through   HiveStatement
> Ex:
> {noformat}
>   add jar /root/invalidpath/testjar.jar
> {noformat}
> Here testjar.jar is not exist so it should throw exception.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2052) PostHook and PreHook API to add flag to indicate it is pre or post hook plus cache for content summary

[Siying Dong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siying Dong updated HIVE-2052:
--

Attachment: HIVE-2052.3.patch

> PostHook and PreHook API to add flag to indicate it is pre or post hook plus 
> cache for content summary
> --
>
> Key: HIVE-2052
> URL: https://issues.apache.org/jira/browse/HIVE-2052
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.3.patch, HIVE-2052.1.patch, HIVE-2052.2.patch, 
> HIVE-2052.3.patch
>
>
> This will allow hooks to share some information better and reduce their 
> latency

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2052) PostHook and PreHook API to add flag to indicate it is pre or post hook plus cache for content summary

[Siying Dong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008641#comment-13008641
 ] 

Siying Dong commented on HIVE-2052:
---

the name is wrong but the content is right. I'll fix it now.

> PostHook and PreHook API to add flag to indicate it is pre or post hook plus 
> cache for content summary
> --
>
> Key: HIVE-2052
> URL: https://issues.apache.org/jira/browse/HIVE-2052
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.3.patch, HIVE-2052.1.patch, HIVE-2052.2.patch
>
>
> This will allow hooks to share some information better and reduce their 
> latency

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Attachment: HIVE-2011.3.patch.txt

HIVE-2011.3.patch.txt: 

* Fixed errors in Derby schema scripts caused by accidentally picking up wrong 
DataNucleus (and wrong IdentifierFactory) during original schema generation run.
* Added a couple usage notes to the Derby and MySQL READMEs related to the ij 
and mysqldump utilities.

Testing:
# Created Derby and MySQL schemas for 0.3.0 through 0.7.0 by starting Hive with 
autoCreateSchema=true.
# Applied hive-schema-x.y.z.xxx.sql scripts to each of the previously created 
DBs. Verified that these scripts complete without error, and that the resulting 
DB schemas contain all tables.
# Applied the upgrade scripts to validated 0.5.0 and 0.6.0 DBs for Derby and 
MySQL, and verified that all scripts complete without error.
# Validated the upgraded DBs by running Hive against the upgraded DB schema 
with datanucleus.autoCreateSchema=false and datanucleus.validateSchema=true




> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt, HIVE-2011.3.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2011) upgrade-0.6.0.mysql.sql script attempts to increase size of PK COLUMNS.TYPE_NAME to 4000

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2011:
-

Attachment: HIVE-2011.2.patch.txt

> upgrade-0.6.0.mysql.sql script attempts to increase size of PK 
> COLUMNS.TYPE_NAME to 4000
> 
>
> Key: HIVE-2011
> URL: https://issues.apache.org/jira/browse/HIVE-2011
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.6.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: HIVE-2011.1.patch.txt, HIVE-2011.2.patch.txt, 
> HIVE-2011.2.patch.txt
>
>
> {code}
> # mysql flumenewresearch < upgrade-0.6.0.mysql.sql 
> ERROR 1071 (42000) at line 16: Specified key was too long; max key length is 
> 767 bytes
> {code}
> Here's the cause of the problem from upgrade-0.6.0.mysql.sql:
> {code}
> ...
> ALTER TABLE `COLUMNS` MODIFY `TYPE_NAME` VARCHAR(4000);
> ...
> ALTER TABLE `COLUMNS` DROP PRIMARY KEY;
> ALTER TABLE `COLUMNS` ADD PRIMARY KEY (`SD_ID`, `COLUMN_NAME`);
> ...
> {code}
> We need to make sure that the PK on COLUMNS.TYPE_NAME is dropped before the 
> size of the column is bumped to 4000.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Jenkins build is back to normal : Hive-0.7.0-h0.20 #43

[Apache Hudson Server]

See 

Build failed in Jenkins: Hive-trunk-h0.20 #620

[Apache Hudson Server]

See 

--
[...truncated 27976 lines...]
[junit] PREHOOK: type: CREATETABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: CREATETABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] PREHOOK: type: LOAD
[junit] Copying data from 

[junit] Loading data to table default.testhivedrivertable
[junit] POSTHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] POSTHOOK: type: LOAD
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select count(1) as cnt from testhivedrivertable
[junit] PREHOOK: type: QUERY
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_12-16-01_628_4311003007288564311/-mr-1
[junit] Total MapReduce jobs = 1
[junit] Launching Job 1 out of 1
[junit] Number of reduce tasks determined at compile time: 1
[junit] In order to change the average load for a reducer (in bytes):
[junit]   set hive.exec.reducers.bytes.per.reducer=
[junit] In order to limit the maximum number of reducers:
[junit]   set hive.exec.reducers.max=
[junit] In order to set a constant number of reducers:
[junit]   set mapred.reduce.tasks=
[junit] Job running in-process (local Hadoop)
[junit] Hadoop job information for null: number of mappers: 0; number of 
reducers: 0
[junit] 2011-03-18 12:16:04,715 null map = 100%,  reduce = 100%
[junit] Ended Job = job_local_0001
[junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable
[junit] POSTHOOK: type: QUERY
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_12-16-01_628_4311003007288564311/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: CREATETABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: CREATETABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] PREHOOK: type: LOAD
[junit] Copying data from 

[junit] Loading data to table default.testhivedrivertable
[junit] POSTHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] POSTHOOK: type: LOAD
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select * from testhivedrivertable limit 10
[junit] PREHOOK: type: QUERY
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_12-16-06_212_7468792470509886855/-mr-1
[junit] POSTHOOK: query: select * from testhivedrivertable limit 10
[junit] POSTHOOK: type: QUERY
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2011-03-18_12-16-06_212_7468792470509886855/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: defa

[jira] Commented: (HIVE-2061) Create a hive_contrib.jar symlink to hive-contrib-{version}.jar for backward compatibility

[Paul Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008585#comment-13008585
 ] 

Paul Yang commented on HIVE-2061:
-

Committed. Thanks Ning!

> Create a hive_contrib.jar symlink to hive-contrib-{version}.jar for backward 
> compatibility
> --
>
> Key: HIVE-2061
> URL: https://issues.apache.org/jira/browse/HIVE-2061
> Project: Hive
>  Issue Type: Bug
>Reporter: Ning Zhang
>Assignee: Ning Zhang
>Priority: Minor
> Attachments: HIVE-2061.patch
>
>
> We have seen a use case where in the user's script, it run 'add jar 
> hive_contrib.jar'. Since Hive has moved the jar file to be 
> hive-contrib-{version}.jar, it introduced backward incompatibility. If we as 
> the user to change the script and when Hive upgrade version again, the user 
> need to change the script again. Creating a symlink seems to be the best 
> solution. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Eli Collins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins updated HIVE-2064:
--

Status: Patch Available  (was: Open)

> Make call to SecurityUtil.getServerPrincipal unambiguous
> 
>
> Key: HIVE-2064
> URL: https://issues.apache.org/jira/browse/HIVE-2064
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 0.7.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: hive-2064-1.patch
>
>
> HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes 
> null for the 2nd arg. When building against the hadoop security branch this 
> is a compilation error as it matches the signatures of both 
> getServerPrincipal methods (one takes a String for the 2nd arg, one an 
> InetAddress). This call needs to be made unambiguous eg by passing "0.0.0.0" 
> instead of null, which per the getServerPrincipal javadoc is equivalent:
> {quote}
> It replaces hostname pattern with hostname, which should be
>* fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
>* dynamically looked-up fqdn of the current host instead.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2063) jdbc return only 1 collumn

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-2063:
-

Status: Open  (was: Patch Available)

> jdbc return only 1 collumn
> --
>
> Key: HIVE-2063
> URL: https://issues.apache.org/jira/browse/HIVE-2063
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2063.patch
>
>
> we not set separator for data and all data return in first columns and all 
> other fields set NULL
> addition we get WARNING: Missing fields! Expected 27 fields but only got 1! 
> Ignoring similar problems.
> it's regresion after HIVE-1378 
> bug:
> use delimiter '\t' for fields on server side
> use default delimiter with code '1' on client side

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2063) jdbc return only 1 collumn

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008564#comment-13008564
 ] 

Namit Jain commented on HIVE-2063:
--

Can you use Utilities.tabCode instead of hardcoding "9" ?



> jdbc return only 1 collumn
> --
>
> Key: HIVE-2063
> URL: https://issues.apache.org/jira/browse/HIVE-2063
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2063.patch
>
>
> we not set separator for data and all data return in first columns and all 
> other fields set NULL
> addition we get WARNING: Missing fields! Expected 27 fields but only got 1! 
> Ignoring similar problems.
> it's regresion after HIVE-1378 
> bug:
> use delimiter '\t' for fields on server side
> use default delimiter with code '1' on client side

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: RCFile - some queries

[yongqiang he]

>> but the recordLength is not the actual on-disk length of the record.
It is acutal on-disk length. It is compressed key length plus the
compressed value length

>>Similarly, the next field - key length - is not the on-disk length of the 
>>compressed key.

There are two keyLengths, one is compressed key length, the other is
uncompressed keyLength

For 2, it wo't be a problem. record length is compressed length

>>Thread-Safety.
It is not thread safe. Application should do it themselves.
 It is initially designed for Hive. Thread safety is there at first
time, and then removed because Hive does not need that, and
'synchronized' may need extra overhead

>>3.1
Reader.nextBlock() is later added for file merge. So the normal reader
should not use this method.
>>3.2.
True.

On Fri, Mar 18, 2011 at 8:30 AM, Krishna Kumar  wrote:
> Hello,
>
>    I was looking into the RCFile format, esp when used with compression; a
> picture of the file layout as I understand it in this case is attached.
>
>    Some queries/potential issues:
>
>    1. RCFile makes a claim of being sequence file compatible; but the
> recordLength is not the actual on-disk length of the record. As shown in the
> picture, it is the uncompressed key length plus the compressed value length.
> Similarly, the next field - key length - is not the on-disk length of the
> compressed key.
>
>    2. Record Length is also used for seeking on the inputstream. See
> Reader.seekToNextKeyBuffer(). Since record length is overstated for
> compressed records, this can result in incorrect positioning.
>
>    3. Thread-Safety: Is the RCFile.Reader class meant to be thread-safe?
> Some public methods are marked synchronized which gives that appearance but
> there are a few thread-safety issues I think.
>
>        3.1 Other public methods, such as Reader.nextBlock() are not
> synchronized which operate on the same data structures.
>
>        3.2. Callbacks such as LazyDecompressionCallbackImpl.decompress
> operates on the valuebuffer currentValue, which can be simultaneously
> modified by the public methods on the Reader.
>
> Cheers,
>  Krishna
>
>

[jira] Commented: (HIVE-2052) PostHook and PreHook API to add flag to indicate it is pre or post hook plus cache for content summary

[Joydeep Sen Sarma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008558#comment-13008558
 ] 

Joydeep Sen Sarma commented on HIVE-2052:
-

looks like the new patch is for HIVE-2051 not HIVE-2052!

> PostHook and PreHook API to add flag to indicate it is pre or post hook plus 
> cache for content summary
> --
>
> Key: HIVE-2052
> URL: https://issues.apache.org/jira/browse/HIVE-2052
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.3.patch, HIVE-2052.1.patch, HIVE-2052.2.patch
>
>
> This will allow hooks to share some information better and reduce their 
> latency

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2051) getInputSummary() to call FileSystem.getContentSummary() in parallel

[Joydeep Sen Sarma (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008549#comment-13008549
 ] 

Joydeep Sen Sarma commented on HIVE-2051:
-

Siying - i think we shouldn't ignore ExecutionException. The best part of 
checking for each task status seems to be that we can find out if any of them 
failed (indicated by ExecutionException). Also we can remove the 
executor.awaitTermination() call as well (same feedback as the comments above).

also - do you want to make the core of this routine synchronized (perhaps on 
the context object - which is one per query)? there really is no point running 
more than one of these per query at a time. (we can move this whole routine to 
the Context object if that seems like a better place (or at least make the call 
from the Context object where it can be marked as a synchronized method).

otherwise looks good. please upload a new patch and i will test and commit.

> getInputSummary() to call FileSystem.getContentSummary() in parallel
> 
>
> Key: HIVE-2051
> URL: https://issues.apache.org/jira/browse/HIVE-2051
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
>Priority: Minor
> Attachments: HIVE-2051.1.patch, HIVE-2051.2.patch, HIVE-2051.3.patch, 
> HIVE-2051.4.patch
>
>
> getInputSummary() now call FileSystem.getContentSummary() one by one, which 
> can be extremely slow when the number of input paths are huge. By calling 
> those functions in parallel, we can cut latency in most cases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008543#comment-13008543
 ] 

Carl Steinbach commented on HIVE-2064:
--

+1. Will commit if tests pass.

> Make call to SecurityUtil.getServerPrincipal unambiguous
> 
>
> Key: HIVE-2064
> URL: https://issues.apache.org/jira/browse/HIVE-2064
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 0.7.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: hive-2064-1.patch
>
>
> HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes 
> null for the 2nd arg. When building against the hadoop security branch this 
> is a compilation error as it matches the signatures of both 
> getServerPrincipal methods (one takes a String for the 2nd arg, one an 
> InetAddress). This call needs to be made unambiguous eg by passing "0.0.0.0" 
> instead of null, which per the getServerPrincipal javadoc is equivalent:
> {quote}
> It replaces hostname pattern with hostname, which should be
>* fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
>* dynamically looked-up fqdn of the current host instead.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Todd Lipcon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008535#comment-13008535
 ] 

Todd Lipcon commented on HIVE-2064:
---

This broke with HADOOP-7104.

Patch looks good to me.

> Make call to SecurityUtil.getServerPrincipal unambiguous
> 
>
> Key: HIVE-2064
> URL: https://issues.apache.org/jira/browse/HIVE-2064
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 0.7.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: hive-2064-1.patch
>
>
> HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes 
> null for the 2nd arg. When building against the hadoop security branch this 
> is a compilation error as it matches the signatures of both 
> getServerPrincipal methods (one takes a String for the 2nd arg, one an 
> InetAddress). This call needs to be made unambiguous eg by passing "0.0.0.0" 
> instead of null, which per the getServerPrincipal javadoc is equivalent:
> {quote}
> It replaces hostname pattern with hostname, which should be
>* fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
>* dynamically looked-up fqdn of the current host instead.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-2054: fix for IOException on the jdbc driver on windows.

[Bennie Schut]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/513/
---

(Updated 2011-03-18 01:08:21.290554)


Review request for hive.


Changes
---

New patch because of merge from HIVE-1815.


Summary
---

HIVE-2054: fix for IOException on the jdbc driver on windows.


This addresses bug HIVE-2054.
https://issues.apache.org/jira/browse/HIVE-2054


Diffs (updated)
-

  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 1082843 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java 
1082843 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java 1082843 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcSessionState.java 1081782 

Diff: https://reviews.apache.org/r/513/diff


Testing
---


Thanks,

Bennie

[jira] Updated: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Eli Collins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eli Collins updated HIVE-2064:
--

Attachment: hive-2064-1.patch

Patch attached.

> Make call to SecurityUtil.getServerPrincipal unambiguous
> 
>
> Key: HIVE-2064
> URL: https://issues.apache.org/jira/browse/HIVE-2064
> Project: Hive
>  Issue Type: Bug
>  Components: Security
>Affects Versions: 0.7.0
>Reporter: Eli Collins
>Assignee: Eli Collins
>Priority: Blocker
> Fix For: 0.7.0
>
> Attachments: hive-2064-1.patch
>
>
> HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes 
> null for the 2nd arg. When building against the hadoop security branch this 
> is a compilation error as it matches the signatures of both 
> getServerPrincipal methods (one takes a String for the 2nd arg, one an 
> InetAddress). This call needs to be made unambiguous eg by passing "0.0.0.0" 
> instead of null, which per the getServerPrincipal javadoc is equivalent:
> {quote}
> It replaces hostname pattern with hostname, which should be
>* fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
>* dynamically looked-up fqdn of the current host instead.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (HIVE-2064) Make call to SecurityUtil.getServerPrincipal unambiguous

[Eli Collins (JIRA)]

Make call to SecurityUtil.getServerPrincipal unambiguous


 Key: HIVE-2064
 URL: https://issues.apache.org/jira/browse/HIVE-2064
 Project: Hive
  Issue Type: Bug
  Components: Security
Affects Versions: 0.7.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Blocker
 Fix For: 0.7.0


HadoopThriftAuthBridge20S calls SecurityUtil.getServerPrincipal and passes null 
for the 2nd arg. When building against the hadoop security branch this is a 
compilation error as it matches the signatures of both getServerPrincipal 
methods (one takes a String for the 2nd arg, one an InetAddress). This call 
needs to be made unambiguous eg by passing "0.0.0.0" instead of null, which per 
the getServerPrincipal javadoc is equivalent:

{quote}
It replaces hostname pattern with hostname, which should be
   * fully-qualified domain name. If hostname is null or "0.0.0.0", it uses
   * dynamically looked-up fqdn of the current host instead.
{quote}


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

RCFile - some queries

[Krishna Kumar]

Hello,

I was looking into the RCFile format, esp when used with compression; a
picture of the file layout as I understand it in this case is attached.

Some queries/potential issues:

1. RCFile makes a claim of being sequence file compatible; but the
recordLength is not the actual on-disk length of the record. As shown in the
picture, it is the uncompressed key length plus the compressed value length.
Similarly, the next field - key length - is not the on-disk length of the
compressed key.

2. Record Length is also used for seeking on the inputstream. See
Reader.seekToNextKeyBuffer(). Since record length is overstated for
compressed records, this can result in incorrect positioning.

3. Thread-Safety: Is the RCFile.Reader class meant to be thread-safe?
Some public methods are marked synchronized which gives that appearance but
there are a few thread-safety issues I think.

3.1 Other public methods, such as Reader.nextBlock() are not
synchronized which operate on the same data structures.

3.2. Callbacks such as LazyDecompressionCallbackImpl.decompress
operates on the valuebuffer currentValue, which can be simultaneously
modified by the public methods on the Reader.

Cheers,
 Krishna

[jira] Updated: (HIVE-2063) jdbc return only 1 collumn

[Alexey Diomin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Diomin updated HIVE-2063:


Status: Patch Available  (was: Open)

> jdbc return only 1 collumn
> --
>
> Key: HIVE-2063
> URL: https://issues.apache.org/jira/browse/HIVE-2063
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2063.patch
>
>
> we not set separator for data and all data return in first columns and all 
> other fields set NULL
> addition we get WARNING: Missing fields! Expected 27 fields but only got 1! 
> Ignoring similar problems.
> it's regresion after HIVE-1378 
> bug:
> use delimiter '\t' for fields on server side
> use default delimiter with code '1' on client side

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (HIVE-2063) jdbc return only 1 collumn

[Alexey Diomin (JIRA)]

jdbc return only 1 collumn
--

 Key: HIVE-2063
 URL: https://issues.apache.org/jira/browse/HIVE-2063
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 0.7.0
Reporter: Alexey Diomin
Priority: Critical
 Attachments: HIVE-2063.patch

we not set separator for data and all data return in first columns and all 
other fields set NULL

addition we get WARNING: Missing fields! Expected 27 fields but only got 1! 
Ignoring similar problems.

it's regresion after HIVE-1378 

bug:

use delimiter '\t' for fields on server side

use default delimiter with code '1' on client side

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2063) jdbc return only 1 collumn

[Alexey Diomin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Diomin updated HIVE-2063:


Attachment: HIVE-2063.patch

> jdbc return only 1 collumn
> --
>
> Key: HIVE-2063
> URL: https://issues.apache.org/jira/browse/HIVE-2063
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2063.patch
>
>
> we not set separator for data and all data return in first columns and all 
> other fields set NULL
> addition we get WARNING: Missing fields! Expected 27 fields but only got 1! 
> Ignoring similar problems.
> it's regresion after HIVE-1378 
> bug:
> use delimiter '\t' for fields on server side
> use default delimiter with code '1' on client side

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2062) HivePreparedStatement.executeImmediate always throw exception

[Alexey Diomin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Diomin updated HIVE-2062:


Attachment: HIVE-2062.patch

> HivePreparedStatement.executeImmediate always throw exception
> -
>
> Key: HIVE-2062
> URL: https://issues.apache.org/jira/browse/HIVE-2062
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2062.patch
>
>
> executeImmediate:
> try {
>   clearWarnings();
>   resultSet = null;
>   client.execute(sql);
> }
> but:
>   public void clearWarnings() throws SQLException {
> // TODO Auto-generated method stub
> throw new SQLException("Method not supported");
>   }
> in result all calls executeQuery() for prepared statement return exception

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2062) HivePreparedStatement.executeImmediate always throw exception

[Alexey Diomin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Diomin updated HIVE-2062:


Assignee: Alexey Diomin
  Status: Patch Available  (was: Open)

> HivePreparedStatement.executeImmediate always throw exception
> -
>
> Key: HIVE-2062
> URL: https://issues.apache.org/jira/browse/HIVE-2062
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.7.0
>Reporter: Alexey Diomin
>Assignee: Alexey Diomin
>Priority: Critical
> Attachments: HIVE-2062.patch
>
>
> executeImmediate:
> try {
>   clearWarnings();
>   resultSet = null;
>   client.execute(sql);
> }
> but:
>   public void clearWarnings() throws SQLException {
> // TODO Auto-generated method stub
> throw new SQLException("Method not supported");
>   }
> in result all calls executeQuery() for prepared statement return exception

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Created: (HIVE-2062) HivePreparedStatement.executeImmediate always throw exception

[Alexey Diomin (JIRA)]

HivePreparedStatement.executeImmediate always throw exception
-

 Key: HIVE-2062
 URL: https://issues.apache.org/jira/browse/HIVE-2062
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Affects Versions: 0.7.0
Reporter: Alexey Diomin
Priority: Critical


executeImmediate:

try {
  clearWarnings();
  resultSet = null;
  client.execute(sql);
}

but:
  public void clearWarnings() throws SQLException {
// TODO Auto-generated method stub
throw new SQLException("Method not supported");
  }

in result all calls executeQuery() for prepared statement return exception

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2054) Exception on windows when using the jdbc driver. "IOException: The system cannot find the path specified"

[Bennie Schut (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bennie Schut updated HIVE-2054:
---

Attachment: HIVE-2054.2.patch.txt

New patch because of merge from HIVE-1815.

> Exception on windows when using the jdbc driver. "IOException: The system 
> cannot find the path specified"
> -
>
> Key: HIVE-2054
> URL: https://issues.apache.org/jira/browse/HIVE-2054
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.8.0
>Reporter: Bennie Schut
>Assignee: Bennie Schut
>Priority: Minor
> Fix For: 0.8.0
>
> Attachments: HIVE-2054.1.patch.txt, HIVE-2054.2.patch.txt
>
>
> It seems something recently changed on the jdbc driver which causes this 
> IOException on windows.
> java.lang.RuntimeException: java.io.IOException: The system cannot find the 
> path specified
>   at 
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:237)
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:73)
>   at org.apache.hadoop.hive.jdbc.HiveDriver.connect(HiveDriver.java:110)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HIVE-2061) Create a hive_contrib.jar symlink to hive-contrib-{version}.jar for backward compatibility

[Paul Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008363#comment-13008363
 ] 

Paul Yang commented on HIVE-2061:
-

Looks good, will test and commit.

> Create a hive_contrib.jar symlink to hive-contrib-{version}.jar for backward 
> compatibility
> --
>
> Key: HIVE-2061
> URL: https://issues.apache.org/jira/browse/HIVE-2061
> Project: Hive
>  Issue Type: Bug
>Reporter: Ning Zhang
>Assignee: Ning Zhang
>Priority: Minor
> Attachments: HIVE-2061.patch
>
>
> We have seen a use case where in the user's script, it run 'add jar 
> hive_contrib.jar'. Since Hive has moved the jar file to be 
> hive-contrib-{version}.jar, it introduced backward incompatibility. If we as 
> the user to change the script and when Hive upgrade version again, the user 
> need to change the script again. Creating a symlink seems to be the best 
> solution. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (HIVE-2028) Performance instruments for client side execution

[Paul Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Yang updated HIVE-2028:


   Resolution: Fixed
Fix Version/s: 0.8.0
   Status: Resolved  (was: Patch Available)

Committed. Thanks Ning!

> Performance instruments for client side execution
> -
>
> Key: HIVE-2028
> URL: https://issues.apache.org/jira/browse/HIVE-2028
> Project: Hive
>  Issue Type: Improvement
>Reporter: Ning Zhang
>Assignee: Ning Zhang
> Fix For: 0.8.0
>
> Attachments: HIVE-2028.2.patch, HIVE-2028.3.patch, HIVE-2028.patch
>
>
> Hive client side execution could sometimes takes a long time. This task is to 
> instrument the client side code to measure the time spent in the most likely 
> expensive components. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira