[jira] [Updated] (HIVE-3537) release locks at the end of move tasks

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3537:
-

Component/s: Locking

> release locks at the end of move tasks
> --
>
> Key: HIVE-3537
> URL: https://issues.apache.org/jira/browse/HIVE-3537
> Project: Hive
>  Issue Type: Bug
>  Components: Locking, Query Processor
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> Look at HIVE-3106 for details.
> In order to make sure that concurrency is not an issue for multi-table 
> inserts, the current option is to introduce a dependency task, which thereby
> delays the creation of all partitions. It would be desirable to release the
> locks for the outputs as soon as the move task is completed. That way, for
> multi-table inserts, the concurrency can be enabled without delaying any 
> table.
> Currently, the movetask contains a input/output, but they do not seem to be
> populated correctly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-3537) release locks at the end of move tasks

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain reassigned HIVE-3537:


Assignee: Namit Jain

> release locks at the end of move tasks
> --
>
> Key: HIVE-3537
> URL: https://issues.apache.org/jira/browse/HIVE-3537
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> Look at HIVE-3106 for details.
> In order to make sure that concurrency is not an issue for multi-table 
> inserts, the current option is to introduce a dependency task, which thereby
> delays the creation of all partitions. It would be desirable to release the
> locks for the outputs as soon as the move task is completed. That way, for
> multi-table inserts, the concurrency can be enabled without delaying any 
> table.
> Currently, the movetask contains a input/output, but they do not seem to be
> populated correctly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3537) release locks at the end of move tasks

[Namit Jain (JIRA)]

Namit Jain created HIVE-3537:


 Summary: release locks at the end of move tasks
 Key: HIVE-3537
 URL: https://issues.apache.org/jira/browse/HIVE-3537
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Namit Jain


Look at HIVE-3106 for details.

In order to make sure that concurrency is not an issue for multi-table 
inserts, the current option is to introduce a dependency task, which thereby
delays the creation of all partitions. It would be desirable to release the
locks for the outputs as soon as the move task is completed. That way, for
multi-table inserts, the concurrency can be enabled without delaying any 
table.

Currently, the movetask contains a input/output, but they do not seem to be
populated correctly.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470032#comment-13470032
 ] 

Phabricator commented on HIVE-3536:
---

njain has accepted the revision "HIVE-3536 [jira] Output of sort merge join is 
no longer bucketed".

REVISION DETAIL
  https://reviews.facebook.net/D5907

To: JIRA, njain, kevinwilfong


> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470031#comment-13470031
 ] 

Namit Jain commented on HIVE-3536:
--

+1

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3514) Refactor Partition Pruner so that logic can be reused.

[Gang Tim Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470025#comment-13470025
 ] 

Gang Tim Liu commented on HIVE-3514:


Namit thank you very much

Best wishes

Tim Liu (Liu , Gang)

Sent from my iPhone 




> Refactor Partition Pruner so that logic can be reused.
> --
>
> Key: HIVE-3514
> URL: https://issues.apache.org/jira/browse/HIVE-3514
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Gang Tim Liu
>Assignee: Gang Tim Liu
>Priority: Minor
> Attachments: HIVE-3514.patch, HIVE-3514.patch.2, HIVE-3514.patch.3, 
> HIVE-3514.patch.4, HIVE-3514.patch.5
>
>
> Partition Pruner has logic reusable like
> 1. walk through operator tree
> 2. walk through operation tree
> 3. create pruning predicate
> The first candidate is list bucketing pruner.
> Some consideration:
> 1. refactor for general use case not just list bucketing
> 2. avoid over-refactor by focusing on pieces targeted for reuse

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3536:
--

Attachment: HIVE-3536.D5907.2.patch

kevinwilfong updated the revision "HIVE-3536 [jira] Output of sort merge join 
is no longer bucketed".
Reviewers: JIRA, njain

  Updated test case.

REVISION DETAIL
  https://reviews.facebook.net/D5907

AFFECTED FILES
  ql/src/test/results/clientpositive/smb_mapjoin_11.q.out
  ql/src/test/queries/clientpositive/smb_mapjoin_11.q
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

To: JIRA, njain, kevinwilfong


> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3536:


Attachment: HIVE-3536.2.patch.txt

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470018#comment-13470018
 ] 

Kevin Wilfong commented on HIVE-3536:
-

Updated the new test.

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3536:


Status: Patch Available  (was: Open)

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt, HIVE-3536.2.patch.txt, 
> HIVE-3536.D5907.2.patch
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470017#comment-13470017
 ] 

Namit Jain commented on HIVE-2228:
--

Isn't it true for other clauses also ?
I mean, order by, sort by, distribute by etc.
It is OK if you want to do that in a follow-up, but wanted to confirm.

> Can't use DB qualified column names in WHERE or GROUP BY clauses
> 
>
> Key: HIVE-2228
> URL: https://issues.apache.org/jira/browse/HIVE-2228
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Affects Versions: 0.7.0
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-2228.1.patch.txt
>
>
> Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP 
> BY clauses. The workaround is to define a table alias:
> {noformat}
> hive> CREATE DATABASE db1;
> OK
> hive> CREATE TABLE db1.t(a INT, b INT);
> OK
> hive> SELECT * FROM db1.t WHERE db1.t.a > 100;
> FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t WHERE t.a > 100;
> OK
> hive> SELECT * FROM db1.t GROUP BY db1.t.a;
> FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t GROUP BY t.a;
> OK
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-1977:
-

Status: Open  (was: Patch Available)

minor comments

> DESCRIBE TABLE syntax doesn't support specifying a database qualified table 
> name
> 
>
> Key: HIVE-1977
> URL: https://issues.apache.org/jira/browse/HIVE-1977
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-1977.1.patch.txt, HIVE-1977.2.patch.txt, 
> HIVE-1977.3.patch.txt
>
>
> The syntax for DESCRIBE is broken. It should be:
> {code}
> DESCRIBE [EXTENDED] [database DOT]table [column]
> {code}
> but is actually
> {code}
> DESCRIBE [EXTENDED] table[DOT col_name]
> {code}
> Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470011#comment-13470011
 ] 

Namit Jain commented on HIVE-3536:
--

I will start running the tests.
Once you update the new test, I can just run the new test (assuming all tests 
run fine).

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3514) Refactor Partition Pruner so that logic can be reused.

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3514:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed. Thanks Tim

> Refactor Partition Pruner so that logic can be reused.
> --
>
> Key: HIVE-3514
> URL: https://issues.apache.org/jira/browse/HIVE-3514
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Gang Tim Liu
>Assignee: Gang Tim Liu
>Priority: Minor
> Attachments: HIVE-3514.patch, HIVE-3514.patch.2, HIVE-3514.patch.3, 
> HIVE-3514.patch.4, HIVE-3514.patch.5
>
>
> Partition Pruner has logic reusable like
> 1. walk through operator tree
> 2. walk through operation tree
> 3. create pruning predicate
> The first candidate is list bucketing pruner.
> Some consideration:
> 1. refactor for general use case not just list bucketing
> 2. avoid over-refactor by focusing on pieces targeted for reuse

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470009#comment-13470009
 ] 

Phabricator commented on HIVE-3536:
---

njain has commented on the revision "HIVE-3536 [jira] Output of sort merge join 
is no longer bucketed".

  Otherwise, it looks good

INLINE COMMENTS
  ql/src/test/queries/clientpositive/smb_mapjoin_11.q:32 Unfortunately, this 
does not verify that the data is bucketed.
  Can you perform a join between test_table3 and test_table1 for bucket 2 for 
both of them.
  That would return 0 rows if the data was not bucketed correctly.

REVISION DETAIL
  https://reviews.facebook.net/D5907

To: JIRA, njain, kevinwilfong


> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-3536:
-

Status: Open  (was: Patch Available)

small comment on phabricator

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470007#comment-13470007
 ] 

Phabricator commented on HIVE-3536:
---

njain has commented on the revision "HIVE-3536 [jira] Output of sort merge join 
is no longer bucketed".

  This was certainty a desired feature.

REVISION DETAIL
  https://reviews.facebook.net/D5907

To: JIRA, njain, kevinwilfong


> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-2228:
---

Status: Patch Available  (was: Open)

> Can't use DB qualified column names in WHERE or GROUP BY clauses
> 
>
> Key: HIVE-2228
> URL: https://issues.apache.org/jira/browse/HIVE-2228
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Affects Versions: 0.7.0
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-2228.1.patch.txt
>
>
> Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP 
> BY clauses. The workaround is to define a table alias:
> {noformat}
> hive> CREATE DATABASE db1;
> OK
> hive> CREATE TABLE db1.t(a INT, b INT);
> OK
> hive> SELECT * FROM db1.t WHERE db1.t.a > 100;
> FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t WHERE t.a > 100;
> OK
> hive> SELECT * FROM db1.t GROUP BY db1.t.a;
> FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t GROUP BY t.a;
> OK
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470002#comment-13470002
 ] 

Zhenxiao Luo commented on HIVE-2228:


propose to add dbName into ColumnInfo. The logic is mainly in  
TypeCheckProcFactory. While, lots of updates in TestParse.

Review Request submitted at:
https://reviews.facebook.net/D5913

> Can't use DB qualified column names in WHERE or GROUP BY clauses
> 
>
> Key: HIVE-2228
> URL: https://issues.apache.org/jira/browse/HIVE-2228
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Affects Versions: 0.7.0
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-2228.1.patch.txt
>
>
> Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP 
> BY clauses. The workaround is to define a table alias:
> {noformat}
> hive> CREATE DATABASE db1;
> OK
> hive> CREATE TABLE db1.t(a INT, b INT);
> OK
> hive> SELECT * FROM db1.t WHERE db1.t.a > 100;
> FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t WHERE t.a > 100;
> OK
> hive> SELECT * FROM db1.t GROUP BY db1.t.a;
> FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t GROUP BY t.a;
> OK
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2228) Can't use DB qualified column names in WHERE or GROUP BY clauses

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-2228:
---

Attachment: HIVE-2228.1.patch.txt

> Can't use DB qualified column names in WHERE or GROUP BY clauses
> 
>
> Key: HIVE-2228
> URL: https://issues.apache.org/jira/browse/HIVE-2228
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Affects Versions: 0.7.0
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-2228.1.patch.txt
>
>
> Hive doesn't allow you to use DB qualified column names in the WHERE or GROUP 
> BY clauses. The workaround is to define a table alias:
> {noformat}
> hive> CREATE DATABASE db1;
> OK
> hive> CREATE TABLE db1.t(a INT, b INT);
> OK
> hive> SELECT * FROM db1.t WHERE db1.t.a > 100;
> FAILED: Error in semantic analysis: Line 1:26 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t WHERE t.a > 100;
> OK
> hive> SELECT * FROM db1.t GROUP BY db1.t.a;
> FAILED: Error in semantic analysis: Line 1:29 Invalid table alias or column 
> reference 'db1'
> hive> SELECT * FROM db1.t t GROUP BY t.a;
> OK
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-3528

[Sean Busbey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7431/
---

(Updated Oct. 5, 2012, 3:23 a.m.)


Review request for hive.


Changes
---

WIP solution. 1 failing case remains, nullable enums.


Summary (updated)
-

HIVE-3528


Description (updated)
---

Changes AvroSerDe to properly give the non-null schema to serialization 
routines when using Nullable complex types


Diffs (updated)
-

  /trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java 
1394121 
  
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java
 1394121 
  
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java
 1394121 

Diff: https://reviews.apache.org/r/7431/diff/


Testing (updated)
---

Adds tests that check each of the Avro types that Serialization needs to use a 
user-provided schema, both as top level fields and as nested members of a 
complex type.


Thanks,

Sean Busbey

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3536:


Status: Patch Available  (was: Open)

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469976#comment-13469976
 ] 

Kevin Wilfong commented on HIVE-3536:
-

https://reviews.facebook.net/D5907

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Wilfong updated HIVE-3536:


Attachment: HIVE-3536.1.patch.txt

> Output of sort merge join is no longer bucketed
> ---
>
> Key: HIVE-3536
> URL: https://issues.apache.org/jira/browse/HIVE-3536
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3536.1.patch.txt
>
>
> I don't know if this was a feature or a happy coincidence, but before 
> HIVE-3230, the output of a sort merge join on two partitions would be 
> bucketed, even if hive.enforce.bucketing was set to false.  This could 
> potentially save a reduce phase when inserting into a bucketed table.
> This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3519) partition to directory comparison in CombineHiveInputFormat needs to accept partitions dir without scheme

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3519:


Status: Patch Available  (was: Open)

> partition to directory comparison in CombineHiveInputFormat needs to accept 
> partitions dir without scheme
> -
>
> Key: HIVE-3519
> URL: https://issues.apache.org/jira/browse/HIVE-3519
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3519.1.patch, HIVE-3519.2.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test 
> case is just printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir = 
> file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
>  in pathToPartitionInfo: 
> [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
>  
> /Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
> at 
> org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:232)
> at junit.framework.TestSuite.run(TestSuite.java:227)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3519) partition to directory comparison in CombineHiveInputFormat needs to accept partitions dir without scheme

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3519:


Status: Open  (was: Patch Available)

> partition to directory comparison in CombineHiveInputFormat needs to accept 
> partitions dir without scheme
> -
>
> Key: HIVE-3519
> URL: https://issues.apache.org/jira/browse/HIVE-3519
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3519.1.patch, HIVE-3519.2.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test 
> case is just printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir = 
> file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
>  in pathToPartitionInfo: 
> [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
>  
> /Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
> at 
> org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:232)
> at junit.framework.TestSuite.run(TestSuite.java:227)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3519) partition to directory comparison in CombineHiveInputFormat needs to accept partitions dir without scheme

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3519:


Attachment: HIVE-3519.2.patch

HIVE-3519.2.patch - addresses the NPE issue described above.


> partition to directory comparison in CombineHiveInputFormat needs to accept 
> partitions dir without scheme
> -
>
> Key: HIVE-3519
> URL: https://issues.apache.org/jira/browse/HIVE-3519
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3519.1.patch, HIVE-3519.2.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test 
> case is just printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir = 
> file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
>  in pathToPartitionInfo: 
> [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
>  
> /Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
> at 
> org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:232)
> at junit.framework.TestSuite.run(TestSuite.java:227)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
> at 
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3536) Output of sort merge join is no longer bucketed

[Kevin Wilfong (JIRA)]

Kevin Wilfong created HIVE-3536:
---

 Summary: Output of sort merge join is no longer bucketed
 Key: HIVE-3536
 URL: https://issues.apache.org/jira/browse/HIVE-3536
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.10.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong


I don't know if this was a feature or a happy coincidence, but before 
HIVE-3230, the output of a sort merge join on two partitions would be bucketed, 
even if hive.enforce.bucketing was set to false.  This could potentially save a 
reduce phase when inserting into a bucketed table.

This would be good to have back.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3519) partition to directory comparison in CombineHiveInputFormat needs to accept partitions dir without scheme

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469962#comment-13469962
 ] 

Thejas M Nair commented on HIVE-3519:
-

The previous patch exposed another problem with related code on windows. 
TestSymlinkTextInputFormat throws the following exception -

{code}

java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.doGetWorksFromPath(HiveFileFormatUtils.java:420)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:349)
at 
org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
PREHOOK: query: drop table text_symlink_text

{code}

This is because in HiveFileFormatUtils.populateNewPartitionDesc it was looking 
up a path whose string representation has an additional "/" before the drive 
letter. While the strings that were keys in the Map that did not have the "/C".
Eg - It was doing lookup on map- 
{C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1=[text_symlink_text],
 
C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1=[text_symlink_text]}
 with
with 
file:/C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
with 
/C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1



> partition to directory comparison in CombineHiveInputFormat needs to accept 
> partitions dir without scheme
> -
>
> Key: HIVE-3519
> URL: https://issues.apache.org/jira/browse/HIVE-3519
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3519.1.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test 
> case is just printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir = 
> file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
>  in pathToPartitionInfo: 
> [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
>  
> /Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
> at 
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
> at 
> org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:232)
>  

[jira] [Updated] (HIVE-3441) testcases escape1,escape2 fail on windows

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3441:


Status: Patch Available  (was: Open)

> testcases escape1,escape2 fail on windows
> -
>
> Key: HIVE-3441
> URL: https://issues.apache.org/jira/browse/HIVE-3441
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3441.1.patch, HIVE-3441.2.patch, HIVE-3441.3.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3441) testcases escape1,escape2 fail on windows

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3441:


Attachment: HIVE-3441.3.patch

HIVE-3441.3.patch - fixed duplicate lines in previous patch


> testcases escape1,escape2 fail on windows
> -
>
> Key: HIVE-3441
> URL: https://issues.apache.org/jira/browse/HIVE-3441
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3441.1.patch, HIVE-3441.2.patch, HIVE-3441.3.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3441) testcases escape1,escape2 fail on windows

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3441:


Status: Open  (was: Patch Available)

HIVE-3441.2.patch  has some duplicate lines. Creating new patch.


> testcases escape1,escape2 fail on windows
> -
>
> Key: HIVE-3441
> URL: https://issues.apache.org/jira/browse/HIVE-3441
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3441.1.patch, HIVE-3441.2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3441) testcases escape1,escape2 fail on windows

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3441:


Attachment: HIVE-3441.2.patch

HIVE-3441.2.patch - because of the backward compatibility issue that has been 
identified, this patch uses different escape chars only if the OS in windows. 
The patch adds support for INCLUDE/EXCLUDE hints in .q files for windows 
similar to the options for hadoop versions.


> testcases escape1,escape2 fail on windows
> -
>
> Key: HIVE-3441
> URL: https://issues.apache.org/jira/browse/HIVE-3441
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3441.1.patch, HIVE-3441.2.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3437) 0.23 compatibility: fix unit tests when building against 0.23

[Chris Drome (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Drome updated HIVE-3437:
--

Attachment: HIVE-3437-0.9-4.patch

Fixed tests: join14.q, sample_islocalmode_hook.q, local_mapred_error_cache.q.

> 0.23 compatibility: fix unit tests when building against 0.23
> -
>
> Key: HIVE-3437
> URL: https://issues.apache.org/jira/browse/HIVE-3437
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Fix For: 0.9.0, 0.10.0
>
> Attachments: HIVE-3437-0.9-1.patch, HIVE-3437-0.9-2.patch, 
> HIVE-3437-0.9-3.patch, HIVE-3437-0.9-4.patch, HIVE-3437-0.9.patch, 
> HIVE-3437-trunk-1.patch, HIVE-3437-trunk-2.patch, HIVE-3437-trunk-3.patch, 
> HIVE-3437-trunk-4.patch, HIVE-3437-trunk.patch
>
>
> Many unit tests fail as a result of building the code against hadoop 0.23. 
> Initial focus will be to fix 0.9.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469857#comment-13469857
 ] 

Navis commented on HIVE-3531:
-

@Edward: As stated in description, we delegates execution of all the queries to 
a single hive server. In this case, external lock manager like a zookeeper 
could be used but it's not required thing. But it does not mean locking itself 
is not needed. Multiple queries could be executed concurrently and still need a 
appropriate locking system. 

I didn't think of locking mechanism of hive deeply yet. It's just in-memory 
version of zookeeper lock manager. The part "locking does not working well" in 
your comment should be an another issue.

> Simple lock manager for dedicated hive server
> -
>
> Key: HIVE-3531
> URL: https://issues.apache.org/jira/browse/HIVE-3531
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3531.D5871.1.patch
>
>
> In many cases, we uses hive server as a sole proxy for executing all the 
> queries. For that, current default lock manager based on zookeeper seemed a 
> little heavy. Simple in-memory lock manager could be enough.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3437) 0.23 compatibility: fix unit tests when building against 0.23

[Chris Drome (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Drome updated HIVE-3437:
--

Attachment: (was: HIVE-3437-0.9-4.patch)

> 0.23 compatibility: fix unit tests when building against 0.23
> -
>
> Key: HIVE-3437
> URL: https://issues.apache.org/jira/browse/HIVE-3437
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Fix For: 0.9.0, 0.10.0
>
> Attachments: HIVE-3437-0.9-1.patch, HIVE-3437-0.9-2.patch, 
> HIVE-3437-0.9-3.patch, HIVE-3437-0.9.patch, HIVE-3437-trunk-1.patch, 
> HIVE-3437-trunk-2.patch, HIVE-3437-trunk-3.patch, HIVE-3437-trunk-4.patch, 
> HIVE-3437-trunk.patch
>
>
> Many unit tests fail as a result of building the code against hadoop 0.23. 
> Initial focus will be to fix 0.9.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1722 - Still Failing

[Apache Jenkins Server]

Changes for Build #1708

Changes for Build #1709
[namit] HIVE-3515 metadata_export_drop.q causes failure of other tests
(Ivan Gorbachev via namit)


Changes for Build #1710

Changes for Build #1711
[heyongqiang] HIVE-2206:add a new optimizer for query correlation discovery and 
optimization (Yin Huai via He Yongqiang)

[namit] HIVE-1367 cluster by multiple columns does not work if parenthesis is 
present
(Zhenxiao Luo via namit)


Changes for Build #1712
[cws] add instrumentation to capture if there is skew in reducers (Arun Dobriya 
via cws)

[namit] HIVE-3493 aggName of SemanticAnalyzer.getGenericUDAFEvaluator is 
generated in two
different ways (Yin Huai via namit)

[heyongqiang] revert r1392105 due to bylaw requirement mentioned by Carl 
Steinbach


Changes for Build #1713

Changes for Build #1714
[kevinwilfong] HIVE-3484. RetryingRawStore logic needs to be significantly 
reworked to support retries within transactions (Jean Xu via kevinwilfong)


Changes for Build #1715
[namit] HIVE-3495 For UDAFs, when generating a plan without 
map-side-aggregation, constant 
agg parameters will be replaced by ExprNodeColumnDesc (Yin Huai via namit)


Changes for Build #1716

Changes for Build #1717
[kevinwilfong] HIVE-3458. Parallel test script doesnt run all tests. (Ivan 
Gorbachev via kevinwilfong)

[hashutosh] HIVE-3481: : Hiveserver is not closing the existing 
driver handle before executing the next command. It results in to file handle 
leaks. (Kanna Karanam via Ashutosh Chauhan)


Changes for Build #1718

Changes for Build #1719
[kevinwilfong] HIVE-3498. hivetest.py fails with --revision option. (Ivan 
Gorbachev via kevinwilfong)


Changes for Build #1720

Changes for Build #1721

Changes for Build #1722



No tests ran.

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1722)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1722/ to 
view the results.

[jira] [Commented] (HIVE-3441) testcases escape1,escape2 fail on windows

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469818#comment-13469818
 ] 

Ashutosh Chauhan commented on HIVE-3441:


I tried testing this patch. I was able to read partitions having old names 
which are escaped with patch fine, but I was not able to delete them. Looks 
like comment holds only w.r.t reading partitions with old names but dropping 
them is a problem.

> testcases escape1,escape2 fail on windows
> -
>
> Key: HIVE-3441
> URL: https://issues.apache.org/jira/browse/HIVE-3441
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3441.1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1721 - Still Failing

[Apache Jenkins Server]

Changes for Build #1708

Changes for Build #1709
[namit] HIVE-3515 metadata_export_drop.q causes failure of other tests
(Ivan Gorbachev via namit)


Changes for Build #1710

Changes for Build #1711
[heyongqiang] HIVE-2206:add a new optimizer for query correlation discovery and 
optimization (Yin Huai via He Yongqiang)

[namit] HIVE-1367 cluster by multiple columns does not work if parenthesis is 
present
(Zhenxiao Luo via namit)


Changes for Build #1712
[cws] add instrumentation to capture if there is skew in reducers (Arun Dobriya 
via cws)

[namit] HIVE-3493 aggName of SemanticAnalyzer.getGenericUDAFEvaluator is 
generated in two
different ways (Yin Huai via namit)

[heyongqiang] revert r1392105 due to bylaw requirement mentioned by Carl 
Steinbach


Changes for Build #1713

Changes for Build #1714
[kevinwilfong] HIVE-3484. RetryingRawStore logic needs to be significantly 
reworked to support retries within transactions (Jean Xu via kevinwilfong)


Changes for Build #1715
[namit] HIVE-3495 For UDAFs, when generating a plan without 
map-side-aggregation, constant 
agg parameters will be replaced by ExprNodeColumnDesc (Yin Huai via namit)


Changes for Build #1716

Changes for Build #1717
[kevinwilfong] HIVE-3458. Parallel test script doesnt run all tests. (Ivan 
Gorbachev via kevinwilfong)

[hashutosh] HIVE-3481: : Hiveserver is not closing the existing 
driver handle before executing the next command. It results in to file handle 
leaks. (Kanna Karanam via Ashutosh Chauhan)


Changes for Build #1718

Changes for Build #1719
[kevinwilfong] HIVE-3498. hivetest.py fails with --revision option. (Ivan 
Gorbachev via kevinwilfong)


Changes for Build #1720

Changes for Build #1721



2 tests failed.
FAILED:  
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1

Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... 
-Dtest.silent=false" to get more logs.

Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
more logs.
at junit.framework.Assert.fail(Assert.java:47)
at 
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:11512)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)


REGRESSION:  org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1

Error Message:
One or more queries failed

Stack Trace:
junit.framework.AssertionFailedError: One or more queries failed
at junit.framework.Assert.fail(Assert.java:47)
at 
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)




The Apache Jenkins build system has built Hive-trunk-h0.2

[jira] [Updated] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3535:
---

Status: Patch Available  (was: Open)

> derby metastore upgrade script throw errors when updating from 0.7 to 0.8
> -
>
> Key: HIVE-3535
> URL: https://issues.apache.org/jira/browse/HIVE-3535
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.9.0
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3535.1.patch.txt
>
>
> $DERBY_HOME/bin/ij upgrade.sql
> ij version 10.4
> ij> CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db';
> ij> RUN 
> '/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql';
> ij> -- Upgrade MetaStore schema from 0.7.0 to 0.8.0
> RUN '008-HIVE-2246.derby.sql';
> ij> /*
>  * Creates the following tables:
>  *  - CDS
>  *  - COLUMNS_V2
>  * The new columns table is called COLUMNS_V2
>  * because many columns are removed, and the schema is changed.
>  * It'd take too long to migrate and keep the same table.
>  */
> CREATE TABLE "CDS" (
>   "CD_ID" bigint NOT NULL,
>   PRIMARY KEY ("CD_ID")
> );
> 0 rows inserted/updated/deleted
> ij> CREATE TABLE "COLUMNS_V2" (
>   "CD_ID" bigint NOT NULL,
>   "COMMENT" varchar(4000),
>   "COLUMN_NAME" varchar(128) NOT NULL,
>   "TYPE_NAME" varchar(4000),
>   "INTEGER_IDX" INTEGER NOT NULL,
>   PRIMARY KEY ("CD_ID", "COLUMN_NAME")
> );
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE "COLUMNS_V2" 
>   ADD CONSTRAINT "COLUMNS_V2_FK1"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
>   ON DELETE NO ACTION ON UPDATE NO ACTION
> ;
> 0 rows inserted/updated/deleted
> ij> /* Alter the SDS table to:
>  *  - add the column CD_ID
>  *  - add a foreign key on CD_ID
>  *  - create an index on CD_ID
>  */ 
> ALTER TABLE SDS
>   ADD COLUMN "CD_ID" bigint
> ;
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE SDS
>   ADD CONSTRAINT "SDS_FK2"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
> ;
> 0 rows inserted/updated/deleted
> ij> /*
>  * Migrate the TBLS table
>  * Add entries into CDS.
>  * Populate the CD_ID field in SDS for tables
>  * Add entires to COLUMNS_V2 based on this table's sd's columns
>  */ 
> /* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
>  * for tables. For speed, just let CD_ID = SD_ID for tables 
>  */
> INSERT INTO CDS (CD_ID)
> SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 13, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   TBLS t
> ON
>   t.SD_ID = c.SD_ID
> ;
> ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key 
> constraint 'COLUMNS_V2_FK1' for key (1).  The statement has been rolled back.
> ij> /*
>  * Migrate the partitions.
>  * Update the partitions' SDS to use the parent tables' CD_ID  BEGIN
>  * Derby does not allow joins in update statements, 
>  * so we have to make a temporary tableh
>  */
> DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
>   "SD_ID" bigint not null,
>   "CD_ID" bigint not null
> ) ON COMMIT PRESERVE ROWS NOT LOGGED;
> 0 rows inserted/updated/deleted
> ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
>   p.SD_ID, sds.CD_ID
>   FROM PARTITIONS p
>   JOIN TBLS t ON t.TBL_ID = p.TBL_ID
>   JOIN SDS sds on t.SD_ID = sds.SD_ID
>   WHERE p.SD_ID IS NOT NULL;
> ERROR 23502: Column 'CD_ID'  cannot accept a NULL value.
> ij> UPDATE SDS sd
>   SET sd.CD_ID = 
> (SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
>   WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
> 0 rows inserted/updated/deleted
> WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of 
> a query is an empty table.
> ij> /*
>  * Migrate IDXS
>  */
> INSERT INTO CDS (CD_ID)
> SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 5, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   IDXS i
> ON
>   i.SD_ID = c.SD_ID
> ;
> ERROR 42X05: Table/

[jira] [Commented] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469778#comment-13469778
 ] 

Zhenxiao Luo commented on HIVE-3535:


The new upgrade script gets rid of all the errors:

ij> run 'upgrade-0.7.0-to-0.8.0.derby.sql';
ij> – Upgrade MetaStore schema from 0.7.0 to 0.8.0
RUN '008-HIVE-2246.derby.sql';
ij> /*
Creates the following tables:
- CDS
- COLUMNS_V2
The new columns table is called COLUMNS_V2
because many columns are removed, and the schema is changed.
It'd take too long to migrate and keep the same table.
*/
CREATE TABLE "CDS" (
"CD_ID" bigint NOT NULL,
PRIMARY KEY ("CD_ID")
);
0 rows inserted/updated/deleted
ij> CREATE TABLE "COLUMNS_V2" (
"CD_ID" bigint NOT NULL,
"COMMENT" varchar(4000),
"COLUMN_NAME" varchar(128) NOT NULL,
"TYPE_NAME" varchar(4000),
"INTEGER_IDX" INTEGER NOT NULL,
PRIMARY KEY ("CD_ID", "COLUMN_NAME")
);
0 rows inserted/updated/deleted
ij> ALTER TABLE "COLUMNS_V2" 
ADD CONSTRAINT "COLUMNS_V2_FK1"
FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
ON DELETE NO ACTION ON UPDATE NO ACTION
;
0 rows inserted/updated/deleted
ij> /* Alter the SDS table to:
- add the column CD_ID
- add a foreign key on CD_ID
- create an index on CD_ID
*/ 
ALTER TABLE SDS
ADD COLUMN "CD_ID" bigint
;
0 rows inserted/updated/deleted
ij> ALTER TABLE SDS
ADD CONSTRAINT "SDS_FK2"
FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
;
0 rows inserted/updated/deleted
ij> /*
Migrate the TBLS table
Add entries into CDS.
Populate the CD_ID field in SDS for tables
Add entires to COLUMNS_V2 based on this table's sd's columns
*/
/* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
for tables. For speed, just let CD_ID = SD_ID for tables
*/
INSERT INTO CDS (CD_ID)
SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL;
1 row inserted/updated/deleted
ij> UPDATE SDS
SET CD_ID = SD_ID
WHERE SD_ID in 
(SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL);
1 row inserted/updated/deleted
ij> INSERT INTO COLUMNS_V2
(CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
SELECT 
c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
FROM
COLUMNS c
JOIN
TBLS t
ON
t.SD_ID = c.SD_ID
;
2 rows inserted/updated/deleted
ij> /*
Migrate the partitions.
Update the partitions' SDS to use the parent tables' CD_ID BEGIN
Derby does not allow joins in update statements,
so we have to make a temporary tableh
*/
DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
"SD_ID" bigint not null,
"CD_ID" bigint not null
) ON COMMIT PRESERVE ROWS NOT LOGGED;
0 rows inserted/updated/deleted
ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
p.SD_ID, sds.CD_ID
FROM PARTITIONS p
JOIN TBLS t ON t.TBL_ID = p.TBL_ID
JOIN SDS sds on t.SD_ID = sds.SD_ID
WHERE p.SD_ID IS NOT NULL AND sds.CD_ID IS NOT NULL;
0 rows inserted/updated/deleted
ij> UPDATE SDS sd
SET sd.CD_ID = 
(SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
0 rows inserted/updated/deleted
WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of a 
query is an empty table.
ij> /*
Migrate IDXS
*/
INSERT INTO CDS (CD_ID)
SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL;
0 rows inserted/updated/deleted
ij> UPDATE SDS
SET CD_ID = SD_ID
WHERE SD_ID in 
(SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL);
0 rows inserted/updated/deleted
WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of a 
query is an empty table.
ij> INSERT INTO COLUMNS_V2
(CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
SELECT 
c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
FROM
COLUMNS c
JOIN
IDXS i
ON
i.SD_ID = c.SD_ID
;
0 rows inserted/updated/deleted
ij> /*
rename the old COLUMNS table
*/
RENAME TABLE COLUMNS TO COLUMNS_OLD;
0 rows inserted/updated/deleted
ij> RUN '009-HIVE-2215.derby.sql';
ij> – Table PARTITION_EVENTS for classes 
[org.apache.hadoop.hive.metastore.model.MPartitionEvent]
CREATE TABLE PARTITION_EVENTS
(
PART_NAME_ID BIGINT NOT NULL,
DB_NAME VARCHAR(128),
EVENT_TIME BIGINT NOT NULL,
EVENT_TYPE INTEGER NOT NULL,
PARTITION_NAME VARCHAR(767),
TBL_NAME VARCHAR(128)
);
0 rows inserted/updated/deleted
ij> ALTER TABLE PARTITION_EVENTS ADD CONSTRAINT PARTITION_EVENTS_PK PRIMARY KEY 
(PART_NAME_ID);
0 rows inserted/updated/deleted
ij>

> derby metastore upgrade script throw errors when updating from 0.7 to 0.8
> -
>
> Key: HIVE-3535
> URL: https://issues.apache.org/jira/browse/HIVE-3535
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.9.0
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3535.1.patch.txt
>
>
> $DERBY_HOME/bin/ij upgrade.sql
> ij version 10.4
> ij> CONNECT 'jdbc:derby:/var/lib/hive/metasto

[jira] [Updated] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3535:
---

Attachment: HIVE-3535.1.patch.txt

> derby metastore upgrade script throw errors when updating from 0.7 to 0.8
> -
>
> Key: HIVE-3535
> URL: https://issues.apache.org/jira/browse/HIVE-3535
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.9.0
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3535.1.patch.txt
>
>
> $DERBY_HOME/bin/ij upgrade.sql
> ij version 10.4
> ij> CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db';
> ij> RUN 
> '/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql';
> ij> -- Upgrade MetaStore schema from 0.7.0 to 0.8.0
> RUN '008-HIVE-2246.derby.sql';
> ij> /*
>  * Creates the following tables:
>  *  - CDS
>  *  - COLUMNS_V2
>  * The new columns table is called COLUMNS_V2
>  * because many columns are removed, and the schema is changed.
>  * It'd take too long to migrate and keep the same table.
>  */
> CREATE TABLE "CDS" (
>   "CD_ID" bigint NOT NULL,
>   PRIMARY KEY ("CD_ID")
> );
> 0 rows inserted/updated/deleted
> ij> CREATE TABLE "COLUMNS_V2" (
>   "CD_ID" bigint NOT NULL,
>   "COMMENT" varchar(4000),
>   "COLUMN_NAME" varchar(128) NOT NULL,
>   "TYPE_NAME" varchar(4000),
>   "INTEGER_IDX" INTEGER NOT NULL,
>   PRIMARY KEY ("CD_ID", "COLUMN_NAME")
> );
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE "COLUMNS_V2" 
>   ADD CONSTRAINT "COLUMNS_V2_FK1"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
>   ON DELETE NO ACTION ON UPDATE NO ACTION
> ;
> 0 rows inserted/updated/deleted
> ij> /* Alter the SDS table to:
>  *  - add the column CD_ID
>  *  - add a foreign key on CD_ID
>  *  - create an index on CD_ID
>  */ 
> ALTER TABLE SDS
>   ADD COLUMN "CD_ID" bigint
> ;
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE SDS
>   ADD CONSTRAINT "SDS_FK2"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
> ;
> 0 rows inserted/updated/deleted
> ij> /*
>  * Migrate the TBLS table
>  * Add entries into CDS.
>  * Populate the CD_ID field in SDS for tables
>  * Add entires to COLUMNS_V2 based on this table's sd's columns
>  */ 
> /* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
>  * for tables. For speed, just let CD_ID = SD_ID for tables 
>  */
> INSERT INTO CDS (CD_ID)
> SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 13, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   TBLS t
> ON
>   t.SD_ID = c.SD_ID
> ;
> ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key 
> constraint 'COLUMNS_V2_FK1' for key (1).  The statement has been rolled back.
> ij> /*
>  * Migrate the partitions.
>  * Update the partitions' SDS to use the parent tables' CD_ID  BEGIN
>  * Derby does not allow joins in update statements, 
>  * so we have to make a temporary tableh
>  */
> DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
>   "SD_ID" bigint not null,
>   "CD_ID" bigint not null
> ) ON COMMIT PRESERVE ROWS NOT LOGGED;
> 0 rows inserted/updated/deleted
> ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
>   p.SD_ID, sds.CD_ID
>   FROM PARTITIONS p
>   JOIN TBLS t ON t.TBL_ID = p.TBL_ID
>   JOIN SDS sds on t.SD_ID = sds.SD_ID
>   WHERE p.SD_ID IS NOT NULL;
> ERROR 23502: Column 'CD_ID'  cannot accept a NULL value.
> ij> UPDATE SDS sd
>   SET sd.CD_ID = 
> (SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
>   WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
> 0 rows inserted/updated/deleted
> WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of 
> a query is an empty table.
> ij> /*
>  * Migrate IDXS
>  */
> INSERT INTO CDS (CD_ID)
> SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 5, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   IDXS i
> ON
>   i.SD_ID = c.SD_ID
> ;
> ERROR 42X05: Table/Vie

[jira] [Commented] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469776#comment-13469776
 ] 

Zhenxiao Luo commented on HIVE-3535:


Review request submitted at:
https://reviews.facebook.net/D5889

> derby metastore upgrade script throw errors when updating from 0.7 to 0.8
> -
>
> Key: HIVE-3535
> URL: https://issues.apache.org/jira/browse/HIVE-3535
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.9.0
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
>
> $DERBY_HOME/bin/ij upgrade.sql
> ij version 10.4
> ij> CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db';
> ij> RUN 
> '/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql';
> ij> -- Upgrade MetaStore schema from 0.7.0 to 0.8.0
> RUN '008-HIVE-2246.derby.sql';
> ij> /*
>  * Creates the following tables:
>  *  - CDS
>  *  - COLUMNS_V2
>  * The new columns table is called COLUMNS_V2
>  * because many columns are removed, and the schema is changed.
>  * It'd take too long to migrate and keep the same table.
>  */
> CREATE TABLE "CDS" (
>   "CD_ID" bigint NOT NULL,
>   PRIMARY KEY ("CD_ID")
> );
> 0 rows inserted/updated/deleted
> ij> CREATE TABLE "COLUMNS_V2" (
>   "CD_ID" bigint NOT NULL,
>   "COMMENT" varchar(4000),
>   "COLUMN_NAME" varchar(128) NOT NULL,
>   "TYPE_NAME" varchar(4000),
>   "INTEGER_IDX" INTEGER NOT NULL,
>   PRIMARY KEY ("CD_ID", "COLUMN_NAME")
> );
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE "COLUMNS_V2" 
>   ADD CONSTRAINT "COLUMNS_V2_FK1"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
>   ON DELETE NO ACTION ON UPDATE NO ACTION
> ;
> 0 rows inserted/updated/deleted
> ij> /* Alter the SDS table to:
>  *  - add the column CD_ID
>  *  - add a foreign key on CD_ID
>  *  - create an index on CD_ID
>  */ 
> ALTER TABLE SDS
>   ADD COLUMN "CD_ID" bigint
> ;
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE SDS
>   ADD CONSTRAINT "SDS_FK2"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
> ;
> 0 rows inserted/updated/deleted
> ij> /*
>  * Migrate the TBLS table
>  * Add entries into CDS.
>  * Populate the CD_ID field in SDS for tables
>  * Add entires to COLUMNS_V2 based on this table's sd's columns
>  */ 
> /* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
>  * for tables. For speed, just let CD_ID = SD_ID for tables 
>  */
> INSERT INTO CDS (CD_ID)
> SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 13, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   TBLS t
> ON
>   t.SD_ID = c.SD_ID
> ;
> ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key 
> constraint 'COLUMNS_V2_FK1' for key (1).  The statement has been rolled back.
> ij> /*
>  * Migrate the partitions.
>  * Update the partitions' SDS to use the parent tables' CD_ID  BEGIN
>  * Derby does not allow joins in update statements, 
>  * so we have to make a temporary tableh
>  */
> DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
>   "SD_ID" bigint not null,
>   "CD_ID" bigint not null
> ) ON COMMIT PRESERVE ROWS NOT LOGGED;
> 0 rows inserted/updated/deleted
> ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
>   p.SD_ID, sds.CD_ID
>   FROM PARTITIONS p
>   JOIN TBLS t ON t.TBL_ID = p.TBL_ID
>   JOIN SDS sds on t.SD_ID = sds.SD_ID
>   WHERE p.SD_ID IS NOT NULL;
> ERROR 23502: Column 'CD_ID'  cannot accept a NULL value.
> ij> UPDATE SDS sd
>   SET sd.CD_ID = 
> (SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
>   WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
> 0 rows inserted/updated/deleted
> WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of 
> a query is an empty table.
> ij> /*
>  * Migrate IDXS
>  */
> INSERT INTO CDS (CD_ID)
> SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 5, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   IDXS i
> ON
> 

[jira] [Commented] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469774#comment-13469774
 ] 

Zhenxiao Luo commented on HIVE-3535:


Derby does not support ORDER BY in subquery:

http://old.nabble.com/Problem-with-%22order-by%22-td16775867.html

"Using a order by clause in a subquery is actually not allowed in the SQL 
standard (neither SQL-99 nor SQL-2003)"

> derby metastore upgrade script throw errors when updating from 0.7 to 0.8
> -
>
> Key: HIVE-3535
> URL: https://issues.apache.org/jira/browse/HIVE-3535
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.9.0
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
>
> $DERBY_HOME/bin/ij upgrade.sql
> ij version 10.4
> ij> CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db';
> ij> RUN 
> '/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql';
> ij> -- Upgrade MetaStore schema from 0.7.0 to 0.8.0
> RUN '008-HIVE-2246.derby.sql';
> ij> /*
>  * Creates the following tables:
>  *  - CDS
>  *  - COLUMNS_V2
>  * The new columns table is called COLUMNS_V2
>  * because many columns are removed, and the schema is changed.
>  * It'd take too long to migrate and keep the same table.
>  */
> CREATE TABLE "CDS" (
>   "CD_ID" bigint NOT NULL,
>   PRIMARY KEY ("CD_ID")
> );
> 0 rows inserted/updated/deleted
> ij> CREATE TABLE "COLUMNS_V2" (
>   "CD_ID" bigint NOT NULL,
>   "COMMENT" varchar(4000),
>   "COLUMN_NAME" varchar(128) NOT NULL,
>   "TYPE_NAME" varchar(4000),
>   "INTEGER_IDX" INTEGER NOT NULL,
>   PRIMARY KEY ("CD_ID", "COLUMN_NAME")
> );
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE "COLUMNS_V2" 
>   ADD CONSTRAINT "COLUMNS_V2_FK1"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
>   ON DELETE NO ACTION ON UPDATE NO ACTION
> ;
> 0 rows inserted/updated/deleted
> ij> /* Alter the SDS table to:
>  *  - add the column CD_ID
>  *  - add a foreign key on CD_ID
>  *  - create an index on CD_ID
>  */ 
> ALTER TABLE SDS
>   ADD COLUMN "CD_ID" bigint
> ;
> 0 rows inserted/updated/deleted
> ij> ALTER TABLE SDS
>   ADD CONSTRAINT "SDS_FK2"
>   FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
> ;
> 0 rows inserted/updated/deleted
> ij> /*
>  * Migrate the TBLS table
>  * Add entries into CDS.
>  * Populate the CD_ID field in SDS for tables
>  * Add entires to COLUMNS_V2 based on this table's sd's columns
>  */ 
> /* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
>  * for tables. For speed, just let CD_ID = SD_ID for tables 
>  */
> INSERT INTO CDS (CD_ID)
> SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 13, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
> SELECT 
>   c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
> FROM
>   COLUMNS c
> JOIN
>   TBLS t
> ON
>   t.SD_ID = c.SD_ID
> ;
> ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key 
> constraint 'COLUMNS_V2_FK1' for key (1).  The statement has been rolled back.
> ij> /*
>  * Migrate the partitions.
>  * Update the partitions' SDS to use the parent tables' CD_ID  BEGIN
>  * Derby does not allow joins in update statements, 
>  * so we have to make a temporary tableh
>  */
> DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
>   "SD_ID" bigint not null,
>   "CD_ID" bigint not null
> ) ON COMMIT PRESERVE ROWS NOT LOGGED;
> 0 rows inserted/updated/deleted
> ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
>   p.SD_ID, sds.CD_ID
>   FROM PARTITIONS p
>   JOIN TBLS t ON t.TBL_ID = p.TBL_ID
>   JOIN SDS sds on t.SD_ID = sds.SD_ID
>   WHERE p.SD_ID IS NOT NULL;
> ERROR 23502: Column 'CD_ID'  cannot accept a NULL value.
> ij> UPDATE SDS sd
>   SET sd.CD_ID = 
> (SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
>   WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
> 0 rows inserted/updated/deleted
> WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of 
> a query is an empty table.
> ij> /*
>  * Migrate IDXS
>  */
> INSERT INTO CDS (CD_ID)
> SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID;
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 5, column 54.
> ij> UPDATE SDS
>   SET CD_ID = SD_ID
> WHERE SD_ID in 
> (SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID);
> ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
> ij> INSERT INTO COLUMNS_V2
>   (CD_ID, COMM

[jira] [Created] (HIVE-3535) derby metastore upgrade script throw errors when updating from 0.7 to 0.8

[Zhenxiao Luo (JIRA)]

Zhenxiao Luo created HIVE-3535:
--

 Summary: derby metastore upgrade script throw errors when updating 
from 0.7 to 0.8
 Key: HIVE-3535
 URL: https://issues.apache.org/jira/browse/HIVE-3535
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.9.0
Reporter: Zhenxiao Luo
Assignee: Zhenxiao Luo
 Fix For: 0.10.0


$DERBY_HOME/bin/ij upgrade.sql
ij version 10.4
ij> CONNECT 'jdbc:derby:/var/lib/hive/metastore/metastore_db';
ij> RUN 
'/usr/lib/hive/scripts/metastore/upgrade/derby/upgrade-0.7.0-to-0.8.0.derby.sql';
ij> -- Upgrade MetaStore schema from 0.7.0 to 0.8.0
RUN '008-HIVE-2246.derby.sql';
ij> /*
 * Creates the following tables:
 *  - CDS
 *  - COLUMNS_V2
 * The new columns table is called COLUMNS_V2
 * because many columns are removed, and the schema is changed.
 * It'd take too long to migrate and keep the same table.
 */
CREATE TABLE "CDS" (
  "CD_ID" bigint NOT NULL,
  PRIMARY KEY ("CD_ID")
);
0 rows inserted/updated/deleted
ij> CREATE TABLE "COLUMNS_V2" (
  "CD_ID" bigint NOT NULL,
  "COMMENT" varchar(4000),
  "COLUMN_NAME" varchar(128) NOT NULL,
  "TYPE_NAME" varchar(4000),
  "INTEGER_IDX" INTEGER NOT NULL,
  PRIMARY KEY ("CD_ID", "COLUMN_NAME")
);
0 rows inserted/updated/deleted
ij> ALTER TABLE "COLUMNS_V2" 
  ADD CONSTRAINT "COLUMNS_V2_FK1"
  FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
  ON DELETE NO ACTION ON UPDATE NO ACTION
;
0 rows inserted/updated/deleted
ij> /* Alter the SDS table to:
 *  - add the column CD_ID
 *  - add a foreign key on CD_ID
 *  - create an index on CD_ID
 */ 
ALTER TABLE SDS
  ADD COLUMN "CD_ID" bigint
;
0 rows inserted/updated/deleted
ij> ALTER TABLE SDS
  ADD CONSTRAINT "SDS_FK2"
  FOREIGN KEY ("CD_ID") REFERENCES "CDS" ("CD_ID")
;
0 rows inserted/updated/deleted
ij> /*
 * Migrate the TBLS table
 * Add entries into CDS.
 * Populate the CD_ID field in SDS for tables
 * Add entires to COLUMNS_V2 based on this table's sd's columns
 */ 

/* In the migration, there is a 1:1 mapping between CD_ID and SD_ID
 * for tables. For speed, just let CD_ID = SD_ID for tables 
 */
INSERT INTO CDS (CD_ID)
SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID;
ERROR 42X01: Syntax error: Encountered "ORDER" at line 13, column 54.
ij> UPDATE SDS
  SET CD_ID = SD_ID
WHERE SD_ID in 
(SELECT t.SD_ID FROM TBLS t WHERE t.SD_ID IS NOT NULL ORDER BY t.SD_ID);
ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
ij> INSERT INTO COLUMNS_V2
  (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
SELECT 
  c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
FROM
  COLUMNS c
JOIN
  TBLS t
ON
  t.SD_ID = c.SD_ID
;
ERROR 23503: INSERT on table 'COLUMNS_V2' caused a violation of foreign key 
constraint 'COLUMNS_V2_FK1' for key (1).  The statement has been rolled back.
ij> /*
 * Migrate the partitions.
 * Update the partitions' SDS to use the parent tables' CD_ID  BEGIN
 * Derby does not allow joins in update statements, 
 * so we have to make a temporary tableh
 */
DECLARE GLOBAL TEMPORARY TABLE "TMP_TBL" (
  "SD_ID" bigint not null,
  "CD_ID" bigint not null
) ON COMMIT PRESERVE ROWS NOT LOGGED;
0 rows inserted/updated/deleted
ij> INSERT INTO "SESSION"."TMP_TBL" SELECT
  p.SD_ID, sds.CD_ID
  FROM PARTITIONS p
  JOIN TBLS t ON t.TBL_ID = p.TBL_ID
  JOIN SDS sds on t.SD_ID = sds.SD_ID
  WHERE p.SD_ID IS NOT NULL;
ERROR 23502: Column 'CD_ID'  cannot accept a NULL value.
ij> UPDATE SDS sd
  SET sd.CD_ID = 
(SELECT tt.CD_ID FROM SESSION.TMP_TBL tt WHERE tt.SD_ID = sd.SD_ID)
  WHERE sd.SD_ID IN (SELECT SD_ID FROM SESSION.TMP_TBL);
0 rows inserted/updated/deleted
WARNING 02000: No row was found for FETCH, UPDATE or DELETE; or the result of a 
query is an empty table.
ij> /*
 * Migrate IDXS
 */
INSERT INTO CDS (CD_ID)
SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID;
ERROR 42X01: Syntax error: Encountered "ORDER" at line 5, column 54.
ij> UPDATE SDS
  SET CD_ID = SD_ID
WHERE SD_ID in 
(SELECT i.SD_ID FROM IDXS i WHERE i.SD_ID IS NOT NULL ORDER BY i.SD_ID);
ERROR 42X01: Syntax error: Encountered "ORDER" at line 4, column 55.
ij> INSERT INTO COLUMNS_V2
  (CD_ID, COMMENT, COLUMN_NAME, TYPE_NAME, INTEGER_IDX)
SELECT 
  c.SD_ID, c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME, c.INTEGER_IDX
FROM
  COLUMNS c
JOIN
  IDXS i
ON
  i.SD_ID = c.SD_ID
;
ERROR 42X05: Table/View 'IDXS' does not exist.
ij> /*
 * rename the old COLUMNS table
 */
RENAME TABLE COLUMNS TO COLUMNS_OLD;
0 rows inserted/updated/deleted
ij> RUN '009-HIVE-2215.derby.sql';
ij> -- Table PARTITION_EVENTS for classes 
[org.apache.hadoop.hive.metastore.model.MPartitionEvent]
CREATE TABLE PARTITION_EVENTS
(
PART_NAME_ID BIGINT NOT NULL,
DB_NAME VARCHAR(128),
EVENT_TIME BIGINT NOT NULL,
EVENT_TYPE INTEGER NOT NULL,
PARTITION_NAME VARCHAR(767),
TBL_NAME VARCHAR(128)
);
0 rows inserted/updated/deleted
ij> ALTER TA

[jira] [Assigned] (HIVE-3534) Add left Lateral view syntax

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo reassigned HIVE-3534:
-

Assignee: Edward Capriolo

> Add left Lateral view syntax
> 
>
> Key: HIVE-3534
> URL: https://issues.apache.org/jira/browse/HIVE-3534
> Project: Hive
>  Issue Type: New Feature
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
>
> A lateral view is like a join and if the function in question (like 
> explode()) produces zero rows then you lose the row
> whereas conceivably you could keep the row and just fill all entry with nulls.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3534) Add left Lateral view syntax

[Edward Capriolo (JIRA)]

Edward Capriolo created HIVE-3534:
-

 Summary: Add left Lateral view syntax
 Key: HIVE-3534
 URL: https://issues.apache.org/jira/browse/HIVE-3534
 Project: Hive
  Issue Type: New Feature
Reporter: Edward Capriolo


A lateral view is like a join and if the function in question (like explode()) 
produces zero rows then you lose the row
whereas conceivably you could keep the row and just fill all entry with nulls.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3533) ZooKeeperHiveLockManager does not respect the option to keep locks alive even after the current session has closed

[Matt Martin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Martin updated HIVE-3533:
--

Attachment: HIVE-3533.1.patch.txt

Uploading the very simple patch for this issue.  Replacing the hardcoded value 
of "false" with the boolean parameter "keepAlive".

> ZooKeeperHiveLockManager does not respect the option to keep locks alive even 
> after the current session has closed
> --
>
> Key: HIVE-3533
> URL: https://issues.apache.org/jira/browse/HIVE-3533
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.9.0
>Reporter: Matt Martin
>Priority: Minor
> Attachments: HIVE-3533.1.patch.txt
>
>
> The HiveLockManager interface defines the following method:
> public List lock(List objs,
>   boolean keepAlive) throws LockException;
> ZooKeeperHiveLockManager implements HiveLockManager, but the current 
> implementation of the "lock" method never actually references the "keepAlive" 
> parameter.  As a result, all of the locks acquired by the "lock" method are 
> ephemeral.  In other words, Zookeeper-based locks only exist as long as the 
> underlying Zookeeper session exists.  As soon as the Zookeeper session ends, 
> any Zookeeper-based locks are automatically released.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3533) ZooKeeperHiveLockManager does not respect the option to keep locks alive even after the current session has closed

[Matt Martin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Martin updated HIVE-3533:
--

Status: Patch Available  (was: Open)

> ZooKeeperHiveLockManager does not respect the option to keep locks alive even 
> after the current session has closed
> --
>
> Key: HIVE-3533
> URL: https://issues.apache.org/jira/browse/HIVE-3533
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.9.0
>Reporter: Matt Martin
>Priority: Minor
> Attachments: HIVE-3533.1.patch.txt
>
>
> The HiveLockManager interface defines the following method:
> public List lock(List objs,
>   boolean keepAlive) throws LockException;
> ZooKeeperHiveLockManager implements HiveLockManager, but the current 
> implementation of the "lock" method never actually references the "keepAlive" 
> parameter.  As a result, all of the locks acquired by the "lock" method are 
> ephemeral.  In other words, Zookeeper-based locks only exist as long as the 
> underlying Zookeeper session exists.  As soon as the Zookeeper session ends, 
> any Zookeeper-based locks are automatically released.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-1362) column level statistics

[Shreepadma Venugopalan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469616#comment-13469616
 ] 

Shreepadma Venugopalan commented on HIVE-1362:
--

@Namit: I've addressed your review comments in the latest revision to the 
patch. Its available on review board.

> column level statistics
> ---
>
> Key: HIVE-1362
> URL: https://issues.apache.org/jira/browse/HIVE-1362
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Ning Zhang
>Assignee: Shreepadma Venugopalan
> Attachments: HIVE-1362.1.patch.txt, HIVE-1362.2.patch.txt, 
> HIVE-1362.3.patch.txt, HIVE-1362.4.patch.txt, 
> HIVE-1362-gen_thrift.1.patch.txt, HIVE-1362-gen_thrift.2.patch.txt, 
> HIVE-1362-gen_thrift.3.patch.txt, HIVE-1362-gen_thrift.4.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3533) ZooKeeperHiveLockManager does not respect the option to keep locks alive even after the current session has closed

[Matt Martin (JIRA)]

Matt Martin created HIVE-3533:
-

 Summary: ZooKeeperHiveLockManager does not respect the option to 
keep locks alive even after the current session has closed
 Key: HIVE-3533
 URL: https://issues.apache.org/jira/browse/HIVE-3533
 Project: Hive
  Issue Type: Bug
  Components: Locking
Affects Versions: 0.9.0
Reporter: Matt Martin
Priority: Minor


The HiveLockManager interface defines the following method:

public List lock(List objs,
  boolean keepAlive) throws LockException;

ZooKeeperHiveLockManager implements HiveLockManager, but the current 
implementation of the "lock" method never actually references the "keepAlive" 
parameter.  As a result, all of the locks acquired by the "lock" method are 
ephemeral.  In other words, Zookeeper-based locks only exist as long as the 
underlying Zookeeper session exists.  As soon as the Zookeeper session ends, 
any Zookeeper-based locks are automatically released.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1362) column level statistics

[Shreepadma Venugopalan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shreepadma Venugopalan updated HIVE-1362:
-

Status: Patch Available  (was: Open)

> column level statistics
> ---
>
> Key: HIVE-1362
> URL: https://issues.apache.org/jira/browse/HIVE-1362
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Ning Zhang
>Assignee: Shreepadma Venugopalan
> Attachments: HIVE-1362.1.patch.txt, HIVE-1362.2.patch.txt, 
> HIVE-1362.3.patch.txt, HIVE-1362.4.patch.txt, 
> HIVE-1362-gen_thrift.1.patch.txt, HIVE-1362-gen_thrift.2.patch.txt, 
> HIVE-1362-gen_thrift.3.patch.txt, HIVE-1362-gen_thrift.4.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-1362) column level statistics

[Shreepadma Venugopalan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469614#comment-13469614
 ] 

Shreepadma Venugopalan commented on HIVE-1362:
--

@Shrikanth: Thank you for your comments. We can certainly add a new UDAF with 
the Flajolet-Martin sketch that returns a serialized numDV estimator. I've 
already filed a new JIRA (HIVE-3516) for the incremental stats computation 
work. I'll add the UDAF as part of that JIRA. 

A couple of reasons why we decided to create a new compute_stats aggregation 
operator instead of generating more expressions in the SQL,

1. We felt its a lot cleaner to encapsulate the stats for a column within a 
single UDAF. The compute_stats UDAF returns a struct with the relevant stats 
depending on the data type of the column and as a result makes the parsing as 
well as the SQL we generate simple.

2. Adding a new compute_stats UDAF allows the gathering of statistical 
summaries of the underlying data even outside of the column stats framework. 
One use I can think of is, it can be used to model the statistical properties 
of data which in turn can be used to generate data whose statistical properties 
mimic that of the underlying data.

Even though max, min, total count exist as UDAFs today, we need these to be 
part of the histogram UDAF. Estimating quantiles for equi-height histogram is a 
lot more efficient if we know the range of values the column can take. We need 
to know the total_count to generate the histogram bins. Given that we need 
these stats for generating histograms, I think its a good idea to encapsulate 
all of these stats within the compute_stats UDAF. Thanks.

> column level statistics
> ---
>
> Key: HIVE-1362
> URL: https://issues.apache.org/jira/browse/HIVE-1362
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Ning Zhang
>Assignee: Shreepadma Venugopalan
> Attachments: HIVE-1362.1.patch.txt, HIVE-1362.2.patch.txt, 
> HIVE-1362.3.patch.txt, HIVE-1362.4.patch.txt, 
> HIVE-1362-gen_thrift.1.patch.txt, HIVE-1362-gen_thrift.2.patch.txt, 
> HIVE-1362-gen_thrift.3.patch.txt, HIVE-1362-gen_thrift.4.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-1977:
---

Status: Patch Available  (was: Open)

> DESCRIBE TABLE syntax doesn't support specifying a database qualified table 
> name
> 
>
> Key: HIVE-1977
> URL: https://issues.apache.org/jira/browse/HIVE-1977
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-1977.1.patch.txt, HIVE-1977.2.patch.txt, 
> HIVE-1977.3.patch.txt
>
>
> The syntax for DESCRIBE is broken. It should be:
> {code}
> DESCRIBE [EXTENDED] [database DOT]table [column]
> {code}
> but is actually
> {code}
> DESCRIBE [EXTENDED] table[DOT col_name]
> {code}
> Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-1977:
---

Attachment: HIVE-1977.3.patch.txt

> DESCRIBE TABLE syntax doesn't support specifying a database qualified table 
> name
> 
>
> Key: HIVE-1977
> URL: https://issues.apache.org/jira/browse/HIVE-1977
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-1977.1.patch.txt, HIVE-1977.2.patch.txt, 
> HIVE-1977.3.patch.txt
>
>
> The syntax for DESCRIBE is broken. It should be:
> {code}
> DESCRIBE [EXTENDED] [database DOT]table [column]
> {code}
> but is actually
> {code}
> DESCRIBE [EXTENDED] table[DOT col_name]
> {code}
> Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469605#comment-13469605
 ] 

Zhenxiao Luo commented on HIVE-1977:


Get bug fixed in the previous patch.

Get tests passed now. Review Request submitted at:
https://reviews.facebook.net/D5763

> DESCRIBE TABLE syntax doesn't support specifying a database qualified table 
> name
> 
>
> Key: HIVE-1977
> URL: https://issues.apache.org/jira/browse/HIVE-1977
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-1977.1.patch.txt, HIVE-1977.2.patch.txt
>
>
> The syntax for DESCRIBE is broken. It should be:
> {code}
> DESCRIBE [EXTENDED] [database DOT]table [column]
> {code}
> but is actually
> {code}
> DESCRIBE [EXTENDED] table[DOT col_name]
> {code}
> Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-1977) DESCRIBE TABLE syntax doesn't support specifying a database qualified table name

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-1977:
---

Status: Open  (was: Patch Available)

> DESCRIBE TABLE syntax doesn't support specifying a database qualified table 
> name
> 
>
> Key: HIVE-1977
> URL: https://issues.apache.org/jira/browse/HIVE-1977
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Query Processor, SQL
>Reporter: Carl Steinbach
>Assignee: Zhenxiao Luo
> Attachments: HIVE-1977.1.patch.txt, HIVE-1977.2.patch.txt
>
>
> The syntax for DESCRIBE is broken. It should be:
> {code}
> DESCRIBE [EXTENDED] [database DOT]table [column]
> {code}
> but is actually
> {code}
> DESCRIBE [EXTENDED] table[DOT col_name]
> {code}
> Ref: http://dev.mysql.com/doc/refman/5.0/en/describe.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3437) 0.23 compatibility: fix unit tests when building against 0.23

[Chris Drome (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Drome updated HIVE-3437:
--

Attachment: HIVE-3437-trunk-4.patch
HIVE-3437-0.9-4.patch

Fixed tests: join14.q, sample_islocalmode_hook.q.

> 0.23 compatibility: fix unit tests when building against 0.23
> -
>
> Key: HIVE-3437
> URL: https://issues.apache.org/jira/browse/HIVE-3437
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.9.0, 0.10.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Fix For: 0.9.0, 0.10.0
>
> Attachments: HIVE-3437-0.9-1.patch, HIVE-3437-0.9-2.patch, 
> HIVE-3437-0.9-3.patch, HIVE-3437-0.9-4.patch, HIVE-3437-0.9.patch, 
> HIVE-3437-trunk-1.patch, HIVE-3437-trunk-2.patch, HIVE-3437-trunk-3.patch, 
> HIVE-3437-trunk-4.patch, HIVE-3437-trunk.patch
>
>
> Many unit tests fail as a result of building the code against hadoop 0.23. 
> Initial focus will be to fix 0.9.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469559#comment-13469559
 ] 

Edward Capriolo commented on HIVE-3531:
---

Why wouldnt I just want to turn that off?

> Simple lock manager for dedicated hive server
> -
>
> Key: HIVE-3531
> URL: https://issues.apache.org/jira/browse/HIVE-3531
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3531.D5871.1.patch
>
>
> In many cases, we uses hive server as a sole proxy for executing all the 
> queries. For that, current default lock manager based on zookeeper seemed a 
> little heavy. Simple in-memory lock manager could be enough.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469558#comment-13469558
 ] 

Edward Capriolo commented on HIVE-3531:
---

@Carl Why? See my comments above. I am worried about code bloat in hive (No 
offense Navis your code is great). What is the case for having a lock manager 
that only does half the things a lock manager is supposed to do?

> Simple lock manager for dedicated hive server
> -
>
> Key: HIVE-3531
> URL: https://issues.apache.org/jira/browse/HIVE-3531
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3531.D5871.1.patch
>
>
> In many cases, we uses hive server as a sole proxy for executing all the 
> queries. For that, current default lock manager based on zookeeper seemed a 
> little heavy. Simple in-memory lock manager could be enough.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469552#comment-13469552
 ] 

Carl Steinbach commented on HIVE-3531:
--

@Navis: This is awesome. We were planning to write something like this for 
HiveServer2, but you beat us to it :)

I'll try to review this in the next day or so. Thanks!

> Simple lock manager for dedicated hive server
> -
>
> Key: HIVE-3531
> URL: https://issues.apache.org/jira/browse/HIVE-3531
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3531.D5871.1.patch
>
>
> In many cases, we uses hive server as a sole proxy for executing all the 
> queries. For that, current default lock manager based on zookeeper seemed a 
> little heavy. Simple in-memory lock manager could be enough.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469550#comment-13469550
 ] 

Edward Capriolo commented on HIVE-3531:
---

[~navis] Interesting.  I do know that the locking does not work well with 
thrift service currently as I have had issues with it. 

What is your use case for this? Why would you want a lock manager that does not 
actually work? One possibility is that it may be much nicer to make this the 
default for unit tests, but other then that I can not think of one.

> Simple lock manager for dedicated hive server
> -
>
> Key: HIVE-3531
> URL: https://issues.apache.org/jira/browse/HIVE-3531
> Project: Hive
>  Issue Type: Improvement
>  Components: Server Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-3531.D5871.1.patch
>
>
> In many cases, we uses hive server as a sole proxy for executing all the 
> queries. For that, current default lock manager based on zookeeper seemed a 
> little heavy. Simple in-memory lock manager could be enough.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-3530) warnings in Hive.g

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo reassigned HIVE-3530:
--

Assignee: Zhenxiao Luo

> warnings in Hive.g
> --
>
> Key: HIVE-3530
> URL: https://issues.apache.org/jira/browse/HIVE-3530
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Zhenxiao Luo
>
>  Building Grammar 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g  
> 
>  ANTLR Parser Generator  Version 3.0.1 (August 13, 2007)  1989-2007
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:578:5:
>  Decision can match input such as "Identifier KW_RENAME KW_TO" using multiple 
> alternatives: 1, 10
>  As a result, alternative(s) 10 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1607:5:
>  Decision can m
> atch input such as "Identifier DOT Identifier" using multiple alternatives: 
> 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1823:5:
>  Decision can match input such as "KW_ORDER KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1836:5:
>  Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1848:5:
>  Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1859:5:
>  Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
> Most of these seem to be due to HIVE-1367

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3530) warnings in Hive.g

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469528#comment-13469528
 ] 

Zhenxiao Luo commented on HIVE-3530:


Sure. I will take a look.

> warnings in Hive.g
> --
>
> Key: HIVE-3530
> URL: https://issues.apache.org/jira/browse/HIVE-3530
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Namit Jain
>
>  Building Grammar 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g  
> 
>  ANTLR Parser Generator  Version 3.0.1 (August 13, 2007)  1989-2007
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:578:5:
>  Decision can match input such as "Identifier KW_RENAME KW_TO" using multiple 
> alternatives: 1, 10
>  As a result, alternative(s) 10 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1607:5:
>  Decision can m
> atch input such as "Identifier DOT Identifier" using multiple alternatives: 
> 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1823:5:
>  Decision can match input such as "KW_ORDER KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1836:5:
>  Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1848:5:
>  Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
>  warning(200): 
> /Users/njain/hive/hive3/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g:1859:5:
>  Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple 
> alternatives: 1, 2
>  As a result, alternative(s) 2 were disabled for that input
> Most of these seem to be due to HIVE-1367

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2935) Implement HiveServer2

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469526#comment-13469526
 ] 

Carl Steinbach commented on HIVE-2935:
--

The attached patches contain a complete, working version of HiveServer2. Before 
going into details about the contents of the patch I want to first quickly 
review how to apply the patch and try out the new server:

1) Download and apply HIVE-2935.2.nothrift.patch.txt
2) Run the Thrift code generator (make sure $THRIFT_HOME points to thrift 
version 0.7.0)
{{% ant thriftif -Dthrift.home=$THRIFT_HOME}}
3) Optionally download and unpack the beeline test outputs in the 
ql/src/test/results directory. 
4) Build Hive:
{{% ant clean package}}
5) Start HiveServer2
{{% hiveserver2}}
6) From another window start the beeline CLI and connect to HiveServer2:
{noformat}
% beeline
Hive version 0.10.0-SNAPSHOT by Apache
beeline> !connect jdbc:hive2://localhost:1 scott tiger 
org.apache.hive.jdbc.HiveDriver
!connect jdbc:hive2://localhost:1 scott tiger 
org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://localhost:1
Connected to: Hive (version 0.10.0)
Driver: Hive (version 0.10.0-SNAPSHOT)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:1> show tables;
show tables;
+---+
| tab_name  |
+---+
| primitives|
| src   |
| src1  |
| src_json  |
| src_sequencefile  |
| src_thrift|
| srcbucket |
| srcbucket2|
| srcpart   |
+---+
9 rows selected (1.079 seconds)
{noformat}

If you downloaded the test outputs you can also try running the new 
BeeLineTestDriver:

% ant test -Dtestcase=TestBeeLineDriver -Dtest.concurrency.num.threads=10

Patch contents:
* {{cli/src/java/org/apache/hive/cli/beeline/*.java}}
** This a JDBC CLI for Hive based on the SQLLine CLI. An earlier version of 
BeeLine which had a dependency on SQLLine was previously added in HIVE-3100. 
However, while working on HS2 we discovered bugs in SQLLine that we needed to 
fix, and also needed to build in some extensions in order to support the 
BeeLine test driver. Adding the code directly to Hive seemed like the best 
option since the upstream project is no longer actively maintained.
* {{cli/src/java/org/apache/hive/cli/beeline/util/QFileClient.java}}
** This is a beeline test client used by TestBeeLineDriver. This class also 
provides an improved version of the output masking functionality currently 
located in QTestUtil.
* {{common/src/java/org/apache/hive/common/util/*.java}}
** Utility classes borrowed from Hadoop.
* {{data/files/types/primitives/*}}
** Data files for a 'primitives' table that contains all Hive primitive types 
along with NULLs.
* {{data/scripts/q_test_*.sql}}
** Test initialization scripts that are used to create and initialize all of 
the tables that are referenced by CliDriver tests. Called from QFileClient.
* {{jdbc/src/java/org/apache/hive/jdbc/*}}
** The HS2 JDBC driver.
* {{service/if/cli_service.thrift}}
** The CliService Thrift IDL file.
* {{service/src/java/org/apache/hive/service/*}}
** Service infrastructure classes borrowed from o.a.hadoop.yarn.service
* {{service/src/java/org/apache/hive/service/auth/*}}
** Kerberos/LDAP/SASL auth code for HS2
* {{service/src/java/org/apache/hive/service/cli/*}}
** Implementation classes for CLIService. These classes form the core of 
HiveServer2.
* {{service/src/java/org/apache/hive/service/server/HiveServer2.java}}
** HiveServer2 class. At the moment it just starts the CLIService, but we plan 
to extend it with other pluggable services in the future.
* {{testutils/junit/*}
** Utility classes for running concurrent JUnit tests. Most of this code was 
borrowed from tempus-fugit.

This code was a collaborative effort between me and my colleague Prasad 
Mujumdar. Prasad is wholly responsible for the new JDBC driver and 
authorization code. We worked together on the other parts.

We plan to break this monolithic patch up into several smaller patches in order 
to make the review process easier. Here's our initial plan for how to do this:

# Review/commit the beeline CLI (note that this can be used with the existing 
JDBC driver and HiveServer1)
# Review/commit the BeeLine test driver and new test outputs
# Review/commit the HiveServer2 core component.
# Review/commit the HiveServer2 JDBC driver.

We would appreciate receiving feedback from the Hive committers about whether 
or not this plan makes sense.

Thanks!



> Implement HiveServer2
> -
>
> Key: HIVE-2935
> URL: https://issues.apache.org/jira/browse/HIVE-2935
> Project: Hive
>  Issue Type: New Feature
>  Components: Server Infrastructure
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>  Labe

[jira] [Updated] (HIVE-2935) Implement HiveServer2

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2935:
-

Attachment: HIVE-2935.2.nothrift.patch.txt

> Implement HiveServer2
> -
>
> Key: HIVE-2935
> URL: https://issues.apache.org/jira/browse/HIVE-2935
> Project: Hive
>  Issue Type: New Feature
>  Components: Server Infrastructure
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>  Labels: HiveServer2
> Attachments: beelinepositive.tar.gz, HIVE-2935.1.notest.patch.txt, 
> HIVE-2935.2.notest.patch.txt, HIVE-2935.2.nothrift.patch.txt
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3532) ALTER TABLE SET LOCATION can be used to circumvent authorization checks

[Carl Steinbach (JIRA)]

Carl Steinbach created HIVE-3532:


 Summary: ALTER TABLE SET LOCATION can be used to circumvent 
authorization checks
 Key: HIVE-3532
 URL: https://issues.apache.org/jira/browse/HIVE-3532
 Project: Hive
  Issue Type: Bug
  Components: Authorization, Security
Reporter: Carl Steinbach




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2094) CREATE and DROP DATABASE doesn't check user permission for doing it

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2094:
-

Component/s: Authorization

> CREATE and DROP DATABASE doesn't check user permission for doing it
> ---
>
> Key: HIVE-2094
> URL: https://issues.apache.org/jira/browse/HIVE-2094
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Metastore, Security
>Reporter: Siying Dong
>Assignee: He Yongqiang
>
> We need to make sure only users with system permission to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2818) Create table checks the current database privilege

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2818:
-

Component/s: Security

> Create table checks the current database privilege
> --
>
> Key: HIVE-2818
> URL: https://issues.apache.org/jira/browse/HIVE-2818
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Security
>Affects Versions: 0.7.1
>Reporter: Benyi Wang
>
> Hive seems check the current database to determine the privilege of a 
> statement when you use fully qualified name like 'database.table'
>  
> {code}
> hive> set hive.security.authorization.enabled=true;
> hive> create database test_db;
> hive> grant all on database test_db to user test_user;
> hive> revoke all on database default from test_user;
> hive> use default;
> hive> create table test_db.new_table (id int);
> Authorization failed:No privilege 'Create' found for outputs { 
> database:default}. Use show grant to get more details.
> hive> use test_db;
> hive> create table test_db.new_table (id int);
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2538) Database level authorization can be circumvented by first "using" a database that one has ALL privileges to

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2538:
-

Component/s: Authorization

> Database level authorization can be circumvented by first "using" a database 
> that one has ALL privileges to
> ---
>
> Key: HIVE-2538
> URL: https://issues.apache.org/jira/browse/HIVE-2538
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Security
>Affects Versions: 0.7.1
>Reporter: Sriram Krishnan
>
> You can replicate this bug by recreating my setup, which I describe below:
> I have our "default" database set up to only allow SELECT for user 
> "skrishnan". But user skrishnan has "ALL" privileges on database "skrishnan".
> The following works correctly (i.e user shouldn't be able to create a table 
> in the default database):
> hive> use default;
> OK
> Time taken: 0.043 seconds
> hive> create table skrishnan_test(i int);
> Authorization failed:No privilege 'Create' found for outputs { 
> database:default}. Use show grant to get more details. (*Correct Behavior*)
> However, user skrishnan can indeed create tables in the default database by 
> doing this:
> hive> use skrishnan; 
> OK
> Time taken: 0.038 seconds
> hive> create table *default.skrishnan_test(i int);*
> OK
> Time taken: 0.34 seconds (*Incorrect behavior*)
> That means that the database level authorization is basically circumvented by 
> first using a database that a user has all privileges to. And then using the 
> fully qualified table name (db\_name.table\_name) for a database that a user 
> doesn't have permissions to.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-2240) Set hive.security.authorization.createtable.owner.grants to ALL by default

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-2240:
-

Component/s: Authorization

> Set hive.security.authorization.createtable.owner.grants to ALL by default
> --
>
> Key: HIVE-2240
> URL: https://issues.apache.org/jira/browse/HIVE-2240
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Configuration, Security
>Reporter: Carl Steinbach
>
> The default value is currently 'null', which means that users will not have 
> the ability to access tables that they created.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #158

[Apache Jenkins Server]

See 


--
[...truncated 10112 lines...]
 [echo] Project: odbc
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/odbc/src/conf
 does not exist.

ivy-resolve-test:
 [echo] Project: odbc

ivy-retrieve-test:
 [echo] Project: odbc

compile-test:
 [echo] Project: odbc

create-dirs:
 [echo] Project: serde
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/serde/src/test/resources
 does not exist.

init:
 [echo] Project: serde

ivy-init-settings:
 [echo] Project: serde

ivy-resolve:
 [echo] Project: serde
[ivy:resolve] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/ivy/ivysettings.xml
[ivy:report] Processing 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/resolution-cache/org.apache.hive-hive-serde-default.xml
 to 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/report/org.apache.hive-hive-serde-default.html

ivy-retrieve:
 [echo] Project: serde

dynamic-serde:

compile:
 [echo] Project: serde

ivy-resolve-test:
 [echo] Project: serde

ivy-retrieve-test:
 [echo] Project: serde

compile-test:
 [echo] Project: serde
[javac] Compiling 26 source files to 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/serde/test/classes
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.

create-dirs:
 [echo] Project: service
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/service/src/test/resources
 does not exist.

init:
 [echo] Project: service

ivy-init-settings:
 [echo] Project: service

ivy-resolve:
 [echo] Project: service
[ivy:resolve] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/ivy/ivysettings.xml
[ivy:report] Processing 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/resolution-cache/org.apache.hive-hive-service-default.xml
 to 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/report/org.apache.hive-hive-service-default.html

ivy-retrieve:
 [echo] Project: service

compile:
 [echo] Project: service

ivy-resolve-test:
 [echo] Project: service

ivy-retrieve-test:
 [echo] Project: service

compile-test:
 [echo] Project: service
[javac] Compiling 2 source files to 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/service/test/classes

test:
 [echo] Project: hive

test-shims:
 [echo] Project: hive

test-conditions:
 [echo] Project: shims

gen-test:
 [echo] Project: shims

create-dirs:
 [echo] Project: shims
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/test/resources
 does not exist.

init:
 [echo] Project: shims

ivy-init-settings:
 [echo] Project: shims

ivy-resolve:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/ivy/ivysettings.xml
[ivy:report] Processing 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/resolution-cache/org.apache.hive-hive-shims-default.xml
 to 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/ivy/report/org.apache.hive-hive-shims-default.html

ivy-retrieve:
 [echo] Project: shims

compile:
 [echo] Project: shims
 [echo] Building shims 0.20

build_shims:
 [echo] Project: shims
 [echo] Compiling 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/common/java;/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20/java
 against hadoop 0.20.2 
(/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/build/hadoopcore/hadoop-0.20.2)

ivy-init-settings:
 [echo] Project: shims

ivy-resolve-hadoop-shim:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/ivy/ivysettings.xml

ivy-retrieve-hadoop-shim:
 [echo] Project: shims
 [echo] Building shims 0.20S

build_shims:
 [echo] Project: shims
 [echo] Compiling 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/common/ja

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #158

[Apache Jenkins Server]

See 

--
[...truncated 5441 lines...]
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hwi/test/src
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hwi/test/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hwi/test/resources
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/hwi/src/test/resources
 does not exist.

init:
 [echo] Project: hwi

create-dirs:
 [echo] Project: hbase-handler
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler/test
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler/test/src
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler/test/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/hbase-handler/test/resources
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/hbase-handler/src/test/resources
 does not exist.

init:
 [echo] Project: hbase-handler

create-dirs:
 [echo] Project: pdk
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk/test
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk/test/src
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk/test/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/pdk/test/resources
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/pdk/src/test/resources
 does not exist.

init:
 [echo] Project: pdk

create-dirs:
 [echo] Project: builtins
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins/test
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins/test/src
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins/test/classes
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/builtins/test/resources
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/builtins/src/test/resources
 does not exist.

init:
 [echo] Project: builtins

jar:
 [echo] Project: hive

create-dirs:
 [echo] Project: shims
 [copy] Warning: 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/shims/src/test/resources
 does not exist.

init:
 [echo] Project: shims

ivy-init-settings:
 [echo] Project: shims

ivy-resolve:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/ivy/ivysettings.xml
[ivy:resolve] downloading 
http://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.4.3/zookeeper-3.4.3.jar
 ...
[ivy:resolve] 
 
(749kB)
[ivy:resolve] .. (0kB)
[ivy:resolve]   [SUCCESSFUL ] 
org.apache.zookeeper#zookeeper;3.4.3!zookeeper.jar (154ms)
[ivy:resolve] downloading 
http://repo1.maven.org/maven2/org/apache/thrift/libthrift/0.7.0/libthrift-0.7.0.jar
 ...
[ivy:resolve] .. (294kB)
[ivy:resolve] .. (0kB)
[ivy:resolve]   [SUCCESSFUL ] org.apache.thrift#libthrift;0.7.0!libthrift.jar 
(103ms)
[ivy:resolve] downloading 
http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar
 ...
[ivy:resolve] . (37kB)
[ivy:resolve] .. (0kB)
[ivy:resolve]   [SUCCESSFUL ] 
commons-logging#commons-logging;1.0.4!commons-logging.jar (42ms)
[ivy:resolve] downloading 
http://repo1.maven.org/maven2/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
 ...
[ivy:resolve] ... (25kB)
[ivy:resolve] .. (0kB)
[ivy:resolve]   [SUCCESSFUL ] 
commons-logging#commons-logging-api;1.0.4!commons-logging-

[jira] [Updated] (HIVE-2874) Renaming external partition changes location

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-2874:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed. Thanks Zhenxiao

> Renaming external partition changes location
> 
>
> Key: HIVE-2874
> URL: https://issues.apache.org/jira/browse/HIVE-2874
> Project: Hive
>  Issue Type: Bug
>Reporter: Kevin Wilfong
>Assignee: Zhenxiao Luo
> Attachments: HIVE-2874.1.patch.txt, HIVE-2874.2.patch.txt, 
> HIVE-2874.3.patch.txt
>
>
> Renaming an external partition will change the location of that partition to 
> the default location of a managed partition with the same name.
> E.g. If ex_table is external and has partition part=1 with location 
> /.../managed_table/part=1
> Calling ALTER TABLE ex_table PARTITION (part = '1') RENAME TO PARTITION (part 
> = '2');
> Will change the location of the partition to /.../ex_table/part=2

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3525) Avro Maps with Nullable Values fail with NPE

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HIVE-3525:
--

Status: Patch Available  (was: Open)

> Avro Maps with Nullable Values fail with NPE
> 
>
> Key: HIVE-3525
> URL: https://issues.apache.org/jira/browse/HIVE-3525
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Sean Busbey
> Attachments: HIVE-3525.1.patch.txt, HIVE-3525.2.patch.txt
>
>
> When working against current trunk@1393794, using a backing Avro schema that 
> has a Map field with nullable values causes a NPE on deserialization when the 
> map contains a null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3525) Avro Maps with Nullable Values fail with NPE

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HIVE-3525:
--

Attachment: HIVE-3525.2.patch.txt

Patch that corresponds to review board #7430

> Avro Maps with Nullable Values fail with NPE
> 
>
> Key: HIVE-3525
> URL: https://issues.apache.org/jira/browse/HIVE-3525
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Sean Busbey
> Attachments: HIVE-3525.1.patch.txt, HIVE-3525.2.patch.txt
>
>
> When working against current trunk@1393794, using a backing Avro schema that 
> has a Map field with nullable values causes a NPE on deserialization when the 
> map contains a null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: HIVE-3525

[Sean Busbey]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7430/
---

(Updated Oct. 4, 2012, 1:56 p.m.)


Review request for hive.


Changes
---

Last update missed the changes to AvroSerializer


Description
---

Changes Avro SerDe to use HashMap when copying out the Avro Map 
to Map. fixes HIVE-3525.


Diffs (updated)
-

  
/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java 
1394076 
  /trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java 
1394076 
  
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java
 1394076 
  
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java
 1394076 
  
/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java
 1394076 

Diff: https://reviews.apache.org/r/7430/diff/


Testing
---

Includes unit tests for 

* AvroObjectInspectorGenerator to verify that the Nullable value type is 
presented as just the non-null type.
* AvroDeserializer to verify that Maps with null are properly handled
* AvroSerializer to verify that Maps with null can round trip.


Thanks,

Sean Busbey

[jira] [Commented] (HIVE-3527) Allow CREATE TABLE LIKE command to take TBLPROPERTIES

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469239#comment-13469239
 ] 

Kevin Wilfong commented on HIVE-3527:
-

Ignore the that Phabricator diff and patch, they should be the same as the 
original, I just wanted to see if I could make it work.

> Allow CREATE TABLE LIKE command to take TBLPROPERTIES
> -
>
> Key: HIVE-3527
> URL: https://issues.apache.org/jira/browse/HIVE-3527
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3527.1.patch.txt, HIVE-3527.D5883.1.patch
>
>
> CREATE TABLE ... LIKE ... commands currently don't take TBLPROPERTIES.  I 
> think it would be a useful feature.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3527) Allow CREATE TABLE LIKE command to take TBLPROPERTIES

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3527:
--

Attachment: HIVE-3527.D5883.1.patch

kevinwilfong requested code review of "HIVE-3527 [jira] Allow CREATE TABLE LIKE 
command to take TBLPROPERTIES".
Reviewers: JIRA

  test

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D5883

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java
  ql/src/test/queries/clientpositive/create_like_tbl_props.q
  ql/src/test/results/clientpositive/create_like_tbl_props.q.out

MANAGE HERALD DIFFERENTIAL RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/13935/

To: JIRA, kevinwilfong


> Allow CREATE TABLE LIKE command to take TBLPROPERTIES
> -
>
> Key: HIVE-3527
> URL: https://issues.apache.org/jira/browse/HIVE-3527
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3527.1.patch.txt, HIVE-3527.D5883.1.patch
>
>
> CREATE TABLE ... LIKE ... commands currently don't take TBLPROPERTIES.  I 
> think it would be a useful feature.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3527) Allow CREATE TABLE LIKE command to take TBLPROPERTIES

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469237#comment-13469237
 ] 

Phabricator commented on HIVE-3527:
---

kevinwilfong has abandoned the revision "HIVE-3527 [jira] Allow CREATE TABLE 
LIKE command to take TBLPROPERTIES".

REVISION DETAIL
  https://reviews.facebook.net/D5883

To: JIRA, kevinwilfong


> Allow CREATE TABLE LIKE command to take TBLPROPERTIES
> -
>
> Key: HIVE-3527
> URL: https://issues.apache.org/jira/browse/HIVE-3527
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3527.1.patch.txt, HIVE-3527.D5883.1.patch
>
>
> CREATE TABLE ... LIKE ... commands currently don't take TBLPROPERTIES.  I 
> think it would be a useful feature.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3527) Allow CREATE TABLE LIKE command to take TBLPROPERTIES

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469218#comment-13469218
 ] 

Namit Jain commented on HIVE-3527:
--

+1

> Allow CREATE TABLE LIKE command to take TBLPROPERTIES
> -
>
> Key: HIVE-3527
> URL: https://issues.apache.org/jira/browse/HIVE-3527
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
> Attachments: HIVE-3527.1.patch.txt
>
>
> CREATE TABLE ... LIKE ... commands currently don't take TBLPROPERTIES.  I 
> think it would be a useful feature.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1720 - Still Failing

[Apache Jenkins Server]

Changes for Build #1708

Changes for Build #1709
[namit] HIVE-3515 metadata_export_drop.q causes failure of other tests
(Ivan Gorbachev via namit)


Changes for Build #1710

Changes for Build #1711
[heyongqiang] HIVE-2206:add a new optimizer for query correlation discovery and 
optimization (Yin Huai via He Yongqiang)

[namit] HIVE-1367 cluster by multiple columns does not work if parenthesis is 
present
(Zhenxiao Luo via namit)


Changes for Build #1712
[cws] add instrumentation to capture if there is skew in reducers (Arun Dobriya 
via cws)

[namit] HIVE-3493 aggName of SemanticAnalyzer.getGenericUDAFEvaluator is 
generated in two
different ways (Yin Huai via namit)

[heyongqiang] revert r1392105 due to bylaw requirement mentioned by Carl 
Steinbach


Changes for Build #1713

Changes for Build #1714
[kevinwilfong] HIVE-3484. RetryingRawStore logic needs to be significantly 
reworked to support retries within transactions (Jean Xu via kevinwilfong)


Changes for Build #1715
[namit] HIVE-3495 For UDAFs, when generating a plan without 
map-side-aggregation, constant 
agg parameters will be replaced by ExprNodeColumnDesc (Yin Huai via namit)


Changes for Build #1716

Changes for Build #1717
[kevinwilfong] HIVE-3458. Parallel test script doesnt run all tests. (Ivan 
Gorbachev via kevinwilfong)

[hashutosh] HIVE-3481: : Hiveserver is not closing the existing 
driver handle before executing the next command. It results in to file handle 
leaks. (Kanna Karanam via Ashutosh Chauhan)


Changes for Build #1718

Changes for Build #1719
[kevinwilfong] HIVE-3498. hivetest.py fails with --revision option. (Ivan 
Gorbachev via kevinwilfong)


Changes for Build #1720



No tests ran.

The Apache Jenkins build system has built Hive-trunk-h0.21 (build #1720)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1720/ to 
view the results.

[jira] [Updated] (HIVE-3501) Track table and keys used in joins and group bys for logging

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3501:
-

   Resolution: Fixed
Fix Version/s: 0.10.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks Sambavi!

> Track table and keys used in joins and group bys for logging
> 
>
> Key: HIVE-3501
> URL: https://issues.apache.org/jira/browse/HIVE-3501
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Sambavi Muthukrishnan
>Assignee: Sambavi Muthukrishnan
>Priority: Minor
> Fix For: 0.10.0
>
> Attachments: table_access_keys.1.patch, table_access_keys.2.patch, 
> table_access_keys.3.patch, table_access_keys.4.patch, 
> table_access_keys.5.patch
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> For all operators that could benefit from bucketing, it will be useful to 
> keep track of and log the table names and key column names in order for the 
> operator to be converted to the bucketed version. This task is to track this 
> information for joins and group bys when the keys can be directly mapped back 
> to table scans and columns on that table. This information will be tracked on 
> the QueryPlan object so it is available to any pre/post execution hooks for 
> logging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira