[jira] [Commented] (HIVE-2828) make timestamp accessible in the hbase KeyValue

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167775#comment-14167775
 ] 

Phabricator commented on HIVE-2828:
---

ndimiduk added a subscriber: ndimiduk.
ndimiduk accepted this revision.
ndimiduk added a reviewer: ndimiduk.
ndimiduk added a comment.
This revision is now accepted and ready to land.

+1

BRANCH
  DPAL-861

INLINE COMMENTS
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java:154 
This is as good a decision as any. Maybe down the road someone will want to 
specify which cell's timestamp maps to which column?

REVISION DETAIL
  https://reviews.facebook.net/D1989

To: navis, JIRA, ndimiduk
Cc: ndimiduk


> make timestamp accessible in the hbase KeyValue 
> 
>
> Key: HIVE-2828
> URL: https://issues.apache.org/jira/browse/HIVE-2828
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.2.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.3.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.4.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-2828.D1989.5.patch, HIVE-2828.6.patch.txt, 
> HIVE-2828.7.patch.txt, HIVE-2828.8.patch.txt
>
>
> Originated from HIVE-2781 and not accepted, but I think this could be helpful 
> to someone.
> By using special column notation ':timestamp' in HBASE_COLUMNS_MAPPING, user 
> might access timestamp value in hbase KeyValue.
> {code}
> CREATE TABLE hbase_table (key int, value string, time timestamp)
>   STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
>   WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf:string,:timestamp")
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-5859) Create view does not captures inputs

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885104#comment-13885104
 ] 

Phabricator commented on HIVE-5859:
---

navis has abandoned the revision "HIVE-5859 [jira] Create view does not 
captures inputs".

REVISION DETAIL
  https://reviews.facebook.net/D14235

To: JIRA, navis
Cc: thejas


> Create view does not captures inputs
> 
>
> Key: HIVE-5859
> URL: https://issues.apache.org/jira/browse/HIVE-5859
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D14235.1.patch, HIVE-5859.2.patch.txt, 
> HIVE-5859.3.patch.txt
>
>
> For example, 
> CREATE VIEW view_j5jbymsx8e_1 as SELECT * FROM tbl_j5jbymsx8e;
> should capture "default.tbl_j5jbymsx8e" as input entity for authorization 
> process but currently it's not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5859) Create view does not captures inputs

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874143#comment-13874143
 ] 

Phabricator commented on HIVE-5859:
---

thejas has commented on the revision "HIVE-5859 [jira] Create view does not 
captures inputs".

INLINE COMMENTS
  ql/src/test/results/clientnegative/deletejar.q.out:1 This test result change 
does not look related to this change.
  ql/src/test/results/clientnegative/invalid_columns.q.out:1 This test result 
change does not look related to this change.

REVISION DETAIL
  https://reviews.facebook.net/D14235

To: JIRA, navis
Cc: thejas


> Create view does not captures inputs
> 
>
> Key: HIVE-5859
> URL: https://issues.apache.org/jira/browse/HIVE-5859
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D14235.1.patch, HIVE-5859.2.patch.txt
>
>
> For example, 
> CREATE VIEW view_j5jbymsx8e_1 as SELECT * FROM tbl_j5jbymsx8e;
> should capture "default.tbl_j5jbymsx8e" as input entity for authorization 
> process but currently it's not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859592#comment-13859592
 ] 

Phabricator commented on HIVE-5414:
---

thejas has commented on the revision "HIVE-5414 [jira] The result of show grant 
is not visible via JDBC".

  +1

REVISION DETAIL
  https://reviews.facebook.net/D13209

To: JIRA, navis
Cc: thejas


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, 
> D13209.4.patch, HIVE-5414.4.patch.txt, HIVE-5414.5.patch.txt
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Updated] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5414:
--

Attachment: D13209.4.patch

navis updated the revision "HIVE-5414 [jira] The result of show grant is not 
visible via JDBC".

  Addressed comments

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13209

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13209?vs=44271&id=46389#toc

AFFECTED FILES
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/unset_table_property.q.out
  ql/src/test/results/clientnegative/unset_view_property.q.out
  ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out
  ql/src/test/results/clientpositive/authorization_1.q.out
  ql/src/test/results/clientpositive/authorization_2.q.out
  ql/src/test/results/clientpositive/authorization_3.q.out
  ql/src/test/results/clientpositive/authorization_4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/authorization_6.q.out
  ql/src/test/results/clientpositive/keyword_1.q.out
  ql/src/test/results/clientpositive/show_tblproperties.q.out
  ql/src/test/results/clientpositive/unset_table_view_property.q.out

To: JIRA, navis
Cc: thejas


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, 
> D13209.4.patch, HIVE-5414.4.patch.txt, HIVE-5414.5.patch.txt
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849706#comment-13849706
 ] 

Phabricator commented on HIVE-5414:
---

thejas has commented on the revision "HIVE-5414 [jira] The result of show grant 
is not visible via JDBC".

  The patch does not apply on trunk anymore. Can you also please rebase ?
  Sorry about the delay in reviewing this patch.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java:36 I think 
"property" would be a better name for column, than "key".
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:1816 
can you also add test for column name and column type of result ?
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:1814 
can you also add a test for 'show roles' ?

REVISION DETAIL
  https://reviews.facebook.net/D13209

To: JIRA, navis
Cc: Unknown Object (Phabricator User), thejas


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch, 
> HIVE-5414.4.patch.txt
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846046#comment-13846046
 ] 

Phabricator commented on HIVE-2093:
---

thejas has commented on the revision "HIVE-2093 [jira] create/drop database 
should populate inputs/outputs and check concurrency and user permission".

  +1

REVISION DETAIL
  https://reviews.facebook.net/D12807

To: JIRA, navis
Cc: thejas


> create/drop database should populate inputs/outputs and check concurrency and 
> user permission
> -
>
> Key: HIVE-2093
> URL: https://issues.apache.org/jira/browse/HIVE-2093
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Locking, Metastore, Security
>Reporter: Namit Jain
>Assignee: Navis
> Attachments: D12807.3.patch, D12807.4.patch, HIVE-2093.6.patch, 
> HIVE-2093.7.patch.txt, HIVE-2093.8.patch.txt, HIVE-2093.9.patch.txt, 
> HIVE-2093.D12807.1.patch, HIVE-2093.D12807.2.patch, HIVE.2093.1.patch, 
> HIVE.2093.2.patch, HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch
>
>
> concurrency and authorization are needed for create/drop table. Also to make 
> concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS 
> DATABASE



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-2093:
--

Attachment: D12807.4.patch

navis updated the revision "HIVE-2093 [jira] create/drop database should 
populate inputs/outputs and check concurrency and user permission".

  addressed comments

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12807

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12807?vs=41577&id=45177#toc

AFFECTED FILES
  hbase-handler/src/test/results/negative/cascade_dbdrop_hadoop20.q.out
  
hcatalog/core/src/main/java/org/apache/hcatalog/security/HdfsAuthorizationProvider.java
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java
  ql/src/test/queries/clientnegative/authorization_fail_create_db.q
  ql/src/test/queries/clientnegative/authorization_fail_drop_db.q
  ql/src/test/queries/clientnegative/lockneg_query_tbl_in_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_db_lock_conflict.q
  ql/src/test/queries/clientnegative/lockneg_try_drop_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_lock_db_in_use.q
  ql/src/test/results/clientnegative/authorization_fail_2.q.out
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_6.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_fail_create_db.q.out
  ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty_restrict.q.out
  ql/src/test/results/clientnegative/exim_22_export_authfail.q.out
  ql/src/test/results/clientnegative/exim_23_import_exist_authfail.q.out
  ql/src/test/results/clientnegative/exim_24_import_part_authfail.q.out
  ql/src/test/results/clientnegative/exim_25_import_nonexist_authfail.q.out
  ql/src/test/results/clientnegative/join_nonexistent_part.q.out
  ql/src/test/results/clientnegative/load_exist_part_authfail.q.out
  ql/src/test/results/clientnegative/load_nonpart_authfail.q.out
  ql/src/test/results/clientnegative/load_part_authfail.q.out
  ql/src/test/results/clientnegative/lockneg_query_tbl_in_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_db_lock_conflict.q.out
  ql/src/test/results/clientnegative/lockneg_try_drop_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_lock_db_in_use.q.out
  ql/src/test/results/clientpositive/alter1.q.out
  ql/src/test/results/clientpositive/alter2.q.out
  ql/src/test/results/clientpositive/alter4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/database.q.out
  ql/src/test/results/clientpositive/database_drop.q.out
  ql/src/test/results/clientpositive/describe_database_json.q.out
  ql/src/test/results/clientpositive/drop_database_removes_partition_dirs.q.out
  ql/src/test/results/clientpositive/exim_00_nonpart_empty.q.out
  ql/src/test/results/clientpositive/exim_01_nonpart.q.out
  ql/src/test/results/clientpositive/exim_02_00_part_empty.q.out
  ql/src/test/results/clientpositive/exim_02_part.q.out
  ql/src/test/results/clientpositive/exim_03_nonpart_over_compat.q.out
  ql/src/test/results/clientpositive/exim_04_all_part.q.out
  ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out
  ql/src/test/results/clientpositive/exim_05_some_part.q.out
  ql/src/test/results/clientpositive/exim_06_one_part.q.out
  ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out
  ql/src/test/results/clientpositive/exim_

[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843900#comment-13843900
 ] 

Phabricator commented on HIVE-2093:
---

navis has commented on the revision "HIVE-2093 [jira] create/drop database 
should populate inputs/outputs and check concurrency and user permission".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:510 I've copied it from 
somewhere, forget where it was. Could you suggest better state code?
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:737 Ah, sure.
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:257 This method 
seemed not used in anywhere in current codebase, so it's hard to say the intent 
of initial contributor(Siying Dong). But this method seemed useful and I'll 
change it to return URI for database, too.
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:83 Some complex 
operations like "import" need different privilege checks with that of 
initiative operation. I think this part is from HIVE-2818 and mistakenly mixed 
with this issue. I'll remove it.
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:2290 ok.
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:1233 ok.
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java:104 
ok.

REVISION DETAIL
  https://reviews.facebook.net/D12807

To: JIRA, navis
Cc: thejas


> create/drop database should populate inputs/outputs and check concurrency and 
> user permission
> -
>
> Key: HIVE-2093
> URL: https://issues.apache.org/jira/browse/HIVE-2093
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Locking, Metastore, Security
>Reporter: Namit Jain
>Assignee: Navis
> Attachments: D12807.3.patch, HIVE-2093.6.patch, 
> HIVE-2093.7.patch.txt, HIVE-2093.8.patch.txt, HIVE-2093.D12807.1.patch, 
> HIVE-2093.D12807.2.patch, HIVE.2093.1.patch, HIVE.2093.2.patch, 
> HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch
>
>
> concurrency and authorization are needed for create/drop table. Also to make 
> concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS 
> DATABASE



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13841839#comment-13841839
 ] 

Phabricator commented on HIVE-2093:
---

thejas has commented on the revision "HIVE-2093 [jira] create/drop database 
should populate inputs/outputs and check concurrency and user permission".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:257 why not return 
the location uri here ?
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:1233 
what qnName means becomes clear only after reading the code, can you expand the 
variable name or add a javadoc comment ?
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:2290 can 
you update comment to say "SHOW LOCKS DATABASE [database] [extended]"
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java:104 
you can use - new Path(toURI)
  Its there since hadoop 0.20.2
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:83 why is this block 
of changes needed ?
  It does not seem to be used anyway. I think the separation between entity and 
privileges is a good thing.

REVISION DETAIL
  https://reviews.facebook.net/D12807

To: JIRA, navis
Cc: thejas


> create/drop database should populate inputs/outputs and check concurrency and 
> user permission
> -
>
> Key: HIVE-2093
> URL: https://issues.apache.org/jira/browse/HIVE-2093
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Locking, Metastore, Security
>Reporter: Namit Jain
>Assignee: Navis
> Attachments: D12807.3.patch, HIVE-2093.6.patch, 
> HIVE-2093.7.patch.txt, HIVE-2093.8.patch.txt, HIVE-2093.D12807.1.patch, 
> HIVE-2093.D12807.2.patch, HIVE.2093.1.patch, HIVE.2093.2.patch, 
> HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch
>
>
> concurrency and authorization are needed for create/drop table. Also to make 
> concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS 
> DATABASE



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840943#comment-13840943
 ] 

Phabricator commented on HIVE-2093:
---

thejas has commented on the revision "HIVE-2093 [jira] create/drop database 
should populate inputs/outputs and check concurrency and user permission".

  I have reviewed half of it, looks good mostly. I will complete the review 
tomorrow.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:510 What is the basis of 
the SQLState value ? what does it mean?
  Isn't class 01 warning ? But this is an error.
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:737 can you update the 
javadoc for this param ?

REVISION DETAIL
  https://reviews.facebook.net/D12807

To: JIRA, navis
Cc: thejas


> create/drop database should populate inputs/outputs and check concurrency and 
> user permission
> -
>
> Key: HIVE-2093
> URL: https://issues.apache.org/jira/browse/HIVE-2093
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Locking, Metastore, Security
>Reporter: Namit Jain
>Assignee: Navis
> Attachments: D12807.3.patch, HIVE-2093.6.patch, 
> HIVE-2093.7.patch.txt, HIVE-2093.8.patch.txt, HIVE-2093.D12807.1.patch, 
> HIVE-2093.D12807.2.patch, HIVE.2093.1.patch, HIVE.2093.2.patch, 
> HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch
>
>
> concurrency and authorization are needed for create/drop table. Also to make 
> concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS 
> DATABASE



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5342) Remove pre hadoop-0.20.0 related codes

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840787#comment-13840787
 ] 

Phabricator commented on HIVE-5342:
---

navis has abandoned the revision "HIVE-5342 [jira] Remove pre hadoop-0.20.0 
related codes".

REVISION DETAIL
  https://reviews.facebook.net/D13047

To: JIRA, navis


> Remove pre hadoop-0.20.0 related codes
> --
>
> Key: HIVE-5342
> URL: https://issues.apache.org/jira/browse/HIVE-5342
> Project: Hive
>  Issue Type: Task
>  Components: Shims
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13047.1.patch
>
>
> Recently, we discussed not supporting hadoop-0.20.0. If it would be done like 
> that or not, 0.17 related codes would be removed before that.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4790:
--

Attachment: D11511.4.patch

navis updated the revision "HIVE-4790 [jira] MapredLocalTask task does not make 
virtual columns".

  Fix test fails

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11511

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11511?vs=44409&id=44589#toc

AFFECTED FILES
  ql/pom.xml
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
  ql/src/test/queries/clientpositive/join_vc.q
  ql/src/test/results/clientpositive/join_vc.q.out

To: JIRA, navis


> MapredLocalTask task does not make virtual columns
> --
>
> Key: HIVE-4790
> URL: https://issues.apache.org/jira/browse/HIVE-4790
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11511.3.patch, D11511.4.patch, 
> HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
>
>
> From mailing list, 
> http://www.mail-archive.com/user@hive.apache.org/msg08264.html
> {noformat}
> SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON 
> b.rownumber = a.number;
> fails with this error:
>  
> > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber = 
> a.number;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> setting HADOOP_USER_NAMEpmarron
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property 
> hive.metastore.local no longer has any effect. Make sure to provide a valid 
> value for hive.metastore.uris if you are connecting to a remote metastore.
> Execution log at: /tmp/pmarron/.log
> 2013-06-25 10:52:56 Starting to launch local task to process map join;
>   maximum memory = 932118528
> java.lang.RuntimeException: cannot find field block__offset__inside__file 
> from [0:rownumber, 1:offset]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> at 
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
> at 
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at 
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
> at 
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Execution failed with exit status: 2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5025) Column aliases for input argument of GenericUDFs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5025:
--

Attachment: D12093.3.patch

navis updated the revision "HIVE-5025 [jira] Column aliases for input argument 
of GenericUDFs".

  Fixed test fails

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12093

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12093?vs=44403&id=44517#toc

AFFECTED FILES
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFColumnNameTest.java
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFColumnNameTest.java
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFColumnNameTest.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
  ql/src/test/queries/clientpositive/udf_col_names.q
  ql/src/test/results/clientpositive/udf_col_names.q.out
  ql/src/test/results/compiler/plan/groupby1.q.xml
  ql/src/test/results/compiler/plan/groupby2.q.xml
  ql/src/test/results/compiler/plan/groupby3.q.xml
  ql/src/test/results/compiler/plan/groupby5.q.xml
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java

To: JIRA, navis
Cc: reconnect.grayhat


> Column aliases for input argument of GenericUDFs 
> -
>
> Key: HIVE-5025
> URL: https://issues.apache.org/jira/browse/HIVE-5025
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D12093.2.patch, D12093.3.patch, HIVE-5025.D12093.1.patch
>
>
> In some cases, column aliases for input argument are very useful to know. But 
> I cannot sure of this in the sense that UDFs should not be dependent to 
> contextual information like column alias.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4790:
--

Attachment: D11511.3.patch

navis updated the revision "HIVE-4790 [jira] MapredLocalTask task does not make 
virtual columns".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11511

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11511?vs=35637&id=44409#toc

AFFECTED FILES
  ql/pom.xml
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java
  ql/src/test/queries/clientpositive/join_vc.q
  ql/src/test/results/clientpositive/join_vc.q.out

To: JIRA, navis


> MapredLocalTask task does not make virtual columns
> --
>
> Key: HIVE-4790
> URL: https://issues.apache.org/jira/browse/HIVE-4790
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11511.3.patch, HIVE-4790.D11511.1.patch, 
> HIVE-4790.D11511.2.patch
>
>
> From mailing list, 
> http://www.mail-archive.com/user@hive.apache.org/msg08264.html
> {noformat}
> SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON 
> b.rownumber = a.number;
> fails with this error:
>  
> > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber = 
> a.number;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> setting HADOOP_USER_NAMEpmarron
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property 
> hive.metastore.local no longer has any effect. Make sure to provide a valid 
> value for hive.metastore.uris if you are connecting to a remote metastore.
> Execution log at: /tmp/pmarron/.log
> 2013-06-25 10:52:56 Starting to launch local task to process map join;
>   maximum memory = 932118528
> java.lang.RuntimeException: cannot find field block__offset__inside__file 
> from [0:rownumber, 1:offset]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
> at 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
> at 
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> at 
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
> at 
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at 
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
> at 
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Execution failed with exit status: 2
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5025) Column aliases for input argument of GenericUDFs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5025:
--

Attachment: D12093.2.patch

navis updated the revision "HIVE-5025 [jira] Column aliases for input argument 
of GenericUDFs".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12093

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12093?vs=37359&id=44403#toc

AFFECTED FILES
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFColumnNameTest.java
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFColumnNameTest.java
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFColumnNameTest.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDeserializer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java
  ql/src/test/queries/clientpositive/udf_col_names.q
  ql/src/test/results/clientpositive/udf_col_names.q.out
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java

To: JIRA, navis


> Column aliases for input argument of GenericUDFs 
> -
>
> Key: HIVE-5025
> URL: https://issues.apache.org/jira/browse/HIVE-5025
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D12093.2.patch, HIVE-5025.D12093.1.patch
>
>
> In some cases, column aliases for input argument are very useful to know. But 
> I cannot sure of this in the sense that UDFs should not be dependent to 
> contextual information like column alias.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5414:
--

Attachment: D13209.3.patch

navis updated the revision "HIVE-5414 [jira] The result of show grant is not 
visible via JDBC".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13209

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13209?vs=40941&id=44271#toc

AFFECTED FILES
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/unset_table_property.q.out
  ql/src/test/results/clientnegative/unset_view_property.q.out
  ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out
  ql/src/test/results/clientpositive/authorization_1.q.out
  ql/src/test/results/clientpositive/authorization_2.q.out
  ql/src/test/results/clientpositive/authorization_3.q.out
  ql/src/test/results/clientpositive/authorization_4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/authorization_6.q.out
  ql/src/test/results/clientpositive/keyword_1.q.out
  ql/src/test/results/clientpositive/show_tblproperties.q.out
  ql/src/test/results/clientpositive/unset_table_view_property.q.out
  service/src/java/org/apache/hive/service/CompositeService.java

To: JIRA, navis


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch, D13209.3.patch
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3286) Explicit skew join on user provided condition

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3286:
--

Attachment: D4287.11.patch

navis updated the revision "HIVE-3286 [jira] Explicit skew join on user 
provided condition".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D4287

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D4287?vs=38511&id=44265#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
  ql/src/java/org/apache/hadoop/hive/ql/io/SkewedKeyPartitioner.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/InlineSkewJoinOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/JoinReorder.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBJoinTree.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/SkewContext.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java
  ql/src/test/queries/clientnegative/skewjoin_explicit_invalid1.q
  ql/src/test/queries/clientnegative/skewjoin_explicit_invalid2.q
  ql/src/test/queries/clientnegative/skewjoin_explicit_invalid3.q
  ql/src/test/queries/clientnegative/skewjoin_explicit_invalid4.q
  ql/src/test/queries/clientpositive/skewjoin_explicit.q
  ql/src/test/results/clientnegative/skewjoin_explicit_invalid1.q.out
  ql/src/test/results/clientnegative/skewjoin_explicit_invalid2.q.out
  ql/src/test/results/clientnegative/skewjoin_explicit_invalid3.q.out
  ql/src/test/results/clientnegative/skewjoin_explicit_invalid4.q.out
  ql/src/test/results/clientpositive/skewjoin_explicit.q.out

To: JIRA, navis
Cc: njain


> Explicit skew join on user provided condition
> -
>
> Key: HIVE-3286
> URL: https://issues.apache.org/jira/browse/HIVE-3286
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D4287.11.patch, HIVE-3286.D4287.10.patch, 
> HIVE-3286.D4287.5.patch, HIVE-3286.D4287.6.patch, HIVE-3286.D4287.7.patch, 
> HIVE-3286.D4287.8.patch, HIVE-3286.D4287.9.patch
>
>
> Join operation on table with skewed data takes most of execution time 
> handling the skewed keys. But mostly we already know about that and even know 
> what is look like the skewed keys.
> If we can explicitly assign reducer slots for the skewed keys, total 
> execution time could be greatly shortened.
> As for a start, I've extended join grammar something like this.
> {code}
> select * from src a join src b on a.key=b.key skew on (a.key+1 < 50, a.key+1 
> < 100, a.key < 150);
> {code}
> which means if above query is executed by 20 reducers, one reducer for 
> a.key+1 < 50, one reducer for 50 <= a.key+1 < 100, one reducer for 99 <= 
> a.key < 150, and 17 reducers for others (could be extended to assign more 
> than one reducer later)
> This can be only used with common-inner-equi joins. And skew condition should 
> be composed of join keys only.
> Work till done now will be updated shortly after code cleanup.
> 
> Skew expressions* in "SKEW ON (expr, expr, ...)" are evaluated sequentially 
> at runtime, and first 'true' one decides skew group for the row. Each skew 
> group has reserved partition slot(s), to which all rows in a group would be 
> assigned. 
> The number of partition slot reserved for each group is decided also at 
> runtime by simple calculation of percentage. If a skew group is "CLUSTER BY 
> 20 PERCENT" and total partition slot (=number of reducer) is 20, that group 
> will reserve 4 partition slots, etc.
> "DISTRIBUTE BY" decides how the rows in a group is dispersed in the range of 
> reserved slots (If there is only one slot for a group, this is meaningless). 
> Currently, three distribution policies are available: RANDOM, KEYS, 
> . 
> 1. RANDOM : rows of driver** alias are dispersed by random and rows of 
> non-driver alias are duplicated for all the slots (default if not specified)
> 2. KEYS : determined by hash value of keys (same with previous)
> 3. expression : determined by hash of 

[jira] [Updated] (HIVE-5859) Create view does not captures inputs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5859:
--

Attachment: D14235.1.patch

navis requested code review of "HIVE-5859 [jira] Create view does not captures 
inputs".

Reviewers: JIRA

HIVE-5859 Create view does not captures inputs

For example,
CREATE VIEW view_j5jbymsx8e_1 as SELECT * FROM tbl_j5jbymsx8e;

should capture "default.tbl_j5jbymsx8e" as input entity for authorization 
process but currently it's not.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D14235

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
  ql/src/test/results/clientnegative/alter_view_as_select_with_partition.q.out
  ql/src/test/results/clientnegative/alter_view_failure.q.out
  ql/src/test/results/clientnegative/alter_view_failure2.q.out
  ql/src/test/results/clientnegative/alter_view_failure4.q.out
  ql/src/test/results/clientnegative/alter_view_failure5.q.out
  ql/src/test/results/clientnegative/alter_view_failure6.q.out
  ql/src/test/results/clientnegative/alter_view_failure7.q.out
  ql/src/test/results/clientnegative/alter_view_failure9.q.out
  ql/src/test/results/clientnegative/analyze_view.q.out
  ql/src/test/results/clientnegative/create_or_replace_view1.q.out
  ql/src/test/results/clientnegative/create_or_replace_view2.q.out
  ql/src/test/results/clientnegative/create_or_replace_view4.q.out
  ql/src/test/results/clientnegative/create_or_replace_view5.q.out
  ql/src/test/results/clientnegative/create_or_replace_view6.q.out
  ql/src/test/results/clientnegative/create_or_replace_view7.q.out
  ql/src/test/results/clientnegative/create_or_replace_view8.q.out
  ql/src/test/results/clientnegative/create_view_failure1.q.out
  ql/src/test/results/clientnegative/create_view_failure2.q.out
  ql/src/test/results/clientnegative/create_view_failure4.q.out
  ql/src/test/results/clientnegative/deletejar.q.out
  ql/src/test/results/clientnegative/drop_table_failure2.q.out
  ql/src/test/results/clientnegative/insert_view_failure.q.out
  ql/src/test/results/clientnegative/invalid_columns.q.out
  ql/src/test/results/clientnegative/invalidate_view1.q.out
  ql/src/test/results/clientnegative/load_view_failure.q.out
  ql/src/test/results/clientnegative/recursive_view.q.out
  ql/src/test/results/clientnegative/unset_view_property.q.out
  ql/src/test/results/clientpositive/alter_view_as_select.q.out
  ql/src/test/results/clientpositive/alter_view_rename.q.out
  ql/src/test/results/clientpositive/authorization_8.q.out
  ql/src/test/results/clientpositive/create_big_view.q.out
  ql/src/test/results/clientpositive/create_like_view.q.out
  ql/src/test/results/clientpositive/create_or_replace_view.q.out
  ql/src/test/results/clientpositive/create_view.q.out
  ql/src/test/results/clientpositive/create_view_partitioned.q.out
  ql/src/test/results/clientpositive/create_view_translate.q.out
  ql/src/test/results/clientpositive/ctas_char.q.out
  ql/src/test/results/clientpositive/ctas_date.q.out
  ql/src/test/results/clientpositive/ctas_varchar.q.out
  ql/src/test/results/clientpositive/database_drop.q.out
  ql/src/test/results/clientpositive/describe_formatted_view_partitioned.q.out
  
ql/src/test/results/clientpositive/describe_formatted_view_partitioned_json.q.out
  ql/src/test/results/clientpositive/explain_dependency.q.out
  ql/src/test/results/clientpositive/explain_logical.q.out
  ql/src/test/results/clientpositive/join_view.q.out
  ql/src/test/results/clientpositive/lateral_view_noalias.q.out
  ql/src/test/results/clientpositive/ppd_union_view.q.out
  ql/src/test/results/clientpositive/ptf.q.out
  ql/src/test/results/clientpositive/show_create_table_view.q.out
  ql/src/test/results/clientpositive/subquery_exists.q.out
  ql/src/test/results/clientpositive/unset_table_view_property.q.out
  ql/src/test/results/clientpositive/view.q.out
  ql/src/test/results/clientpositive/view_cast.q.out
  ql/src/test/results/clientpositive/view_inputs.q.out
  ql/src/test/results/clientpositive/windowing.q.out

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/45537/

To: JIRA, navis


> Create view does not captures inputs
> 
>
> Key: HIVE-5859
> URL: https://issues.apache.org/jira/browse/HIVE-5859
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D14235.1.patch
>
>
> For example, 
> CREATE VIEW view_j5jbymsx8e_1 as SELECT * FROM tbl_j5jbymsx8e;
> should capture "def

[jira] [Updated] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4632:
--

Attachment: D11001.4.patch

navis updated the revision "HIVE-4632 [jira] Use hadoop counter as a stat 
publisher".

  1. Fixed tests
  2. Made StatsFactory thread-safe

Reviewers: ashutoshc, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11001

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11001?vs=43839&id=43989#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  data/conf/hive-site.xml
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java
  itests/qtest/pom.xml
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/KeyVerifyingStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsSetupConst.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java
  ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
  ql/src/test/queries/clientnegative/stats_aggregator_error_2.q
  ql/src/test/queries/clientnegative/stats_publisher_error_1.q
  ql/src/test/queries/clientnegative/stats_publisher_error_2.q
  ql/src/test/queries/clientpositive/stats19.q
  ql/src/test/queries/clientpositive/stats_aggregator_error_1.q
  ql/src/test/queries/clientpositive/stats_counter.q
  ql/src/test/queries/clientpositive/stats_publisher_error_1.q
  ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out
  ql/src/test/results/clientpositive/stats_counter.q.out
  ql/src/test/results/compiler/plan/case_sensitivity.q.xml
  ql/src/test/results/compiler/plan/groupby1.q.xml
  ql/src/test/results/compiler/plan/input1.q.xml
  ql/src/test/results/compiler/plan/input2.q.xml
  ql/src/test/results/compiler/plan/input3.q.xml
  ql/src/test/results/compiler/plan/input4.q.xml
  ql/src/test/results/compiler/plan/input5.q.xml
  ql/src/test/results/compiler/plan/input6.q.xml
  ql/src/test/results/compiler/plan/input7.q.xml
  ql/src/test/results/compiler/plan/input9.q.xml
  ql/src/test/results/compiler/plan/input_testsequencefile.q.xml
  ql/src/test/results/compiler/plan/join1.q.xml
  ql/src/test/results/compiler/plan/join2.q.xml
  ql/src/test/results/compiler/plan/join3.q.xml
  ql/src/test/results/compiler/plan/sample2.q.xml
  ql/src/test/results/compiler/plan/sample3.q.xml
  ql/src/test/results/compiler/plan/sample4.q.xml
  ql/src/test/results/compiler/plan/sample5.q.xml
  ql/src/test/results/compiler/plan/sample6.q.xml
  ql/src/test/results/compiler/plan/sample7.q.xml

To: JIRA, ashutoshc, navis
Cc: ashutoshc


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11001.4.patch, HIVE-4632.4.patch.txt, 
> HIVE-4632.5.patch.txt, HIVE-4632.6.patch.txt
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825175#comment-13825175
 ] 

Phabricator commented on HIVE-4632:
---

ashutoshc has requested changes to the revision "HIVE-4632 [jira] Use hadoop 
counter as a stat publisher".

  Thanks for making changes. Lets also have counter as default in HiveConf.java

INLINE COMMENTS
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:606 As I said on 
jira, lets have counter as default.

REVISION DETAIL
  https://reviews.facebook.net/D11001

BRANCH
  HIVE-4632

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis
Cc: ashutoshc


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-4632.4.patch.txt, HIVE-4632.5.patch.txt
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4632:
--

Attachment: D11001.3.patch

navis updated the revision "HIVE-4632 [jira] Use hadoop counter as a stat 
publisher".

  Addressed comments

Reviewers: ashutoshc, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11001

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11001?vs=43617&id=43839#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  data/conf/hive-site.xml
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java
  itests/qtest/pom.xml
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/KeyVerifyingStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsSetupConst.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java
  ql/src/test/queries/clientpositive/stats_counter.q
  ql/src/test/results/clientpositive/stats_counter.q.out

To: JIRA, ashutoshc, navis
Cc: ashutoshc


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11001.2.patch, D11001.3.patch, HIVE-4632.D11001.1.patch
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825094#comment-13825094
 ] 

Phabricator commented on HIVE-4632:
---

navis has commented on the revision "HIVE-4632 [jira] Use hadoop counter as a 
stat publisher".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java:43 I 
was not sure it's always a JobConf. I'll check the type of conf and make new 
JobConf only if needed.
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java:69 
done.
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java:48 I 
always wondered why there is no string or double type counters, which might be 
useful for complex MR jobs if exists. I'll remove this part.

REVISION DETAIL
  https://reviews.facebook.net/D11001

BRANCH
  HIVE-4632

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis
Cc: ashutoshc


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11001.2.patch, HIVE-4632.D11001.1.patch
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824206#comment-13824206
 ] 

Phabricator commented on HIVE-4632:
---

ashutoshc has requested changes to the revision "HIVE-4632 [jira] Use hadoop 
counter as a stat publisher".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java:43 
Instead of new JobConf(hconf, ExecDriver.class), doing (JobConf) hconf is 
better, ie instead of creating new JobConf object, cast hconf to JobConf.
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java:48 I 
don't think we need to do this check isDecimals(). Counters by definition are 
countable, so they must be of numeric type. If they are not, than its bug, 
which we shouldn't hide.
  Thing is this method will be called on every row, so its lots of wasted 
compute here.
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java:69 
Good to do LOG.error(e) here.

REVISION DETAIL
  https://reviews.facebook.net/D11001

BRANCH
  HIVE-4632

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis
Cc: ashutoshc


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11001.2.patch, HIVE-4632.D11001.1.patch
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4632) Use hadoop counter as a stat publisher

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4632:
--

Attachment: D11001.2.patch

navis updated the revision "HIVE-4632 [jira] Use hadoop counter as a stat 
publisher".

  Rebased to trunk & added a test case

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11001

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11001?vs=34113&id=43617#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java
  itests/qtest/pom.xml
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/stats/KeyVerifyingStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsSetupConst.java
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestStatsPublisherEnhanced.java
  ql/src/test/queries/clientpositive/stats_counter.q
  ql/src/test/results/clientpositive/stats_counter.q.out

To: JIRA, navis


> Use hadoop counter as a stat publisher
> --
>
> Key: HIVE-4632
> URL: https://issues.apache.org/jira/browse/HIVE-4632
> Project: Hive
>  Issue Type: Improvement
>  Components: Statistics
>Affects Versions: 0.12.0
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11001.2.patch, HIVE-4632.D11001.1.patch
>
>
> Currently stats are all long/aggregation type and can be safely acquired by 
> hadoop counter without other db or hbase.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4293) Predicates following UDTF operator are removed by PPD

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4293:
--

Attachment: D9933.6.patch

navis updated the revision "HIVE-4293 [jira] Predicates following UDTF operator 
are removed by PPD".

  Rebased to trunk & improve test case a little

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D9933

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D9933?vs=38595&id=43329#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java
  ql/src/test/queries/clientpositive/lateral_view_ppd.q
  ql/src/test/queries/clientpositive/ppd_udtf.q
  ql/src/test/results/clientpositive/cluster.q.out
  ql/src/test/results/clientpositive/ctas_colname.q.out
  ql/src/test/results/clientpositive/lateral_view_ppd.q.out
  ql/src/test/results/clientpositive/ppd2.q.out
  ql/src/test/results/clientpositive/ppd_gby.q.out
  ql/src/test/results/clientpositive/ppd_gby2.q.out
  ql/src/test/results/clientpositive/ppd_udtf.q.out
  ql/src/test/results/clientpositive/udtf_json_tuple.q.out
  ql/src/test/results/clientpositive/udtf_parse_url_tuple.q.out
  ql/src/test/results/compiler/plan/join1.q.xml
  ql/src/test/results/compiler/plan/join2.q.xml
  ql/src/test/results/compiler/plan/join3.q.xml
  ql/src/test/results/compiler/plan/join4.q.xml
  ql/src/test/results/compiler/plan/join5.q.xml
  ql/src/test/results/compiler/plan/join6.q.xml
  ql/src/test/results/compiler/plan/join7.q.xml
  ql/src/test/results/compiler/plan/join8.q.xml

To: JIRA, navis


> Predicates following UDTF operator are removed by PPD
> -
>
> Key: HIVE-4293
> URL: https://issues.apache.org/jira/browse/HIVE-4293
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
> Attachments: D9933.6.patch, HIVE-4293.D9933.1.patch, 
> HIVE-4293.D9933.2.patch, HIVE-4293.D9933.3.patch, HIVE-4293.D9933.4.patch, 
> HIVE-4293.D9933.5.patch
>
>
> For example, 
> {noformat}
> explain SELECT value from (
>   select explode(array(key, value)) as (value) from (
> select * FROM src WHERE key > 200
>   ) A
> ) B WHERE value > 300
> ;
> {noformat}
> Makes plan like this, removing last predicates
> {noformat}
>   TableScan
> alias: src
> Filter Operator
>   predicate:
>   expr: (key > 200.0)
>   type: boolean
>   Select Operator
> expressions:
>   expr: array(key,value)
>   type: array
> outputColumnNames: _col0
> UDTF Operator
>   function name: explode
>   Select Operator
> expressions:
>   expr: col
>   type: string
> outputColumnNames: _col0
> File Output Operator
>   compressed: false
>   GlobalTableId: 0
>   table:
>   input format: org.apache.hadoop.mapred.TextInputFormat
>   output format: 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5722) Skip generating vectorization code if possible

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5722:
--

Attachment: D13941.1.patch

navis requested code review of "HIVE-5722 [jira] Skip generating vectorization 
code if possible".

Reviewers: JIRA

HIVE-5722 Skip generating vectorization code if possible

Currently, ql module always generates new vectorization code, which might not 
be changed so frequently.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13941

AFFECTED FILES
  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java
  ant/src/org/apache/hadoop/hive/ant/GenVectorTestCode.java

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/42663/

To: JIRA, navis


> Skip generating vectorization code if possible
> --
>
> Key: HIVE-5722
> URL: https://issues.apache.org/jira/browse/HIVE-5722
> Project: Hive
>  Issue Type: Improvement
>  Components: Build Infrastructure
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D13941.1.patch, HIVE-5722.1.patch.txt
>
>
> Currently, ql module always generates new vectorization code, which might not 
> be changed so frequently.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3990) Provide input threshold for direct-fetcher (HIVE-2925)

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3990:
--

Attachment: D8415.3.patch

navis updated the revision "HIVE-3990 [jira] Provide input threshold for 
direct-fetcher (HIVE-2925)".

  Rebased to trunk & Applied InputEstimator to MapJoin big table decision 
process

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8415

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8415?vs=42879&id=43035#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
  ql/src/java/org/apache/hadoop/hive/ql/metadata/InputEstimator.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q
  ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out

To: JIRA, navis


> Provide input threshold for direct-fetcher (HIVE-2925)
> --
>
> Key: HIVE-3990
> URL: https://issues.apache.org/jira/browse/HIVE-3990
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D8415.2.patch, D8415.3.patch, HIVE-3990.D8415.1.patch
>
>
> As a followup of HIVE-2925, add input threshold for fetch task conversion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4880) Rearrange explain order of stages simpler

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4880:
--

Attachment: D11733.4.patch

navis updated the revision "HIVE-4880 [jira] Rearrange explain order of stages 
simpler".

  Rebased to trunk & fixed test result (was partially merged with other 
patch(HIVE-3925))

Reviewers: hagleitn, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11733

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11733?vs=42009&id=42975#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/ObjectPair.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
  ql/src/test/queries/clientpositive/explain_rearrange.q
  ql/src/test/results/clientpositive/explain_rearrange.q.out

To: JIRA, hagleitn, navis
Cc: brock


> Rearrange explain order of stages simpler
> -
>
> Key: HIVE-4880
> URL: https://issues.apache.org/jira/browse/HIVE-4880
> Project: Hive
>  Issue Type: Improvement
>  Components: Diagnosability
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D11733.2.patch, D11733.3.patch, D11733.4.patch, 
> HIVE-4880.D11733.1.patch
>
>
> Current explain order of stages is not best form to read (and to debug)
> {noformat}
> Before:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
>   Stage-4
>   Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
>   Stage-9 depends on stages: Stage-0
>   Stage-2 depends on stages: Stage-9
>   Stage-3
>   Stage-5
>   Stage-6 depends on stages: Stage-5
> After:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-2 depends on stages: Stage-1 , consists of Stage-3, Stage-4, Stage-5
>   Stage-3
>   Stage-4
>   Stage-5
>   Stage-6 depends on stages: Stage-5
>   Stage-7 depends on stages: Stage-3, Stage-4, Stage-6
>   Stage-8 depends on stages: Stage-7
>   Stage-9 depends on stages: Stage-8
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5718:
--

Attachment: D13857.3.patch

navis updated the revision "HIVE-5718 [jira] Support direct fetch for lateral 
views, sub queries, etc.".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13857

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13857?vs=42957&id=42963#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
  ql/src/test/queries/clientpositive/nonmr_fetch.q
  ql/src/test/results/clientpositive/explain_logical.q.out
  ql/src/test/results/clientpositive/lateral_view_noalias.q.out
  ql/src/test/results/clientpositive/nonmr_fetch.q.out
  ql/src/test/results/clientpositive/udf_explode.q.out
  ql/src/test/results/clientpositive/udf_inline.q.out
  ql/src/test/results/clientpositive/udf_reflect2.q.out
  ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out
  ql/src/test/results/clientpositive/udtf_explode.q.out

To: JIRA, navis


> Support direct fetch for lateral views, sub queries, etc.
> -
>
> Key: HIVE-5718
> URL: https://issues.apache.org/jira/browse/HIVE-5718
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch
>
>
> Extend HIVE-2925 with LV and SubQ.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5718:
--

Attachment: D13857.2.patch

navis updated the revision "HIVE-5718 [jira] Support direct fetch for lateral 
views, sub queries, etc.".

  1. Fixed trivial test changes
  2. Fixed missing view dependency
  3. Use OperatorFactory for creating ListSinkOperator

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13857

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13857?vs=42867&id=42957#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
  ql/src/test/queries/clientpositive/nonmr_fetch.q
  ql/src/test/results/clientpositive/explain_logical.q.out
  ql/src/test/results/clientpositive/lateral_view_noalias.q.out
  ql/src/test/results/clientpositive/nonmr_fetch.q.out
  ql/src/test/results/clientpositive/udf_explode.q.out
  ql/src/test/results/clientpositive/udf_inline.q.out
  ql/src/test/results/clientpositive/udf_reflect2.q.out
  ql/src/test/results/clientpositive/udf_to_unix_timestamp.q.out
  ql/src/test/results/clientpositive/udtf_explode.q.out

To: JIRA, navis


> Support direct fetch for lateral views, sub queries, etc.
> -
>
> Key: HIVE-5718
> URL: https://issues.apache.org/jira/browse/HIVE-5718
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13857.1.patch, D13857.2.patch
>
>
> Extend HIVE-2925 with LV and SubQ.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3990) Provide input threshold for direct-fetcher (HIVE-2925)

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3990:
--

Attachment: D8415.2.patch

navis updated the revision "HIVE-3990 [jira] Provide input threshold for 
direct-fetcher (HIVE-2925)".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8415

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8415?vs=27291&id=42879#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
  ql/src/java/org/apache/hadoop/hive/ql/metadata/InputEstimator.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/test/queries/clientpositive/nonmr_fetch_threshold.q
  ql/src/test/results/clientpositive/nonmr_fetch_threshold.q.out

To: JIRA, navis


> Provide input threshold for direct-fetcher (HIVE-2925)
> --
>
> Key: HIVE-3990
> URL: https://issues.apache.org/jira/browse/HIVE-3990
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D8415.2.patch, HIVE-3990.D8415.1.patch
>
>
> As a followup of HIVE-2925, add input threshold for fetch task conversion.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5718:
--

Attachment: D13857.1.patch

navis requested code review of "HIVE-5718 [jira] Support direct fetch for 
lateral views, sub queries, etc.".

Reviewers: JIRA

HIVE-5718 Support direct fetch for lateral views, sub queries, etc.

Extend HIVE-2925 with LV and SubQ.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13857

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
  ql/src/test/queries/clientpositive/nonmr_fetch.q
  ql/src/test/results/clientpositive/lateral_view_noalias.q.out
  ql/src/test/results/clientpositive/nonmr_fetch.q.out
  ql/src/test/results/clientpositive/udf_explode.q.out
  ql/src/test/results/clientpositive/udtf_explode.q.out

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/42153/

To: JIRA, navis


> Support direct fetch for lateral views, sub queries, etc.
> -
>
> Key: HIVE-5718
> URL: https://issues.apache.org/jira/browse/HIVE-5718
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13857.1.patch
>
>
> Extend HIVE-2925 with LV and SubQ.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5707) Validate values for ConfVar

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5707:
--

Attachment: D13821.2.patch

navis updated the revision "HIVE-5707 [jira] Validate values for ConfVar".

  Fixed orc_create (cannot reproduce fail of bucket_num_reducers)

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13821

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13821?vs=42753&id=42861#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java
  ql/src/test/queries/clientnegative/set_hiveconf_validation2.q
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientnegative/set_hiveconf_validation2.q.out

To: JIRA, navis


> Validate values for ConfVar
> ---
>
> Key: HIVE-5707
> URL: https://issues.apache.org/jira/browse/HIVE-5707
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13821.1.patch, D13821.2.patch
>
>
> with set hive.conf.validation=true, hive validates new value can be changed 
> to the type. But it does not check value itself.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5657) TopN produces incorrect results with count(distinct)

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810688#comment-13810688
 ] 

Phabricator commented on HIVE-5657:
---

sershe has commented on the revision "HIVE-5657 [jira] TopN produces incorrect 
results with count(distinct)".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:387 this 
changes the meaning to now refer to the field "this.value". Is it intended? 
Previously it was not used. Perhaps the first "if" should be removed if not 
used on purpose

REVISION DETAIL
  https://reviews.facebook.net/D13797

To: JIRA, navis
Cc: sershe


> TopN produces incorrect results with count(distinct)
> 
>
> Key: HIVE-5657
> URL: https://issues.apache.org/jira/browse/HIVE-5657
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Navis
>Priority: Critical
> Attachments: D13797.1.patch, D13797.2.patch, HIVE-5657.1.patch.txt, 
> example.patch
>
>
> Attached patch illustrates the problem.
> limit_pushdown test has various other cases of aggregations and distincts, 
> incl. count-distinct, that work correctly (that said, src dataset is bad for 
> testing these things because every count, for example, produces one record 
> only), so something must be special about this.
> I am not very familiar with distinct- code and these nuances; if someone 
> knows a quick fix feel free to take this, otherwise I will probably start 
> looking next week. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5707) Validate values for ConfVar

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5707:
--

Attachment: D13821.1.patch

navis requested code review of "HIVE-5707 [jira] Validate values for ConfVar".

Reviewers: JIRA

HIVE-5707 Validate values for ConfVar

with set hive.conf.validation=true, hive validates new value can be changed to 
the type. But it does not check value itself.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13821

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java
  ql/src/test/queries/clientnegative/set_hiveconf_validation2.q
  ql/src/test/results/clientnegative/set_hiveconf_validation2.q.out

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/41961/

To: JIRA, navis


> Validate values for ConfVar
> ---
>
> Key: HIVE-5707
> URL: https://issues.apache.org/jira/browse/HIVE-5707
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13821.1.patch
>
>
> with set hive.conf.validation=true, hive validates new value can be changed 
> to the type. But it does not check value itself.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5657) TopN produces incorrect results with count(distinct)

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5657:
--

Attachment: D13797.2.patch

navis updated the revision "HIVE-5657 [jira] TopN produces incorrect results 
with count(distinct)".

  1. Minimized diff
  2. Support multi-distinct cases

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13797

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13797?vs=42645&id=42747#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveKey.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
  ql/src/test/queries/clientpositive/limit_pushdown.q
  ql/src/test/queries/clientpositive/limit_pushdown_negative.q
  ql/src/test/results/clientpositive/limit_pushdown.q.out
  ql/src/test/results/clientpositive/limit_pushdown_negative.q.out

To: JIRA, navis
Cc: sershe


> TopN produces incorrect results with count(distinct)
> 
>
> Key: HIVE-5657
> URL: https://issues.apache.org/jira/browse/HIVE-5657
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Navis
>Priority: Critical
> Attachments: D13797.1.patch, D13797.2.patch, example.patch, 
> HIVE-5657.1.patch.txt
>
>
> Attached patch illustrates the problem.
> limit_pushdown test has various other cases of aggregations and distincts, 
> incl. count-distinct, that work correctly (that said, src dataset is bad for 
> testing these things because every count, for example, produces one record 
> only), so something must be special about this.
> I am not very familiar with distinct- code and these nuances; if someone 
> knows a quick fix feel free to take this, otherwise I will probably start 
> looking next week. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5657) TopN produces incorrect results with count(distinct)

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809811#comment-13809811
 ] 

Phabricator commented on HIVE-5657:
---

navis has commented on the revision "HIVE-5657 [jira] TopN produces incorrect 
results with count(distinct)".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:268 Right. 
it should be -1. I did mistake doing some refactoring.
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java:255 For distinct, it 
does not store values. Check the key and decide to forward all or exclude all. 
I'm not sure that the previous version was better. In this time, I've focused 
simplifying the flow of RS-op.
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:255 Yes 
right. Previously, the key was like this
  [distributeKey:distinctKey1]
  [distributeKey:distinctKey2]
  and each row is serialized in whole by OI
  structOI[structOI(distributeKey):UnionOI(distinctKey)]

  Now the key is prepared like this and
  [distributeKey]
  [distinctKey1,distinctKey2]

  serialized for each part directly by inner OI : structOI(distributeKey) and 
UnionOI(distinctKey)

  I'm not feel good introducing new interface KeySerializer. But serializing 
distributeKey multiple time seemed worse than that.
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:125 
yes.
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java:211 Changed 
the name because it was confusing that RS is for MapAggr GBY, which is not.
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:243 I 
didn't know there was VectorReduceSinkOperator when I've started this, which 
made me include more refactorings than just amount of fixing the problem. I 
think current version of patch is way simpler than that of original. But if it 
makes merging of vectorization hard, I might create minimal patch just for fix.

REVISION DETAIL
  https://reviews.facebook.net/D13797

To: JIRA, navis
Cc: sershe


> TopN produces incorrect results with count(distinct)
> 
>
> Key: HIVE-5657
> URL: https://issues.apache.org/jira/browse/HIVE-5657
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Navis
>Priority: Critical
> Attachments: D13797.1.patch, example.patch, HIVE-5657.1.patch.txt
>
>
> Attached patch illustrates the problem.
> limit_pushdown test has various other cases of aggregations and distincts, 
> incl. count-distinct, that work correctly (that said, src dataset is bad for 
> testing these things because every count, for example, produces one record 
> only), so something must be special about this.
> I am not very familiar with distinct- code and these nuances; if someone 
> knows a quick fix feel free to take this, otherwise I will probably start 
> looking next week. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5657) TopN produces incorrect results with count(distinct)

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809784#comment-13809784
 ] 

Phabricator commented on HIVE-5657:
---

sershe has commented on the revision "HIVE-5657 [jira] TopN produces incorrect 
results with count(distinct)".

INLINE COMMENTS
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:125 
so this now supports any number of distincts?
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java:255 right now this 
only returns forward... is this by design?
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:243 should 
all of this also be done for vectorized path?
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:268 I 
fixed it in my patch for vectorized... why is hash needed here?
  If row is excluded we don't need hash, it's only needed when we store the 
value or collect
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:297 if 
index >= 0 this should store value
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java:255 
Previously there was just key, which was some columns and optionally one 
distinct.
  Do I read correctly that distribution key is now the same, just without 
distinct?

REVISION DETAIL
  https://reviews.facebook.net/D13797

To: JIRA, navis
Cc: sershe


> TopN produces incorrect results with count(distinct)
> 
>
> Key: HIVE-5657
> URL: https://issues.apache.org/jira/browse/HIVE-5657
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Navis
>Priority: Critical
> Attachments: D13797.1.patch, example.patch, HIVE-5657.1.patch.txt
>
>
> Attached patch illustrates the problem.
> limit_pushdown test has various other cases of aggregations and distincts, 
> incl. count-distinct, that work correctly (that said, src dataset is bad for 
> testing these things because every count, for example, produces one record 
> only), so something must be special about this.
> I am not very familiar with distinct- code and these nuances; if someone 
> knows a quick fix feel free to take this, otherwise I will probably start 
> looking next week. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5657) TopN produces incorrect results with count(distinct)

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5657:
--

Attachment: D13797.1.patch

navis requested code review of "HIVE-5657 [jira] TopN produces incorrect 
results with count(distinct)".

Reviewers: JIRA

HIVE-5657 TopN produces incorrect results with count(distinct)

Attached patch illustrates the problem.
limit_pushdown test has various other cases of aggregations and distincts, 
incl. count-distinct, that work correctly (that said, src dataset is bad for 
testing these things because every count, for example, produces one record 
only), so something must be special about this.
I am not very familiar with distinct- code and these nuances; if someone knows 
a quick fix feel free to take this, otherwise I will probably start looking 
next week.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13797

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNHash.java
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
  ql/src/test/queries/clientpositive/limit_pushdown.q
  ql/src/test/queries/clientpositive/limit_pushdown_negative.q
  ql/src/test/results/clientpositive/limit_pushdown.q.out
  ql/src/test/results/clientpositive/limit_pushdown_negative.q.out
  serde/src/java/org/apache/hadoop/hive/serde2/KeySerializer.java
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
  
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/41811/

To: JIRA, navis


> TopN produces incorrect results with count(distinct)
> 
>
> Key: HIVE-5657
> URL: https://issues.apache.org/jira/browse/HIVE-5657
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Navis
>Priority: Critical
> Attachments: D13797.1.patch, example.patch, HIVE-5657.1.patch.txt
>
>
> Attached patch illustrates the problem.
> limit_pushdown test has various other cases of aggregations and distincts, 
> incl. count-distinct, that work correctly (that said, src dataset is bad for 
> testing these things because every count, for example, produces one record 
> only), so something must be special about this.
> I am not very familiar with distinct- code and these nuances; if someone 
> knows a quick fix feel free to take this, otherwise I will probably start 
> looking next week. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5690) Support subquery for single sourced multi query

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5690:
--

Attachment: D13791.1.patch

navis requested code review of "HIVE-5690 [jira] Support subquery for single 
sourced multi query".

Reviewers: JIRA

logs

Single sourced multi (insert) query is very useful for various ETL processes 
but it does not allow subqueries included. For example,

explain from src
insert overwrite table x1 select * from (select distinct key,value) b order by 
key
insert overwrite table x2 select * from (select distinct key,value) c order by 
value;

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13791

AFFECTED FILES
  build.properties
  ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/test/queries/clientpositive/multi_insert_subquery.q
  ql/src/test/results/clientpositive/multi_insert_subquery.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/41733/

To: JIRA, navis


> Support subquery for single sourced multi query
> ---
>
> Key: HIVE-5690
> URL: https://issues.apache.org/jira/browse/HIVE-5690
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D13791.1.patch
>
>
> Single sourced multi (insert) query is very useful for various ETL processes 
> but it does not allow subqueries included. For example, 
> {noformat}
> explain from src 
> insert overwrite table x1 select * from (select distinct key,value) b order 
> by key
> insert overwrite table x2 select * from (select distinct key,value) c order 
> by value;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5350) Cleanup exception handling around parallel orderby

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13802567#comment-13802567
 ] 

Phabricator commented on HIVE-5350:
---

brock has accepted the revision "HIVE-5350 [jira] Cleanup exception handling 
around parallel orderby".

REVISION DETAIL
  https://reviews.facebook.net/D13617

BRANCH
  HIVE-5350

ARCANIST PROJECT
  hive

To: JIRA, brock, navis


> Cleanup exception handling around parallel orderby
> --
>
> Key: HIVE-5350
> URL: https://issues.apache.org/jira/browse/HIVE-5350
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Navis
>Priority: Minor
> Attachments: D13617.1.patch
>
>
> I think we should log the message to the console and the full exception to 
> the log:
> ExecDriver:
> {noformat}
> try {
>   handleSampling(driverContext, mWork, job, conf);
>   job.setPartitionerClass(HiveTotalOrderPartitioner.class);
> } catch (Exception e) {
>   console.printInfo("Not enough sampling data.. Rolling back to 
> single reducer task");
>   rWork.setNumReduceTasks(1);
>   job.setNumReduceTasks(1);
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5350) Cleanup exception handling around parallel orderby

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5350:
--

Attachment: D13617.1.patch

navis requested code review of "HIVE-5350 [jira] Cleanup exception handling 
around parallel orderby".

Reviewers: JIRA

HIVE-5350 Cleanup exception handling around parallel orderby

I think we should log the message to the console and the full exception to the 
log:

ExecDriver:

try {
  handleSampling(driverContext, mWork, job, conf);
  job.setPartitionerClass(HiveTotalOrderPartitioner.class);
} catch (Exception e) {
  console.printInfo("Not enough sampling data.. Rolling back to single 
reducer task");
  rWork.setNumReduceTasks(1);
  job.setNumReduceTasks(1);
}

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13617

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/41013/

To: JIRA, navis


> Cleanup exception handling around parallel orderby
> --
>
> Key: HIVE-5350
> URL: https://issues.apache.org/jira/browse/HIVE-5350
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Navis
>Priority: Minor
> Attachments: D13617.1.patch
>
>
> I think we should log the message to the console and the full exception to 
> the log:
> ExecDriver:
> {noformat}
> try {
>   handleSampling(driverContext, mWork, job, conf);
>   job.setPartitionerClass(HiveTotalOrderPartitioner.class);
> } catch (Exception e) {
>   console.printInfo("Not enough sampling data.. Rolling back to 
> single reducer task");
>   rWork.setNumReduceTasks(1);
>   job.setNumReduceTasks(1);
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3972) Support using multiple reducer for fetching order by results

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3972:
--

Attachment: D8349.7.patch

navis updated the revision "HIVE-3972 [jira] Support using multiple reducer for 
fetching order by results".

  Fix build fail

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8349

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8349?vs=42123&id=42177#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MergeSortingFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/RowFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
  ql/src/test/queries/clientpositive/orderby_query_bucketing.q
  ql/src/test/results/clientpositive/orderby_query_bucketing.q.out

To: JIRA, navis
Cc: njain


> Support using multiple reducer for fetching order by results
> 
>
> Key: HIVE-3972
> URL: https://issues.apache.org/jira/browse/HIVE-3972
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D8349.5.patch, D8349.6.patch, D8349.7.patch, 
> HIVE-3972.D8349.1.patch, HIVE-3972.D8349.2.patch, HIVE-3972.D8349.3.patch, 
> HIVE-3972.D8349.4.patch
>
>
> Queries for fetching results which have lastly "order by" clause make final 
> MR run with single reducer, which can be too much. For example, 
> {code}
> select value, sum(key) as sum from src group by value order by sum;
> {code}
> If number of reducer is reasonable, multiple result files could be merged 
> into single sorted stream in the fetcher level.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5220) Add option for removing intermediate directory for partition, which is empty

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5220:
--

Attachment: D12729.2.patch

navis updated the revision "HIVE-5220 [jira] Add option for removing 
intermediate directory for partition, which is empty".

  Addressed comments & rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12729

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12729?vs=39537&id=42171#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java

To: JIRA, navis


> Add option for removing intermediate directory for partition, which is empty
> 
>
> Key: HIVE-5220
> URL: https://issues.apache.org/jira/browse/HIVE-5220
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D12729.2.patch, HIVE-5220.D12729.1.patch
>
>
> For deeply nested partitioned table, intermediate directories are not removed 
> even if there is no partitions in it by removing them.
> {noformat}
> /deep_part/c=09/d=01
> /deep_part/c=09/d=01/e=01
> /deep_part/c=09/d=01/e=02
> /deep_part/c=09/d=02
> /deep_part/c=09/d=02/e=01
> /deep_part/c=09/d=02/e=02
> {noformat}
> After removing partition (c='09'), directory remains like this, 
> {noformat}
> /deep_part/c=09/d=01
> /deep_part/c=09/d=02
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5454) HCatalog runs a partition listing with an empty filter

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13801981#comment-13801981
 ] 

Phabricator commented on HIVE-5454:
---

brock has commented on the revision "HIVE-5454 [jira] HCatalog runs a partition 
listing with an empty filter".

  +1

REVISION DETAIL
  https://reviews.facebook.net/D13317

To: JIRA, QwertyManiac
Cc: brock


> HCatalog runs a partition listing with an empty filter
> --
>
> Key: HIVE-5454
> URL: https://issues.apache.org/jira/browse/HIVE-5454
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0
>Reporter: Harsh J
> Attachments: D13317.1.patch, D13317.2.patch, D13317.3.patch
>
>
> This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of 
> calling HCatInputFormat causes it to do 2x partition lookups - once without 
> the filter, and then again with the filter.
> For tables with large number partitions (10, say), the non-filter lookup 
> proves fatal both to the client ("Read timed out" errors from 
> ThriftMetaStoreClient cause the server doesn't respond) and to the server 
> (too much data loaded into the cache, OOME, or slowdown).
> The fix would be to use a single call that also passes a partition filter 
> information, as was in the case of HCatalog 0.4 sources before HCATALOG-527.
> (HCatalog-release-wise, this affects all 0.5.x users)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3972) Support using multiple reducer for fetching order by results

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3972:
--

Attachment: D8349.6.patch

navis updated the revision "HIVE-3972 [jira] Support using multiple reducer for 
fetching order by results".

  Fix test fail

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8349

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8349?vs=42027&id=42123#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MergeSortingFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/RowFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
  ql/src/test/queries/clientpositive/orderby_query_bucketing.q
  ql/src/test/results/clientpositive/orderby_query_bucketing.q.out

To: JIRA, navis
Cc: njain


> Support using multiple reducer for fetching order by results
> 
>
> Key: HIVE-3972
> URL: https://issues.apache.org/jira/browse/HIVE-3972
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D8349.5.patch, D8349.6.patch, HIVE-3972.D8349.1.patch, 
> HIVE-3972.D8349.2.patch, HIVE-3972.D8349.3.patch, HIVE-3972.D8349.4.patch
>
>
> Queries for fetching results which have lastly "order by" clause make final 
> MR run with single reducer, which can be too much. For example, 
> {code}
> select value, sum(key) as sum from src group by value order by sum;
> {code}
> If number of reducer is reasonable, multiple result files could be merged 
> into single sorted stream in the fetcher level.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5276) Skip useless string encoding stage for hiveserver2

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5276:
--

Attachment: D12879.3.patch

navis updated the revision "HIVE-5276 [jira] Skip useless string encoding stage 
for hiveserver2".

  Fixed possible IndexOutOfBoundsException in FetchTask

Reviewers: cwsteinbach, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12879

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12879?vs=40023&id=42033#toc

BRANCH
  HIVE-5276

ARCANIST PROJECT
  hive

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DefaultFetchFormatter.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchFormatter.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java

To: JIRA, cwsteinbach, navis
Cc: cwsteinbach


> Skip useless string encoding stage for hiveserver2
> --
>
> Key: HIVE-5276
> URL: https://issues.apache.org/jira/browse/HIVE-5276
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D12879.2.patch, D12879.3.patch, HIVE-5276.3.patch.txt, 
> HIVE-5276.D12879.1.patch
>
>
> Current hiveserver2 acquires rows in string format which is used for cli 
> output. Then convert them into row again and convert to final format lastly. 
> This is inefficient and memory consuming. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3972) Support using multiple reducer for fetching order by results

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-3972:
--

Attachment: D8349.5.patch

navis updated the revision "HIVE-3972 [jira] Support using multiple reducer for 
fetching order by results".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D8349

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D8349?vs=28557&id=42027#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  conf/hive-default.xml.template
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MergeSortingFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/RowFetcher.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
  ql/src/test/queries/clientpositive/orderby_query_bucketing.q
  ql/src/test/results/clientpositive/orderby_query_bucketing.q.out

To: JIRA, navis
Cc: njain


> Support using multiple reducer for fetching order by results
> 
>
> Key: HIVE-3972
> URL: https://issues.apache.org/jira/browse/HIVE-3972
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D8349.5.patch, HIVE-3972.D8349.1.patch, 
> HIVE-3972.D8349.2.patch, HIVE-3972.D8349.3.patch, HIVE-3972.D8349.4.patch
>
>
> Queries for fetching results which have lastly "order by" clause make final 
> MR run with single reducer, which can be too much. For example, 
> {code}
> select value, sum(key) as sum from src group by value order by sum;
> {code}
> If number of reducer is reasonable, multiple result files could be merged 
> into single sorted stream in the fetcher level.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5560) Hive produces incorrect results on multi-distinct query

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5560:
--

Attachment: D13599.2.patch

navis updated the revision "HIVE-5560 [jira] Hive produces incorrect results on 
multi-distinct query".

  Fixed test results

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13599

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13599?vs=41997&id=42021#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/test/queries/clientpositive/groupby2_map_multi_distinct.q
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out
  ql/src/test/results/clientpositive/count.q.out
  ql/src/test/results/clientpositive/groupby2_map_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby2_noskew_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby3_map_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby3_noskew_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
  ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out
  ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_sort_11.q.out
  ql/src/test/results/clientpositive/join18_multi_distinct.q.out
  ql/src/test/results/clientpositive/limit_pushdown.q.out

To: JIRA, navis


> Hive produces incorrect results on multi-distinct query
> ---
>
> Key: HIVE-5560
> URL: https://issues.apache.org/jira/browse/HIVE-5560
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Attachments: D13599.1.patch, D13599.2.patch
>
>
> {noformat}
> select key, count(distinct key) + count(distinct value) from src tablesample 
> (10 ROWS) group by key
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@src
>  A masked pattern was here 
> 165 1
> val_165 1
> 238 1
> val_238 1
> 255 1
> val_255 1
> 27  1
> val_27  1
> 278 1
> val_278 1
> 311 1
> val_311 1
> 409 1
> val_409 1
> 484 1
> val_484 1
> 86  1
> val_86  1
> 98  1
> val_98  1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4144) Add "select database()" command to show the current database

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4144:
--

Attachment: D9597.5.patch

navis updated the revision "HIVE-4144 [jira] Add "select database()" command to 
show the current database".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D9597

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D9597?vs=38637&id=42015#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
  ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/io/OneNullRowInputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UDFCurrentDB.java
  ql/src/test/queries/clientpositive/select_dummy_source.q
  ql/src/test/queries/clientpositive/udf_current_database.q
  ql/src/test/results/clientpositive/select_dummy_source.q.out
  ql/src/test/results/clientpositive/show_functions.q.out
  ql/src/test/results/clientpositive/udf_current_database.q.out

To: JIRA, navis


> Add "select database()" command to show the current database
> 
>
> Key: HIVE-4144
> URL: https://issues.apache.org/jira/browse/HIVE-4144
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Reporter: Mark Grover
>Assignee: Navis
> Attachments: D9597.5.patch, HIVE-4144.D9597.1.patch, 
> HIVE-4144.D9597.2.patch, HIVE-4144.D9597.3.patch, HIVE-4144.D9597.4.patch
>
>
> A recent hive-user mailing list conversation asked about having a command to 
> show the current database.
> http://mail-archives.apache.org/mod_mbox/hive-user/201303.mbox/%3CCAMGr+0i+CRY69m3id=DxthmUCWLf0NxpKMCtROb=uauh2va...@mail.gmail.com%3E
> MySQL seems to have a command to do so:
> {code}
> select database();
> {code}
> http://dev.mysql.com/doc/refman/5.0/en/information-functions.html#function_database
> We should look into having something similar in Hive.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4880) Rearrange explain order of stages simpler

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4880:
--

Attachment: D11733.3.patch

navis updated the revision "HIVE-4880 [jira] Rearrange explain order of stages 
simpler".

  Fix test fails (temporary fix before HIVE-3925, which will show dependencies 
for fetch task properly)

Reviewers: hagleitn, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11733

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11733?vs=41883&id=42009#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/ObjectPair.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
  ql/src/test/queries/clientpositive/explain_rearrange.q
  ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out
  ql/src/test/results/clientpositive/explain_rearrange.q.out

To: JIRA, hagleitn, navis
Cc: brock


> Rearrange explain order of stages simpler
> -
>
> Key: HIVE-4880
> URL: https://issues.apache.org/jira/browse/HIVE-4880
> Project: Hive
>  Issue Type: Improvement
>  Components: Diagnosability
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D11733.2.patch, D11733.3.patch, HIVE-4880.D11733.1.patch
>
>
> Current explain order of stages is not best form to read (and to debug)
> {noformat}
> Before:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
>   Stage-4
>   Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
>   Stage-9 depends on stages: Stage-0
>   Stage-2 depends on stages: Stage-9
>   Stage-3
>   Stage-5
>   Stage-6 depends on stages: Stage-5
> After:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-2 depends on stages: Stage-1 , consists of Stage-3, Stage-4, Stage-5
>   Stage-3
>   Stage-4
>   Stage-5
>   Stage-6 depends on stages: Stage-5
>   Stage-7 depends on stages: Stage-3, Stage-4, Stage-6
>   Stage-8 depends on stages: Stage-7
>   Stage-9 depends on stages: Stage-8
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5560) Hive produces incorrect results on multi-distinct query

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5560:
--

Attachment: D13599.1.patch

navis requested code review of "HIVE-5560 [jira] Hive produces incorrect 
results on multi-distinct query".

Reviewers: JIRA

HIVE-5560 Hive produces incorrect results on multi-distinct query

select key, count(distinct key) + count(distinct value) from src tablesample 
(10 ROWS) group by key
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
 A masked pattern was here 
165 1
val_165 1
238 1
val_238 1
255 1
val_255 1
27  1
val_27  1
278 1
val_278 1
311 1
val_311 1
409 1
val_409 1
484 1
val_484 1
86  1
val_86  1
98  1
val_98  1

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13599

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/test/queries/clientpositive/groupby2_map_multi_distinct.q
  ql/src/test/results/clientpositive/auto_join18_multi_distinct.q.out
  ql/src/test/results/clientpositive/count.q.out
  ql/src/test/results/clientpositive/groupby2_map_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby2_noskew_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out
  ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out
  ql/src/test/results/clientpositive/groupby_sort_11.q.out
  ql/src/test/results/clientpositive/join18_multi_distinct.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40725/

To: JIRA, navis


> Hive produces incorrect results on multi-distinct query
> ---
>
> Key: HIVE-5560
> URL: https://issues.apache.org/jira/browse/HIVE-5560
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Attachments: D13599.1.patch
>
>
> {noformat}
> select key, count(distinct key) + count(distinct value) from src tablesample 
> (10 ROWS) group by key
> POSTHOOK: type: QUERY
> POSTHOOK: Input: default@src
>  A masked pattern was here 
> 165 1
> val_165 1
> 238 1
> val_238 1
> 255 1
> val_255 1
> 27  1
> val_27  1
> 278 1
> val_278 1
> 311 1
> val_311 1
> 409 1
> val_409 1
> 484 1
> val_484 1
> 86  1
> val_86  1
> 98  1
> val_98  1
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-784) Support uncorrelated subqueries in the WHERE clause

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798696#comment-13798696
 ] 

Phabricator commented on HIVE-784:
--

hbutani has commented on the revision "HIVE-784 [jira] Support uncorrelated 
subqueries in the WHERE clause".

  Tried to address all the issues you raised, except for the Nested SubQuery 
check. In addition:
  - added a bunch of -ve tests: subquery with or, check for exists/not exists 
with implicit Group By, subquery with multiple columns in select

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g:391 Changed 
the grammar so we support both these forms:

  select ...
  from ...
  where x not in (select ...)

  or

  select...
  from ...
  where not x in (select ...)
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:470 this is 
checking the where clause in the SubQuery.
  It can be null. When it is null, it implies this is an uncorrelated SubQuery.
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:496 yes this is 
allowed. Works in mysql. Can look into this in a later jira
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:537 
sqNewSarchCond can be null. In case there is only 1 conjunct, no copy is made.
  ASTNode equals doesn't do a deep check.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1804 No 
this feature is missing. Can this be a separate jira
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6526 i 
looked at this again:
  - the initial switch is different, because this method is based on 
subQuery.joinType
  - the way the left and right aliases are setup is different.
  - the code for initializing expressions,filters and filterFrPushing can be 
refactored out. Doesn't seem worth it though.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java:101 this gets 
caught in the remove method below, because then the SubQuery is not a top level 
conjunct.
  Have added a -ve test for this.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java:226-233 this 
method differs from SemanticAnalyzer::doPhase1GetAllAggregations in:
  - doesn't throw SemanticException
  - doesn't pass the ASTNode to the UnparseTranslator.
  - returns an int instead of the Agg/Windowing ASTNodes

REVISION DETAIL
  https://reviews.facebook.net/D13443

To: JIRA, ashutoshc, hbutani


> Support uncorrelated subqueries in the WHERE clause
> ---
>
> Key: HIVE-784
> URL: https://issues.apache.org/jira/browse/HIVE-784
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Harish Butani
> Attachments: D13443.1.patch, D13443.2.patch, HIVE-784.1.patch.txt, 
> HIVE-784.2.patch, SubQuerySpec.pdf, tpchQueriesUsingSubQueryClauses.sql
>
>
> Hive currently only support views in the FROM-clause, some Facebook use cases 
> suggest that Hive should support subqueries such as those connected by 
> IN/EXISTS in the WHERE-clause. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4880) Rearrange explain order of stages simpler

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4880:
--

Attachment: D11733.2.patch

navis updated the revision "HIVE-4880 [jira] Rearrange explain order of stages 
simpler".

  Optionized explain ouput

Reviewers: hagleitn, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D11733

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D11733?vs=35871&id=41883#toc

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/common/ObjectPair.java
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
  ql/src/test/queries/clientpositive/explain_rearrange.q
  ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out
  ql/src/test/results/clientpositive/explain_rearrange.q.out

To: JIRA, hagleitn, navis
Cc: brock


> Rearrange explain order of stages simpler
> -
>
> Key: HIVE-4880
> URL: https://issues.apache.org/jira/browse/HIVE-4880
> Project: Hive
>  Issue Type: Improvement
>  Components: Diagnosability
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D11733.2.patch, HIVE-4880.D11733.1.patch
>
>
> Current explain order of stages is not best form to read (and to debug)
> {noformat}
> Before:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
>   Stage-4
>   Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
>   Stage-9 depends on stages: Stage-0
>   Stage-2 depends on stages: Stage-9
>   Stage-3
>   Stage-5
>   Stage-6 depends on stages: Stage-5
> After:
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-2 depends on stages: Stage-1 , consists of Stage-3, Stage-4, Stage-5
>   Stage-3
>   Stage-4
>   Stage-5
>   Stage-6 depends on stages: Stage-5
>   Stage-7 depends on stages: Stage-3, Stage-4, Stage-6
>   Stage-8 depends on stages: Stage-7
>   Stage-9 depends on stages: Stage-8
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-784) Support uncorrelated subqueries in the WHERE clause

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-784:
-

Attachment: D13443.2.patch

hbutani updated the revision "HIVE-784 [jira] Support uncorrelated subqueries 
in the WHERE clause".

Reviewers: ashutoshc, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13443

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13443?vs=41457&id=41871#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
  ql/src/test/queries/clientnegative/subquery_exists_implicit_gby.q
  ql/src/test/queries/clientnegative/subquery_in_groupby.q
  ql/src/test/queries/clientnegative/subquery_in_select.q
  ql/src/test/queries/clientnegative/subquery_multiple_cols_in_select.q
  ql/src/test/queries/clientnegative/subquery_notexists_implicit_gby.q
  ql/src/test/queries/clientnegative/subquery_subquery_chain.q
  ql/src/test/queries/clientnegative/subquery_windowing_corr.q
  ql/src/test/queries/clientnegative/subquery_with_or_cond.q
  ql/src/test/queries/clientpositive/subquery_exists.q
  ql/src/test/queries/clientpositive/subquery_in.q
  ql/src/test/queries/clientpositive/subquery_multiinsert.q
  ql/src/test/queries/clientpositive/subquery_notexists.q
  ql/src/test/queries/clientpositive/subquery_notin.q
  ql/src/test/results/clientnegative/subquery_exists_implicit_gby.q.out
  ql/src/test/results/clientnegative/subquery_in_groupby.q.out
  ql/src/test/results/clientnegative/subquery_in_select.q.out
  ql/src/test/results/clientnegative/subquery_multiple_cols_in_select.q.out
  ql/src/test/results/clientnegative/subquery_notexists_implicit_gby.q.out
  ql/src/test/results/clientnegative/subquery_subquery_chain.q.out
  ql/src/test/results/clientnegative/subquery_windowing_corr.q.out
  ql/src/test/results/clientnegative/subquery_with_or_cond.q.out
  ql/src/test/results/clientpositive/subquery_exists.q.out
  ql/src/test/results/clientpositive/subquery_in.q.out
  ql/src/test/results/clientpositive/subquery_multiinsert.q.out
  ql/src/test/results/clientpositive/subquery_notexists.q.out
  ql/src/test/results/clientpositive/subquery_notin.q.out

To: JIRA, ashutoshc, hbutani


> Support uncorrelated subqueries in the WHERE clause
> ---
>
> Key: HIVE-784
> URL: https://issues.apache.org/jira/browse/HIVE-784
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Harish Butani
> Attachments: D13443.1.patch, D13443.2.patch, HIVE-784.1.patch.txt, 
> HIVE-784.2.patch, SubQuerySpec.pdf, tpchQueriesUsingSubQueryClauses.sql
>
>
> Hive currently only support views in the FROM-clause, some Facebook use cases 
> suggest that Hive should support subqueries such as those connected by 
> IN/EXISTS in the WHERE-clause. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5580) push down predicates with an and-operator between non-SARGable predicates will get NPE

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798563#comment-13798563
 ] 

Phabricator commented on HIVE-5580:
---

hagleitn has accepted the revision "HIVE-5580 [jira] push down predicates with 
an and-operator between non-SARGable predicates will get NPE".

  LGTM +1

REVISION DETAIL
  https://reviews.facebook.net/D13533

BRANCH
  h-5580

ARCANIST PROJECT
  hive

To: JIRA, hagleitn, omalley


> push down predicates with an and-operator between non-SARGable predicates 
> will get NPE
> --
>
> Key: HIVE-5580
> URL: https://issues.apache.org/jira/browse/HIVE-5580
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13533.1.patch
>
>
> When all of the predicates in an AND-operator in a SARG expression get 
> removed by the SARG builder, evaluation can end up with a NPE. 
> Sub-expressions are typically removed from AND-operators because they aren't 
> SARGable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5568) count(*) on ORC tables with predicate pushdown on partition columns fail

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798564#comment-13798564
 ] 

Phabricator commented on HIVE-5568:
---

hagleitn has accepted the revision "HIVE-5568 [jira] count(*) on ORC tables 
with predicate pushdown on partition columns fail".

  LGTM +1

REVISION DETAIL
  https://reviews.facebook.net/D13485

BRANCH
  h-5568

ARCANIST PROJECT
  hive

To: JIRA, hagleitn, omalley


> count(*) on ORC tables with predicate pushdown on partition columns fail
> 
>
> Key: HIVE-5568
> URL: https://issues.apache.org/jira/browse/HIVE-5568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.12.1
>
> Attachments: D13485.1.patch, D13485.2.patch, D13485.3.patch
>
>
> If the query is:
> {code}
> select count(*) from orc_table where x = 10;
> {code}
> where x is a partition column and predicate pushdown is enabled, you'll get 
> an array out of bounds exception.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5568) count(*) on ORC tables with predicate pushdown on partition columns fail

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5568:
--

Attachment: D13485.3.patch

omalley updated the revision "HIVE-5568 [jira] count(*) on ORC tables with 
predicate pushdown on partition columns fail".

  Previous rebase missed the code change. oops.

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13485

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13485?vs=41799&id=41805#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

To: JIRA, omalley


> count(*) on ORC tables with predicate pushdown on partition columns fail
> 
>
> Key: HIVE-5568
> URL: https://issues.apache.org/jira/browse/HIVE-5568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.12.1
>
> Attachments: D13485.1.patch, D13485.2.patch, D13485.3.patch
>
>
> If the query is:
> {code}
> select count(*) from orc_table where x = 10;
> {code}
> where x is a partition column and predicate pushdown is enabled, you'll get 
> an array out of bounds exception.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5568) count(*) on ORC tables with predicate pushdown on partition columns fail

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5568:
--

Attachment: D13485.2.patch

omalley updated the revision "HIVE-5568 [jira] count(*) on ORC tables with 
predicate pushdown on partition columns fail".

  rebased to trunk (after HIVE-5567)

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13485

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13485?vs=41661&id=41799#toc

AFFECTED FILES
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

To: JIRA, omalley


> count(*) on ORC tables with predicate pushdown on partition columns fail
> 
>
> Key: HIVE-5568
> URL: https://issues.apache.org/jira/browse/HIVE-5568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.12.1
>
> Attachments: D13485.1.patch, D13485.2.patch
>
>
> If the query is:
> {code}
> select count(*) from orc_table where x = 10;
> {code}
> where x is a partition column and predicate pushdown is enabled, you'll get 
> an array out of bounds exception.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5580) push down predicates with an and-operator between non-SARGable predicates will get NPE

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5580:
--

Attachment: D13533.1.patch

omalley requested code review of "HIVE-5580 [jira] push down predicates with an 
and-operator between non-SARGable predicates will get NPE".

Reviewers: JIRA

HIVE-5580

When all of the predicates in an AND-operator in a SARG expression get removed 
by the SARG builder, evaluation can end up with a NPE. Sub-expressions are 
typically removed from AND-operators because they aren't SARGable.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13533

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
  ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40461/

To: JIRA, omalley


> push down predicates with an and-operator between non-SARGable predicates 
> will get NPE
> --
>
> Key: HIVE-5580
> URL: https://issues.apache.org/jira/browse/HIVE-5580
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13533.1.patch
>
>
> When all of the predicates in an AND-operator in a SARG expression get 
> removed by the SARG builder, evaluation can end up with a NPE. 
> Sub-expressions are typically removed from AND-operators because they aren't 
> SARGable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5567) Add better protection code for SARGs

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798462#comment-13798462
 ] 

Phabricator commented on HIVE-5567:
---

hagleitn has commented on the revision "HIVE-5567 [jira] Add better protection 
code for SARGs".

  LGTM

REVISION DETAIL
  https://reviews.facebook.net/D13479

BRANCH
  h-5567

ARCANIST PROJECT
  hive

To: JIRA, hagleitn, omalley


> Add better protection code for SARGs
> 
>
> Key: HIVE-5567
> URL: https://issues.apache.org/jira/browse/HIVE-5567
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13479.1.patch, D13479.2.patch, D13479.3.patch
>
>
> Currently, the SARG parser gets a NPE when the push down predicate uses a 
> type like decimal that isn't supported.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5567) Add better protection code for SARGs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5567:
--

Attachment: D13479.3.patch

omalley updated the revision "HIVE-5567 [jira] Add better protection code for 
SARGs".

  Fixed the ordering to fix TestSearchArgumentImpl

Reviewers: hagleitn, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13479

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13479?vs=41685&id=41781#toc

BRANCH
  h-5567

ARCANIST PROJECT
  hive

AFFECTED FILES
  data/files/orc_create_people.txt
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

To: JIRA, hagleitn, omalley


> Add better protection code for SARGs
> 
>
> Key: HIVE-5567
> URL: https://issues.apache.org/jira/browse/HIVE-5567
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13479.1.patch, D13479.2.patch, D13479.3.patch
>
>
> Currently, the SARG parser gets a NPE when the push down predicate uses a 
> type like decimal that isn't supported.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5575) ZooKeeper connection closed when unlock with retry

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5575:
--

Attachment: D13515.1.patch

chenchun requested code review of "HIVE-5575 [jira] ZooKeeper connection closed 
when unlock with retry".

Reviewers: JIRA

lock

See the attachment, I have encountered a scenario that hive retries to  unlock 
all locks, but zookeeper session is closed. If there are hundreds of locks, say 
dynamic partition, the process will hang up for several days.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13515

AFFECTED FILES
  
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40365/

To: JIRA, chenchun


> ZooKeeper connection closed when unlock with retry
> --
>
> Key: HIVE-5575
> URL: https://issues.apache.org/jira/browse/HIVE-5575
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Chun Chen
>Assignee: Chun Chen
> Fix For: 0.13.0
>
> Attachments: D13515.1.patch
>
>
> See the attachment, I have encountered a scenario that hive retries to  
> unlock all locks, but zookeeper session is closed. If there are hundreds of 
> locks, say dynamic partition, the process will hang up for several days.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5567) Add better protection code for SARGs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5567:
--

Attachment: D13479.2.patch

omalley updated the revision "HIVE-5567 [jira] Add better protection code for 
SARGs".

  Fixed the test case to keep the int column

Reviewers: hagleitn, JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13479

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13479?vs=41637&id=41685#toc

BRANCH
  h-5567

ARCANIST PROJECT
  hive

AFFECTED FILES
  data/files/orc_create_people.txt
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

To: JIRA, hagleitn, omalley


> Add better protection code for SARGs
> 
>
> Key: HIVE-5567
> URL: https://issues.apache.org/jira/browse/HIVE-5567
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13479.1.patch, D13479.2.patch
>
>
> Currently, the SARG parser gets a NPE when the push down predicate uses a 
> type like decimal that isn't supported.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5567) Add better protection code for SARGs

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797422#comment-13797422
 ] 

Phabricator commented on HIVE-5567:
---

hagleitn has accepted the revision "HIVE-5567 [jira] Add better protection code 
for SARGs".

  Looks good except for minor test issue.

INLINE COMMENTS
  ql/src/test/queries/clientpositive/orc_create.q:69 did you mean to change 
this to decimal? i'm guessing you want an int test too.

REVISION DETAIL
  https://reviews.facebook.net/D13479

BRANCH
  h-5567

ARCANIST PROJECT
  hive

To: JIRA, hagleitn, omalley


> Add better protection code for SARGs
> 
>
> Key: HIVE-5567
> URL: https://issues.apache.org/jira/browse/HIVE-5567
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13479.1.patch
>
>
> Currently, the SARG parser gets a NPE when the push down predicate uses a 
> type like decimal that isn't supported.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5568) count(*) on ORC tables with predicate pushdown on partition columns fail

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5568:
--

Attachment: D13485.1.patch

omalley requested code review of "HIVE-5568 [jira] count(*) on ORC tables with 
predicate pushdown on partition columns fail".

Reviewers: JIRA

HIVE-5568

If the query is:

select count(*) from orc_table where x = 10;

where x is a partition column and predicate pushdown is enabled, you'll get an 
array out of bounds exception.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13485

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40317/

To: JIRA, omalley


> count(*) on ORC tables with predicate pushdown on partition columns fail
> 
>
> Key: HIVE-5568
> URL: https://issues.apache.org/jira/browse/HIVE-5568
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.12.1
>
> Attachments: D13485.1.patch
>
>
> If the query is:
> {code}
> select count(*) from orc_table where x = 10;
> {code}
> where x is a partition column and predicate pushdown is enabled, you'll get 
> an array out of bounds exception.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5567) Add better protection code for SARGs

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5567:
--

Attachment: D13479.1.patch

omalley requested code review of "HIVE-5567 [jira] Add better protection code 
for SARGs".

Reviewers: JIRA

HIVE-5567. Add better protection code for SARGs.

Currently, the SARG parser gets a NPE when the push down predicate uses a type 
like decimal that isn't supported.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13479

AFFECTED FILES
  data/files/orc_create_people.txt
  ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40305/

To: JIRA, omalley


> Add better protection code for SARGs
> 
>
> Key: HIVE-5567
> URL: https://issues.apache.org/jira/browse/HIVE-5567
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13479.1.patch
>
>
> Currently, the SARG parser gets a NPE when the push down predicate uses a 
> type like decimal that isn't supported.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-2093:
--

Attachment: D12807.3.patch

navis updated the revision "HIVE-2093 [jira] create/drop database should 
populate inputs/outputs and check concurrency and user permission".

  Rebased to trunk

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12807

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12807?vs=39813&id=41577#toc

AFFECTED FILES
  hbase-handler/src/test/results/negative/cascade_dbdrop_hadoop20.q.out
  
hcatalog/core/src/main/java/org/apache/hcatalog/security/HdfsAuthorizationProvider.java
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java
  ql/src/test/queries/clientnegative/authorization_fail_create_db.q
  ql/src/test/queries/clientnegative/authorization_fail_drop_db.q
  ql/src/test/queries/clientnegative/lockneg_query_tbl_in_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_db_lock_conflict.q
  ql/src/test/queries/clientnegative/lockneg_try_drop_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_lock_db_in_use.q
  ql/src/test/results/clientnegative/authorization_fail_2.q.out
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_6.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_fail_create_db.q.out
  ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty_restrict.q.out
  ql/src/test/results/clientnegative/exim_22_export_authfail.q.out
  ql/src/test/results/clientnegative/exim_23_import_exist_authfail.q.out
  ql/src/test/results/clientnegative/exim_24_import_part_authfail.q.out
  ql/src/test/results/clientnegative/exim_25_import_nonexist_authfail.q.out
  ql/src/test/results/clientnegative/join_nonexistent_part.q.out
  ql/src/test/results/clientnegative/load_exist_part_authfail.q.out
  ql/src/test/results/clientnegative/load_nonpart_authfail.q.out
  ql/src/test/results/clientnegative/load_part_authfail.q.out
  ql/src/test/results/clientnegative/lockneg_query_tbl_in_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_db_lock_conflict.q.out
  ql/src/test/results/clientnegative/lockneg_try_drop_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_lock_db_in_use.q.out
  ql/src/test/results/clientpositive/alter1.q.out
  ql/src/test/results/clientpositive/alter2.q.out
  ql/src/test/results/clientpositive/alter4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/database.q.out
  ql/src/test/results/clientpositive/database_drop.q.out
  ql/src/test/results/clientpositive/describe_database_json.q.out
  ql/src/test/results/clientpositive/drop_database_removes_partition_dirs.q.out
  ql/src/test/results/clientpositive/exim_00_nonpart_empty.q.out
  ql/src/test/results/clientpositive/exim_01_nonpart.q.out
  ql/src/test/results/clientpositive/exim_02_00_part_empty.q.out
  ql/src/test/results/clientpositive/exim_02_part.q.out
  ql/src/test/results/clientpositive/exim_03_nonpart_over_compat.q.out
  ql/src/test/results/clientpositive/exim_04_all_part.q.out
  ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out
  ql/src/test/results/clientpositive/exim_05_some_part.q.out
  ql/src/test/results/clientpositive/exim_06_one_part.q.out
  ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out
  ql/src/test/results/clientpositive/exim_08

[jira] [Commented] (HIVE-784) Support uncorrelated subqueries in the WHERE clause

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794530#comment-13794530
 ] 

Phabricator commented on HIVE-784:
--

ashutoshc has requested changes to the revision "HIVE-784 [jira] Support 
uncorrelated subqueries in the WHERE clause".

  Design looks good. Mostly implementation related comments.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g:391 It would 
be nicer if instead of two rules for IN / NOT IN if we could just have one 
rule, which can conditionally generate TOK_SUBQUERY_OP_NOTIN / 
TOK_SUBQUERY_OP_IN token. Not a big deal, but would be nice to have since that 
makes grammar bit more succinct.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1804 You 
mentioned in comment above that we don't support nested / recursive subq, but I 
don't see a check for that. Perhaps, its there but I missed it.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1789 Thanks 
for detailed comments. Very helpful!
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1811 There 
should exactly one subq currently. If so, will be good to add a note for it.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java:101 Since, OR 
is not supported, It will be good to generate an error message here if OR is 
encountered.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java:226-233 Same 
logic exists in SemanticAnalyzer::doPhase1GetAllAggregations, perhaps we can 
create a util method in ParseUtils, instead of repeating code here.
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:289 If you mark 
this as transient, you probably wont need to write Kryo serializer for this.
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:699 I dont think 
its required. We should probably mark all usage of instances of ASTNodeOrigin 
as transient.
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:470 This should 
never be the case. Shall we throw an exception here, instead of silently 
returning?
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:496 Is this 
allowed by standard that subq predicate may refer to Outer? If yes, than in 
future perhaps we can add this predicate as a conjunct for outer query.
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:537 Is this need 
to be .equals() check here instead of == ?
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:539 It will be 
good to add a comment, why we need to have True condition here, instead? 
Probably, because plan gen fails later while generating rest of filter plan.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1821 Good 
to name this method as validateAndRewriteAST() ?
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6526 Quite 
a bit of this code is repeated from genJoinTree(), seems like atleast some bits 
could be refactored out of genJoinTree() which this method can make use of.
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:1827 name 
sqOperator is misleading here, topOp perhaps ?
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:41 This is not an 
operator in classic Hive sense. Perhaps, SubqASTcontainer or something else.
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java:24 SubQueryType 
instead of SubQueryOperatorType ?

REVISION DETAIL
  https://reviews.facebook.net/D13443

BRANCH
  SubQuery

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, hbutani


> Support uncorrelated subqueries in the WHERE clause
> ---
>
> Key: HIVE-784
> URL: https://issues.apache.org/jira/browse/HIVE-784
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Harish Butani
> Attachments: D13443.1.patch, HIVE-784.1.patch.txt, HIVE-784.2.patch, 
> SubQuerySpec.pdf, tpchQueriesUsingSubQueryClauses.sql
>
>
> Hive currently only support views in the FROM-clause, some Facebook use cases 
> suggest that Hive should support subqueries such as those connected by 
> IN/EXISTS in the WHERE-clause. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-784) Support uncorrelated subqueries in the WHERE clause

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-784:
-

Attachment: D13443.1.patch

hbutani requested code review of "HIVE-784 [jira] Support uncorrelated 
subqueries in the WHERE clause".

Reviewers: JIRA, ashutoshc

SubQuery: add tests for distinct

Hive currently only support views in the FROM-clause, some Facebook use cases 
suggest that Hive should support subqueries such as those connected by 
IN/EXISTS in the WHERE-clause.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13443

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java
  ql/src/test/queries/clientnegative/subquery_in_groupby.q
  ql/src/test/queries/clientnegative/subquery_in_select.q
  ql/src/test/queries/clientnegative/subquery_windowing_corr.q
  ql/src/test/queries/clientpositive/subquery_exists.q
  ql/src/test/queries/clientpositive/subquery_in.q
  ql/src/test/queries/clientpositive/subquery_multiinsert.q
  ql/src/test/queries/clientpositive/subquery_notexists.q
  ql/src/test/queries/clientpositive/subquery_notin.q
  ql/src/test/results/clientnegative/subquery_in_groupby.q.out
  ql/src/test/results/clientnegative/subquery_in_select.q.out
  ql/src/test/results/clientnegative/subquery_windowing_corr.q.out
  ql/src/test/results/clientpositive/subquery_exists.q.out
  ql/src/test/results/clientpositive/subquery_in.q.out
  ql/src/test/results/clientpositive/subquery_multiinsert.q.out
  ql/src/test/results/clientpositive/subquery_notexists.q.out
  ql/src/test/results/clientpositive/subquery_notin.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/40029/

To: JIRA, ashutoshc, hbutani


> Support uncorrelated subqueries in the WHERE clause
> ---
>
> Key: HIVE-784
> URL: https://issues.apache.org/jira/browse/HIVE-784
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Harish Butani
> Attachments: D13443.1.patch, HIVE-784.1.patch.txt, HIVE-784.2.patch, 
> SubQuerySpec.pdf, tpchQueriesUsingSubQueryClauses.sql
>
>
> Hive currently only support views in the FROM-clause, some Facebook use cases 
> suggest that Hive should support subqueries such as those connected by 
> IN/EXISTS in the WHERE-clause. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5454) HCatalog runs a partition listing with an empty filter

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5454:
--

Attachment: D13317.3.patch

QwertyManiac updated the revision "HIVE-5454 [jira] HCatalog runs a partition 
listing with an empty filter".

  Fixed 4 reported checkstyle violations. Lint reported fine from arc, so 
didn't notice these earlier.

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13317

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13317?vs=41043&id=41049#toc

AFFECTED FILES
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/transfer/impl/HCatInputFormatReader.java
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java
  hcatalog/src/docs/src/documentation/content/xdocs/inputoutput.xml
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hcatalog/utils/HBaseReadWrite.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/GroupByAge.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadJson.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadRC.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadText.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadWrite.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SimpleRead.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreComplex.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreDemo.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreNumbers.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SumNumbers.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/TypeDataCheck.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteJson.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteRC.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteText.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteTextPartitioned.java
  
hcatalog/storage-handlers/hbase/src/test/org/apache/hive/hcatalog/hbase/TestHBaseInputFormat.java

To: JIRA, QwertyManiac


> HCatalog runs a partition listing with an empty filter
> --
>
> Key: HIVE-5454
> URL: https://issues.apache.org/jira/browse/HIVE-5454
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0
>Reporter: Harsh J
> Attachments: D13317.1.patch, D13317.2.patch, D13317.3.patch
>
>
> This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of 
> calling HCatInputFormat causes it to do 2x partition lookups - once without 
> the filter, and then again with the filter.
> For tables with large number partitions (10, say), the non-filter lookup 
> proves fatal both to the client ("Read timed out" errors from 
> ThriftMetaStoreClient cause the server doesn't respond) and to the server 
> (too much data loaded into the cache, OOME, or slowdown).
> The fix would be to use a single call that also passes a partition filter 
> information, as was in the case of HCatalog 0.4 sources before HCATALOG-527.
> (HCatalog-release-wise, this affects all 0.5.x users)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5454) HCatalog runs a partition listing with an empty filter

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5454:
--

Attachment: D13317.2.patch

QwertyManiac updated the revision "HIVE-5454 [jira] HCatalog runs a partition 
listing with an empty filter".

  Removed usage of removed deprecated methods.

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13317

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13317?vs=41025&id=41043#toc

AFFECTED FILES
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/transfer/impl/HCatInputFormatReader.java
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java
  hcatalog/src/docs/src/documentation/content/xdocs/inputoutput.xml
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hcatalog/utils/HBaseReadWrite.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/GroupByAge.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadJson.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadRC.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadText.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadWrite.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SimpleRead.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreComplex.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreDemo.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreNumbers.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SumNumbers.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/TypeDataCheck.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteJson.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteRC.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteText.java
  
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteTextPartitioned.java
  
hcatalog/storage-handlers/hbase/src/test/org/apache/hive/hcatalog/hbase/TestHBaseInputFormat.java

To: JIRA, QwertyManiac


> HCatalog runs a partition listing with an empty filter
> --
>
> Key: HIVE-5454
> URL: https://issues.apache.org/jira/browse/HIVE-5454
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0
>Reporter: Harsh J
> Attachments: D13317.1.patch, D13317.2.patch
>
>
> This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of 
> calling HCatInputFormat causes it to do 2x partition lookups - once without 
> the filter, and then again with the filter.
> For tables with large number partitions (10, say), the non-filter lookup 
> proves fatal both to the client ("Read timed out" errors from 
> ThriftMetaStoreClient cause the server doesn't respond) and to the server 
> (too much data loaded into the cache, OOME, or slowdown).
> The fix would be to use a single call that also passes a partition filter 
> information, as was in the case of HCatalog 0.4 sources before HCATALOG-527.
> (HCatalog-release-wise, this affects all 0.5.x users)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5454) HCatalog runs a partition listing with an empty filter

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5454:
--

Attachment: D13317.1.patch

QwertyManiac requested code review of "HIVE-5454 [jira] HCatalog runs a 
partition listing with an empty filter".

Reviewers: JIRA

HIVE-5454. HCatalog runs a partition listing with an empty filter.

Modify the input format of HCat to load partitions always with a filter if 
available.

This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of 
calling HCatInputFormat causes it to do 2x partition lookups - once without the 
filter, and then again with the filter.

For tables with large number partitions (10, say), the non-filter lookup 
proves fatal both to the client ("Read timed out" errors from 
ThriftMetaStoreClient cause the server doesn't respond) and to the server (too 
much data loaded into the cache, OOME, or slowdown).

The fix would be to use a single call that also passes a partition filter 
information, as was in the case of HCatalog 0.4 sources before HCATALOG-527.

(HCatalog-release-wise, this affects all 0.5.x users)

TEST PLAN
  Built hcat module and ran all its existing tests

REVISION DETAIL
  https://reviews.facebook.net/D13317

AFFECTED FILES
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/transfer/impl/HCatInputFormatReader.java
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java
  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/39321/

To: JIRA, QwertyManiac


> HCatalog runs a partition listing with an empty filter
> --
>
> Key: HIVE-5454
> URL: https://issues.apache.org/jira/browse/HIVE-5454
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0
>Reporter: Harsh J
> Attachments: D13317.1.patch
>
>
> This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of 
> calling HCatInputFormat causes it to do 2x partition lookups - once without 
> the filter, and then again with the filter.
> For tables with large number partitions (10, say), the non-filter lookup 
> proves fatal both to the client ("Read timed out" errors from 
> ThriftMetaStoreClient cause the server doesn't respond) and to the server 
> (too much data loaded into the cache, OOME, or slowdown).
> The fix would be to use a single call that also passes a partition filter 
> information, as was in the case of HCatalog 0.4 sources before HCATALOG-527.
> (HCatalog-release-wise, this affects all 0.5.x users)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.6.patch

sershe updated the revision "HIVE-4888 [jira] listPartitionsByFilter doesn't 
support lt/gt/lte/gte".

  wrong number in test

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13101

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13101?vs=40959&id=40995#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java
  ql/src/test/queries/clientpositive/filter_numeric.q
  ql/src/test/results/clientpositive/filter_numeric.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, D13101.2.patch, D13101.3.patch, 
> D13101.4.patch, D13101.5.patch, D13101.6.patch, HIVE-4888.00.patch, 
> HIVE-4888.01.patch, HIVE-4888.04.patch, HIVE-4888.05.patch, 
> HIVE-4888.06.patch, HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4542) TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4542:
--

Attachment: D13269.2.patch

vaibhavgumashta updated the revision "HIVE-4542 [jira] 
TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database".

  Rebased on current trunk, some minor changes to make TestJdbcDriver2 fit 
Junit 4.x

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13269

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13269?vs=40953&id=40965#toc

AFFECTED FILES
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java

To: JIRA, vaibhavgumashta
Cc: thejas, ashutoshc


> TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database
> ---
>
> Key: HIVE-4542
> URL: https://issues.apache.org/jira/browse/HIVE-4542
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Thejas M Nair
>Assignee: Vaibhav Gumashta
> Attachments: D13269.1.patch, D13269.2.patch, HIVE-4542.1.patch, 
> HIVE-4542.2.patch
>
>
> The check for database name in TestJdbcDriver2.testMetaDataGetSchemas fails 
> with the error -
> {code}
> junit.framework.ComparisonFailure: expected:<...efault> but was:<...bname>
> {code}
> ie, a database called dbname is found, which it does not expect. This failure 
> will happen depending on the order in which the function get the databases, 
> if "default" database is the first one, it succeeds.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.5.patch

sershe updated the revision "HIVE-4888 [jira] listPartitionsByFilter doesn't 
support lt/gt/lte/gte".

  add more tests

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13101

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13101?vs=40713&id=40959#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
  ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java
  ql/src/test/queries/clientpositive/filter_numeric.q
  ql/src/test/results/clientpositive/filter_numeric.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, D13101.2.patch, D13101.3.patch, 
> D13101.4.patch, D13101.5.patch, HIVE-4888.00.patch, HIVE-4888.01.patch, 
> HIVE-4888.04.patch, HIVE-4888.05.patch, HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4542) TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4542:
--

Attachment: D13269.1.patch

vaibhavgumashta requested code review of "HIVE-4542 [jira] 
TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database".

Reviewers: JIRA

HIVE-4542: The check for database name in 
TestJdbcDriver2.testMetaDataGetSchemas fails with the error -
junit.framework.ComparisonFailure: expected:<...efault> but was:<...bname>
ie, a database called dbname is found, which it does not expect. This failure 
will happen depending on the order in which the function get the databases, if 
"default" database is the first one, it succeeds.

TEST PLAN
  Empty

REVISION DETAIL
  https://reviews.facebook.net/D13269

AFFECTED FILES
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/39051/

To: JIRA, vaibhavgumashta
Cc: thejas, ashutoshc


> TestJdbcDriver2.testMetaDataGetSchemas fails because of unexpected database
> ---
>
> Key: HIVE-4542
> URL: https://issues.apache.org/jira/browse/HIVE-4542
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: D13269.1.patch, HIVE-4542.1.patch, HIVE-4542.2.patch
>
>
> The check for database name in TestJdbcDriver2.testMetaDataGetSchemas fails 
> with the error -
> {code}
> junit.framework.ComparisonFailure: expected:<...efault> but was:<...bname>
> {code}
> ie, a database called dbname is found, which it does not expect. This failure 
> will happen depending on the order in which the function get the databases, 
> if "default" database is the first one, it succeeds.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5414:
--

Attachment: D13209.2.patch

navis updated the revision "HIVE-5414 [jira] The result of show grant is not 
visible via JDBC".

  Fixed outputs

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13209

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13209?vs=40779&id=40941#toc

AFFECTED FILES
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/unset_table_property.q.out
  ql/src/test/results/clientnegative/unset_view_property.q.out
  ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out
  ql/src/test/results/clientpositive/authorization_1.q.out
  ql/src/test/results/clientpositive/authorization_2.q.out
  ql/src/test/results/clientpositive/authorization_3.q.out
  ql/src/test/results/clientpositive/authorization_4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/authorization_6.q.out
  ql/src/test/results/clientpositive/keyword_1.q.out
  ql/src/test/results/clientpositive/show_tblproperties.q.out
  ql/src/test/results/clientpositive/unset_table_view_property.q.out
  service/src/java/org/apache/hive/service/CompositeService.java

To: JIRA, navis


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch, D13209.2.patch
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5428) Direct SQL check fails during tests

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5428:
--

Attachment: D13245.2.patch

sershe updated the revision "HIVE-5428 [jira] Direct SQL check fails during 
tests".

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13245

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13245?vs=40863&id=40899#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java

To: JIRA, sershe


> Direct SQL check fails during tests
> ---
>
> Key: HIVE-5428
> URL: https://issues.apache.org/jira/browse/HIVE-5428
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Sergey Shelukhin
> Attachments: D13245.1.patch, D13245.2.patch, HIVE-5428.01.patch
>
>
> Noticed this while work on mavenization. If you run the following command
> {noformat}
> ant test -Dtestcase=TestCliDriver -Dqfile=udf_case.q -Dtest.silent=false
> {noformat}
> and look at the top of the logs you see the exception below. It looks like 
> something needs to be changed in the initialization order.
> {noformat}
> 2013-10-02 13:42:21,596 INFO  metastore.ObjectStore 
> (ObjectStore.java:initialize(243)) - ObjectStore, initialize called
> 2013-10-02 13:42:22,048 DEBUG bonecp.BoneCPDataSource 
> (BoneCPDataSource.java:maybeInit(148)) - JDBC URL = 
> jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true, 
> Username = APP, partitions = 1, max (per partition) = 0, min (per partition) 
> = 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
> 2013-10-02 13:42:22,051 WARN  bonecp.BoneCPConfig 
> (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
> 2013-10-02 13:42:30,218 INFO  metastore.ObjectStore 
> (ObjectStore.java:getPMF(312)) - Setting MetaStore object pin classes with 
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 2013-10-02 13:42:30,253 DEBUG bonecp.BoneCPDataSource 
> (BoneCPDataSource.java:maybeInit(148)) - JDBC URL = 
> jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true, 
> Username = APP, partitions = 1, max (per partition) = 0, min (per partition) 
> = 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
> 2013-10-02 13:42:30,253 WARN  bonecp.BoneCPConfig 
> (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
> 2013-10-02 13:42:30,262 INFO  metastore.MetaStoreDirectSql 
> (MetaStoreDirectSql.java:(99)) - MySQL check failed, assuming we are 
> not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), 
> after : "".
> 2013-10-02 13:42:30,298 ERROR metastore.MetaStoreDirectSql 
> (MetaStoreDirectSql.java:(112)) - Self-test query [select "DB_ID" from 
> "DBS"] failed; direct SQL is disabled
> javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" 
> from "DBS"".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:108)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:249)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:220)
>   at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
>   at 
> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:418)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:405)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:444)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:329)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:289)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4084)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:126)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru

[jira] [Updated] (HIVE-5428) Direct SQL check fails during tests

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5428:
--

Attachment: D13245.1.patch

sershe requested code review of "HIVE-5428 [jira] Direct SQL check fails during 
tests".

Reviewers: JIRA

patch

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13245

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38973/

To: JIRA, sershe


> Direct SQL check fails during tests
> ---
>
> Key: HIVE-5428
> URL: https://issues.apache.org/jira/browse/HIVE-5428
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
> Attachments: D13245.1.patch
>
>
> Noticed this while work on mavenization. If you run the following command
> {noformat}
> ant test -Dtestcase=TestCliDriver -Dqfile=udf_case.q -Dtest.silent=false
> {noformat}
> and look at the top of the logs you see the exception below. It looks like 
> something needs to be changed in the initialization order.
> {noformat}
> 2013-10-02 13:42:21,596 INFO  metastore.ObjectStore 
> (ObjectStore.java:initialize(243)) - ObjectStore, initialize called
> 2013-10-02 13:42:22,048 DEBUG bonecp.BoneCPDataSource 
> (BoneCPDataSource.java:maybeInit(148)) - JDBC URL = 
> jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true, 
> Username = APP, partitions = 1, max (per partition) = 0, min (per partition) 
> = 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
> 2013-10-02 13:42:22,051 WARN  bonecp.BoneCPConfig 
> (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
> 2013-10-02 13:42:30,218 INFO  metastore.ObjectStore 
> (ObjectStore.java:getPMF(312)) - Setting MetaStore object pin classes with 
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 2013-10-02 13:42:30,253 DEBUG bonecp.BoneCPDataSource 
> (BoneCPDataSource.java:maybeInit(148)) - JDBC URL = 
> jdbc:derby:;databaseName=../build/test/junit_metastore_db;create=true, 
> Username = APP, partitions = 1, max (per partition) = 0, min (per partition) 
> = 0, helper threads = 3, idle max age = 60 min, idle test period = 240 min
> 2013-10-02 13:42:30,253 WARN  bonecp.BoneCPConfig 
> (BoneCPConfig.java:sanitize(1537)) - Max Connections < 1. Setting to 20
> 2013-10-02 13:42:30,262 INFO  metastore.MetaStoreDirectSql 
> (MetaStoreDirectSql.java:(99)) - MySQL check failed, assuming we are 
> not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), 
> after : "".
> 2013-10-02 13:42:30,298 ERROR metastore.MetaStoreDirectSql 
> (MetaStoreDirectSql.java:(112)) - Self-test query [select "DB_ID" from 
> "DBS"] failed; direct SQL is disabled
> javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" 
> from "DBS"".
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
>   at 
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:108)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:249)
>   at 
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:220)
>   at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.hive.metastore.RetryingRawStore.(RetryingRawStore.java:62)
>   at 
> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:71)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:418)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:405)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:444)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:329)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:289)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4084)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:126)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.

[jira] [Commented] (HIVE-5364) NPE on some queries from partitioned orc table

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784523#comment-13784523
 ] 

Phabricator commented on HIVE-5364:
---

hagleitn has commented on the revision "HIVE-5364 [jira] NPE on some queries 
from partitioned orc table".

  LGTM +1

REVISION DETAIL
  https://reviews.facebook.net/D13215

To: JIRA, omalley
Cc: hagleitn


> NPE on some queries from partitioned orc table
> --
>
> Key: HIVE-5364
> URL: https://issues.apache.org/jira/browse/HIVE-5364
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Blocker
> Attachments: D13215.1.patch
>
>
> If you create a partitioned ORC table with:
> {code}
> create table A
> ...
> PARTITIONED BY (
> year int,
> month int,
> day int)
> {code}
> This query will fail:
> select count from A where where year=2013 and month=9 and day=15;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5425) Provide a configuration option to control the default stripe size for ORC

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784470#comment-13784470
 ] 

Phabricator commented on HIVE-5425:
---

hagleitn has accepted the revision "HIVE-5425 [jira] Provide a configuration 
option to control the default stripe size for ORC".

  LGTM +1

REVISION DETAIL
  https://reviews.facebook.net/D13233

BRANCH
  h-5425

ARCANIST PROJECT
  hive

To: JIRA, hagleitn, omalley


> Provide a configuration option to control the default stripe size for ORC
> -
>
> Key: HIVE-5425
> URL: https://issues.apache.org/jira/browse/HIVE-5425
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13233.1.patch
>
>
> We should provide a configuration option to control the default stripe size.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5425) Provide a configuration option to control the default stripe size for ORC

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5425:
--

Attachment: D13233.1.patch

omalley requested code review of "HIVE-5425 [jira] Provide a configuration 
option to control the default stripe size for ORC".

Reviewers: JIRA

HIVE-5425

We should provide a configuration option to control the default stripe size.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13233

AFFECTED FILES
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38901/

To: JIRA, omalley


> Provide a configuration option to control the default stripe size for ORC
> -
>
> Key: HIVE-5425
> URL: https://issues.apache.org/jira/browse/HIVE-5425
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: D13233.1.patch
>
>
> We should provide a configuration option to control the default stripe size.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5364) NPE on some queries from partitioned orc table

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5364:
--

Attachment: D13215.1.patch

omalley requested code review of "HIVE-5364 [jira] NPE on some queries from 
partitioned orc table".

Reviewers: JIRA

HIVE-5364 NPE in OrcInputFormat

If you create a partitioned ORC table with:

create table A (
  id int
) PARTITIONED BY (
year int,
month int,
day int)

This query will fail when predicate pushdown is on:
SELECT id FROM A WHERE id > 90;

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13215

AFFECTED FILES
  data/files/orc_create_people.txt
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
  ql/src/test/queries/clientpositive/orc_create.q
  ql/src/test/results/clientpositive/orc_create.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38853/

To: JIRA, omalley


> NPE on some queries from partitioned orc table
> --
>
> Key: HIVE-5364
> URL: https://issues.apache.org/jira/browse/HIVE-5364
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
>Priority: Blocker
> Attachments: D13215.1.patch
>
>
> If you create a partitioned ORC table with:
> {code}
> create table A
> ...
> PARTITIONED BY (
> year int,
> month int,
> day int)
> {code}
> This query will fail:
> select count from A where where year=2013 and month=9 and day=15;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5414) The result of show grant is not visible via JDBC

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5414:
--

Attachment: D13209.1.patch

navis requested code review of "HIVE-5414 [jira] The result of show grant is 
not visible via JDBC".

Reviewers: JIRA

HIVE-5414 The result of show grant is not visible via JDBC

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13209

AFFECTED FILES
  jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/RoleDDLDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowGrantDesc.java
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out
  ql/src/test/results/clientpositive/authorization_1.q.out
  ql/src/test/results/clientpositive/authorization_2.q.out
  ql/src/test/results/clientpositive/authorization_3.q.out
  ql/src/test/results/clientpositive/authorization_4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/authorization_6.q.out
  ql/src/test/results/clientpositive/keyword_1.q.out
  service/src/java/org/apache/hive/service/CompositeService.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38823/

To: JIRA, navis


> The result of show grant is not visible via JDBC
> 
>
> Key: HIVE-5414
> URL: https://issues.apache.org/jira/browse/HIVE-5414
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: D13209.1.patch
>
>
> Currently, show grant / show role grant does not make fetch task, which 
> provides the result schema for jdbc clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5407) show create table creating unusable DDL when some reserved keywords exist

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5407:
--

Attachment: D13191.1.patch

code6 requested code review of "HIVE-5407 [jira] show create table creating 
unusable DDL when some reserved keywords  exist".

Reviewers: JIRA

HIVE-5407

HIVE-701 already makes most reserved keywords available for 
table/column/partition names and 'show create table' produces usable DDLs.
However I think it's better if we quote table/column/partition names for the 
output of 'show create table', which is how mysql works and seems more robust.

For example, use select as column name will produce unusable DDL:

create table table_select(`select` string);
show create table table_select;

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13191

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/test/results/clientpositive/show_create_table_alter.q.out
  ql/src/test/results/clientpositive/show_create_table_db_table.q.out
  ql/src/test/results/clientpositive/show_create_table_delimited.q.out
  ql/src/test/results/clientpositive/show_create_table_partitioned.q.out
  ql/src/test/results/clientpositive/show_create_table_serde.q.out
  ql/src/test/results/clientpositive/show_create_table_view.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38757/

To: JIRA, code6


> show create table creating unusable DDL when some reserved keywords  exist
> --
>
> Key: HIVE-5407
> URL: https://issues.apache.org/jira/browse/HIVE-5407
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
> Environment: hive 0.11
>Reporter: Zhichun Wu
>Priority: Minor
> Attachments: D13191.1.patch
>
>
> HIVE-701 already makes most reserved keywords available for 
> table/column/partition names and 'show create table' produces usable DDLs.
> However I think it's better if we quote table/column/partition names for the 
> output of 'show create table', which is how mysql works and seems more robust.
> For example, use select as column name will produce unusable DDL:
> {code}
> create table table_select(`select` string);
> show create table table_select;
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.4.patch

sershe updated the revision "HIVE-4888 [jira] listPartitionsByFilter doesn't 
support lt/gt/lte/gte".

  small issue

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13101

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13101?vs=40707&id=40713#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  ql/src/test/queries/clientpositive/filter_numeric.q
  ql/src/test/results/clientpositive/filter_numeric.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, D13101.2.patch, D13101.3.patch, 
> D13101.4.patch, HIVE-4888.00.patch, HIVE-4888.01.patch, 
> HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.3.patch

sershe updated the revision "HIVE-4888 [jira] listPartitionsByFilter doesn't 
support lt/gt/lte/gte".

  Work around Derby issue; add compat with JDO equality compares

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13101

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13101?vs=40503&id=40707#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  ql/src/test/queries/clientpositive/filter_numeric.q
  ql/src/test/results/clientpositive/filter_numeric.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, D13101.2.patch, D13101.3.patch, 
> HIVE-4888.00.patch, HIVE-4888.01.patch, HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5370:
--

Attachment: D13185.2.patch

Amareshwari updated the revision "HIVE-5370 [jira] format_number udf should 
take user specifed format as argument".

  Remove negtive test for string argument

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13185

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13185?vs=40683&id=40689#toc

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java
  ql/src/test/queries/clientnegative/udf_format_number_wrong6.q
  ql/src/test/queries/clientpositive/udf_format_number.q
  ql/src/test/results/clientnegative/udf_format_number_wrong6.q.out
  ql/src/test/results/clientpositive/udf_format_number.q.out

To: JIRA, Amareshwari


> format_number udf should take user specifed format as argument
> --
>
> Key: HIVE-5370
> URL: https://issues.apache.org/jira/browse/HIVE-5370
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Attachments: D13185.1.patch, D13185.2.patch
>
>
> Currently, format_number udf formats the number to #,###,###.##, but it 
> should also take a user specified format as optional input.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5370) format_number udf should take user specifed format as argument

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5370:
--

Attachment: D13185.1.patch

Amareshwari requested code review of "HIVE-5370 [jira] format_number udf should 
take user specifed format as argument".

Reviewers: JIRA

Support format as second argument for format_number udf

Currently, format_number udf formats the number to #,###,###.##, but it should 
also take a user specified format as optional input.

TEST PLAN
  Unit test updated

REVISION DETAIL
  https://reviews.facebook.net/D13185

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java
  ql/src/test/queries/clientpositive/udf_format_number.q
  ql/src/test/results/clientpositive/udf_format_number.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38649/

To: JIRA, Amareshwari


> format_number udf should take user specifed format as argument
> --
>
> Key: HIVE-5370
> URL: https://issues.apache.org/jira/browse/HIVE-5370
> Project: Hive
>  Issue Type: Improvement
>  Components: UDF
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>Priority: Minor
> Attachments: D13185.1.patch
>
>
> Currently, format_number udf formats the number to #,###,###.##, but it 
> should also take a user specified format as optional input.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5381) OOM when using many lead/lag funtions against large partition

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5381:
--

Attachment: D13161.1.patch

navis requested code review of "HIVE-5381 [jira] OOM when using many lead/lag 
funtions against large partition".

Reviewers: JIRA

HIVE-5381 OOM when using many lead/lag funtions against large partition

Currently, columns containing window function are evaluated for all rows in a 
partition, which possibly make OOM if the partition is big (400MB in my case). 
For lead/lag functions, the value can be evaluated per row.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13161

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/exec/LLRowResolver.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFPartition.java
  ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38499/

To: JIRA, navis


> OOM when using many lead/lag funtions against large partition
> -
>
> Key: HIVE-5381
> URL: https://issues.apache.org/jira/browse/HIVE-5381
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D13161.1.patch
>
>
> Currently, columns containing window function are evaluated for all rows in a 
> partition, which possibly make OOM if the partition is big (400MB in my 
> case). For lead/lag functions, the value can be evaluated per row.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-5379) NoClassDefFoundError is thrown when using lead/lag with kryo serialization

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779576#comment-13779576
 ] 

Phabricator commented on HIVE-5379:
---

ashutoshc has accepted the revision "HIVE-5379 [jira] NoClassDefFoundError is 
thrown when using lead/lag with kryo serialization".

  +1

REVISION DETAIL
  https://reviews.facebook.net/D13155

BRANCH
  HIVE-5379

ARCANIST PROJECT
  hive

To: JIRA, ashutoshc, navis


> NoClassDefFoundError is thrown when using lead/lag with kryo serialization
> --
>
> Key: HIVE-5379
> URL: https://issues.apache.org/jira/browse/HIVE-5379
> Project: Hive
>  Issue Type: Bug
>  Components: PTF-Windowing
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D13155.1.patch
>
>
> {noformat}
> java.lang.RuntimeException: Error in configuring object
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:432)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
>   at org.apache.hadoop.mapred.Child.main(Child.java:260)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>   ... 9 more
> Caused by: java.lang.NoClassDefFoundError: 
> org/antlr/runtime/tree/TreeWizard$ContextVisitor
>   at java.lang.ClassLoader.defineClass1(Native Method)
>   at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>   at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>   at java.lang.Class.getDeclaringClass(Native Method)
>   at java.lang.Class.getEnclosingClass(Class.java:1085)
>   at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1054)
>   at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1110)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>   at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
>   at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
>   at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
>   at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
>   at 
> com.esot

[jira] [Updated] (HIVE-5379) NoClassDefFoundError is thrown when using lead/lag with kryo serialization

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5379:
--

Attachment: D13155.1.patch

navis requested code review of "HIVE-5379 [jira] NoClassDefFoundError is thrown 
when using lead/lag with kryo serialization".

Reviewers: JIRA

HIVE-5379 NoClassDefFoundError is thrown when using lead/lag with kryo 
serialization

java.lang.RuntimeException: Error in configuring object
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at 
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at 
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:432)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.NoClassDefFoundError: 
org/antlr/runtime/tree/TreeWizard$ContextVisitor
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.getDeclaringClass(Native Method)
at java.lang.Class.getEnclosingClass(Class.java:1085)
at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1054)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1110)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at 
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at 
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at 
com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at 
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at 
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at 
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:812)
at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:723)
at 
org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:736)
at 
org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:304)
at 
org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:262)
at 
org.apache.hadoop

[jira] [Commented] (HIVE-3011) Test Issue for arc/JIRA integration, ignore

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779432#comment-13779432
 ] 

Phabricator commented on HIVE-3011:
---

epriestley has abandoned the revision "HIVE-3011 [jira] Test Issue for arc/JIRA 
integration, ignore".

REVISION DETAIL
  https://reviews.facebook.net/D3111

To: JIRA, epriestley


> Test Issue for arc/JIRA integration, ignore
> ---
>
> Key: HIVE-3011
> URL: https://issues.apache.org/jira/browse/HIVE-3011
> Project: Hive
>  Issue Type: Bug
>Reporter: Evan Priestley
>Priority: Trivial
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3093.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3093.2.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3111.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3117.1.patch
>
>
> I'm helping @ashutosh debug some arc/Phabricator issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3011) Test Issue for arc/JIRA integration, ignore

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779433#comment-13779433
 ] 

Phabricator commented on HIVE-3011:
---

epriestley has abandoned the revision "HIVE-3011 [jira] Test Issue for arc/JIRA 
integration, ignore".

REVISION DETAIL
  https://reviews.facebook.net/D3117

To: JIRA, epriestley


> Test Issue for arc/JIRA integration, ignore
> ---
>
> Key: HIVE-3011
> URL: https://issues.apache.org/jira/browse/HIVE-3011
> Project: Hive
>  Issue Type: Bug
>Reporter: Evan Priestley
>Priority: Trivial
> Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3093.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3093.2.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3111.1.patch, 
> ASF.LICENSE.NOT.GRANTED--HIVE-3011.D3117.1.patch
>
>
> I'm helping @ashutosh debug some arc/Phabricator issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5358) ReduceSinkDeDuplication should ignore column orders when check overlapping part of keys between parent and child

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5358:
--

Attachment: D13113.1.patch

chenchun requested code review of "HIVE-5358 [jira] ReduceSinkDeDuplication 
should ignore column orders when check overlapping part of keys between parent 
and child".

Reviewers: JIRA

HIVE-5358

select key, value from (select key, value from src group by key, value) t group 
by key, value;

This can be optimized by ReduceSinkDeDuplication

select key, value from (select key, value from src group by key, value) t group 
by value, key;

However the sql above can't be optimized by ReduceSinkDeDuplication currently 
due to different column orders of parent and child operator.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13113

AFFECTED FILES
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeColumnListDesc.java
  ql/src/test/queries/clientpositive/reduce_deduplicate_extended.q
  ql/src/test/results/clientpositive/reduce_deduplicate_extended.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38295/

To: JIRA, chenchun


> ReduceSinkDeDuplication should ignore column orders when check overlapping 
> part of keys between parent and child
> 
>
> Key: HIVE-5358
> URL: https://issues.apache.org/jira/browse/HIVE-5358
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Chun Chen
>Assignee: Chun Chen
> Attachments: D13113.1.patch, HIVE-5358.patch
>
>
> {code}
> select key, value from (select key, value from src group by key, value) t 
> group by key, value;
> {code}
> This can be optimized by ReduceSinkDeDuplication
> {code}
> select key, value from (select key, value from src group by key, value) t 
> group by value, key;
> {code}
> However the sql above can't be optimized by ReduceSinkDeDuplication currently 
> due to different column orders of parent and child operator.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.2.patch

sershe updated the revision "HIVE-4888 [jira] listPartitionsByFilter doesn't 
support lt/gt/lte/gte".

  address all things

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D13101

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D13101?vs=40485&id=40503#toc

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  ql/src/test/queries/clientpositive/filter_numeric.q
  ql/src/test/results/clientpositive/filter_numeric.q.out
  serde/if/serde.thrift
  serde/src/gen/thrift/gen-cpp/serde_constants.cpp
  serde/src/gen/thrift/gen-cpp/serde_constants.h
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
  serde/src/gen/thrift/gen-php/org/apache/hadoop/hive/serde/Types.php
  serde/src/gen/thrift/gen-py/org_apache_hadoop_hive_serde/constants.py
  serde/src/gen/thrift/gen-rb/serde_constants.rb

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, D13101.2.patch, HIVE-4888.00.patch, 
> HIVE-4888.01.patch, HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-5196) ThriftCLIService.java uses stderr to print the stack trace, it should use the logger instead.

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-5196:
--

Attachment: D13107.1.patch

vaibhavgumashta requested code review of "HIVE-5196 [jira] 
ThriftCLIService.java uses stderr to print the stack trace, it should use the 
logger instead.".

Reviewers: JIRA

HIVE-5196: ThriftCLIService should use LOG.error instead of stderr to log error 
description

ThriftCLIService.java uses stderr to print the stack trace, it should use the 
logger instead. Using e.printStackTrace is not suitable for production.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13107

AFFECTED FILES
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38265/

To: JIRA, vaibhavgumashta


> ThriftCLIService.java uses stderr to print the stack trace, it should use the 
> logger instead.
> -
>
> Key: HIVE-5196
> URL: https://issues.apache.org/jira/browse/HIVE-5196
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.12.0
>
> Attachments: D13107.1.patch
>
>
> ThriftCLIService.java uses stderr to print the stack trace, it should use the 
> logger instead. Using e.printStackTrace is not suitable for production.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4888) listPartitionsByFilter doesn't support lt/gt/lte/gte

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4888:
--

Attachment: D13101.1.patch

sershe requested code review of "HIVE-4888 [jira] listPartitionsByFilter 
doesn't support lt/gt/lte/gte".

Reviewers: JIRA

Add better pushdown (numeric compares) to Direct SQL path. DataNucleus is 
missing cast functions, so to add to JDO, a plugin would be necessary.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D13101

AFFECTED FILES
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
  metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
  
serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/38247/

To: JIRA, sershe


> listPartitionsByFilter doesn't support lt/gt/lte/gte
> 
>
> Key: HIVE-4888
> URL: https://issues.apache.org/jira/browse/HIVE-4888
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: D13101.1.patch, HIVE-4888.on-top-of-4914.patch
>
>
> Filter pushdown could be improved. Based on my experiments there's no 
> reasonable way to do it with DN 2.0, due to DN bug in substring and 
> Collection.get(int) not being implemented.
> With version as low as 2.1 we can use values.get on partition to extract 
> values to compare to. Type compatibility is an issue, but is easy for strings 
> and integral values.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira