[jira] [Updated] (HIVE-7482) The execution side changes for SMB join in hive-tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7482:
-
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks [~vikram.dixit]!

> The execution side changes for SMB join in hive-tez
> ---
>
> Key: HIVE-7482
> URL: https://issues.apache.org/jira/browse/HIVE-7482
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-7482.1.patch, HIVE-7482.10.patch, 
> HIVE-7482.11.patch, HIVE-7482.12.patch, HIVE-7482.13.patch, 
> HIVE-7482.14.patch, HIVE-7482.2.patch, HIVE-7482.3.patch, HIVE-7482.4.patch, 
> HIVE-7482.5.patch, HIVE-7482.6.patch, HIVE-7482.7.patch, HIVE-7482.8.patch, 
> HIVE-7482.9.patch, HIVE-7482.WIP.2.patch, HIVE-7482.WIP.3.patch, 
> HIVE-7482.WIP.4.patch, HIVE-7482.WIP.patch
>
>
> A piece of HIVE-7430.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-7481) The planning side changes for SMB join on hive-tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-7481.
--
   Resolution: Fixed
Fix Version/s: 0.14.0

> The planning side changes for SMB join on hive-tez
> --
>
> Key: HIVE-7481
> URL: https://issues.apache.org/jira/browse/HIVE-7481
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-7481.1.patch, HIVE-7481.WIP.patch
>
>
> A piece of HIVE-7430.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7096) Support grouped splits in Tez partitioned broadcast join

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7096:
-
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

> Support grouped splits in Tez partitioned broadcast join
> 
>
> Key: HIVE-7096
> URL: https://issues.apache.org/jira/browse/HIVE-7096
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Gunther Hagleitner
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-7096.1.patch, HIVE-7096.2.patch, HIVE-7096.3.patch, 
> HIVE-7096.4.patch, HIVE-7096.5.patch, HIVE-7096.tez.branch.patch
>
>
> Same checks for schema + deser + file format done in HiveSplitGenerator need 
> to be done in the CustomPartitionVertex.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-6515) Custom vertex in hive-tez should be able to accept multiple MR-inputs

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6515.
--
   Resolution: Fixed
Fix Version/s: 0.14.0

> Custom vertex in hive-tez should be able to accept multiple MR-inputs
> -
>
> Key: HIVE-6515
> URL: https://issues.apache.org/jira/browse/HIVE-6515
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
>
> Custom vertex in hive-tez should be able to accept multiple MR-inputs. 
> Currently this is restricted to only 1 input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-7430) Implement SMB join in tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-7430.
--
   Resolution: Fixed
Fix Version/s: 0.14.0

> Implement SMB join in tez
> -
>
> Key: HIVE-7430
> URL: https://issues.apache.org/jira/browse/HIVE-7430
> Project: Hive
>  Issue Type: New Feature
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
>
> We need to enable SMB joins in hive-tez.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8237) CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145998#comment-14145998
 ] 

Hive QA commented on HIVE-8237:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670816/HIVE-8237.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6325 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/953/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/953/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-953/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670816

> CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)
> -
>
> Key: HIVE-8237
> URL: https://issues.apache.org/jira/browse/HIVE-8237
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8237.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8227) NPE w/ hive on tez when doing unions on empty tables

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8227:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk.

> NPE w/ hive on tez when doing unions on empty tables
> 
>
> Key: HIVE-8227
> URL: https://issues.apache.org/jira/browse/HIVE-8227
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-8227.1.patch, HIVE-8227.2.patch
>
>
> We're looking at aliasToWork.values() to determine input paths etc. This can 
> contain nulls when we're scanning empty tables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7689) Enable Postgres as METASTORE back-end

[Lukas Eder (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146027#comment-14146027
 ] 

Lukas Eder commented on HIVE-7689:
--

This is Lukas from Data Geekery (the company behind jOOQ) chiming in.

{quote}Long term this may be the right answer. I have not looked into the 
options. JOOQ looks like the open source edition only supports open source 
databases, and since Hive already supports Oracle and MSSQL Server this would 
not be an option.{quote}

This is correct. We're aware of the challenges our license model induces to 
other Open Source projects, specifically those of the ASF. We've been talking 
also to other projects about this:

*Apache GORA*:

- https://issues.apache.org/jira/browse/GORA-86
- 
https://mail-archives.apache.org/mod_mbox/gora-user/201311.mbox/%3ccalugr6aiac9tteskmjhiylf7htdyeuts_z_j0hse8z0th2f...@mail.gmail.com%3E

*Apache CloudStack*:

- Unfortunately, there hadn't been too many public discussions about this

>From the GORA mailing list thread, I suspect that only #2 would apply to Hive:

>> 2. In addition to the above, Data Geekery and Apache Gora could make a
>> separate agreement, which grants Apache Gora a perpetual license to use the
>> jOOQ Enterprise Edition for integration testing Apache Gora against SQL
>> Server and other commercial databases, but not to distribute, sublicense or
>> make available the jOOQ Enterprise Edition to end users. End users may
>> again use Apache Gora with the jOOQ Open Source Edition along with Open
>> Source databases, or they may purchase a commercial license from Data
>> Geekery. In this case, Data Geekery would require a prominent backlink
>> indicating that Apache Gora is based on jOOQ. This is how IntelliJ or
>> YourKit Profiler handle free commercial licenses for non-commercial OSS
>> projects as well.

This option was well received by the GORA team, although the GORA project 
itself isn't advancing very fast, so we have not settled this yet.

If you think that this could be a viable option also for Apache Hive, let me 
know (or write to sa...@datageekery.com). We're very open to finding creative 
solutions with other OSS vendors.

And obviously - to get back on topic - the free, ASL 2.0-licensed jOOQ support 
for PostgreSQL makes PostgreSQL even more awesome! :)

> Enable Postgres as METASTORE back-end
> -
>
> Key: HIVE-7689
> URL: https://issues.apache.org/jira/browse/HIVE-7689
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Damien Carol
>Assignee: Damien Carol
>Priority: Minor
>  Labels: metastore, postgres
> Fix For: 0.14.0
>
> Attachments: HIVE-7689.5.patch, HIVE-7689.6.patch, HIVE-7689.7.patch, 
> HIVE-7689.8.patch, HIVE-7889.1.patch, HIVE-7889.2.patch, HIVE-7889.3.patch, 
> HIVE-7889.4.patch
>
>
> I maintain few patches to make Metastore works with Postgres back end in our 
> production environment.
> The main goal of this JIRA is to push upstream these patches.
> This patch enable LOCKS, COMPACTION and fix error in STATS on postgres 
> metastore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8199) CBO Trunk Merge: quote2 test fails due to incorrect literal translation

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146067#comment-14146067
 ] 

Hive QA commented on HIVE-8199:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670824/HIVE-8199.01.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6342 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quote2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/954/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/954/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-954/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670824

> CBO Trunk Merge: quote2 test fails due to incorrect literal translation
> ---
>
> Key: HIVE-8199
> URL: https://issues.apache.org/jira/browse/HIVE-8199
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8199.01.patch, HIVE-8199.patch
>
>
> Quoting of quotes and slashes is lost in translation back from CBO to AST, it 
> seems



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7156) Group-By operator stat-annotation only uses distinct approx to generate rollups

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146069#comment-14146069
 ] 

Hive QA commented on HIVE-7156:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670861/HIVE-7156.7.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/955/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/955/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-955/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-955/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/results/clientpositive/quote2.q.out'
Reverted 'ql/src/test/queries/clientpositive/quote2.q'
Reverted 
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/optiq/translator/ASTBuilder.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/webhcat/svr/target 
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target 
accumulo-handler/target hwi/target common/target common/src/gen contrib/target 
service/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update
Aql/src/test/queries/clientpositive/tez_union_group_by.q
Aql/src/test/results/clientpositive/tez/tez_union_group_by.q.out
Uql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java
Uitests/src/test/resources/testconfiguration.properties

Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1627254.

Updated to revision 1627254.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670861

> Group-By operator stat-annotation only uses distinct approx to generate 
> rollups
> ---
>
> Key: HIVE-7156
> URL: https://issues.apache.org/jira/browse/HIVE-7156
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Prasanth J
> Attachments: HIVE-7156.1.patch, HIVE-7156.2.patch, HIVE-7156.3.patch, 
> HIVE-7156.4.patch, HIVE-7156

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/
---

(Updated Sept. 24, 2014, 9:17 a.m.)


Review request for hive.


Changes
---

Update patch based on the comments from Thejas, Brock, and Swarnim. This patch 
mainly add a public API hasMoreLogs() and modify the internal status tracking 
of HiveStatement.


Repository: hive-git


Description
---

When executing query in Beeline, user should have a option to see the progress 
through the outputs. Beeline could use the API introduced in HIVE-4629 to get 
and display the logs to the client.


Diffs (updated)
-

  beeline/pom.xml 45fa02b 
  beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 
1e66542 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
daf8e9e 
  jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
PRE-CREATION 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 

Diff: https://reviews.apache.org/r/25575/diff/


Testing
---

UT passed.


Thanks,

Dong Chen

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

> On Sept. 19, 2014, 8:52 p.m., Brock Noland wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 768
> > 
> >
> > if ( not null) {
> > 
> > } else {
> > 
> > 
> > is simpler as:
> > 
> > if (null) {
> > 
> > } else {

I think it is better to put the valid common case at first, like Thejas 
metioned in below comments. Shall we keep the order here?


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/#review54023
---


On Sept. 24, 2014, 9:17 a.m., Dong Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25575/
> ---
> 
> (Updated Sept. 24, 2014, 9:17 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs. Beeline could use the API introduced in 
> HIVE-4629 to get and display the logs to the client.
> 
> 
> Diffs
> -
> 
>   beeline/pom.xml 45fa02b 
>   beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  1e66542 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> daf8e9e 
>   jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
> PRE-CREATION 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 
> 
> Diff: https://reviews.apache.org/r/25575/diff/
> 
> 
> Testing
> ---
> 
> UT passed.
> 
> 
> Thanks,
> 
> Dong Chen
> 
>

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

> On Sept. 15, 2014, 6:56 p.m., Brock Noland wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 93
> > 
> >
> > Used internally can be private
> 
> Dong Chen wrote:
> Class Commands also use it. Maybe we have to keep it public.

This enum is removed from patch


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/#review53376
---


On Sept. 24, 2014, 9:17 a.m., Dong Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25575/
> ---
> 
> (Updated Sept. 24, 2014, 9:17 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs. Beeline could use the API introduced in 
> HIVE-4629 to get and display the logs to the client.
> 
> 
> Diffs
> -
> 
>   beeline/pom.xml 45fa02b 
>   beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  1e66542 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> daf8e9e 
>   jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
> PRE-CREATION 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 
> 
> Diff: https://reviews.apache.org/r/25575/diff/
> 
> 
> Testing
> ---
> 
> UT passed.
> 
> 
> Thanks,
> 
> Dong Chen
> 
>

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

> On Sept. 18, 2014, 8:07 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 98
> > 
> >
> > Looks like we just need to check isRunning (ie true or false) for 
> > running.
> > We don't need the whole set of enums, and complexity associated with 
> > getting those values right.
> > Can we just use a boolean instead of the enum ?
> > 
> > Apart from complexity, the reason for using boolean instead of enum -  
> > I am also concerned about people starting to use these enums and then 
> > complain if we change them in future. Even though we haven't documented it 
> > as a public api, people might start using them. So keeping the 
> > functionality we promise through methods here to minimal is safer.

Use the method in below comments to track HiveStatement status.


> On Sept. 18, 2014, 8:07 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 247
> > 
> >
> > set isRunning=true here and set isRunning=false in other place of other 
> > calls to setQueryStatus

Use the method in below comments to track HiveStatement status.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/#review53809
---


On Sept. 24, 2014, 9:17 a.m., Dong Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25575/
> ---
> 
> (Updated Sept. 24, 2014, 9:17 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs. Beeline could use the API introduced in 
> HIVE-4629 to get and display the logs to the client.
> 
> 
> Diffs
> -
> 
>   beeline/pom.xml 45fa02b 
>   beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  1e66542 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> daf8e9e 
>   jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
> PRE-CREATION 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 
> 
> Diff: https://reviews.apache.org/r/25575/diff/
> 
> 
> Testing
> ---
> 
> UT passed.
> 
> 
> Thanks,
> 
> Dong Chen
> 
>

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

> On Sept. 19, 2014, 9:32 p.m., Swarnim Kulkarni wrote:
> > beeline/src/java/org/apache/hive/beeline/Commands.java, line 833
> > 
> >
> > Also show we be checking on the SQL error code vs the message. Grepping 
> > on the error seems a little risky.

Thanks for pointing out this risky code. The new patch return an empty list 
instead.


- Dong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/#review54027
---


On Sept. 24, 2014, 9:17 a.m., Dong Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25575/
> ---
> 
> (Updated Sept. 24, 2014, 9:17 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs. Beeline could use the API introduced in 
> HIVE-4629 to get and display the logs to the client.
> 
> 
> Diffs
> -
> 
>   beeline/pom.xml 45fa02b 
>   beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
>   
> itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java
>  1e66542 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> daf8e9e 
>   jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
> PRE-CREATION 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 
> 
> Diff: https://reviews.apache.org/r/25575/diff/
> 
> 
> Testing
> ---
> 
> UT passed.
> 
> 
> Thanks,
> 
> Dong Chen
> 
>

Re: Review Request 25575: HIVE-7615: Beeline should have an option for user to see the query progress

[Dong Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25575/
---

(Updated Sept. 24, 2014, 9:51 a.m.)


Review request for hive.


Changes
---

a little change on the updated patch. Move the place of cancelled state 
checking in getQueryLog().


Repository: hive-git


Description
---

When executing query in Beeline, user should have a option to see the progress 
through the outputs. Beeline could use the API introduced in HIVE-4629 to get 
and display the logs to the client.


Diffs (updated)
-

  beeline/pom.xml 45fa02b 
  beeline/src/java/org/apache/hive/beeline/Commands.java a92d69f 
  
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java 
1e66542 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
daf8e9e 
  jdbc/src/java/org/apache/hive/jdbc/ClosedOrCancelledStatementException.java 
PRE-CREATION 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 86bc580 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2cbf58c 

Diff: https://reviews.apache.org/r/25575/diff/


Testing
---

UT passed.


Thanks,

Dong Chen

[jira] [Updated] (HIVE-7615) Beeline should have an option for user to see the query progress

[Dong Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Chen updated HIVE-7615:

Attachment: HIVE-7615.3.patch

Hi, [~brocknoland], [~thejas], [~swarnim]. Thanks for your comments to improve 
the quality of the patch. I have updated V3.

The new patch mainly added a public API hasMoreLogs() and modified the internal 
status tracking and methods sync up of HiveStatement.

Thanks very much for your effort. :)

> Beeline should have an option for user to see the query progress
> 
>
> Key: HIVE-7615
> URL: https://issues.apache.org/jira/browse/HIVE-7615
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-7615.1.patch, HIVE-7615.2.patch, HIVE-7615.3.patch, 
> HIVE-7615.patch, complete_logs, simple_logs
>
>
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs.
> Beeline could use the API introduced in HIVE-4629 to get and display the logs 
> to the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8203) ACID operations result in NPE when run through HS2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146165#comment-14146165
 ] 

Hive QA commented on HIVE-8203:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670841/HIVE-8203.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6344 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/957/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/957/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-957/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670841

> ACID operations result in NPE when run through HS2
> --
>
> Key: HIVE-8203
> URL: https://issues.apache.org/jira/browse/HIVE-8203
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8203.patch
>
>
> When accessing Hive via HS2, any operation requiring the DbTxnManager results 
> in an NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8171) Tez and Vectorized Reduce doesn't create scratch columns

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146166#comment-14146166
 ] 

Hive QA commented on HIVE-8171:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670853/HIVE-8171.02.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/958/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/958/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-958/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-958/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java'
Reverted 
'itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTxns.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/webhcat/svr/target 
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target 
accumulo-handler/target hwi/target common/target common/src/gen contrib/target 
service/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1627268.

At revision 1627268.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670853

> Tez and Vectorized Reduce doesn't create scratch columns
> 
>
> Key: HIVE-8171
> URL: https://issues.apache.org/jira/browse/HIVE-8171
> Project: Hive
>  Issue Type: Bug
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-8171.01.patch, HIVE-8171.02.patch
>
>

[jira] [Commented] (HIVE-8223) CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146167#comment-14146167
 ] 

Hive QA commented on HIVE-8223:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670851/HIVE-8223.01.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/959/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/959/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-959/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-959/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1627268.

At revision 1627268.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670851

> CBO Trunk Merge: partition_wise_fileformat2 select result depends on ordering
> -
>
> Key: HIVE-8223
> URL: https://issues.apache.org/jira/browse/HIVE-8223
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8223.01.patch, HIVE-8223.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8086) Disable Trivial Project Removal Rule, Fix Result Schema

[Laljo John Pullokkaran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laljo John Pullokkaran updated HIVE-8086:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Disable Trivial Project Removal Rule, Fix Result Schema
> ---
>
> Key: HIVE-8086
> URL: https://issues.apache.org/jira/browse/HIVE-8086
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8086.1.patch, HIVE-8086.patch
>
>
> Disable trivial project rule till Optiq-407 gets fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8086) CBO: Disable Trivial Project Removal Rule, Fix Result Schema

[Laljo John Pullokkaran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laljo John Pullokkaran updated HIVE-8086:
-
Summary: CBO: Disable Trivial Project Removal Rule, Fix Result Schema  
(was: Disable Trivial Project Removal Rule, Fix Result Schema)

> CBO: Disable Trivial Project Removal Rule, Fix Result Schema
> 
>
> Key: HIVE-8086
> URL: https://issues.apache.org/jira/browse/HIVE-8086
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8086.1.patch, HIVE-8086.patch
>
>
> Disable trivial project rule till Optiq-407 gets fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8111) CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146233#comment-14146233
 ] 

Hive QA commented on HIVE-8111:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670862/HIVE-8111.02.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6343 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/960/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/960/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-960/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670862

> CBO trunk merge: duplicated casts for arithmetic expressions in Hive and CBO
> 
>
> Key: HIVE-8111
> URL: https://issues.apache.org/jira/browse/HIVE-8111
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8111.01.patch, HIVE-8111.02.patch, HIVE-8111.patch
>
>
> Original test failure: looks like column type changes to different decimals 
> in most cases. In one case it causes the integer part to be too big to fit, 
> so the result becomes null it seems.
> What happens is that CBO adds casts to arithmetic expressions to make them 
> type compatible; these casts become part of new AST, and then Hive adds casts 
> on top of these casts. This (the first part) also causes lots of out file 
> changes. It's not clear how to best fix it so far, in addition to incorrect 
> decimal width and sometimes nulls when width is larger than allowed in Hive.
> Option one - don't add those for numeric ops - cannot be done if numeric op 
> is a part of compare, for which CBO needs correct types.
> Option two - unwrap casts when determining type in Hive - hard or impossible 
> to tell apart CBO-added casts and user casts. 
> Option three - don't change types in Hive if CBO has run - seems hacky and 
> hard to ensure it's applied everywhere.
> Option four - map all expressions precisely between two trees and remove 
> casts again after optimization, will be pretty difficult.
> Option five - somehow mark those casts. Not sure about how yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7689) Enable Postgres as METASTORE back-end

[Damien Carol (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146240#comment-14146240
 ] 

Damien Carol commented on HIVE-7689:


[~alangates] I understand more clearly now why you have some concerns regarding 
that JIRA.
bq. Are you concerned that having upper case names for DataNucleus generated 
tables but lower case names for transaction tables is confusing? I don't see 
that as an issue since these tables are not a public interface.
First time I launched the script for posgres, I saw the new tables was lower 
case but all other tables was in upper case. I believed it was a bug. If this 
not an issue, I think we can close this JIRA.

It's just confusing. Every tables in Metastore rely on same upper case policy 
and naming conventions.

The only problem is when we will extend Jdbc stats provider or provide access 
to this lower cased tables though DataNucleus.
Plus if we want to support a new Database for back-end that don't rely on 
"default lower case" behavior.

> Enable Postgres as METASTORE back-end
> -
>
> Key: HIVE-7689
> URL: https://issues.apache.org/jira/browse/HIVE-7689
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Damien Carol
>Assignee: Damien Carol
>Priority: Minor
>  Labels: metastore, postgres
> Fix For: 0.14.0
>
> Attachments: HIVE-7689.5.patch, HIVE-7689.6.patch, HIVE-7689.7.patch, 
> HIVE-7689.8.patch, HIVE-7889.1.patch, HIVE-7889.2.patch, HIVE-7889.3.patch, 
> HIVE-7889.4.patch
>
>
> I maintain few patches to make Metastore works with Postgres back end in our 
> production environment.
> The main goal of this JIRA is to push upstream these patches.
> This patch enable LOCKS, COMPACTION and fix error in STATS on postgres 
> metastore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7107) Clean up HiveDrivers and fix an issue

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HIVE-7107:
---
Affects Version/s: 0.14.0

> Clean up HiveDrivers and fix an issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HIVE-7107:
---
Summary: Fix HiveServer1 JDBC Driver spec compliancy issue  (was: Clean up 
HiveDrivers and fix an issue)

> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 24076: HIVE-7107: Clean up HiveDrivers and fix an issue

[Lars Francke]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24076/
---

(Updated Sept. 24, 2014, 12:06 p.m.)


Review request for hive.


Bugs: HIVE-7107
https://issues.apache.org/jira/browse/HIVE-7107


Repository: hive-git


Description
---

The HiveServer1 driver does not adhere to the JDBC standard of returning null 
when it can't handle a connection URL.

I realize that HS1 is scheduled to be removed but I submitted this before 
0.13.1 was released and maybe 0.13.2 will be released still. Also I think 
there's a chance that some distributions will keep HS1 in for a bit longer and 
this patch fixes an issue with it.


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 
fa8c43b 
  jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java 68c0788 

Diff: https://reviews.apache.org/r/24076/diff/


Testing
---


Thanks,

Lars Francke

[jira] [Updated] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Lars Francke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Francke updated HIVE-7107:
---
Attachment: HIVE-7107.2.patch

This is the patch for the mentioned JDBC issue and without any cleanup.

> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch, HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Lars Francke (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146249#comment-14146249
 ] 

Lars Francke commented on HIVE-7107:


I have also updated Review Board. [~thejas] it should be ready for review now, 
thanks!

> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch, HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8240) VectorColumnAssignFactory throws "Incompatible Bytes vector column and primitive category VARCHAR"

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146298#comment-14146298
 ] 

Hive QA commented on HIVE-8240:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670867/HIVE-8240.01.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6344 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_char_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_varchar_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/961/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/961/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-961/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670867

> VectorColumnAssignFactory throws "Incompatible Bytes vector column and 
> primitive category VARCHAR"
> --
>
> Key: HIVE-8240
> URL: https://issues.apache.org/jira/browse/HIVE-8240
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-8240.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8201) Remove hardwiring to HiveInputFormat in acid qfile tests

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146375#comment-14146375
 ] 

Owen O'Malley commented on HIVE-8201:
-

+1

Bonus point for having "hive.doing.acid"

> Remove hardwiring to HiveInputFormat in acid qfile tests
> 
>
> Key: HIVE-8201
> URL: https://issues.apache.org/jira/browse/HIVE-8201
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-8201.2.patch, HIVE-8201.patch
>
>
> Now that HIVE-7812 is checked in we should remove the hardwiring to 
> HiveInputFormat for the qfile tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6148) Support arbitrary structs stored in HBase

[Swarnim Kulkarni (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Swarnim Kulkarni updated HIVE-6148:
---
Attachment: HIVE-6148.2.patch.txt

Thanks for the review [~brocknoland]. Addressed comments and patch updated.

> Support arbitrary structs stored in HBase
> -
>
> Key: HIVE-6148
> URL: https://issues.apache.org/jira/browse/HIVE-6148
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.12.0
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt
>
>
> We should add support to be able to query arbitrary structs stored in HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146388#comment-14146388
 ] 

Hive QA commented on HIVE-8021:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670884/HIVE-8021.01.patch

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6344 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_correctness
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_correctness
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchAbort
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/962/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/962/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-962/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670884

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6683) Beeline does not accept comments at end of line

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-6683:
--
Status: Open  (was: Patch Available)

> Beeline does not accept comments at end of line
> ---
>
> Key: HIVE-6683
> URL: https://issues.apache.org/jira/browse/HIVE-6683
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.10.0
>Reporter: Jeremy Beard
>Assignee: Sergio Peña
> Fix For: 0.14.0
>
> Attachments: HIVE-6683.1.patch
>
>
> Beeline fails to read queries where lines have comments at the end. This 
> works in the embedded Hive CLI.
> Example:
> SELECT
> 1 -- this is a comment about this value
> FROM
> table;
> Error: Error while processing statement: FAILED: ParseException line 1:36 
> mismatched input '' expecting FROM near '1' in from clause 
> (state=42000,code=4)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8189) A select statement with a subquery is failing with HBaseSerde

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146409#comment-14146409
 ] 

Yongzhi Chen commented on HIVE-8189:


Add the patch to the review board:
https://reviews.apache.org/r/25993/

> A select statement with a subquery is failing with HBaseSerde
> -
>
> Key: HIVE-8189
> URL: https://issues.apache.org/jira/browse/HIVE-8189
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 0.12.0, 0.13.1
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-8189.1.patch, hbase_ppd_join.q
>
>
> Hive tables in the query are hbase tables, and the subquery is a join 
> statement.
> When
> set hive.optimize.ppd=true;
>   and
> set hive.auto.convert.join=false;
> The query does not return data. 
> While hive.optimize.ppd=true and hive.auto.convert.join=true return values 
> back. See attached query file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8182) beeline fails when executing multiple-line queries with trailing spaces

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-8182:
--
Release Note: Re-submit patch to run tests non-related with this fix.
  Status: Patch Available  (was: Open)

> beeline fails when executing multiple-line queries with trailing spaces
> ---
>
> Key: HIVE-8182
> URL: https://issues.apache.org/jira/browse/HIVE-8182
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.1, 0.12.0
>Reporter: Yongzhi Chen
>Assignee: Sergio Peña
> Fix For: 0.14.0
>
> Attachments: HIVE-8181.1.patch
>
>
> As title indicates, when executing a multi-line query with trailing spaces, 
> beeline reports syntax error: 
> Error: Error while compiling statement: FAILED: ParseException line 1:76 
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> If put this query in one single line, beeline succeeds to execute it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6683) Beeline does not accept comments at end of line

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-6683:
--
Release Note: Re-submit patch to run tests non-related with this fix.
  Status: Patch Available  (was: Open)

> Beeline does not accept comments at end of line
> ---
>
> Key: HIVE-6683
> URL: https://issues.apache.org/jira/browse/HIVE-6683
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.10.0
>Reporter: Jeremy Beard
>Assignee: Sergio Peña
> Fix For: 0.14.0
>
> Attachments: HIVE-6683.1.patch
>
>
> Beeline fails to read queries where lines have comments at the end. This 
> works in the embedded Hive CLI.
> Example:
> SELECT
> 1 -- this is a comment about this value
> FROM
> table;
> Error: Error while processing statement: FAILED: ParseException line 1:36 
> mismatched input '' expecting FROM near '1' in from clause 
> (state=42000,code=4)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8182) beeline fails when executing multiple-line queries with trailing spaces

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HIVE-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergio Peña updated HIVE-8182:
--
Status: Open  (was: Patch Available)

> beeline fails when executing multiple-line queries with trailing spaces
> ---
>
> Key: HIVE-8182
> URL: https://issues.apache.org/jira/browse/HIVE-8182
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.1, 0.12.0
>Reporter: Yongzhi Chen
>Assignee: Sergio Peña
> Fix For: 0.14.0
>
> Attachments: HIVE-8181.1.patch
>
>
> As title indicates, when executing a multi-line query with trailing spaces, 
> beeline reports syntax error: 
> Error: Error while compiling statement: FAILED: ParseException line 1:76 
> extraneous input ';' expecting EOF near '' (state=42000,code=4)
> If put this query in one single line, beeline succeeds to execute it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7647) Beeline does not honor --headerInterval and --color when executing with "-e"

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146440#comment-14146440
 ] 

Naveen Gangam commented on HIVE-7647:
-

[~leftylev] Thank you for granting me Wiki-write permissions. I have added a 
link to this JIRA on the Beeline Command Options wiki page. Thanks

> Beeline does not honor --headerInterval and --color when executing with "-e"
> 
>
> Key: HIVE-7647
> URL: https://issues.apache.org/jira/browse/HIVE-7647
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Minor
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7647.1.patch, HIVE-7647.2.patch
>
>
> --showHeader is being honored
> [root@localhost ~]# beeline --showHeader=false -u 
> 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -e "select * from sample_07 limit 10;"
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> -hiveconf (No such file or directory)
> +--+--++-+
> | 00-  | All Occupations  | 135185230  | 42270   |
> | 11-  | Management occupations   | 6152650| 100310  |
> | 11-1011  | Chief executives | 301930 | 160440  |
> | 11-1021  | General and operations managers  | 1697690| 107970  |
> | 11-1031  | Legislators  | 64650  | 37980   |
> | 11-2011  | Advertising and promotions managers  | 36100  | 94720   |
> | 11-2021  | Marketing managers   | 166790 | 118160  |
> | 11-2022  | Sales managers   | 333910 | 110390  |
> | 11-2031  | Public relations managers| 51730  | 101220  |
> | 11-3011  | Administrative services managers | 246930 | 79500   |
> +--+--++-+
> 10 rows selected (0.838 seconds)
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> Closing: org.apache.hive.jdbc.HiveConnection
> --outputFormat is being honored.
> [root@localhost ~]# beeline --outputFormat=csv -u 
> 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -e "select * from sample_07 limit 10;"
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> 'code','description','total_emp','salary'
> '00-','All Occupations','135185230','42270'
> '11-','Management occupations','6152650','100310'
> '11-1011','Chief executives','301930','160440'
> '11-1021','General and operations managers','1697690','107970'
> '11-1031','Legislators','64650','37980'
> '11-2011','Advertising and promotions managers','36100','94720'
> '11-2021','Marketing managers','166790','118160'
> '11-2022','Sales managers','333910','110390'
> '11-2031','Public relations managers','51730','101220'
> '11-3011','Administrative services managers','246930','79500'
> 10 rows selected (0.664 seconds)
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> Closing: org.apache.hive.jdbc.HiveConnection
> both --color & --headerInterval are being honored when executing using "-f" 
> option (reads query from a file rather than the commandline) (cannot really 
> see the color here but use the terminal colors)
> [root@localhost ~]# beeline --showheader=true --color=true --headerInterval=5 
> -u 'jdbc:hive2://localhost:1/default' -n hive -d 
> org.apache.hive.jdbc.HiveDriver -f /tmp/tmp.sql  
> Connecting to jdbc:hive2://localhost:1/default
> Connected to: Apache Hive (version 0.12.0-cdh5.0.1)
> Driver: Hive JDBC (version 0.12.0-cdh5.0.1)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Beeline version 0.12.0-cdh5.1.0 by Apache Hive
> 0: jdbc:hive2://localhost> select * from sample_07 limit 8;
> +--+--++-+
> |   code   | description  | total_emp  | salary  |
> +--+--++-+
> | 00-  | All Occupations  | 135185230  | 42270   |
> | 11-  | Management occupations   | 6152650| 100310  |
> | 11-1011  | Chief executives | 301930 | 160440  |
> | 11-1021  | General and operations managers  | 1697690| 107970  |
> | 11-1031  | Legislators  | 64650  | 37980   |
> +-

Re: Need wiki permission

[Josh Elser]

Thanks, Lefty!

Lefty Leverenz wrote:

Done.  Thanks for contributing to the Hive wiki, Josh!

-- Lefty

On Wed, Sep 24, 2014 at 1:05 AM, Josh Elser  wrote:


Ugh, sorry -- username is 'elserj'


Josh Elser wrote:


Could someone please grant me create permission on Confluence? I need to
circle around on writing some documentation for the recently added
AccumuloStorageHandler (HIVE-7789).

Thanks!

- Josh

[jira] [Updated] (HIVE-8237) CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8237:
---
Fix Version/s: 0.14.0

> CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)
> -
>
> Key: HIVE-8237
> URL: https://issues.apache.org/jira/browse/HIVE-8237
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Fix For: 0.14.0
>
> Attachments: HIVE-8237.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8237) CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8237:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Failure is unrelated. Committed to trunk. Thanks, John!

> CBO: Use Fully qualified table name (db.tablename in ReloptHiveTable)
> -
>
> Key: HIVE-8237
> URL: https://issues.apache.org/jira/browse/HIVE-8237
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Laljo John Pullokkaran
>Assignee: Laljo John Pullokkaran
> Attachments: HIVE-8237.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8244) INSERT/UPDATE/DELETE should return count of rows affected

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-8244:


 Summary: INSERT/UPDATE/DELETE should return count of rows affected
 Key: HIVE-8244
 URL: https://issues.apache.org/jira/browse/HIVE-8244
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.14.0
Reporter: Eugene Koifman


it's common in SQL and JDBC 
[API|http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeUpdate(java.lang.String)]
 to return count of affected rows.

Hive should do the same (it's not as of 9/24/2014)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8169) Windows: alter table ..set location from hcatalog failed with NullPointerException

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146473#comment-14146473
 ] 

Hive QA commented on HIVE-8169:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670859/HIVE-8169.2.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6344 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/963/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/963/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-963/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670859

> Windows: alter table ..set location from hcatalog failed with 
> NullPointerException
> --
>
> Key: HIVE-8169
> URL: https://issues.apache.org/jira/browse/HIVE-8169
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
> Environment: Windows Server 2008 R2
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HIVE-8169.1.patch, HIVE-8169.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8224) Support Char, Varchar in AvroSerDe

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8224:
--
   Resolution: Fixed
Fix Version/s: 0.14.0
 Release Note: To document support of char/varchar for avro.
   Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks to Mohit for the contribution.

> Support Char, Varchar in AvroSerDe
> --
>
> Key: HIVE-8224
> URL: https://issues.apache.org/jira/browse/HIVE-8224
> Project: Hive
>  Issue Type: Task
>  Components: Serializers/Deserializers
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
>  Labels: Avro
> Fix For: 0.14.0
>
> Attachments: HIVE-8224.1.patch, HIVE-8224.patch
>
>
> Both Char and Varchar represented as String primitive type in Avro. 
> Char is persisted without padding, if any.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8224) Support Char, Varchar in AvroSerDe

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8224:
--
Tags: TODOC14

> Support Char, Varchar in AvroSerDe
> --
>
> Key: HIVE-8224
> URL: https://issues.apache.org/jira/browse/HIVE-8224
> Project: Hive
>  Issue Type: Task
>  Components: Serializers/Deserializers
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
>  Labels: Avro
> Fix For: 0.14.0
>
> Attachments: HIVE-8224.1.patch, HIVE-8224.patch
>
>
> Both Char and Varchar represented as String primitive type in Avro. 
> Char is persisted without padding, if any.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8189) A select statement with a subquery is failing with HBaseSerde

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146540#comment-14146540
 ] 

Brock Noland commented on HIVE-8189:


+1

> A select statement with a subquery is failing with HBaseSerde
> -
>
> Key: HIVE-8189
> URL: https://issues.apache.org/jira/browse/HIVE-8189
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Affects Versions: 0.12.0, 0.13.1
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-8189.1.patch, hbase_ppd_join.q
>
>
> Hive tables in the query are hbase tables, and the subquery is a join 
> statement.
> When
> set hive.optimize.ppd=true;
>   and
> set hive.auto.convert.join=false;
> The query does not return data. 
> While hive.optimize.ppd=true and hive.auto.convert.join=true return values 
> back. See attached query file. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8205) Using strings in group type fails in ParquetSerDe

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8205:
--
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks, Mohit.

> Using strings in group type fails in ParquetSerDe
> -
>
> Key: HIVE-8205
> URL: https://issues.apache.org/jira/browse/HIVE-8205
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
>  Labels: parquet
> Fix For: 0.14.0
>
> Attachments: HIVE-8205.1.patch, HIVE-8205.1.patch, HIVE-8205.patch
>
>
> In HIVE-7735, schema info was plumbed to ETypeConverter to disambiguate 
> between hive Char, Varchar and String types, which are all represented as 
> PrimitiveType "binary" and OriginalType "utf8" in parquet.
> However, this does not work for parquet nested types (that map to hive Array, 
> Map, etc.) containing these values, because schema lookup for nested values 
> was not implemented.  It's also non-trivial to do that in the current parquet 
> serde implementation. Instead of plumbing in the schema, we should convert 
> these types to the same Text writeable and let the object inspectors handle 
> the final conversion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8241) With vectorization enabled count(distinct)) fails with ClassCastException

[Mostafa Mokhtar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146554#comment-14146554
 ] 

Mostafa Mokhtar commented on HIVE-8241:
---

Looks like the wrong ObjectInspector is used when vectorization is on.

> With vectorization enabled count(distinct)) fails with ClassCastException
> -
>
> Key: HIVE-8241
> URL: https://issues.apache.org/jira/browse/HIVE-8241
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: cn105
>Reporter: Mostafa Mokhtar
>Assignee: Matt McCline
> Fix For: 0.14.0
>
>
> Repro 
> {code}
> set hive.vectorized.execution.enabled=true;
> select count(distinct(cint)) from alltypesorc;
> {code}
> Exception 
> {code}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardUnionObjectInspector$StandardUnion
>  cannot be cast to org.apache.hadoop.io.IntWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:654)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:602)
>   at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.toHiveKey(ReduceSinkOperator.java:468)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.processOp(VectorReduceSinkOperator.java:283)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.flushOutput(VectorGroupByOperator.java:918)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:891)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$400(VectorGroupByOperator.java:63)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.flush(VectorGroupByOperator.java:463)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.checkHashModeEfficiency(VectorGroupByOperator.java:536)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:363)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:851)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8245) Collect table read entities at same time as view read entities

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-8245:
--

 Summary: Collect table read entities at same time as view read 
entities 
 Key: HIVE-8245
 URL: https://issues.apache.org/jira/browse/HIVE-8245
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8245) Collect table read entities at same time as view read entities

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8245:
---
Attachment: HIVE-8245.patch

> Collect table read entities at same time as view read entities 
> ---
>
> Key: HIVE-8245
> URL: https://issues.apache.org/jira/browse/HIVE-8245
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8245.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8245) Collect table read entities at same time as view read entities

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8245:
---
Status: Patch Available  (was: Open)

> Collect table read entities at same time as view read entities 
> ---
>
> Key: HIVE-8245
> URL: https://issues.apache.org/jira/browse/HIVE-8245
> Project: Hive
>  Issue Type: Improvement
>  Components: CBO
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-8245.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 25997: Collect table read entities at same time as view read entities

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25997/
---

Review request for hive and Thejas Nair.


Bugs: HIVE-8245
https://issues.apache.org/jira/browse/HIVE-8245


Repository: hive-git


Description
---

Collect table read entities at same time as view read entities 


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 2f36f04 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c4dacf9 

Diff: https://reviews.apache.org/r/25997/diff/


Testing
---

Unit tests


Thanks,

Ashutosh Chauhan

[jira] [Commented] (HIVE-8241) With vectorization enabled count(distinct)) fails with ClassCastException

[Mostafa Mokhtar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146573#comment-14146573
 ] 

Mostafa Mokhtar commented on HIVE-8241:
---

keyObject inspector is :

org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector
 
while it should be 

org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector>>

> With vectorization enabled count(distinct)) fails with ClassCastException
> -
>
> Key: HIVE-8241
> URL: https://issues.apache.org/jira/browse/HIVE-8241
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: cn105
>Reporter: Mostafa Mokhtar
>Assignee: Matt McCline
> Fix For: 0.14.0
>
>
> Repro 
> {code}
> set hive.vectorized.execution.enabled=true;
> select count(distinct(cint)) from alltypesorc;
> {code}
> Exception 
> {code}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardUnionObjectInspector$StandardUnion
>  cannot be cast to org.apache.hadoop.io.IntWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:654)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:602)
>   at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.toHiveKey(ReduceSinkOperator.java:468)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.processOp(VectorReduceSinkOperator.java:283)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.flushOutput(VectorGroupByOperator.java:918)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:891)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$400(VectorGroupByOperator.java:63)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.flush(VectorGroupByOperator.java:463)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.checkHashModeEfficiency(VectorGroupByOperator.java:536)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:363)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:851)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang reassigned HIVE-8233:
-

Assignee: Chao

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8233.1-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-8233:
--
Status: Patch Available  (was: Open)

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
> Attachments: HIVE-8233.1-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8221) authorize additional metadata read operations in metastore storage based authorization

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-8221:

Status: Patch Available  (was: Open)

> authorize additional metadata read operations in metastore storage based 
> authorization 
> ---
>
> Key: HIVE-8221
> URL: https://issues.apache.org/jira/browse/HIVE-8221
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.14.0
>
> Attachments: HIVE-8221.1.patch
>
>
> Table and database metadata read operations should also be authorized by 
> storage based authorization, when enabled in hive metastore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7974) Notification Event Listener movement to a new top level repl/ module

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146600#comment-14146600
 ] 

Sushanth Sowmyan commented on HIVE-7974:


1. The Main reason for that is visibility. Sometimes, code changes happen in 
ql/ that don't make appropriate changes in hcatalog-webhcat-java-client, for 
eg., and we have to run around trying to fix it. Strongly connected tests would 
catch these kinds of issues, but when you're a serialize and a deserialize away 
across a message format, it's easier to break something without realizing it. 
If we are to grow this further, I believe it should belong in hive codebase, 
rather than tucked away inside hcatalog-server-extensions. Also, if I now put 
ReplicationTask basing on top of hcatalog-server-extensions, writing that alone 
in a top level repl/ feels like a cyclic dependency issue waiting to happen. If 
I put ReplicationTask inside hcatalog-server-extensions, it risks being seen as 
HCat Replication rather than Hive Replication. I thought that moving it out to 
a top-level made things cleaner in the long run.

2. I thought about this for a bit. Not breaking backward compatibility is a 
huge deal, but if I had to move packages out of hcatalog-server-extensions, 
that potentially breaks Jackson auto-deserialization/serialization because the 
classnames are different. There is no fundamentally different data in it, and 
if we wrote our own serialize/deserialize, we could make it work irrespective 
of classnames, and the messages are format compatible. If we fixed 
serialization/deserialization by not using automatic 
serialization/deserialization, we could stick with one listener, and it 
wouldn't matter if it were a HCatEventMessage or an EventMessage.

What can be done easily, however, is that the hive metastore supports having 
more than one listener loaded, as a comma separated list, and users can manage 
migration development and testing with ReplListener, while still keeping their 
existing NotificationListener active, simply by specifying a different prefix. 
And they can then code-migrate over to the newer listeners over a couple of 
releases, and gain the ability to use replication on top of those events.

I completely understand your question on why it's worth the effort, and it was 
a hard call for me, and I eventually wound up in the side of this. If we decide 
not to, then we can cancel this patch, and simply do ReplicationTask inside 
hcatalog-server-extensions.

3. I've created https://issues.apache.org/jira/browse/HIVE-8165 to mark a 
deprecation leading to a removal, rather than an outright removal, as discussed 
above.

> Notification Event Listener movement to a new top level repl/ module
> 
>
> Key: HIVE-7974
> URL: https://issues.apache.org/jira/browse/HIVE-7974
> Project: Hive
>  Issue Type: Sub-task
>  Components: Import/Export
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-7974.patch
>
>
> We need to create a new hive module (say hive-repl? ) to subsume the 
> NotificationListener from HCatalog.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7605) add more javadoc to HivePrivilegeObject

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146604#comment-14146604
 ] 

Hive QA commented on HIVE-7605:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670910/HIVE-7605.2.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6342 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/964/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/964/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-964/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670910

> add more javadoc to HivePrivilegeObject
> ---
>
> Key: HIVE-7605
> URL: https://issues.apache.org/jira/browse/HIVE-7605
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-7605.1.patch, HIVE-7605.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7605) add more javadoc to HivePrivilegeObject

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146614#comment-14146614
 ] 

Jason Dere commented on HIVE-7605:
--

Looks fine to me. +1

> add more javadoc to HivePrivilegeObject
> ---
>
> Key: HIVE-7605
> URL: https://issues.apache.org/jira/browse/HIVE-7605
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-7605.1.patch, HIVE-7605.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8164) Adding in a ReplicationTask that converts a Notification Event to actionable tasks

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146616#comment-14146616
 ] 

Sushanth Sowmyan commented on HIVE-8164:


> Since the classes in this patch are really for use outside of Hive by third 
> party system such as Falcon, it makes sense to note that in the 
> package-info.java file and give some info on how a tool might use those files.

Agreed, will add info there.

> ReplicationTask.needsStagingDir - It would make more sense for this to be 
> abstract and let each subtype say yes/no rather than encoding the logic for 
> each subtype here.
> ReplicationTask.getSrcWhCommands and getDstWhCommands should be abstract as 
> well.

Of all the event types currently present in the replication design, 
ADD_PARTITION is the only event type that results in needing the staging 
directory. Implementing this in ReplicationTask makes it easy for other 
ReplicationTasks to avoid having some repetitive code. They can still choose to 
@Override, so that flexibility has not been taken from them, it just there is a 
default base implementation. I followed the same logic for getSrcWhCommands and 
getDstWhCommands to provide a default impl of being empty, but calling 
verifyActionable(), but these were intended to be overridden.

I think the default impl for needsStagingDir still has value, and would like to 
keep it as-is unless you object strongly to it. The get*WhCommands though, I 
will go ahead and make abstract.

> At least one real implementation of ReplicationTask would make it easier to 
> determine whether this is the right interface or not.

Agreed. I'll upload the Add/Drop/Alter Partition ones and their test in a 
couple of days - I had to rewire a bunch of them when trying to add in 
AlterPartition.

> Adding in a ReplicationTask that converts a Notification Event to actionable 
> tasks
> --
>
> Key: HIVE-8164
> URL: https://issues.apache.org/jira/browse/HIVE-8164
> Project: Hive
>  Issue Type: Sub-task
>  Components: Import/Export
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-8164.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8217) WebHCat 'jobs' endpoint fails if it runs into issues with any of the jobs

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-8217:
---
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk(0.14 still). Thanks, Eugene!

> WebHCat 'jobs' endpoint fails if it runs into issues with any of the jobs
> -
>
> Key: HIVE-8217
> URL: https://issues.apache.org/jira/browse/HIVE-8217
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.13.1
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.14.0
>
> Attachments: HIVE-8217.patch
>
>
> need to make org.apache.hive.hcatalog.templeton.Server.showJobList() return 
> info about as many jobs as possible instead of failing fast at
> {noformat}
>   if (showDetails) {
> StatusDelegator sd = new StatusDelegator(appConf);
> QueueStatusBean statusBean = sd.run(getDoAsUser(), job);
> jobItem.detail = statusBean;
>   }
> {noformat}
> no precommit tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7482) The execution side changes for SMB join in hive-tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7482:
-
Attachment: HIVE-7482.addendum.patch

> The execution side changes for SMB join in hive-tez
> ---
>
> Key: HIVE-7482
> URL: https://issues.apache.org/jira/browse/HIVE-7482
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.14.0
>
> Attachments: HIVE-7482.1.patch, HIVE-7482.10.patch, 
> HIVE-7482.11.patch, HIVE-7482.12.patch, HIVE-7482.13.patch, 
> HIVE-7482.14.patch, HIVE-7482.2.patch, HIVE-7482.3.patch, HIVE-7482.4.patch, 
> HIVE-7482.5.patch, HIVE-7482.6.patch, HIVE-7482.7.patch, HIVE-7482.8.patch, 
> HIVE-7482.9.patch, HIVE-7482.WIP.2.patch, HIVE-7482.WIP.3.patch, 
> HIVE-7482.WIP.4.patch, HIVE-7482.WIP.patch, HIVE-7482.addendum.patch
>
>
> A piece of HIVE-7430.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8241) With vectorization enabled count(distinct)) fails with ClassCastException

[Mostafa Mokhtar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146667#comment-14146667
 ] 

Mostafa Mokhtar commented on HIVE-8241:
---

Issue starts in 
VectorExpressionWriterFactory.processVectorExpressions(List, 
List, SingleOIDClosure) line: 1044   


> With vectorization enabled count(distinct)) fails with ClassCastException
> -
>
> Key: HIVE-8241
> URL: https://issues.apache.org/jira/browse/HIVE-8241
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: cn105
>Reporter: Mostafa Mokhtar
>Assignee: Matt McCline
> Fix For: 0.14.0
>
>
> Repro 
> {code}
> set hive.vectorized.execution.enabled=true;
> select count(distinct(cint)) from alltypesorc;
> {code}
> Exception 
> {code}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardUnionObjectInspector$StandardUnion
>  cannot be cast to org.apache.hadoop.io.IntWritable
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:654)
>   at 
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:602)
>   at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.toHiveKey(ReduceSinkOperator.java:468)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.processOp(VectorReduceSinkOperator.java:283)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.flushOutput(VectorGroupByOperator.java:918)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:891)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$400(VectorGroupByOperator.java:63)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.flush(VectorGroupByOperator.java:463)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.checkHashModeEfficiency(VectorGroupByOperator.java:536)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.processBatch(VectorGroupByOperator.java:363)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:851)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:139)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7802) Update language manual for insert, update, and delete

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146660#comment-14146660
 ] 

Lefty Leverenz commented on HIVE-7802:
--

151 watchers now, so I'll review very carefully (but not until tonight, after 
traveling "back to America").

> Update language manual for insert, update, and delete
> -
>
> Key: HIVE-7802
> URL: https://issues.apache.org/jira/browse/HIVE-7802
> Project: Hive
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: TODOC14
>
> With the addition of ACID compliant insert, insert...values, update, and 
> delete we need to update the Hive language manual to cover the new features.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7605) add more javadoc to HivePrivilegeObject

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146686#comment-14146686
 ] 

Lefty Leverenz commented on HIVE-7605:
--

+1

Thanks for indulging my nitpick on Hive capitalization (even though Hive is 
lowercase elsewhere in the file, oh well).

> add more javadoc to HivePrivilegeObject
> ---
>
> Key: HIVE-7605
> URL: https://issues.apache.org/jira/browse/HIVE-7605
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-7605.1.patch, HIVE-7605.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 24076: HIVE-7107: Clean up HiveDrivers and fix an issue

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24076/#review54446
---

Ship it!


Ship It!

- Thejas Nair


On Sept. 24, 2014, 12:06 p.m., Lars Francke wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24076/
> ---
> 
> (Updated Sept. 24, 2014, 12:06 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7107
> https://issues.apache.org/jira/browse/HIVE-7107
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL.
> 
> I realize that HS1 is scheduled to be removed but I submitted this before 
> 0.13.1 was released and maybe 0.13.2 will be released still. Also I think 
> there's a chance that some distributions will keep HS1 in for a bit longer 
> and this patch fixes an issue with it.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java
>  fa8c43b 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java 68c0788 
> 
> Diff: https://reviews.apache.org/r/24076/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Lars Francke
> 
>

[jira] [Commented] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146710#comment-14146710
 ] 

Thejas M Nair commented on HIVE-7107:
-

+1

> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch, HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146715#comment-14146715
 ] 

Hive QA commented on HIVE-8233:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670840/HIVE-8233.1-spark.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6503 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/150/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/150/console
Test logs: 
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-150/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670840

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8233.1-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Chao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao updated HIVE-8233:
---
Attachment: HIVE-8233.2-spark.patch

We shouldn't change those qfiles since it will affect results from both MR/Tez.
This patch creates spark-branch specific qfiles, which have these the two 
configuration variables set to false.

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8233.1-spark.patch, HIVE-8233.2-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7615) Beeline should have an option for user to see the query progress

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146759#comment-14146759
 ] 

Hive QA commented on HIVE-7615:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670943/HIVE-7615.3.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6349 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/965/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/965/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-965/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670943

> Beeline should have an option for user to see the query progress
> 
>
> Key: HIVE-7615
> URL: https://issues.apache.org/jira/browse/HIVE-7615
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: 0.14.0
>
> Attachments: HIVE-7615.1.patch, HIVE-7615.2.patch, HIVE-7615.3.patch, 
> HIVE-7615.patch, complete_logs, simple_logs
>
>
> When executing query in Beeline, user should have a option to see the 
> progress through the outputs.
> Beeline could use the API introduced in HIVE-4629 to get and display the logs 
> to the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 26001: HIVE-8233 - multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Chao Sun]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26001/
---

Review request for hive, Brock Noland and Xuefu Zhang.


Bugs: hive-8233
https://issues.apache.org/jira/browse/hive-8233


Repository: hive-git


Description
---

Right now, for multi-table insertion, we will start from multiple 
FileSinkOperators, and break from their lowest common ancestor, adding 
temporary FileSinkOperator and TableScanOperators. A special case is when the 
LCA is a ForwardOperator, in which case we don't break it, since it's already 
been optimized.
However, there's a issue, considering the following plan:
  ...
  RS_0
   |
  FOR
   |
 /   \
   GBY_1  GBY_2
| |
   ...   ...
| |
   RS_1  RS_2
| |
   ...   ...
| |
   FS_1  FS_2
which may result to:
  RW
 /  \
   RWRW
Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
branches will get duplicated (and same) input.


Diffs
-

  itests/src/test/resources/testconfiguration.properties 637fbc1 
  
ql/src/test/queries/clientpositive/spark_groupby7_noskew_multi_single_reducer.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map_skew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_noskew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby9.q PRE-CREATION 
  
ql/src/test/queries/clientpositive/spark_groupby_multi_insert_common_distinct.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_union17.q PRE-CREATION 
  
ql/src/test/results/clientpositive/spark/spark_groupby7_noskew_multi_single_reducer.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map_skew.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_noskew.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby9.q.out PRE-CREATION 
  
ql/src/test/results/clientpositive/spark/spark_groupby_multi_insert_common_distinct.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_union17.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26001/diff/


Testing
---


Thanks,

Chao Sun

[jira] [Updated] (HIVE-7156) Group-By operator stat-annotation only uses distinct approx to generate rollups

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-7156:
-
Attachment: HIVE-7156.8.patch

rebase of .7

> Group-By operator stat-annotation only uses distinct approx to generate 
> rollups
> ---
>
> Key: HIVE-7156
> URL: https://issues.apache.org/jira/browse/HIVE-7156
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Prasanth J
> Attachments: HIVE-7156.1.patch, HIVE-7156.2.patch, HIVE-7156.3.patch, 
> HIVE-7156.4.patch, HIVE-7156.5.patch, HIVE-7156.6.patch, HIVE-7156.7.patch, 
> HIVE-7156.8.patch, hive-debug.log.bz2
>
>
> The stats annotation for a group-by only annotates the reduce-side row-count 
> with the distinct values.
> The map-side gets the row-count as the rows output instead of distinct * 
> parallelism, while the reducer side gets the correct parallelism.
> {code}
> hive> explain select distinct L_SHIPDATE from lineitem;
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: lineitem
>   Statistics: Num rows: 589709 Data size: 4745677733354 
> Basic stats: COMPLETE Column stats: COMPLETE
>   Select Operator
> expressions: l_shipdate (type: string)
> outputColumnNames: l_shipdate
> Statistics: Num rows: 589709 Data size: 4745677733354 
> Basic stats: COMPLETE Column stats: COMPLETE
> Group By Operator
>   keys: l_shipdate (type: string)
>   mode: hash
>   outputColumnNames: _col0
>   Statistics: Num rows: 589709 Data size: 
> 563999032646 Basic stats: COMPLETE Column stats: COMPLETE
>   Reduce Output Operator
> key expressions: _col0 (type: string)
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 589709 Data size: 
> 563999032646 Basic stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized
> Reducer 2 
> Reduce Operator Tree:
>   Group By Operator
> keys: KEY._col0 (type: string)
> mode: mergepartial
> outputColumnNames: _col0
> Statistics: Num rows: 1955 Data size: 183770 Basic stats: 
> COMPLETE Column stats: COMPLETE
> Select Operator
>   expressions: _col0 (type: string)
>   outputColumnNames: _col0
>   Statistics: Num rows: 1955 Data size: 183770 Basic stats: 
> COMPLETE Column stats: COMPLETE
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7073) Implement Binary in ParquetSerDe

[Pratik Khadloya (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146844#comment-14146844
 ] 

Pratik Khadloya commented on HIVE-7073:
---

Hello [~davidzchen] and [~brocknoland]
Can i help with anything here? I am interested in getting this jira moving. 
Please let me know if i can contribute in some way though this will be the 
first time i will be looking into Hive's code.

Thanks!

> Implement Binary in ParquetSerDe
> 
>
> Key: HIVE-7073
> URL: https://issues.apache.org/jira/browse/HIVE-7073
> Project: Hive
>  Issue Type: Sub-task
>Reporter: David Chen
>Assignee: David Chen
>
> The ParquetSerDe currently does not support the BINARY data type. This ticket 
> is to implement the BINARY data type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8246:
---
Affects Version/s: 0.13.0

> HiveServer2 in http-kerberos mode is restrictive on client usernames
> 
>
> Key: HIVE-8246
> URL: https://issues.apache.org/jira/browse/HIVE-8246
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
>
> Unable to use client usernames of the format:
> {code}
> username/host@REALM
> username@FOREIGN_REALM
> {code}
> The following works fine:
> {code}
> username@REALM 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8246:
---
Fix Version/s: 0.14.0

> HiveServer2 in http-kerberos mode is restrictive on client usernames
> 
>
> Key: HIVE-8246
> URL: https://issues.apache.org/jira/browse/HIVE-8246
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
>
> Unable to use client usernames of the format:
> {code}
> username/host@REALM
> username@FOREIGN_REALM
> {code}
> The following works fine:
> {code}
> username@REALM 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-8246:
--

 Summary: HiveServer2 in http-kerberos mode is restrictive on 
client usernames
 Key: HIVE-8246
 URL: https://issues.apache.org/jira/browse/HIVE-8246
 Project: Hive
  Issue Type: Bug
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


Unable to use client usernames of the format:
{code}
username/host@REALM
username@FOREIGN_REALM
{code}

The following works fine:
{code}
username@REALM 
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8246:
---
Component/s: JDBC
 HiveServer2

> HiveServer2 in http-kerberos mode is restrictive on client usernames
> 
>
> Key: HIVE-8246
> URL: https://issues.apache.org/jira/browse/HIVE-8246
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
>
> Unable to use client usernames of the format:
> {code}
> username/host@REALM
> username@FOREIGN_REALM
> {code}
> The following works fine:
> {code}
> username@REALM 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146860#comment-14146860
 ] 

Hive QA commented on HIVE-8233:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12671023/HIVE-8233.2-spark.patch

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 6519 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby7_noskew_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby8_map
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby8_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby8_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_union17
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/151/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/151/console
Test logs: 
http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-151/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12671023

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8233.1-spark.patch, HIVE-8233.2-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8228) CBO: fix couple of issues with partition pruning

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-8228:
---
Status: Patch Available  (was: Open)

> CBO: fix couple of issues with partition pruning
> 
>
> Key: HIVE-8228
> URL: https://issues.apache.org/jira/browse/HIVE-8228
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-8228.1.patch
>
>
> - Pruner doesn't handle non-deterministic UDFs correctly
> - Plan genned after CBO has a Project between TScan and Filter; which 
> prevents PartPruning from triggering in hive post CBO. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8246:
---
Status: Patch Available  (was: Open)

> HiveServer2 in http-kerberos mode is restrictive on client usernames
> 
>
> Key: HIVE-8246
> URL: https://issues.apache.org/jira/browse/HIVE-8246
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
> Attachments: HIVE-8246.1.patch
>
>
> Unable to use client usernames of the format:
> {code}
> username/host@REALM
> username@FOREIGN_REALM
> {code}
> The following works fine:
> {code}
> username@REALM 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8246) HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-8246:
---
Attachment: HIVE-8246.1.patch

> HiveServer2 in http-kerberos mode is restrictive on client usernames
> 
>
> Key: HIVE-8246
> URL: https://issues.apache.org/jira/browse/HIVE-8246
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.14.0
>
> Attachments: HIVE-8246.1.patch
>
>
> Unable to use client usernames of the format:
> {code}
> username/host@REALM
> username@FOREIGN_REALM
> {code}
> The following works fine:
> {code}
> username@REALM 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 26005: HiveServer2 in http-kerberos mode is restrictive on client usernames

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26005/
---

Review request for hive, dilli dorai and Thejas Nair.


Bugs: HIVE-8246
https://issues.apache.org/jira/browse/HIVE-8246


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-8246


Diffs
-

  service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 82093fa 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java 
99ef8bc 

Diff: https://reviews.apache.org/r/26005/diff/


Testing
---

Manual testing on secure cluster.


Thanks,

Vaibhav Gumashta

[jira] [Commented] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146868#comment-14146868
 ] 

Sergey Shelukhin commented on HIVE-8225:


[~ashutoshc] fyi

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8225.inprogress.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8225:
---
Attachment: HIVE-8225.inprogress.patch

Here's partial fix in process... first I tried using Optiq rules but that 
doesn't work due to some problems with transform (described in the patch).
I added a fix to post-CBO plan processing (before AST conversion), which fixes 
the issue for one small query that originally had the problem (see q file), but 
not for the original query. I'd need to take a look at that (probably the fix 
needs to be in a better place, or some check mismatches), for now attaching the 
in-progress patch since I will be on vacation soon

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8225.inprogress.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146869#comment-14146869
 ] 

Hive QA commented on HIVE-7107:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12670956/HIVE-7107.2.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6347 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/966/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/966/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-966/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12670956

> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch, HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-8247) Pig cursor written to Hive via HCat doesn't NULL-fill missing columns

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-8247:


 Summary: Pig cursor written to Hive via HCat doesn't NULL-fill 
missing columns
 Key: HIVE-8247
 URL: https://issues.apache.org/jira/browse/HIVE-8247
 Project: Hive
  Issue Type: Bug
  Components: HCatalog
Affects Versions: 0.13.1
Reporter: Eugene Koifman


This started out as BUG-15650 but in BUG-15650 it's no longer clear what the 
real issue is so I'm filing a new ticket.

Suppose a Hive table has columns (a,b,c,d)
If a Pig script writing to this table produces schema (a,b,c) it works: 'd' 
will be NULL.

If a Pig script writing to this table produces schema (a,b,d) it fails with 
error below.
This is an old issue.  There is nothing in HCatalog documentation that 
indicates whether this should work.

{noformat}
Running org.apache.hive.hcatalog.pig.TestOrcHCatStorer
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.113 sec <<< 
FAILURE! - in org.apache.hive.hcatalog.pig.TestOrcHCatStorer
partialSchemaSepcification(org.apache.hive.hcatalog.pig.TestOrcHCatStorer)  
Time elapsed: 29.886 sec  <<< ERROR!
org.apache.pig.impl.logicalLayer.FrontendException: Unable to store alias ABD
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1635)
at org.apache.pig.PigServer.registerQuery(PigServer.java:575)
at 
org.apache.hive.hcatalog.mapreduce.HCatBaseTest.logAndRegister(HCatBaseTest.java:92)
at 
org.apache.hive.hcatalog.pig.TestHCatStorer.partialSchemaSepcification(TestHCatStorer.java:1035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: org.apache.pig.impl.plan.VisitorException: 
 Output Location Validation Failed for: 'T More info to 
follow:
org.apache.hive.hcatalog.common.HCatException : 2007 : Invalid column position 
in partition schema : Expected column  at position 3, found column 
at 
org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:75)
at 
org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
at 
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
at 
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at 
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at 
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at 
org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
at 
org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at 
org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java

[jira] [Commented] (HIVE-8031) CBO should use per column join selectivity not NDV when applying exponential backoff.

[Mostafa Mokhtar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146874#comment-14146874
 ] 

Mostafa Mokhtar commented on HIVE-8031:
---

Issue is that NDV is not scaled with row count as a results the CE of the join 
ends up being too low.
Another issue is that the join of date_dim x store_sales doesn't go through the 
PK/FK path HiveRelMdUniqueKeys.getUniqueKeys(ProjectRelBase, boolean) is of 
type HepRelVertex not HiveTableScanRel.


> CBO should use per column join selectivity not NDV when applying exponential 
> backoff.
> -
>
> Key: HIVE-8031
> URL: https://issues.apache.org/jira/browse/HIVE-8031
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Mostafa Mokhtar
>Assignee: Harish Butani
> Fix For: 0.14.0
>
>
> Currently CBO uses NDV not join selectivity in computeInnerJoinSelectivity 
> which results in in-accurate estimate number of rows.
> I looked at the plan for TPC-DS Q17 after the latest set of changes and I am 
> concerned that the estimate of rows for the join of store_sales and 
> store_returns is so low, as you can see the estimate is 8461 rows for joining 
> 1.2795706667449066E8 with 1.2922108035889767E7.
> {code}
> HiveJoinRel(condition=[AND(=($130, $3), =($129, $15))], 
> joinType=[inner]): rowcount = 1079.1345153548855, cumulative cost = 
> {8.271845957931738E10 rows, 0.0 cpu, 0.0 io}, id = 517
>   HiveJoinRel(condition=[=($0, $38)], joinType=[inner]): 
> rowcount = 6.669190301841249E7, cumulative cost = {4.300510912631623E10 rows, 
> 0.0 cpu, 0.0 io}, id = 402
> HiveTableScanRel(table=[[catalog_sales]]): rowcount = 
> 4.3005109025E10, cumulative cost = {0}, id = 2
> HiveFilterRel(condition=[in($15, '2000Q1', '2000Q2', 
> '2000Q3')]): rowcount = 101.31622746185853, cumulative cost = {0.0 rows, 0.0 
> cpu, 0.0 io}, id = 181
>   HiveTableScanRel(table=[[d3]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 3
>   HiveJoinRel(condition=[AND(AND(=($3, $61), =($2, $60)), 
> =($9, $67))], joinType=[inner]): rowcount = 8461.27236667537, cumulative cost 
> = {8.26517592150266E10 rows, 0.0 cpu, 0.0 io}, id = 515
> HiveJoinRel(condition=[=($27, $0)], joinType=[inner]): 
> rowcount = 1.2795706667449066E8, cumulative cost = {8.251088004031622E10 
> rows, 0.0 cpu, 0.0 io}, id = 417
>   HiveTableScanRel(table=[[store_sales]]): rowcount = 
> 8.2510879939E10, cumulative cost = {0}, id = 5
>   HiveFilterRel(condition=[=($15, '2000Q1')]): rowcount = 
> 101.31622746185853, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 173
> HiveTableScanRel(table=[[d1]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 0
> HiveJoinRel(condition=[=($0, $24)], joinType=[inner]): 
> rowcount = 1.2922108035889767E7, cumulative cost = {8.332595810316228E9 rows, 
> 0.0 cpu, 0.0 io}, id = 424
>   HiveTableScanRel(table=[[store_returns]]): rowcount = 
> 8.332595709E9, cumulative cost = {0}, id = 7
>   HiveFilterRel(condition=[in($15, '2000Q1', '2000Q2', 
> '2000Q3')]): rowcount = 101.31622746185853, cumulative cost = {0.0 rows, 0.0 
> cpu, 0.0 io}, id = 177
> HiveTableScanRel(table=[[d2]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7957) Revisit event version handling in dynamic partition pruning on Tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7957:
-
Attachment: HIVE-7957.2.patch

review comments: future proof by making sure the event handling is thread safe.

> Revisit event version handling in dynamic partition pruning on Tez
> --
>
> Key: HIVE-7957
> URL: https://issues.apache.org/jira/browse/HIVE-7957
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-7957.1.patch, HIVE-7957.2.patch
>
>
> Once TEZ-1447 is resolved, we should be able to simplify the handing of event 
> versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7107) Fix HiveServer1 JDBC Driver spec compliancy issue

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146878#comment-14146878
 ] 

Thejas M Nair commented on HIVE-7107:
-

The test failures above are unrelated to this change.


> Fix HiveServer1 JDBC Driver spec compliancy issue
> -
>
> Key: HIVE-7107
> URL: https://issues.apache.org/jira/browse/HIVE-7107
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HIVE-7107.2.patch, HIVE-7107.2.patch
>
>
> The HiveServer1 driver does not adhere to the JDBC standard of returning null 
> when it can't handle a connection URL. It instead throws an exception which 
> leads to subsequent exceptions (from other drivers, i.e. the HiveServer2 one) 
> being swallowed.
> This is what you'd see:
> {quote}
> Error: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com 
> (state=08S01,code=0)
> java.sql.SQLException: Invalid URL: 
> jdbc:hive2://localhost:1/default;principal=hive/_h...@example.com
>   at 
> org.apache.hadoop.hive.jdbc.HiveConnection.(HiveConnection.java:86)
> {quote}
> In addition this patch cleans up the drivers a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8233) multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Chao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao updated HIVE-8233:
---
Attachment: HIVE-8233.3-spark.patch

Add these qfiles as spark-only tests.

> multi-table insertion doesn't work with ForwardOperator [Spark Branch]
> --
>
> Key: HIVE-8233
> URL: https://issues.apache.org/jira/browse/HIVE-8233
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chao
>Assignee: Chao
> Attachments: HIVE-8233.1-spark.patch, HIVE-8233.2-spark.patch, 
> HIVE-8233.3-spark.patch
>
>
> Right now, for multi-table insertion, we will start from multiple 
> FileSinkOperators, and break from their lowest common ancestor, adding 
> temporary FileSinkOperator and TableScanOperators. A special case is when the 
> LCA is a ForwardOperator, in which case we don't break it, since it's already 
> been optimized.
> However, there's a issue, considering the following plan:
> {noformat}
>   ...
>   RS_0
>|
>   FOR
>|
>  /   \
>GBY_1  GBY_2
> | |
>...   ...
> | |
>RS_1  RS_2
> | |
>...   ...
> | |
>FS_1  FS_2
> {noformat}
> which may result to:
> {noformat}
>   RW
>  /  \
>RWRW
> {noformat}
> Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
> branches will get duplicated (and same) input.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 26001: HIVE-8233 - multi-table insertion doesn't work with ForwardOperator [Spark Branch]

[Chao Sun]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26001/
---

(Updated Sept. 24, 2014, 9:04 p.m.)


Review request for hive, Brock Noland and Xuefu Zhang.


Changes
---

Made these qfiles as spark-only tests.


Bugs: hive-8233
https://issues.apache.org/jira/browse/hive-8233


Repository: hive-git


Description
---

Right now, for multi-table insertion, we will start from multiple 
FileSinkOperators, and break from their lowest common ancestor, adding 
temporary FileSinkOperator and TableScanOperators. A special case is when the 
LCA is a ForwardOperator, in which case we don't break it, since it's already 
been optimized.
However, there's a issue, considering the following plan:
  ...
  RS_0
   |
  FOR
   |
 /   \
   GBY_1  GBY_2
| |
   ...   ...
| |
   RS_1  RS_2
| |
   ...   ...
| |
   FS_1  FS_2
which may result to:
  RW
 /  \
   RWRW
Hence, because of the issue in HIVE-7731 and HIVE-8118, both downstream 
branches will get duplicated (and same) input.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 637fbc1 
  
ql/src/test/queries/clientpositive/spark_groupby7_noskew_multi_single_reducer.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_map_skew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby8_noskew.q PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_groupby9.q PRE-CREATION 
  
ql/src/test/queries/clientpositive/spark_groupby_multi_insert_common_distinct.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/spark_union17.q PRE-CREATION 
  
ql/src/test/results/clientpositive/spark/spark_groupby7_noskew_multi_single_reducer.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_map_skew.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby8_noskew.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_groupby9.q.out PRE-CREATION 
  
ql/src/test/results/clientpositive/spark/spark_groupby_multi_insert_common_distinct.q.out
 PRE-CREATION 
  ql/src/test/results/clientpositive/spark/spark_union17.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/26001/diff/


Testing
---


Thanks,

Chao Sun

[jira] [Commented] (HIVE-8021) CBO: support CTAS and insert ... select

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146892#comment-14146892
 ] 

Sergey Shelukhin commented on HIVE-8021:


woops wrong jira

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8021:
---
Attachment: HIVE-8225.inprogress.patch

small update. judging by debug logging the fixup is simply never called even 
though aggregates that are lost are of the same pattern, probably need to do it 
separately in ASTConverter after derived table stuff, I just put it there by 
accident because i had some debug logging there :)

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8021) CBO: support CTAS and insert ... select

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8021:
---
Attachment: (was: HIVE-8225.inprogress.patch)

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8225) CBO trunk merge: union11 test fails due to incorrect plan

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-8225:
---
Attachment: HIVE-8225.inprogress.patch

small update. judging by debug logging the fixup is simply never called even 
though aggregates that are lost are of the same pattern, probably need to do it 
separately in ASTConverter after derived table stuff, I just put it there by 
accident because i had some debug logging there 

> CBO trunk merge: union11 test fails due to incorrect plan
> -
>
> Key: HIVE-8225
> URL: https://issues.apache.org/jira/browse/HIVE-8225
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8225.inprogress.patch, HIVE-8225.inprogress.patch
>
>
> The result changes to as if the union didn't have count() inside. The issue 
> can be fixed by using srcunion.value outside the subquery in count (replace 
> count(1) with count(srcunion.value)). Otherwise, it looks like count(1) node 
> from union-ed queries is not present in AST at all, which might cause this 
> result.
> -Interestingly, adding group by to each query in a union produces completely 
> weird result (count(1) is 309 for each key, whereas it should be 1 and the 
> "logical" incorrect value if internal count is lost is 500)- Nm, that groups 
> by table column called key, which is weird but is what Hive does



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 26005: HiveServer2 in http-kerberos mode is restrictive on client usernames

[dilli dorai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26005/#review54455
---

Ship it!


Ship It!

- dilli dorai


On Sept. 24, 2014, 8:53 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26005/
> ---
> 
> (Updated Sept. 24, 2014, 8:53 p.m.)
> 
> 
> Review request for hive, dilli dorai and Thejas Nair.
> 
> 
> Bugs: HIVE-8246
> https://issues.apache.org/jira/browse/HIVE-8246
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-8246
> 
> 
> Diffs
> -
> 
>   service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 82093fa 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java 
> 99ef8bc 
> 
> Diff: https://reviews.apache.org/r/26005/diff/
> 
> 
> Testing
> ---
> 
> Manual testing on secure cluster.
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Comment Edited] (HIVE-8021) CBO: support CTAS and insert ... select

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146889#comment-14146889
 ] 

Sergey Shelukhin edited comment on HIVE-8021 at 9/24/14 9:12 PM:
-

-small update. judging by debug logging the fixup is simply never called even 
though aggregates that are lost are of the same pattern, probably need to do it 
separately in ASTConverter after derived table stuff, I just put it there by 
accident because i had some debug logging there :)-


was (Author: sershe):
small update. judging by debug logging the fixup is simply never called even 
though aggregates that are lost are of the same pattern, probably need to do it 
separately in ASTConverter after derived table stuff, I just put it there by 
accident because i had some debug logging there :)

> CBO: support CTAS and insert ... select
> ---
>
> Key: HIVE-8021
> URL: https://issues.apache.org/jira/browse/HIVE-8021
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-8021.01.patch, HIVE-8021.patch, 
> HIVE-8021.preliminary.patch
>
>
> Need to send only the select part to CBO for now



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8031) CBO needs to scale down NDV with selectivity to avoid underestimating

[Mostafa Mokhtar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mostafa Mokhtar updated HIVE-8031:
--
Summary: CBO needs to scale down NDV with selectivity to avoid 
underestimating   (was: CBO should use per column join selectivity not NDV when 
applying exponential backoff.)

> CBO needs to scale down NDV with selectivity to avoid underestimating 
> --
>
> Key: HIVE-8031
> URL: https://issues.apache.org/jira/browse/HIVE-8031
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Mostafa Mokhtar
>Assignee: Harish Butani
> Fix For: 0.14.0
>
>
> Currently CBO uses NDV not join selectivity in computeInnerJoinSelectivity 
> which results in in-accurate estimate number of rows.
> I looked at the plan for TPC-DS Q17 after the latest set of changes and I am 
> concerned that the estimate of rows for the join of store_sales and 
> store_returns is so low, as you can see the estimate is 8461 rows for joining 
> 1.2795706667449066E8 with 1.2922108035889767E7.
> {code}
> HiveJoinRel(condition=[AND(=($130, $3), =($129, $15))], 
> joinType=[inner]): rowcount = 1079.1345153548855, cumulative cost = 
> {8.271845957931738E10 rows, 0.0 cpu, 0.0 io}, id = 517
>   HiveJoinRel(condition=[=($0, $38)], joinType=[inner]): 
> rowcount = 6.669190301841249E7, cumulative cost = {4.300510912631623E10 rows, 
> 0.0 cpu, 0.0 io}, id = 402
> HiveTableScanRel(table=[[catalog_sales]]): rowcount = 
> 4.3005109025E10, cumulative cost = {0}, id = 2
> HiveFilterRel(condition=[in($15, '2000Q1', '2000Q2', 
> '2000Q3')]): rowcount = 101.31622746185853, cumulative cost = {0.0 rows, 0.0 
> cpu, 0.0 io}, id = 181
>   HiveTableScanRel(table=[[d3]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 3
>   HiveJoinRel(condition=[AND(AND(=($3, $61), =($2, $60)), 
> =($9, $67))], joinType=[inner]): rowcount = 8461.27236667537, cumulative cost 
> = {8.26517592150266E10 rows, 0.0 cpu, 0.0 io}, id = 515
> HiveJoinRel(condition=[=($27, $0)], joinType=[inner]): 
> rowcount = 1.2795706667449066E8, cumulative cost = {8.251088004031622E10 
> rows, 0.0 cpu, 0.0 io}, id = 417
>   HiveTableScanRel(table=[[store_sales]]): rowcount = 
> 8.2510879939E10, cumulative cost = {0}, id = 5
>   HiveFilterRel(condition=[=($15, '2000Q1')]): rowcount = 
> 101.31622746185853, cumulative cost = {0.0 rows, 0.0 cpu, 0.0 io}, id = 173
> HiveTableScanRel(table=[[d1]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 0
> HiveJoinRel(condition=[=($0, $24)], joinType=[inner]): 
> rowcount = 1.2922108035889767E7, cumulative cost = {8.332595810316228E9 rows, 
> 0.0 cpu, 0.0 io}, id = 424
>   HiveTableScanRel(table=[[store_returns]]): rowcount = 
> 8.332595709E9, cumulative cost = {0}, id = 7
>   HiveFilterRel(condition=[in($15, '2000Q1', '2000Q2', 
> '2000Q3')]): rowcount = 101.31622746185853, cumulative cost = {0.0 rows, 0.0 
> cpu, 0.0 io}, id = 177
> HiveTableScanRel(table=[[d2]]): rowcount = 73049.0, 
> cumulative cost = {0}, id = 1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8210) TezJobMonitor should print time spent in Application (RUNNING)

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-8210:
-
   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

> TezJobMonitor should print time spent in Application (RUNNING)
> --
>
> Key: HIVE-8210
> URL: https://issues.apache.org/jira/browse/HIVE-8210
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Trivial
>  Labels: UX
> Fix For: 0.14.0
>
> Attachments: HIVE-8210.1.patch
>
>
> In a fully loaded cluster, there is no easy way to distinguish between a 
> query which takes 2 minutes to run and a query which takes 2 minutes to get 
> an AM.
> Clearly log the amount of time the query spent in RUNNING state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)