[jira] [Commented] (HIVE-6070) document HIVE-6052

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853761#comment-13853761
 ] 

Hive QA commented on HIVE-6070:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619722/HIVE-6070.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/718/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/718/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619722

> document HIVE-6052
> --
>
> Key: HIVE-6070
> URL: https://issues.apache.org/jira/browse/HIVE-6070
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6070.patch
>
>
> See comments in HIVE-6052 - this is the followup jira



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853763#comment-13853763
 ] 

Hive QA commented on HIVE-3611:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12562690/HIVE-3611.1.patch.txt

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/719/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/719/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-719/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'conf/hive-default.xml.template'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit/target itests/custom-serde/target 
itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target 
hcatalog/server-extensions/target hcatalog/core/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen 
service/target contrib/target serde/target beeline/target odbc/target 
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1552519.

At revision 1552519.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12562690

> Hive JDBC doesn't support BINARY column
> ---
>
> Key: HIVE-3611
> URL: https://issues.apache.org/jira/browse/HIVE-3611
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.9.0
>Reporter: Viji
>  Labels: patch
> Attachments: HIVE-3611.1.patch.txt
>
>
> Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type 
> columns.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5176) Wincompat : Changes for allowing various path compatibilities with Windows

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853737#comment-13853737
 ] 

Hive QA commented on HIVE-5176:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619689/HIVE-5176.2.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/716/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/716/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619689

> Wincompat : Changes for allowing various path compatibilities with Windows
> --
>
> Key: HIVE-5176
> URL: https://issues.apache.org/jira/browse/HIVE-5176
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5176.2.patch, HIVE-5176.patch
>
>
> We need to make certain changes across the board to allow us to read/parse 
> windows paths. Some are escaping changes, some are being strict about how we 
> read paths (through URL.encode/decode, etc)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6066) Wincompat: describe_comment_nonascii.q failing on windows

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853709#comment-13853709
 ] 

Hive QA commented on HIVE-6066:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619677/HIVE-6066.1.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/715/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/715/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619677

> Wincompat: describe_comment_nonascii.q failing on windows
> -
>
> Key: HIVE-6066
> URL: https://issues.apache.org/jira/browse/HIVE-6066
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests, Windows
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6066.1.patch
>
>
> describe_comment_nonascii.q failing on Windows. Some strings are not being 
> properly converted to utf-8 bytes, both during formatting and when diffing 
> the results.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5558) Support alternate join syntax

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5558:


Attachment: HIVE-5558.1.patch

> Support alternate join syntax
> -
>
> Key: HIVE-5558
> URL: https://issues.apache.org/jira/browse/HIVE-5558
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5558.1.patch
>
>
> See details in HIVE-
> Allow from clause to join table sources with the `comma' token.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5558) Support alternate join syntax

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5558:


Status: Patch Available  (was: Open)

> Support alternate join syntax
> -
>
> Key: HIVE-5558
> URL: https://issues.apache.org/jira/browse/HIVE-5558
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5558.1.patch
>
>
> See details in HIVE-
> Allow from clause to join table sources with the `comma' token.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5558) Support alternate join syntax

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853670#comment-13853670
 ] 

Harish Butani commented on HIVE-5558:
-

review at: https://reviews.apache.org/r/16413/

> Support alternate join syntax
> -
>
> Key: HIVE-5558
> URL: https://issues.apache.org/jira/browse/HIVE-5558
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5558.1.patch
>
>
> See details in HIVE-
> Allow from clause to join table sources with the `comma' token.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16413: HIVE-5558: Allow from clause to join table sources with the `comma' token.

[Harish Butani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16413/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5558
https://issues.apache.org/jira/browse/HIVE-5558


Repository: hive-git


Description
---

Allow from clause to join table sources with the `comma' token.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/FromClauseParser.g 205604c 
  ql/src/test/queries/clientnegative/join_alt_syntax_comma_on.q PRE-CREATION 
  ql/src/test/queries/clientpositive/join_alt_syntax.q PRE-CREATION 
  ql/src/test/results/clientnegative/join_alt_syntax_comma_on.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/join_alt_syntax.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16413/diff/


Testing
---

added new tests


Thanks,

Harish Butani

[jira] [Commented] (HIVE-6065) orc/FileDump and testRCFile should close on appropriate resources

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853664#comment-13853664
 ] 

Hive QA commented on HIVE-6065:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619713/HIVE-6065.2.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/713/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/713/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619713

> orc/FileDump and testRCFile should close on appropriate resources
> -
>
> Key: HIVE-6065
> URL: https://issues.apache.org/jira/browse/HIVE-6065
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6065.1.patch, HIVE-6065.2.patch
>
>
> On Windows, TestRCFile/TestFileDump were unable to delete some test files 
> because something was still hanging on to them.  Looks like there are a 
> couple of places where we need to close some resources.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16412: HIVE-6048: Hive load data command rejects file with '+' in the name

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16412/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-6048
https://issues.apache.org/jira/browse/HIVE-6048


Repository: hive-git


Description
---

The issue, including its old cousin, is caused by encoding/decoding of 
URI.toString, Path.toString(), file path etc. It's found that the best approach 
is to use URI to represent a file and URI.getPath() gives the correct decoded 
file path. The fix in this patch is mostly about passing URI around so that 
acurate file info isn't lost during passing. 


Diffs
-

  data/files/person c902284 
  data/files/person+age.txt PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 38d97e3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cb492f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java fd811f3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java dbf3f91 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java 
33111e5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
e97d948 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java c2981e8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b9cd65c 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CopyWork.java de31b21 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadDesc.java bada915 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java 40adca7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java 7d555e4 
  ql/src/test/queries/clientpositive/load_file_with_space_in_the_name.q 672d5d2 
  ql/src/test/queries/clientpositive/load_hdfs_file_with_space_in_the_name.q 
d4520e2 
  ql/src/test/results/clientpositive/load_file_with_space_in_the_name.q.out 
af6fd10 
  
ql/src/test/results/clientpositive/load_hdfs_file_with_space_in_the_name.q.out 
1e7fa33 
  ql/src/test/results/compiler/plan/case_sensitivity.q.xml 27d064f 
  ql/src/test/results/compiler/plan/groupby1.q.xml 00500bb 
  ql/src/test/results/compiler/plan/input1.q.xml 28a2237 
  ql/src/test/results/compiler/plan/input2.q.xml d96bfab 
  ql/src/test/results/compiler/plan/input3.q.xml 46fe7f9 
  ql/src/test/results/compiler/plan/input4.q.xml 98e28d4 
  ql/src/test/results/compiler/plan/input5.q.xml 806c3bf 
  ql/src/test/results/compiler/plan/input6.q.xml 8b2e348 
  ql/src/test/results/compiler/plan/input7.q.xml 8ae403b 
  ql/src/test/results/compiler/plan/input9.q.xml f8a2f76 
  ql/src/test/results/compiler/plan/input_testsequencefile.q.xml d8697ff 
  ql/src/test/results/compiler/plan/join1.q.xml 9e4b609 
  ql/src/test/results/compiler/plan/join2.q.xml efcb865 
  ql/src/test/results/compiler/plan/join3.q.xml 9bbe64f 
  ql/src/test/results/compiler/plan/sample2.q.xml 568cea8 
  ql/src/test/results/compiler/plan/sample3.q.xml c23313b 
  ql/src/test/results/compiler/plan/sample4.q.xml 568cea8 
  ql/src/test/results/compiler/plan/sample5.q.xml f60cb96 
  ql/src/test/results/compiler/plan/sample6.q.xml 5bb3dbc 
  ql/src/test/results/compiler/plan/sample7.q.xml 152cc08 
  ql/src/test/results/compiler/plan/subq.q.xml 8990b76 
  ql/src/test/results/compiler/plan/union.q.xml 6cab061 

Diff: https://reviews.apache.org/r/16412/diff/


Testing
---

New unit tests are added. Some old tests have newly generated output.


Thanks,

Xuefu Zhang

[jira] [Updated] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6034:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Sergey!

> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6034.01.patch, HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6048) Hive load data command rejects file with '+' in the name

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853633#comment-13853633
 ] 

Hive QA commented on HIVE-6048:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619669/HIVE-6048.2.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/712/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/712/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619669

> Hive load data command rejects file with '+' in the name
> 
>
> Key: HIVE-6048
> URL: https://issues.apache.org/jira/browse/HIVE-6048
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-6048.1.patch, HIVE-6048.2.patch, HIVE-6048.patch
>
>
> '+' is a valid character in a file name on linux and HDFS. However, loading 
> data from such a file into table results the following error:
> {code}
> hive> load data local inpath '/home/xzhang/temp/t+est.txt' into table test;
> Copying data from file:/home/xzhang/temp/t est.txt
> No files matching path: file:/home/xzhang/temp/t est.txt
> FAILED: Execution Error, return code 3 from 
> org.apache.hadoop.hive.ql.exec.CopyTask
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15654: Rewrite Trim and Pad UDFs based on GenericUDF

[Xuefu Zhang]

> On Dec. 20, 2013, 2:37 a.m., Mohammad Islam wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java, 
> > line 47
> > 
> >
> > Even if I implement it this way won't I still have to cast arguments[0] 
> > to a PrimitiveObjectInspector in order to call getPrimitiveCategory()? If 
> > so then I don't see the advantage of doing it your way you suggest.

Well, that just gives a little clean code without try... catch... block. The 
benefit is marginal, but personally I think it's better to do explicit check 
than relying on runtime exception.


- Xuefu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15654/#review30732
---


On Dec. 18, 2013, 3:16 a.m., Mohammad Islam wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15654/
> ---
> 
> (Updated Dec. 18, 2013, 3:16 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Carl Steinbach, and Jitendra 
> Pandey.
> 
> 
> Bugs: HIVE-5829
> https://issues.apache.org/jira/browse/HIVE-5829
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Rewrite the UDFS *pads and *trim using GenericUDF.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a895d65 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> bca1f26 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLTrim.java dc00cf9 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLpad.java d1da19a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRTrim.java 2bcc5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRpad.java 9652ce2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFTrim.java 490886d 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrim.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java
>  eff251f 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFTrim.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15654/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mohammad Islam
> 
>

[jira] [Commented] (HIVE-6070) document HIVE-6052

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853632#comment-13853632
 ] 

Ashutosh Chauhan commented on HIVE-6070:


+1

> document HIVE-6052
> --
>
> Key: HIVE-6070
> URL: https://issues.apache.org/jira/browse/HIVE-6070
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6070.patch
>
>
> See comments in HIVE-6052 - this is the followup jira



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-3611) Hive JDBC doesn't support BINARY column

[Wee Hiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853625#comment-13853625
 ] 

Wee Hiong commented on HIVE-3611:
-

May I know is there a workaround while waiting for the patch to be ready?

> Hive JDBC doesn't support BINARY column
> ---
>
> Key: HIVE-3611
> URL: https://issues.apache.org/jira/browse/HIVE-3611
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.9.0
>Reporter: Viji
>  Labels: patch
> Attachments: HIVE-3611.1.patch.txt
>
>
> Similar to HIVE-2957, the JDBC HiveDriver should also support BINARY type 
> columns.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-6079) Hadoop 1 tests fail in tez branch

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6079.
--

Resolution: Fixed

Committed to branch. Thanks Vikram!

> Hadoop 1 tests fail in tez branch
> -
>
> Key: HIVE-6079
> URL: https://issues.apache.org/jira/browse/HIVE-6079
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: tez-branch
>
> Attachments: HIVE-6079.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-6081) Dag utils in tez has incorrect dependency on Hadoop20 shims

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6081.
--

   Resolution: Fixed
Fix Version/s: tez-branch

Looks good. Committed to branch. Thanks Vikram!

> Dag utils in tez has incorrect dependency on Hadoop20 shims
> ---
>
> Key: HIVE-6081
> URL: https://issues.apache.org/jira/browse/HIVE-6081
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: tez-branch
>
> Attachments: HIVE-6081.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6047) Permanent UDFs in Hive

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6047:
-

Attachment: PermanentFunctionsinHive.pdf

Attaching initial proposal

> Permanent UDFs in Hive
> --
>
> Key: HIVE-6047
> URL: https://issues.apache.org/jira/browse/HIVE-6047
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: PermanentFunctionsinHive.pdf
>
>
> Currently Hive only supports temporary UDFs which must be re-registered when 
> starting up a Hive session. Provide some support to register permanent UDFs 
> with Hive. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-6078) Choosing conditional task for merging files is not deterministic in tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6078.
--

Resolution: Fixed

Committed to branch. Thanks Vikram!

> Choosing conditional task for merging files is not deterministic in tez
> ---
>
> Key: HIVE-6078
> URL: https://issues.apache.org/jira/browse/HIVE-6078
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: tez-branch
>
> Attachments: HIVE-6078.1.patch
>
>
> In tez, the merging of files does not happen deterministically.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6081) Dag utils in tez has incorrect dependency on Hadoop20 shims

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-6081:


 Summary: Dag utils in tez has incorrect dependency on Hadoop20 
shims
 Key: HIVE-6081
 URL: https://issues.apache.org/jira/browse/HIVE-6081
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
 Attachments: HIVE-6081.1.patch





--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6079) Hadoop 1 tests fail in tez branch

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6079:
-

Fix Version/s: tez-branch

> Hadoop 1 tests fail in tez branch
> -
>
> Key: HIVE-6079
> URL: https://issues.apache.org/jira/browse/HIVE-6079
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: tez-branch
>
> Attachments: HIVE-6079.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6081) Dag utils in tez has incorrect dependency on Hadoop20 shims

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-6081:
-

Attachment: HIVE-6081.1.patch

> Dag utils in tez has incorrect dependency on Hadoop20 shims
> ---
>
> Key: HIVE-6081
> URL: https://issues.apache.org/jira/browse/HIVE-6081
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6081.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6078) Choosing conditional task for merging files is not deterministic in tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6078:
-

Fix Version/s: tez-branch

> Choosing conditional task for merging files is not deterministic in tez
> ---
>
> Key: HIVE-6078
> URL: https://issues.apache.org/jira/browse/HIVE-6078
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: tez-branch
>
> Attachments: HIVE-6078.1.patch
>
>
> In tez, the merging of files does not happen deterministically.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6080) Non-deterministic stage dependencies in tez

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-6080:


 Summary: Non-deterministic stage dependencies in tez
 Key: HIVE-6080
 URL: https://issues.apache.org/jira/browse/HIVE-6080
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K


The dependent stages were non-deterministic in tez.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6079) Hadoop 1 tests fail in tez branch

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-6079:


 Summary: Hadoop 1 tests fail in tez branch
 Key: HIVE-6079
 URL: https://issues.apache.org/jira/browse/HIVE-6079
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
 Attachments: HIVE-6079.1.patch





--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6079) Hadoop 1 tests fail in tez branch

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-6079:
-

Attachment: HIVE-6079.1.patch

> Hadoop 1 tests fail in tez branch
> -
>
> Key: HIVE-6079
> URL: https://issues.apache.org/jira/browse/HIVE-6079
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6079.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6078) Choosing conditional task for merging files is not deterministic in tez

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-6078:


 Summary: Choosing conditional task for merging files is not 
deterministic in tez
 Key: HIVE-6078
 URL: https://issues.apache.org/jira/browse/HIVE-6078
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
 Attachments: HIVE-6078.1.patch

In tez, the merging of files does not happen deterministically.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6078) Choosing conditional task for merging files is not deterministic in tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-6078:
-

Attachment: HIVE-6078.1.patch

> Choosing conditional task for merging files is not deterministic in tez
> ---
>
> Key: HIVE-6078
> URL: https://issues.apache.org/jira/browse/HIVE-6078
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6078.1.patch
>
>
> In tez, the merging of files does not happen deterministically.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-6077) Fixing a couple of orc unit tests on tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6077.
--

Resolution: Fixed

> Fixing a couple of orc unit tests on tez
> 
>
> Key: HIVE-6077
> URL: https://issues.apache.org/jira/browse/HIVE-6077
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-6077.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6077) Fixing a couple of orc unit tests on tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-6077:
-

Attachment: HIVE-6077.1.patch

> Fixing a couple of orc unit tests on tez
> 
>
> Key: HIVE-6077
> URL: https://issues.apache.org/jira/browse/HIVE-6077
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-6077.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15654: Rewrite Trim and Pad UDFs based on GenericUDF

[Mohammad Islam]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15654/#review30732
---



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java


Even if I implement it this way won't I still have to cast arguments[0] to 
a PrimitiveObjectInspector in order to call getPrimitiveCategory()? If so then 
I don't see the advantage of doing it your way you suggest.


- Mohammad Islam


On Dec. 18, 2013, 3:16 a.m., Mohammad Islam wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15654/
> ---
> 
> (Updated Dec. 18, 2013, 3:16 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Carl Steinbach, and Jitendra 
> Pandey.
> 
> 
> Bugs: HIVE-5829
> https://issues.apache.org/jira/browse/HIVE-5829
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Rewrite the UDFS *pads and *trim using GenericUDF.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a895d65 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> bca1f26 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLTrim.java dc00cf9 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLpad.java d1da19a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRTrim.java 2bcc5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRpad.java 9652ce2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFTrim.java 490886d 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrim.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java
>  eff251f 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFTrim.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15654/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mohammad Islam
> 
>

[jira] [Commented] (HIVE-6045) Beeline hivevars is broken for more than one hivevar

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853605#comment-13853605
 ] 

Hive QA commented on HIVE-6045:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619668/HIVE-6045.4.patch

{color:green}SUCCESS:{color} +1 4800 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/711/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/711/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619668

> Beeline hivevars is broken for more than one hivevar
> 
>
> Key: HIVE-6045
> URL: https://issues.apache.org/jira/browse/HIVE-6045
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.13.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-6045.1.patch, HIVE-6045.2.patch, HIVE-6045.4.patch, 
> HIVE-6045.patch
>
>
> HIVE-4568 introduced --hivevar flag.  But if you specify more than one 
> hivevar, for example 
> {code}
> beeline --hivevar file1=/user/szehon/file1 --hivevar file2=/user/szehon/file2
> {code}
> then the variables during runtime get mangled to evaluate to:
> {code}
> file1=/user/szehon/file1&file2=/user/szehon/file2
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6077) Fixing a couple of orc unit tests on tez

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-6077:


 Summary: Fixing a couple of orc unit tests on tez
 Key: HIVE-6077
 URL: https://issues.apache.org/jira/browse/HIVE-6077
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: tez-branch






--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-6055) Cleanup aisle tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-6055.
--

Resolution: Fixed

Committed to branch.

> Cleanup aisle tez
> -
>
> Key: HIVE-6055
> URL: https://issues.apache.org/jira/browse/HIVE-6055
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-6055.1.patch
>
>
> Some of the past merges have led to some dead code. Need to remove this from 
> the tez branch.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5891) Alias conflict when merging multiple mapjoin tasks into their common child mapred task

[Yin Huai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853583#comment-13853583
 ] 

Yin Huai commented on HIVE-5891:


i see. yes, seems getMapJoinContext and getSmbMapJoinContext can also have 
QBJoinTrees. I think it will be good to show meaningful aliases for those 
intermediate results. So, users can know where does an intermediate result come 
from. Since it is not easy to get the correct QB.id, I prefer to use 
QBJoinTree.id right now. Once this bug has been fixed, we can work on a 
followup jira to get rid of INTNAME. Also, I guess that we do not have an unit 
test to cover this bug. Can you add an test query in multiMapJoin2.q and 
comment the reason that we need this test? Thanks.

> Alias conflict when merging multiple mapjoin tasks into their common child 
> mapred task
> --
>
> Key: HIVE-5891
> URL: https://issues.apache.org/jira/browse/HIVE-5891
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Sun Rui
>Assignee: Sun Rui
> Attachments: HIVE-5891.1.patch
>
>
> Use the following test case with HIVE 0.12:
> {code:sql}
> create table src(key int, value string);
> load data local inpath 'src/data/files/kv1.txt' overwrite into table src;
> select * from (
>   select c.key from
> (select a.key from src a join src b on a.key=b.key group by a.key) tmp
> join src c on tmp.key=c.key
>   union all
>   select c.key from
> (select a.key from src a join src b on a.key=b.key group by a.key) tmp
> join src c on tmp.key=c.key
> ) x;
> {code}
> We will get a NullPointerException from Union Operator:
> {noformat}
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row {"_col0":0}
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row {"_col0":0}
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
>   ... 4 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.ql.exec.UnionOperator.processOp(UnionOperator.java:120)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:652)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:655)
>   at 
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
>   at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:220)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   ... 5 more
> {noformat}
>   
> The root cause is in 
> CommonJoinTaskDispatcher.mergeMapJoinTaskIntoItsChildMapRedTask().
> {noformat}
>   +--+  +--+
>   | MapJoin task |  | MapJoin task |
>   +--+  +--+
>  \ /
>   \   /
>  +--+
>  |  Union task  |
>  +--+
> {noformat} 
> CommonJoinTaskDispatcher merges the two MapJoin tasks into their common 
> child: Union task. The two MapJoin tasks have the same alias name for their 
> big tables: $INTNAME, which is the name of the temporary table of a join 
> stream. The aliasToWork map uses alias as key, so eventually only the MapJoin 
> operator tree of one MapJoin task 

Re: Hive CBO - Branch Request

[Brock Noland]

Hi,

Do you have an Apache ID? (I don't see you here
http://people.apache.org/committer-index.html). Without an Apache ID I am
not sure how we'd give you access to commit the branch.

More importantly I don't think we have any provision for "branch committer"
in the Hive ByLaws (https://cwiki.apache.org/confluence/display/Hive/Bylaws)
or really any provisions for branches at all. We have talked about adding a
branch merge provision but that has not occurred at present.

As a side note, Hadoop did recently change their bylaws to include the
concept of a "branch committer".
http://s.apache.org/hadoop-branch-committers

Brock


On Thu, Dec 19, 2013 at 6:19 PM, John Pullokkaran <
jpullokka...@hortonworks.com> wrote:

> Hi,
>
>   I am working on CBO for Hive
> (HIVE-5775
> ).
>
> In order to make code integration easier i would like to do this work on a
> separate branch which can be brought into trunk once code is stable and
> reviewed.
>
> It would also be easier if i could commit in to this branch without having
> to wait for a committer.
>
>
> Thanks
> John
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

[jira] [Commented] (HIVE-6045) Beeline hivevars is broken for more than one hivevar

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853577#comment-13853577
 ] 

Xuefu Zhang commented on HIVE-6045:
---

+1, pending test result.

> Beeline hivevars is broken for more than one hivevar
> 
>
> Key: HIVE-6045
> URL: https://issues.apache.org/jira/browse/HIVE-6045
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.13.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-6045.1.patch, HIVE-6045.2.patch, HIVE-6045.4.patch, 
> HIVE-6045.patch
>
>
> HIVE-4568 introduced --hivevar flag.  But if you specify more than one 
> hivevar, for example 
> {code}
> beeline --hivevar file1=/user/szehon/file1 --hivevar file2=/user/szehon/file2
> {code}
> then the variables during runtime get mangled to evaluate to:
> {code}
> file1=/user/szehon/file1&file2=/user/szehon/file2
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6076) TestStreaming_2 fails because it gets runState 2 (success) instead of runState 3 (failure)

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6076:


Issue Type: Bug  (was: Task)

> TestStreaming_2 fails because it gets runState 2 (success) instead of 
> runState 3 (failure)
> --
>
> Key: HIVE-6076
> URL: https://issues.apache.org/jira/browse/HIVE-6076
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Reporter: Hari Sankar Sivarama Subramaniyan
>
> The templeton controller job should always succeed. The test should be 
> modified to always return a success.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6076) TestStreaming_2 fails because it gets runState 2 (success) instead of runState 3 (failure)

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6076:
---

 Summary: TestStreaming_2 fails because it gets runState 2 
(success) instead of runState 3 (failure)
 Key: HIVE-6076
 URL: https://issues.apache.org/jira/browse/HIVE-6076
 Project: Hive
  Issue Type: Task
  Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan


The templeton controller job should always succeed. The test should be modified 
to always return a success.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6074) Create log4j.properties template for windows build

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6074:
---

 Summary: Create log4j.properties template for windows build
 Key: HIVE-6074
 URL: https://issues.apache.org/jira/browse/HIVE-6074
 Project: Hive
  Issue Type: Task
Reporter: Hari Sankar Sivarama Subramaniyan


Inorder to make Hive work in windows log4j properties file should be compatible 
with the windows naming system. i.e. blackslashes should be supported(without 
interpreting these characters as escape sequences).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6034) vectorized % doesn't handle zeroes the same way as non-vectorized

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853568#comment-13853568
 ] 

Sergey Shelukhin commented on HIVE-6034:


can this be committed?

> vectorized % doesn't handle zeroes the same way as non-vectorized
> -
>
> Key: HIVE-6034
> URL: https://issues.apache.org/jira/browse/HIVE-6034
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-6034.01.patch, HIVE-6034.patch
>
>
> % 0 is NULL, but if vectorized it's NaN



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6075) Remove hive.metastore.warehouse.dir from webhcat config

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6075:
---

 Summary:  Remove hive.metastore.warehouse.dir from webhcat config 
 Key: HIVE-6075
 URL: https://issues.apache.org/jira/browse/HIVE-6075
 Project: Hive
  Issue Type: Task
  Components: WebHCat
Reporter: Hari Sankar Sivarama Subramaniyan






--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853564#comment-13853564
 ] 

Sergey Shelukhin commented on HIVE-6052:


Added release note, added HIVE-6070 with somewhat different text (due to 
context) 

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6073) Local Map Reduce task will fail for Windows OS

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6073:
---

 Summary: Local Map Reduce task will fail for Windows OS
 Key: HIVE-6073
 URL: https://issues.apache.org/jira/browse/HIVE-6073
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan


Local Mapreduce task will be spawned a child jvm where HADOOP_CLASSPATH 
variable will not be exported if Hive is started as a windows service. The fix 
is to introduce a new hive parameter which should be passed as part of 
hiveserver script. This new parameter should be parsed in MapRedLocalTask and 
added to the system variables.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6073) MapJoin will fail for Windows OS

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6073:


Summary: MapJoin will fail for Windows OS  (was: Local Map Reduce task will 
fail for Windows OS)

> MapJoin will fail for Windows OS
> 
>
> Key: HIVE-6073
> URL: https://issues.apache.org/jira/browse/HIVE-6073
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>
> Local Mapreduce task will be spawned a child jvm where HADOOP_CLASSPATH 
> variable will not be exported if Hive is started as a windows service. The 
> fix is to introduce a new hive parameter which should be passed as part of 
> hiveserver script. This new parameter should be parsed in MapRedLocalTask and 
> added to the system variables.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6070) document HIVE-6052

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6070:
---

Status: Patch Available  (was: Open)

> document HIVE-6052
> --
>
> Key: HIVE-6070
> URL: https://issues.apache.org/jira/browse/HIVE-6070
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6070.patch
>
>
> See comments in HIVE-6052 - this is the followup jira



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6070) document HIVE-6052

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6070:
---

Attachment: HIVE-6070.patch

tiny doc patch

> document HIVE-6052
> --
>
> Key: HIVE-6070
> URL: https://issues.apache.org/jira/browse/HIVE-6070
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6070.patch
>
>
> See comments in HIVE-6052 - this is the followup jira



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6052:
---

Release Note: JDO pushdown for integers in metastore didn't work correctly 
for some legacy data in partition columns in Hive 0.12. In 0.13, 
hive.metastore.integral.jdo.pushdown setting is added to enable it, and it's 
disabled by default. Enabling it improves metastore perf for integral columns, 
especially if there's a large number of partitions. However, it doesn't work 
correctly with integral values that are not normalized (e.g. have leading 
zeroes, like 0012). If metastore direct SQL is enabled and works, this 
optimization is also irrelevant.  (was: JDO pushdown for integers in metastore 
didn't work correctly in Hive 0.12. In 0.13, 
hive.metastore.integral.jdo.pushdown setting is added to enable it, and it's 
disabled by default. Enabling it improves metastore perf for integral columns, 
especially if there's a large number of partitions. However, it doesn't work 
correctly with integral values that are not normalized (e.g. have leading 
zeroes, like 0012). If metastore direct SQL is enabled and works, this 
optimization is also irrelevant.)

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6052:
---

Release Note: JDO pushdown for integers in metastore didn't work correctly 
in Hive 0.12. In 0.13, hive.metastore.integral.jdo.pushdown setting is added to 
enable it, and it's disabled by default. Enabling it improves metastore perf 
for integral columns, especially if there's a large number of partitions. 
However, it doesn't work correctly with integral values that are not normalized 
(e.g. have leading zeroes, like 0012). If metastore direct SQL is enabled and 
works, this optimization is also irrelevant.

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6072) With HCatalog refactoring, Hadoop_HBase e2e will fail

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6072:
---

 Summary: With HCatalog refactoring, Hadoop_HBase e2e will fail
 Key: HIVE-6072
 URL: https://issues.apache.org/jira/browse/HIVE-6072
 Project: Hive
  Issue Type: Bug
  Components: HBase Handler, HCatalog
Reporter: Hari Sankar Sivarama Subramaniyan






--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853559#comment-13853559
 ] 

Lefty Leverenz commented on HIVE-6052:
--

When you write it up, I'll also add hive.metastore.integral.jdo.pushdown to the 
wiki.  (Just wanted to name the config variable here, for future searches.)

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6070) document HIVE-6052

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6070:
--

 Summary: document HIVE-6052
 Key: HIVE-6070
 URL: https://issues.apache.org/jira/browse/HIVE-6070
 Project: Hive
  Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Trivial


See comments in HIVE-6052 - this is the followup jira



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6071) Create hive command script for windows, hive.cmd

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6071:
---

 Summary: Create hive command script for windows, hive.cmd
 Key: HIVE-6071
 URL: https://issues.apache.org/jira/browse/HIVE-6071
 Project: Hive
  Issue Type: Task
Reporter: Hari Sankar Sivarama Subramaniyan






--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853555#comment-13853555
 ] 

Ashutosh Chauhan commented on HIVE-6052:


Also add same info on config variable in release notes field of jira as Xuefu 
suggested.

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853554#comment-13853554
 ] 

Ashutosh Chauhan commented on HIVE-6052:


Bug itself is I think sufficiently documented in description. I think there is 
need to document new config variable. Its off by default. We should document in 
what cases its safe for user to turn on this optimization. That usually goes in 
{{hive-default.xml.template}} Lets follow-up with a jira for that.

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6067) Implement vectorized decimal column-scalar comparison filters

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6067:
--

Attachment: HIVE-6067.02.patch

remove trailing white space

> Implement vectorized decimal column-scalar comparison filters
> -
>
> Key: HIVE-6067
> URL: https://issues.apache.org/jira/browse/HIVE-6067
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6067.01.patch, HIVE-6067.02.patch
>
>
> Using the new DecimalColumnVector type, implement a template to generate 
> VectorExpression subclasses for Decimal comparison filters (<, <=, >, >=, =, 
> !=).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5679) add date support to metastore JDO/SQL

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853552#comment-13853552
 ] 

Sergey Shelukhin commented on HIVE-5679:


[~ashutoshc] [~hagleitn] do you want to review? 

> add date support to metastore JDO/SQL
> -
>
> Key: HIVE-5679
> URL: https://issues.apache.org/jira/browse/HIVE-5679
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-5679.01.patch, HIVE-5679.patch
>
>
> Metastore supports strings and integral types in filters.
> It could also support dates.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6067) Implement vectorized decimal column-scalar comparison filters

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853548#comment-13853548
 ] 

Eric Hanson commented on HIVE-6067:
---

Review at https://reviews.apache.org/r/16410/

> Implement vectorized decimal column-scalar comparison filters
> -
>
> Key: HIVE-6067
> URL: https://issues.apache.org/jira/browse/HIVE-6067
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6067.01.patch
>
>
> Using the new DecimalColumnVector type, implement a template to generate 
> VectorExpression subclasses for Decimal comparison filters (<, <=, >, >=, =, 
> !=).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5774) INSERT OVERWRITE DYNAMIC PARTITION on LARGE DATA

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-5774:


Assignee: (was: Szehon Ho)

> INSERT OVERWRITE DYNAMIC PARTITION on LARGE DATA
> 
>
> Key: HIVE-5774
> URL: https://issues.apache.org/jira/browse/HIVE-5774
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
> Environment: debian 6.0.7
>Reporter: Danny Teok
>Priority: Critical
>  Labels: dynamic, hive, insert, overwrite, partition
>
> After several forensic analysis, we are convinced that there is a bug when 
> rebuilding using dynamic partition over more than 30 days. Row counts do not 
> match.
> In details:
> Part A -- original_table
> 2013-01-01; 394,755 rows
> 2013-01-02; 424,448
> 2013-01-03; 427,201
> ...
> 2013-10-30; 3,234,472
> Part B -- copy_of_original_table_new
> 2013-01-01; 372,628 rows
> 2013-01-02; 400,553
> 2013-01-03; 403,495
> ...
> 2013-10-30; 2,865,877
> The query that is used to populate the original table is the same for 
> populating the "copy_of_original_table_new" table. When we rebuilt for 1 day, 
> e.g. 2013-01-01, the number of row counts of the copy_of_original_table_new  
> matched up exactly with orignal_table.
> When we rebuilt for 7 days, the number of row counts matched up exactly.
> When we rebuilt for 15 days, the number of row counts matched up exactly.
> When we rebuilt for 303 days (10 months), everything fxxked up. No matches.
> When we rebuilt for 35 days, 80% matched up exactly. The other 20% are out 
> from hundreds to tens of thousands of rows (a variance of up to 3%)
> In other words, the more days that are specified in the WHERE dt BETWEEN 
> dateStart AND dateEnd, the dates will be out, i.e. no matching row count with 
> original_table.
> However, of those 20% that are out, we rebuilt each of them statically with 
> the corresponding date. The result is astonishingly surprising -- they 
> matched the original_table row count!
> Apologize in advance if this is not technical enough, but I hope the message 
> is clear. We believe there is a bug. Not sure how to check our Hive version, 
> but our Hadoop's version is "Hadoop 2.0.0-cdh4.1.1"
> For a glimpse of the INSERT OVERWRITE sql, it's here -- 
> http://pastebin.com/g1qxsUm2



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Assigned] (HIVE-5774) INSERT OVERWRITE DYNAMIC PARTITION on LARGE DATA

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho reassigned HIVE-5774:
---

Assignee: Szehon Ho

> INSERT OVERWRITE DYNAMIC PARTITION on LARGE DATA
> 
>
> Key: HIVE-5774
> URL: https://issues.apache.org/jira/browse/HIVE-5774
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
> Environment: debian 6.0.7
>Reporter: Danny Teok
>Assignee: Szehon Ho
>Priority: Critical
>  Labels: dynamic, hive, insert, overwrite, partition
>
> After several forensic analysis, we are convinced that there is a bug when 
> rebuilding using dynamic partition over more than 30 days. Row counts do not 
> match.
> In details:
> Part A -- original_table
> 2013-01-01; 394,755 rows
> 2013-01-02; 424,448
> 2013-01-03; 427,201
> ...
> 2013-10-30; 3,234,472
> Part B -- copy_of_original_table_new
> 2013-01-01; 372,628 rows
> 2013-01-02; 400,553
> 2013-01-03; 403,495
> ...
> 2013-10-30; 2,865,877
> The query that is used to populate the original table is the same for 
> populating the "copy_of_original_table_new" table. When we rebuilt for 1 day, 
> e.g. 2013-01-01, the number of row counts of the copy_of_original_table_new  
> matched up exactly with orignal_table.
> When we rebuilt for 7 days, the number of row counts matched up exactly.
> When we rebuilt for 15 days, the number of row counts matched up exactly.
> When we rebuilt for 303 days (10 months), everything fxxked up. No matches.
> When we rebuilt for 35 days, 80% matched up exactly. The other 20% are out 
> from hundreds to tens of thousands of rows (a variance of up to 3%)
> In other words, the more days that are specified in the WHERE dt BETWEEN 
> dateStart AND dateEnd, the dates will be out, i.e. no matching row count with 
> original_table.
> However, of those 20% that are out, we rebuilt each of them statically with 
> the corresponding date. The result is astonishingly surprising -- they 
> matched the original_table row count!
> Apologize in advance if this is not technical enough, but I hope the message 
> is clear. We believe there is a bug. Not sure how to check our Hive version, 
> but our Hadoop's version is "Hadoop 2.0.0-cdh4.1.1"
> For a glimpse of the INSERT OVERWRITE sql, it's here -- 
> http://pastebin.com/g1qxsUm2



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6065) orc/FileDump and testRCFile should close on appropriate resources

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853543#comment-13853543
 ] 

Ashutosh Chauhan commented on HIVE-6065:


No worries. Latest patch looks good.

> orc/FileDump and testRCFile should close on appropriate resources
> -
>
> Key: HIVE-6065
> URL: https://issues.apache.org/jira/browse/HIVE-6065
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6065.1.patch, HIVE-6065.2.patch
>
>
> On Windows, TestRCFile/TestFileDump were unable to delete some test files 
> because something was still hanging on to them.  Looks like there are a 
> couple of places where we need to close some resources.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853540#comment-13853540
 ] 

Sergey Shelukhin commented on HIVE-6052:


You mean documentation of the bug? From user perspective the feature itself is 
not visible. 

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6065) orc/FileDump and testRCFile should close on appropriate resources

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6065:
-

Attachment: HIVE-6065.2.patch

Mind if I sneak in a couple other file deletion-related changes needed for 
Windows tests?

> orc/FileDump and testRCFile should close on appropriate resources
> -
>
> Key: HIVE-6065
> URL: https://issues.apache.org/jira/browse/HIVE-6065
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6065.1.patch, HIVE-6065.2.patch
>
>
> On Windows, TestRCFile/TestFileDump were unable to delete some test files 
> because something was still hanging on to them.  Looks like there are a 
> couple of places where we need to close some resources.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853532#comment-13853532
 ] 

Ashutosh Chauhan commented on HIVE-6052:


Agreed. [~sershe] Can you provide documentation for this change?

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6067) Implement vectorized decimal column-scalar comparison filters

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-6067:
--

Attachment: HIVE-6067.01.patch

Completed template, code gen changes, and unit test.

> Implement vectorized decimal column-scalar comparison filters
> -
>
> Key: HIVE-6067
> URL: https://issues.apache.org/jira/browse/HIVE-6067
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-6067.01.patch
>
>
> Using the new DecimalColumnVector type, implement a template to generate 
> VectorExpression subclasses for Decimal comparison filters (<, <=, >, >=, =, 
> !=).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-hadoop2 - Build # 616 - Still Failing

[Apache Jenkins Server]

Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #582
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #583
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #584
[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)

[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #585
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)


Changes for Build #586
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #587
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shelukhin via Ashutosh Chauhan)


Changes for Build #588

Changes for Build #589

Changes for Build #590
[brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by 
Prasad)


Changes for Build #591
[xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate 
precision/scale for decimal types (reviewed by Sergey Shelukhin)

[hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath 
Pandey via Ashutosh Chauhan)

[hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery 
contains nulls in matching column (Harish Butani via Ashutosh Chauhan)

[hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands 
(Navis via Ashutosh Chauhan)


Changes for Build #592
[hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in 
FileSink (Ashutosh Chauhan via Thejas Nair)

[navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J 
via Navis)

[brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs 
(Brock Noland reviewed by Navis)


Changes for Build #593
[omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between 
non-SARGable predicates cause a NPE. (omalley)


Changes for Build #594
[gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther 
Hagleitner

[gunther] HIVE-2093: UPDATE - add two missing files from previous commit 
(Gunther Hagleitner)

[thejas] HIVE-2093 : create/drop database should populate inputs/outputs and 
check concurrency and user permission (Navis via Thejas Nair)

[hashutosh] HIVE-6016 : Hadoop23Shims has a bug in listLocatedStatus impl. 
(Pra

[jira] [Commented] (HIVE-6064) Wincompat: windows path substitutions overridden by MiniMrShim.getConfiguration() on hadoop-2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853530#comment-13853530
 ] 

Ashutosh Chauhan commented on HIVE-6064:


+1

> Wincompat: windows path substitutions overridden by 
> MiniMrShim.getConfiguration() on hadoop-2
> -
>
> Key: HIVE-6064
> URL: https://issues.apache.org/jira/browse/HIVE-6064
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests, Windows
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6064.1.patch
>
>
> On Windows, HiveConf setting hive.exec.scratchdir is changed to remove the 
> drive letter (i.e. "c:") from the start of its path. However, in 
> HadoopShims23, MiniMrShim.setupConfiguration() subsequently overwrites the 
> HiveConf settings and the drive letter is added back to hive.exec.scratchdir, 
> causing path issues in the MiniMR tests.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6064) Wincompat: windows path substitutions overridden by MiniMrShim.getConfiguration() on hadoop-2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853526#comment-13853526
 ] 

Hive QA commented on HIVE-6064:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619660/HIVE-6064.1.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/710/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/710/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619660

> Wincompat: windows path substitutions overridden by 
> MiniMrShim.getConfiguration() on hadoop-2
> -
>
> Key: HIVE-6064
> URL: https://issues.apache.org/jira/browse/HIVE-6064
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests, Windows
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6064.1.patch
>
>
> On Windows, HiveConf setting hive.exec.scratchdir is changed to remove the 
> drive letter (i.e. "c:") from the start of its path. However, in 
> HadoopShims23, MiniMrShim.setupConfiguration() subsequently overwrites the 
> HiveConf settings and the drive letter is added back to hive.exec.scratchdir, 
> causing path issues in the MiniMR tests.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6065) orc/FileDump and testRCFile should close on appropriate resources

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853525#comment-13853525
 ] 

Ashutosh Chauhan commented on HIVE-6065:


+1

> orc/FileDump and testRCFile should close on appropriate resources
> -
>
> Key: HIVE-6065
> URL: https://issues.apache.org/jira/browse/HIVE-6065
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6065.1.patch
>
>
> On Windows, TestRCFile/TestFileDump were unable to delete some test files 
> because something was still hanging on to them.  Looks like there are a 
> couple of places where we need to close some resources.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6060) Define API for RecordUpdater and UpdateReader

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-6060:


Attachment: h-5317.patch

Rowids are not unique across buckets and thus the unique identifier is thus: 
(transaction id, bucket id, row id). Alan suggested offline that I add bucket 
id to API so that we aren't forced to maintain the current restriction of one 
HDFS file per a bucket. I've also added my thoughts on what the reader would 
look like.

I also need to look at what the API looks like for vectorization.

> Define API for RecordUpdater and UpdateReader
> -
>
> Key: HIVE-6060
> URL: https://issues.apache.org/jira/browse/HIVE-6060
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: h-5317.patch, h-5317.patch
>
>
> We need to define some new APIs for how Hive interacts with the file formats 
> since it needs to be much richer than the current RecordReader and 
> RecordWriter.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6057) Enable bucketed sorted merge joins of arbitrary subqueries

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6057:
--

Description: 
Currently, you cannot use bucketed SMJ when joining subquery results. It would 
make sense to be able to explicitly specify bucketing of the intermediate 
output from a subquery to enable bucketed SMJ.

For example, the following query will NOT use bucketed SMJ:
(gameends and dummymapping are clustered and sorted by hashid into 128 buckets)
{code}
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid) e join dummymapping m on e.hashid=m.hashid
{code}
Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
bucketed SMJ:
{code}
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m on 
e.hashid=m.hashid
{code}

  was:
Currently, you cannot use bucketed SMJ when joining subquery results. It would 
make sense to be able to explicitly specify bucketing of the intermediate 
output from a subquery to enable bucketed SMJ.

For example, the following query will NOT use bucketed SMJ:
(gameends and dummymapping are clustered and sorted by hashid into 128 buckets)
{code}
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid) e join dummymapping m on e.hashid=m.hashid

Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
bucketed SMJ:
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m on 
e.hashid=m.hashid
{code}


> Enable bucketed sorted merge joins of arbitrary subqueries
> --
>
> Key: HIVE-6057
> URL: https://issues.apache.org/jira/browse/HIVE-6057
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Jan-Erik Hedbom
>Priority: Minor
>
> Currently, you cannot use bucketed SMJ when joining subquery results. It 
> would make sense to be able to explicitly specify bucketing of the 
> intermediate output from a subquery to enable bucketed SMJ.
> For example, the following query will NOT use bucketed SMJ:
> (gameends and dummymapping are clustered and sorted by hashid into 128 
> buckets)
> {code}
> select * from (select hashid,count(*) as c from gameends group by hashid 
> distribute by hashid sort by hashid) e join dummymapping m on 
> e.hashid=m.hashid
> {code}
> Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
> bucketed SMJ:
> {code}
> select * from (select hashid,count(*) as c from gameends group by hashid 
> distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m 
> on e.hashid=m.hashid
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6057) Enable bucketed sorted merge joins of arbitrary subqueries

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6057:
--

Description: 
Currently, you cannot use bucketed SMJ when joining subquery results. It would 
make sense to be able to explicitly specify bucketing of the intermediate 
output from a subquery to enable bucketed SMJ.

For example, the following query will NOT use bucketed SMJ:
(gameends and dummymapping are clustered and sorted by hashid into 128 buckets)
{code}
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid) e join dummymapping m on e.hashid=m.hashid

Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
bucketed SMJ:
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m on 
e.hashid=m.hashid
{code}

  was:
Currently, you cannot use bucketed SMJ when joining subquery results. It would 
make sense to be able to explicitly specify bucketing of the intermediate 
output from a subquery to enable bucketed SMJ.

For example, the following query will NOT use bucketed SMJ:
(gameends and dummymapping are clustered and sorted by hashid into 128 buckets)

select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid) e join dummymapping m on e.hashid=m.hashid

Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
bucketed SMJ:
select * from (select hashid,count(*) as c from gameends group by hashid 
distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m on 
e.hashid=m.hashid


> Enable bucketed sorted merge joins of arbitrary subqueries
> --
>
> Key: HIVE-6057
> URL: https://issues.apache.org/jira/browse/HIVE-6057
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Jan-Erik Hedbom
>Priority: Minor
>
> Currently, you cannot use bucketed SMJ when joining subquery results. It 
> would make sense to be able to explicitly specify bucketing of the 
> intermediate output from a subquery to enable bucketed SMJ.
> For example, the following query will NOT use bucketed SMJ:
> (gameends and dummymapping are clustered and sorted by hashid into 128 
> buckets)
> {code}
> select * from (select hashid,count(*) as c from gameends group by hashid 
> distribute by hashid sort by hashid) e join dummymapping m on 
> e.hashid=m.hashid
> Suggestion: Implement an INTO n BUCKETS syntax for subqueries to enable 
> bucketed SMJ:
> select * from (select hashid,count(*) as c from gameends group by hashid 
> distribute by hashid sort by hashid INTO 128 BUCKETS) e join dummymapping m 
> on e.hashid=m.hashid
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6056) The AvroSerDe gives out BadSchemaException if a partition is added to the table

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6056:
--

Description: 
While creating an external table if I do not add a partition, I am able to read 
files using following format: 
{code}
CREATE external TABLE event
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/2013/14/10'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');
{code}
but if I add a partition based on date
{code}
CREATE external TABLE event
PARTITIONED BY (ds STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');

ALTER TABLE event ADD IF NOT EXISTS PARTITION (ds = '2013_12_16') LOCATION 
'2013/12/16/';
{code}
I get the following exception:
{code}
java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
{code}

  was:
While creating an external table if I do not add a partition, I am able to read 
files using following format: 

CREATE external TABLE event
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/2013/14/10'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');

but if I add a partition based on date

CREATE external TABLE event
PARTITIONED BY (ds STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 's3n://test-event/input/'
TBLPROPERTIES ('avro.schema.literal' = '..some schema..');

ALTER TABLE event ADD IF NOT EXISTS PARTITION (ds = '2013_12_16') LOCATION 
'2013/12/16/';

I get the following exception:
java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException


> The AvroSerDe gives out BadSchemaException if a partition is added to the 
> table
> ---
>
> Key: HIVE-6056
> URL: https://issues.apache.org/jira/browse/HIVE-6056
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 0.11.0
> Environment: amazon EMR (hadoop Amazon 1.0.3), avro-1.7.5
>Reporter: Rushil Gupta
>
> While creating an external table if I do not add a partition, I am able to 
> read files using following format: 
> {code}
> CREATE external TABLE event
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT 
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 's3n://test-event/input/2013/14/10'
> TBLPROPERTIES ('avro.schema.literal' = '..some schema..');
> {code}
> but if I add a partition based on date
> {code}
> CREATE external TABLE event
> PARTITIONED BY (ds STRING)
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT 
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 's3n://test-event/input/'
> TBLPROPERTIES ('avro.schema.literal' = '..some schema..');
> ALTER TABLE event ADD IF NOT EXISTS PARTITION (ds = '2013_12_16') LOCATION 
> '2013/12/16/';
> {code}
> I get the following exception:
> {code}
> java.io.IOException:org.apache.hadoop.hive.serde2.avro.BadSchemaException
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-h0.21 - Build # 2517 - Still Failing

[Apache Jenkins Server]

Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayachandran via Harish Butani)


Changes for Build #2484
[brock] HIVE-5880 - (Rename HCatalog HBase Storage Handler artifact id) breaks 
packaging (Xuefu Zhang via Brock Noland)


Changes for Build #2485
[xuefu] HIVE-5866: Hive divide operator generates wrong results in certain 
cases (reviewed by Prasad)

[ehans] HIVE-5877: Implement vectorized support for IN as boolean-valued 
expression (Eric Hanson)


Changes for Build #2486
[ehans] HIVE-5895: vectorization handles division by zero differently from 
normal execution (Sergey Shelukhin via Eric Hanson)

[hashutosh] HIVE-5938 : Remove apache.mina dependency for test (Navis via 
Ashutosh Chauhan)

[xuefu] HIVE-5912: Show partition command doesn't support db.table (Yu Zhao via 
Xuefu)

[brock] HIVE-5906 - TestGenericUDFPower should use delta to compare doubles 
(Szehon Ho via Brock Noland)

[brock] HIVE-5855 - Add deprecated methods back to ColumnProjectionUtils (Brock 
Noland reviewed by Navis)

[brock] HIVE-5915 - Shade Kryo dependency (Brock Noland reviewed by Ashutosh 
Chauhan)


Changes for Build #2487
[hashutosh] HIVE-5916 : No need to aggregate statistics collected via counter 
mechanism (Ashutosh Chauhan via Navis)

[xuefu] HIVE-5947: Fix test failure in decimal_udf.q (reviewed by Brock)

[thejas] HIVE-5550 : Import fails for tables created with default text, 
sequence and orc file formats using HCatalog API (Sushanth Sowmyan via Thejas 
Nair)


Changes for Build #2488
[hashutosh] HIVE-5935 : hive.query.string is not provided to FetchTask (Navis 
via Ashutosh Chauhan)

[navis] HIVE-3455 : ANSI CORR(X,Y) is incorrect (Maxim Bolotin via Navis)

[hashutosh] HIVE-5921 : Better heuristics for worst case statistics estimates 
for join, limit and filter operator (Prasanth J via Harish Butani)

[rhbutani] HIVE-5899 NPE during explain extended with char/varchar columns 
(Jason Dere via Harish Butani)


Changes for Build #2489
[xuefu] HIVE-3181: getDatabaseMajor/Minor version does not return values 
(Szehon via Xuefu, reviewed by Navis)

[brock] HIVE-5641 - BeeLineOpts ignores Throwable (Brock Noland reviewed by 
Prasad and Thejas)

[hashutosh] HIVE-5909 : locate and instr throw 
java.nio.BufferUnderflowException when empty string as substring (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5686 : partition column type validation doesn't quite work for 
dates (Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5887 : metastore direct sql doesn't work with oracle (Sergey 
Shelukhin via Ashutosh Chauhan)


Changes for Build #2490

Changes for Build #2491

Changes for Build #2492
[brock] HIVE-5981 - Add hive-unit back to itests pom (Brock Noland reviewed by 
Prasad)


Changes for Build #2493
[xuefu] HIVE-5872: Make UDAFs such as GenericUDAFSum report accurate 
precision/scale for decimal types (reviewed by Sergey Shelukhin)

[hashutosh] HIVE-5978 : Rollups not supported in vector mode. (Jitendra Nath 
Pandey via Ashutosh Chauhan)

[hashutosh] HIVE-5830 : SubQuery: Not In subqueries should check if subquery 
contains nulls in matching column (Harish Butani via Ashutosh Chauhan)

[hashutosh] HIVE-5598 : Remove dummy new line at the end of non-sql commands 
(Navis via Ashutosh Chauhan)


Changes for Build #2494
[hashutosh] HIVE-5982 : Remove redundant filesystem operations and methods in 
FileSink (Ashutosh Chauhan via Thejas Nair)

[navis] HIVE-5955 : decimal_precision.q test case fails in trunk (Prasanth J 
via Navis)

[brock] HIVE-5983 - Fix name of ColumnProjectionUtils.appendReadColumnIDs 
(Brock Noland reviewed by Navis)


Changes for Build #2495
[omalley] HIVE-5580. Predicate pushdown predicates with an and-operator between 
non-SARGable predicates cause a NPE. (omalley)


Changes for Build #2496
[gunther] HIVE-6000: Hive build broken on hadoop2 (Vikram Dixit K via Gunther 
Hagleitner

[gunther] HIVE-2093: UPDATE - add two missing files from previous commit 
(Gunther Hagleitner)

[thejas] HIVE-2093 : create/drop database should populate inputs/outputs and 
check concurrency and user permission (Navis via Thejas Nair)

[hashutosh] HIVE-6016 : Hadoop23Sh

Hive CBO - Branch Request

[John Pullokkaran]

Hi,

  I am working on CBO for Hive
(HIVE-5775
).

In order to make code integration easier i would like to do this work on a
separate branch which can be brought into trunk once code is stable and
reviewed.

It would also be easier if i could commit in to this branch without having
to wait for a committer.


Thanks
John

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Updated] (HIVE-6065) orc/FileDump and testRCFile should close on appropriate resources

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6065:
-

Description: On Windows, TestRCFile/TestFileDump were unable to delete some 
test files because something was still hanging on to them.  Looks like there 
are a couple of places where we need to close some resources.  (was: On 
Windows, TestSchemaTool/TestMetastoreVersion were unable to delete some test 
files because something was still hanging on to them.  Looks like there are a 
couple of places where we need to close some resources.)

> orc/FileDump and testRCFile should close on appropriate resources
> -
>
> Key: HIVE-6065
> URL: https://issues.apache.org/jira/browse/HIVE-6065
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6065.1.patch
>
>
> On Windows, TestRCFile/TestFileDump were unable to delete some test files 
> because something was still hanging on to them.  Looks like there are a 
> couple of places where we need to close some resources.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6006) Add UDF to calculate distance between geographic coordinates

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853497#comment-13853497
 ] 

Xuefu Zhang commented on HIVE-6006:
---

This UDF seems so specific that I'm wondering if Hive should include it as a 
build-in UDF. 

> Add UDF to calculate distance between geographic coordinates
> 
>
> Key: HIVE-6006
> URL: https://issues.apache.org/jira/browse/HIVE-6006
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: hive-6006.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice to have Hive UDF to calculate distance between two points on 
> Earth. Haversine formula seems to be good enough to overcome this issue
> The next function is proposed:
> HaversineDistance(lat1, lon1, lat2, lon2) - calculate Harvesine Distance 
> between 2 points with coordinates (lat1, lon1) and (lat2, lon2)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6006) Add UDF to calculate distance between geographic coordinates

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853491#comment-13853491
 ] 

Jason Dere commented on HIVE-6006:
--

Hi Kostiantyn, took a look at the patch, couple comments:

TestHaversineDistance.java:
- missing Apache header at the top of the file
- perhaps use assertEquals(double expected, double actual, double delta) here 
to validate the values

UDFHaversineDistance.java:
- UDF description: What units is this distance for, km? Will be useful to add 
to description.
- Rather than creating new DoubleWritable each time, take a look at how UDFCos 
creates a result variable once and re-uses it.
- evaluate(): 
  - maybe check that lat/lon is within [-90,90] and [-180,180]
  - looks like you have a tab in there, replace with spaces
  - maybe use double rather than Double?




> Add UDF to calculate distance between geographic coordinates
> 
>
> Key: HIVE-6006
> URL: https://issues.apache.org/jira/browse/HIVE-6006
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.13.0
>Reporter: Kostiantyn Kudriavtsev
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: hive-6006.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> It would be nice to have Hive UDF to calculate distance between two points on 
> Earth. Haversine formula seems to be good enough to overcome this issue
> The next function is proposed:
> HaversineDistance(lat1, lon1, lat2, lon2) - calculate Harvesine Distance 
> between 2 points with coordinates (lat1, lon1) and (lat2, lon2)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853490#comment-13853490
 ] 

Xuefu Zhang commented on HIVE-6052:
---

I think this JIRA has documentation impact. Release note should probably be 
updated.

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15654: Rewrite Trim and Pad UDFs based on GenericUDF

[Xuefu Zhang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15654/#review30726
---



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java


I was thinking of doing this:

if (!(auguments[0] instanceof PrimitiveObjectInspector)) {
   ...
}



- Xuefu Zhang


On Dec. 18, 2013, 3:16 a.m., Mohammad Islam wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15654/
> ---
> 
> (Updated Dec. 18, 2013, 3:16 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Carl Steinbach, and Jitendra 
> Pandey.
> 
> 
> Bugs: HIVE-5829
> https://issues.apache.org/jira/browse/HIVE-5829
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Rewrite the UDFS *pads and *trim using GenericUDF.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a895d65 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> bca1f26 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLTrim.java dc00cf9 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLpad.java d1da19a 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRTrim.java 2bcc5fa 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFRpad.java 9652ce2 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/UDFTrim.java 490886d 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrim.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java
>  eff251f 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRTrim.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFRpad.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFTrim.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/15654/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mohammad Islam
> 
>

[jira] [Commented] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853484#comment-13853484
 ] 

Prasad Mujumdar commented on HIVE-5992:
---

+1


> Hive inconsistently converts timestamp in AVG and SUM UDAF's
> 
>
> Key: HIVE-5992
> URL: https://issues.apache.org/jira/browse/HIVE-5992
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5992.patch
>
>
> {code}
> hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
> ...
> OK
> 1977-03-15 12:34:22.345678 227306062  1  227306062
> 2.27306062345678E8
> {code}
> As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
> with fractional part ignored in sum, while preserved in avg. As a further 
> result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16328: HIVE-5992: Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Prasad Mujumdar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16328/#review30725
---

Ship it!


LGTM

- Prasad Mujumdar


On Dec. 17, 2013, 9:23 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16328/
> ---
> 
> (Updated Dec. 17, 2013, 9:23 p.m.)
> 
> 
> Review request for hive and Prasad Mujumdar.
> 
> 
> Bugs: HIVE-5992
> https://issues.apache.org/jira/browse/HIVE-5992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The fix is to make the two UDAFs report convert timestamp to double in terms 
> of seconds and the fraction of the second.
> Test is added to cover the case.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 
> 41d5efd 
>   ql/src/test/queries/clientpositive/timestamp_3.q e5a4345 
>   ql/src/test/results/clientpositive/timestamp_3.q.out 8544307 
> 
> Diff: https://reviews.apache.org/r/16328/diff/
> 
> 
> Testing
> ---
> 
> Unit test. New unit test. Regression suite.
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

[jira] [Commented] (HIVE-3616) Use Paths consistently

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853481#comment-13853481
 ] 

Hive QA commented on HIVE-3616:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619665/HIVE-3616.patch

{color:green}SUCCESS:{color} +1 4799 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/709/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/709/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619665

> Use Paths consistently
> --
>
> Key: HIVE-3616
> URL: https://issues.apache.org/jira/browse/HIVE-3616
> Project: Hive
>  Issue Type: Improvement
>  Components: Diagnosability
>Reporter: Ashutosh Chauhan
> Attachments: HIVE-3616.incomplete.patch, HIVE-3616.patch
>
>
> Currently, we interchangeably use Path, Uri and Strings in various parts of 
> codebases. This may results in subtle bugs. We should consistently use Path 
> in the codebase. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6069) Improve error message in GenericUDFRound

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-6069:
-

 Summary: Improve error message in GenericUDFRound
 Key: HIVE-6069
 URL: https://issues.apache.org/jira/browse/HIVE-6069
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Xuefu Zhang
Priority: Trivial


Suggested in HIVE-6039 review board.
https://reviews.apache.org/r/16329/



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16329: HIVE-6039: Round, AVG and SUM functions reject char/varch input while accepting string input

[Xuefu Zhang]

> On Dec. 19, 2013, 6:54 a.m., Prasad Mujumdar wrote:
> > The patch overall looks fine to. A minor suggestion below.
> > 
> > I am wondering if the string value (string, char or varchar) is in the 
> > decimal format (scientific or non-scientific), the converting to double 
> > won't work. Do you think it's better to convert the string values to 
> > decimal in all these cases ?
> > If not, it might be helpful to document that behavior.
> >

In hive, String type used in the numeric context is consistently converted to 
double, an approximate type. If scientific number cannot be converted, that 
should be solved in a separate JIRA. I don't think we should convert an 
approximate type to an exact type which is the principle that Hive is following.


> On Dec. 19, 2013, 6:54 a.m., Prasad Mujumdar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java, 
> > line 151
> > 
> >
> > I do agree that "Only numeric data types allowed" is a bit misleading 
> > since we do support string types as well ..
> 
> Xuefu Zhang wrote:
> Guys, I appreciated your feedback. However, your comment is about the 
> code that I didn't change. If you feel like that a change is needed, I guess 
> a different JIRA is more appropriate. Thanks.
> 
> Prasad Mujumdar wrote:
> that's true. This is all these issues were applicable to STRING type as 
> well.
> A followup jira would be helpful, especially to consider decimal 
> conversion for string types. Thanks!

Okay. I will create a JIRA for this.


- Xuefu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16329/#review30678
---


On Dec. 17, 2013, 9:26 p.m., Xuefu Zhang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16329/
> ---
> 
> (Updated Dec. 17, 2013, 9:26 p.m.)
> 
> 
> Review request for hive and Prasad Mujumdar.
> 
> 
> Bugs: HIVE-6039
> https://issues.apache.org/jira/browse/HIVE-6039
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Allow input to these UDFs for char and varchar.
> 
> 
> Diffs
> -
> 
>   data/files/char_varchar_udf.txt PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 
> 4b219bd 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 
> 41d5efd 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java 
> fc9c1b2 
>   ql/src/test/queries/clientpositive/char_varchar_udf.q PRE-CREATION 
>   ql/src/test/results/clientpositive/char_varchar_udf.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/16329/diff/
> 
> 
> Testing
> ---
> 
> Unit tested. New test added. Test suite passed.
> 
> 
> Thanks,
> 
> Xuefu Zhang
> 
>

[jira] [Commented] (HIVE-5176) Wincompat : Changes for allowing various path compatibilities with Windows

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853450#comment-13853450
 ] 

Jason Dere commented on HIVE-5176:
--

RB at https://reviews.apache.org/r/16403/

> Wincompat : Changes for allowing various path compatibilities with Windows
> --
>
> Key: HIVE-5176
> URL: https://issues.apache.org/jira/browse/HIVE-5176
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5176.2.patch, HIVE-5176.patch
>
>
> We need to make certain changes across the board to allow us to read/parse 
> windows paths. Some are escaping changes, some are being strict about how we 
> read paths (through URL.encode/decode, etc)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853448#comment-13853448
 ] 

Jason Dere commented on HIVE-4448:
--

The same changes are also a part of HIVE-5176, trying to get that one pushed 
through.

> Fix metastore warehouse incorrect location on Windows in unit tests
> ---
>
> Key: HIVE-4448
> URL: https://issues.apache.org/jira/browse/HIVE-4448
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-4448.1.patch
>
>
> Unit test cases which not using QTestUtil will pass incompatible Windows path 
> of METASTOREWAREHOUSE to HiveConf which result in creating the 
> /test/data/warehouse folder in the wrong location in Windows. This folder 
> will not be deleted at the beginning of the unit test and the content will 
> cause failure of unit tests if run the same test case repeatedly. The root 
> cause of this problem is for path like this 
> "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\" part 
> will be parsed as authority of the path and removed from the path string. The 
> patch will fix this problem and make the unit test result consistent between 
> Windows and Linux.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Resolved] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere resolved HIVE-4448.
--

Resolution: Duplicate

> Fix metastore warehouse incorrect location on Windows in unit tests
> ---
>
> Key: HIVE-4448
> URL: https://issues.apache.org/jira/browse/HIVE-4448
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-4448.1.patch
>
>
> Unit test cases which not using QTestUtil will pass incompatible Windows path 
> of METASTOREWAREHOUSE to HiveConf which result in creating the 
> /test/data/warehouse folder in the wrong location in Windows. This folder 
> will not be deleted at the beginning of the unit test and the content will 
> cause failure of unit tests if run the same test case repeatedly. The root 
> cause of this problem is for path like this 
> "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\" part 
> will be parsed as authority of the path and removed from the path string. The 
> patch will fix this problem and make the unit test result consistent between 
> Windows and Linux.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16403: HIVE-5176: Wincompat : Changes for allowing various path compatibilities with Windows

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16403/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5176
https://issues.apache.org/jira/browse/HIVE-5176


Repository: hive-git


Description
---

We need to make certain changes across the board to allow us to read/parse 
windows paths. Some are escaping changes, some are being strict about how we 
read paths (through URL.encode/decode, etc)


Diffs
-

  cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java f08a8b6 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fa3e048 
  common/src/test/org/apache/hadoop/hive/conf/TestHiveConf.java a31238b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 38d97e3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 5cb492f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 9afc80b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java d0be73e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java b9cd65c 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 0684aac 
  ql/src/test/org/apache/hadoop/hive/ql/WindowsPathUtil.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java d4ad931 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
69d1896 

Diff: https://reviews.apache.org/r/16403/diff/


Testing
---


Thanks,

Jason Dere

[jira] [Commented] (HIVE-4417) Make unit tests pass on Windows which use Linux script in query files

[Shuaishuai Nie (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853442#comment-13853442
 ] 

Shuaishuai Nie commented on HIVE-4417:
--

Thank [~jdere] for re-basing the patch.

> Make unit tests pass on Windows which use Linux script in query files
> -
>
> Key: HIVE-4417
> URL: https://issues.apache.org/jira/browse/HIVE-4417
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.11.0
> Environment: Windows 8
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-4417.1.patch, HIVE-4417.2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Make unit tests input20.q, input33.q in TestCliDriver working on both Linux 
> and Windows by implementing "input20_script" in python. Also make newline.q 
> working on Windows by using "cat" in consistent way in query file.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5176) Wincompat : Changes for allowing various path compatibilities with Windows

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5176:
-

Attachment: HIVE-5176.2.patch

Rebasing patch with trunk

> Wincompat : Changes for allowing various path compatibilities with Windows
> --
>
> Key: HIVE-5176
> URL: https://issues.apache.org/jira/browse/HIVE-5176
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5176.2.patch, HIVE-5176.patch
>
>
> We need to make certain changes across the board to allow us to read/parse 
> windows paths. Some are escaping changes, some are being strict about how we 
> read paths (through URL.encode/decode, etc)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5176) Wincompat : Changes for allowing various path compatibilities with Windows

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-5176:
-

Status: Patch Available  (was: Open)

> Wincompat : Changes for allowing various path compatibilities with Windows
> --
>
> Key: HIVE-5176
> URL: https://issues.apache.org/jira/browse/HIVE-5176
> Project: Hive
>  Issue Type: Sub-task
>  Components: Windows
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5176.2.patch, HIVE-5176.patch
>
>
> We need to make certain changes across the board to allow us to read/parse 
> windows paths. Some are escaping changes, some are being strict about how we 
> read paths (through URL.encode/decode, etc)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6068) HiveServer2 beeline client on windows does not handle the non-ascii characters properly

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-6068:
---

Environment: Windows 

> HiveServer2 beeline client on windows does not handle the non-ascii 
> characters properly
> ---
>
> Key: HIVE-6068
> URL: https://issues.apache.org/jira/browse/HIVE-6068
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.13.0
> Environment: Windows 
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
>
> When running a select query against a table which contains rows with 
> non-ascii characters HiveServer2 Beeline client returns them wrong. Example:
> {noformat}
> 738;Garçu, Le (1995);Drama
> 741;Ghost in the Shell (Kôkaku kidôtai) (1995);Animation|Sci-Fi
> {noformat}
> come out from a HiveServer2 beeline client as:
> {noformat}
> '738' 'Gar?u, Le (1995)'  'Drama'
> '741' 'Ghost in the Shell (K?kaku kid?tai) (1995)''Animation|Sci-Fi'
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6068) HiveServer2 beeline client on windows does not handle the non-ascii characters properly

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-6068:
--

 Summary: HiveServer2 beeline client on windows does not handle the 
non-ascii characters properly
 Key: HIVE-6068
 URL: https://issues.apache.org/jira/browse/HIVE-6068
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 0.13.0
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta
 Fix For: 0.13.0


When running a select query against a table which contains rows with non-ascii 
characters HiveServer2 Beeline client returns them wrong. Example:
{noformat}
738;Garçu, Le (1995);Drama
741;Ghost in the Shell (Kôkaku kidôtai) (1995);Animation|Sci-Fi
{noformat}
come out from a HiveServer2 beeline client as:
{noformat}
'738'   'Gar?u, Le (1995)'  'Drama'
'741'   'Ghost in the Shell (K?kaku kid?tai) (1995)''Animation|Sci-Fi'
{noformat}




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5795) Hive should be able to skip header and footer rows when reading data file for a table

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853425#comment-13853425
 ] 

Hive QA commented on HIVE-5795:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619655/HIVE-5795.1.patch

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/708/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/708/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-708/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/results/clientpositive/input20.q.out'
Reverted 'ql/src/test/results/clientpositive/input33.q.out'
Reverted 'ql/src/test/results/beelinepositive/input33.q.out'
Reverted 'ql/src/test/results/beelinepositive/newline.q.out'
Reverted 'ql/src/test/results/beelinepositive/input20.q.out'
Reverted 'ql/src/test/queries/clientpositive/input20.q'
Reverted 'ql/src/test/queries/clientpositive/input33.q'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
data/scripts/input20_script.py itests/target itests/hcatalog-unit/target 
itests/test-serde/target itests/qtest/target itests/hive-unit/target 
itests/custom-serde/target itests/util/target hcatalog/target 
hcatalog/storage-handlers/hbase/target hcatalog/server-extensions/target 
hcatalog/core/target hcatalog/webhcat/svr/target 
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target 
hwi/target common/target common/src/gen contrib/target service/target 
serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update
Uitests/qtest/pom.xml
Ucommon/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Umetastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
Umetastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
U
metastore/src/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
Umetastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
Dql/src/test/results/clientnegative/invalid_columns.q.out
Uql/src/test/results/clientnegative/invalidate_view1.q.out
Uql/src/test/results/clientnegative/regex_col_1.q.out
Uql/src/test/results/clientnegative/regex_col_2.q.out
Uql/src/test/results/clientnegative/regex_col_groupby.q.out
Aql/src/test/results/clientpositive/quotedid_alter.q.out
Uql/src/test/results/clientpositive/quote1.q.out
Uql/src/test/results/clientpositive/escape_orderby1.q.out
Uql/src/test/results/clientpositive/index_bitmap3.q.out
Uql/src/test/results/clientpositive/auto_join_reordering_values.q.out
Aql/src/test/results/clientpositive/quotedid_tblproperty.q.out
Aql/src/test/results/clientpositive/quotedid_skew.q.out
Uql/src/test/results/clientpositive/annotate_stats_part.q.out
Uql/src/test/results/clientpositive/dynamic_partition_skip_default.q.out
Aql/src/test/results/clientpositive/quotedid_basic.q.out
Aql/src/test/results/clientpositive/quotedid_partition.q.out
Aql/src/test/results/clientpositive/quotedid_smb.q.out
Uql/src/test/results/clientpositive/index_bitmap_auto.q.out
Uql/src/test/results/clientpositive/escape_distributeby1.q.out
Uql/src/test/results/clientpositive/alter_partition_coltype.q.out
Uql/src/test/results/clientpositive/escape_clusterby1.q.out
Uql/src/test/results/clientpositive/escape_sortby1.q.out
Dql/src/test/queries/clientnegative/invalid_columns.q
Uql/src/test/queries/clientnegative/regex_col_1.q
Uql/src/test/queries/clientnegative/regex_col_groupby.q
Uql/src/test/queries/clientnegative/regex_col_2.q
Uql/src/test/queries/clientnegative/ambiguous_col1.q
Uql/src/test/queries/clientnegative/ambiguous_col2.q
Uql/src/test/queries/clientpositive/amb

[jira] [Commented] (HIVE-4417) Make unit tests pass on Windows which use Linux script in query files

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853423#comment-13853423
 ] 

Hive QA commented on HIVE-4417:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12619641/HIVE-4417.2.patch

{color:green}SUCCESS:{color} +1 4795 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/707/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/707/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12619641

> Make unit tests pass on Windows which use Linux script in query files
> -
>
> Key: HIVE-4417
> URL: https://issues.apache.org/jira/browse/HIVE-4417
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.11.0
> Environment: Windows 8
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-4417.1.patch, HIVE-4417.2.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Make unit tests input20.q, input33.q in TestCliDriver working on both Linux 
> and Windows by implementing "input20_script" in python. Also make newline.q 
> working on Windows by using "cat" in consistent way in query file.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6013) Supporting Quoted Identifiers in Column Names

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6013:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Harish!

> Supporting Quoted Identifiers in Column Names
> -
>
> Key: HIVE-6013
> URL: https://issues.apache.org/jira/browse/HIVE-6013
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Fix For: 0.13.0
>
> Attachments: HIVE-6013.1.patch, HIVE-6013.2.patch, HIVE-6013.3.patch, 
> HIVE-6013.4.patch, HIVE-6013.5.patch, HIVE-6013.6.patch, HIVE-6013.7.patch, 
> QuotedIdentifier.html
>
>
> Hive's current behavior on Quoted Identifiers is different from the normal 
> interpretation. Quoted Identifier (using backticks) has a special 
> interpretation for Select expressions(as Regular Expressions). Have 
> documented current behavior and proposed a solution in attached doc.
> Summary of solution is:
> - Introduce 'standard' quoted identifiers for columns only. 
> - At the langauage level this is turned on by a flag.
> - At the metadata level we relax the constraint on column names.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-6067) Implement vectorized decimal column-scalar comparison filters

[Eric Hanson (JIRA)]

Eric Hanson created HIVE-6067:
-

 Summary: Implement vectorized decimal column-scalar comparison 
filters
 Key: HIVE-6067
 URL: https://issues.apache.org/jira/browse/HIVE-6067
 Project: Hive
  Issue Type: Sub-task
Reporter: Eric Hanson
Assignee: Eric Hanson


Using the new DecimalColumnVector type, implement a template to generate 
VectorExpression subclasses for Decimal comparison filters (<, <=, >, >=, =, 
!=).



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6052) metastore JDO filter pushdown for integers may produce unexpected results with non-normalized integer columns

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6052:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Sergey!

> metastore JDO filter pushdown for integers may produce unexpected results 
> with non-normalized integer columns
> -
>
> Key: HIVE-6052
> URL: https://issues.apache.org/jira/browse/HIVE-6052
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.13.0
>
> Attachments: HIVE-6052.01.patch, HIVE-6052.02.patch, HIVE-6052.patch
>
>
> If integer partition columns have values stores in non-canonical form, for 
> example with leading zeroes, the integer filter doesn't work. That is because 
> JDO pushdown uses substrings to compare for equality, and SQL pushdown is 
> intentionally crippled to do the same to produce same results.
> Probably, since both SQL pushdown and integers pushdown are just perf 
> optimizations, we can remove it for JDO (or make configurable and disable by 
> default), and uncripple SQL.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-6066) Wincompat: describe_comment_nonascii.q failing on windows

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6066:
-

Status: Patch Available  (was: Open)

> Wincompat: describe_comment_nonascii.q failing on windows
> -
>
> Key: HIVE-6066
> URL: https://issues.apache.org/jira/browse/HIVE-6066
> Project: Hive
>  Issue Type: Sub-task
>  Components: Tests, Windows
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6066.1.patch
>
>
> describe_comment_nonascii.q failing on Windows. Some strings are not being 
> properly converted to utf-8 bytes, both during formatting and when diffing 
> the results.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)