[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422856#comment-13422856
 ] 

Carl Steinbach commented on HIVE-3303:
--

We should use the [INCLUDE|EXCLUDE]_HADOOP_MAJOR_VERSIONS macros to fix this. 
The 0.23 behavior should be the standard going forward, so please create 
mapreduce_stack_trace_h20.q and use the INCLUDE macro, and EXCLUDE 0.20 from 
mapreduce_stack_trace.q

> Fix error code inconsistency bug in mapreduce_stack_trace.q and 
> mapreduce_stack_trace_turnoff.q when running hive on hadoop23
> -
>
> Key: HIVE-3303
> URL: https://issues.apache.org/jira/browse/HIVE-3303
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> when running hive on hadoop23, mapreduce_stack_trace.q and 
> mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs:
> [junit] diff -a 
> /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out
>  
> /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
> [junit] < FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask
> [junit] > FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script.
> [junit] diff -a 
> /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out
>  
> /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
> [junit] 5c5
> [junit] < FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask
> [junit] —
> [junit] > FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script
> The error code 2(which indicates unable to initialize custom script) 
> could not be retrieved. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-2693) Add DECIMAL data type

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach reassigned HIVE-2693:


Assignee: Prasad Mujumdar  (was: Josh Wills)

> Add DECIMAL data type
> -
>
> Key: HIVE-2693
> URL: https://issues.apache.org/jira/browse/HIVE-2693
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor, Types
>Reporter: Carl Steinbach
>Assignee: Prasad Mujumdar
> Attachments: HIVE-2693.patch
>
>
> Add support for the DECIMAL data type. HIVE-2272 (TIMESTAMP) provides a nice 
> template for how to do this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Status: Patch Available  (was: Open)

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt, 
> HIVE-3301.3.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Status: Open  (was: Patch Available)

patch has prefix in it

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt, 
> HIVE-3301.3.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Attachment: HIVE-3301.3.patch.txt

updated patch without prefix
could apply cleanly

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt, 
> HIVE-3301.3.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422832#comment-13422832
 ] 

Zhenxiao Luo commented on HIVE-3301:


@ashutosh: Thanks a lot for the comments.

I made updates and resubmitted review request at:
https://reviews.facebook.net/D4353

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422831#comment-13422831
 ] 

Zhenxiao Luo commented on HIVE-3301:


@Edward: related HIVE tickets are linked. I will add more whenever any new bugs 
filed. Do we need a separate upper level JIRA to trace all the hadoop23 
integration bugs?

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Attachment: HIVE-3301.2.patch.txt

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Status: Patch Available  (was: Open)

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt, HIVE-3301.2.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3262) bucketed mapjoin silently ignores mapjoin hint

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3262:
-

Status: Open  (was: Patch Available)

@Kevin: Canceling this patch too. I'm not really sure why you need to +1 both 
this and HIVE-3289 since apparently the patch (which is not attached here!) has 
been combined with HIVE-3289. There should be a 1:1 correspondence between 
committed patches and "Fixed" JIRA tickets.

> bucketed mapjoin silently ignores mapjoin hint
> --
>
> Key: HIVE-3262
> URL: https://issues.apache.org/jira/browse/HIVE-3262
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> If the bucketed mapjoin is not performed, it is silently ignored.
> Atleast under strict mode, it should lead to an error.
> Would wait for HIVE-3210 before working on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3289) sort merge join may not work silently

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422819#comment-13422819
 ] 

Carl Steinbach commented on HIVE-3289:
--

Two more points which are tangentially related:

* The patch is not attached to this ticket, and it looks like Phabricator 
stopped automatically attaching patches some time ago. Is anyone at Facebook 
looking into fixing this?
* Part of the agreement when we started using Phabricator was that the tool 
would automatically copy review comments back to JIRA. This feature hasn't 
worked in months, and unless it starts working soon I think we should stop 
using Phabricator and switch back to ReviewBoard. Is anyone looking into fixing 
this? If not we should probably just switch back now.

> sort merge join may not work silently
> -
>
> Key: HIVE-3289
> URL: https://issues.apache.org/jira/browse/HIVE-3289
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> The user does not know, if the sort-merge join is working or not.
> create table table_asc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key asc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> create table table_desc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key desc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> set hive.enforce.sorting = true;
> insert overwrite table table_asc select key, value from src;
> insert overwrite table table_desc select key, value from src;
> set hive.optimize.bucketmapjoin = true;
> set hive.optimize.bucketmapjoin.sortedmerge = true;
> set hive.input.format = 
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> explain 
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> explain
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> In the above test, the sort-merge join is not obeyed as expected.
> If you user explicitly asked for sort-merge join, and it is not being
> obeyed, the operation should fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422816#comment-13422816
 ] 

Zhenxiao Luo commented on HIVE-3303:


The problem is, hadoop23 is getting Task-diagnostics differently from hadoop20.

In hadoop20, Task-diagnostics is retrieved via jobSubmitClient in 
JobClient.java:

 public String[] getTaskDiagnostics(TaskAttemptID id) throws IOException {
  return jobSubmitClient.getTaskDiagnostics(id);
}

And in JobTracker.java, all the related logs are put into diagnostic info:


public synchronized String[] getTaskDiagnostics(TaskAttemptID taskId)
throws IOException {

JobID jobId = taskId.getJobID();
TaskID tipId = taskId.getTaskID();
JobInProgress job = jobs.get(jobId);
if (job == null) {
  throw new IllegalArgumentException("Job " + jobId + " not found.");
}
TaskInProgress tip = job.getTaskInProgress(tipId);
if (tip == null) {
  throw new IllegalArgumentException("TIP " + tipId + " not found.");
}
List taskDiagnosticInfo = tip.getDiagnosticInfo(taskId);
return ((taskDiagnosticInfo == null) ? null
: taskDiagnosticInfo.toArray(new String[0]));
  }

Here is the diagnostic info in hadoop20:

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row {"key":"238","value":"val_238"}
[junit] at 
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
[junit] at 
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
[junit] at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
[junit] at org.apache.hadoop.mapred.Child.main(Child.java:170)
[junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive 
Runtime Error while processing row {"key":"238","value":"val_238"}
[junit] at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548)
[junit] at 
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
[junit] ... 4 more
[junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 
2]: Unable to initialize custom script.
[junit] at 
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:346)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at 
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at 
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
[junit] ... 5 more
[junit] Caused by: java.io.IOException: Cannot run program 
"script_does_not_exist": java.io.IOException: error=2, No such file or directory
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
[junit] at 
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:305)
[junit] ... 14 more
[junit] Caused by: java.io.IOException: java.io.IOException: error=2, No 
such file or directory
[junit] at java.lang.UNIXProcess.(UNIXProcess.java:164)
[junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81)
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
[junit] ... 15 more

The error code [2] appears in the diagnostic info, and could be retrieved 
by Hive.

While, in hadoop23, in Job.java, a different execution path is:


public String[] getTaskDiagnostics(final TaskAttemptID taskid)
  throws IOException, InterruptedException {
ensureState(JobState.RUNNING);
return ugi.doAs(new PrivilegedExceptionAction() {
  @Override
  public String[] run() throws IOException, InterruptedException {
return cluster.getClient().getTaskDiagnostics(taskid);
  }
});
  }

Here is the diagnostic info in hadoop23:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row {"key":"238","value":"val_238"}
[junit] at 
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at 
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
[junit] at 
org.apache.hadoop.mapred.MapTask.runOl

[jira] [Comment Edited] (HIVE-3289) sort merge join may not work silently

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422812#comment-13422812
 ] 

Carl Steinbach edited comment on HIVE-3289 at 7/26/12 1:32 AM:
---

-1

bq. I also am not a fan of hive.mapred.mode  If you turn it off, you may 
unintentionally turn off other checks, and it uses strict/nonstrict instead of 
true/false which is easier to validate. That's, at best, a problem for another 
JIRA, though, as it's fairly well established.

I agree with Kevin, but I don't think this should be postponed for another 
JIRA. Please add a new configuration property now instead of further 
overloading what is an already ill-defined and poorly documented configuration 
property.


  was (Author: cwsteinbach):
-1

bq. I also am not a fan of hive.mapred.mode  If you turn it off, you may 
unintentionally turn off other checks, and it uses strict/nonstrict instead of 
true/false which is easier to validate. That's, at best, a problem for another 
JIRA, though, as it's fairly well established.

I agree with Kevin, but I don't think this should be postponed for another 
JIRA. Please add a new configuration property now instead of further 
overloading what is already ill-defined and poorly documented configuration 
property.

  
> sort merge join may not work silently
> -
>
> Key: HIVE-3289
> URL: https://issues.apache.org/jira/browse/HIVE-3289
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> The user does not know, if the sort-merge join is working or not.
> create table table_asc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key asc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> create table table_desc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key desc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> set hive.enforce.sorting = true;
> insert overwrite table table_asc select key, value from src;
> insert overwrite table table_desc select key, value from src;
> set hive.optimize.bucketmapjoin = true;
> set hive.optimize.bucketmapjoin.sortedmerge = true;
> set hive.input.format = 
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> explain 
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> explain
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> In the above test, the sort-merge join is not obeyed as expected.
> If you user explicitly asked for sort-merge join, and it is not being
> obeyed, the operation should fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3289) sort merge join may not work silently

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422812#comment-13422812
 ] 

Carl Steinbach commented on HIVE-3289:
--

-1

bq. I also am not a fan of hive.mapred.mode  If you turn it off, you may 
unintentionally turn off other checks, and it uses strict/nonstrict instead of 
true/false which is easier to validate. That's, at best, a problem for another 
JIRA, though, as it's fairly well established.

I agree with Kevin, but I don't think this should be postponed for another 
JIRA. Please add a new configuration property now instead of further 
overloading what is already ill-defined and poorly documented configuration 
property.


> sort merge join may not work silently
> -
>
> Key: HIVE-3289
> URL: https://issues.apache.org/jira/browse/HIVE-3289
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> The user does not know, if the sort-merge join is working or not.
> create table table_asc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key asc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> create table table_desc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key desc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> set hive.enforce.sorting = true;
> insert overwrite table table_asc select key, value from src;
> insert overwrite table table_desc select key, value from src;
> set hive.optimize.bucketmapjoin = true;
> set hive.optimize.bucketmapjoin.sortedmerge = true;
> set hive.input.format = 
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> explain 
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> explain
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> In the above test, the sort-merge join is not obeyed as expected.
> If you user explicitly asked for sort-merge join, and it is not being
> obeyed, the operation should fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3289) sort merge join may not work silently

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-3289:
-

Status: Open  (was: Patch Available)

> sort merge join may not work silently
> -
>
> Key: HIVE-3289
> URL: https://issues.apache.org/jira/browse/HIVE-3289
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> The user does not know, if the sort-merge join is working or not.
> create table table_asc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key asc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> create table table_desc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key desc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> set hive.enforce.sorting = true;
> insert overwrite table table_asc select key, value from src;
> insert overwrite table table_desc select key, value from src;
> set hive.optimize.bucketmapjoin = true;
> set hive.optimize.bucketmapjoin.sortedmerge = true;
> set hive.input.format = 
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> explain 
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> explain
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> In the above test, the sort-merge join is not obeyed as expected.
> If you user explicitly asked for sort-merge join, and it is not being
> obeyed, the operation should fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3301:
---

Status: Open  (was: Patch Available)

Zhenxiao I left comments on Phabricator.

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3262) bucketed mapjoin silently ignores mapjoin hint

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422807#comment-13422807
 ] 

Kevin Wilfong commented on HIVE-3262:
-

+1 running tests

> bucketed mapjoin silently ignores mapjoin hint
> --
>
> Key: HIVE-3262
> URL: https://issues.apache.org/jira/browse/HIVE-3262
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> If the bucketed mapjoin is not performed, it is silently ignored.
> Atleast under strict mode, it should lead to an error.
> Would wait for HIVE-3210 before working on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3289) sort merge join may not work silently

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422808#comment-13422808
 ] 

Kevin Wilfong commented on HIVE-3289:
-

+1 running tests

> sort merge join may not work silently
> -
>
> Key: HIVE-3289
> URL: https://issues.apache.org/jira/browse/HIVE-3289
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> The user does not know, if the sort-merge join is working or not.
> create table table_asc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key asc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> create table table_desc(key int, value string) CLUSTERED BY (key) SORTED BY 
> (key desc) 
> INTO 1 BUCKETS STORED AS RCFILE; 
> set hive.enforce.sorting = true;
> insert overwrite table table_asc select key, value from src;
> insert overwrite table table_desc select key, value from src;
> set hive.optimize.bucketmapjoin = true;
> set hive.optimize.bucketmapjoin.sortedmerge = true;
> set hive.input.format = 
> org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
> explain 
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(a)*/ * from table_asc a join table_desc b on a.key = b.key;
> explain
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> select /*+mapjoin(b)*/ * from table_asc a join table_desc b on a.key = b.key;
> In the above test, the sort-merge join is not obeyed as expected.
> If you user explicitly asked for sort-merge join, and it is not being
> obeyed, the operation should fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422800#comment-13422800
 ] 

Edward Capriolo commented on HIVE-3301:
---

I am not saying we need to shim layer all the fixes, but having a reasonably 
exhaustive list of the problems linked together in jira would make me more 
confident that we are taking the right plan of action. 

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

Zhenxiao Luo created HIVE-3303:
--

 Summary: Fix error code inconsistency bug in 
mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive 
on hadoop23
 Key: HIVE-3303
 URL: https://issues.apache.org/jira/browse/HIVE-3303
 Project: Hive
  Issue Type: Bug
Reporter: Zhenxiao Luo
Assignee: Zhenxiao Luo


when running hive on hadoop23, mapreduce_stack_trace.q and 
mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs:

[junit] diff -a 
/home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out
 
/home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out
[junit] < FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
[junit] > FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script.


[junit] diff -a 
/home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out
 
/home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out
[junit] 5c5
[junit] < FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
[junit] —
[junit] > FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script

The error code 2(which indicates unable to initialize custom script) could 
not be retrieved. 



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422795#comment-13422795
 ] 

Zhenxiao Luo commented on HIVE-3301:


@Edward: oh yes. As I know, HIVE-3301, HIVE-3275, HIVE-3273, HIVE-3242, 
HIVE-3240, HIVE-3257, HIVE-3249 and HIVE-2804 are all hadoop 23 bugs. I am 
fixing these one by one. Thanks for your advice. I will try to put them into a 
larger shim layer.

I just found an Error Code retrieval inconsistency between hadoop20 and 
hadoop23. Will file another one soon.

Thanks,
Zhenxiao

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Status: Patch Available  (was: Open)

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Attachment: HIVE-3301.1.patch.txt

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3301.1.patch.txt
>
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422790#comment-13422790
 ] 

Zhenxiao Luo commented on HIVE-3301:


review request submitted at:
https://reviews.facebook.net/D4353

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422789#comment-13422789
 ] 

Edward Capriolo commented on HIVE-3301:
---

You know these hadoop 23 jiras are like death of a thousand paper cuts, if I 
had known we were going to face so many issues i would have proposed making a 
larger shim layer. Can we come up with a definitive list of all the 23 problems?

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422787#comment-13422787
 ] 

Zhenxiao Luo commented on HIVE-3301:


The problem is:

In hadoop23, TaskLogServlet.java is using a new utility HtmlQuoting.java to 
print Task Log.

In TaskLogServlet.java, printTaskLog() function:

result = taskLogReader.read(b);
if (result > 0) {
  if (plainText) {
out.write(b, 0, result);
  } else {
HtmlQuoting.quoteHtmlChars(out, b, 0, result);
  }
} else {
  break;
}


While, in hadoop20,  TaskLogServlet.java is using its own utility(there is no 
such HtmlQuoting.java at all) to print Task Log:

In TaskLogServlet.java, printTaskLog fucntion:

result = taskLogReader.read(b);
if (result > 0) {
  if (plainText) {
out.write(b, 0, result);
  } else {
quotedWrite(out, b, 0, result);
  }
} else {
  break;
}


And in Hive, TaskLogProcessor.java is generating stack trace by reading the raw 
taskAttemptLog.

In ql/src/java/org/apache/hadoop/hive/ql/exec/errors/TaskLogProcessor.java, 
getStackTraces() fuction:


List stackTrace = null;

// Patterns that match the middle/end of stack traces
Pattern stackTracePattern = Pattern.compile("^\tat .*", 
Pattern.CASE_INSENSITIVE);
Pattern endStackTracePattern =
Pattern.compile("^\t... [0-9]+ more.*", Pattern.CASE_INSENSITIVE);

while ((inputLine = in.readLine()) != null) {

  if (stackTracePattern.matcher(inputLine).matches() ||
  endStackTracePattern.matcher(inputLine).matches()) {


To have Hive working for both hadoop20 and hadoop23, we should use different 
mechanisms when hive TaskLogProcessor is parsing TaskAttemptLog.

My plan is creating a shim, which have different implementations for hadoop20 
and hadoop23.

In hadoop23, HtmlQuoting.unquoteHtmlChars() is used to parse the TaskAttemptLog.

> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3302) Race condition in query plan for merging at the end of a query

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422782#comment-13422782
 ] 

Kevin Wilfong commented on HIVE-3302:
-

JIRA screwed up the formatting of my diagrams, the first is supposed to show 
MoveTask(1) and MergeTask stemming from ConditionalTask and MoveTask(2) 
dependent upon MoveTask(1) and MergeTask.

The second is supposed to show MergeTask stemming from ConditionalTask, 
MoveTask(1) and MoveTask(2) dependent on MergeTask, and most importantly, no 
dependency relationship between MoveTask(1) and MoveTask(2).

> Race condition in query plan for merging at the end of a query
> --
>
> Key: HIVE-3302
> URL: https://issues.apache.org/jira/browse/HIVE-3302
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
>Priority: Critical
>
> In the query plan that's used to merge files at the end of a query, the 
> dependency tree looks something like:
>MoveTask(1)
>   /   \
> ...ConditionalTask MoveTask(2)...
>   \   /
>MergeTask
> Here MoveTask(1) moves the partition data to a temporary location, and 
> MoveTask(2) moves it to the final location.
> However if there are dynamic partitions generated and some of these 
> partitions are merged and others are moved, the dependency tree is changed at 
> runtime to:
> ...ConditionalTask   MoveTask(2)...
>   \ /
>MergeTask
> \
>  MoveTask(1)
> This produces a race condition between the two MoveTasks where if MoveTask(2) 
> runs before MoveTask(1) the partitions moved by MoveTask(1) will get moved to 
> an intermediate location and never moved to the final location.  In this case 
> those partitions are quietly lost.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Description: 
When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
quote printing bug:

quote is printed as: '"', instead of "

Seems not able to state the bug clearly in html:

quote is printed as 'address sign' + 'quot' + semicolon
not the expected 'quote sign'

  was:
When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
quote printing bug:

quote is printed as: '"', instead of "


> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "
> Seems not able to state the bug clearly in html:
> quote is printed as 'address sign' + 'quot' + semicolon
> not the expected 'quote sign'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3302) Race condition in query plan for merging at the end of a query

[Kevin Wilfong (JIRA)]

Kevin Wilfong created HIVE-3302:
---

 Summary: Race condition in query plan for merging at the end of a 
query
 Key: HIVE-3302
 URL: https://issues.apache.org/jira/browse/HIVE-3302
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.10.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
Priority: Critical


In the query plan that's used to merge files at the end of a query, the 
dependency tree looks something like:
   MoveTask(1)
  /   \
...ConditionalTask MoveTask(2)...
  \   /
   MergeTask

Here MoveTask(1) moves the partition data to a temporary location, and 
MoveTask(2) moves it to the final location.

However if there are dynamic partitions generated and some of these partitions 
are merged and others are moved, the dependency tree is changed at runtime to:
...ConditionalTask   MoveTask(2)...
  \ /
   MergeTask
\
 MoveTask(1)

This produces a race condition between the two MoveTasks where if MoveTask(2) 
runs before MoveTask(1) the partitions moved by MoveTask(1) will get moved to 
an intermediate location and never moved to the final location.  In this case 
those partitions are quietly lost.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhenxiao Luo updated HIVE-3301:
---

Description: 
When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
quote printing bug:

quote is printed as: '"', instead of "

  was:
When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
quote printing bug:

quote is printed as: """, instead of "


> Fix quote printing bug in mapreduce_stack_trace.q testcase failure when 
> running hive on hadoop23
> 
>
> Key: HIVE-3301
> URL: https://issues.apache.org/jira/browse/HIVE-3301
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
>
> When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
> quote printing bug:
> quote is printed as: '"', instead of "

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3301) Fix quote printing bug in mapreduce_stack_trace.q testcase failure when running hive on hadoop23

[Zhenxiao Luo (JIRA)]

Zhenxiao Luo created HIVE-3301:
--

 Summary: Fix quote printing bug in mapreduce_stack_trace.q 
testcase failure when running hive on hadoop23
 Key: HIVE-3301
 URL: https://issues.apache.org/jira/browse/HIVE-3301
 Project: Hive
  Issue Type: Bug
Reporter: Zhenxiao Luo
Assignee: Zhenxiao Luo


When running hive on hadoop0.23, mapreduce_stack_trace.q is failing due to 
quote printing bug:

quote is printed as: """, instead of "

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3084) Hive CI failing due to script_broken_pipe1.q

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422752#comment-13422752
 ] 

Ashutosh Chauhan commented on HIVE-3084:


Yeah its really annoying that irregular and inconsistent failure of this test 
introduce huge irregularity in hudson test outputs. Quite a few svn commits are 
reported as failure where actual culprit is this test which makes hudson less 
reliable. If no one wants to investigate this further, I will commit Travis's 
patch and will open a follow-up jira for later investigation.

> Hive CI failing due to script_broken_pipe1.q
> 
>
> Key: HIVE-3084
> URL: https://issues.apache.org/jira/browse/HIVE-3084
> Project: Hive
>  Issue Type: Bug
>Reporter: Travis Crawford
>Assignee: Travis Crawford
> Attachments: HIVE-3084_disable_failing_test.1.patch
>
>
> Hive's [CI job|https://builds.apache.org/job/Hive-trunk-h0.21/] regularly 
> fails due to this test:
> {code}
> org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
> {code}
> Running just that test in a loop I'm not able to reproduce the issue locally. 
> This may be related to state from a previous test causing this one to fail?
> {code}
> ant clean package
> for x in $(seq 1 10); 
>   do ant test -Dtestcase=TestNegativeCliDriver -Dqfile=script_broken_pipe1.q;
> done
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3275) Fix autolocal1.q testcase failure when building hive on hadoop0.23 MR2

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422739#comment-13422739
 ] 

Ashutosh Chauhan commented on HIVE-3275:


Even between 0.20 and 1.x series, Exception type has changed from 
RuntimeException to IllegalArgumentTypeException as well as exception message 
has changed. I don't see any easy way to keep our testcases to succeed with 
such changes given our diff based comparisons. I think its fine to just make 
sure it run against one version and gives desired behavior. +1. will commit if 
tests pass.

> Fix autolocal1.q testcase failure when building hive on hadoop0.23 MR2
> --
>
> Key: HIVE-3275
> URL: https://issues.apache.org/jira/browse/HIVE-3275
> Project: Hive
>  Issue Type: Bug
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Attachments: HIVE-3275.1.patch.txt
>
>
> autolocal1.q is failing only on hadoop0.23 MR2, due to cluster initialization 
> problem:
> Begin query: autolocal1.q
> diff -a 
> /var/lib/jenkins/workspace/zhenxiao-CDH4-Hive-0.9.0/build/ql/test/logs/clientnegative/autolocal1.q.out
>  
> /var/lib/jenkins/workspace/zhenxiao-CDH4-Hive-0.9.0/ql/src/test/results/clientnegative/autolocal1.q.out
> 5c5
> < Job Submission failed with exception 'java.io.IOException(Cannot initialize 
> Cluster. Please check your configuration for mapreduce.framework.name and the 
> correspond server addresses.)'
> —
> > Job Submission failed with exception 
> > 'java.lang.IllegalArgumentException(Does not contain a valid host:port 
> > authority: abracadabra)'
> Exception: Client execution results failed with error code = 1
> See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
> more logs.
> Failed query: autolocal1.q

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1568 - Still Failing

[Apache Jenkins Server]

Changes for Build #1566
[cws] HIVE-3249 [jira] Upgrade guava to 11.0.2
(Zhenxiao Luo via Carl Steinbach)

Summary:
HIVE-3249: Upgrade guava to 11.0.2

Hadoop has upgraded to this new version of Guava. We should, too, so we don't 
have compatibility issues running on Hadoop 2.0+

currently, hive is using guava-r09.jar
we should update it to guava-11.0.2.jar

Test Plan: EMPTY

Reviewers: JIRA

Differential Revision: https://reviews.facebook.net/D4083


Changes for Build #1567
[hashutosh] HIVE-3126 : Generate & build the velocity based Hive tests on 
windows by fixing the path issues (Kanna Karanam via Ashutosh Chauhan)

[kevinwilfong] HIVE-3295. HIVE-3128 introduced bug causing dynamic partitioning 
to fail. (kevinwilfong reviewed by njain, ashutoshc)


Changes for Build #1568



5 tests failed.
REGRESSION:  
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisherOneStat

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at 
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisherOneStat(TestStatsPublisherEnhanced.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)


REGRESSION:  
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisher

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at 
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisher(TestStatsPublisherEnhanced.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)


REGRESSION:  
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisherMultipleUpdates

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at 
org.apache.hadoop.hive.ql.exec.TestStatsPublisherEnhanced.testStatsPublisherMultipleUpdates(TestStatsPublisherEnhanced.java:190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #86

[Apache Jenkins Server]

See 

--
[...truncated 36555 lines...]
[junit] POSTHOOK: query: select count(1) as cnt from testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2012-07-25_15-14-11_813_111749518353264222/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: DROPTABLE
[junit] Copying file: 

[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] Copying data from 

[junit] Loading data to table default.testhivedrivertable
[junit] POSTHOOK: query: load data local inpath 
'
 into table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select * from testhivedrivertable limit 10
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/hudson/hive_2012-07-25_15-14-16_163_8038277319393064641/-mr-1
[junit] POSTHOOK: query: select * from testhivedrivertable limit 10
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/hudson/hive_2012-07-25_15-14-16_163_8038277319393064641/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=
[junit] Hive history 
file=
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] PO

Re: Custom UDF can not handle constant string parameter

[Edward Capriolo]

Compare your code to the ascii udf and then make your code more ascii like :)

On Wed, Jul 25, 2012 at 5:14 PM, Ablimit Aji  wrote:
> Thanks Edward.
>
> However the following statement works without any problem, and my UDF is no
> different from ascii UDF.
>
> SELECT ascii("A") FROM markup LIMIT 5;
>
> How can I write the correct UDF to achieve original simplicity ?
>
>
>
> On Wed, Jul 25, 2012 at 5:02 PM, Edward Capriolo wrote:
>
>> It depends on how you wrote the UDF. One trick you can do trick hive
>> into thinking it is non-constant: select a column of 0 length and then
>> concat it.
>>
>> SELECT concat(substring(colx),0,0), myUDF("constant string") ) from
>>
>> Now its not constant!
>>
>> On Wed, Jul 25, 2012 at 4:52 PM, Ablimit Aji  wrote:
>> > Hi,
>> >
>> > I wrote several UDF to handle a string typed column. However, when I
>> feed a
>> > constant value to the UDF, it's not recognizing it.
>> > What went wrong ? How can I fix it ?
>> >
>> > Example SQL: SELECT MBB("Polygon ((0 1, 1 1, 0 1, 0 0))") FROM myTable
>> > limit 5;
>> >
>> > MBB(Text) returns Text value
>> >
>> > However the error I get is like this:
>> >
>> > FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments
>> '"Polygon
>> > ((0 1, 1 1, 0 1, 0 0))"':
>> org.apache.hadoop.hive.ql.metadata.HiveException:
>> > Unable to execute method public org.apache.hadoop.io.Text
>> > edu.emory.cci.hive.udf.UDFMBB.evaluate(org.apache.hadoop.io.Text)  on
>> > object edu.emory.cci.hive.udf.UDFMBB@213e2b4a of class
>> > edu.emory.cci.hive.udf.UDFMBB with arguments {Polygon ((0 1, 1 1, 0 1, 0
>> > 0)):org.apache.hadoop.io.Text} of size 1
>>

Re: Custom UDF can not handle constant string parameter

[Ablimit Aji]

Thanks Edward.

However the following statement works without any problem, and my UDF is no
different from ascii UDF.

SELECT ascii("A") FROM markup LIMIT 5;

How can I write the correct UDF to achieve original simplicity ?



On Wed, Jul 25, 2012 at 5:02 PM, Edward Capriolo wrote:

> It depends on how you wrote the UDF. One trick you can do trick hive
> into thinking it is non-constant: select a column of 0 length and then
> concat it.
>
> SELECT concat(substring(colx),0,0), myUDF("constant string") ) from
>
> Now its not constant!
>
> On Wed, Jul 25, 2012 at 4:52 PM, Ablimit Aji  wrote:
> > Hi,
> >
> > I wrote several UDF to handle a string typed column. However, when I
> feed a
> > constant value to the UDF, it's not recognizing it.
> > What went wrong ? How can I fix it ?
> >
> > Example SQL: SELECT MBB("Polygon ((0 1, 1 1, 0 1, 0 0))") FROM myTable
> > limit 5;
> >
> > MBB(Text) returns Text value
> >
> > However the error I get is like this:
> >
> > FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments
> '"Polygon
> > ((0 1, 1 1, 0 1, 0 0))"':
> org.apache.hadoop.hive.ql.metadata.HiveException:
> > Unable to execute method public org.apache.hadoop.io.Text
> > edu.emory.cci.hive.udf.UDFMBB.evaluate(org.apache.hadoop.io.Text)  on
> > object edu.emory.cci.hive.udf.UDFMBB@213e2b4a of class
> > edu.emory.cci.hive.udf.UDFMBB with arguments {Polygon ((0 1, 1 1, 0 1, 0
> > 0)):org.apache.hadoop.io.Text} of size 1
>

[jira] [Commented] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422614#comment-13422614
 ] 

Edward Capriolo commented on HIVE-3068:
---

I will look this over tonight.

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
> Attachments: HIVE-3068.2.patch.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422612#comment-13422612
 ] 

Andrew Chalfant commented on HIVE-3068:
---

New revision is here: https://reviews.facebook.net/D4341

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
> Attachments: HIVE-3068.2.patch.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Custom UDF can not handle constant string parameter

[Edward Capriolo]

It depends on how you wrote the UDF. One trick you can do trick hive
into thinking it is non-constant: select a column of 0 length and then
concat it.

SELECT concat(substring(colx),0,0), myUDF("constant string") ) from

Now its not constant!

On Wed, Jul 25, 2012 at 4:52 PM, Ablimit Aji  wrote:
> Hi,
>
> I wrote several UDF to handle a string typed column. However, when I feed a
> constant value to the UDF, it's not recognizing it.
> What went wrong ? How can I fix it ?
>
> Example SQL: SELECT MBB("Polygon ((0 1, 1 1, 0 1, 0 0))") FROM myTable
> limit 5;
>
> MBB(Text) returns Text value
>
> However the error I get is like this:
>
> FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '"Polygon
> ((0 1, 1 1, 0 1, 0 0))"': org.apache.hadoop.hive.ql.metadata.HiveException:
> Unable to execute method public org.apache.hadoop.io.Text
> edu.emory.cci.hive.udf.UDFMBB.evaluate(org.apache.hadoop.io.Text)  on
> object edu.emory.cci.hive.udf.UDFMBB@213e2b4a of class
> edu.emory.cci.hive.udf.UDFMBB with arguments {Polygon ((0 1, 1 1, 0 1, 0
> 0)):org.apache.hadoop.io.Text} of size 1

[jira] [Updated] (HIVE-3068) Add ability to export table metadata as JSON on table drop

[Andrew Chalfant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Chalfant updated HIVE-3068:
--

Attachment: HIVE-3068.2.patch.txt

Updated patch

> Add ability to export table metadata as JSON on table drop
> --
>
> Key: HIVE-3068
> URL: https://issues.apache.org/jira/browse/HIVE-3068
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Serializers/Deserializers
>Reporter: Andrew Chalfant
>Assignee: Andrew Chalfant
>Priority: Minor
>  Labels: features, newbie
> Attachments: HIVE-3068.2.patch.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When a table is dropped, the contents go to the users trash but the metadata 
> is lost. It would be super neat to be able to save the metadata as well so 
> that tables could be trivially re-instantiated via thrift.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Custom UDF can not handle constant string parameter

[Ablimit Aji]

Hi,

I wrote several UDF to handle a string typed column. However, when I feed a
constant value to the UDF, it's not recognizing it.
What went wrong ? How can I fix it ?

Example SQL: SELECT MBB("Polygon ((0 1, 1 1, 0 1, 0 0))") FROM myTable
limit 5;

MBB(Text) returns Text value

However the error I get is like this:

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '"Polygon
((0 1, 1 1, 0 1, 0 0))"': org.apache.hadoop.hive.ql.metadata.HiveException:
Unable to execute method public org.apache.hadoop.io.Text
edu.emory.cci.hive.udf.UDFMBB.evaluate(org.apache.hadoop.io.Text)  on
object edu.emory.cci.hive.udf.UDFMBB@213e2b4a of class
edu.emory.cci.hive.udf.UDFMBB with arguments {Polygon ((0 1, 1 1, 0 1, 0
0)):org.apache.hadoop.io.Text} of size 1

[jira] [Commented] (HIVE-3025) Fix Hive ARCHIVE command on 0.22 and 0.23

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422589#comment-13422589
 ] 

Vikram Dixit K commented on HIVE-3025:
--

After digging more into this with @hashutosh's help, we see the following 
issues:

1. The hadoop archive command line has changed.
2. There is no way in the current set of commands supported by hive for a user 
to specify a parent directory for the archive.
3. The api createHadoopArchive in all shims is the same which is 
counter-intuitive.

The hadoop archive command has changed between versions 0.20 and 
0.20S/1.0/0.23. There is a compulsory command line parameter -p that is 
required in the latter versions. Since these versions are driving the same 
command line as 0.20 (without the -p), they fail. This needs to be fixed in the 
createHadoopArchive api.

The createHadoopArchive has the issue that it checks 
hive.archive.har.parentdir.settable. The user, in the current set of commands 
available, has no way of setting a parent directory for the creation of the 
archive. So, in the future when that ability is added, we need to revisit the 
createHadoopArchive api itself or derive it from conf.

The createHadoopArchive api is the same across all the shims, i.e. 
Hadoop20Shims.java and the HadoopShimsSecure.java have the exact same 
implementation of this api which is counter-intuitive considering the shims are 
supposed to be specific for versions of hadoop. 

So, I propose at this time, we should fix the createHadoopArchive in the 
HadoopShimsSecure to adhere to the new command line expected by those versions 
of Hadoop. We should also fix the Hadoop20Shims api to not worry about the -p 
parameter since it cannot use it. 

Please let me know if I am missing something.

> Fix Hive ARCHIVE command on 0.22 and 0.23
> -
>
> Key: HIVE-3025
> URL: https://issues.apache.org/jira/browse/HIVE-3025
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.9.0
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Attachments: HIVE-3025.D3195.1.patch
>
>
> archive.q and archive_multi.q fail when Hive is run on top of Hadoop 0.22 or 
> 0.23.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: HBaseSerDe

[Ted Yu]

The ctor is used in TestHBaseSerDe.java

So maybe change it to package private ?

On Wed, Jul 25, 2012 at 12:43 PM, kulkarni.swar...@gmail.com <
kulkarni.swar...@gmail.com> wrote:

> While going through some code for HBase/Hive Integration, I came across
> this constructor:
>
> public HBaseSerDe() throws SerDeException {
>
> }
>
> Basically, the constructor is doing nothing but throwing an exception.
> Problem is fixing this now will be a non-passive change.
>
> I couldn't really find an obvious reason for this to be there. Are there
> any objections if I file a JIRA to remove this constructor?
> --
> Swarnim
>

[jira] [Commented] (HIVE-3153) Release codecs and output streams between flushes of RCFile

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422565#comment-13422565
 ] 

Ashutosh Chauhan commented on HIVE-3153:


Couple of questions:

* I see that you have moved initialization of codecs and output stream from 
init() to flushRecords(). How is it helping in decreasing the memory footprint?
* Also, can you explain which kind of queries will gain from this decrease of 
memory footprint?

> Release codecs and output streams between flushes of RCFile
> ---
>
> Key: HIVE-3153
> URL: https://issues.apache.org/jira/browse/HIVE-3153
> Project: Hive
>  Issue Type: Improvement
>  Components: Compression
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: hive-3153.patch
>
>
> Currently, RCFile writer holds a compression codec per a file and a 
> compression output stream per a column. Especially for queries that use 
> dynamic partitions this quickly consumes a lot of memory.
> I'd like flushRecords to get a codec from the pool and create the compression 
> output stream in flushRecords.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3276) optimize union sub-queries

[Nadeem Moidu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422533#comment-13422533
 ] 

Nadeem Moidu commented on HIVE-3276:


A wiki page has been added for the same here 
https://cwiki.apache.org/confluence/display/Hive/Union+Optimization . I have 
started work on this and will be uploading the patch soon. Feel free to give 
any feedback on the same. Thanks.

> optimize union sub-queries
> --
>
> Key: HIVE-3276
> URL: https://issues.apache.org/jira/browse/HIVE-3276
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Nadeem Moidu
>
> It might be a good idea to optimize simple union queries containing 
> map-reduce jobs in at least one of the sub-qeuries.
> For eg:
> a query like:
> insert overwrite table T1 partition P1
> select * from 
> (
>   subq1
> union all
>   subq2
> ) u;
> today creates 3 map-reduce jobs, one for subq1, another for subq2 and 
> the final one for the union. 
> It might be a good idea to optimize this. Instead of creating the union 
> task, it might be simpler to create a move task (or something like a move
> task), where the outputs of the two sub-queries will be moved to the final 
> directory. This can easily extend to more than 2 sub-queries in the union.
> This is only useful if there is a select * followed by filesink after the
> union. This can be independently useful, and also be used to optimize the
> skewed joins https://cwiki.apache.org/Hive/skewed-join-optimization.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (HIVE-3276) optimize union sub-queries

[Nadeem Moidu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-3276 started by Nadeem Moidu.

> optimize union sub-queries
> --
>
> Key: HIVE-3276
> URL: https://issues.apache.org/jira/browse/HIVE-3276
> Project: Hive
>  Issue Type: Bug
>Reporter: Namit Jain
>Assignee: Nadeem Moidu
>
> It might be a good idea to optimize simple union queries containing 
> map-reduce jobs in at least one of the sub-qeuries.
> For eg:
> a query like:
> insert overwrite table T1 partition P1
> select * from 
> (
>   subq1
> union all
>   subq2
> ) u;
> today creates 3 map-reduce jobs, one for subq1, another for subq2 and 
> the final one for the union. 
> It might be a good idea to optimize this. Instead of creating the union 
> task, it might be simpler to create a move task (or something like a move
> task), where the outputs of the two sub-queries will be moved to the final 
> directory. This can easily extend to more than 2 sub-queries in the union.
> This is only useful if there is a select * followed by filesink after the
> union. This can be independently useful, and also be used to optimize the
> skewed joins https://cwiki.apache.org/Hive/skewed-join-optimization.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-1653) Ability to enforce correct stats

[Shreepadma Venugopalan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422486#comment-13422486
 ] 

Shreepadma Venugopalan commented on HIVE-1653:
--

@Namit: Added comments on phabricator. Thanks!

> Ability to enforce correct stats
> 
>
> Key: HIVE-1653
> URL: https://issues.apache.org/jira/browse/HIVE-1653
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Namit Jain
>
> This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361.
> If one of the mappers/reducers cannot publish stats, it may lead to wrong 
> aggregated stats.
> There should be a way to avoid this - at the least, a configuration variable 
> which fails the 
> task if stats cannot be published

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2101) mapjoin sometimes gives wrong results if there is a filter in the on condition

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422468#comment-13422468
 ] 

Edward Capriolo commented on HIVE-2101:
---

Which is pretty major because hive now auto-map joins most queries.

> mapjoin sometimes gives wrong results if there is a filter in the on condition
> --
>
> Key: HIVE-2101
> URL: https://issues.apache.org/jira/browse/HIVE-2101
> Project: Hive
>  Issue Type: Bug
>Reporter: He Yongqiang
>
> "SELECT / * + mapjoin(src1, src2) * / * FROM src src1 RIGHT OUTER JOIN src 
> src2 ON (src1.key = src2.key AND src1.key < 10 AND src2.key > 10) JOIN src 
> src3 ON (src2.key = src3.key AND src3.key < 10) SORT BY src1.key, src1.value, 
> src2.key, src2.value, src3.key, src3.value;" will give wrong results in 
> today's hive

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2101) mapjoin sometimes gives wrong results if there is a filter in the on condition

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422411#comment-13422411
 ] 

Namit Jain commented on HIVE-2101:
--

This is different from the bug above.

The basic problem is that the filters from the join condition are not executed 
for map-joins.

> mapjoin sometimes gives wrong results if there is a filter in the on condition
> --
>
> Key: HIVE-2101
> URL: https://issues.apache.org/jira/browse/HIVE-2101
> Project: Hive
>  Issue Type: Bug
>Reporter: He Yongqiang
>
> "SELECT / * + mapjoin(src1, src2) * / * FROM src src1 RIGHT OUTER JOIN src 
> src2 ON (src1.key = src2.key AND src1.key < 10 AND src2.key > 10) JOIN src 
> src3 ON (src2.key = src3.key AND src3.key < 10) SORT BY src1.key, src1.value, 
> src2.key, src2.value, src3.key, src3.value;" will give wrong results in 
> today's hive

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422396#comment-13422396
 ] 

Swarnim Kulkarni commented on HIVE-2599:


For all parts of your key? yes. For binary, using ":key#b" in the columns 
mapping should work. 

> Support Composit/Compound Keys with HBaseStorageHandler
> ---
>
> Key: HIVE-2599
> URL: https://issues.apache.org/jira/browse/HIVE-2599
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.8.0
>Reporter: Hans Uhlig
>
> It would be really nice for hive to be able to understand composite keys from 
> an underlying HBase schema. Currently we have to store key fields twice to be 
> able to both key and make data available. I noticed John Sichi mentioned in 
> HIVE-1228 that this would be a separate issue but I cant find any follow up. 
> How feasible is this in the HBaseStorageHandler?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3279) Table schema not being copied to Partitions with no columns

[Travis Crawford (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422385#comment-13422385
 ] 

Travis Crawford commented on HIVE-3279:
---

Differential review: https://reviews.facebook.net/D4329

> Table schema not being copied to Partitions with no columns
> ---
>
> Key: HIVE-3279
> URL: https://issues.apache.org/jira/browse/HIVE-3279
> Project: Hive
>  Issue Type: Bug
>Reporter: Travis Crawford
>Assignee: Travis Crawford
> Attachments: HIVE-3279_serde_reported_partition_schema.1.patch
>
>
> Hive has a feature where {{Partition}}'s without any defined columns use the 
> {{Table}} schema. This happens in 
> {{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
> {code}
> // set default if columns are not set
> if (tPartition.getSd().getCols() == null) {
>   if (table.getCols() != null) {
> tPartition.getSd().setCols(table.getCols());
>   }
> }
> {code}
> There's an issue though, because 
> {{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
>  initializes cols to an empty array, which of course is not null, causing the 
> above feature to not work as expected.
> I'm not sure of the fix - is there a case where cols can indeed be null? I 
> think the best thing to do here is:
> {code}
> -if (tPartition.getSd().getCols() == null) {
> +if (tPartition.getSd().getCols() == null || 
> tPartition.getSd().getCols().size() == 0) {
> {code}
> Thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3279) Table schema not being copied to Partitions with no columns

[Travis Crawford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Travis Crawford updated HIVE-3279:
--

Status: Patch Available  (was: Open)

> Table schema not being copied to Partitions with no columns
> ---
>
> Key: HIVE-3279
> URL: https://issues.apache.org/jira/browse/HIVE-3279
> Project: Hive
>  Issue Type: Bug
>Reporter: Travis Crawford
>Assignee: Travis Crawford
> Attachments: HIVE-3279_serde_reported_partition_schema.1.patch
>
>
> Hive has a feature where {{Partition}}'s without any defined columns use the 
> {{Table}} schema. This happens in 
> {{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
> {code}
> // set default if columns are not set
> if (tPartition.getSd().getCols() == null) {
>   if (table.getCols() != null) {
> tPartition.getSd().setCols(table.getCols());
>   }
> }
> {code}
> There's an issue though, because 
> {{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
>  initializes cols to an empty array, which of course is not null, causing the 
> above feature to not work as expected.
> I'm not sure of the fix - is there a case where cols can indeed be null? I 
> think the best thing to do here is:
> {code}
> -if (tPartition.getSd().getCols() == null) {
> +if (tPartition.getSd().getCols() == null || 
> tPartition.getSd().getCols().size() == 0) {
> {code}
> Thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3279) Table schema not being copied to Partitions with no columns

[Travis Crawford (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Travis Crawford updated HIVE-3279:
--

Attachment: HIVE-3279_serde_reported_partition_schema.1.patch

> Table schema not being copied to Partitions with no columns
> ---
>
> Key: HIVE-3279
> URL: https://issues.apache.org/jira/browse/HIVE-3279
> Project: Hive
>  Issue Type: Bug
>Reporter: Travis Crawford
>Assignee: Travis Crawford
> Attachments: HIVE-3279_serde_reported_partition_schema.1.patch
>
>
> Hive has a feature where {{Partition}}'s without any defined columns use the 
> {{Table}} schema. This happens in 
> {{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
> {code}
> // set default if columns are not set
> if (tPartition.getSd().getCols() == null) {
>   if (table.getCols() != null) {
> tPartition.getSd().setCols(table.getCols());
>   }
> }
> {code}
> There's an issue though, because 
> {{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
>  initializes cols to an empty array, which of course is not null, causing the 
> above feature to not work as expected.
> I'm not sure of the fix - is there a case where cols can indeed be null? I 
> think the best thing to do here is:
> {code}
> -if (tPartition.getSd().getCols() == null) {
> +if (tPartition.getSd().getCols() == null || 
> tPartition.getSd().getCols().size() == 0) {
> {code}
> Thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #86

[Apache Jenkins Server]

See 


--
[...truncated 10115 lines...]
 [echo] Project: odbc
 [copy] Warning: 

 does not exist.

ivy-resolve-test:
 [echo] Project: odbc

ivy-retrieve-test:
 [echo] Project: odbc

compile-test:
 [echo] Project: odbc

create-dirs:
 [echo] Project: serde
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: serde

ivy-init-settings:
 [echo] Project: serde

ivy-resolve:
 [echo] Project: serde
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: serde

dynamic-serde:

compile:
 [echo] Project: serde

ivy-resolve-test:
 [echo] Project: serde

ivy-retrieve-test:
 [echo] Project: serde

compile-test:
 [echo] Project: serde
[javac] Compiling 26 source files to 

[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.

create-dirs:
 [echo] Project: service
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: service

ivy-init-settings:
 [echo] Project: service

ivy-resolve:
 [echo] Project: service
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: service

compile:
 [echo] Project: service

ivy-resolve-test:
 [echo] Project: service

ivy-retrieve-test:
 [echo] Project: service

compile-test:
 [echo] Project: service
[javac] Compiling 2 source files to 


test:
 [echo] Project: hive

test-shims:
 [echo] Project: hive

test-conditions:
 [echo] Project: shims

gen-test:
 [echo] Project: shims

create-dirs:
 [echo] Project: shims
 [copy] Warning: 

 does not exist.

init:
 [echo] Project: shims

ivy-init-settings:
 [echo] Project: shims

ivy-resolve:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 

[ivy:report] Processing 

 to 


ivy-retrieve:
 [echo] Project: shims

compile:
 [echo] Project: shims
 [echo] Building shims 0.20

build_shims:
 [echo] Project: shims
 [echo] Compiling 

 against hadoop 0.20.2 
(

ivy-init-settings:
 [echo] Project: shims

ivy-resolve-hadoop-shim:
 [echo] Project: shims
[ivy:resolve] :: loading settings :: file = 


ivy-retrieve-hadoop-shim:
 [echo] Project: shims
 [echo] Building shims 0.20S

build_shims:
 [echo] Project: shims
 [echo] Compiling 

[jira] [Created] (HIVE-3300) LOAD DATA INPATH fails if a hdfs file with same name is added to table

[Bejoy KS (JIRA)]

Bejoy KS created HIVE-3300:
--

 Summary: LOAD DATA INPATH fails if a hdfs file with same name is 
added to table
 Key: HIVE-3300
 URL: https://issues.apache.org/jira/browse/HIVE-3300
 Project: Hive
  Issue Type: Bug
  Components: Import/Export
Affects Versions: 0.9.0
 Environment: ubuntu linux, hadoop 1.0.3, hive 0.9
Reporter: Bejoy KS


If we are loading data from local fs to hive tables using 'LOAD DATA LOCAL 
INPATH' and if a file with the same name exists in the table's location then 
the new file will be suffixed by *_copy_1.

But if we do the 'LOAD DATA INPATH'  for a file in hdfs then there is no rename 
happening but just a move task is getting triggered. Since a file with same 
name exists in same hdfs location, hadoop fs move operation throws an error.


hive> LOAD DATA INPATH '/userdata/bejoy/site.txt' INTO TABLE test.site;
Loading data to table test.site
Failed with exception null
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MoveTask
hive> 



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler

[Hans Uhlig (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422361#comment-13422361
 ] 

Hans Uhlig commented on HIVE-2599:
--

Do I need to do this for all fields then? Any suggestions for binary?

> Support Composit/Compound Keys with HBaseStorageHandler
> ---
>
> Key: HIVE-2599
> URL: https://issues.apache.org/jira/browse/HIVE-2599
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.8.0
>Reporter: Hans Uhlig
>
> It would be really nice for hive to be able to understand composite keys from 
> an underlying HBase schema. Currently we have to store key fields twice to be 
> able to both key and make data available. I noticed John Sichi mentioned in 
> HIVE-1228 that this would be a separate issue but I cant find any follow up. 
> How feasible is this in the HBaseStorageHandler?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

sorry if i was way too graphical...

[Tonci Korsano]

i just wanted to help...

please, click on these images to get information about each image web page.

kind regards,

tonko.

Hive-trunk-h0.21 - Build # 1567 - Still Failing

[Apache Jenkins Server]

Changes for Build #1566
[cws] HIVE-3249 [jira] Upgrade guava to 11.0.2
(Zhenxiao Luo via Carl Steinbach)

Summary:
HIVE-3249: Upgrade guava to 11.0.2

Hadoop has upgraded to this new version of Guava. We should, too, so we don't 
have compatibility issues running on Hadoop 2.0+

currently, hive is using guava-r09.jar
we should update it to guava-11.0.2.jar

Test Plan: EMPTY

Reviewers: JIRA

Differential Revision: https://reviews.facebook.net/D4083


Changes for Build #1567
[hashutosh] HIVE-3126 : Generate & build the velocity based Hive tests on 
windows by fixing the path issues (Kanna Karanam via Ashutosh Chauhan)

[kevinwilfong] HIVE-3295. HIVE-3128 introduced bug causing dynamic partitioning 
to fail. (kevinwilfong reviewed by njain, ashutoshc)




1 tests failed.
FAILED:  
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1

Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... 
-Dtest.silent=false" to get more logs.

Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
more logs.
at junit.framework.Assert.fail(Assert.java:50)
at 
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)




The Apache Jenkins build system has built Hive-trunk-h0.21 (build 
#$BUILD_NUMBER)

Status: Still Failing

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1567/ to 
view the results.

[jira] [Commented] (HIVE-3298) select 1+1 from dual; if dual is an empty table this statement will return no result

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422325#comment-13422325
 ] 

Edward Capriolo commented on HIVE-3298:
---

That is what it is supposed to do. If you are selecting a table with no rows 
you are going to get no rows. Consider looking at the duelinputformat I 
created. It always returns a single row regardless of the data.

https://github.com/edwardcapriolo/DualInputFormat

> select 1+1 from dual; if dual is an empty table this statement will return no 
> result 
> -
>
> Key: HIVE-3298
> URL: https://issues.apache.org/jira/browse/HIVE-3298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.8.1
> Environment: Linux  3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 
> 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.8.1
>Reporter: caofangkun
>Priority: Minor
>
> hive>drop table if exists dual;
> hive>create table dual (dummy string);
> hive>select 1+1 from dual;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201206081154_0458, Tracking URL = 
> http://dwtest-93-61:50030/jobdetails.jsp?jobid=job_201206081154_0458
> Kill Command = /home/hive/hadoop-0.20.2-cdh3u0/bin/hadoop job  
> -Dmapred.job.tracker=dwtest-93-61:9001 -kill job_201206081154_0458
> Hadoop job information for Stage-1: number of mappers: 0; number of reducers: > 0
> 2012-07-25 16:58:15,793 Stage-1 map = 0%,  reduce = 0%
> 2012-07-25 16:58:17,817 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201206081154_0458
> MapReduce Jobs Launched: 
> Job 0:  HDFS Read: 0 HDFS Write: 0 SUCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 6.607 seconds

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-3298) select 1+1 from dual; if dual is an empty table this statement will return no result

[Edward Capriolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Edward Capriolo resolved HIVE-3298.
---

Resolution: Won't Fix

> select 1+1 from dual; if dual is an empty table this statement will return no 
> result 
> -
>
> Key: HIVE-3298
> URL: https://issues.apache.org/jira/browse/HIVE-3298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.8.1
> Environment: Linux  3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 
> 20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
> java version "1.6.0_25"
> hadoop-0.20.2-cdh3u0
> hive-0.8.1
>Reporter: caofangkun
>Priority: Minor
>
> hive>drop table if exists dual;
> hive>create table dual (dummy string);
> hive>select 1+1 from dual;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201206081154_0458, Tracking URL = 
> http://dwtest-93-61:50030/jobdetails.jsp?jobid=job_201206081154_0458
> Kill Command = /home/hive/hadoop-0.20.2-cdh3u0/bin/hadoop job  
> -Dmapred.job.tracker=dwtest-93-61:9001 -kill job_201206081154_0458
> Hadoop job information for Stage-1: number of mappers: 0; number of reducers: > 0
> 2012-07-25 16:58:15,793 Stage-1 map = 0%,  reduce = 0%
> 2012-07-25 16:58:17,817 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201206081154_0458
> MapReduce Jobs Launched: 
> Job 0:  HDFS Read: 0 HDFS Write: 0 SUCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 6.607 seconds

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422324#comment-13422324
 ] 

Hudson commented on HIVE-3295:
--

Integrated in Hive-trunk-h0.21 #1567 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1567/])
HIVE-3295. HIVE-3128 introduced bug causing dynamic partitioning to fail. 
(kevinwilfong reviewed by njain, ashutoshc) (Revision 1365460)

 Result = FAILURE
kevinwilfong : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365460
Files : 
* 
/hive/trunk/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
* /hive/trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java


> HIVE-3128 introduced bug causing dynamic partitioning to fail
> -
>
> Key: HIVE-3295
> URL: https://issues.apache.org/jira/browse/HIVE-3295
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
>Priority: Blocker
> Fix For: 0.10.0
>
> Attachments: HIVE-3295.1.patch.txt, HIVE-3295.2.patch.txt
>
>
> HIVE-3128 introduced a new commons-compress jar and imports classes from it 
> in FileUtils.java  The FileUtils class is accessed by dynamic partitioning in 
> the map reduce cluster where the jar is not available, causing the query to 
> fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3128) use commons-compress instead of forking tar process

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422322#comment-13422322
 ] 

Hudson commented on HIVE-3128:
--

Integrated in Hive-trunk-h0.21 #1567 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1567/])
HIVE-3295. HIVE-3128 introduced bug causing dynamic partitioning to fail. 
(kevinwilfong reviewed by njain, ashutoshc) (Revision 1365460)

 Result = FAILURE
kevinwilfong : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365460
Files : 
* 
/hive/trunk/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
* /hive/trunk/common/src/java/org/apache/hadoop/hive/common/FileUtils.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java


> use commons-compress instead of forking tar process
> ---
>
> Key: HIVE-3128
> URL: https://issues.apache.org/jira/browse/HIVE-3128
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Query Processor
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
> Fix For: 0.10.0
>
> Attachments: HIVE-3128.1.patch.txt, HIVE-3128.2.patch.txt
>
>
> TAR tool doesn’t exist by default on windows systems so use the CAB files on 
> windows

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3126) Generate & build the velocity based Hive tests on windows by fixing the path issues

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422323#comment-13422323
 ] 

Hudson commented on HIVE-3126:
--

Integrated in Hive-trunk-h0.21 #1567 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1567/])
HIVE-3126 : Generate & build the velocity based Hive tests on windows by 
fixing the path issues (Kanna Karanam via Ashutosh Chauhan) (Revision 1365467)

 Result = FAILURE
hashutosh : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365467
Files : 
* /hive/trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java
* /hive/trunk/build-common.xml
* /hive/trunk/build.xml
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* 
/hive/trunk/contrib/src/test/org/apache/hadoop/hive/contrib/mr/TestGenericMR.java
* /hive/trunk/data/conf/hive-site.xml
* /hive/trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm
* /hive/trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm
* /hive/trunk/odbc/build.xml
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
* 
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
* /hive/trunk/ql/src/test/templates/TestCliDriver.vm
* /hive/trunk/ql/src/test/templates/TestNegativeCliDriver.vm
* /hive/trunk/ql/src/test/templates/TestParse.vm
* /hive/trunk/ql/src/test/templates/TestParseNegative.vm
* /hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java
* 
/hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java
* /hive/trunk/testutils/hadoop.cmd


> Generate & build the velocity based Hive tests on windows by fixing the path 
> issues
> ---
>
> Key: HIVE-3126
> URL: https://issues.apache.org/jira/browse/HIVE-3126
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 0.9.0, 0.10.0, 0.9.1
>Reporter: Kanna Karanam
>Assignee: Kanna Karanam
>  Labels: Windows, test
> Fix For: 0.10.0
>
> Attachments: HIVE-3126.1.patch.txt, HIVE-3126.10.patch.txt, 
> HIVE-3126.2.patch.txt, HIVE-3126.3.patch.txt, HIVE-3126.4.patch.txt, 
> HIVE-3126.5.patch.txt, HIVE-3126.6.patch.txt, HIVE-3126.7.patch.txt, 
> HIVE-3126.8.patch.txt, HIVE-3126.9.patch.txt
>
>
> 1)Escape the backward slash in Canonical Path if unit test runs on windows.
> 2)Diff comparison – 
>  a.   Ignore the extra spacing on windows
>  b.   Ignore the different line endings on windows & Unix
>  c.   Convert the file paths to windows specific. (Handle spaces 
> etc..)
> 3)Set the right file scheme & class path separators while invoking the junit 
> task from 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3295) HIVE-3128 introduced bug causing dynamic partitioning to fail

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422309#comment-13422309
 ] 

Edward Capriolo commented on HIVE-3295:
---

you live and you learn. :)

> HIVE-3128 introduced bug causing dynamic partitioning to fail
> -
>
> Key: HIVE-3295
> URL: https://issues.apache.org/jira/browse/HIVE-3295
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0
>Reporter: Kevin Wilfong
>Assignee: Kevin Wilfong
>Priority: Blocker
> Fix For: 0.10.0
>
> Attachments: HIVE-3295.1.patch.txt, HIVE-3295.2.patch.txt
>
>
> HIVE-3128 introduced a new commons-compress jar and imports classes from it 
> in FileUtils.java  The FileUtils class is accessed by dynamic partitioning in 
> the map reduce cluster where the jar is not available, causing the query to 
> fail.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3299) UDF DAYNAME(date) to HIVE

[Bennie Schut (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422296#comment-13422296
 ] 

Bennie Schut commented on HIVE-3299:


Currently you can get this trough a unix_timestamp function so if you don't 
want to wait for an implementation do something like this.
select from_unixtime(unix_timestamp() , '') from dummytable limit 1;
or
select from_unixtime(unix_timestamp('2012-07-25 00:00:00'), '') from 
dummytable limit 1;

However simply having a nice udf for it would be nice, like they have in mysql: 
http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html#function_dayname

> UDF  DAYNAME(date) to HIVE 
> ---
>
> Key: HIVE-3299
> URL: https://issues.apache.org/jira/browse/HIVE-3299
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Namitha Babychan
> Fix For: 0.10.0
>
>
> Current releases of Hive lacks a function which would return the day name 
> corresponding to a date / timestamp value which might be a part of a column.  
>  
> The function -DAYNAME (date) would return the day name from a date / 
> timestamp or column which would be useful while using HiveQL. This would find 
> its use  in various business sectors like retail, which would help in  
> identifying the trends and sales datails for a particular weekday for entire 
> year,month or week.
> Functionality :-
> Function Name: DAYNAME (date)
>
> Returns the name of the weekday for date. 
> Example: hive> SELECT DAYNAME('2012-07-25');
>-> 'Wednesday'
> Usage :-
> Case 1 : To find DAY NAME corresponding to a particular date 
> hive> SELECT DAYNAME('2012-07-25');
>-> 'Wednesday'
> Case 2 : To query a table to find details based on a particular day name
> Table :-
> date  |item id|store id|value|unit|price
> 01/07/2012|110001|00003|0.99|1.00|0.99
> 02/07/2012|110001|00008|0.99|0.00|0.00
> 03/07/2012|110001|00009|0.99|0.00|0.00
> 04/07/2012|110001|001112002|0.99|0.00|0.00
> 05/07/2012|110001|001112003|0.99|0.00|0.00
> 06/07/2012|110001|001112006|0.99|1.00|0.99
> 07/07/2012|110001|001112007|0.99|0.00|0.00
> 08/07/2012|110001|001112008|0.99|0.00|0.00
> 09/07/2012|110001|001112009|0.99|0.00|0.00
> 10/07/2012|110001|001112010|0.99|0.00|0.00
> 11/07/2012|110001|001113003|0.99|0.00|0.00
> 12/07/2012|110001|001113006|0.99|0.00|0.00
> 13/07/2012|110001|001113008|0.99|0.00|0.00
> 14/07/2012|110001|001113010|0.99|0.00|0.00
> 15/07/2012|110001|001114002|0.99|0.00|0.00
> 16/07/2012|110001|001114004|0.99|1.00|0.99
> 17/07/2012|110001|001114005|0.99|0.00|0.00
> 18/07/2012|110001|001121004|0.99|0.00|0.00
> Query : select * from sales where dayname(date)='wednesday';
> Result :-
> 04/07/2012|110001|001112002|0.99|0.00|0.00
> 11/07/2012|110001|001113003|0.99|0.00|0.00
> 18/07/2012|110001|001121004|0.99|0.00|0.00

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler

[Swarnim Kulkarni (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422237#comment-13422237
 ] 

Swarnim Kulkarni commented on HIVE-2599:


If your composite keys are delimited by a separator, here is a possible way to 
query them in hive:

{noformat}
CREATE EXTERNAL TABLE hbase_table_1(key struct, 
value string) 
ROW FORMAT DELIMITED
COLLECTION ITEMS TERMINATED BY '~'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,test-family:test-qual")
TBLPROPERTIES ("hbase.table.name" = "SIMPLE_TABLE");
{noformat}

Basically what this means is that the composite key here should be mapped to a 
struct and we specifying that the keys in the composite key are separated by a 
"~". After doing this, querying for the individual keys in the composite keys 
should be as simple as:

{noformat}
select key.a,key.b,key.c from hbase_table_1;
{noformat}

> Support Composit/Compound Keys with HBaseStorageHandler
> ---
>
> Key: HIVE-2599
> URL: https://issues.apache.org/jira/browse/HIVE-2599
> Project: Hive
>  Issue Type: Improvement
>  Components: HBase Handler
>Affects Versions: 0.8.0
>Reporter: Hans Uhlig
>
> It would be really nice for hive to be able to understand composite keys from 
> an underlying HBase schema. Currently we have to store key fields twice to be 
> able to both key and make data available. I noticed John Sichi mentioned in 
> HIVE-1228 that this would be a separate issue but I cant find any follow up. 
> How feasible is this in the HBaseStorageHandler?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-3299) UDF DAYNAME(date) to HIVE

[shalish (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shalish updated HIVE-3299:
--

Description: 
Current releases of Hive lacks a function which would return the day name 
corresponding to a date / timestamp value which might be a part of a column.   
The function -DAYNAME (date) would return the day name from a date / timestamp 
or column which would be useful while using HiveQL. This would find its use  in 
various business sectors like retail, which would help in  identifying the 
trends and sales datails for a particular weekday for entire year,month or week.


Functionality :-

Function Name: DAYNAME (date)
 
Returns the name of the weekday for date. 
Example: hive> SELECT DAYNAME('2012-07-25');
   -> 'Wednesday'

Usage :-

Case 1 : To find DAY NAME corresponding to a particular date 

hive> SELECT DAYNAME('2012-07-25');

   -> 'Wednesday'



Case 2 : To query a table to find details based on a particular day name
Table :-
date  |item id|store id|value|unit|price

01/07/2012|110001|00003|0.99|1.00|0.99

02/07/2012|110001|00008|0.99|0.00|0.00

03/07/2012|110001|00009|0.99|0.00|0.00

04/07/2012|110001|001112002|0.99|0.00|0.00

05/07/2012|110001|001112003|0.99|0.00|0.00

06/07/2012|110001|001112006|0.99|1.00|0.99

07/07/2012|110001|001112007|0.99|0.00|0.00

08/07/2012|110001|001112008|0.99|0.00|0.00

09/07/2012|110001|001112009|0.99|0.00|0.00

10/07/2012|110001|001112010|0.99|0.00|0.00

11/07/2012|110001|001113003|0.99|0.00|0.00

12/07/2012|110001|001113006|0.99|0.00|0.00

13/07/2012|110001|001113008|0.99|0.00|0.00

14/07/2012|110001|001113010|0.99|0.00|0.00

15/07/2012|110001|001114002|0.99|0.00|0.00

16/07/2012|110001|001114004|0.99|1.00|0.99

17/07/2012|110001|001114005|0.99|0.00|0.00

18/07/2012|110001|001121004|0.99|0.00|0.00




Query : select * from sales where dayname(date)='wednesday';


Result :-
04/07/2012|110001|001112002|0.99|0.00|0.00

11/07/2012|110001|001113003|0.99|0.00|0.00

18/07/2012|110001|001121004|0.99|0.00|0.00

  was:
Function Name: DAYNAME (date)
 
Returns the name of the weekday for date. 
Example: hive> SELECT DAYNAME('2012-07-25');
   -> 'Wednesday'


> UDF  DAYNAME(date) to HIVE 
> ---
>
> Key: HIVE-3299
> URL: https://issues.apache.org/jira/browse/HIVE-3299
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Namitha Babychan
> Fix For: 0.10.0
>
>
> Current releases of Hive lacks a function which would return the day name 
> corresponding to a date / timestamp value which might be a part of a column.  
>  
> The function -DAYNAME (date) would return the day name from a date / 
> timestamp or column which would be useful while using HiveQL. This would find 
> its use  in various business sectors like retail, which would help in  
> identifying the trends and sales datails for a particular weekday for entire 
> year,month or week.
> Functionality :-
> Function Name: DAYNAME (date)
>
> Returns the name of the weekday for date. 
> Example: hive> SELECT DAYNAME('2012-07-25');
>-> 'Wednesday'
> Usage :-
> Case 1 : To find DAY NAME corresponding to a particular date 
> hive> SELECT DAYNAME('2012-07-25');
>-> 'Wednesday'
> Case 2 : To query a table to find details based on a particular day name
> Table :-
> date  |item id|store id|value|unit|price
> 01/07/2012|110001|00003|0.99|1.00|0.99
> 02/07/2012|110001|00008|0.99|0.00|0.00
> 03/07/2012|110001|00009|0.99|0.00|0.00
> 04/07/2012|110001|001112002|0.99|0.00|0.00
> 05/07/2012|110001|001112003|0.99|0.00|0.00
> 06/07/2012|110001|001112006|0.99|1.00|0.99
> 07/07/2012|110001|001112007|0.99|0.00|0.00
> 08/07/2012|110001|001112008|0.99|0.00|0.00
> 09/07/2012|110001|001112009|0.99|0.00|0.00
> 10/07/2012|110001|001112010|0.99|0.00|0.00
> 11/07/2012|110001|001113003|0.99|0.00|0.00
> 12/07/2012|110001|001113006|0.99|0.00|0.00
> 13/07/2012|110001|001113008|0.99|0.00|0.00
> 14/07/2012|110001|001113010|0.99|0.00|0.00
> 15/07/2012|110001|001114002|0.99|0.00|0.00
> 16/07/2012|110001|001114004|0.99|1.00|0.99
> 17/07/2012|110001|001114005|0.99|0.00|0.00
> 18/07/2012|110001|001121004|0.99|0.00|0.00
> Query : select * from sales where dayname(date)='wednesday';
> Result :-
> 04/07/2012|110001|001112002|0.99|0.00|0.00
> 11/07/2012|110001|001113003|0.99|0.00|0.00
> 18/07/2012|110001|001121004|0.99|0.00|0.00

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/Conta

[jira] [Commented] (HIVE-3299) UDF DAYNAME(date) to HIVE

[shalish (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422156#comment-13422156
 ] 

shalish commented on HIVE-3299:
---

Current releases of Hive lacks a function which would return the day name 
corresponding to a date / timestamp value which might be a part of a column.   
The function -DAYNAME (date) would return the day name from a date / timestamp 
or column which would be useful while using HiveQL. This would find its use  in 
various business sectors like retail, which would help in  identifying the 
trends and sales datails for a particular weekday for entire year,month or week.


Functionality :-

Function Name: DAYNAME (date)
 
Returns the name of the weekday for date. 
Example: hive> SELECT DAYNAME('2012-07-25');
   -> 'Wednesday'

Usage :-

Case 1 : To find DAY NAME corresponding to a particular date 

hive> SELECT DAYNAME('2012-07-25');

   -> 'Wednesday'



Case 2 : To query a table to find details based on a particular day name
Table :-
date  |item id|store id|value|unit|price

01/07/2012|110001|00003|0.99|1.00|0.99

02/07/2012|110001|00008|0.99|0.00|0.00

03/07/2012|110001|00009|0.99|0.00|0.00

04/07/2012|110001|001112002|0.99|0.00|0.00

05/07/2012|110001|001112003|0.99|0.00|0.00

06/07/2012|110001|001112006|0.99|1.00|0.99

07/07/2012|110001|001112007|0.99|0.00|0.00

08/07/2012|110001|001112008|0.99|0.00|0.00

09/07/2012|110001|001112009|0.99|0.00|0.00

10/07/2012|110001|001112010|0.99|0.00|0.00

11/07/2012|110001|001113003|0.99|0.00|0.00

12/07/2012|110001|001113006|0.99|0.00|0.00

13/07/2012|110001|001113008|0.99|0.00|0.00

14/07/2012|110001|001113010|0.99|0.00|0.00

15/07/2012|110001|001114002|0.99|0.00|0.00

16/07/2012|110001|001114004|0.99|1.00|0.99

17/07/2012|110001|001114005|0.99|0.00|0.00

18/07/2012|110001|001121004|0.99|0.00|0.00




Query : select * from sales where dayname(date)='wednesday';


Result :-
04/07/2012|110001|001112002|0.99|0.00|0.00

11/07/2012|110001|001113003|0.99|0.00|0.00

18/07/2012|110001|001121004|0.99|0.00|0.00

> UDF  DAYNAME(date) to HIVE 
> ---
>
> Key: HIVE-3299
> URL: https://issues.apache.org/jira/browse/HIVE-3299
> Project: Hive
>  Issue Type: New Feature
>  Components: UDF
>Affects Versions: 0.9.0
>Reporter: Namitha Babychan
> Fix For: 0.10.0
>
>
> Function Name: DAYNAME (date)
>
> Returns the name of the weekday for date. 
> Example: hive> SELECT DAYNAME('2012-07-25');
>-> 'Wednesday'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3299) UDF DAYNAME(date) to HIVE

[Namitha Babychan (JIRA)]

Namitha Babychan created HIVE-3299:
--

 Summary: UDF  DAYNAME(date) to HIVE 
 Key: HIVE-3299
 URL: https://issues.apache.org/jira/browse/HIVE-3299
 Project: Hive
  Issue Type: New Feature
  Components: UDF
Affects Versions: 0.9.0
Reporter: Namitha Babychan
 Fix For: 0.10.0


Function Name: DAYNAME (date)
 
Returns the name of the weekday for date. 
Example: hive> SELECT DAYNAME('2012-07-25');
   -> 'Wednesday'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

small typo

[Bertrand Dechoux]

Hi,

Small typo in description of configuration :


  hive.metastore.local
  true
  controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM



I guess it is "remote metastore".

I could open a ticket, create a diff and ask for validation. But for
something like I hope someone with right access could do it...

Regards

Bertrand

[jira] [Commented] (HIVE-3278) Create wrappers around Thrift Metastore objects for improved readability

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422106#comment-13422106
 ] 

Ashutosh Chauhan commented on HIVE-3278:


Totally agree with Travis. We already have metastore objects represented via 2 
different ways in: {{org.apache.hadoop.hive.ql.metadata}} and 
{{org.apache.hadoop.hive.metastore.api}} with logic spread across both the 
layers. Adding another set of wrapper objects will only increase the confusion.

> Create wrappers around Thrift Metastore objects for improved readability
> 
>
> Key: HIVE-3278
> URL: https://issues.apache.org/jira/browse/HIVE-3278
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3279) Table schema not being copied to Partitions with no columns

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422103#comment-13422103
 ] 

Ashutosh Chauhan commented on HIVE-3279:


@Travis,
Yeah, this fix is required. Can you prepare a patch for it. Also, include the 
testcase which you have in 
https://issues.apache.org/jira/browse/HIVE-3279?focusedCommentId=13420724&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13420724
 in the patch

> Table schema not being copied to Partitions with no columns
> ---
>
> Key: HIVE-3279
> URL: https://issues.apache.org/jira/browse/HIVE-3279
> Project: Hive
>  Issue Type: Bug
>Reporter: Travis Crawford
>Assignee: Travis Crawford
>
> Hive has a feature where {{Partition}}'s without any defined columns use the 
> {{Table}} schema. This happens in 
> {{[Partition.initialize|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java#L167]}}
> {code}
> // set default if columns are not set
> if (tPartition.getSd().getCols() == null) {
>   if (table.getCols() != null) {
> tPartition.getSd().setCols(table.getCols());
>   }
> }
> {code}
> There's an issue though, because 
> {{[Table.getEmptyTable|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L121]}}
>  initializes cols to an empty array, which of course is not null, causing the 
> above feature to not work as expected.
> I'm not sure of the fix - is there a case where cols can indeed be null? I 
> think the best thing to do here is:
> {code}
> -if (tPartition.getSd().getCols() == null) {
> +if (tPartition.getSd().getCols() == null || 
> tPartition.getSd().getCols().size() == 0) {
> {code}
> Thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-3298) select 1+1 from dual; if dual is an empty table this statement will return no result

[caofangkun (JIRA)]

caofangkun created HIVE-3298:


 Summary: select 1+1 from dual; if dual is an empty table this 
statement will return no result 
 Key: HIVE-3298
 URL: https://issues.apache.org/jira/browse/HIVE-3298
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.8.1
 Environment: Linux  3.0.0-14-generic #23-Ubuntu SMP Mon Nov 21 
20:34:47 UTC 2011 i686 i686 i386 GNU/Linux
java version "1.6.0_25"
hadoop-0.20.2-cdh3u0
hive-0.8.1
Reporter: caofangkun
Priority: Minor


hive>drop table if exists dual;
hive>create table dual (dummy string);
hive>select 1+1 from dual;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206081154_0458, Tracking URL = 
http://dwtest-93-61:50030/jobdetails.jsp?jobid=job_201206081154_0458
Kill Command = /home/hive/hadoop-0.20.2-cdh3u0/bin/hadoop job  
-Dmapred.job.tracker=dwtest-93-61:9001 -kill job_201206081154_0458
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2012-07-25 16:58:15,793 Stage-1 map = 0%,  reduce = 0%
2012-07-25 16:58:17,817 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201206081154_0458
MapReduce Jobs Launched: 
Job 0:  HDFS Read: 0 HDFS Write: 0 SUCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 6.607 seconds


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-h0.21 - Build # 1566 - Failure

[Apache Jenkins Server]

Changes for Build #1566
[cws] HIVE-3249 [jira] Upgrade guava to 11.0.2
(Zhenxiao Luo via Carl Steinbach)

Summary:
HIVE-3249: Upgrade guava to 11.0.2

Hadoop has upgraded to this new version of Guava. We should, too, so we don't 
have compatibility issues running on Hadoop 2.0+

currently, hive is using guava-r09.jar
we should update it to guava-11.0.2.jar

Test Plan: EMPTY

Reviewers: JIRA

Differential Revision: https://reviews.facebook.net/D4083




1 tests failed.
REGRESSION:  
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1

Error Message:
Unexpected exception See build/ql/tmp/hive.log, or try "ant test ... 
-Dtest.silent=false" to get more logs.

Stack Trace:
junit.framework.AssertionFailedError: Unexpected exception
See build/ql/tmp/hive.log, or try "ant test ... -Dtest.silent=false" to get 
more logs.
at junit.framework.Assert.fail(Assert.java:50)
at 
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1(TestNegativeCliDriver.java:10725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052)
at 
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906)




The Apache Jenkins build system has built Hive-trunk-h0.21 (build 
#$BUILD_NUMBER)

Status: Failure

Check console output at https://builds.apache.org/job/Hive-trunk-h0.21/1566/ to 
view the results.

[jira] [Commented] (HIVE-3249) Upgrade guava to 11.0.2

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422071#comment-13422071
 ] 

Hudson commented on HIVE-3249:
--

Integrated in Hive-trunk-h0.21 #1566 (See 
[https://builds.apache.org/job/Hive-trunk-h0.21/1566/])
HIVE-3249 [jira] Upgrade guava to 11.0.2
(Zhenxiao Luo via Carl Steinbach)

Summary:
HIVE-3249: Upgrade guava to 11.0.2

Hadoop has upgraded to this new version of Guava. We should, too, so we don't 
have compatibility issues running on Hadoop 2.0+

currently, hive is using guava-r09.jar
we should update it to guava-11.0.2.jar

Test Plan: EMPTY

Reviewers: JIRA

Differential Revision: https://reviews.facebook.net/D4083 (Revision 1365371)

 Result = FAILURE
cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1365371
Files : 
* /hive/trunk/ivy/libraries.properties


> Upgrade guava to 11.0.2
> ---
>
> Key: HIVE-3249
> URL: https://issues.apache.org/jira/browse/HIVE-3249
> Project: Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Reporter: Zhenxiao Luo
>Assignee: Zhenxiao Luo
> Fix For: 0.10.0
>
> Attachments: HIVE-3249.1.patch.txt
>
>
> Hadoop has upgraded to this new version of Guava. We should, too, so we don't 
> have compatibility issues running on Hadoop 2.0+
> currently, hive is using guava-r09.jar
> we should update it to guava-11.0.2.jar

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Lisk Bucketing DDL Patch

[Gang Tim Liu]

Dear all hive developers,

Please review the documentation:

https://cwiki.apache.org/confluence/display/Hive/ListBucketing

Since we are close to release the first patch DDL.

We will continue to update the wiki about new information and in the
meanwhile want to collect your feedback.

Thanks

Tim