[jira] [Updated] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev

2013-06-08 Thread Navis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-4690:


Status: Patch Available  (was: Open)

> stats_partscan_1.q makes different result with different hadhoop.mr.rev 
> 
>
> Key: HIVE-4690
> URL: https://issues.apache.org/jira/browse/HIVE-4690
> Project: Hive
>  Issue Type: Test
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-4690.D11163.1.patch
>
>
> stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, 
> which can be different with different hadoop.mr.rev.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4692) Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases

2013-06-08 Thread Navis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-4692:


Status: Patch Available  (was: Open)

> Constant agg parameters will be replaced by ExprNodeColumnDesc with 
> single-sourced multi-gby cases
> --
>
> Key: HIVE-4692
> URL: https://issues.apache.org/jira/browse/HIVE-4692
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-4692.D11169.1.patch
>
>
> HIVE-3495 fixed this for single gbys but not for multi-gbys. For example,
>  
> {noformat}
> FROM (select key, cast(key as double) as value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT COUNT(*)
> INSERT OVERWRITE TABLE e2
> SELECT percentile_approx(value, 0.5);
> FAILED: UDFArgumentTypeException The second argument must be a constant, but 
> double was passed instead.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4692) Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases

2013-06-08 Thread Phabricator (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4692:
--

Attachment: HIVE-4692.D11169.1.patch

navis requested code review of "HIVE-4692 [jira] Constant agg parameters will 
be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases".

Reviewers: JIRA

HIVE-4692 Constant agg parameters will be replaced by ExprNodeColumnDesc with 
single-sourced multi-gby cases

HIVE-3495 fixed this for single gbys but not for multi-gbys. For example,

FROM (select key, cast(key as double) as value from src order by key) a
INSERT OVERWRITE TABLE e1
SELECT COUNT(*)
INSERT OVERWRITE TABLE e2
SELECT percentile_approx(value, 0.5);

FAILED: UDFArgumentTypeException The second argument must be a constant, but 
double was passed instead.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D11169

AFFECTED FILES
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/test/queries/clientpositive/multi_insert_gby2.q
  ql/src/test/results/clientpositive/multi_insert_gby2.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/26607/

To: JIRA, navis


> Constant agg parameters will be replaced by ExprNodeColumnDesc with 
> single-sourced multi-gby cases
> --
>
> Key: HIVE-4692
> URL: https://issues.apache.org/jira/browse/HIVE-4692
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-4692.D11169.1.patch
>
>
> HIVE-3495 fixed this for single gbys but not for multi-gbys. For example,
>  
> {noformat}
> FROM (select key, cast(key as double) as value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT COUNT(*)
> INSERT OVERWRITE TABLE e2
> SELECT percentile_approx(value, 0.5);
> FAILED: UDFArgumentTypeException The second argument must be a constant, but 
> double was passed instead.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HIVE-4692) Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases

2013-06-08 Thread Navis (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis reassigned HIVE-4692:
---

Assignee: Navis

> Constant agg parameters will be replaced by ExprNodeColumnDesc with 
> single-sourced multi-gby cases
> --
>
> Key: HIVE-4692
> URL: https://issues.apache.org/jira/browse/HIVE-4692
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
>
> HIVE-3495 fixed this for single gbys but not for multi-gbys. For example,
>  
> {noformat}
> FROM (select key, cast(key as double) as value from src order by key) a
> INSERT OVERWRITE TABLE e1
> SELECT COUNT(*)
> INSERT OVERWRITE TABLE e2
> SELECT percentile_approx(value, 0.5);
> FAILED: UDFArgumentTypeException The second argument must be a constant, but 
> double was passed instead.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4692) Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases

2013-06-08 Thread Navis (JIRA) 
Navis created HIVE-4692:
---

 Summary: Constant agg parameters will be replaced by 
ExprNodeColumnDesc with single-sourced multi-gby cases
 Key: HIVE-4692
 URL: https://issues.apache.org/jira/browse/HIVE-4692
 Project: Hive
  Issue Type: Bug
Reporter: Navis
Priority: Minor


HIVE-3495 fixed this for single gbys but not for multi-gbys. For example,
 
{noformat}
FROM (select key, cast(key as double) as value from src order by key) a
INSERT OVERWRITE TABLE e1
SELECT COUNT(*)
INSERT OVERWRITE TABLE e2
SELECT percentile_approx(value, 0.5);

FAILED: UDFArgumentTypeException The second argument must be a constant, but 
double was passed instead.
{noformat}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4691) orc_createas1.q has minor inconsistency

2013-06-08 Thread Brock Noland (JIRA) 
Brock Noland created HIVE-4691:
--

 Summary: orc_createas1.q has minor inconsistency
 Key: HIVE-4691
 URL: https://issues.apache.org/jira/browse/HIVE-4691
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 0.11.0
Reporter: Brock Noland
Priority: Trivial


orc_createas1.q  has the following two statements:

{noformat}
EXPLAIN SELECT * FROM orc_createas1b LIMIT 5;

SELECT * FROM orc_createas1b ORDER BY key LIMIT 5;
{noformat}

Since the pattern is:

EXPLAIN QUERY

QUERY

This is probably a mistake. We should add ORDER BY to the explain statement.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev

2013-06-08 Thread Phabricator (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-4690:
--

Attachment: HIVE-4690.D11163.1.patch

navis requested code review of "HIVE-4690 [jira] stats_partscan_1.q makes 
different result with different hadhoop.mr.rev".

Reviewers: JIRA

HIVE-4690 stats_partscan_1.q makes different result with different 
hadhoop.mr.rev

stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, 
which can be different with different hadoop.mr.rev.

TEST PLAN
  EMPTY

REVISION DETAIL
  https://reviews.facebook.net/D11163

AFFECTED FILES
  ql/src/test/queries/clientpositive/stats_partscan_1.q
  ql/src/test/queries/clientpositive/stats_partscan_1_23.q
  ql/src/test/results/clientpositive/stats_partscan_1.q.out
  ql/src/test/results/clientpositive/stats_partscan_1_23.q.out

MANAGE HERALD RULES
  https://reviews.facebook.net/herald/view/differential/

WHY DID I GET THIS EMAIL?
  https://reviews.facebook.net/herald/transcript/26601/

To: JIRA, navis


> stats_partscan_1.q makes different result with different hadhoop.mr.rev 
> 
>
> Key: HIVE-4690
> URL: https://issues.apache.org/jira/browse/HIVE-4690
> Project: Hive
>  Issue Type: Test
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-4690.D11163.1.patch
>
>
> stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, 
> which can be different with different hadoop.mr.rev.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HIVE-4690) stats_partscan_1.q makes different result with different hadhoop.mr.rev

2013-06-08 Thread Navis (JIRA) 
Navis created HIVE-4690:
---

 Summary: stats_partscan_1.q makes different result with different 
hadhoop.mr.rev 
 Key: HIVE-4690
 URL: https://issues.apache.org/jira/browse/HIVE-4690
 Project: Hive
  Issue Type: Test
Reporter: Navis
Assignee: Navis
Priority: Trivial


stats_partscan_1.q uses mapred.min/max.split.size and logs number of files, 
which can be different with different hadoop.mr.rev.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-3589) describe/show partition/show tblproperties command should accept database name

2013-06-08 Thread caofangkun (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678935#comment-13678935
 ] 

caofangkun commented on HIVE-3589:
--

Hi [~navis]
desc statements had been fixed in trunk.
But 'show partition ' and 'show tblproperties' statements have not been fixed 
yet .
I find simpley modify HiveParser.g file will fix this issue.
Please have a look at this rb:
https://reviews.apache.org/r/11753/
thank you

> describe/show partition/show tblproperties command should accept database name
> --
>
> Key: HIVE-3589
> URL: https://issues.apache.org/jira/browse/HIVE-3589
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Query Processor
>Affects Versions: 0.8.1
>Reporter: Sujesh Chirackkal
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-3589.D6075.1.patch, HIVE-3589.D6075.2.patch
>
>
> describe command not giving the details when called as describe 
> dbname.tablename.
> Throwing the error "Table dbname not found".
> Ex: hive -e "describe masterdb.table1" will throw error
> "Table masterdb not found"

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: HIVE-3589, show partition/show tblproperties command should accept database name

2013-06-08 Thread fangkun cao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11753/
---

Review request for hive and Navis Ryu.


Description
---

https://issues.apache.org/jira/browse/HIVE-3589


This addresses bug HIVE-3589.
https://issues.apache.org/jira/browse/HIVE-3589


Diffs
-

  
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
 1489269 
  
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/show_partitions.q
 1489269 
  
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/show_tblproperties.q
 1489269 
  
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/show_partitions.q.out
 1489269 
  
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/show_tblproperties.q.out
 1489269 

Diff: https://reviews.apache.org/r/11753/diff/


Testing
---

hive (db1)> show TBLPROPERTIES  default.srcpart;
OK
 
numPartitions   6
numFiles6
last_modified_bykun.cao
last_modified_time  1369040765
transient_lastDdlTime   1369040765
numRows 3000
totalSize   34872
rawDataSize 31872
Time taken: 0.392 seconds, Fetched: 9 row(s)


hive (db1)> show partitions default.srcpart;
OK
ds=2008-04-08/hr=16
ds=2008-04-08/hr=20
ds=2008-04-08/hr=21
ds=2008-04-09/hr=11
ds=2008-04-09/hr=12
ds=2008-05-11/hr=12
ds=2008-12-31/hr=11
ds=2008-12-31/hr=12
Time taken: 0.194 seconds, Fetched: 8 row(s)


Thanks,

fangkun cao

[jira] [Created] (HIVE-4689) For outerjoins, joinEmitInterval might make different result

2013-06-08 Thread Navis (JIRA) 
Navis created HIVE-4689:
---

 Summary: For outerjoins, joinEmitInterval might make different 
result
 Key: HIVE-4689
 URL: https://issues.apache.org/jira/browse/HIVE-4689
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Navis
Assignee: Navis


Alias filter tag is calculated for each group and used for outer joins. But if 
joinEmitInterval is smaller than the group size, pre-matured alias filter tag 
would be used and might introduce different(wrong) result.

It can be observed in join_1to1.q test but I cannot imagine proper solution 
which does not override intention of joinEmitInterval. Should it be disabled 
for outer joins?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails

2013-06-08 Thread Bruce Nelson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678929#comment-13678929
 ] 

Bruce Nelson commented on HIVE-4502:


Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170

> NPE - subquery smb joins fails
> --
>
> Key: HIVE-4502
> URL: https://issues.apache.org/jira/browse/HIVE-4502
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, 
> smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that 
> causes this error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4650) Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x

2013-06-08 Thread Bruce Nelson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678928#comment-13678928
 ] 

Bruce Nelson commented on HIVE-4650:


Correction - typo on my part its hive-0.11.0.1.3.0.0-107 not *-170

> Getting Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after 
> upgrade to Hive-0.11.0.x from hive-0.10.0.x
> --
>
> Key: HIVE-4650
> URL: https://issues.apache.org/jira/browse/HIVE-4650
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0
> Environment: HortonWorks 1.3 distro on x86_64 Centos 6 
>Reporter: Bruce Nelson
>
> working from a simple table in Hive 
> hive> desc cmnt
> > ;
> OK
> x1  int None
> x2  int None
> x3  int None
> x4  int None
> y   double  None 
> hive> select * from cmnt;
> OK
> 7   26  6   60  78.5
> 1   29  15  52  74.3
> 11  56  8   20  104.3
> 11  31  8   47  87.6
> 7   52  6   33  95.9
> 11  55  9   22  109.2
> 3   71  17  6   102.7
> 1   31  22  44  72.5
> 2   54  18  22  93.1
> 21  47  4   26  115.9
> 1   40  23  34  83.8
> 11  66  9   12  113.3
> 10  68  8   12  109.4
> A query that joins and transforms against this table : 
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( 
> select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 
> 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) 
> VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1 
> VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join 
> (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) 
> mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001) 
> sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004) 
> sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 
> VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) 
> obj1_6 ;
> Generates during Stage-3 : 
> setting HADOOP_USER_NAMEtest
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55 Starting to launch local task to process map join;
>   maximum memory = 1065484288
> 2013-06-03 12:40:56 Processing rows:1   Hashtable size: 1 
>   Memory usage:   7175528   rate:   0.007
> 2013-06-03 12:40:56 Dump the hashtable into file: 
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56 Upload 1 File to: 
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
>  File size: 334
> 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL = 
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job  -kill 
> job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%,  reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%,  reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: 
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_02 (and more) from job 
> job_201306022123_0045
> Task with the most failures(4): 
> -
> Task ID:
>   task_201306022123_0045_m_00
> URL:
>   
> http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_00
> -
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
> at org.apache.hadoop.mapred.

[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails

2013-06-08 Thread Bruce Nelson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678926#comment-13678926
 ] 

Bruce Nelson commented on HIVE-4502:


I verified that HIVE-4502-1.patch for HIVE-4650 has been applied to HWorx 
hive-0.11.0.1.3.0.0-170 and the MapJoin issue in HIVE-4650 still exits. 

> NPE - subquery smb joins fails
> --
>
> Key: HIVE-4502
> URL: https://issues.apache.org/jira/browse/HIVE-4502
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Attachments: HIVE-4502-1.patch, HIVE-4502.D10695.1.patch, 
> smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that 
> causes this error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4650) Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x

2013-06-08 Thread Bruce Nelson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678925#comment-13678925
 ] 

Bruce Nelson commented on HIVE-4650:


Vikram : I took a look at JIRA HIVE-4502 and Hortonworks 
hive-0.11.0.1.3.0.0-170 (HWorx 1.3.0) has HIVE-4502-1.patch already applied in 
the src. I did verify compiling the source and running the May 20th hive-0.11.0 
package and the same failure exists. The HIVE-4502.D10695.patch is not possible 
because of differences in code. I will report this on JIRA HIVE-4502 as well. 
So it appears that 4502 is not a fix for this particular issue.  

> Getting Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after 
> upgrade to Hive-0.11.0.x from hive-0.10.0.x
> --
>
> Key: HIVE-4650
> URL: https://issues.apache.org/jira/browse/HIVE-4650
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.11.0
> Environment: HortonWorks 1.3 distro on x86_64 Centos 6 
>Reporter: Bruce Nelson
>
> working from a simple table in Hive 
> hive> desc cmnt
> > ;
> OK
> x1  int None
> x2  int None
> x3  int None
> x4  int None
> y   double  None 
> hive> select * from cmnt;
> OK
> 7   26  6   60  78.5
> 1   29  15  52  74.3
> 11  56  8   20  104.3
> 11  31  8   47  87.6
> 7   52  6   33  95.9
> 11  55  9   22  109.2
> 3   71  17  6   102.7
> 1   31  22  44  72.5
> 2   54  18  22  93.1
> 21  47  4   26  115.9
> 1   40  23  34  83.8
> 11  66  9   12  113.3
> 10  68  8   12  109.4
> A query that joins and transforms against this table : 
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( 
> select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 
> 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) 
> VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1 
> VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join 
> (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) 
> mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001) 
> sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004) 
> sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 
> VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) 
> obj1_6 ;
> Generates during Stage-3 : 
> setting HADOOP_USER_NAMEtest
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55 Starting to launch local task to process map join;
>   maximum memory = 1065484288
> 2013-06-03 12:40:56 Processing rows:1   Hashtable size: 1 
>   Memory usage:   7175528   rate:   0.007
> 2013-06-03 12:40:56 Dump the hashtable into file: 
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56 Upload 1 File to: 
> file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
>  File size: 334
> 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL = 
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job  -kill 
> job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: > 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%,  reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%,  reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: 
> http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_02 (and more) from job 
> job_201306022123_0045
> Task with the most failures(4): 
> -
> Task ID:
>   task_201306022123_0045_m_00
> URL:
>   
> ht

Hive-trunk-h0.21 - Build # 2134 - Still Failing

2013-06-08 Thread Apache Jenkins Server 
Changes for Build #2107
[omalley] HIVE-4486 FetchOperator slows down SMB map joins by 50% when there 
are many 
partitions (Gopal V via omalley)


Changes for Build #2108

Changes for Build #2109

Changes for Build #2110

Changes for Build #2111
[omalley] HIVE-4475 Switch RCFile default to LazyBinaryColumnarSerDe. (Guther 
Hagleitner
via omalley)

[omalley] HIVE-4521 Auto join conversion fails in certain cases (Gunther 
Hagleitner via
omalley)


Changes for Build #2112

Changes for Build #2113
[gates] HIVE-4578 Changes to Pig's test harness broke HCat e2e tests (gates)


Changes for Build #2114
[gates] HIVE-4581 HCat e2e tests broken by changes to Hive's describe table 
formatting (gates)


Changes for Build #2115

Changes for Build #2116
[navis] JDBC2: HiveDriver should not throw RuntimeException when passed an 
invalid URL (Richard Ding via Navis)


Changes for Build #2117

Changes for Build #2118

Changes for Build #2119

Changes for Build #2120

Changes for Build #2121
[navis] HIVE-4572 ColumnPruner cannot preserve RS key columns corresponding to 
un-selected join keys in columnExprMap (Yin Huai via Navis)

[navis] HIVE-4540 JOIN-GRP BY-DISTINCT fails with NPE when 
mapjoin.mapreduce=true (Gunther Hagleitner via Navis)


Changes for Build #2122

Changes for Build #2123

Changes for Build #2124
[gates] HIVE-4543 Broken link in HCat doc (Reader and Writer Interfaces) (Lefty 
Leverenz via gates)


Changes for Build #2125
[daijy] PIG-3337: Fix remaining Window e2e tests


Changes for Build #2126
[hashutosh] HIVE-4615 : Invalid column names allowed when created dynamically 
by a SerDe (Gabriel Reid via Ashutosh Chauhan)

[hashutosh] HIVE-3846 : alter view rename NPEs with authorization on. (Teddy 
Choi via Ashutosh Chauhan)

[hashutosh] HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings 
related to overriding final parameters (Chu Tong via Ashutosh Chauhan)

[hashutosh] HIVE-4610 : HCatalog checkstyle violation after HIVE4578 (Brock 
Noland via Ashutosh Chauhan)

[hashutosh] HIVE-4636 : Failing on TestSemanticAnalysis.testAddReplaceCols in 
trunk (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4626 : join_vc.q is not deterministic (Navis via Ashutosh 
Chauhan)

[hashutosh] HIVE-4562 : HIVE3393 brought in Jackson library,and these four jars 
should be packed into hive-exec.jar (caofangkun via Ashutosh Chauhan)

[hashutosh] HIVE-4489 : beeline always return the same error message twice 
(Chaoyu Tang via Ashutosh Chauhan)

[hashutosh] HIVE-4510 : HS2 doesn't nest exceptions properly (fun debug times) 
(Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4535 : hive build fails with hadoop 0.20 (Thejas Nair via 
Ashutosh Chauhan)


Changes for Build #2127
[hashutosh] HIVE-4585 : Remove unused MR Temp file localization from Tasks 
(Gunther Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4418 : TestNegativeCliDriver failure message if cmd succeeds 
is misleading (Thejas Nair via Ashutosh Chauhan)

[navis] HIVE-4620 MR temp directory conflicts in case of parallel execution 
mode (Prasad Mujumdar via Navis)


Changes for Build #2128
[hashutosh] HIVE-4646 : skewjoin.q is failing in hadoop2 (Navis via Ashutosh 
Chauhan)

[hashutosh] HIVE-4377 : Add more comment to https://reviews.facebook.net/D1209 
(HIVE2340) :  (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4546 : Hive CLI leaves behind the per session resource 
directory on non-interactive invocation (Prasad Mujumdar via Ashutosh Chauhan)

[gates] HIVE-2670 A cluster test utility for Hive (gates and Johnny Zhang via 
gates)


Changes for Build #2129
[hashutosh] HIVE-2304 : Support PreparedStatement.setObject (Ido Hadanny via 
Ashutosh Chauhan)

[hashutosh] HIVE-4526 : auto_sortmerge_join_9.q throws NPE but test is 
succeeded (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4516 : Fix concurrency bug in 
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon 
Hartlaub and Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4566 : NullPointerException if typeinfo and nativesql commands 
are executed at beeline before a DB connection is established (Xuefu Zhang via 
Ashutosh Chauhan)


Changes for Build #2130

Changes for Build #2131
[hashutosh] HIVE-4422 : Test output need to be updated for Windows only unit 
test in TestCliDriver (Shuaishuai Nie via Ashutosh Chauhan)

[hashutosh] HIVE-4657 : HCatalog checkstyle violation after HIVE-2670 
(Shreepadma Venugopalan via Ashutosh Chauhan)

[hashutosh] HIVE-4348 : Unit test compile fail at hbase-handler project on 
Windows becuase of illegal escape character (Shuaishuai Nie via Ashutosh 
Chauhan)

[hashutosh] HIVE-4554 : Failed to create a table from existing file if file 
path has spaces (Xuefu Zhang via Ashutosh Chauhan)

[hashutosh] HIVE-4459 : Script hcat is overriding HIVE_CONF_DIR variable (Jarek 
Jarcec Cecho via Ashutosh Chauhan)


Changes for Build #2132
[hashutosh] HIVE-4435 : Column stats: Distinct value estimator should use hash 
functions that are pairwise independent (Shreepadma Ven

Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #397

2013-06-08 Thread Apache Jenkins Server 
See 

--
[...truncated 36425 lines...]
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/jenkins/hive_2013-06-08_13-43-38_731_909711795034216731/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/service/tmp/hive_job_log_jenkins_201306081343_1747240030.txt
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Copying file: 
file:/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/data/files/kv1.txt
[junit] PREHOOK: query: load data local inpath 
'/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/data/files/kv1.txt'
 into table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] Copying data from 
file:/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/data/files/kv1.txt
[junit] Loading data to table default.testhivedrivertable
[junit] POSTHOOK: query: load data local inpath 
'/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/data/files/kv1.txt'
 into table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: select * from testhivedrivertable limit 10
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: 
file:/tmp/jenkins/hive_2013-06-08_13-43-42_778_7720513823812777632/-mr-1
[junit] POSTHOOK: query: select * from testhivedrivertable limit 10
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: 
file:/tmp/jenkins/hive_2013-06-08_13-43-42_778_7720513823812777632/-mr-1
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/service/tmp/hive_job_log_jenkins_201306081343_688819881.txt
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] OK
[junit] PREHOOK: query: create table testhivedrivertable (num int)
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: create table testhivedrivertable (num int)
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] PREHOOK: Input: default@testhivedrivertable
[junit] PREHOOK: Output: default@testhivedrivertable
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK: type: DROPTABLE
[junit] POSTHOOK: Input: default@testhivedrivertable
[junit] POSTHOOK: Output: default@testhivedrivertable
[junit] OK
[junit] Hive history 
file=/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/service/tmp/hive_job_log_jenkins_201306081343_1175237629.txt
[junit] Copying file: 
file:/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/data/files/kv1.txt
[junit] Hive history 
file=/x1/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21/hive/build/service/tmp/hive_job_log_jenkins_201306081343_1075316503.txt
[junit] PREHOOK: query: drop table testhivedrivertable
[junit] PREHOOK: type: DROPTABLE
[junit] POSTHOOK: query: drop table testhivedrivertable
[junit] POSTHOOK:

[jira] [Updated] (HIVE-4606) Implement vector group by hash spill

2013-06-08 Thread Remus Rusanu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-4606:
---

Attachment: HIVE-4606.2.patch.txt

> Implement vector group by hash spill
> 
>
> Key: HIVE-4606
> URL: https://issues.apache.org/jira/browse/HIVE-4606
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: vectorization-branch
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Fix For: vectorization-branch
>
> Attachments: HIVE-4606.1.patch.txt, HIVE-4606.2.patch.txt
>
>
> the row mode GBy operator is capable of spilling the hash map (by emitting it 
> into the M/R and deleting the keys emitted) to conform to emmory constraints. 
> The VGBy should be able to do the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4688) NPE in writing null values.

2013-06-08 Thread Jitendra Nath Pandey (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-4688:
---

Attachment: HIVE-4688.3.patch

> NPE in writing null values.
> ---
>
> Key: HIVE-4688
> URL: https://issues.apache.org/jira/browse/HIVE-4688
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-4688.1.patch, HIVE-4688.2.patch, HIVE-4688.3.patch
>
>
> VectorExpressionWriter throws NPE when writing null values.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4688) NPE in writing null values.

2013-06-08 Thread Jitendra Nath Pandey (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-4688:
---

Attachment: HIVE-4688.2.patch

> NPE in writing null values.
> ---
>
> Key: HIVE-4688
> URL: https://issues.apache.org/jira/browse/HIVE-4688
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-4688.1.patch, HIVE-4688.2.patch
>
>
> VectorExpressionWriter throws NPE when writing null values.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HIVE-4678) second clause of AND, OR filter not applied for vectorized execution

2013-06-08 Thread Jitendra Nath Pandey (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey resolved HIVE-4678.


Resolution: Fixed

The fix for this was included in the patch for HIVE-4667.

> second clause of AND, OR filter not applied for vectorized execution
> 
>
> Key: HIVE-4678
> URL: https://issues.apache.org/jira/browse/HIVE-4678
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: vectorization-branch
>Reporter: Eric Hanson
>Assignee: Jitendra Nath Pandey
>
> Query
> select ddate, dnumbertables23008 from factsqlengineam_vec_orc where ddate = 
> "2013-01-08 00:00:00" and dnumbertables23008 = 1052436;
> returns rows where dnumbertables23008 != 1052436.
> Actual results:
> 636087 rows 
> Sample:
> ...
> 2013-01-08 00:00:00 0
> 2013-01-08 00:00:00 0
> 2013-01-08 00:00:00 108
> 2013-01-08 00:00:00 0
> 2013-01-08 00:00:00 0
> 2013-01-08 00:00:00 1625
> 2013-01-08 00:00:00 210
> 2013-01-08 00:00:00 0
> 2013-01-08 00:00:00 209
> 2013-01-08 00:00:00 0
> ...
> Expected results:
> Either no rows returned, or all rows have 1052436 in second column.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4606) Implement vector group by hash spill

2013-06-08 Thread Remus Rusanu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678755#comment-13678755
 ] 

Remus Rusanu commented on HIVE-4606:


https://reviews.apache.org/r/11747/

> Implement vector group by hash spill
> 
>
> Key: HIVE-4606
> URL: https://issues.apache.org/jira/browse/HIVE-4606
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: vectorization-branch
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Fix For: vectorization-branch
>
> Attachments: HIVE-4606.1.patch.txt
>
>
> the row mode GBy operator is capable of spilling the hash map (by emitting it 
> into the M/R and deleting the keys emitted) to conform to emmory constraints. 
> The VGBy should be able to do the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4606) Implement vector group by hash spill

2013-06-08 Thread Remus Rusanu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-4606:
---

Attachment: HIVE-4606.1.patch.txt

> Implement vector group by hash spill
> 
>
> Key: HIVE-4606
> URL: https://issues.apache.org/jira/browse/HIVE-4606
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: vectorization-branch
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Fix For: vectorization-branch
>
> Attachments: HIVE-4606.1.patch.txt
>
>
> the row mode GBy operator is capable of spilling the hash map (by emitting it 
> into the M/R and deleting the keys emitted) to conform to emmory constraints. 
> The VGBy should be able to do the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HIVE-4606) Implement vector group by hash spill

2013-06-08 Thread Remus Rusanu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-4606:
---

Fix Version/s: vectorization-branch
   Status: Patch Available  (was: Open)

> Implement vector group by hash spill
> 
>
> Key: HIVE-4606
> URL: https://issues.apache.org/jira/browse/HIVE-4606
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: vectorization-branch
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
> Fix For: vectorization-branch
>
> Attachments: HIVE-4606.1.patch.txt
>
>
> the row mode GBy operator is capable of spilling the hash map (by emitting it 
> into the M/R and deleting the keys emitted) to conform to emmory constraints. 
> The VGBy should be able to do the same.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hive-trunk-hadoop2 - Build # 231 - Still Failing

2013-06-08 Thread Apache Jenkins Server 
Changes for Build #199
[omalley] HIVE-4550 local_mapred_error_cache fails on some hadoop versions 
(Gunther 
Hagleitner via omalley)

[omalley] HIVE-4440 SMB Operator spills to disk like it's 1999 (Gunther 
Hagleitner via
omalley)


Changes for Build #200

Changes for Build #201
[omalley] HIVE-4486 FetchOperator slows down SMB map joins by 50% when there 
are many 
partitions (Gopal V via omalley)


Changes for Build #202

Changes for Build #203

Changes for Build #204

Changes for Build #205
[omalley] HIVE-4475 Switch RCFile default to LazyBinaryColumnarSerDe. (Guther 
Hagleitner
via omalley)

[omalley] HIVE-4521 Auto join conversion fails in certain cases (Gunther 
Hagleitner via
omalley)


Changes for Build #206
[gates] HIVE-4578 Changes to Pig's test harness broke HCat e2e tests (gates)


Changes for Build #207
[gates] HIVE-4581 HCat e2e tests broken by changes to Hive's describe table 
formatting (gates)


Changes for Build #208

Changes for Build #209
[navis] JDBC2: HiveDriver should not throw RuntimeException when passed an 
invalid URL (Richard Ding via Navis)


Changes for Build #210

Changes for Build #211

Changes for Build #212

Changes for Build #213

Changes for Build #214
[navis] HIVE-4572 ColumnPruner cannot preserve RS key columns corresponding to 
un-selected join keys in columnExprMap (Yin Huai via Navis)

[navis] HIVE-4540 JOIN-GRP BY-DISTINCT fails with NPE when 
mapjoin.mapreduce=true (Gunther Hagleitner via Navis)


Changes for Build #215

Changes for Build #216

Changes for Build #217
[gates] HIVE-4543 Broken link in HCat doc (Reader and Writer Interfaces) (Lefty 
Leverenz via gates)


Changes for Build #218

Changes for Build #219
[daijy] PIG-3337: Fix remaining Window e2e tests


Changes for Build #220

Changes for Build #221

Changes for Build #222

Changes for Build #223
[hashutosh] HIVE-4610 : HCatalog checkstyle violation after HIVE4578 (Brock 
Noland via Ashutosh Chauhan)

[hashutosh] HIVE-4636 : Failing on TestSemanticAnalysis.testAddReplaceCols in 
trunk (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4626 : join_vc.q is not deterministic (Navis via Ashutosh 
Chauhan)

[hashutosh] HIVE-4562 : HIVE3393 brought in Jackson library,and these four jars 
should be packed into hive-exec.jar (caofangkun via Ashutosh Chauhan)

[hashutosh] HIVE-4489 : beeline always return the same error message twice 
(Chaoyu Tang via Ashutosh Chauhan)

[hashutosh] HIVE-4510 : HS2 doesn't nest exceptions properly (fun debug times) 
(Thejas Nair via Ashutosh Chauhan)

[hashutosh] HIVE-4535 : hive build fails with hadoop 0.20 (Thejas Nair via 
Ashutosh Chauhan)


Changes for Build #224
[hashutosh] HIVE-4615 : Invalid column names allowed when created dynamically 
by a SerDe (Gabriel Reid via Ashutosh Chauhan)

[hashutosh] HIVE-3846 : alter view rename NPEs with authorization on. (Teddy 
Choi via Ashutosh Chauhan)

[hashutosh] HIVE-4403 : Running Hive queries on Yarn (MR2) gives warnings 
related to overriding final parameters (Chu Tong via Ashutosh Chauhan)


Changes for Build #225
[gates] HIVE-2670 A cluster test utility for Hive (gates and Johnny Zhang via 
gates)

[hashutosh] HIVE-4585 : Remove unused MR Temp file localization from Tasks 
(Gunther Hagleitner via Ashutosh Chauhan)

[hashutosh] HIVE-4418 : TestNegativeCliDriver failure message if cmd succeeds 
is misleading (Thejas Nair via Ashutosh Chauhan)

[navis] HIVE-4620 MR temp directory conflicts in case of parallel execution 
mode (Prasad Mujumdar via Navis)


Changes for Build #226
[hashutosh] HIVE-2304 : Support PreparedStatement.setObject (Ido Hadanny via 
Ashutosh Chauhan)

[hashutosh] HIVE-4526 : auto_sortmerge_join_9.q throws NPE but test is 
succeeded (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4516 : Fix concurrency bug in 
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java (Jon 
Hartlaub and Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4566 : NullPointerException if typeinfo and nativesql commands 
are executed at beeline before a DB connection is established (Xuefu Zhang via 
Ashutosh Chauhan)

[hashutosh] HIVE-4646 : skewjoin.q is failing in hadoop2 (Navis via Ashutosh 
Chauhan)

[hashutosh] HIVE-4377 : Add more comment to https://reviews.facebook.net/D1209 
(HIVE2340) :  (Navis via Ashutosh Chauhan)

[hashutosh] HIVE-4546 : Hive CLI leaves behind the per session resource 
directory on non-interactive invocation (Prasad Mujumdar via Ashutosh Chauhan)


Changes for Build #227
[hashutosh] HIVE-4422 : Test output need to be updated for Windows only unit 
test in TestCliDriver (Shuaishuai Nie via Ashutosh Chauhan)

[hashutosh] HIVE-4657 : HCatalog checkstyle violation after HIVE-2670 
(Shreepadma Venugopalan via Ashutosh Chauhan)

[hashutosh] HIVE-4348 : Unit test compile fail at hbase-handler project on 
Windows becuase of illegal escape character (Shuaishuai Nie via Ashutosh 
Chauhan)

[hashutosh] HIVE-4554 : Failed to create a table from existing file if file 
path has spaces (Xuefu Zhang via Ashutosh Chauhan

[jira] [Resolved] (HIVE-4667) tpch query 1 fails with java.lang.ClassCastException

2013-06-08 Thread Ashutosh Chauhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan resolved HIVE-4667.


Resolution: Fixed

Committed to branch. Thanks, Jitendra!

> tpch query 1 fails with java.lang.ClassCastException
> 
>
> Key: HIVE-4667
> URL: https://issues.apache.org/jira/browse/HIVE-4667
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: vectorization-branch
>Reporter: Tony Murphy
>Assignee: Jitendra Nath Pandey
> Fix For: vectorization-branch
>
> Attachments: HIVE-4667.1.patch
>
>
> {noformat}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.DoubleColumnVector
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DoubleColSubtractLongScalar.evaluate(DoubleColSubtractLongScalar.java:46)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression.evaluateChildren(VectorExpression.java:69)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.DoubleColMultiplyDoubleColumn.evaluate(DoubleColMultiplyDoubleColumn.java:41)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFSumDouble.aggregateInputSelection(VectorUDAFSumDouble.java:98)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processAggregators(VectorGroupByOperator.java:174)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.processOp(VectorGroupByOperator.java:151)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:104)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.processOp(VectorFilterOperator.java:91)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
>   at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:502)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:832)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:717)
>   ... 9 more
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request: HIVE-4606 Add memory pressure flush for VectorGroupByOperator

2013-06-08 Thread Remus Rusanu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11747/
---

Review request for hive, Jitendra Pandey and Eric Hanson.


Description
---

Implement the flush under memory pressure, modeled much after the 
GroupByOperator implementation.


This addresses bug HIVE-4606.
https://issues.apache.org/jira/browse/HIVE-4606


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/JavaObjectSizeUtil.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java
 030a73c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 
4634731 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapper.java 
01dd7be 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorHashKeyWrapperBatch.java
 5e547c7 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorAggregateExpression.java
 8ab9f43 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCount.java
 bb4800e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCountStar.java
 607e3ad 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFAvgDouble.java
 432b12e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFAvgLong.java
 b282aeb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMaxDouble.java
 bc7f852 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMaxLong.java
 6ba416e 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMaxString.java
 c5e92b6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMinDouble.java
 d982fc2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMinLong.java
 a8f5531 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFMinString.java
 ca1b840 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFStdPopDouble.java
 4d408b3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFStdPopLong.java
 7e16a1c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFStdSampDouble.java
 00cd04d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFStdSampLong.java
 ea4d894 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFSumDouble.java
 e609a19 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFSumLong.java
 f120db9 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFVarPopDouble.java
 0ac7677 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFVarPopLong.java
 37cfda6 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFVarSampDouble.java
 c016160 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen/VectorUDAFVarSampLong.java
 8aa9621 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFAvg.txt
 1295cfd 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFCount.txt
 69ff67b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFMinMax.txt
 d00d9ae 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFMinMaxString.txt
 96ce80c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFSum.txt
 c4e6a19 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/templates/VectorUDAFVar.txt
 9c8ad94 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java
 0050ebc 

Diff: https://reviews.apache.org/r/11747/diff/


Testing
---


Thanks,

Remus Rusanu

[jira] [Updated] (HIVE-4688) NPE in writing null values.

2013-06-08 Thread Jitendra Nath Pandey (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-4688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-4688:
---

Attachment: HIVE-4688.1.patch

Patch uploaded. For null values, we should return NullWritable instead of null.

> NPE in writing null values.
> ---
>
> Key: HIVE-4688
> URL: https://issues.apache.org/jira/browse/HIVE-4688
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-4688.1.patch
>
>
> VectorExpressionWriter throws NPE when writing null values.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira