[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1785:
-

Attachment: hive_1758_5.patch

1) make the old interface be deprecated
2) let the existing Prehook and Posthook implements the new interface.
3) the task tag for each task

> change Pre/Post Query Hooks to take in 1 parameter: HookContext
> ---
>
> Key: HIVE-1785
> URL: https://issues.apache.org/jira/browse/HIVE-1785
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Liyin Tang
> Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1758_5.patch, 
> hive_1785_1.patch, hive_1785_2.patch
>
>
> This way, it would be possible to add new parameters to the hooks without 
> changing the existing hooks.
> This will be a incompatible change, and all the hooks need to change to the 
> new API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1801) HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice

2010-11-22 Thread He Yongqiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934536#action_12934536
 ] 

He Yongqiang commented on HIVE-1801:


can you put "if (recordReader instanceof RCFileRecordReader)" at the same level 
with "else if (inputFormatClass.getName().contains("RCFile")) {"?

 } else if (inputFormatClass.getName().contains("RCFile")) {
-  RCFile.Reader in = new RCFile.Reader(fs, path, job);
   blockPointer = true;
-  in.sync(fileSplit.getStart());
-  blockStart = in.getPosition();
-  in.close();
+
+  if (recordReader instanceof RCFileRecordReader) {
+blockStart = ((RCFileRecordReader)recordReader).getStart();
+  } else {
+RCFile.Reader in = new RCFile.Reader(fs, path, job);
+in.sync(fileSplit.getStart());
+blockStart = in.getPosition();
+in.close();
+  }


> HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
> 
>
> Key: HIVE-1801
> URL: https://issues.apache.org/jira/browse/HIVE-1801
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
> Attachments: HIVE-1801.1.patch
>
>
> HiveInputFormat or CombineHiveInputFormat RCFile.Reader.sync() twice. One in 
> getReader() and one in initIOContext(). We can avoid the latter one by read 
> the sync() position of the former one.
> We also sync() twice for SequenceFile but since SequenceFileReader is not a 
> part of Hive code, maybe we should be careful when depending on the 
> implementation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1801) HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice

2010-11-22 Thread He Yongqiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

He Yongqiang updated HIVE-1801:
---

Status: Open  (was: Patch Available)

> HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
> 
>
> Key: HIVE-1801
> URL: https://issues.apache.org/jira/browse/HIVE-1801
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
> Attachments: HIVE-1801.1.patch
>
>
> HiveInputFormat or CombineHiveInputFormat RCFile.Reader.sync() twice. One in 
> getReader() and one in initIOContext(). We can avoid the latter one by read 
> the sync() position of the former one.
> We also sync() twice for SequenceFile but since SequenceFileReader is not a 
> part of Hive code, maybe we should be careful when depending on the 
> implementation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1804) Mapjoin will fail if there are no files associating with the join tables

2010-11-22 Thread Liyin Tang (JIRA) 
Mapjoin will fail if there are no files associating with the join tables


 Key: HIVE-1804
 URL: https://issues.apache.org/jira/browse/HIVE-1804
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.7.0
Reporter: Liyin Tang
Assignee: Liyin Tang
 Fix For: 0.7.0


If there are some empty tables without any file associated, the map join will 
fail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Build failed in Hudson: Hive-trunk-h0.20 #430

2010-11-22 Thread Apache Hudson Server 
See 

Changes:

[namit] HIVE-1787 optimize the code path when there are no outer joins
(Siying Dong via namit)

--
[...truncated 15439 lines...]
[junit] POSTHOOK: Output: defa...@src1
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_sequencefile
[junit] POSTHOOK: Output: defa...@src_sequencefile
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_thrift
[junit] POSTHOOK: Output: defa...@src_thrift
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_json
[junit] POSTHOOK: Output: defa...@src_json
[junit] OK
[junit] diff 

 

[junit] Done query: unknown_table1.q
[junit] Begin query: unknown_table2.q
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-08, hr=11)
[junit] rmr: cannot remove 
p:
 No such file or directory.
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-08/hr=11
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-08, hr=12)
[junit] rmr: cannot remove 
p:
 No such file or directory.
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-08/hr=12
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-09, hr=11)
[junit] rmr: cannot remove 
p:
 No such file or directory.
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-09/hr=11
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-09, hr=12)
[junit] rmr: cannot remove 
p:
 No such file or directory.
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-09/hr=12
[junit] OK
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src
[junit] POSTHOOK: Output: defa...@src
[junit] OK
[junit] Copying data from 

[junit] Loading data to

[jira] Updated: (HIVE-138) Provide option to export a HEADER

2010-11-22 Thread Paul Butler (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Butler updated HIVE-138:
-

Attachment: HIVE-138.patch

> Provide option to export a HEADER
> -
>
> Key: HIVE-138
> URL: https://issues.apache.org/jira/browse/HIVE-138
> Project: Hive
>  Issue Type: Improvement
>  Components: Clients, Query Processor
>Reporter: Adam Kramer
>Priority: Minor
> Attachments: HIVE-138.patch
>
>
> When writing data to directories or files for later analysis, or when 
> exploring data in the hive CLI with raw SELECT statements, it'd be great if 
> we could get a "header" or something so we know which columns our output 
> comes from. Any chance this is easy to add? Just print the column names (or 
> formula used to generate them) in the first row?
> SELECT foo.* WITH HEADER FROM some_table foo limit 3;
> col1col2col3
> 1   9   6
> 7   5   0
> 7   5   3
> SELECT f.col1-f.col2, col3 WITH HEADER FROM some_table foo limit 3;
> f.col1-f.col2 col3
> -8 6
> 2 0
> 2 3
> ...etc

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-138) Provide option to export a HEADER

2010-11-22 Thread Paul Butler (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paul Butler updated HIVE-138:
-

Assignee: Paul Butler
  Status: Patch Available  (was: Open)

> Provide option to export a HEADER
> -
>
> Key: HIVE-138
> URL: https://issues.apache.org/jira/browse/HIVE-138
> Project: Hive
>  Issue Type: Improvement
>  Components: Clients, Query Processor
>Reporter: Adam Kramer
>Assignee: Paul Butler
>Priority: Minor
> Attachments: HIVE-138.patch
>
>
> When writing data to directories or files for later analysis, or when 
> exploring data in the hive CLI with raw SELECT statements, it'd be great if 
> we could get a "header" or something so we know which columns our output 
> comes from. Any chance this is easy to add? Just print the column names (or 
> formula used to generate them) in the first row?
> SELECT foo.* WITH HEADER FROM some_table foo limit 3;
> col1col2col3
> 1   9   6
> 7   5   0
> 7   5   3
> SELECT f.col1-f.col2, col3 WITH HEADER FROM some_table foo limit 3;
> f.col1-f.col2 col3
> -8 6
> 2 0
> 2 3
> ...etc

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

2010-11-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934619#action_12934619
 ] 

Namit Jain commented on HIVE-1648:
--

I haven't taken a look at the code, but here are the comments for the tests


Instead of:


desc extended  in the tests,
please use
show table extended like ``;


This will dump stats in a new line and can be easily compared.
The non-deterministic stats are ignored.


Add a test for limit in the sub-query.

Dont select from existing tables: src/src1 for your stats tests.
Create new tables and then set hive.stats.autogather.read to true.
This was, you are sure that the remaining tests will not be affected.

Add another test for 3-way join where the join keys are not the same: something 
like:

select .. from A join B on A.key1 = B.key1 join C on B.key2 = C.key2 where 


> Automatically gathering stats when reading a table/partition
> 
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Ning Zhang
>Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to 
> gathering stats. This requires additional scan of the data. Stats gathering 
> can be piggy-backed on TableScanOperator whenever a table/partition is 
> scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread HBase Review Board (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934640#action_12934640
 ] 

HBase Review Board commented on HIVE-1526:
--

Message from: "Ning Zhang" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1242/#review1958
---



.classpath._hbase


Is deleting of this file because of we eliminated hadoop pre-0.20 versions? 

John, can you confirm if this file can be deleted?



ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java


This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends 
ExecDriver which implements getType(), this function should not be here in 
MapRedTask.



ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java


It would be better to define a new OperatorType for SMBMapJoinOperator.



ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java


commnet: -1 -> null


- Ning





> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Review Request: HIVE-1526: Hive should depend on a release version of Thrift

2010-11-22 Thread Ning Zhang 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1242/#review1958
---



.classpath._hbase


Is deleting of this file because of we eliminated hadoop pre-0.20 versions? 

John, can you confirm if this file can be deleted?



ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java


This should not be MAPREDLOCAL, but MAPRED. Since MapRedTask extends 
ExecDriver which implements getType(), this function should not be here in 
MapRedTask.



ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java


It would be better to define a new OperatorType for SMBMapJoinOperator.



ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java


commnet: -1 -> null


- Ning


On 2010-11-18 01:23:52, Carl Steinbach wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1242/
> ---
> 
> (Updated 2010-11-18 01:23:52)
> 
> 
> Review request for Hive Developers.
> 
> 
> Summary
> ---
> 
> Review request for HIVE-1526: upgrade to Thrift 0.5.0
> 
> This review request does not include the code generated by the Thrift 
> compiler.
> 
> 
> This addresses bug HIVE-1526.
> http://issues.apache.org/jira/browse/HIVE-1526
> 
> 
> Diffs
> -
> 
>   .classpath._hbase 15fd1c5 
>   build-common.xml 53fd1b1 
>   build.xml dc20725 
>   eclipse-templates/.classpath d06d6bd 
>   eclipse-templates/.classpath._hbase b81c24a 
>   ivy/libraries.properties fdd54c8 
>   lib/README 72d90b7 
>   lib/libfb303.LICENSE 6b5c3b1 
>   lib/libfb303.jar 516b597 
>   lib/libthrift.LICENSE 6b5c3b1 
>   lib/libthrift.jar 7e7f90e 
>   lib/thrift-0.5.0.jar PRE-CREATION 
>   lib/thrift-fb303-0.5.0.jar PRE-CREATION 
>   lib/thrift-fb303.LICENSE PRE-CREATION 
>   lib/thrift.LICENSE PRE-CREATION 
>   metastore/build.xml ffb86c1 
>   metastore/if/hive_metastore.thrift 9c314cc 
>   metastore/ivy.xml 2e39eb8 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> a4f1853 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 4f5623c 
>   odbc/Makefile 816ffd4 
>   odbc/build.xml 64cf988 
>   odbc/src/cpp/HiveRowSet.cpp b4dc2c0 
>   odbc/src/cpp/hiveclienthelper.cpp e2d48f3 
>   ql/build.xml a025a2b 
>   ql/if/queryplan.thrift aca0e8f 
>   ql/ivy.xml e72a450 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
> 5ccb909 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0a9cdf4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java af6e0af 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 
> 87fc61c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> 9fe35ec 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java 
> c070aca 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 
> 5eb9098 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 98571dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd 
>   ql/src/java/org/apache/hadoop

Re: Review Request: HIVE-1526: Hive should depend on a release version of Thrift

2010-11-22 Thread Carl Steinbach 


> On 2010-11-22 14:37:20, Ning Zhang wrote:
> > .classpath._hbase, line 1
> > 
> >
> > Is deleting of this file because of we eliminated hadoop pre-0.20 
> > versions? 
> > 
> > John, can you confirm if this file can be deleted?

This is an eclipse project classpath file. It should not have been checked in. 


- Carl


---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1242/#review1958
---


On 2010-11-18 01:23:52, Carl Steinbach wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1242/
> ---
> 
> (Updated 2010-11-18 01:23:52)
> 
> 
> Review request for Hive Developers.
> 
> 
> Summary
> ---
> 
> Review request for HIVE-1526: upgrade to Thrift 0.5.0
> 
> This review request does not include the code generated by the Thrift 
> compiler.
> 
> 
> This addresses bug HIVE-1526.
> http://issues.apache.org/jira/browse/HIVE-1526
> 
> 
> Diffs
> -
> 
>   .classpath._hbase 15fd1c5 
>   build-common.xml 53fd1b1 
>   build.xml dc20725 
>   eclipse-templates/.classpath d06d6bd 
>   eclipse-templates/.classpath._hbase b81c24a 
>   ivy/libraries.properties fdd54c8 
>   lib/README 72d90b7 
>   lib/libfb303.LICENSE 6b5c3b1 
>   lib/libfb303.jar 516b597 
>   lib/libthrift.LICENSE 6b5c3b1 
>   lib/libthrift.jar 7e7f90e 
>   lib/thrift-0.5.0.jar PRE-CREATION 
>   lib/thrift-fb303-0.5.0.jar PRE-CREATION 
>   lib/thrift-fb303.LICENSE PRE-CREATION 
>   lib/thrift.LICENSE PRE-CREATION 
>   metastore/build.xml ffb86c1 
>   metastore/if/hive_metastore.thrift 9c314cc 
>   metastore/ivy.xml 2e39eb8 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> a4f1853 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 4f5623c 
>   odbc/Makefile 816ffd4 
>   odbc/build.xml 64cf988 
>   odbc/src/cpp/HiveRowSet.cpp b4dc2c0 
>   odbc/src/cpp/hiveclienthelper.cpp e2d48f3 
>   ql/build.xml a025a2b 
>   ql/if/queryplan.thrift aca0e8f 
>   ql/ivy.xml e72a450 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
> 5ccb909 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java 7daea2d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java bf7b7a5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 46b8bf4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 0a9cdf4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java af6e0af 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java a572bcb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExtractOperator.java 7bf8886 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java de2c42d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java d510f95 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 554bf25 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 2a35c96 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1e6941e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 8423178 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 
> 87fc61c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> 9fe35ec 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java e16df36 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java 
> c070aca 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 
> 5eb9098 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java da5dbb9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 98571dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 887ad30 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 0fab63c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java c5aa3d6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7fbc586 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 6c6ea89 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 592f4f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 001d1f0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 4f8b4bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java d644a33 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 47adbed 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 96e63b8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 893530c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java 682f38a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 2c42ed7 
>   
> ql/

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread HBase Review Board (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934643#action_12934643
 ] 

HBase Review Board commented on HIVE-1526:
--

Message from: "Carl Steinbach" 


bq.  On 2010-11-22 14:37:20, Ning Zhang wrote:
bq.  > .classpath._hbase, line 1
bq.  > 
bq.  >
bq.  > Is deleting of this file because of we eliminated hadoop pre-0.20 
versions? 
bq.  > 
bq.  > John, can you confirm if this file can be deleted?

This is an eclipse project classpath file. It should not have been checked in. 


- Carl


---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1242/#review1958
---





> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Ning Zhang 
I see. That's OK then. 

On Nov 22, 2010, at 2:46 PM, HBase Review Board (JIRA) wrote:

> 
>[ 
> https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934643#action_12934643
>  ] 
> 
> HBase Review Board commented on HIVE-1526:
> --
> 
> Message from: "Carl Steinbach" 
> 
> 
> bq.  On 2010-11-22 14:37:20, Ning Zhang wrote:
> bq.  > .classpath._hbase, line 1
> bq.  > 
> bq.  >
> bq.  > Is deleting of this file because of we eliminated hadoop pre-0.20 
> versions? 
> bq.  > 
> bq.  > John, can you confirm if this file can be deleted?
> 
> This is an eclipse project classpath file. It should not have been checked 
> in. 
> 
> 
> - Carl
> 
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/1242/#review1958
> ---
> 
> 
> 
> 
> 
>> Hive should depend on a release version of Thrift
>> -
>> 
>>Key: HIVE-1526
>>URL: https://issues.apache.org/jira/browse/HIVE-1526
>>Project: Hive
>> Issue Type: Task
>> Components: Build Infrastructure, Clients
>>   Reporter: Carl Steinbach
>>   Assignee: Carl Steinbach
>>Fix For: 0.7.0
>> 
>>Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
>> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
>> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>> 
>> 
>> Hive should depend on a release version of Thrift, and ideally it should use 
>> Ivy to resolve this dependency.
>> The Thrift folks are working on adding Thrift artifacts to a maven 
>> repository here: https://issues.apache.org/jira/browse/THRIFT-363
> 
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Ning Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934650#action_12934650
 ] 

Ning Zhang commented on HIVE-1526:
--

I see. That's OK then. 






> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Ning Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934651#action_12934651
 ] 

Ning Zhang commented on HIVE-1526:
--

I see. That's OK then. 






> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1805) Ability to create dynamic partitions atomically

2010-11-22 Thread Namit Jain (JIRA) 
Ability to create dynamic partitions atomically
---

 Key: HIVE-1805
 URL: https://issues.apache.org/jira/browse/HIVE-1805
 Project: Hive
  Issue Type: New Feature
Reporter: Namit Jain




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread He Yongqiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934656#action_12934656
 ] 

He Yongqiang commented on HIVE-1526:


can we get this in after HIVE-78 since this is mostly auto-generated files?

> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1805) Ability to create dynamic partitions atomically

2010-11-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934657#action_12934657
 ] 

Namit Jain commented on HIVE-1805:
--

Currently, if a query creates partitions dynamically, some of them may be 
created and some others fail.
It will be useful to have an atomic way to running the query - either all the 
partitions should be created or none of them.

The same problem exists for multi-table inserts, but it is not a very common 
scenario.

> Ability to create dynamic partitions atomically
> ---
>
> Key: HIVE-1805
> URL: https://issues.apache.org/jira/browse/HIVE-1805
> Project: Hive
>  Issue Type: New Feature
>Reporter: Namit Jain
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1797) Compressed the hashtable dump file before put into distributed cache

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1797:
-

Status: Patch Available  (was: Open)

> Compressed the hashtable dump file before put into distributed cache
> 
>
> Key: HIVE-1797
> URL: https://issues.apache.org/jira/browse/HIVE-1797
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Attachments: hive-1797.patch, hive-1797_3.patch
>
>
> Clearly, the size of small table is the performance bottleneck for map join.
> Because the size of the small table will affect the memory usage and dumped 
> hashtable file.
> That means there are 2 boundaries of the map join performance.
> 1)The memory usage for local task and mapred task
> 2)The dumped hashtable file size for distributed cache
> The reason that test case in last email spends most of the execution time on 
> initializing is because it hits the second boundary.
> Since we have already bound the memory usage, one thing we can do is to let 
> the performance never hits the secondary bound before it hits the first 
> boundary.
> Assuming the heap size is 1.6 G and the small table file size is 15M 
> compressed (75M uncompressed),
> local  task can roughly hold that 1.5M unique rows in memory. 
> Roughly the dumped file size will be 150M, which is too large to put into the 
> distributed cache.
>  
> From experiments, we can basically conclude when the dumped file size is 
> smaller than 30M. 
> The distributed cache works well and all the mappers will  be initialized in 
> a short time (less than 30 secs).
> One easy implementation is to compress the hashtable file. 
> I use the gzip to compress the hashtable file and the file size is compressed 
> from 100M to 13M.
> After several tests, all the mappers will be initialized in less than 23 secs.
> But this solution adds some decompression overhead to each mapper.
> Mappers on the same machine will do the duplicated decompression work.
> Maybe in the future, we can let the distributed cache to support this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1785:
-

Status: Patch Available  (was: Open)

> change Pre/Post Query Hooks to take in 1 parameter: HookContext
> ---
>
> Key: HIVE-1785
> URL: https://issues.apache.org/jira/browse/HIVE-1785
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Namit Jain
>Assignee: Liyin Tang
> Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1758_5.patch, 
> hive_1785_1.patch, hive_1785_2.patch
>
>
> This way, it would be possible to add new parameters to the hooks without 
> changing the existing hooks.
> This will be a incompatible change, and all the hooks need to change to the 
> new API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Carl Steinbach (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934677#action_12934677
 ] 

Carl Steinbach commented on HIVE-1526:
--

@Yongqiang: that's fine with me, assuming that HIVE-78 is close to getting 
committed. I think this ticket is blocking some of the security work.

> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext

2010-11-22 Thread John Sichi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-1785:
-

Fix Version/s: 0.7.0
Affects Version/s: 0.7.0
   Status: Open  (was: Patch Available)

* It looks like you mixed in some changes from other patches?  (And the patch 
name is HIVE-1758 instead of HIVE-1785.)  Please resubmit with only the changes 
relevant to HIVE-1785.

Also, I don't see where you addressed two of my previous comments:

* rewrite existing hook implementations in Hive codebase to implement new 
interface
* add Javadoc for Hook, HookContext, and ExecuteWithHookContext

One new one while we're at it:

* The old hook interface provided SessionState, Set, 
Set, LineageInfo and UserGroupInformation.  HookContext should 
provide getters for these as well.  I realize that some of these are obtainable 
via QueryPlan and HiveConf, but we should not force users to become dependent 
on internals like QueryPlan unless they want to be; also, providing the 
equivalent getters makes upgrade to the new interface easier.  You can make the 
HookContext getters call through to QueryPlan getters etc.


> change Pre/Post Query Hooks to take in 1 parameter: HookContext
> ---
>
> Key: HIVE-1785
> URL: https://issues.apache.org/jira/browse/HIVE-1785
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Namit Jain
>Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: hive-1785_3.patch, hive-1785_4.patch, hive_1758_5.patch, 
> hive_1785_1.patch, hive_1785_2.patch
>
>
> This way, it would be possible to add new parameters to the hooks without 
> changing the existing hooks.
> This will be a incompatible change, and all the hooks need to change to the 
> new API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Devaraj Das (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934681#action_12934681
 ] 

Devaraj Das commented on HIVE-1526:
---

Guys, it will be really appreciated if this patch can be committed now. This 
blocks the other jiras on Security - HIVE-842 and HIVE-1696. Since Carl and 
others have done so much work on the thrift patch already, I think it makes 
sense to have this patch committed now. Thanks!

> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

2010-11-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934682#action_12934682
 ] 

Namit Jain commented on HIVE-1648:
--

In SemanticAnalyzer:addStatsTask:

 } else {
6177  List children = (List) op.getChildren();
6178  if (children != null) {
6179for (Node child : children) {
6180  opsToProcess.add((Operator) 
child);
6181}
6182  }


why is the above code block needed ? TableScan can only be at the top.


Also, can you check for Conditional Tasks in addition to MapRedTask ?


> Automatically gathering stats when reading a table/partition
> 
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Ning Zhang
>Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to 
> gathering stats. This requires additional scan of the data. Stats gathering 
> can be piggy-backed on TableScanOperator whenever a table/partition is 
> scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1648) Automatically gathering stats when reading a table/partition

2010-11-22 Thread Namit Jain (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934683#action_12934683
 ] 

Namit Jain commented on HIVE-1648:
--

Otherwise, it looks OK

> Automatically gathering stats when reading a table/partition
> 
>
> Key: HIVE-1648
> URL: https://issues.apache.org/jira/browse/HIVE-1648
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Ning Zhang
>Assignee: Paul Butler
> Attachments: HIVE-1648.2.patch, HIVE-1648.3.patch, HIVE-1648.patch
>
>
> HIVE-1361 introduces a new command 'ANALYZE TABLE T COMPUTE STATISTICS' to 
> gathering stats. This requires additional scan of the data. Stats gathering 
> can be piggy-backed on TableScanOperator whenever a table/partition is 
> scanned (given not LIMIT operator). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread Carl Steinbach (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-1526:
-

Status: Open  (was: Patch Available)

> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1526) Hive should depend on a release version of Thrift

2010-11-22 Thread He Yongqiang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934685#action_12934685
 ] 

He Yongqiang commented on HIVE-1526:


ok. i am fine to get this committed before HIVE-78.

> Hive should depend on a release version of Thrift
> -
>
> Key: HIVE-1526
> URL: https://issues.apache.org/jira/browse/HIVE-1526
> Project: Hive
>  Issue Type: Task
>  Components: Build Infrastructure, Clients
>Reporter: Carl Steinbach
>Assignee: Carl Steinbach
> Fix For: 0.7.0
>
> Attachments: HIVE-1526-no-codegen.3.patch.txt, HIVE-1526.2.patch.txt, 
> HIVE-1526.3.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar, 
> serde2_test.patch, svn_rm.sh, thrift-0.5.0.jar, thrift-fb303-0.5.0.jar
>
>
> Hive should depend on a release version of Thrift, and ideally it should use 
> Ivy to resolve this dependency.
> The Thrift folks are working on adding Thrift artifacts to a maven repository 
> here: https://issues.apache.org/jira/browse/THRIFT-363

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1804) Mapjoin will fail if there are no files associating with the join tables

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1804:
-

Status: Patch Available  (was: Open)

If the parition desc is empty, just create a empty hash table file

> Mapjoin will fail if there are no files associating with the join tables
> 
>
> Key: HIVE-1804
> URL: https://issues.apache.org/jira/browse/HIVE-1804
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Fix For: 0.7.0
>
>
> If there are some empty tables without any file associated, the map join will 
> fail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1804) Mapjoin will fail if there are no files associating with the join tables

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1804:
-

Attachment: hive-1804-1.patch

If the partition desc is empty, then create an empty hashtable file for it.

> Mapjoin will fail if there are no files associating with the join tables
> 
>
> Key: HIVE-1804
> URL: https://issues.apache.org/jira/browse/HIVE-1804
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Liyin Tang
>Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: hive-1804-1.patch
>
>
> If there are some empty tables without any file associated, the map join will 
> fail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext

2010-11-22 Thread Liyin Tang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liyin Tang updated HIVE-1785:
-

Attachment: hive-1785_6.patch

Thanks for the careful review and sorry to submit the wrong patch before.
This patch makes the all changes according to the discussion before and clears 
irrelevant code.
Please review.

> change Pre/Post Query Hooks to take in 1 parameter: HookContext
> ---
>
> Key: HIVE-1785
> URL: https://issues.apache.org/jira/browse/HIVE-1785
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Namit Jain
>Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, 
> hive_1758_5.patch, hive_1785_1.patch, hive_1785_2.patch
>
>
> This way, it would be possible to add new parameters to the hooks without 
> changing the existing hooks.
> This will be a incompatible change, and all the hooks need to change to the 
> new API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext

2010-11-22 Thread John Sichi (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-1785:
-

Attachment: (was: hive_1758_5.patch)

> change Pre/Post Query Hooks to take in 1 parameter: HookContext
> ---
>
> Key: HIVE-1785
> URL: https://issues.apache.org/jira/browse/HIVE-1785
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: Namit Jain
>Assignee: Liyin Tang
> Fix For: 0.7.0
>
> Attachments: hive-1785_3.patch, hive-1785_4.patch, hive-1785_6.patch, 
> hive_1785_1.patch, hive_1785_2.patch
>
>
> This way, it would be possible to add new parameters to the hooks without 
> changing the existing hooks.
> This will be a incompatible change, and all the hooks need to change to the 
> new API

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-78) Authorization infrastructure for Hive

2010-11-22 Thread He Yongqiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

He Yongqiang updated HIVE-78:
-

Status: Patch Available  (was: Open)

> Authorization infrastructure for Hive
> -
>
> Key: HIVE-78
> URL: https://issues.apache.org/jira/browse/HIVE-78
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Ashish Thusoo
>Assignee: He Yongqiang
> Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, 
> hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, 
> HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, 
> HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, 
> HIVE-78.5.no_thrift.patch, hive-78.diff
>
>
> Allow hive to integrate with existing user repositories for authentication 
> and authorization infromation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-78) Authorization infrastructure for Hive

2010-11-22 Thread He Yongqiang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-78?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

He Yongqiang updated HIVE-78:
-

Attachment: HIVE-78.5.no_thrift.patch
HIVE-78.5.complete.patch

A new patch.

Had an internal group code review, the main changes are:
1) instead of calling metastore again to get partition's privilege information, 
pack user's privileges in Partition object when getting partition.
2) added a few configs for grant behavior on new tables.


  hive.exec.security.authorization.table.owner.grants
  
  the privileges automatically granted to the owner



  hive.exec.security.authorization.table.user.grants
  
  the privileges automatically granted to some users whenenve a 
table gets created. 
   An example like "userX,userY:select;userZ:create" will grant select 
privilege to userX and userY, 
   and grant create privilege to userZ whenenve a new table 
created.



  hive.exec.security.authorization.table.group.grants
  
  the privileges automatically granted to some groups whenenve a 
table gets created. 
   An example like "groupX,groupY:select;groupZ:create" will grant select 
privilege to groupX and groupY, 
   and grant create privilege to groupZ whenenve a new table 
created.



  hive.exec.security.authorization.table.role.grants
  
  the privileges automatically granted to some groups whenenve a 
table gets created. 
   An example like "roleX,roleY:select;roleZ:create" will grant select 
privilege to roleX and roleY, 
   and grant create privilege to roleZ whenenve a new table 
created.


3) changed privilege 'Overwrite' to 'update'

> Authorization infrastructure for Hive
> -
>
> Key: HIVE-78
> URL: https://issues.apache.org/jira/browse/HIVE-78
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Ashish Thusoo
>Assignee: He Yongqiang
> Attachments: createuser-v1.patch, hive-78-metadata-v1.patch, 
> hive-78-syntax-v1.patch, HIVE-78.1.nothrift.patch, HIVE-78.1.thrift.patch, 
> HIVE-78.2.nothrift.patch, HIVE-78.2.thrift.patch, HIVE-78.4.complete.patch, 
> HIVE-78.4.no_thrift.patch, HIVE-78.5.complete.patch, 
> HIVE-78.5.no_thrift.patch, hive-78.diff
>
>
> Allow hive to integrate with existing user repositories for authentication 
> and authorization infromation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1801) HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice

2010-11-22 Thread Siying Dong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siying Dong updated HIVE-1801:
--

Attachment: HIVE-1802.1.patch

address Yongqiang's comment.

> HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
> 
>
> Key: HIVE-1801
> URL: https://issues.apache.org/jira/browse/HIVE-1801
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
> Attachments: HIVE-1801.1.patch, HIVE-1802.1.patch
>
>
> HiveInputFormat or CombineHiveInputFormat RCFile.Reader.sync() twice. One in 
> getReader() and one in initIOContext(). We can avoid the latter one by read 
> the sync() position of the former one.
> We also sync() twice for SequenceFile but since SequenceFileReader is not a 
> part of Hive code, maybe we should be careful when depending on the 
> implementation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1801) HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice

2010-11-22 Thread Siying Dong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siying Dong updated HIVE-1801:
--

Status: Patch Available  (was: Open)

> HiveInputFormat or CombineHiveInputFormat always sync blocks of RCFile twice
> 
>
> Key: HIVE-1801
> URL: https://issues.apache.org/jira/browse/HIVE-1801
> Project: Hive
>  Issue Type: Improvement
>Reporter: Siying Dong
>Assignee: Siying Dong
> Attachments: HIVE-1801.1.patch, HIVE-1802.1.patch
>
>
> HiveInputFormat or CombineHiveInputFormat RCFile.Reader.sync() twice. One in 
> getReader() and one in initIOContext(). We can avoid the latter one by read 
> the sync() position of the former one.
> We also sync() twice for SequenceFile but since SequenceFileReader is not a 
> part of Hive code, maybe we should be careful when depending on the 
> implementation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.