[jira] Updated: (HIVE-1655) Adding consistency check at jobClose() when committing dynamic partitions

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-1655:
-

   Status: Resolved  (was: Patch Available)
 Hadoop Flags: [Reviewed]
Fix Version/s: 0.7.0
   Resolution: Fixed

committed. Thanks Ning

> Adding consistency check at jobClose() when committing dynamic partitions
> -
>
> Key: HIVE-1655
> URL: https://issues.apache.org/jira/browse/HIVE-1655
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Ning Zhang
>Assignee: Ning Zhang
> Fix For: 0.7.0
>
> Attachments: HIVE-1655.patch
>
>
> In case of dynamic partition insert, FileSinkOperator generated a directory 
> for a new partition and the files in the directory is named with '_tmp*'. 
> When a task succeed, the file is renamed to remove the "_tmp", which 
> essentially implement the "commit" semantics. A lot of exceptions could 
> happen (process got killed, machine dies etc.) could left the _tmp files 
> exist in the DP directory. These _tmp files should be deleted ("rolled back") 
> at successful jobClose(). After the deletion, we should also delete any empty 
> directories.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Build failed in Hudson: Hive-trunk-h0.18 #546

[Apache Hudson Server]

See 

Changes:

[jvs] HIVE-1656. All TestJdbcDriver test cases fail in Eclipse unless a
property is added in run config
(Steven Wong via jvs)

[jvs] HIVE-1628. Fix Base64TextInputFormat to be compatible with commons
codec 1.4
(Todd Lipcon via jvs)

[namit] HIVE-558. Make describe output better
(Thiruvel Thirumoolan via namit)

--
[...truncated 30279 lines...]
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src
[junit] POSTHOOK: Output: defa...@src
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src1
[junit] POSTHOOK: Output: defa...@src1
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_sequencefile
[junit] POSTHOOK: Output: defa...@src_sequencefile
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_thrift
[junit] POSTHOOK: Output: defa...@src_thrift
[junit] OK
[junit] Copying data from 

[junit] Loading data to table src_json
[junit] POSTHOOK: Output: defa...@src_json
[junit] OK
[junit] diff 

 

[junit] Done query: unknown_table1.q
[junit] Begin query: unknown_table2.q
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-08, hr=11)
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-08/hr=11
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-08, hr=12)
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-08/hr=12
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-09, hr=11)
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-09/hr=11
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcpart partition (ds=2008-04-09, hr=12)
[junit] POSTHOOK: Output: defa...@srcpart@ds=2008-04-09/hr=12
[junit] OK
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket
[junit] POSTHOOK: Output: defa...@srcbucket
[junit] OK
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

[junit] Loading data to table srcbucket2
[junit] POSTHOOK: Output: defa...@srcbucket2
[junit] OK
[junit] Copying data from 

Hudson build is back to normal : Hive-trunk-h0.20 #369

[Apache Hudson Server]

See 

Re: Review Request: Fix HIVE-1378: Return value for map, array, and struct needs to return a string.

[Steven Wong]

> On 2010-09-20 18:14:21, Zheng Shao wrote:
> >

Thanks for taking the time to review it! See my comments below.


> On 2010-09-20 18:14:21, Zheng Shao wrote:
> > trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java, 
> > line 179
> > 
> >
> > While this function works for most cases, there is no guarantee that it 
> > will work for nested map/array/struct, because toString() is not required 
> > to be implemented correctly for Objects in hive.
> > 
> > The preferred way would be to use 
> > ObjectInspectorUtils/ObjectInspectorConvertors to copy/convert  > ObjectInspector> to "Standard Object", and then convert to JDBC objects.
> >

It is a good point regarding toString. I didn't try nested container types; I'm 
trying nested map for the first time now and it doesn't work. Will fix. Will 
see how ObjectInspector* can help.


> On 2010-09-20 18:14:21, Zheng Shao wrote:
> > trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java, 
> > line 54
> > 
> >
> > Can we change the name to serializeField (here and in LazySimpleSerDe)?
> >

OK.


- Steven


---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/828/#review1273
---


On 2010-09-20 12:55:10, Steven Wong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/828/
> ---
> 
> (Updated 2010-09-20 12:55:10)
> 
> 
> Review request for Hive Developers.
> 
> 
> Summary
> ---
> 
> HIVE-1606 is also fixed as a side effect.
> 
> 
> This addresses bug HIVE-1378.
> http://issues.apache.org/jira/browse/HIVE-1378
> 
> 
> Diffs
> -
> 
>   trunk/build.xml 997983 
>   trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 997983 
>   trunk/conf/hive-default.xml 997983 
>   trunk/data/files/datatypes.txt PRE-CREATION 
>   trunk/data/scripts/input20_script 997983 
>   trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java 
> 997983 
>   trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 997983 
>   trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 
> 997983 
>   trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java 
> 997983 
>   trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcColumn.java 997983 
>   trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 997983 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 997983 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 997983 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 997983 
>   trunk/ql/src/test/results/clientpositive/binary_output_format.q.out 997983 
>   trunk/ql/src/test/results/compiler/plan/input20.q.xml 997983 
>   trunk/ql/src/test/results/compiler/plan/input4.q.xml 997983 
>   trunk/ql/src/test/results/compiler/plan/input5.q.xml 997983 
>   trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 
> PRE-CREATION 
>   
> trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
> 997983 
> 
> Diff: http://review.cloudera.org/r/828/diff
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Steven
> 
>

Re: Review Request: Fix HIVE-1378: Return value for map, array, and struct needs to return a string.

[Steven Wong]

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/828/
---

(Updated 2010-09-21 17:55:31.694691)


Review request for Hive Developers.


Changes
---

Fixed nested map/array/struct.


Summary
---

HIVE-1606 is also fixed as a side effect.


This addresses bug HIVE-1378.
http://issues.apache.org/jira/browse/HIVE-1378


Diffs (updated)
-

  trunk/build.xml 997983 
  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 997983 
  trunk/conf/hive-default.xml 997983 
  trunk/data/files/datatypes.txt PRE-CREATION 
  trunk/data/scripts/input20_script 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 
997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java 
997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcColumn.java 997983 
  trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 997983 
  trunk/ql/src/test/results/clientpositive/binary_output_format.q.out 997983 
  trunk/ql/src/test/results/compiler/plan/input20.q.xml 997983 
  trunk/ql/src/test/results/compiler/plan/input4.q.xml 997983 
  trunk/ql/src/test/results/compiler/plan/input5.q.xml 997983 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 
PRE-CREATION 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
997983 

Diff: http://review.cloudera.org/r/828/diff


Testing
---


Thanks,

Steven

[jira] Resolved: (HIVE-1621) Disable join filters for outer joins.

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu resolved HIVE-1621.
---

Resolution: Won't Fix

Since HIVE-1534 is committed, resolving this as won't fix.

> Disable join filters for outer joins.
> -
>
> Key: HIVE-1621
> URL: https://issues.apache.org/jira/browse/HIVE-1621
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>
> As suggested at [comment 
> |https://issues.apache.org/jira/browse/HIVE-1534?focusedCommentId=12907001&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12907001],
>  SemanticAnalyzer should give out error if join filter is specified for outer 
> joins.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1538) FilterOperator is applied twice with ppd on.

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913354#action_12913354
 ] 

John Sichi commented on HIVE-1538:
--

It would be cool to get this fixed; without it the predicate decomposition I 
added for HIVE-1226 is pointless.

> FilterOperator is applied twice with ppd on.
> 
>
> Key: HIVE-1538
> URL: https://issues.apache.org/jira/browse/HIVE-1538
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>
> With hive.optimize.ppd set to true, FilterOperator is applied twice. And it 
> seems second operator is always filtering zero rows.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HIVE-1654) select distinct should allow column name regex

[John Sichi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi reassigned HIVE-1654:


Assignee: Arvind Prabhakar  (was: John Sichi)

Arvind, this one is a good one to work on together with HIVE-1459 if you're up 
for it.

I think we should rewrite the tree when we see * and regex, replacing them with 
the actual column reference list, so that downstream code like DISTINCT doesn't 
have to special-case it.


> select distinct should allow column name regex
> --
>
> Key: HIVE-1654
> URL: https://issues.apache.org/jira/browse/HIVE-1654
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.6.0
>Reporter: John Sichi
>Assignee: Arvind Prabhakar
> Fix For: 0.7.0
>
>
> This works (matching column name foo):
> select `fo.*` from pokes;
> but this
> select distinct `fo.*` from pokes;
> gives
> FAILED: Error in semantic analysis: line 1:16 Invalid Table Alias or Column 
> Reference `fo.*`
> It should work consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1654) select distinct should allow column name regex

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913351#action_12913351
 ] 

John Sichi commented on HIVE-1654:
--

select distinct *

also chokes.


> select distinct should allow column name regex
> --
>
> Key: HIVE-1654
> URL: https://issues.apache.org/jira/browse/HIVE-1654
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.6.0
>Reporter: John Sichi
>Assignee: John Sichi
> Fix For: 0.7.0
>
>
> This works (matching column name foo):
> select `fo.*` from pokes;
> but this
> select distinct `fo.*` from pokes;
> gives
> FAILED: Error in semantic analysis: line 1:16 Invalid Table Alias or Column 
> Reference `fo.*`
> It should work consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Updated: (HIVE-1609) Support partition filtering in metastore

[John Sichi]

Oops, yeah, this happened the last time we upgraded datanucleus also (see 
HIVE-1373 where we fixed it).  If someone posts a patch which fixes .classpath, 
I'll commit it.

JVS

On Sep 21, 2010, at 6:51 PM, Steven Wong wrote:

> Did this check-in break the Eclipse build?
> 
> 
> -Original Message-
> From: John Sichi (JIRA) [mailto:j...@apache.org] 
> Sent: Tuesday, September 21, 2010 2:16 PM
> To: hive-dev@hadoop.apache.org
> Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore
> 
> 
> [ 
> https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>  ]
> 
> John Sichi updated HIVE-1609:
> -
> 
>  Status: Resolved  (was: Patch Available)
>Hadoop Flags: [Reviewed]
>  Resolution: Fixed
> 
> Committed.  Thanks Ajay!
> 
> 
>> Support partition filtering in metastore
>> 
>> 
>>Key: HIVE-1609
>>URL: https://issues.apache.org/jira/browse/HIVE-1609
>>Project: Hadoop Hive
>> Issue Type: New Feature
>> Components: Metastore
>>   Reporter: Ajay Kidave
>>   Assignee: Ajay Kidave
>>Fix For: 0.7.0
>> 
>>Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>> 
>> 
>> The metastore needs to have support for returning a list of partitions based 
>> on user specified filter conditions. This will be useful for tools which 
>> need to do partition pruning. Howl is one such use case. The way partition 
>> pruning is done during hive query execution need not be changed.
> 
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
> 
> 

RE: [jira] Updated: (HIVE-1609) Support partition filtering in metastore

[Steven Wong]

Did this check-in break the Eclipse build?


-Original Message-
From: John Sichi (JIRA) [mailto:j...@apache.org] 
Sent: Tuesday, September 21, 2010 2:16 PM
To: hive-dev@hadoop.apache.org
Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore


 [ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-1609:
-

  Status: Resolved  (was: Patch Available)
Hadoop Flags: [Reviewed]
  Resolution: Fixed

Committed.  Thanks Ajay!


> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

[Steven Wong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Wong updated HIVE-1378:
--

Attachment: HIVE-1378.4.patch

Regenerated patch based on r999712.

> Return value for map, array, and struct needs to return a string 
> -
>
> Key: HIVE-1378
> URL: https://issues.apache.org/jira/browse/HIVE-1378
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Drivers
>Reporter: Jerome Boulon
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1378.1.patch, HIVE-1378.2.patch, HIVE-1378.3.patch, 
> HIVE-1378.4.patch, HIVE-1378.patch
>
>
> In order to be able to select/display any data from JDBC Hive driver, return 
> value for map, array, and struct needs to return a string

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1378) Return value for map, array, and struct needs to return a string

[Ning Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913338#action_12913338
 ] 

Ning Zhang commented on HIVE-1378:
--

Steven, there are conflicts when applying to the latest trunk. Can you 
regenerate the patch?

> Return value for map, array, and struct needs to return a string 
> -
>
> Key: HIVE-1378
> URL: https://issues.apache.org/jira/browse/HIVE-1378
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Drivers
>Reporter: Jerome Boulon
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1378.1.patch, HIVE-1378.2.patch, HIVE-1378.3.patch, 
> HIVE-1378.patch
>
>
> In order to be able to select/display any data from JDBC Hive driver, return 
> value for map, array, and struct needs to return a string

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1378) Return value for map, array, and struct needs to return a string

[HBase Review Board (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913336#action_12913336
 ] 

HBase Review Board commented on HIVE-1378:
--

Message from: "Steven Wong" 

---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/828/
---

(Updated 2010-09-21 17:55:31.694691)


Review request for Hive Developers.


Changes
---

Fixed nested map/array/struct.


Summary
---

HIVE-1606 is also fixed as a side effect.


This addresses bug HIVE-1378.
http://issues.apache.org/jira/browse/HIVE-1378


Diffs (updated)
-

  trunk/build.xml 997983 
  trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 997983 
  trunk/conf/hive-default.xml 997983 
  trunk/data/files/datatypes.txt PRE-CREATION 
  trunk/data/scripts/input20_script 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java 
997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveResultSetMetaData.java 
997983 
  trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/JdbcColumn.java 997983 
  trunk/jdbc/src/test/org/apache/hadoop/hive/jdbc/TestJdbcDriver.java 997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
997983 
  trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 997983 
  trunk/ql/src/test/results/clientpositive/binary_output_format.q.out 997983 
  trunk/ql/src/test/results/compiler/plan/input20.q.xml 997983 
  trunk/ql/src/test/results/compiler/plan/input4.q.xml 997983 
  trunk/ql/src/test/results/compiler/plan/input5.q.xml 997983 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java 
PRE-CREATION 
  trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
997983 

Diff: http://review.cloudera.org/r/828/diff


Testing
---


Thanks,

Steven




> Return value for map, array, and struct needs to return a string 
> -
>
> Key: HIVE-1378
> URL: https://issues.apache.org/jira/browse/HIVE-1378
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Drivers
>Reporter: Jerome Boulon
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1378.1.patch, HIVE-1378.2.patch, HIVE-1378.3.patch, 
> HIVE-1378.patch
>
>
> In order to be able to select/display any data from JDBC Hive driver, return 
> value for map, array, and struct needs to return a string

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

[Steven Wong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steven Wong updated HIVE-1378:
--

Attachment: HIVE-1378.3.patch

New patch includes fixing nested map/array/struct.

> Return value for map, array, and struct needs to return a string 
> -
>
> Key: HIVE-1378
> URL: https://issues.apache.org/jira/browse/HIVE-1378
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Drivers
>Reporter: Jerome Boulon
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1378.1.patch, HIVE-1378.2.patch, HIVE-1378.3.patch, 
> HIVE-1378.patch
>
>
> In order to be able to select/display any data from JDBC Hive driver, return 
> value for map, array, and struct needs to return a string

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HIVE-1624) Patch to allows scripts in S3 location

[Vaibhav Aggarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Aggarwal reassigned HIVE-1624:
--

Assignee: Vaibhav Aggarwal

> Patch to allows scripts in S3 location
> --
>
> Key: HIVE-1624
> URL: https://issues.apache.org/jira/browse/HIVE-1624
> Project: Hadoop Hive
>  Issue Type: New Feature
>Reporter: Vaibhav Aggarwal
>Assignee: Vaibhav Aggarwal
> Attachments: HIVE-1624.patch
>
>
> I want to submit a patch which allows user to run scripts located in S3.
> This patch enables Hive to download the hive scripts located in S3 buckets 
> and execute them. This saves users the effort of copying scripts to HDFS 
> before executing them.
> Thanks
> Vaibhav

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1655) Adding consistency check at jobClose() when committing dynamic partitions

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913317#action_12913317
 ] 

Namit Jain commented on HIVE-1655:
--

+1

> Adding consistency check at jobClose() when committing dynamic partitions
> -
>
> Key: HIVE-1655
> URL: https://issues.apache.org/jira/browse/HIVE-1655
> Project: Hadoop Hive
>  Issue Type: Improvement
>Reporter: Ning Zhang
>Assignee: Ning Zhang
> Attachments: HIVE-1655.patch
>
>
> In case of dynamic partition insert, FileSinkOperator generated a directory 
> for a new partition and the files in the directory is named with '_tmp*'. 
> When a task succeed, the file is renamed to remove the "_tmp", which 
> essentially implement the "commit" semantics. A lot of exceptions could 
> happen (process got killed, machine dies etc.) could left the _tmp files 
> exist in the DP directory. These _tmp files should be deleted ("rolled back") 
> at successful jobClose(). After the deletion, we should also delete any empty 
> directories.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1658) Fix describe [extended] column formatting

[Paul Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913315#action_12913315
 ] 

Paul Yang commented on HIVE-1658:
-

Maybe another option is to have a pretty qualifier? To get your output, the 
user would have to type something like 'pretty describe '. Otherwise 
everything defaults to normal.

> Fix describe [extended] column formatting
> -
>
> Key: HIVE-1658
> URL: https://issues.apache.org/jira/browse/HIVE-1658
> Project: Hadoop Hive
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Paul Yang
>Assignee: Thiruvel Thirumoolan
>
> When displaying the column schema, the formatting should follow should be 
> nametypecomment
> to be inline with the previous formatting style for backward compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1627) Hive Join returns incorrect results if the join is (bigint = string)

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913310#action_12913310
 ] 

John Sichi commented on HIVE-1627:
--

To help verify this, could you provide a specific example?


> Hive Join returns incorrect results if the join is (bigint = string)
> 
>
> Key: HIVE-1627
> URL: https://issues.apache.org/jira/browse/HIVE-1627
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.5.0
>Reporter: Abhinav Gupta
>
> I was running a query joining on bigint column with string column.
> And, result was incorrect because only "16 bytes seemed to be compared". The 
> length of value more than 16 bytes when represented as base-10. 
> The problem was fixed once I changed the join to (bigint = cast (string as 
> bigint))
> Is the bug because of type conversion on join keys?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1651) ScriptOperator should not forward any output to downstream operators if an exception is happened

[Ning Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913301#action_12913301
 ] 

Ning Zhang commented on HIVE-1651:
--

Discussed with Joydeep offline. The side effects of failed task should be 
cleaned after the job finished. _tmp* files are already taken care of in the 
current code base. The only side effect that need to be taken care of is the 
empty directories created by failed dynamic partition inserts. This issue is 
addressed in HIVE-1655. 


> ScriptOperator should not forward any output to downstream operators if an 
> exception is happened
> 
>
> Key: HIVE-1651
> URL: https://issues.apache.org/jira/browse/HIVE-1651
> Project: Hadoop Hive
>  Issue Type: Bug
>Reporter: Ning Zhang
>Assignee: Ning Zhang
> Attachments: HIVE-1651.patch
>
>
> ScriptOperator spawns 2 threads for getting the stdout and stderr from the 
> script and then forward the output from stdout to downstream operators. In 
> case of any exceptions to the script (e.g., got killed), the ScriptOperator 
> got an exception and throw it to upstream operators until MapOperator got it 
> and call close(abort). Before the ScriptOperator.close() is called the script 
> output stream can still forward output to downstream operators. We should 
> terminate it immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1361) table/partition level statistics

[Ning Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ning Zhang updated HIVE-1361:
-

Status: Patch Available  (was: Open)

> table/partition level statistics
> 
>
> Key: HIVE-1361
> URL: https://issues.apache.org/jira/browse/HIVE-1361
> Project: Hadoop Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Ahmed M Aly
> Fix For: 0.7.0
>
> Attachments: HIVE-1361.2.patch, HIVE-1361.2_java_only.patch, 
> HIVE-1361.java_only.patch, HIVE-1361.patch, stats0.patch
>
>
> At the first step, we gather table-level stats for non-partitioned table and 
> partition-level stats for partitioned table. Future work could extend the 
> table level stats to partitioned table as well. 
> There are 3 major milestones in this subtask: 
>  1) extend the insert statement to gather table/partition level stats 
> on-the-fly.
>  2) extend metastore API to support storing and retrieving stats for a 
> particular table/partition. 
>  3) add an ANALYZE TABLE [PARTITION] statement in Hive QL to gather stats for 
> existing tables/partitions. 
> The proposed stats are:
> Partition-level stats: 
>   - number of rows
>   - total size in bytes
>   - number of files
>   - max, min, average row sizes
>   - max, min, average file sizes
> Table-level stats in addition to partition level stats:
>   - number of partitions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1361) table/partition level statistics

[Ning Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ning Zhang updated HIVE-1361:
-

Attachment: HIVE-1361.2.patch
HIVE-1361.2_java_only.patch

Uploading a new patch (including a full version and a Java_only version 
including XML build files) for review. This is against the latest trunk.

The major changes from the last patch include: 
  1) Make JDBC update/insert/select using PreparedStatement(). 
  2) In HBase, use HTable.delete(ArrayList) to speed up delete, and 
flushCommit() to batch update. 
  3) Refactor StatsTask to put stats into PartitionStatistics and 
TableStatistics so that it is easier to add new stats later. 
  4) Move WriteEntity creation from StatsTask to compile-time.

 I'm running tests again after refreshed to the latest trunk.

> table/partition level statistics
> 
>
> Key: HIVE-1361
> URL: https://issues.apache.org/jira/browse/HIVE-1361
> Project: Hadoop Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Ning Zhang
>Assignee: Ahmed M Aly
> Fix For: 0.7.0
>
> Attachments: HIVE-1361.2.patch, HIVE-1361.2_java_only.patch, 
> HIVE-1361.java_only.patch, HIVE-1361.patch, stats0.patch
>
>
> At the first step, we gather table-level stats for non-partitioned table and 
> partition-level stats for partitioned table. Future work could extend the 
> table level stats to partitioned table as well. 
> There are 3 major milestones in this subtask: 
>  1) extend the insert statement to gather table/partition level stats 
> on-the-fly.
>  2) extend metastore API to support storing and retrieving stats for a 
> particular table/partition. 
>  3) add an ANALYZE TABLE [PARTITION] statement in Hive QL to gather stats for 
> existing tables/partitions. 
> The proposed stats are:
> Partition-level stats: 
>   - number of rows
>   - total size in bytes
>   - number of files
>   - max, min, average row sizes
>   - max, min, average file sizes
> Table-level stats in addition to partition level stats:
>   - number of partitions

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1663) ql/src/java/org/apache/hadoop/hive/ql/parse/SamplePruner.java is empty

[He Yongqiang (JIRA)]

ql/src/java/org/apache/hadoop/hive/ql/parse/SamplePruner.java is empty
--

 Key: HIVE-1663
 URL: https://issues.apache.org/jira/browse/HIVE-1663
 Project: Hadoop Hive
  Issue Type: Bug
Reporter: He Yongqiang
Assignee: He Yongqiang


we should remove this empty file

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1662) Add file pruning into Hive.

[He Yongqiang (JIRA)]

Add file pruning into Hive.
---

 Key: HIVE-1662
 URL: https://issues.apache.org/jira/browse/HIVE-1662
 Project: Hadoop Hive
  Issue Type: New Feature
Reporter: He Yongqiang


now hive support filename virtual column. 
if a file name filter presents in a query, hive should be able to only add 
files which passed the filter to input paths.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1609) Support partition filtering in metastore

[John Sichi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Sichi updated HIVE-1609:
-

  Status: Resolved  (was: Patch Available)
Hadoop Flags: [Reviewed]
  Resolution: Fixed

Committed.  Thanks Ajay!


> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1534) Join filters do not work correctly with outer joins

[Namit Jain (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namit Jain updated HIVE-1534:
-

  Status: Resolved  (was: Patch Available)
Hadoop Flags: [Reviewed]
  Resolution: Fixed

Committed. Thanks Amareshwari

> Join filters do not work correctly with outer joins
> ---
>
> Key: HIVE-1534
> URL: https://issues.apache.org/jira/browse/HIVE-1534
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
> Attachments: patch-1534-1.txt, patch-1534-2.txt, patch-1534-3.txt, 
> patch-1534-4.txt, patch-1534.txt
>
>
>  SELECT * FROM T1 LEFT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T1.c1 < 10)
> and  SELECT * FROM T1 RIGHT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T2.c1 < 10)
> do not give correct results.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1661) Default values for parameters

[Namit Jain (JIRA)]

Default values for parameters
-

 Key: HIVE-1661
 URL: https://issues.apache.org/jira/browse/HIVE-1661
 Project: Hadoop Hive
  Issue Type: New Feature
  Components: Query Processor
Reporter: Namit Jain
Assignee: Siying Dong
 Fix For: 0.7.0


It would be good to have a default value for some hive parameters:

say RETENTION to be 30 days.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[Paul Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913239#action_12913239
 ] 

Paul Yang commented on HIVE-1609:
-

@Yongqiang - The new thrift function seems benign - what problems are you 
running into?

> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[He Yongqiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913240#action_12913240
 ] 

He Yongqiang commented on HIVE-1609:


[by several partition functions in my previous comment, i mean the existing 
partition functions.] So just want to make sure the ones added in this jira 
will work finely for python client. 

@john, pls go ahead commit this. This is a really good one to have. We can fix 
problems later if there are any.

> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913238#action_12913238
 ] 

John Sichi commented on HIVE-1609:
--

@Yongqiang:  ant test just passed for me.  Let me know if I should hold off on 
the commit until the python issues are resolved.

> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[He Yongqiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913234#action_12913234
 ] 

He Yongqiang commented on HIVE-1609:


Just want to make sure that this will aslo work in python client. I found there 
are some small problems when calling several partition functions from python.


> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1633) CombineHiveInputFormat fails with "cannot find dir for emptyFile"

[He Yongqiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913230#action_12913230
 ] 

He Yongqiang commented on HIVE-1633:


Amareshwari, by adding a testcase in TestHiveFileFormatUtils, you will be able 
to find out the underlying problem, and then can you post a patch for it?

> CombineHiveInputFormat fails with "cannot find dir for emptyFile"
> -
>
> Key: HIVE-1633
> URL: https://issues.apache.org/jira/browse/HIVE-1633
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Clients
>Reporter: Amareshwari Sriramadasu
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1657) join results are displayed wrongly for some complex joins

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913221#action_12913221
 ] 

John Sichi commented on HIVE-1657:
--

I've confirmed this as a bug by comparing the behavior with MySQL.

{noformat}
mysql> select * from table1;
+--+--+
| k| v|
+--+--+
|   12 |   35 | 
| NULL |   36 | 
|   48 |   40 | 
|   40 |   40 | 
| NULL |   45 | 
|  100 |  100 | 
+--+--+
6 rows in set (0.00 sec)

mysql> select * from myinput1;
+--+--+
| k| v|
+--+--+
|   12 |   35 | 
| NULL |   40 | 
|   48 | NULL | 
|  100 |  100 | 
+--+--+
4 rows in set (0.00 sec)

mysql> select * from table1 a right outer join myinput1 b on (a.v=b.v) left 
outer join table1 c on (b.k=c.k);
+--+--+--+--+--+--+
| k| v| k| v| k| v|
+--+--+--+--+--+--+
|   12 |   35 |   12 |   35 |   12 |   35 | 
|   48 |   40 | NULL |   40 | NULL | NULL | 
|   40 |   40 | NULL |   40 | NULL | NULL | 
| NULL | NULL |   48 | NULL |   48 |   40 | 
|  100 |  100 |  100 |  100 |  100 |  100 | 
+--+--+--+--+--+--+
5 rows in set (0.00 sec)
{noformat}


> join results are displayed wrongly for some complex joins
> -
>
> Key: HIVE-1657
> URL: https://issues.apache.org/jira/browse/HIVE-1657
> Project: Hadoop Hive
>  Issue Type: Bug
>Reporter: Amareshwari Sriramadasu
>
> Noticed that the output of the join is displayed wrongly for queries 
> involving more than one table and mixture of left and outer joins, with 
> different join conditions.
> For ex: SELECT * from T1 a RIGHT OUTER JOIN T2 b ON (a.value=b.value) LEFT 
> OUTER JOIN T3 c ON (b.key=c.key);
> displays the table T2 first, then T1 and T3.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1378) Return value for map, array, and struct needs to return a string

[HBase Review Board (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913164#action_12913164
 ] 

HBase Review Board commented on HIVE-1378:
--

Message from: "Steven Wong" 


bq.  On 2010-09-20 18:14:21, Zheng Shao wrote:
bq.  >

Thanks for taking the time to review it! See my comments below.


bq.  On 2010-09-20 18:14:21, Zheng Shao wrote:
bq.  > trunk/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java, 
line 179
bq.  > 
bq.  >
bq.  > While this function works for most cases, there is no guarantee that 
it will work for nested map/array/struct, because toString() is not required to 
be implemented correctly for Objects in hive.
bq.  > 
bq.  > The preferred way would be to use 
ObjectInspectorUtils/ObjectInspectorConvertors to copy/convert  to "Standard Object", and then convert to JDBC objects.
bq.  >

It is a good point regarding toString. I didn't try nested container types; I'm 
trying nested map for the first time now and it doesn't work. Will fix. Will 
see how ObjectInspector* can help.


bq.  On 2010-09-20 18:14:21, Zheng Shao wrote:
bq.  > 
trunk/serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java, 
line 54
bq.  > 
bq.  >
bq.  > Can we change the name to serializeField (here and in 
LazySimpleSerDe)?
bq.  >

OK.


- Steven


---
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/828/#review1273
---





> Return value for map, array, and struct needs to return a string 
> -
>
> Key: HIVE-1378
> URL: https://issues.apache.org/jira/browse/HIVE-1378
> Project: Hadoop Hive
>  Issue Type: Improvement
>  Components: Drivers
>Reporter: Jerome Boulon
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1378.1.patch, HIVE-1378.2.patch, HIVE-1378.patch
>
>
> In order to be able to select/display any data from JDBC Hive driver, return 
> value for map, array, and struct needs to return a string

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

[Steven Wong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913139#action_12913139
 ] 

Steven Wong commented on HIVE-1656:
---

Thanks, John!

> All TestJdbcDriver test cases fail in Eclipse unless a property is added in 
> run config
> --
>
> Key: HIVE-1656
> URL: https://issues.apache.org/jira/browse/HIVE-1656
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Build Infrastructure
>Affects Versions: 0.7.0
>Reporter: Steven Wong
>Assignee: Steven Wong
> Fix For: 0.7.0
>
> Attachments: HIVE-1656.1.patch
>
>
> All TestJdbcDriver test cases fail in Eclipse, unless I add the following 
> property in the TestJdbc run configuration ("Arguments" tab --> "VM 
> arguments" box):
> -Dtest.warehouse.dir="${workspace_loc:trunk}/build/ql/test/data/warehouse"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[John Sichi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913137#action_12913137
 ] 

John Sichi commented on HIVE-1609:
--

Running this one through tests now.


> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[Ajay Kidave (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913132#action_12913132
 ] 

Ajay Kidave commented on HIVE-1609:
---

@Paul : I have created HIVE-1660 for the optimizations to get_partitions_ps.

> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1624) Patch to allows scripts in S3 location

[Vaibhav Aggarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913124#action_12913124
 ] 

Vaibhav Aggarwal commented on HIVE-1624:


Thanks for looking at this. I will experiment with the suggested approach today.

> Patch to allows scripts in S3 location
> --
>
> Key: HIVE-1624
> URL: https://issues.apache.org/jira/browse/HIVE-1624
> Project: Hadoop Hive
>  Issue Type: New Feature
>Reporter: Vaibhav Aggarwal
> Attachments: HIVE-1624.patch
>
>
> I want to submit a patch which allows user to run scripts located in S3.
> This patch enables Hive to download the hive scripts located in S3 buckets 
> and execute them. This saves users the effort of copying scripts to HDFS 
> before executing them.
> Thanks
> Vaibhav

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1534) Join filters do not work correctly with outer joins

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913105#action_12913105
 ] 

Namit Jain commented on HIVE-1534:
--

+1

> Join filters do not work correctly with outer joins
> ---
>
> Key: HIVE-1534
> URL: https://issues.apache.org/jira/browse/HIVE-1534
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
> Attachments: patch-1534-1.txt, patch-1534-2.txt, patch-1534-3.txt, 
> patch-1534-4.txt, patch-1534.txt
>
>
>  SELECT * FROM T1 LEFT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T1.c1 < 10)
> and  SELECT * FROM T1 RIGHT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T2.c1 < 10)
> do not give correct results.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1660) Change get_partitions_ps to pass partition filter to database

[Ajay Kidave (JIRA)]

Change get_partitions_ps to pass partition filter to database
-

 Key: HIVE-1660
 URL: https://issues.apache.org/jira/browse/HIVE-1660
 Project: Hadoop Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Ajay Kidave
Assignee: Paul Yang


Support for doing partition pruning by passing the partition filter to the 
database is added by HIVE-1609. Changing get_partitions_ps to use this could 
result in performance improvement  for tables having large number of 
partitions. A listPartitionNamesByFilter API might be required for implementing 
this for use from Hive.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url

[Ning Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913081#action_12913081
 ] 

Ning Zhang commented on HIVE-1659:
--

parse_url currently support 2 signatures: parse_url(fullurl, 
'[QUERY|PATH|HOST|...]') and parse_url(fullurl, 'QUERY', '[ref|sk|...]'). In 
parse_url_tuple, the syntax is consolidated as parse_url_tuple(fullurl, 'HOST', 
'PATH', 'QUERY:ref', 'QUERY:sk',...). 

> parse_url_tuple:  a UDTF version of parse_url
> -
>
> Key: HIVE-1659
> URL: https://issues.apache.org/jira/browse/HIVE-1659
> Project: Hadoop Hive
>  Issue Type: New Feature
>Reporter: Ning Zhang
>
> The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. 
> However it can only extract an atomic value from the URL. If we want to 
> extract multiple piece of information, we need to call the function many 
> times. It is desirable to parse the URL once and extract all needed 
> information and return a tuple in a UDTF. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (HIVE-1659) parse_url_tuple: a UDTF version of parse_url

[Ning Zhang (JIRA)]

parse_url_tuple:  a UDTF version of parse_url
-

 Key: HIVE-1659
 URL: https://issues.apache.org/jira/browse/HIVE-1659
 Project: Hadoop Hive
  Issue Type: New Feature
Reporter: Ning Zhang


The UDF parse_url take s a URL, parse it and extract QUERY/PATH etc from it. 
However it can only extract an atomic value from the URL. If we want to extract 
multiple piece of information, we need to call the function many times. It is 
desirable to parse the URL once and extract all needed information and return a 
tuple in a UDTF. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1658) Fix describe [extended] column formatting

[Thiruvel Thirumoolan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913043#action_12913043
 ] 

Thiruvel Thirumoolan commented on HIVE-1658:


ok, will revert back the formatting of the columns (partitions also) and will 
leave the rest of the changes as is. That would mean the headers also will go 
away, the formatting doesnt look good with them and tab alone as separator.

> Fix describe [extended] column formatting
> -
>
> Key: HIVE-1658
> URL: https://issues.apache.org/jira/browse/HIVE-1658
> Project: Hadoop Hive
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Paul Yang
>Assignee: Thiruvel Thirumoolan
>
> When displaying the column schema, the formatting should follow should be 
> nametypecomment
> to be inline with the previous formatting style for backward compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HIVE-1609) Support partition filtering in metastore

[Namit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913001#action_12913001
 ] 

Namit Jain commented on HIVE-1609:
--

I meant, exposing it via the Hive QL directly.
I don't think there is a way to do that currently.

> Support partition filtering in metastore
> 
>
> Key: HIVE-1609
> URL: https://issues.apache.org/jira/browse/HIVE-1609
> Project: Hadoop Hive
>  Issue Type: New Feature
>  Components: Metastore
>Reporter: Ajay Kidave
>Assignee: Ajay Kidave
> Fix For: 0.7.0
>
> Attachments: hive_1609.patch, hive_1609_2.patch, hive_1609_3.patch
>
>
> The metastore needs to have support for returning a list of partitions based 
> on user specified filter conditions. This will be useful for tools which need 
> to do partition pruning. Howl is one such use case. The way partition pruning 
> is done during hive query execution need not be changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HIVE-1534) Join filters do not work correctly with outer joins

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated HIVE-1534:
--

Attachment: patch-1534-4.txt

Added hive.outerjoin.supports.filter configuration with above semantics. 
Updated testcase to repeat the filter queries in join_filters.q with 
configuration turned off.

> Join filters do not work correctly with outer joins
> ---
>
> Key: HIVE-1534
> URL: https://issues.apache.org/jira/browse/HIVE-1534
> Project: Hadoop Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
> Attachments: patch-1534-1.txt, patch-1534-2.txt, patch-1534-3.txt, 
> patch-1534-4.txt, patch-1534.txt
>
>
>  SELECT * FROM T1 LEFT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T1.c1 < 10)
> and  SELECT * FROM T1 RIGHT OUTER JOIN T2 ON (T1.c1=T2.c2 AND T2.c1 < 10)
> do not give correct results.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.