RE: wrong number of records loaded to a table is returned by Hive

2010-10-01 Thread Steven Wong
Based on my cursory code inspection, the non-final row count is set when ExecDriver.progress calls ss.getHiveHistory().setCounters(...) inside the while loop, and we need to add the same call after the while loop (after the last updateCounters call at the end) to set the final row count. From:

[jira] Resolved: (HIVE-1606) For a null value in a string column, JDBC driver returns the string "NULL"

2010-09-27 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong resolved HIVE-1606. --- Fix Version/s: 0.7.0 Resolution: Fixed Fixed by HIVE-1378. > For a null value in a str

[jira] Commented: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-27 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915452#action_12915452 ] Steven Wong commented on HIVE-1378: --- Thanks, Ning/Zheng/John! > Return value

RE: Logging level

2010-09-27 Thread Steven Wong
Hmm, it doesn't seem possible before MAPREDUCE-336 is fixed. -Original Message- From: Steven Wong [mailto:sw...@netflix.com] Sent: Monday, September 27, 2010 10:31 AM To: hive-u...@hadoop.apache.org Cc: Subject: RE: Logging level Yes, I need a way to change the logging level o

RE: Logging level

2010-09-27 Thread Steven Wong
] Sent: Sunday, September 26, 2010 1:51 AM To: hive-u...@hadoop.apache.org Cc: Subject: Re: Logging level You need to set hadoop looging level to debug if you are looking at map task logs. I guess hadoop.root.logger is your friend. On Sun, Sep 26, 2010 at 12:06 AM, Steven Wong wrote: > >

RE: Logging level

2010-09-26 Thread Steven Wong
hive -hiveconf hive.root.logger=DEBUG,DRFA On Sep 25, 2010, at 11:27 PM, Steven Wong wrote: How can I control the logging level of Hive code that runs in the mappers/reducers? For example, how to set it to DEBUG? Thanks. Steven

Logging level

2010-09-25 Thread Steven Wong
How can I control the logging level of Hive code that runs in the mappers/reducers? For example, how to set it to DEBUG? Thanks. Steven

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-24 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.7.patch This new patch should be compatible with Hadoop 0.17.2.1. > Return va

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-23 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.6.patch Regenerated patch based on r1000539. Interestingly, I haven't ha

[jira] Commented: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-23 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914288#action_12914288 ] Steven Wong commented on HIVE-1378: --- Agh. What keeps conflicting every time? I'

Re: Review Request: Fix HIVE-1378: Return value for map, array, and struct needs to return a string.

2010-09-23 Thread Steven Wong
--- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/828/ --- (Updated 2010-09-23 14:06:40.052031) Review request for Hive Developers. Change

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-23 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.5.patch New patch moves ArrayList construction and adds back schema mismatch

[jira] Updated: (HIVE-1664) Eclipse build broken

2010-09-22 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1664: -- Status: Patch Available (was: Open) Fix Version/s: 0.7.0 > Eclipse build bro

[jira] Updated: (HIVE-1664) Eclipse build broken

2010-09-22 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1664: -- Attachment: HIVE-1664.classpath.2.patch Here you go. > Eclipse build bro

RE: [jira] Updated: (HIVE-1609) Support partition filtering in metastore

2010-09-22 Thread Steven Wong
John, I patched .classpath, but the build still fails, with other errors. Please look at https://issues.apache.org/jira/browse/HIVE-1664. Steven -Original Message- From: John Sichi [mailto:jsi...@facebook.com] Sent: Tuesday, September 21, 2010 7:04 PM To: Cc: Steven Wong Subject: Re

[jira] Updated: (HIVE-1664) Eclipse build broken

2010-09-22 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1664: -- Attachment: HIVE-1664.classpath.patch The attached HIVE-1664.classpath.patch fixes the .classpath

[jira] Created: (HIVE-1664) Eclipse build broken

2010-09-22 Thread Steven Wong (JIRA)
: Steven Wong After updating trunk to r999644, Eclipse build is broken. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

Re: Review Request: Fix HIVE-1378: Return value for map, array, and struct needs to return a string.

2010-09-21 Thread Steven Wong
ally generated e-mail. To reply, visit: http://review.cloudera.org/r/828/#review1273 --- On 2010-09-20 12:55:10, Steven Wong wrote: > > --- > This is an automatical

Re: Review Request: Fix HIVE-1378: Return value for map, array, and struct needs to return a string.

2010-09-21 Thread Steven Wong
--- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/828/ --- (Updated 2010-09-21 17:55:31.694691) Review request for Hive Developers. Change

RE: [jira] Updated: (HIVE-1609) Support partition filtering in metastore

2010-09-21 Thread Steven Wong
Did this check-in break the Eclipse build? -Original Message- From: John Sichi (JIRA) [mailto:j...@apache.org] Sent: Tuesday, September 21, 2010 2:16 PM To: hive-dev@hadoop.apache.org Subject: [jira] Updated: (HIVE-1609) Support partition filtering in metastore [ https://issues.a

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-21 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.4.patch Regenerated patch based on r999712. > Return value for map, array,

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-21 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.3.patch New patch includes fixing nested map/array/struct. > Return value

[jira] Commented: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-21 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12913139#action_12913139 ] Steven Wong commented on HIVE-1656: --- Thanks, John! > All TestJdbcDriver test cas

[jira] Commented: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-20 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12912697#action_12912697 ] Steven Wong commented on HIVE-1656: --- Please review the attached patch. &

[jira] Updated: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-20 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1656: -- Attachment: HIVE-1656.1.patch > All TestJdbcDriver test cases fail in Eclipse unless a property

[jira] Assigned: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-20 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong reassigned HIVE-1656: - Assignee: Steven Wong > All TestJdbcDriver test cases fail in Eclipse unless a property is ad

[jira] Updated: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-20 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1656: -- Status: Patch Available (was: Open) > All TestJdbcDriver test cases fail in Eclipse unless a prope

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-20 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.2.patch Add config to hive-default.xml. > Return value for map, array,

[jira] Created: (HIVE-1656) All TestJdbcDriver test cases fail in Eclipse unless a property is added in run config

2010-09-17 Thread Steven Wong (JIRA)
: Hadoop Hive Issue Type: Bug Components: Build Infrastructure Affects Versions: 0.7.0 Reporter: Steven Wong All TestJdbcDriver test cases fail in Eclipse, unless I add the following property in the TestJdbc run configuration ("Arguments" tab -->

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-17 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.1.patch Regenerated patch based on r997983. > Return value for map, array,

[jira] Commented: (HIVE-1646) Hive 0.5 Build Crashing

2010-09-16 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910400#action_12910400 ] Steven Wong commented on HIVE-1646: --- @Stephen I just looked at the saved output o

[jira] Commented: (HIVE-1646) Hive 0.5 Build Crashing

2010-09-16 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910285#action_12910285 ] Steven Wong commented on HIVE-1646: --- I ran ant package and ant test as 2 sepa

[jira] Commented: (HIVE-1646) Hive 0.5 Build Crashing

2010-09-16 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910266#action_12910266 ] Steven Wong commented on HIVE-1646: --- A suggestion: Search through your ant test outpu

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-13 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Status: Patch Available (was: Open) Fix Version/s: 0.7.0 > Return value for map, array,

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-13 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Attachment: HIVE-1378.patch > Return value for map, array, and struct needs to return a str

RE: Help with setting up Eclipse for Hive

2010-09-10 Thread Steven Wong
Adding -Dtest.warehouse.dir="${workspace_loc:trunk}/build/ql/test/data/warehouse" to the VM arguments in run config solves the problem. From: Steven Wong Sent: Thursday, August 19, 2010 10:44 AM To: hive-dev@hadoop.apache.org Subject: Help with setting up Eclipse for Hive I fol

[jira] Created: (HIVE-1631) JDBC driver returns wrong precision, scale, or column size for some data types

2010-09-10 Thread Steven Wong (JIRA)
Issue Type: Bug Components: Drivers Affects Versions: 0.7.0 Reporter: Steven Wong Priority: Minor For some data types, these methods return values that do not conform to the JDBC spec: org.apache.hadoop.hive.jdbc.HiveResultSetMetaData.getPrecision(int

[jira] Updated: (HIVE-1378) Return value for map, array, and struct needs to return a string

2010-09-10 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Wong updated HIVE-1378: -- Summary: Return value for map, array, and struct needs to return a string (was: Return value for

RE: Deserializing map column via JDBC (HIVE-1378)

2010-09-03 Thread Steven Wong
OK, I misunderstood that the input and output serdes for script must be the same type. -Original Message- From: Zheng Shao [mailto:zs...@facebook.com] Sent: Friday, September 03, 2010 6:17 AM To: Steven Wong; hive-dev@hadoop.apache.org Subject: RE: Deserializing map column via JDBC

RE: Deserializing map column via JDBC (HIVE-1378)

2010-09-02 Thread Steven Wong
essage- From: Zheng Shao [mailto:zs...@facebook.com] Sent: Thursday, September 02, 2010 7:19 PM To: Steven Wong; hive-dev@hadoop.apache.org Subject: RE: Deserializing map column via JDBC (HIVE-1378) Earlier there was no multi-level delimited format - the only way is first-level delimited, and t

RE: Deserializing map column via JDBC (HIVE-1378)

2010-09-02 Thread Steven Wong
Why was/is useJSONforLazy needed? What's the historical background? -Original Message- From: Zheng Shao [mailto:zs...@facebook.com] Sent: Thursday, September 02, 2010 7:11 PM To: Steven Wong; hive-dev@hadoop.apache.org Subject: RE: Deserializing map column via JDBC (HIVE-1378)

RE: Deserializing map column via JDBC (HIVE-1378)

2010-09-02 Thread Steven Wong
nticAnalyzer.getTableDescFromSerDe -> PlanUtis.getTableDesc What is it all about and how should we untangle it (ideally get rid of SERIALIZATION_USE_JSON_OBJECTS)? Thanks. Steven -Original Message- From: Zheng Shao [mailto:zs...@facebook.com] Sent: Wednesday, September 01, 2

RE: Deserializing map column via JDBC (HIVE-1378)

2010-09-01 Thread Steven Wong
[mailto:zs...@facebook.com] Sent: Wednesday, September 01, 2010 3:22 AM To: Steven Wong; hive-dev@hadoop.apache.org; John Sichi Cc: Jerome Boulon Subject: RE: Deserializing map column via JDBC (HIVE-1378) Hi Steven, Sorry for the late reply. The email slipped my eye... This issue was brought up

RE: Deserializing map column via JDBC (HIVE-1378)

2010-08-31 Thread Steven Wong
remaining concern is that, for "select * from partitioned_table", 3(b) might require fixing HIVE-1573 together, because I hit some partition-column problem when I tried 3(b) in the debugger. I hope HIVE-1573 can be fixed separately, but I don't know yet, I'll have to see. Steven -Origi

[jira] Commented: (HIVE-1606) For a null value in a string column, JDBC driver returns the string "NULL"

2010-08-30 Thread Steven Wong (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904443#action_12904443 ] Steven Wong commented on HIVE-1606: --- Some other data types may have a similar pro

RE: Deserializing map column via JDBC (HIVE-1378)

2010-08-30 Thread Steven Wong
Any guidance on how I/we should proceed on HIVE-1378 and HIVE-1606? -Original Message- From: Steven Wong Sent: Friday, August 27, 2010 2:24 PM To: hive-dev@hadoop.apache.org; 'John Sichi' Cc: Zheng Shao; Jerome Boulon Subject: RE: Deserializing map column via JDBC (HIVE-1378)

RE: Deserializing map column via JDBC (HIVE-1378)

2010-08-27 Thread Steven Wong
usly require a server-side change. Both HIVE-1606 and HIVE-1378 (the jira at hand) can share some server-side change, if HIVE-1378 ends up changing the sever side too. Steven -Original Message- From: John Sichi [mailto:jsi...@facebook.com] Sent: Friday, August 27, 2010 11:2

[jira] Created: (HIVE-1606) For a null value in a string column, JDBC driver returns the string "NULL"

2010-08-27 Thread Steven Wong (JIRA)
adoop Hive Issue Type: Bug Components: Drivers Affects Versions: 0.7.0 Reporter: Steven Wong It should return null instead of "NULL". -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.

RE: Deserializing map column via JDBC (HIVE-1378)

2010-08-26 Thread Steven Wong
2010 12:11 PM To: Steven Wong Cc: Zheng Shao; hive-dev@hadoop.apache.org; Jerome Boulon Subject: Re: Deserializing map column via JDBC (HIVE-1378) If you replace DynamicSerDe with LazySimpleSerDe on the JDBC client side, can't you then tell it to expect JSON serialization for the maps? That w

RE: Deserializing map column via JDBC (HIVE-1378)

2010-08-24 Thread Steven Wong
on the client side, per Zheng's suggestion. Steven From: Zheng Shao [mailto:zs...@facebook.com] Sent: Monday, August 16, 2010 3:57 PM To: Steven Wong; hive-dev@hadoop.apache.org Cc: Jerome Boulon Subject: RE: Deserializing map column via JDBC (HIVE-1378) I think the call to client.fetchOne s

[jira] Created: (HIVE-1575) get_json_object does not support JSON array at the root level

2010-08-19 Thread Steven Wong (JIRA)
Components: UDF Affects Versions: 0.7.0 Reporter: Steven Wong Currently, get_json_object(json_txt, path) always returns null if json_txt is not a JSON object (e.g. is a JSON array) at the root level. I have a table column of JSON arrays at the root level, but I can't

[jira] Created: (HIVE-1573) select * from partitioned table via JDBC does not return partition columns

2010-08-19 Thread Steven Wong (JIRA)
Issue Type: Bug Components: Drivers Affects Versions: 0.7.0 Reporter: Steven Wong select * from partitioned table via JDBC does not return partition columns. This behavior is different from via CLI. -- This message is automatically generated by JIRA. - You can reply to

Help with setting up Eclipse for Hive

2010-08-19 Thread Steven Wong
I followed http://wiki.apache.org/hadoop/Hive/GettingStarted/EclipseSetup and, when I tried to run the unit tests, Eclipse spit out the error "Variable references non-existent resource : ${workspace_loc:trunk}". I think the reason may be that the project is in /somepath/hive/trunk but the works

Deserializing map column via JDBC (HIVE-1378)

2010-08-13 Thread Steven Wong
Trying to work on HIVE-1378. My first step is to get the Hive JDBC driver to return actual values for mapcol in the result set of "select mapcol, bigintcol, stringcol from foo", where mapcol is a map column, instead of the current behavior of complaining that mapcol's column type is not recogniz