Re: MongoDB storage handler for HIVE

2011-11-17 Thread YC Huang
We only use their java driver, but do not change any code, is that also aware of GPL license? I'm glad if it could be in hive-contribs, thanks. YC On Friday, November 18, 2011, Ashutosh Chauhan wrote: > Great work YC. Alas, mongodb is GPL licensed, otherwise we could have it > in hive-contribs

Re: MongoDB storage handler for HIVE

2011-11-17 Thread Ashutosh Chauhan
Great work YC. Alas, mongodb is GPL licensed, otherwise we could have it in hive-contribs (ofcourse only if you have decided it to contribute). Ashutosh On Thu, Nov 17, 2011 at 18:47, Stephen Boesch wrote: > Nice idea! I have worked a bit with Mongo and am leaning towards hive . > This could be

Re: Severely hit by "curse of last reducer"

2011-11-17 Thread Mark Grover
Rohan, I took a look at the source code and wanted to share a couple of things: 1) Make sure the following 2 properties are being set to true (they are false by default): hive.optimize.skewjoin hive.auto.convert.join 2) The Hive source code that is causing the exception is: String path =

Re: MongoDB storage handler for HIVE

2011-11-17 Thread Stephen Boesch
Nice idea! I have worked a bit with Mongo and am leaning towards hive . This could be a nice combo. will check it out (pun intended) 2011/11/17 YC Huang > I just have a quick and dirty implementation of a MongoDB storage handler > for HIVE, the project is hosted on GitHub: > https://github.com/

MongoDB storage handler for HIVE

2011-11-17 Thread YC Huang
I just have a quick and dirty implementation of a MongoDB storage handler for HIVE, the project is hosted on GitHub: https://github.com/yc-huang/Hive-mongo. Since Hive table do not support 'update' efficiently, we use MongoDB to store those data, e.g. 'meta' data like user profile info, which need

Re: Severely hit by "curse of last reducer"

2011-11-17 Thread rohan monga
Hi Mark, Apologies for the thin details on the query :) Here is the error log http://pastebin.com/pqxh4d1u the job tracker doesn't show any errors. I am using hive-0.7, I did set a threshold for the query and sadly i couldn't find any more documentation on skewjoins other than the wiki. Thanks, --

Re: Severely hit by "curse of last reducer"

2011-11-17 Thread Mark Grover
Rohan, The short answer is: I don't know:-) If you could paste the log, I or someone else of the mailing list could be able to help. BTW, What version of Hive were you using? Did you set the threshold before running the query? Try to find some documentation online if can tell what all propertie

Re: Severely hit by "curse of last reducer"

2011-11-17 Thread rohan monga
Hi Mark, I have tried setting hive.optimize.skewjoin=true, but it get a NullPointerException after the first stage of the query completes. Why does that happen? Thanks, -- Rohan Monga On Thu, Nov 17, 2011 at 1:37 PM, Mark Grover wrote: > Ayon, > I see. From what you explained, skew join seems

Re: Severely hit by "curse of last reducer"

2011-11-17 Thread Mark Grover
Ayon, I see. From what you explained, skew join seems like what you want. Have you tried that already? Details on how skew join works are in this presentation. Jump to 15 minute mark if you want to just listen about skew joins. http://www.youtube.com/watch?v=OB4H3Yt5VWM I bet you could also fin

Re: Hive HBase wiki

2011-11-17 Thread John Sichi
It has been quite a while since those instructions were written, so maybe something has broken. There is a unit test for it (hbase-handler/src/test/queries/hbase_bulk.m) which is still passing. If you're running via CLI, logs by default go in /tmp/ Long-term, energy best expended on this wo

Hive HBase wiki

2011-11-17 Thread Ben West
Hey all, I'm having some trouble with the HBase bulk load, following the instructions from https://cwiki.apache.org/confluence/display/Hive/HBaseBulkLoad. In the last step ("Sort Data") I get: java.lang.RuntimeException: Hive Runtime Error while closing operators: java.io.IOException: No files

Re: Converting Array to a String

2011-11-17 Thread Tucker, Matt
I'm running into the same issue, and I see that it's addressed in HIVE-2223. In the meantime, I'm getting an error when trying to use the reflect() function : SELECT reflect("org.apache.commons.lang.StringUtils", "join", collectedSet), ... FAILED: Error in semantic analysis: Line 1:69 Argument t

Need Help In HBASE

2011-11-17 Thread Vikas Srivastava
hey Folks, I have some Queries regarding Hbase 1) When ever i try to update the latest record and read later from the client the reading time increases as the frequency of updates increase to more than 1 times. Is there any way we can improve the performance or reduce this delay. It is al

RE: hive0.6 metastore Problem for help

2011-11-17 Thread Chinna Rao Lalam
Hi, Pls check this property " javax.jdo.option.Multithreaded " it is introduced in trunk in 0.8 version as part of HIVE-2022 . If the value is true it will make JDO thread safe. Analyze this property can be added to 0.6 version or not if it works. Hope it helps, Chinna Rao Lalam _

hive0.6 metastore Problem for help

2011-11-17 Thread 黄逗
Hello everybody I had a problem when I used hive0.6 metastore. Is such There are a lot of timing task in my application run at 09 am to 11am every day, i Don't know from when to begin , a task got a exception, and then, all of the tasks are exception from this time. The content of the ta