help running pig on ec2 cluster +S3

Hi, I am trying to run pig job to read HAR data from S3 and run the job on ec2 cluster and I am getting the following error: Any ideas on what could be running Error before Pig is launched ERROR 2999: Unexpected internal error. Failed to create DataStorage java.lang.

Re: hive queries from pig

Oh so "sh" does this then right? Make sure you 'exec" before running the "sh" command (that way you ensure the store finished before the sh is executed) D On Mon, Nov 14, 2011 at 2:47 PM, Stan Rosenberg wrote: > On Mon, Nov 14, 2011 at 5:30 PM, Dmitriy Ryaboy wrote: >> If you manually create th

Re: register udf

Hi Dan, to use UDFs you just need to put the relative class files in a jar, and register it in the pig script. No need to import or anything else. Cheers, -- Gianmarco On Mon, Nov 14, 2011 at 23:40, Dan Yi wrote: > hi, all > > i have a big projects, one of my project have all the pig scripts,

Re: hive queries from pig

On Mon, Nov 14, 2011 at 5:30 PM, Dmitriy Ryaboy wrote: > If you manually create the hive table + partitions to match the format > Pig writes things in, it should just work. Hive table already exists. However, we don't want to write directly into its warehouse location because it may result in a

register udf

hi, all i have a big projects, one of my project have all the pig scripts, and other one have all the java file as udfs for the pig scripts. i use eclipse/maven to manage all the projects, just wonder how i can use the udfs java file under other project for my pig scripts? i can only register t

Re: hive queries from pig

If you manually create the hive table + partitions to match the format Pig writes things in, it should just work.  HCatalog is about doing the deep integration; if you want deeper integration than just matching up formats and metadata, you will pretty much wind up rewriting HCat... For your second

Re: hive queries from pig

On Mon, Nov 14, 2011 at 3:08 PM, Dmitriy Ryaboy wrote: > My lack of imagination is showing -- can you explain what you mean by > integrating hive queries with pig, For example, we implemented a storage function which creates path partitioning based on a given sequence of columns; the output is st

Re: hive queries from pig

My lack of imagination is showing -- can you explain what you mean by integrating hive queries with pig, and what the problems with hcatalog are? For thrift, you might want to check jruby integration (https://issues.apache.org/jira/browse/PIG-2317) -Dmitriy On Mon, Nov 14, 2011 at 7:44 AM, Stan

Re: creating a graph over time

Just a heads up: I have a cleaner version (with tests!) here: https://issues.apache.org/jira/browse/PIG-2364 If you're still using this, I heavily suggest using the new version. 2011/11/4 Marco Cadetg > Yeha, that is awesome. Thank you very much Jonathan. > -Marco > > On Wed, Nov 2, 2011 at 7:5

hive queries from pig

Hi, We are trying to brainstorm on how best to integrate hive queries into pig. All suggestions are greatly appreciated! Note, we are trying to use hcatalog but there are a couple of problems with that approach. We also considered using jython to communicate with a thrift server but jython seems

Re: store json in pig?

A JsonStorage UDF has recently been added to Pig (see https://issues.apache.org/jira/browse/PIG-2332). It will be released as part of 0.10, but you can get it now by building the latest code. Alan. On Nov 13, 2011, at 10:14 PM, Kat Huang wrote: > Is there any way of storing in Json format in

store json in pig?

Is there any way of storing in Json format in pig? I haven't been able to find any, but thought I'd put this question out there if anyone has come up with a workaround