installing PIG

2010-07-20 Thread Ifeanyichukwu Osuji
Hello everyone, I am having problems running pig with hadoop. Every time i try to run pig in hadoopmode/mapreducemode i get this: pig-0.7.0$ bin/pig 10/07/20 13:03:11 INFO pig.Main: Logging error messages to: /home/hadoop/pig-0.7.0/pig_1279645391480.log 2010-07-20 13:03:11,809 [ma

RE: installing PIG

2010-07-20 Thread Xuefu Zhang
Did you setup env variable $HADOOP_CONF_DIR, which should points where your cluster configuration directory? --Xuefu -Original Message- From: Ifeanyichukwu Osuji [mailto:osujii...@potsdam.edu] Sent: Tuesday, July 20, 2010 10:04 AM To: pig-user@hadoop.apache.org Subject: installing PIG

including hadoop/conf directory

2010-07-20 Thread Ifeanyichukwu Osuji
Hi, I dont know how to include my hadoop/conf directory to my classpath. Can someone help? Thanks. Ifeanyichukwu Osuji

Re: including hadoop/conf directory

2010-07-20 Thread Dave Viner
Try this: # export PATH=/usr/local/pig-0.7.0/bin:$PATH # export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ # export HADOOPDIR=/usr/local/hadoop/conf # export PIG_PATH=/usr/local/pig-0.7.0/ # export PIG_CLASSPATH=$HADOOPDIR # export PIG_HADOOP_VERSION=0.20.2 JAVA_HOME must match what's in the /usr/l

how to set the conf in pig to use a cached file.

2010-07-20 Thread Kochis, Allan
Hi I would like to distribute cache a file of key value pairs. Mridul pointed that to do this set mapred.cached.archives=hdfs://host:port/mypath/file#link mapred.create.symlink=yes My question is how do I set these in pig? Thanks, Allan

Using Regex in Embedded Pig in Java

2010-07-20 Thread Matthew Smith
All, I am using pig embedded in Java and need to use matches in my pig job. However when I try to use escape characters in the pig line, the compiler complains. How do I use complex regex while embedding? Sample code that is throwing errors: myServer.registerQuery("filtered = FILTER fir

Using Regex

2010-07-20 Thread Matthew Smith
All, I am using pig embedded in Java and need to use matches in my pig job. However when I try to use escape characters in the pig line, the compiler complains. How do I use complex regex while embedding? Sample code that is throwing errors: myServer.registerQuery("filtered = FILTER fir

RE: Using Regex

2010-07-20 Thread Brian Adams
double escape, in our setup so a word break that is usually \b, needs to be '\\bthe\\b' -Original Message- From: Matthew Smith [mailto:matthew.sm...@g2-inc.com] Sent: Tue 7/20/2010 5:03 PM To: pig-user@hadoop.apache.org Subject: Using Regex All, I am using pig embedded in Java and

Re: pig + BigDecimal + DataType

2010-07-20 Thread ToddG
Follow Up: Thanks Dmitriy, that worked out really well. I just followed lead of builtin/IntAvg.java. In my case, I wound up storing intermediate BigDecimal values as chararrays...expensive to create all those objects, but conceptually simple. -Todd On 7/8/10 5:15 PM, Dmitriy Ryaboy wrote: th

PIG + Junit

2010-07-20 Thread ToddG
I'd like to include running various PIG scripts in my continuous build system. Of course, I'll only use small datasets for this, and in the beginning, I'll only target a local machine instance. However, this brings up several questions: Q: Whats the best way to run PIG from java? Here's what

RE: Using Regex

2010-07-20 Thread Matthew Smith
myServer.registerQuery("filtered = FILTER firstcut BY dIP matches '\\Q32.21.12.\\E*';"); throws runtime error: Exception in thread "main" org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at line 1, column 45. Encountered: "Q" (81), after : "\'\\" at org.apache.pig.im

Re: Using Regex

2010-07-20 Thread Dmitriy Ryaboy
It's a terrible thing, but keep adding slashes. Seriously. First, you need to escape the slash so Java passes it through. Then you need to escape each of those slashes so the pig parser passes it through. So 4 slashes should do it. On Tue, Jul 20, 2010 at 2:40 PM, Matthew Smith wrote: > myServer.

RE: Using Regex

2010-07-20 Thread Matthew Smith
Four slashes did it. Thanks! -Original Message- From: Dmitriy Ryaboy [mailto:dvrya...@gmail.com] Sent: Tuesday, July 20, 2010 6:04 PM To: pig-user@hadoop.apache.org Subject: Re: Using Regex It's a terrible thing, but keep adding slashes. Seriously. First, you need to escape the slash so

Re: PIG + Junit

2010-07-20 Thread Jeff Zhang
Hi Todd, The method registerQuery can not handle register and define statement. You should use method registerJar and and registerFunction instead. Another way is to put your script in a file and then use registerScript to execute the pig script. On Wed, Jul 21, 2010 at 5:26 AM, ToddG wrote:

Re: Using Regex in Embedded Pig in Java

2010-07-20 Thread Anthony Urso
You need to escape the backslashes, e.g. \\Q will be converted to \Q by javac before it is passed to Pig. On Tue, Jul 20, 2010 at 1:55 PM, Matthew Smith wrote: > All, > > > > I am using pig embedded in Java and need to use matches in my pig job. > However when I try to use escape characters in th

Re: PIG + Junit

2010-07-20 Thread Corbin Hoenes
Hey Todd we run against entire pig scripts with some helper classes we built basically they preprocess the variables then call register script but the test looks like this: @Before public void setUp() throws Exception { Helper.delete(OUT_FILE); runner = new PigRunner();

Re: PIG + Junit

2010-07-20 Thread Corbin Hoenes
Trying to attach the PigRunner class in case that helps give you a start using register script. On Jul 20, 2010, at 11:56 PM, Corbin Hoenes wrote: > Hey Todd we run against entire pig scripts with some helper classes we built > basically they preprocess the variables then call register script

Re: PIG + Junit

2010-07-20 Thread Corbin Hoenes
okay no attachments...try this gist: http://gist.github.com/484135 On Jul 21, 2010, at 12:02 AM, Corbin Hoenes wrote: > Trying to attach the PigRunner class in case that helps give you a start > using register script. > > > > On Jul 20, 2010, at 11:56 PM, Corbin Hoenes wrote: > >> Hey Todd