Hello everyone,
I am having problems running pig with hadoop. Every time i try
to run pig in hadoopmode/mapreducemode i get this:
pig-0.7.0$ bin/pig
10/07/20 13:03:11 INFO pig.Main: Logging error messages to:
/home/hadoop/pig-0.7.0/pig_1279645391480.log
2010-07-20 13:03:11,809 [ma
Did you setup env variable $HADOOP_CONF_DIR, which should points where your
cluster configuration directory?
--Xuefu
-Original Message-
From: Ifeanyichukwu Osuji [mailto:osujii...@potsdam.edu]
Sent: Tuesday, July 20, 2010 10:04 AM
To: pig-user@hadoop.apache.org
Subject: installing PIG
Hi,
I dont know how to include my hadoop/conf directory to my
classpath. Can someone help? Thanks.
Ifeanyichukwu Osuji
Try this:
# export PATH=/usr/local/pig-0.7.0/bin:$PATH
# export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
# export HADOOPDIR=/usr/local/hadoop/conf
# export PIG_PATH=/usr/local/pig-0.7.0/
# export PIG_CLASSPATH=$HADOOPDIR
# export PIG_HADOOP_VERSION=0.20.2
JAVA_HOME must match what's in the /usr/l
Hi
I would like to distribute cache a file of key value pairs.
Mridul pointed that to do this set
mapred.cached.archives=hdfs://host:port/mypath/file#link
mapred.create.symlink=yes
My question is how do I set these in pig?
Thanks,
Allan
All,
I am using pig embedded in Java and need to use matches in my pig job.
However when I try to use escape characters in the pig line, the
compiler complains. How do I use complex regex while embedding?
Sample code that is throwing errors:
myServer.registerQuery("filtered = FILTER fir
All,
I am using pig embedded in Java and need to use matches in my pig job.
However when I try to use escape characters in the pig line, the
compiler complains. How do I use complex regex while embedding?
Sample code that is throwing errors:
myServer.registerQuery("filtered = FILTER fir
double escape, in our setup
so a word break that is usually \b, needs to be '\\bthe\\b'
-Original Message-
From: Matthew Smith [mailto:matthew.sm...@g2-inc.com]
Sent: Tue 7/20/2010 5:03 PM
To: pig-user@hadoop.apache.org
Subject: Using Regex
All,
I am using pig embedded in Java and
Follow Up: Thanks Dmitriy, that worked out really well. I just followed
lead of builtin/IntAvg.java. In my case, I wound up storing intermediate
BigDecimal values as chararrays...expensive to create all those objects,
but conceptually simple.
-Todd
On 7/8/10 5:15 PM, Dmitriy Ryaboy wrote:
th
I'd like to include running various PIG scripts in my continuous build
system. Of course, I'll only use small datasets for this, and in the
beginning, I'll only target a local machine instance. However, this
brings up several questions:
Q: Whats the best way to run PIG from java? Here's what
myServer.registerQuery("filtered = FILTER firstcut BY dIP matches
'\\Q32.21.12.\\E*';");
throws runtime error:
Exception in thread "main"
org.apache.pig.impl.logicalLayer.parser.TokenMgrError: Lexical error at
line 1, column 45. Encountered: "Q" (81), after : "\'\\"
at
org.apache.pig.im
It's a terrible thing, but keep adding slashes. Seriously.
First, you need to escape the slash so Java passes it through. Then you need
to escape each of those slashes so the pig parser passes it through. So 4
slashes should do it.
On Tue, Jul 20, 2010 at 2:40 PM, Matthew Smith wrote:
> myServer.
Four slashes did it. Thanks!
-Original Message-
From: Dmitriy Ryaboy [mailto:dvrya...@gmail.com]
Sent: Tuesday, July 20, 2010 6:04 PM
To: pig-user@hadoop.apache.org
Subject: Re: Using Regex
It's a terrible thing, but keep adding slashes. Seriously.
First, you need to escape the slash so
Hi Todd,
The method registerQuery can not handle register and define statement. You
should use method registerJar and and registerFunction instead.
Another way is to put your script in a file and then use registerScript to
execute the pig script.
On Wed, Jul 21, 2010 at 5:26 AM, ToddG wrote:
You need to escape the backslashes, e.g. \\Q will be converted to \Q
by javac before it is passed to Pig.
On Tue, Jul 20, 2010 at 1:55 PM, Matthew Smith wrote:
> All,
>
>
>
> I am using pig embedded in Java and need to use matches in my pig job.
> However when I try to use escape characters in th
Hey Todd we run against entire pig scripts with some helper classes we built
basically they preprocess the variables then call register script but the test
looks like this:
@Before
public void setUp() throws Exception {
Helper.delete(OUT_FILE);
runner = new PigRunner();
Trying to attach the PigRunner class in case that helps give you a start using
register script.
On Jul 20, 2010, at 11:56 PM, Corbin Hoenes wrote:
> Hey Todd we run against entire pig scripts with some helper classes we built
> basically they preprocess the variables then call register script
okay no attachments...try this gist:
http://gist.github.com/484135
On Jul 21, 2010, at 12:02 AM, Corbin Hoenes wrote:
> Trying to attach the PigRunner class in case that helps give you a start
> using register script.
>
>
>
> On Jul 20, 2010, at 11:56 PM, Corbin Hoenes wrote:
>
>> Hey Todd
18 matches
Mail list logo