t to the jar file for pig 0.8. once I remove it, everything works
fine.
Thanks Cheolsoo and Dmitriy for your help.
Dan
-Original Message-
From: Danfeng Li [mailto:d...@operasolutions.com]
Sent: Tuesday, March 12, 2013 11:26 PM
To: user@pig.apache.org
Subject: RE: pig 0.12.0 ERROR 2998:
're getting this error.
Thanks,
Cheolsoo
On Tue, Mar 12, 2013 at 4:45 PM, Danfeng Li wrote:
> Here's the output for the secretDebugCmd
>
> dli2@sig1:~/$ pig12 -secretDebugCmd -param input="s" -param output="t"
> -c b224G_1.pig Find hadoop at /usr/bin/h
ugCmd
this will show you what's actually on the classpath.
On Tue, Mar 12, 2013 at 3:32 PM, Danfeng Li wrote:
> I download version r11 (
> https://alexoutilities.googlecode.com/svn-history/r358/repository/com/
> google/common/google-guava/r11/google-guava-r11.jar) and put it in my
.
com.google.common.collect.ImmutableSet.of
11.0 is currently required.
On Tue, Mar 12, 2013 at 2:54 PM, Danfeng Li wrote:
> Thanks for the quick repsonse, which guava version I should use?
>
> -Original Message-
> From: Dmitriy Ryaboy [mailto:dvrya...@gmail.com]
> Sent: Tuesday, March 1
.
com.google.common.collect.ImmutableSet.of
Sounds like you have a bad (older? newer?) version of guava on the classpath.
On Tue, Mar 12, 2013 at 2:50 PM, Danfeng Li wrote:
> When I try to run pig 0.12.0, I got the following error
>
> $ pig12 -param input="t" -param output="s" -c b224G_1.pig
When I try to run pig 0.12.0, I got the following error
$ pig12 -param input="t" -param output="s" -c b224G_1.pig
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
2013-03-12 17:44:17,724 [main] INFO org.apache.pi
> Hi Dan,
>
> Are you using 0.11 or trunk?
>
> If you're using trunk, please take a look at PIG-3014.
> https://issues.apache.org/jira/browse/PIG-3014
>
> Thanks,
> Cheolsoo
>
>
>
> On Tue, Feb 26, 2013 at 4:09 PM, Danfeng Li
> wrote:
>
> >
On Fri, Feb 15, 2013 at 3:21 PM, Danfeng Li wrote:
> I got the following error when using the new built in function
> CurrentTime()
>
> 2013-02-15 14:42:37,228 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 2997: Unable to recrea
I got the following error when using the new built in function CurrentTime()
2013-02-15 14:42:37,228 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2997: Unable to recreate exception from backed error:
org.apache.pig.backend.executionengine.ExecException: ERROR 2080: Foreach
currently do
When I try to use the following statement
explain -brief A;
I got the following error
2013-02-06 19:18:34,250 [Low Memory Detector] INFO
org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call -
Collection threshold init = 526123008(513792K) used = 349984528(341781K)
comm
If I self cross a relation, I got the original relation, which is not expected.
The input: A.txt
1
2
3
The code:
A = load 'A.txt' as (id:chararray);
B = cross A, A;
dump B;
(1)
(2)
(3)
C = foreach A generate id;
D = cross A, C;
dump D;
(1,1)
(1,2)
(1,3)
(2,1)
(2,2)
(2,3)
(3,1)
(3,2)
(3,3)
Is th
tml#define-udfs especially the
section on SHIP.
Alan.
On Dec 20, 2012, at 10:01 AM, Danfeng Li wrote:
> I read alot of about pig can ship a tar file and untar it before execution.
> However, I couldn't find any example. Can someone provide an example?
>
> What I would like to do
regex. Then each slash needs to be escaped again to be
> used in a java string. Hence . Painful, yes.
>
>
> On Fri, Sep 7, 2012 at 2:06 PM, Danfeng Li wrote:
>
> > Hi,
> >
> > I'm little bit puzzled about REPLACE when there is backslash involved.
> >
Hi,
I'm little bit puzzled about REPLACE when there is backslash involved.
I want to replace all the "dir" in the string with "\\test\sub",
After a lot of try and error, I finally got it done, but I'm not sure why it is
the case. Here's my code
A = load 'a.txt' as (name:chararray);
B = foreach
Piggybank for a while and eventually retire it.
I am wondering what other people think. Please let me know if it is not a good
idea to move AvroStorage to core Pig from Piggybank.
Thanks,
Cheolsoo
On Tue, Aug 21, 2012 at 5:47 PM, Danfeng Li wrote:
> Thanks, Cheolsoo. That solve my problems.
>
ct the following string in the frontend:
"1#<1st avro schema>,2#<2nd avro schema>"
and pass it to backend via UdfContext. Now in backend, tasks parse this string
to get output schema for each store.
Thanks,
Cheolsoo
On Tue, Aug 21, 2012 at 4:38 PM, Danfeng Li wrote:
&g
Ok, I found the solution
Replace
Schema tupleSchema = new Schema(input.getFields());
With
Schema tupleSchema = new
Schema(input.getField(0).schema.getField(0).schema.getFields());
Will to the trick.
Thanks.
Dan
-Original Message-
From: Danfeng Li [mailto:d...@operasolutions.com
e.org
Subject: Re: how to get input schema in UDF
Chapter 10 in Alan Gates' excellent book "Programmin Pig" discusses this issue.
Robert Yerex
Data Scientist
Civitas Leaning
On Mon, Aug 13, 2012 at 3:43 PM, Danfeng Li wrote:
> I have a big, e.g. A: {(name: chararray,age: i
I have a big, e.g. A: {(name: chararray,age: int)}, I wrote a udf which adds 1
more field in the tuple inside the bag. E.g. B: {(name: chararray,age: int,
rank:int)}. Because the number of fields in the original bag is not fixed, e.g
I can have one more field such as gender:int.
In my udf, in o
he degradation in performance seems to have a knee as 4 or 5
> conditionals works as expected but as presented, the script takes
> about 6 minutes at the
> GRUNT> prompt after hitting enter; before any Hadoop execution.
>
> -Clay
>
>
> On Tue, 26 Jun 2012, Danfeng Li
We found the following simple logic will cause very long compiling time for pig
0.10.0, while using pig 0.8.1, everything is fine.
A = load 'A.txt' using PigStorage() AS (m: int);
B = FOREACH A {
days_str = (chararray)
(m == 1 ? 31:
(m == 2 ? 28:
(m == 3 ? 31:
18, 2012 at 2:51 PM, Danfeng Li wrote:
> This is the jstack output during the setup time, not exactly sure how to
> interoperate it.
>
> Thanks.
> Dan
>
> [dli@hmaster run]$ jstack 15640
> 2012-06-18 17:32:47
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b17
oy [mailto:dvrya...@gmail.com]
Sent: Saturday, June 16, 2012 8:24 AM
To: user@pig.apache.org
Subject: Re: Job setup for a pig run takes ages
What loader are you using? Jt is not the place to look at, try jstacking your
pig process. Most likely it's talking to the NamaNode most of the time becaus
We also run into the long setup time issue, but our problem is different
1. The setup time takes about 20minutes, we can't see anything on the
jobtracker during this setup time.
2. Our data is saved in flat file, uncompressed.
3. Our code consists of many small pig files, they are used in the fol
I try to follow the example on
http://pig.apache.org/docs/r0.8.1/udf.html#Python+UDFs for Python UDF, but I
got the following error
* My command:
pig -x local ptest.pig
* error message
2011-10-06 15:15:19,995 [main] INFO org.apache.pig.Main - Logging error
messages to: /home/dli/pigExamples/P
25 matches
Mail list logo
