Re: help : error run pig
I thought it was good to go. Hey, have you tried maybe just doing a simple load test? I mean just loading a file into grunt with the LOAD command, and then doing a DUMP on it. So after that, we could see if there is actually something wrong with your installation. Renato M. 2010/9/27 Ngô Văn Vĩ > "192.168.230.130" is IP of my machine > @JeffZhang: can you explain clearly? > Thanks > > On Tue, Sep 28, 2010 at 8:39 AM, Jeff Zhang wrote: > > > It seems you have connected to the right hadoop when you start pig > > grunt. But connect to the wrong hadoop when you run pig script. > > Try to search whether there's other configuration files that mess up > > with your default configuration. And what is machine "192.168.230.130" > > ? > > > > > > On Tue, Sep 28, 2010 at 9:23 AM, Ngô Văn Vĩ > > wrote: > > > have you help me? > > > i have configuration > > > *- bin/pig* > > > export JAVA_HOME=/home/ngovi/jdk1.6.0_21 > > > export PIG_INSTALL=/home/ngovi/pig-0.7.0 > > > export PATH=$PATH:$PIG_INSTALL/bin > > > export PIG_HADOOP_VERSION=0.20.2 > > > export PIG_CLASSPATH=/home/ngovi/hadoop-0.20.2/conf/ > > > > > > *- conf/pig.properties* > > > fs.default.name=hdfs://localhost:9000/ > > > mapred.job.tracker=localhost:9001 > > > # log4jconf log4j configuration file > > > i run pig that have error > > > > > > *- in hadoop-0.20.2/conf* > > > *core-site.xml* > > > > > > > > > fs.default.name > > > hdfs://localhost:9000 > > > > > > the name of the default file system > > > > > > > > > > > > *hdfs-site.xml* > > > > > > > > > dfs.replication > > > 1 > > > Default block replication > > > > > > > > > > > > *mapred-site.xml* > > > > > > > > > > > > mapred.job.tracker > > > localhost:9001 > > > > > > the host and port that the mapreduce job tracker run at > > > > > > > > > > > > > > > I run pig that have error?? > > > *ng...@master:~/pig-0.7.0$ bin/pig -x mapreduce > > > 10/09/27 18:16:29 INFO pig.Main: Logging error messages to: > > > /home/ngovi/pig-0.7.0/pig_1285636589590.log > > > 2010-09-27 18:16:30,029 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > > Connecting > > > to hadoop file system at: hdfs://localhost:9000/ > > > 2010-09-27 18:16:30,347 [main] INFO > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > > Connecting > > > to map-reduce job tracker at: localhost:9001 > > > grunt> * > > > > > > > > > thanks all > > > > > > On Mon, Sep 27, 2010 at 1:14 PM, Alan Gates > wrote: > > > > > >> Pig is failing to connect to your namenode. Is the address Pig is > > trying > > >> to use (hdfs://master:54310/) correct? Can you connect using that > > string > > >> from the same machine using bin/hadoop? > > >> > > >> Alan. > > >> > > >> > > >> On Sep 27, 2010, at 8:45 AM, Ngô Văn Vĩ wrote: > > >> > > >> I run Pig at Hadoop Mode > > >>> (Pig-0.7.0 and hadoop-0.20.2) > > >>> have error? > > >>> ng...@master:~/pig-0.7.0$ bin/pig > > >>> 10/09/27 08:39:40 INFO pig.Main: Logging error messages to: > > >>> /home/ngovi/pig-0.7.0/pig_1285601980268.log > > >>> 2010-09-27 08:39:40,538 [main] INFO > > >>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > > >>> Connecting > > >>> to hadoop file system at: hdfs://master:54310/ > > >>> 2010-09-27 08:39:41,760 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 0 > > time(s). > > >>> 2010-09-27 08:39:42,762 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 1 > > time(s). > > >>> 2010-09-27 08:39:43,763 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 2 > > time(s). > > >>> 2010-09-27 08:39:44,765 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 3 > > time(s). > > >>> 2010-09-27 08:39:45,766 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 4 > > time(s). > > >>> 2010-09-27 08:39:46,767 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 5 > > time(s). > > >>> 2010-09-27 08:39:47,768 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 6 > > time(s). > > >>> 2010-09-27 08:39:48,769 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 7 > > time(s). > > >>> 2010-09-27 08:39:49,770 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.130:54310. Already tried 8 > > time(s). > > >>> 2010-09-27 08:39:50,771 [main] INFO org.apache.hadoop.ipc.Client - > > >>> Retrying > > >>> connect to server: master/192.168.230.
Re: Pig optimizer
Hi Daniel, Yeah that is, but like there are two types of optimizations right? I mean physical and logical optimizations. The physical ones are the ones on how the operators are distributed along mapreduce jobs and the logical ones are the ones similar to relational algebra right? Do you have any tips on how to get a quick grasp on pig logical optimizations? Thanks again. Renato M. 2010/8/26 Daniel Dai > Hi, Renato, > I think you are talking about how we organize different operators into > map-reduce jobs. Unfortunately there is no document currently. Basically we > will put as much operators into one map-reduce job as possible. > Co-group/Group, Join, Order, Distinct, Cross, Stream will create a > map-reduce boundary; Most others we will put into existing jobs. The main > logic is inside MRCompiler.java. > > > Daniel > > Renato Marroquín Mogrovejo wrote: > >> Anyone, please? >> >> Renato M. >> >> 2010/8/24 Renato Marroquín Mogrovejo >> >> >> >>> Hi Daniel, >>> >>> Thanks, but that was not what I was actually looking. What I want to know >>> is for example, how the optimizer work when the bags' logical plans are >>> combined, or if all commands are reduced at the end to CO-GROUP commands, >>> how is this handled? I know from Pig's paper that the ORDER, and LOAD, >>> commands generate new MapReduce jobs, are there any optimizations for the >>> physical plans? >>> Thanks in advanced. >>> >>> >>> Renato M. >>> >>> 2010/8/23 Daniel Dai >>> >>> Hi, Renato, >>> >>> >>>> There is a description of optimization rule in Pig Latin reference menu: >>>> >>>> http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules >>>> . >>>> Is that enough? >>>> >>>> Daniel >>>> >>>> >>>> Renato Marroquín Mogrovejo wrote: >>>> >>>> >>>> >>>>> Hey everyone, I was wondering if anybody has any references or >>>>> suggestion >>>>> on >>>>> how to learn about Pig's optimizer besides the source code or Pig's >>>>> paper. >>>>> Thanks in advance. >>>>> >>>>> >>>>> Renato M. >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >
Re: Pig optimizer
Anyone, please? Renato M. 2010/8/24 Renato Marroquín Mogrovejo > Hi Daniel, > > Thanks, but that was not what I was actually looking. What I want to know > is for example, how the optimizer work when the bags' logical plans are > combined, or if all commands are reduced at the end to CO-GROUP commands, > how is this handled? I know from Pig's paper that the ORDER, and LOAD, > commands generate new MapReduce jobs, are there any optimizations for the > physical plans? > Thanks in advanced. > > > Renato M. > > 2010/8/23 Daniel Dai > > Hi, Renato, >> There is a description of optimization rule in Pig Latin reference menu: >> http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules. >> Is that enough? >> >> Daniel >> >> >> Renato Marroquín Mogrovejo wrote: >> >>> Hey everyone, I was wondering if anybody has any references or suggestion >>> on >>> how to learn about Pig's optimizer besides the source code or Pig's >>> paper. >>> Thanks in advance. >>> >>> >>> Renato M. >>> >>> >> >> >
Re: Pig optimizer
Hi Daniel, Thanks, but that was not what I was actually looking. What I want to know is for example, how the optimizer work when the bags' logical plans are combined, or if all commands are reduced at the end to CO-GROUP commands, how is this handled? I know from Pig's paper that the ORDER, and LOAD, commands generate new MapReduce jobs, are there any optimizations for the physical plans? Thanks in advanced. Renato M. 2010/8/23 Daniel Dai > Hi, Renato, > There is a description of optimization rule in Pig Latin reference menu: > http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules. > Is that enough? > > Daniel > > > Renato Marroquín Mogrovejo wrote: > >> Hey everyone, I was wondering if anybody has any references or suggestion >> on >> how to learn about Pig's optimizer besides the source code or Pig's paper. >> Thanks in advance. >> >> >> Renato M. >> >> > >
Pig optimizer
Hey everyone, I was wondering if anybody has any references or suggestion on how to learn about Pig's optimizer besides the source code or Pig's paper. Thanks in advance. Renato M.
Re: Bug in new logical optimizer framework?
Thanks Daniel and Dmitriy for your answers, now I have a much clear idea of what type of optimization work is being done on PIG. And just another quick question, do you guys know where in the code those optimizations are? I just want to give it a closer look (: Renato M. 2010/7/1 Dmitriy Ryaboy > Renato, > I just want to make sure folks know -- Pig already has a number of such > optimizations. Daniel's work is aimed at making it (much) easier to write > such rules and to add a couple new ones. But some of the classic > optimizations like projection and filter push-down already exist in the > released versions of Pig. > > D > > On Thu, Jul 1, 2010 at 5:32 PM, Daniel Dai wrote: > > > Yes, they are classic logical optimization plus some Pig only > optimization. > > All these are rule based. > https://issues.apache.org/jira/browse/PIG-1319is an umbrella Jira to track > all new optimization rules. > > > > Daniel > > > > > > Renato Marroquín Mogrovejo wrote: > > > >> Hi, > >> > >> I am also interested in this logical plan optimization framework > >> functionality. You mentioned that rules are being developed, could you > >> explain a little bit more about them? Are they like the classic logical > >> optimizations (early projection, early filtering, among others)? > >> Thanks in advance. > >> > >> Renato M. > >> > >> logical plan optimization framework > >> > >> 2010/6/28 Alan Gates > >> > >> > >> > >>> On Jun 28, 2010, at 12:36 AM, Swati Jain wrote: > >>> > >>> Thanks for the prompt reply. As you mentioned optimization is in its > >>> > >>> > >>>> developing stage, does it mean optimization framework is not complete > or > >>>> only rules are in developing stage? In addition to that, I would > really > >>>> appreciate if you could give a rough idea when the patch will be > >>>> available > >>>> and what functionality will it contain? > >>>> > >>>> At this point we believe the framework is complete and rules are > being > >>>> > >>>> > >>> developed. But the framework has never been used in user testing > >>> situations > >>> (alpha or beta testing) so there will be a whole round of bugs to fix > >>> once > >>> that testing is done. > >>> > >>> The current plan is to switch to this code as the actual optimizer with > >>> 0.8, which we hope to release late this year (no promises). > >>> > >>> Alan. > >>> > >>> > >>> > >> > > >
Re: Bug in new logical optimizer framework?
Hi, I am also interested in this logical plan optimization framework functionality. You mentioned that rules are being developed, could you explain a little bit more about them? Are they like the classic logical optimizations (early projection, early filtering, among others)? Thanks in advance. Renato M. logical plan optimization framework 2010/6/28 Alan Gates > > On Jun 28, 2010, at 12:36 AM, Swati Jain wrote: > > Thanks for the prompt reply. As you mentioned optimization is in its >> developing stage, does it mean optimization framework is not complete or >> only rules are in developing stage? In addition to that, I would really >> appreciate if you could give a rough idea when the patch will be available >> and what functionality will it contain? >> >> At this point we believe the framework is complete and rules are being > developed. But the framework has never been used in user testing situations > (alpha or beta testing) so there will be a whole round of bugs to fix once > that testing is done. > > The current plan is to switch to this code as the actual optimizer with > 0.8, which we hope to release late this year (no promises). > > Alan. >
About PigPen
Hi, does anybody know which the PigPen release is? I found two links. The first one is from the wiki and the second one is from the jira. http://issues.apache.org/jira/secure/attachment/12393772/org.apache.pig.pigpen_0.0.1.jar https://issues.apache.org/jira/secure/attachment/12400858/PigPen.tgz Thanks in advance. Renato M.
Re: error while compiling pig
Thanks Jeff, do I need to do this? I mean I downloaded it and put it on my jvm/lib/ext, then I run "ant" and it worked fine :) Hey do you know if we can run PIG in mapreduce mode inside eclipse? I haven't been able to even though I have my env variables set, and I configured the run properties in the IDE. Thanks again. Renato M. 2010/5/24 Jeff Zhang > After you run "ant jar", the jline 0.94 will be in build/ivy/lib/Pig > You don't need to download jline by yourself. > > > > On Mon, May 24, 2010 at 10:49 AM, Renato Marroquín Mogrovejo > wrote: > > Hi everyone, I fixed using this version of the JLine library ---> > > jline-0.9.94 > > I dunno why I downloaded a previous one, but I would like know which the > > actual used version is. > > Thanks again. > > > > Renato M. > > > > > > -- > Best Regards > > Jeff Zhang >
Re: error while compiling pig
Hi everyone, I fixed using this version of the JLine library ---> jline-0.9.94 I dunno why I downloaded a previous one, but I would like know which the actual used version is. Thanks again. Renato M.
error while compiling pig
Hi everyone, does anybody know which JLine version I should use? I tried to
use the version jline-0_9_5 and I get an error while I tried to compile it
using ant
I also tried using Eclipse but it says that my jline library doesn't have a
methos setDefaultPrompt.
Thanks in advance.
Renato M.
[echo] *** Building Main Sources ***
[echo] *** To compile with all warnings enabled, supply
-Dall.warnings=1 on command line ***
[echo] *** If all.warnings property is supplied,
compile-sources-all-warnings target will be executed ***
[echo] *** Else, compile-sources (which only warns about deprecations)
target will be executed ***
compile-sources:
[javac] Compiling 601 source files to
/home/rmogrovejo/workspace/pig-0.7.0/build/classes
[javac]
/home/rmogrovejo/workspace/pig-0.7.0/src/org/apache/pig/Main.java:397:
cannot find symbol
[javac] symbol : method setDefaultPrompt(java.lang.String)
[javac] location: class jline.ConsoleReader
[javac] reader.setDefaultPrompt("grunt> ");
[javac] ^
[javac]
/home/rmogrovejo/workspace/pig-0.7.0/src/org/apache/pig/tools/grunt/GruntParser.java:213:
cannot find symbol
[javac] symbol : method setDefaultPrompt(java.lang.String)
[javac] location: class jline.ConsoleReader
[javac] mConsoleReader.setDefaultPrompt("grunt> ");
[javac] ^
[javac]
/home/rmogrovejo/workspace/pig-0.7.0/src-gen/org/apache/pig/tools/pigscript/parser/PigScriptParserTokenManager.java:28:
cannot find symbol
[javac] symbol : method setDefaultPrompt(java.lang.String)
[javac] location: class jline.ConsoleReader
[javac] consoleReader.setDefaultPrompt(">> ");
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 3 errors
About Zebra
Hi out-there! Is there any other documentation like papers or articles about Zebra and / or its use? Thanks in advance. Renato M.
Code Repository
Hi, is there a PIG coding standard? or any type of documentation I could follow? Thanks. Renato M.
