Re: Embedded pig in java
thanks for your help :)
Le Vendredi 4 avril 2014 2h36, Jay Vyas jayunit...@gmail.com a écrit :
You can run the maven profile on bigpetstore as an example as well:
(see PigCSVCleaner.java):
https://github.com/jayunit100/bigpetstore/
And for a visual demo you can watch how we use maven to run local pig tasks
w/ PigServer in maven (somewhere in the middle of the video) here:
https://www.youtube.com/watch?v=OVB3nEKN94k
On Thu, Apr 3, 2014 at 7:23 AM, David LaBarbera
davidlabarb...@localresponse.com wrote:
Try running with
java -cp pathpig.jar idlocal.Idlocal
David
On Apr 3, 2014, at 7:54 AM, Junior Tsire jajao2...@yahoo.fr wrote:
Hi everybody,
I have been following a tutorial on how to embed pig in java because I
am currently learning it. I used the sample code that I found on
http://pig.apache.org/docs/r0.11.0/cont.html to teach myself and here is
the code that I took and paste it on netbeans
package idlocal;
import java.io.IOException;
import org.apache.pig.PigServer;
public class Idlocal{
public static void main(String[] args) {
try {
PigServer pigServer = new PigServer(local);
runIdQuery(pigServer, passwd);
}
catch(Exception e) {
}
}
public static void runIdQuery(PigServer pigServer, String inputFile)
throws IOException {
pigServer.registerQuery(A = load ' + inputFile + ' using
PigStorage(':'););
pigServer.registerQuery(B = foreach A generate $0 as id;);
pigServer.store(B, id.out);
}
}
I managed to compile it on cygwin by doing javac -cp pathpig.jar
idlocal.java. It successfully created the class file. Now when I tried to
run it (on cygwin) by doing java -cp pathpig.jar idlocal, it gave me the
this Error: Could not find or load main class Idlocal.
I really need some help because I don't know what is going on :(
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Embedded pig in java
Try running with
java -cp pathpig.jar idlocal.Idlocal
David
On Apr 3, 2014, at 7:54 AM, Junior Tsire jajao2...@yahoo.fr wrote:
Hi everybody,
I have been following a tutorial on how to embed pig in java because I am
currently learning it. I used the sample code that I found on
http://pig.apache.org/docs/r0.11.0/cont.html to teach myself and here is the
code that I took and paste it on netbeans
package idlocal;
import java.io.IOException;
import org.apache.pig.PigServer;
public class Idlocal{
public static void main(String[] args) {
try {
PigServer pigServer = new PigServer(local);
runIdQuery(pigServer, passwd);
}
catch(Exception e) {
}
}
public static void runIdQuery(PigServer pigServer, String inputFile) throws
IOException {
pigServer.registerQuery(A = load ' + inputFile + ' using
PigStorage(':'););
pigServer.registerQuery(B = foreach A generate $0 as id;);
pigServer.store(B, id.out);
}
}
I managed to compile it on cygwin by doing javac -cp pathpig.jar
idlocal.java. It successfully created the class file. Now when I tried to run
it (on cygwin) by doing java -cp pathpig.jar idlocal, it gave me the this
Error: Could not find or load main class Idlocal.
I really need some help because I don't know what is going on :(
Re: Embedded pig in java
You can run the maven profile on bigpetstore as an example as well:
(see PigCSVCleaner.java):
https://github.com/jayunit100/bigpetstore/
And for a visual demo you can watch how we use maven to run local pig tasks
w/ PigServer in maven (somewhere in the middle of the video) here:
https://www.youtube.com/watch?v=OVB3nEKN94k
On Thu, Apr 3, 2014 at 7:23 AM, David LaBarbera
davidlabarb...@localresponse.com wrote:
Try running with
java -cp pathpig.jar idlocal.Idlocal
David
On Apr 3, 2014, at 7:54 AM, Junior Tsire jajao2...@yahoo.fr wrote:
Hi everybody,
I have been following a tutorial on how to embed pig in java because I
am currently learning it. I used the sample code that I found on
http://pig.apache.org/docs/r0.11.0/cont.html to teach myself and here is
the code that I took and paste it on netbeans
package idlocal;
import java.io.IOException;
import org.apache.pig.PigServer;
public class Idlocal{
public static void main(String[] args) {
try {
PigServer pigServer = new PigServer(local);
runIdQuery(pigServer, passwd);
}
catch(Exception e) {
}
}
public static void runIdQuery(PigServer pigServer, String inputFile)
throws IOException {
pigServer.registerQuery(A = load ' + inputFile + ' using
PigStorage(':'););
pigServer.registerQuery(B = foreach A generate $0 as id;);
pigServer.store(B, id.out);
}
}
I managed to compile it on cygwin by doing javac -cp pathpig.jar
idlocal.java. It successfully created the class file. Now when I tried to
run it (on cygwin) by doing java -cp pathpig.jar idlocal, it gave me the
this Error: Could not find or load main class Idlocal.
I really need some help because I don't know what is going on :(
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Embedded Pig with Java
Thanks Daniel. I did not see an exception either. Is log generation
with PigServer something that we could add in Pig 0.9.1? Debugging
would be a lot easier while converting existing if scripts to java and
otherwise.
Sent from my iPhone
On Dec 5, 2011, at 6:43 PM, Daniel Dai da...@hortonworks.com wrote:
There is no log if you embed PigServer in code, however, you will see
exception.
Daniel
On Mon, Dec 5, 2011 at 2:58 PM, Prashant Kommireddi
prash1...@gmail.comwrote:
FYI, I am aware the error is that I am using registerQuery and not
registerJar. Would just like to know where the error logs can be found.
Thanks,
Prashant Kommireddi
On Mon, Dec 5, 2011 at 2:21 PM, Prashant Kommireddi prash1...@gmail.com
wrote:
I am embedding Pig Latin in Java, and want to check the error logs. Where
can I find them?
The program below runs fine if I don't use the line
*pigServer.registerQuery(REGISTER
' +lib+ ';);
But fails when I use this line and I can't find the logs in the directory
I ran this from.
*
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.pig.PigServer;
public class idmapreduce {
public static void main(String[] args) {
try {
PigServer pigServer = new PigServer(mapreduce);
runIdQuery(pigServer, passwd,
/home/pkommireddi/dev/apps/gridforce/main/pigassist/dist/pigassist.jar);
} catch (Exception e) {}
}
public static void runIdQuery(PigServer pigServer, String inputFile,
String lib) throws IOException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path output = new Path(idout);
fs.delete(output, true);
*pigServer.registerQuery(REGISTER ' +lib+ ';);*
pigServer.registerQuery(A = load ' + inputFile + ' using
PigStorage(':'););
pigServer.registerQuery(B = foreach A generate $0 as id;);
pigServer.store(B, idout);
}
}
Thanks,
Prashant Kommireddi
Re: Embedded Pig with Java
Daniel, I have filed a bug https://issues.apache.org/jira/browse/PIG-2401
-Prashant
On Mon, Dec 5, 2011 at 7:06 PM, Prashant Kommireddi prash1...@gmail.comwrote:
That's correct. I will file a bug with the details soon. Basically the
MR job never begins, it fails client-side without an exception
message.
-Prashant
On Dec 5, 2011, at 7:00 PM, Daniel Dai da...@hortonworks.com wrote:
What the log does is capture the exception, and dump to a file. In java
mode, you should capture exception in your code. You said you didn't see
an
exception, you mean the mapreduce job started but fail without exception?
That's certainly unexpected.
Daniel
On Mon, Dec 5, 2011 at 6:47 PM, Prashant Kommireddi prash1...@gmail.com
wrote:
Thanks Daniel. I did not see an exception either. Is log generation
with PigServer something that we could add in Pig 0.9.1? Debugging
would be a lot easier while converting existing if scripts to java and
otherwise.
Sent from my iPhone
On Dec 5, 2011, at 6:43 PM, Daniel Dai da...@hortonworks.com wrote:
There is no log if you embed PigServer in code, however, you will see
exception.
Daniel
On Mon, Dec 5, 2011 at 2:58 PM, Prashant Kommireddi
prash1...@gmail.com
wrote:
FYI, I am aware the error is that I am using registerQuery and not
registerJar. Would just like to know where the error logs can be
found.
Thanks,
Prashant Kommireddi
On Mon, Dec 5, 2011 at 2:21 PM, Prashant Kommireddi
prash1...@gmail.com
wrote:
I am embedding Pig Latin in Java, and want to check the error logs.
Where
can I find them?
The program below runs fine if I don't use the line
*pigServer.registerQuery(REGISTER
' +lib+ ';);
But fails when I use this line and I can't find the logs in the
directory
I ran this from.
*
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.pig.PigServer;
public class idmapreduce {
public static void main(String[] args) {
try {
PigServer pigServer = new PigServer(mapreduce);
runIdQuery(pigServer, passwd,
/home/pkommireddi/dev/apps/gridforce/main/pigassist/dist/pigassist.jar);
} catch (Exception e) {}
}
public static void runIdQuery(PigServer pigServer, String inputFile,
String lib) throws IOException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path output = new Path(idout);
fs.delete(output, true);
*pigServer.registerQuery(REGISTER ' +lib+ ';);*
pigServer.registerQuery(A = load ' + inputFile + ' using
PigStorage(':'););
pigServer.registerQuery(B = foreach A generate $0 as id;);
pigServer.store(B, idout);
}
}
Thanks,
Prashant Kommireddi
