Thanks Alan, this was helpful.
On Thu, Jan 24, 2013 at 9:46 AM, Alan Gates wrote:
> You might want to look at webhcat's code. It produces a servlet that it
> embeds in a jetty server. You may be able to copy paste this to get what
> you want.
>
> The code of interest is in the hcat repository
Thank you for the suggestions. I will file a jira and add our discussion
there.
On Fri, Jan 25, 2013 at 4:23 PM, Rohini Palaniswamy wrote:
> Jon,
> Those are good areas to check. Few things I have seen regarding those are
>
> 1) JythonScriptEngine -PythonInterpreter is static and is not suit
Jon,
Those are good areas to check. Few things I have seen regarding those are
1) JythonScriptEngine -PythonInterpreter is static and is not suitable for
multiple runs if the script names are same (hit this issue in PIG-2433 unit
tests).
2) QueryParserDriver - There is a static cache with macr
user to bcc, +dev
Cheolsoo,
Can you make a JIRA for this? I can imagine a slightly heavier test suite,
but I like where you started. If it's not far off, then I think it'll be a
win to make it thread safe. But we need to make sure to test the most
advanced features...UDF's (esp the same name but
>> if you have multiple threads that run a query via PigServer, there is a
great chance of the internals clashing because of the use of static
variable within Pig.
Recently, I spent some time on this, and what I found is that the Pig
front-end is quite thread-safe. Here is how I tested it:
1) Wro
That clarifies it for me, thanks a lot.
Regards,
Rama.
On Fri, Jan 25, 2013 at 10:09 AM, Jonathan Coveney wrote:
> Well, when I say that Pig is not multi-threaded, what I mean is that if you
> have multiple threads that run a query via PigServer, there is a great
> chance of the internals clash
Well, when I say that Pig is not multi-threaded, what I mean is that if you
have multiple threads that run a query via PigServer, there is a great
chance of the internals clashing because of the use of static variables
within Pig. Pig itself, when running a single query, is multi-threaded.
It's jus
Hi Jonathan,
Pardon if it's a naive question, but Interesting that you say Pig is not
multithreaded.
We're using Pig 0.10.0, and looking at the code, it seems to do the right
things to handle multi threaded requests (ThreadLocal for ScriptState for
eg).
Would be great if you can point out to the
You might want to look at webhcat's code. It produces a servlet that it embeds
in a jetty server. You may be able to copy paste this to get what you want.
The code of interest is in the hcat repository under webhcat/svr.
Alan.
On Jan 24, 2013, at 9:42 AM, Prashant Kommireddi wrote:
> Thanks
Thanks Alan. We are trying to plug Pig into our existing app server.
We have already done this for Java MR. The difficulty we are facing is
with the fact that we can use JobClient.submitJob and jobtracker's job
end notification to run jobs async, whereas PigServer.executeBatch
blocks until pig job
If you're looking for an app server for Pig I'd take a look at a couple of
other projects already out there that can do this:
1) webhcat (fka Templeton, now part of the HCatalog project). It provides a
REST API that launches Pig, Hive, or MR jobs and allows you to manage them, get
results, etc
Are there any plans on making the pigserver multi-threaded?
since there is "PigProcessNotificationListener" to subscribe for async
callbacks when the pig job completes, is there any real need to keep the
pig job submitting thread waiting until the job completes?
Is this just a shortcoming today o
I think whatever way you slice it, handling thousands of pig jobs
asynchronously is going to be a bear. I mean, this is essentially what the
job tracker does, albeit with a lot less information.
Either way, Pig is not multi-threaded so having more than one instance of
Pig in the same JVM is going
Both. Think of it as an app server handling all of these requests.
Sent from my iPhone
On Jan 23, 2013, at 9:09 PM, Jonathan Coveney wrote:
> Thousands of requests, or thousands of Pig jobs? Or both?
>
>
> 2013/1/23 Prashant Kommireddi
>
>> Did not want to have several threads launched for thi
Thousands of requests, or thousands of Pig jobs? Or both?
2013/1/23 Prashant Kommireddi
> Did not want to have several threads launched for this. We might have
> thousands of requests coming in, and the app is doing a lot more than only
> Pig.
>
> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coven
Did not want to have several threads launched for this. We might have
thousands of requests coming in, and the app is doing a lot more than only
Pig.
On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney wrote:
> start a separate Process which runs Pig?
>
>
> 2013/1/23 Prashant Kommireddi
>
> > Hey
start a separate Process which runs Pig?
2013/1/23 Prashant Kommireddi
> Hey guys,
>
> I am trying to do the following:
>
>1. Launch a pig job asynchronously via Java program
>2. Get a notification once the job is complete (something similar to
>Hadoop callback with a servlet)
>
> I
You can create in instance of PigProcessNotificationListener that calls
back when the job finishes.
On Wed, Jan 23, 2013 at 4:48 PM, Prashant Kommireddi wrote:
> Hey guys,
>
> I am trying to do the following:
>
>1. Launch a pig job asynchronously via Java program
>2. Get a notification o
Hey guys,
I am trying to do the following:
1. Launch a pig job asynchronously via Java program
2. Get a notification once the job is complete (something similar to
Hadoop callback with a servlet)
I looked at PigServer.executeBatch() and it seems to be waiting until job
completes.This is
19 matches
Mail list logo
