Well, when I say that Pig is not multi-threaded, what I mean is that if you have multiple threads that run a query via PigServer, there is a great chance of the internals clashing because of the use of static variables within Pig. Pig itself, when running a single query, is multi-threaded. It's just not "multi-threaded" in the sense that multiple instances can safely be run in the same JVM.
2013/1/24 Ramakrishna Nalam <nramakris...@gmail.com> > Hi Jonathan, > > Pardon if it's a naive question, but Interesting that you say Pig is not > multithreaded. > We're using Pig 0.10.0, and looking at the code, it seems to do the right > things to handle multi threaded requests (ThreadLocal for ScriptState for > eg). > > Would be great if you can point out to the kind of issues there could be. > > > Regards, > Rama. > > > > On Thu, Jan 24, 2013 at 8:32 PM, Praveen M <lefthandma...@gmail.com> > wrote: > > > Are there any plans on making the pigserver multi-threaded? > > > > since there is "PigProcessNotificationListener" to subscribe for async > > callbacks when the pig job completes, is there any real need to keep the > > pig job submitting thread waiting until the job completes? > > > > Is this just a shortcoming today or are there more concrete reasons > against > > providing with a pigserver which can submit to the cluster in mapreduce > > mode async? > > > > Thanks, > > Praveen > > > > > > > > On Wed, Jan 23, 2013 at 10:56 PM, Jonathan Coveney <jcove...@gmail.com > > >wrote: > > > > > I think whatever way you slice it, handling thousands of pig jobs > > > asynchronously is going to be a bear. I mean, this is essentially what > > the > > > job tracker does, albeit with a lot less information. > > > > > > Either way, Pig is not multi-threaded so having more than one instance > of > > > Pig in the same JVM is going to start causing problems (which is why, I > > > imagine, there is no async way to call Pig). So multiple processes is > > > really the only way around it that I know of. > > > > > > At Twitter we have a deployment of mesos, and our long term solution is > > > going to be running all of our pig jobs on mesos, in the short term by > > > deploying daemons that run pig jobs as local processes. > > > > > > > > > 2013/1/23 Prashant Kommireddi <prash1...@gmail.com> > > > > > > > Both. Think of it as an app server handling all of these requests. > > > > > > > > Sent from my iPhone > > > > > > > > On Jan 23, 2013, at 9:09 PM, Jonathan Coveney <jcove...@gmail.com> > > > wrote: > > > > > > > > > Thousands of requests, or thousands of Pig jobs? Or both? > > > > > > > > > > > > > > > 2013/1/23 Prashant Kommireddi <prash1...@gmail.com> > > > > > > > > > >> Did not want to have several threads launched for this. We might > > have > > > > >> thousands of requests coming in, and the app is doing a lot more > > than > > > > only > > > > >> Pig. > > > > >> > > > > >> On Wed, Jan 23, 2013 at 5:44 PM, Jonathan Coveney < > > jcove...@gmail.com > > > > >>> wrote: > > > > >> > > > > >>> start a separate Process which runs Pig? > > > > >>> > > > > >>> > > > > >>> 2013/1/23 Prashant Kommireddi <prash1...@gmail.com> > > > > >>> > > > > >>>> Hey guys, > > > > >>>> > > > > >>>> I am trying to do the following: > > > > >>>> > > > > >>>> 1. Launch a pig job asynchronously via Java program > > > > >>>> 2. Get a notification once the job is complete (something > > similar > > > to > > > > >>>> Hadoop callback with a servlet) > > > > >>>> > > > > >>>> I looked at PigServer.executeBatch() and it seems to be waiting > > > until > > > > >> job > > > > >>>> completes.This is not what I would like my app to do. > > > > >>>> > > > > >>>> Any ideas? > > > > >>>> > > > > >>>> Thanks, > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > -- > > -Praveen > > >
