On Feb 13, 2013, at 5:56 PM, Anthonius deBoer wrote: > Interesting! > But maybe it is not the manager that is the bottleneck, but which process is > the one that takes the inputs from the webpage (or in the API) and prepares > all the files that are produced by the workflows? > > I guess it is the MAIN server? > THAT is the one that is the bottleneck I think then... > > Are there any plans to make that process parallel?
Well, you can run as many web servers as you'd like, which should help to alleviate this problem. Whichever server happens to be doing the work will be slow, but the rest should be unaffected. > What I am facing, is that I have 100 FASTQ pairs or so, for a single > flowcell, I can start the analysis of that set from the UI, but it will just > crank through them and takes about 2-4 minutues for each pair to be > processed, so with 100 pairs or so, you are looking 3-4 hours of an hourglass > before control is given back to the user... Is this in the context of starting a workflow with hundreds of inputs? --nate > It would be ideal if we would farm out the parsing of the web input to the > cluster, so the creation of the histories and datasets is no longer the > bottleneck.... > > How do others deal with large sequencing projects on this list? > Are there any groups that routinely do 100-200 pairs of FASTQ files from > within galaxy? > > I would like to hear their experience... > > Thanks > > Thon > > On Feb 13, 2013, at 01:44 PM, Nate Coraor <n...@bx.psu.edu> wrote: > >> On Feb 13, 2013, at 4:12 PM, Anthonius deBoer wrote: >> >> > Would this problem go away if I only used ONE webserver in my proxy >> > setting and not use a balancer? >> > I may not need it and if it means I could not use the toolshed >> > effectively, it may be worth it not to use balancing.. >> >> No, because your manager/handlers will still need to be restarted to pick up >> the new tools. The only way you can have it work without restarting is to >> have everything in a single process. >> >> > What i really need is the manager being balanced, since submitting a >> > workflow with 100's of BAM files can take hours just to get started...ANy >> > change we can run multiple managers? >> >> I'm surprised the manager is the holdup here since all it does is assign a >> handler. Regardless, in the development branch (which will be the next >> stable/release branch), the manager has been killed, and handlers are >> assigned directly by the web processes. >> >> --nate >> >> > >> > Thanks >> > >> > On Feb 13, 2013, at 07:59 AM, Nate Coraor <n...@bx.psu.edu> wrote: >> > >> >> On Feb 7, 2013, at 2:43 PM, Anthonius deBoer wrote: >> >> >> >> > That's very unfortunate... >> >> > I have a ton of tools and I guess now I have to create a package for >> >> > them in a local toolshed to update them in a running galaxy server? >> >> > >> >> > In any case...The toolshed installation also does not work for me...I >> >> > still have to restart galaxy, even after using the toolshed approach to >> >> > install a tool...It either does not show up at all or give a bunch of >> >> > errors, about not being able to find the tool... >> >> > >> >> > Is this also related to the fact I have two webservers and am behind a >> >> > proxy server as well? >> >> >> >> Hi Thon, >> >> >> >> Essentially yes, there is no way for one web process to communicate with >> >> others that it has installed a tool. We'd like to allow for this sort of >> >> notification via a message queue, but we don't have a proper message >> >> queue in Galaxy right now. >> >> >> >> --nate >> >> >> >> > >> >> > Thon >> >> > >> >> > On Feb 07, 2013, at 05:29 AM, Dannon Baker <dannonba...@me.com> wrote: >> >> > >> >> >> Unfortunately not, and with the migration of tools to the toolshed >> >> >> installation mechanism I don't imagine this will be addressed (at >> >> >> least by the team) anytime soon. If you wanted you could probably >> >> >> write a script that would reload a specified tool in each of the >> >> >> separate web processes, or just implement a complete rolling restart >> >> >> of your web processes to avoid service disruption while still loading >> >> >> the tool updates. >> >> >> >> >> >> -Dannon >> >> >> >> >> >> >> >> >> On Feb 6, 2013, at 8:40 PM, Anthonius deBoer <thondeb...@me.com> wrote: >> >> >> >> >> >> > I am indeed using multiple web processes and I guess I am talking >> >> >> > about the "old" admine tool reloader... >> >> >> > Is there any other way to do this for your own tools that you just >> >> >> > manually place in tools etc.? >> >> >> > >> >> >> > Thon >> >> >> > >> >> >> > On Feb 05, 2013, at 06:22 PM, Dannon Baker <dannonba...@me.com> >> >> >> > wrote: >> >> >> > >> >> >> >> Are you using multiple web processes, and are you referring to the >> >> >> >> old admin tool reloader or the toolshed reloading interface? >> >> >> >> >> >> >> >> -Dannon >> >> >> >> >> >> >> >> On Feb 5, 2013, at 9:13 PM, Anthonius deBoer <thondeb...@me.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >> > Hi, >> >> >> >> > >> >> >> >> > I find that reloading a tool's configuration file does not really >> >> >> >> > work. >> >> >> >> > First, you have to click the reload buttow twice to actually have >> >> >> >> > it update the VERSION number (so it does read something)... >> >> >> >> > But when I try to run my tool, the old bug is still there... >> >> >> >> > >> >> >> >> > I am using proxy server so something may still be cached, but I >> >> >> >> > have to restart my server for it actually to pick up the >> >> >> >> > changes... >> >> >> >> > >> >> >> >> > Any ideas? >> >> >> >> > >> >> >> >> > Thon >> >> >> >> > ___________________________________________________________ >> >> >> >> > Please keep all replies on the list by using "reply all" >> >> >> >> > in your mail client. To manage your subscriptions to this >> >> >> >> > and other Galaxy lists, please use the interface at: >> >> >> >> > >> >> >> >> > http://lists.bx.psu.edu/ >> >> >> >> >> >> >> >> >> > ___________________________________________________________ >> >> > Please keep all replies on the list by using "reply all" >> >> > in your mail client. To manage your subscriptions to this >> >> > and other Galaxy lists, please use the interface at: >> >> > >> >> > http://lists.bx.psu.edu/ >> >> >> ___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
