Re: [galaxy-dev] Workflow param error leads to template error [patch]
Hi Dannon, Here is the full trace from the log: === 138.253.25.15 - - [29/Aug/2014:19:16:01 +0100] "POST /galaxy/workflow/run?id=f597429621d6eb2b HTTP/1.1" 500 - "http://138.253.25.15/galaxy/workflow/run?id=f597429621d6eb2b"; "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" Error - : 'WorkflowStep' object has no attribute 'module' URL: http://138.253.25.15/galaxy/workflow/run?id=f597429621d6eb2b File '/usr/lib/galaxy-server/lib/galaxy/web/framework/middleware/error.py', line 149 in __call__ app_iter = self.application(environ, sr_checker) File '/usr/lib/galaxy-server/eggs/Paste-1.7.5.1-py2.7.egg/paste/recursive.py', line 84 in __call__ return self.application(environ, start_response) File '/usr/lib/galaxy-server/lib/galaxy/web/framework/middleware/remoteuser.py', line 94 in __call__ return self.app( environ, start_response ) File '/usr/lib/galaxy-server/eggs/Paste-1.7.5.1-py2.7.egg/paste/httpexceptions.py', line 633 in __call__ return self.application(environ, start_response) File '/usr/lib/galaxy-server/lib/galaxy/web/framework/base.py', line 132 in __call__ return self.handle_request( environ, start_response ) File '/usr/lib/galaxy-server/lib/galaxy/web/framework/base.py', line 190 in handle_request body = method( trans, **kwargs ) File '/usr/lib/galaxy-server/lib/galaxy/webapps/galaxy/controllers/workflow.py', line 1410 in run enable_unique_defaults=trans.app.config.enable_unique_workflow_defaults) File '/usr/lib/galaxy-server/lib/galaxy/web/framework/__init__.py', line 1194 in fill_template return self.fill_template_mako( filename, **kwargs ) File '/usr/lib/galaxy-server/lib/galaxy/web/framework/__init__.py', line 1209 in fill_template_mako return template.render( **data ) File '/usr/lib/galaxy-server/eggs/Mako-0.4.1-py2.7.egg/mako/template.py', line 296 in render return runtime._render(self, self.callable_, args, data) File '/usr/lib/galaxy-server/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 660 in _render **_kwargs_for_callable(callable_, data)) File '/usr/lib/galaxy-server/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '/usr/lib/galaxy-server/eggs/Mako-0.4.1-py2.7.egg/mako/runtime.py', line 718 in _exec_template callable_(context, *args, **kwargs) File '/usr/lib/galaxy-server/database/compiled_templates/base.mako.py', line 66 in render_body __M_writer(unicode(next.body())) File '/usr/lib/galaxy-server/database/compiled_templates/workflow/run.mako.py', line 257 in render_body for ic in oc.input_step.module.get_data_inputs(): AttributeError: 'WorkflowStep' object has no attribute 'module' === Cheers, TIM On Thu, 2014-09-18 at 22:12 +0100, Dannon Baker wrote: > Hi Tim, > > > Thanks for this. If you still have it handy, what was the exact > attribute error? > > > -Dannon > > On Thu, Sep 18, 2014 at 10:28 AM, Tim Booth wrote: > Dear Devs, > > I recently ran into a problem where attempting to run a > workflow > resulted in an uninformative error and a traceback in the > logs, that I > put into Google. This led me to the message below from 2012. > As far as > I can see it got no reply at the time and I have hit the same > issue. > > I modified database/compiled_templates/workflow/run.mako.py > and this > resulted in a much more useful error report appearing in the > browser. > Obviously this is a generated file, so attached is a patch for > templates/webapps/galaxy/workflow/run.mako that I propose you > add to > Galaxy. As you'll see it just adds a single "try" block and > then > ignores the exception, which I would say in this case is not > actually a > bad thing to do even if it is not the ideal fix. Should be > fairly > explanatory from the patch. > > Cheers, > > TIM > > Way way back in Jun 2012, Paul-Michael.Agapow at hpa.org.uk > wrote: > > > Looking for pointers on what might be causing this problem: > > > > A user had a moderately complicated workflow that when run > under certain parameter values results in a traceback that > ends: > > > >Module workflow_run_mako:232 in render_body > >>> for ic in oc.input_step.module.get_data_inputs(): > >AttributeError: 'Wo
Re: [galaxy-dev] Workflow param error leads to template error [patch]
Dear Devs, I recently ran into a problem where attempting to run a workflow resulted in an uninformative error and a traceback in the logs, that I put into Google. This led me to the message below from 2012. As far as I can see it got no reply at the time and I have hit the same issue. I modified database/compiled_templates/workflow/run.mako.py and this resulted in a much more useful error report appearing in the browser. Obviously this is a generated file, so attached is a patch for templates/webapps/galaxy/workflow/run.mako that I propose you add to Galaxy. As you'll see it just adds a single "try" block and then ignores the exception, which I would say in this case is not actually a bad thing to do even if it is not the ideal fix. Should be fairly explanatory from the patch. Cheers, TIM Way way back in Jun 2012, Paul-Michael.Agapow at hpa.org.uk wrote: > Looking for pointers on what might be causing this problem: > > A user had a moderately complicated workflow that when run under certain > parameter values results in a traceback that ends: > >Module workflow_run_mako:232 in render_body >>> for ic in oc.input_step.module.get_data_inputs(): >AttributeError: 'WorkflowStep' object has no attribute 'module' > > After some puzzling, I realized one of the parameters was outside the allowed > range (a max on an integer param) and that Galaxy was trying to render > "run.mako" flagging up the error but erroring out. Why Galaxy tries to call > this non-existent member is unclear to me. Any insight or places I should > start exploring? > > Galaxy version: various (started with an 6-month old one, upgraded to latest > production to see if it would fix error, it didn't) > Hosting OS: CentOS 6 > > -- > Paul Agapow (paul-michael.agapow at hpa.org.uk) > Bioinformatics, Health Protection Agency (UK) > -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 # This patch addresses http://lists.bx.psu.edu/pipermail/galaxy-dev/2012-June/010303.html --- a/templates/webapps/galaxy/workflow/run.mako +++ b/templates/webapps/galaxy/workflow/run.mako @@ -582,9 +582,12 @@ # Filter possible inputs to data types that are valid for subsequent steps type_filter = [] for oc in step.output_connections: +try: for ic in oc.input_step.module.get_data_inputs(): if 'extensions' in ic and ic['name'] == oc.input_name: type_filter += ic['extensions'] +except AttributeError: + pass if not type_filter: type_filter = ['data'] %> ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi Nate, It's great to hear that my rearrangements for packaging Galaxy are somewhat in line with what you are doing. My package is a bit of a rats-nest of symlinks right now but I'll clean these up as things become more configurable. One thing I'm a bit stuck on is the tool_data_table_conf.xml and shed_tool_data_table_conf.xml files. If the former is installed and updated by my package and the latter is managed by the internal tool-shed logic then this leaves nowhere for the user to make manual edits (we can use the debian config-files system to share ownership but it's awkward). Also I've made a galaxy-tools-bl package that adds some standard tool definitions in a file called bl_tool_conf.xml. I can set things up so this is seen by Galaxy: [app:main] ... tool_config_file = tool_conf.xml,shed_tool_conf.xml,bl_tool_conf.xml So I can split out tool_conf.xml but currently I can't split out items in tool_data_table_conf.xml (unless I make a startup script that compiles it on the fly from XML fragments, in the style of build_universe_config.py). Now if it were me I'd try and eliminate this XML file altogether, and have Galaxy just collect up all the available .loc files on startup without the need for a central registry file. This would involve embedding the couple of bits of metadata from the tool_data_table_conf.xml XML file into the .loc files themselves, and maybe there is some reason I've not spotted why this would be a bad idea. Any thoughts on this? Is it something you have considered or would consider? Cheers, TIM On Fri, 2014-03-28 at 18:49 +, Nate Coraor wrote: > Hi Tim, > > > I have recently been working on getting Galaxy Main's configs and > server-modified files and directories out of the galaxy-dist > directory, so our goals are aligning. Not everything can be moved > without some trickery (e.g. symlinks) but most paths, including the > paths to shed_*.xml are configurable in universe_wsgi.ini (which > itself need not live in the galaxy-dist directory). > > > --nate > > > On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth wrote: > Hi, > > This is mainly a message to Carlos and Eric who volunteered to > get > involved with my Debian packaging, but of course any other > help will > also be appreciated. If either of you can spare time now it > is a good > moment to do so. > > I've been working on my package stuff over the last couple of > weeks and > have uploaded the latest to Debian-Med ready to build: > > > http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log > > Note that you need the re-packed tarball as generated by > get-orig-source.sh and if you have a problem generating that > you can > grab a copy of it here: > > > https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz > > I've only built this for Ubuntu, and I know that to get it > working on > Debian you'll at least need to replace the upstart job with > an /etc/init.d script. After that I think you should have > something > working (see my commit notes). > > My latest efforts have been to try and get tool-shed installs > working. > Galaxy expects to be able to write to its own > shed_tools_*_conf.xml > files as well as to shed_tools and the tool-data directory. > It looks > like there is work to have a separate shed_tool-data folder > but this is > not fully working so I'm seeing if I can patch it. Either > way, it's > vital for packaging that the files managed by DPKG and the > files > (over)written by the Galaxy server are separated out into /var > and /usr > respectively. > > Cheers, > > TIM > > On Mon, 2013-12-16 at 16:27 +, Carlos Borroto wrote: > > > Hi Tim, > > > > This sounds great. I'll be happy to help testing and > hopefully find > > some time to help packaging once it gets into Debian Med(are > you > > submitting all your packages there?). > > > > One question, for apache/nginx configuration why not use > something ala > > phpMyAdmin which ask you if you want to preconfigure the > package with > > a webserver in particular. The name of the DEB packagin
Re: [galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi, This is mainly a message to Carlos and Eric who volunteered to get involved with my Debian packaging, but of course any other help will also be appreciated. If either of you can spare time now it is a good moment to do so. I've been working on my package stuff over the last couple of weeks and have uploaded the latest to Debian-Med ready to build: http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log Note that you need the re-packed tarball as generated by get-orig-source.sh and if you have a problem generating that you can grab a copy of it here: https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz I've only built this for Ubuntu, and I know that to get it working on Debian you'll at least need to replace the upstart job with an /etc/init.d script. After that I think you should have something working (see my commit notes). My latest efforts have been to try and get tool-shed installs working. Galaxy expects to be able to write to its own shed_tools_*_conf.xml files as well as to shed_tools and the tool-data directory. It looks like there is work to have a separate shed_tool-data folder but this is not fully working so I'm seeing if I can patch it. Either way, it's vital for packaging that the files managed by DPKG and the files (over)written by the Galaxy server are separated out into /var and /usr respectively. Cheers, TIM On Mon, 2013-12-16 at 16:27 +, Carlos Borroto wrote: > Hi Tim, > > This sounds great. I'll be happy to help testing and hopefully find > some time to help packaging once it gets into Debian Med(are you > submitting all your packages there?). > > One question, for apache/nginx configuration why not use something ala > phpMyAdmin which ask you if you want to preconfigure the package with > a webserver in particular. The name of the DEB packaging technology to > ask these kind of questions is evading me now. I think using something > like that could open many possibilities in the future, like database > backend to use, home URL, admin user/password, etc... > > Thanks for your work on this, > Carlos > > > On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth wrote: > > Hi All, > > > > As previously mentioned, I'm back working on packaging the Galaxy server > > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately > > pushing towards something that could be Debian compliant. There's a way > > to go in that regard, but I do now have an updated package for Bio-Linux > > in final testing and it also has a new trick: doing "apt-get install > > galaxy-server-apache-proxy" will set up just that with no further > > configuration needed. The galaxy server appears at > > http://localhost/galaxy and users log in with their regular system > > username and password. Uploads are enabled via regular SFTP so no > > special FTP server configuration is needed. > > > > It's a little hacky in parts but I'm generally pleased with the result. > > If anyone want to take a look I'd welcome comments. It's not in the > > main BL repo yet but can be found here: > > > > https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra > > > > Cheers, > > > > TIM > > > > -- > > Tim Booth > > NERC Environmental Bioinformatics Centre > > > > Centre for Ecology and Hydrology > > Maclean Bldg, Benson Lane > > Crowmarsh Gifford > > Wallingford, England > > OX10 8BB > > > > http://nebc.nerc.ac.uk > > +44 1491 69 2705 > > > > _______ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Remote User Logout
Hi,
I'm currently using one of those hacks, and it seems to work nicely for
the user (Chrome + FF at least) but it does need some messy setting up
in Apache and some cunning redirects in place. I've pasted the relevant
file fragments below. It's somewhat confounded with my stuff to enable
SFTP uploads but hopefully you get the idea and the original explanation
on Stackoverflow is pretty good. The remote_user_logout_href is
something I got to by trial and error.
Cheers,
TIM
===
% cat /usr/share/galaxy-server/logout/.htaccess
# HaCk based on
http://stackoverflow.com/questions/4163122/http-basic-authentication-log-out
# Authname must match the one in ../proxy/.htaccess
AuthType Basic
AuthName Galaxy_Server
AuthUserFile /usr/share/galaxy-server/logout/.htpasswd
Require user logout
===
% cat /usr/share/galaxy-server/logout/.htpasswd
#Password is logout. This in not a secret.
logout:$apr1$0eB1iURY$kwqa0c8tXksbjPQLYqr6s.
===
% cat /usr/share/galaxy-server/proxy/.htaccess
# Security settings for Galaxy proxied via Apache. Note the actual
# proxy config is under /etc/apache2/conf.d/galaxy. If for some
# reason you wanted Apache proxy with internal Galaxy authentication
# then you could remove this file and Apache would no longer insist on
# authentication.
AuthBasicProvider external
AuthExternal pwauth
AuthType Basic
AuthName Galaxy_Server
#I'd like to do this, but it upsets Firefox. Use ErrorDocument instead.
# AuthName "Galaxy Server: \
# Log in with regular username and password. \
# Users need to be in the galaxy system group."
ErrorDocument 401 "\
401 Authorization Required\
Log-in to Galaxy failed\
You should have been prompted to log into the Galaxy server. \
You need to give your regular system username and password. \
Please reload this page to try again.\
If this fails, check that you are a member of the galaxy system
group, by \
running groups on the command line.\
To add a user, eg. user1, to this group, you may use the
command:\
sudo usermod -aG galaxy user1\
"
# You may want to comment these 2 lines out or to
# change the group required, but users still need to
# be in the galaxy group for SFTP uploads to work properly.
AuthzUnixgroup on
Require group galaxy
# This is needed to tell Galaxy about the remote
# user.
RequestHeader set REMOTE_USER %{RU}e env=RU
RequestHeader unset Authorization env=RU
===
% cat /etc/galaxy-server/universe_wsgi.d/31_apache-proxy.ini
# Settings added by debian-galaxy-apache-proxy to switch Galaxy over to
# authenticating by real user accounts and also permitting uploads.
[app:main]
# Other scripts assume that maildomain is localhsot, so you can't just
# change the setting below and expect everythig to work.
use_remote_user = True
remote_user_maildomain = localhost
# Users may copy files here directly or upload via SFTP/SCP
ftp_upload_dir = /var/lib/galaxy-server/transfer
ftp_upload_site = *** Transfer files via SCP or SFTP to
/var/lib/galaxy-server/transfer/... ***
# There is no neat way to log out a user with Basic Auth, but here is a
non-neat way.
# Not yet tested on IE.
remote_user_logout_href = javascript:var r=new
XMLHttpRequest();r.onreadystatechange=function(){if(r.readyState==4)window.location.replace('logout.html')};r.open('get','logout.html',true,'logout','logout');r.send();
===
--
Tim Booth
NERC Environmental Bioinformatics Centre
Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB
http://nebc.nerc.ac.uk
+44 1491 69 2705
___
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi John and Carlos, Thanks for your replies regarding my DEB packaging work. Some quick answers to questions you raised. > > In response to the eggs comment in the changelog - there are upsides > and downsides to the way Galaxy handles dependency fetching but I > think there should at least be an off switch. I thought that would be > easy, but it sounds like not: > ... > I have created a Trello card for this. > https://trello.com/c/lthRjVZq > Of course the ultimate goal for Debian compliance is that Galaxy uses 100% system Python libs and DPKG looks after all the interdependencies. I'm not sure how realistic that is, but I'm working on it. I'll need good regression tests for sure. On Mon, 2013-12-16 at 16:27 +, Carlos Borroto wrote: > Hi Tim, > > This sounds great. I'll be happy to help testing and hopefully find > some time to help packaging once it gets into Debian Med(are you > submitting all your packages there?). Yes, it will all go in the Debian-Med SVN but for it to actually work on Debian I'll need a non-Upstart init script and probably some other changes. Maybe you can help me with that? I'll ping you in the New Year. > One question, for apache/nginx configuration why not use something ala > phpMyAdmin which ask you if you want to preconfigure the package with > a webserver in particular. The name of the DEB packaging technology to > ask these kind of questions is evading me now. I think using something > like that could open many possibilities in the future, like database > backend to use, home URL, admin user/password, etc... I think you mean Debconf, which I'll probably use for a couple of things like the initial list of admin users but which I try to limit. If I put the packages out on the live DVD/USB or the Cloud image then I have to configure them with default settings in any case, and then it's easier for the user to just edit the config files than to remember a command like: % sudo dpkg-reconfigure -p medium galaxy-server-apache-proxy As a case in point, the apache2 package itself no longer uses debconf at all. I'm not currently planning to add Nginx to this packaging, as the priority was to enable uploads via SFTP, and this required use of system accounts in Galaxy, which called for authentication via PAM, which needed the relevant Apache2 modules. Cheers, TIM > Thanks for your work on this, > Carlos > > > On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth wrote: > > Hi All, > > > > As previously mentioned, I'm back working on packaging the Galaxy server > > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately > > pushing towards something that could be Debian compliant. There's a way > > to go in that regard, but I do now have an updated package for Bio-Linux > > in final testing and it also has a new trick: doing "apt-get install > > galaxy-server-apache-proxy" will set up just that with no further > > configuration needed. The galaxy server appears at > > http://localhost/galaxy and users log in with their regular system > > username and password. Uploads are enabled via regular SFTP so no > > special FTP server configuration is needed. > > > > It's a little hacky in parts but I'm generally pleased with the result. > > If anyone want to take a look I'd welcome comments. It's not in the > > main BL repo yet but can be found here: > > > > https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra > > > > Cheers, > > > > TIM > > > > -- > > Tim Booth > > NERC Environmental Bioinformatics Centre > > > > Centre for Ecology and Hydrology > > Maclean Bldg, Benson Lane > > Crowmarsh Gifford > > Wallingford, England > > OX10 8BB > > > > http://nebc.nerc.ac.uk > > +44 1491 69 2705 > > > > _______ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy packages for Bio-Linux - an update + Apache proxying
Hi All, As previously mentioned, I'm back working on packaging the Galaxy server as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately pushing towards something that could be Debian compliant. There's a way to go in that regard, but I do now have an updated package for Bio-Linux in final testing and it also has a new trick: doing "apt-get install galaxy-server-apache-proxy" will set up just that with no further configuration needed. The galaxy server appears at http://localhost/galaxy and users log in with their regular system username and password. Uploads are enabled via regular SFTP so no special FTP server configuration is needed. It's a little hacky in parts but I'm generally pleased with the result. If anyone want to take a look I'd welcome comments. It's not in the main BL repo yet but can be found here: https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra Cheers, TIM -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Uploads stalling on local Galaxy installation
Hi Devs, Just to let you know, I've nailed down my uploading issue to a reproducible test case which may be worth addressing if only by tweaking the notes in universe.wsgi.sample to be clearer. It was nothing to do with AJAX uploading in the end, and probably unrelated to the intermittent Chrome/localhost issue. To reproduce: 1) In universe_wsgi.ini, set "cookie_path = /galaxy". 2) Point Galaxy to an empty database and start it up so it builds a new empty DB. 3) Browse to localhost:8080 and try to upload a file, and the upload fails. Now you'll probably tell me there should be no / in the cookie path, but here's the weird thing... 4) Set "cookie_path = galaxy" (or back to None) and restart. 5) Upload a single file. 6) Set "cookie_path = /galaxy" and restart. 7) Upload a second file - and now it works! So it appears that setting the cookie_path to /whatever breaks the very first upload on a new server. This explains why I couldn't easily spot the problem, since I'd already had this setting on an existing installation and it had seemed to work. Does anyone have any insight into what might be happening? Cheers, TIM On Fri, 2013-10-04 at 18:05 +0100, Nate Coraor wrote: > Tim, > > Just to be certain, if you have AJAX upload enabled, you'll see this message > after you click Execute on the upload tool: > > http://www.bx.psu.edu/~nate/galaxy/ajax.png > > Whereas with it disabled, you'll see: > > http://www.bx.psu.edu/~nate/galaxy/noajax.png > > --nate > > On Oct 4, 2013, at 11:48 AM, Tim Booth wrote: > > > Hi Guys, > > > > Thanks for the replies. In that case it looks like the problem with my > > DEB package is not the same as the transient upload issue with Chrome. > > I was indeed using Chrome for most of the testing, but if I try to use > > the DEB package I've made then uploads still don't work at all in either > > Chrome or FF. > > > > The file appears in database/file/000/ but I don't see any job in the > > history. Setting ajax-upload to false doesn't seem to change anything > > whatsoever - (to reassure myself the XML was actually being reloaded I > > changed the version number and saw that changed). > > > > I've rolled back all the changes in my package that I think would > > reasonably impact the issue, in particular my patch that enables Galaxy > > to use certain system libraries rather than it's own eggs, but the > > problem still persists so I feel I have to try and understand it rather > > than blindly switching things around. > > > > In terms of debugging, I can use tcpflow and FireBug to dig into the > > network communication but how do I get Galaxy to give me more info about > > what it is up to? I can't see any obvious errors in the server log, and > > when if comes to the internals of the thing I don't know what is > > actually supposed to happening. What is the expected sequence of events > > when a file is uploaded? How is the status of the job signalled to the > > right-hand pane? Where do I start adding debugging statements and > > assertions to help me see what is happening? Can the in-browser > > debugging mentioned in the config file help me out at all, and where can > > I find out more? > > > > I'm happy to RTFM here but I've not yet found the right bit of the FM to > > read! > > > > Cheers, > > > > TIM > > > > On Fri, 2013-10-04 at 15:12 +0100, Nate Coraor wrote: > >> On Oct 4, 2013, at 10:00 AM, John Chilton wrote: > >> > >>> I have seen this behavior too and I too have switched to Firefox, but > >>> I have only ever seen this with chrome on localhost, never with chrome > >>> on a remote server behind a proxy. > >> > >> I've seen this issue before as well, a few of us have worked on it in the > >> past, but it seems to be transient and not easy to reproduce. In my > >> testing a couple years ago, I found that the call to ajaxSubmit() to > >> actually POST the data never occurs, and I couldn't get Firebug to tell me > >> why. > >> > >> You can disable the AJAX upload method in tools/data_source/upload.xml, > >> but that makes the UI a lot less usable, especially for large files. > >> > >> --nate > >> > >>> > >>> -John > >>> > >>> On Fri, Oct 4, 2013 at 8:55 AM, Carlos Borroto > >>> wrote: > >>>> On Fri, Oct 4, 2013 at 8:11 AM, Tim Booth wrote: > >>>>> When uploading a file, no job appea
Re: [galaxy-dev] Uploads stalling on local Galaxy installation
Hi Sam, I just want to try and get a working Galaxy on Bio-Linux, and unless your new code is coming out imminently (ie. next week) I need to make progress before then. If I can't debug the problem I think I'm going to have to revert changes until I get to something that happens to work. The result will be far removed from anything that could ever become a proper Debian package, and given the progress I've made I'd prefer not to quit now. By the time your next release comes out I'll have other jobs on my plate. Is it really that hard to peek into the system and try to see where the upload is jamming? Cheers, TIM On Fri, 2013-10-04 at 17:05 +0100, sam guerler wrote: > Hey, > > > We are currently refactoring and redesigning the front-end javascript > code. If the bug you are observing is 1) related to specifics of the > current front-end implementation and 2) if fixing the bug is time > consuming it might be reasonable to not invest that time and wait > until we complete the new client code for the upload process. > > > Thanks, > Sam > > > On Fri, Oct 4, 2013 at 11:48 AM, Tim Booth wrote: > Hi Guys, > > Thanks for the replies. In that case it looks like the > problem with my > DEB package is not the same as the transient upload issue with > Chrome. > I was indeed using Chrome for most of the testing, but if I > try to use > the DEB package I've made then uploads still don't work at all > in either > Chrome or FF. > > The file appears in database/file/000/ but I don't see any job > in the > history. Setting ajax-upload to false doesn't seem to change > anything > whatsoever - (to reassure myself the XML was actually being > reloaded I > changed the version number and saw that changed). > > I've rolled back all the changes in my package that I think > would > reasonably impact the issue, in particular my patch that > enables Galaxy > to use certain system libraries rather than it's own eggs, but > the > problem still persists so I feel I have to try and understand > it rather > than blindly switching things around. > > In terms of debugging, I can use tcpflow and FireBug to dig > into the > network communication but how do I get Galaxy to give me more > info about > what it is up to? I can't see any obvious errors in the > server log, and > when if comes to the internals of the thing I don't know what > is > actually supposed to happening. What is the expected sequence > of events > when a file is uploaded? How is the status of the job > signalled to the > right-hand pane? Where do I start adding debugging statements > and > assertions to help me see what is happening? Can the > in-browser > debugging mentioned in the config file help me out at all, and > where can > I find out more? > > I'm happy to RTFM here but I've not yet found the right bit of > the FM to > read! > > Cheers, > > TIM > > On Fri, 2013-10-04 at 15:12 +0100, Nate Coraor wrote: > > On Oct 4, 2013, at 10:00 AM, John Chilton wrote: > > > > > I have seen this behavior too and I too have switched to > Firefox, but > > > I have only ever seen this with chrome on localhost, never > with chrome > > > on a remote server behind a proxy. > > > > I've seen this issue before as well, a few of us have worked > on it in the past, but it seems to be transient and not easy > to reproduce. In my testing a couple years ago, I found that > the call to ajaxSubmit() to actually POST the data never > occurs, and I couldn't get Firebug to tell me why. > > > > You can disable the AJAX upload method in > tools/data_source/upload.xml, but that makes the UI a lot less > usable, especially for large files. > > > > --nate > > > > > > > > -John > > > > > > On Fri, Oct 4, 2013 at 8:55 AM, Carlos Borroto > wrote: > > >> On Fri, Oct 4, 2013 at 8:11 AM, Tim Booth > wrote: > > >>> When
Re: [galaxy-dev] Uploads stalling on local Galaxy installation
Hi Guys, Thanks for the replies. In that case it looks like the problem with my DEB package is not the same as the transient upload issue with Chrome. I was indeed using Chrome for most of the testing, but if I try to use the DEB package I've made then uploads still don't work at all in either Chrome or FF. The file appears in database/file/000/ but I don't see any job in the history. Setting ajax-upload to false doesn't seem to change anything whatsoever - (to reassure myself the XML was actually being reloaded I changed the version number and saw that changed). I've rolled back all the changes in my package that I think would reasonably impact the issue, in particular my patch that enables Galaxy to use certain system libraries rather than it's own eggs, but the problem still persists so I feel I have to try and understand it rather than blindly switching things around. In terms of debugging, I can use tcpflow and FireBug to dig into the network communication but how do I get Galaxy to give me more info about what it is up to? I can't see any obvious errors in the server log, and when if comes to the internals of the thing I don't know what is actually supposed to happening. What is the expected sequence of events when a file is uploaded? How is the status of the job signalled to the right-hand pane? Where do I start adding debugging statements and assertions to help me see what is happening? Can the in-browser debugging mentioned in the config file help me out at all, and where can I find out more? I'm happy to RTFM here but I've not yet found the right bit of the FM to read! Cheers, TIM On Fri, 2013-10-04 at 15:12 +0100, Nate Coraor wrote: > On Oct 4, 2013, at 10:00 AM, John Chilton wrote: > > > I have seen this behavior too and I too have switched to Firefox, but > > I have only ever seen this with chrome on localhost, never with chrome > > on a remote server behind a proxy. > > I've seen this issue before as well, a few of us have worked on it in the > past, but it seems to be transient and not easy to reproduce. In my testing > a couple years ago, I found that the call to ajaxSubmit() to actually POST > the data never occurs, and I couldn't get Firebug to tell me why. > > You can disable the AJAX upload method in tools/data_source/upload.xml, but > that makes the UI a lot less usable, especially for large files. > > --nate > > > > > -John > > > > On Fri, Oct 4, 2013 at 8:55 AM, Carlos Borroto > > wrote: > >> On Fri, Oct 4, 2013 at 8:11 AM, Tim Booth wrote: > >>> When uploading a file, no job appears in the right-hand panel. If I > >>> click refresh I might see a job but it stays in the blue "Dataset is > >>> uploading" state forever. > >> > >> Hi Tim, > >> > >> It is great news to hear Galaxy deb packages could be a possibility in > >> the near future. Regarding this issue, I can confirm I see it a lot on > >> Chrome but never on Firefox. It has been so bad for me that I almost > >> completely moved to using only Firefox when working on Galaxy. I > >> thought it could be a cookies issue, but clearing out galaxy cookies > >> from Chrome doesn't help in my case. > >> > >> I'll be happy to do further testing as this is something I would love > >> to see fixed. > >> > >> Best, > >> Carlos > >> ___ > >> Please keep all replies on the list by using "reply all" > >> in your mail client. To manage your subscriptions to this > >> and other Galaxy lists, please use the interface at: > >> http://lists.bx.psu.edu/ > >> > >> To search Galaxy mailing lists use the unified search at: > >> http://galaxyproject.org/search/mailinglists/ > > ___ > > Please keep all replies on the list by using "reply all" > > in your mail client. To manage your subscriptions to this > > and other Galaxy lists, please use the interface at: > > http://lists.bx.psu.edu/ > > > > To search Galaxy mailing lists use the unified search at: > > http://galaxyproject.org/search/mailinglists/ > > > ___ > Please keep all replies on the list by using "reply all" > in your mail client. To manage your subscriptions to this > and other Galaxy lists, please use the interface at: > http://lists.bx.psu.edu/ > > To search Galaxy mailing lists use the unified search at: > http://galaxyproject.org/search/mailinglists/ --
[galaxy-dev] Uploads stalling on local Galaxy installation
Hi All, Some of the developers will know me already but it's my first time being properly on this list as I'm now working concertedly on Galaxy. I'm responsible for Bio-Linux which currently ships with an out-of-date and slightly flakey galaxy, but I'm working to fix that. I've hit a problem. This may or may not be the same as the issue seen by Luobin Yang. When uploading a file, no job appears in the right-hand panel. If I click refresh I might see a job but it stays in the blue "Dataset is uploading" state forever. I'm working on packaging Galaxy for Debian/Ubuntu/Bio-Linux, specifically Ubuntu 12.04 LTS (python 2.7.3). To rearrange Galaxy into something that resembles a Debian package takes a lot of tinkering and I've patched a few bits of the code, so when uploads started freezing like this I assumed it was something I'd done and started reverting patches. But it turns out I can trigger this problem on a pristine download of the latest server release (20130926 from Bitbucket). All I do is: grab and unpack latest version python ./scripts/fetch_eggs.py -c universe_wsgi.ini.sample python ./scripts/fetch_eggs.py -c universe_wsgi.ini.sample -e psycopg2 (note - I'm not actually using Posgres in this case and I'm assuming this is irrelevant but reporting for completenes) mkdir eggs-cache ; export PYTHON_EGG_CACHE=`pwd`/eggs-cache ./run.sh (wait for SQLite DB to build) Ctrl-C, ./run.sh, test, Ctrl-C, ./run.sh, test, etc... After a few restarts the problem occurs. Failing that, it seems I can make failure more likely by tarring up the whole directory and untarring it again. I suspect all this is doing is affecting the disk cache timing and thus triggering some underlying race condition. Once the server is started, behaviour is apparently consistent - ie. all uploads either fail or all work. Other tasks seem to work, so long as they don't depend on the stalled uploads, but I've only tried a couple of these (just a basic "select first lines"). I'm now convinced there is some sort of race condition on start-up that is responsible. On my DEB package it fails like this all the time, but an issue is clearly there in the unmolested code, just rarer. I don't mind getting into the code to try and debug this but I don't know where to start, even with the source code docs and my rapidly increasing Python-fu. Trying to fathom the interactions between the threads and the browser and the SQLAlchemy layer I either need a good outline document or a developer to hold my hand for a bit. Can someone help me out with this one? Thanks in advance, TIM -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy Server on Bio-Linux
Hi Carlos, Yes, I'm an active member of Deb Med but unfortunately not finding time to read the mailing list properly nor to actually run Debian (or even the Ubuntu development release). We've actually factored time into the packaging project to contribute all relevant packaging into the Debian-Med repositories. This has to be done with a bit of care as there are now real differences between standards for Debian Sid and Ubuntu LTS for which we are packaging. My plan is to complete a bunch of packages and then review and commit what we have in a batch later this month. I do have a package for TopHat in my own PPA that I added late last year: https://launchpad.net/~tbooth/+archive/ppa1 I made a couple of tweaks to get the thing to link properly, but again I need to check that these aren't Ubuntu specific before I push any patches back to Alex, who seems to be actively committing changes to Git right now. Cheers, TIM On Wed, 2012-01-04 at 20:22 +, Carlos Borroto wrote: > On Tue, Jan 3, 2012 at 11:36 AM, Nate Coraor wrote: > > On Dec 20, 2011, at 1:09 PM, Bicak, Mesude wrote: > > > >> Dear Galaxy Developers, > >> > >> We work in Professor Dawn Field's group (Molecular Evolution and > Bioinformatics Research Group) at the NERC Environmental > Bioinformatics Centre (NEBC) of the Centre for Ecology and Hydrology > (CEH) Research Institute based in Oxford. > >> > >> We develop and distribute Bio-Linux > (http://nebc.nerc.ac.uk/tools/bio-linux), which is a customised Ubuntu > distribution that comes with 500+ bioinformatics packages. Within our > research group we also provide bioinformatics analysis for NERC-funded > researchers, and recently started looking into Galaxy as well. It > didn't take us long to discover its power and we would like to enable > Bio-Linux users to install, run and maintain the Galaxy server with > minimal effort, also with the aim to spread the word on Galaxy in > Europe! > >> > >> Recently we took on a project with Dr. Casey Bergman from > University of Manchester as the Principal Investigator, to package all > the necessary Galaxy dependencies for Ubuntu/Bio-Linux. As many > pre-requisities are already included with Bio-Linux, we are already > some way down this path. New packages that we create will appear in a > Launchpad PPA (https://launchpad.net/~nebc/+archive/galaxy). > >> > >> We will be happy to hear any comments regarding these efforts and > we hope that this will be a useful resource for all Galaxy users. Once > the initial packaging is done, we hope to collaborate with Galaxy team > in maintaining and improving this resource. > >> > >> Best wishes, > >> Tim, Soon and Mesude > > > > Hi Mesude, > > > > This is fantastic, thanks for letting us know, and please do post up > if there is anything we can help with. Also, if you're not aware, > Galaxy's cloud offering is built on CloudBioLinux, which itself is > built on Bio-Linux. > > > > --nate > > > > This is very good news, I'm personally a big fan of avoiding as much > as possible installing from source on production systems, as things > can get easily out of hand when trying to keep everything updated. > > Mesude, I recently joined Debian Med > (http://www.debian.org/devel/debian-med/ and > https://launchpad.net/~debian-med) which the exact same goal of > getting as much tools as possible packaged for Debian/Ubuntu. I see > Tim Booth is an active member in Debian Med and I was wondering if > your group is planning to keep the development of these packages in > the git/svn repositories of Debian Med. I wouldn't want to duplicate > your efforts. > > I like very much the use of PPAs while waiting for packages to be > officially included in Debian and trickle down to Ubuntu, process that > could be somehow slow at times. Just today I was able to upload > Tophat's package being developed at Debian Med to the PPA, I'm pretty > sure it needs more work, but you might be interested in taking a look. > > Kind regards, > Carlos -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 ___ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
