Re: [galaxy-dev] How to upload local files in Galaxy
Hi, There is also another solution if you don't want let users being able to create libraries. We have implemented this solution in our local production server here at Institut Curie. We add a tool call local upload file, that takes as entry parameters the name of the dataset, the type of file and the path to the file you want to upload. At the execution, the bash script behind will remove the output dataset created by Galaxy and replaced it by a symbolic link to the file (so you don't duplicate files). Just to warn you, there could be a security failure with this tool, because if it is locally executed, it will be by the Galaxy applicative user that can potentially have more rights on file than the current user. To go through this failure, we execute this tool on our local cluster (with pbs/torque) as the current user (so if the current user try to upload a file that he doesn't owned, the tool will not be able to create the symbolic link and the user will receive an error on the Galaxy interface). Tell me if you're interested in such a tool, and I send you the xml file and bash script associated. Bests, Alban Le 07/06/2012 20:11, Mehmet Belgin a écrit : Brad, Thank you for your fast reply! Looks like library_import_dir is for admins and there is another library option for users. I will try with that one and see if the files appear in the GUI. Thanks! = Mehmet Belgin, Ph.D. (mehmet.bel...@oit.gatech.edu mailto:mehmet.bel...@oit.gatech.edu) Scientific Computing Consultant | OIT - Academic and Research Technologies Georgia Institute of Technology 258 Fourth Street, Rich Building, Room 326 Atlanta, GA 30332-0700 Office: (404) 385-0665 On Jun 7, 2012, at 2:04 PM, Langhorst, Brad wrote: Mehmet: It's not important how the files get there, they could be moved via ftp, scp, cp, smb - whatever. Galaxy will use that directory to import from no matter how the files arrive. I found that confusing at first too. Brad On Jun 7, 2012, at 1:55 PM, Mehmet Belgin wrote: Hi Everyone, I am helping a research group to use Galaxy on our clusters. Unfortunately I have no previous experience with Galaxy, but learning along the way. We are almost there, but cannot figure out one particular issue. This is about configuration of Galaxy, so I thought developers list is a better place to submit than the user list. The galaxy web interface allows for either copy/paste of text, or a URL. Unfortunately we cannot setup a FTP server as instructed due to restrictions on the cluster. The files we are trying to upload are large; around 2GB in size. It does not make sense to upload these files to a remote location (which we can provide an URL for) and download them back, since the data and galaxy are on the same system. However, I could not find a way to open these files locally. I did some reading, and hoped that library_import_dir in universe_wsgi.ini would do the trick, but it didn't. Therefore, I will really appreciate any suggestions. Thanks a lot in advance! -Mehmet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com mailto:langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] How to upload local files in Galaxy
Hi all, Despite the possible security failure, I push the upload_local_tool in main toolshed under the data source repository. I notice about the possible security failure and the way to fix it. I also have a simple 5 minutes method to modify existing tools to allow users to choose the directory where to write the outputs (to keep working with your own tree whitout breaking Galaxy database links). Tell me if you are interested in this method.. Bests, Alban Hello, I am interested in this local upload file tool. Can you please send the files to me as well. Thanks. -Raj On 6/8/12 4:36 AM, Alban Lermine wrote: Hi, There is also another solution if you don't want let users being able to create libraries. We have implemented this solution in our local production server here at Institut Curie. We add a tool call local upload file, that takes as entry parameters the name of the dataset, the type of file and the path to the file you want to upload. At the execution, the bash script behind will remove the output dataset created by Galaxy and replaced it by a symbolic link to the file (so you don't duplicate files). Just to warn you, there could be a security failure with this tool, because if it is locally executed, it will be by the Galaxy applicative user that can potentially have more rights on file than the current user. To go through this failure, we execute this tool on our local cluster (with pbs/torque) as the current user (so if the current user try to upload a file that he doesn't owned, the tool will not be able to create the symbolic link and the user will receive an error on the Galaxy interface). Tell me if you're interested in such a tool, and I send you the xml file and bash script associated. Bests, Alban Le 07/06/2012 20:11, Mehmet Belgin a écrit : Brad, Thank you for your fast reply! Looks like library_import_dir is for admins and there is another library option for users. I will try with that one and see if the files appear in the GUI. Thanks! = Mehmet Belgin, Ph.D. (mehmet.bel...@oit.gatech.edu mailto:mehmet.bel...@oit.gatech.edu) Scientific Computing Consultant | OIT - Academic and Research Technologies Georgia Institute of Technology 258 Fourth Street, Rich Building, Room 326 Atlanta, GA 30332-0700 Office: (404) 385-0665 On Jun 7, 2012, at 2:04 PM, Langhorst, Brad wrote: Mehmet: It's not important how the files get there, they could be moved via ftp, scp, cp, smb - whatever. Galaxy will use that directory to import from no matter how the files arrive. I found that confusing at first too. Brad On Jun 7, 2012, at 1:55 PM, Mehmet Belgin wrote: Hi Everyone, I am helping a research group to use Galaxy on our clusters. Unfortunately I have no previous experience with Galaxy, but learning along the way. We are almost there, but cannot figure out one particular issue. This is about configuration of Galaxy, so I thought developers list is a better place to submit than the user list. The galaxy web interface allows for either copy/paste of text, or a URL. Unfortunately we cannot setup a FTP server as instructed due to restrictions on the cluster. The files we are trying to upload are large; around 2GB in size. It does not make sense to upload these files to a remote location (which we can provide an URL for) and download them back, since the data and galaxy are on the same system. However, I could not find a way to open these files locally. I did some reading, and hoped that library_import_dir in universe_wsgi.ini would do the trick, but it didn't. Therefore, I will really appreciate any suggestions. Thanks a lot in advance! -Mehmet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com mailto:langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Bio-informatics consultant GGF (http://dna.uga.edu) and QBCG (http://qbcg.uga.edu) 706-542-6092 (8-12 Tuesday, Thursday
Re: [galaxy-dev] How to upload local files in Galaxy
Re, My bad: upload_local_tool - upload_local_file tool Bests, Alban Hi all, Despite the possible security failure, I push the upload_local_tool in main toolshed under the data source repository. I notice about the possible security failure and the way to fix it. I also have a simple 5 minutes method to modify existing tools to allow users to choose the directory where to write the outputs (to keep working with your own tree whitout breaking Galaxy database links). Tell me if you are interested in this method.. Bests, Alban Hello, I am interested in this local upload file tool. Can you please send the files to me as well. Thanks. -Raj On 6/8/12 4:36 AM, Alban Lermine wrote: Hi, There is also another solution if you don't want let users being able to create libraries. We have implemented this solution in our local production server here at Institut Curie. We add a tool call local upload file, that takes as entry parameters the name of the dataset, the type of file and the path to the file you want to upload. At the execution, the bash script behind will remove the output dataset created by Galaxy and replaced it by a symbolic link to the file (so you don't duplicate files). Just to warn you, there could be a security failure with this tool, because if it is locally executed, it will be by the Galaxy applicative user that can potentially have more rights on file than the current user. To go through this failure, we execute this tool on our local cluster (with pbs/torque) as the current user (so if the current user try to upload a file that he doesn't owned, the tool will not be able to create the symbolic link and the user will receive an error on the Galaxy interface). Tell me if you're interested in such a tool, and I send you the xml file and bash script associated. Bests, Alban Le 07/06/2012 20:11, Mehmet Belgin a écrit : Brad, Thank you for your fast reply! Looks like library_import_dir is for admins and there is another library option for users. I will try with that one and see if the files appear in the GUI. Thanks! = Mehmet Belgin, Ph.D. (mehmet.bel...@oit.gatech.edu mailto:mehmet.bel...@oit.gatech.edu) Scientific Computing Consultant | OIT - Academic and Research Technologies Georgia Institute of Technology 258 Fourth Street, Rich Building, Room 326 Atlanta, GA 30332-0700 Office: (404) 385-0665 On Jun 7, 2012, at 2:04 PM, Langhorst, Brad wrote: Mehmet: It's not important how the files get there, they could be moved via ftp, scp, cp, smb - whatever. Galaxy will use that directory to import from no matter how the files arrive. I found that confusing at first too. Brad On Jun 7, 2012, at 1:55 PM, Mehmet Belgin wrote: Hi Everyone, I am helping a research group to use Galaxy on our clusters. Unfortunately I have no previous experience with Galaxy, but learning along the way. We are almost there, but cannot figure out one particular issue. This is about configuration of Galaxy, so I thought developers list is a better place to submit than the user list. The galaxy web interface allows for either copy/paste of text, or a URL. Unfortunately we cannot setup a FTP server as instructed due to restrictions on the cluster. The files we are trying to upload are large; around 2GB in size. It does not make sense to upload these files to a remote location (which we can provide an URL for) and download them back, since the data and galaxy are on the same system. However, I could not find a way to open these files locally. I did some reading, and hoped that library_import_dir in universe_wsgi.ini would do the trick, but it didn't. Therefore, I will really appreciate any suggestions. Thanks a lot in advance! -Mehmet ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Brad Langhorst langho...@neb.com mailto:langho...@neb.com 978-380-7564 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Bio-informatics consultant GGF (http
Re: [galaxy-dev] Sharing data in Cistrome
Hi, Did you have special characters (é, à,...) in the email of the person you try to share with? If so, try whitout this special characters (è - e or à - a), it should works.. Bests, Alban Le 07/06/2012 16:26, Päivi Pihlajamaa a écrit : Hi, I am trying to share my files/history in Galaxy/Cistrome with my colleague, but the 'Share with user' link gives the following error. Could you please help me, am i doing something wrong or is there a bug in the system? Best regards, Päivi Server Error URL: http://cistrome.org/ap/history/share?use_panels=Falseid=7b2014ae6b4b7c3f Module paste.exceptions.errormiddleware:143 in __call__ app_iter = self.application(environ, start_response) Module paste.debug.prints:98 in __call__ environ, self.app) Module paste.wsgilib:539 in intercept_output app_iter = application(environ, replacement_start_response) Module paste.recursive:80 in __call__ return self.application(environ, start_response) Module paste.httpexceptions:632 in __call__ return self.application(environ, start_response) Module galaxy.web.framework.base:160 in __call__ body = method( trans, **kwargs ) Module galaxy.web.framework:90 in decorator return func( self, trans, *args, **kwargs ) Module galaxy.web.controllers.history:960 in share send_to_err=send_to_err ) Module galaxy.web.framework:733 in fill_template return self.fill_template_mako( filename, **kwargs ) Module galaxy.web.framework:744 in fill_template_mako return template.render( **data ) Module mako.template:296 in render return runtime._render(self, self.callable_, args, data) Module mako.runtime:660 in _render **_kwargs_for_callable(callable_, data)) Module mako.runtime:692 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) Module mako.runtime:718 in _exec_template callable_(context, *args, **kwargs) Module _base_mako:42 in render_body __M_writer(unicode(next.body())) Module _history_share_mako:74 in render_body __M_writer(unicode(history.name)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error: Job output not returned from cluster
) ofileExistsSTR = str(ofileExists) while efileExistsSTR == False: time.sleep( 1 ) efileExists = os.path.isfile(efile) efileExistsSTR = str(efileExists) while ofileExistsSTR == False: time.sleep( 1 ) ofileExists = os.path.isfile(ofile) ofileExistsSTR = str(ofileExists) # Back to original code ofh = file(ofile, r) efh = file(efile, r) stdout = ofh.read( 32768 ) stderr = efh.read( 32768 ) except: stdout = '' stderr = 'Job output not returned by PBS: the output datasets were deleted while the job was running, the job was manually dequeued or there was a cluster error.' log.debug(stderr) * Last step is to allow galaxy user to run sudo Hope it can help you finding your problem.. See you, Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Same error in multiple tools
Hi, I'm running these tools on a cluster.. Alban Emil Bouvier wrote: Alban, Are you running these tools on a cluster, or on a local system? Dave B. On 3/19/12 6:55 AM, Alban Lermine wrote: Hi, I have the same error message for multiple tools (GFFtoBED, BEDtoBIGBED,BAMtoSAM,...): /from galaxy import eggs ImportError: No module named galaxy It's happening only since the last upgrade, do you know what is going wrong? Thanks, Alban / -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Same error in multiple tools
Hi, I have the same error message for multiple tools (GFFtoBED, BEDtoBIGBED,BAMtoSAM,...): /from galaxy import eggs ImportError: No module named galaxy It's happening only since the last upgrade, do you know what is going wrong? Thanks, Alban / -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] Error starting Galaxy after update of March 2012
Nate Coraor wrote: On Mar 13, 2012, at 11:08 AM, Alban Lermine wrote: Hi Alban, Is your Galaxy server behind a firewall or without network access such that it cannot access http://eggs.g2.bx.psu.edu/ ? What happens if you run `python scripts/fetch_eggs.py` by hand? Is there a way to downgrade Galaxy? Yes, in most cases you can downgrade with `hg update -r old changeset id`. --nate Thanks by advance for your answer.. Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Re, Thanks for your fast answer.. I run fetch_eggs.py by hand, here is the return: Fetched http://eggs.g2.bx.psu.edu/boto/boto-2.2.2-py2.6.egg Removed conflicting egg: galaxy_dir/galaxy-dist/eggs/boto-1.8d-py2.6.egg I tried then to start Galaxy, and I also have a new error message: Traceback (most recent call last): File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/web/buildapp.py, line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/app.py, line 24, in __init__ self.config.check() File /bioinfo/http/prod/hosted/galaxy.curie.fr/galaxy-dist/lib/galaxy/config.py, line 254, in check raise ConfigurationError( Unable to create missing directory: %s\n%s % ( path, e ) ) ConfigurationError: Unable to create missing directory: ./../shed_tools [Errno 13] Permission denied: './../shed_tools' It seems it's now only a permission problem.. (I only have rights on galaxy-dist directory), I will ask our system engineer to rerun update.. Hi Alban, You can fix this one by changing the path to the shed_tools directory in shed_tool_conf.xml. It can be any directory you can write to, but should not be underneath the galaxy-dist directory. --nate Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 It's working again, great thanks, Alban -- Alban Lermine Unité 900 : Inserm - Mines ParisTech - Institut Curie « Bioinformatics and Computational Systems Biology of Cancer » 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France Tel : +33 (0) 1 56 24 69 84 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] Error starting Galaxy after update of March 2012
Hi,
I run upgrade: hg pull -u -r 40f1816d6857 on our local instance, and I'm
now unable to start Galaxy..
Here is the content of Galaxy_log:
Traceback (most recent call last):
File galaxy_dir/galaxy-dist/lib/galaxy/web/buildapp.py, line 81,
in app_factory
from galaxy.app import UniverseApplication
File galaxy_dir/galaxy-dist/lib/galaxy/app.py, line 11, in module
from galaxy.objectstore import build_object_store_from_config
File galaxy_dir/galaxy-dist/lib/galaxy/objectstore/__init__.py,
line 27, in module
from galaxy.objectstore.s3_multipart_upload import multipart_upload
File
galaxy_dir/galaxy-dist/lib/galaxy/objectstore/s3_multipart_upload.py,
line 22, in module
eggs.require('boto')
File galaxy_dir/galaxy-dist/lib/galaxy/eggs/__init__.py, line 415,
in require
raise EggNotFetchable( str( [ egg.name for egg in e.eggs ] ) )
EggNotFetchable: ['boto']
Removing PID file galaxy_dir/galaxy-dist/galaxy.pid
Is that an error specific to our instance?
Is there a way to downgrade Galaxy?
Thanks by advance for your answer..
Alban
--
Alban Lermine
Unité 900 : Inserm - Mines ParisTech - Institut Curie
« Bioinformatics and Computational Systems Biology of Cancer »
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel : +33 (0) 1 56 24 69 84
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
