Re: [galaxy-dev] Salmon references and data manager

[Ignacio EGUINOA]

Hi Christopher and Björn, 

I have some comments about this because I also came up with these questions 
some time ago... 

> From: "Björn Grüning" 
> To: "Previti" , "galaxy-dev"
> 
> Sent: Friday, September 7, 2018 9:56:41 AM
> Subject: Re: [galaxy-dev] Salmon references and data manager

> Hi Christopher!

> > Dear Björn,

> > I just installed Salmon on our Galaxy instance and I have a couple of
> > basic questions.

> Sure, thanks for getting in touch!

> > Currently the reference transcriptomes are put in the same data table as
> > the genomes, would it be of interest to separate this and give the

> > transcriptomes their own table? I could probably try to do this...

> That I don't understand?
> Salmon is using this one here, isn't it?

> https://github.com/bgruening/galaxytools/blob/master/tools/salmon/salmon.xml#L233
What he means, I think, is the table to build the index from. Data managers 
that take a transcriptome as input get it from the all_fasta table, I think 
that is what he means by the genomes table. 
As I said at some point I also thought it may be useful to have a separate 
table (e.g all_transcriptomes) so that the genome and transcriptome entries of 
the same build don't get mixed. I think it would be good to have a way of 
listing only the transcriptomes from the all_gff but that would requiere some 
kind of standard on the naming to filter. We had this in our instance at some 
point but didn't help at all so I just modified the data manger to use the 
all_fasta and that is what I published. 
So, @Christopher ...having a separate table is not the solution although it 
would be easier for the GUI. For now just giving the entries a descriptive name 
to indicate the entries correspond to a transcriptome is enough and works ok 
for us. In any case this is not for users and at least for us its all handled 
through the API so, again, it's just a matter of taking care of the entries 
names and you are fine with using the all_fasta table. 

> > There is a data manager available that unfortunately has a bug. We fixed
> > that and it now populates the reference genome data table.

> Do you mean this one?

> https://github.com/ieguinoa/data_manager_salmon_index_builder

> > I would probably modify this as well use the new table. Could this be
> > useful? I'm not sure how to proceed...would I give you the modified
> > Salmon wrapper for inclusion in the package?

> If you can, please feel free to create PRs to the repositories, so we
> can all reviewed it. And then, when we merge, it gets automatically
> updated to the Tool Shed :)

As Björn said, if that's the one you are talking about please create a PR or an 
isssue or contact me. 

Cheers, 
Ignacio 

> Thanks!
> Bjoern

> > Best regards,

> > Christopher


> > --
> > *Dr. Christopher Previti*
> > Genomics and Proteomics Core Facility
> > High Throughput Sequencing (W190)
> > Bioinformatician

> > German Cancer Research Center (DKFZ)
> > Foundation under Public Law
> > Im Neuenheimer Feld 580
> > 69120 Heidelberg
> > Germany
> > Room: B2.102 (INF580/TP3)
> > Phone: +49 6221 42-4661

> > christopher.prev...@dkfz.de 
> > www.dkfz.de 

> > Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> > VAT-ID No.: DE143293537

> > Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> > Personen bestimmt, an die sie adressiert ist.
> > Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> > Informationen enthalten. Sollten Sie nicht
> > der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
> > Absender und löschen Sie die Mitteilung.
> > Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> > untersagt.


> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/

> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Uploads via Nginx proxy fail after upgrade to current release

[Ignacio EGUINOA]

Hi again, 

I'm still not sure if this is the same error as Hans-Georg Sommer but in my 
case I've just noticed that the error only occurs with files SMALLER than ~1KB 

> From: "Ignacio Eguinoa" 
> To: "Hans-Georg Sommer" 
> Cc: "galaxy-dev" 
> Sent: Wednesday, June 6, 2018 3:43:47 PM
> Subject: Re: [galaxy-dev] Uploads via Nginx proxy fail after upgrade to 
> current
> release

> Hi,

> I'm also experiencing what I think is the same problem after upgrading to 
> v18.05
> (from 17.09). Some more info on this:
> In my case I've tried this with and without the proxy (Apache) and using 
> either
> uWSGI or Paste as http servers and this changes nothing.
> I have 2 servers and in one of them the upload works perfectly fine after the
> upgrade to v18.05. here are barely any differences in the framework code (just
> a few custom changes for the private instance) and it also works with Apache 
> as
> proxy so I really don't understand where is the error coming from.
> If i try to upload a file to a data library from a path in the server then it
> works ok.
> I get the same error as Hans-Georg Somme in the upload -> Warning: Internal
> Server Error (500)
> The error I get in the logs is:

> galaxy.web.framework.decorators ERROR 2018-06-06 15:17:40,976 
> [p:17537,w:1,m:0]
> [uWSGIWorker1Core3] Uncaught exception in exposed API method:
> Traceback (most recent call last):
> File "lib/galaxy/web/framework/decorators.py", line 154, in decorator
> rval = func(self, trans, *args, **kwargs)
> File "lib/galaxy/webapps/galaxy/api/uploads.py", line 43, in create
> chunk_size = os.fstat(session_chunk.file.fileno()).st_size
> AttributeError: 'cStringIO.StringO' object has no attribute 'fileno'

> Hope this clarifies a bit.
> Ignacio

>> From: "Hans-Georg Sommer" 
>> To: "galaxy-dev" 
>> Sent: Wednesday, June 6, 2018 11:13:40 AM
>> Subject: [galaxy-dev] Uploads via Nginx proxy fail after upgrade to current
>> release

>> Hi,

>> after updating Galaxy to 18.05 (from 17.05) the file upload doesn't work
>> anymore, but shows an Internal Server Error (500) in the status column
>> of the upload form.

>> We use a nginx proxy also for uploads, configured as described in [1],
>> galaxy.ini contains the line 'nginx_upload_path = /_upload'.

>> The nginx access logs show, that the upload requests seem to go to the
>> wrong location ('/api/uploads' instead of '/_upload'):
>> 0.0.0.0 - - [06/Jun/2018:10:44:43 +0200] "POST /api/uploads HTTP/1.1"
>> 500 193 "https://galaxy.gwdg.de/;

>> I searched the release notes and the github issues, but wasn't able to
>> find anything besides [2], which sounds similar but does not give much
>> information.

>> Thanks for your help,

>> Hans-Georg

>> [1] https://docs.galaxyproject.org/en/latest/admin/nginx.html
>> [2] https://biostar.usegalaxy.org/p/25870/

>> --
>> 
>> Hans-Georg Sommer
>> Arbeitsgruppe "Anwendungs- und Informationssysteme"
>> E-Mail: hans-georg.som...@gwdg.de
>> 
>> Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen
>> (GWDG)
>> Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de
>> Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: g...@gwdg.de
>> Service-Hotline: Tel.: +49 551 201-1523, E-Mail: supp...@gwdg.de

>> Geschäftsführer: Prof. Dr. Ramin Yahyapour
>> Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lassau
>> Sitz der Gesellschaft: Göttingen
>> Registergericht: Göttingen, Handelsregister-Nr. B 598
>> 
>> Zertifiziert nach ISO 9001
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> https://lists.galaxyproject.org/

>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/

> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/

> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Uploads via Nginx proxy fail after upgrade to current release

[Ignacio EGUINOA]

Hi, 

I'm also experiencing what I think is the same problem after upgrading to 
v18.05 (from 17.09). Some more info on this: 
In my case I've tried this with and without the proxy (Apache) and using either 
uWSGI or Paste as http servers and this changes nothing. 
I have 2 servers and in one of them the upload works perfectly fine after the 
upgrade to v18.05. here are barely any differences in the framework code (just 
a few custom changes for the private instance) and it also works with Apache as 
proxy so I really don't understand where is the error coming from. 
If i try to upload a file to a data library from a path in the server then it 
works ok. 
I get the same error as Hans-Georg Somme in the upload -> Warning: Internal 
Server Error (500) 
The error I get in the logs is: 

galaxy.web.framework.decorators ERROR 2018-06-06 15:17:40,976 [p:17537,w:1,m:0] 
[uWSGIWorker1Core3] Uncaught exception in exposed API method: 
Traceback (most recent call last): 
File "lib/galaxy/web/framework/decorators.py", line 154, in decorator 
rval = func(self, trans, *args, **kwargs) 
File "lib/galaxy/webapps/galaxy/api/uploads.py", line 43, in create 
chunk_size = os.fstat(session_chunk.file.fileno()).st_size 
AttributeError: 'cStringIO.StringO' object has no attribute 'fileno' 

Hope this clarifies a bit. 
Ignacio 

> From: "Hans-Georg Sommer" 
> To: "galaxy-dev" 
> Sent: Wednesday, June 6, 2018 11:13:40 AM
> Subject: [galaxy-dev] Uploads via Nginx proxy fail after upgrade to current
> release

> Hi,

> after updating Galaxy to 18.05 (from 17.05) the file upload doesn't work
> anymore, but shows an Internal Server Error (500) in the status column
> of the upload form.

> We use a nginx proxy also for uploads, configured as described in [1],
> galaxy.ini contains the line 'nginx_upload_path = /_upload'.

> The nginx access logs show, that the upload requests seem to go to the
> wrong location ('/api/uploads' instead of '/_upload'):
> 0.0.0.0 - - [06/Jun/2018:10:44:43 +0200] "POST /api/uploads HTTP/1.1"
> 500 193 "https://galaxy.gwdg.de/;

> I searched the release notes and the github issues, but wasn't able to
> find anything besides [2], which sounds similar but does not give much
> information.

> Thanks for your help,

> Hans-Georg

> [1] https://docs.galaxyproject.org/en/latest/admin/nginx.html
> [2] https://biostar.usegalaxy.org/p/25870/

> --
> 
> Hans-Georg Sommer
> Arbeitsgruppe "Anwendungs- und Informationssysteme"
> E-Mail: hans-georg.som...@gwdg.de
> 
> Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen
> (GWDG)
> Am Faßberg 11, 37077 Göttingen, URL: http://www.gwdg.de
> Tel.: +49 551 201-1510, Fax: +49 551 201-2150, E-Mail: g...@gwdg.de
> Service-Hotline: Tel.: +49 551 201-1523, E-Mail: supp...@gwdg.de

> Geschäftsführer: Prof. Dr. Ramin Yahyapour
> Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lassau
> Sitz der Gesellschaft: Göttingen
> Registergericht: Göttingen, Handelsregister-Nr. B 598
> 
> Zertifiziert nach ISO 9001
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client. To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> https://lists.galaxyproject.org/

> To search Galaxy mailing lists use the unified search at:
> http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Combine different resolvers for one tool

[Ignacio EGUINOA]

Hi all, 

I'm setting up a Galaxy server which consists of a Galaxy head node and a 
Docker engine (swarm) running on top of an OpenNebula cloud. So, ideally I 
would like to run everything using this Docker engine. 

For tools that only have 1 dependency, the Docker resolver works perfectly. 
But, in the case of tools having more than 1 dependency, if I activate mulled 
containers (enable_beta_mulled_containers = True) it will just try to run the 
tool script using the container corresponding to the 1st dependency found, 
which of course fails. 

What is the correct approach for resolving several dependencies using Docker? 
I tried activating the involucro which (as far as I understood) builds the 
docker container on-the-go based on the conda environment which results from 
the merge of all the dependencies. I know it's a beta feature but still, 
couldn't make it work...besides setting the path and auto_init to True, what am 
I supposed to put in the containers_resolvers_config_file ? 

Beside this, would it be possible to combine the dependencies in another way? 
Is it possible in the current version to load the conda environment inside a 
I know it may sound redundant and unnecessary in terms of dependency resolution 
but in cases like I the one I described, where you have the Docker swarm 
already running. 
I tried this with a few tools, using one of the containers as base and loading 
the rest of the dependencies using conda environment. Would it be possible to 
do this in an automatic way, using a general base container? 

Anyway, the general question is where are things going regarding dependency 
resolution, so that I can get an idea of what to expect in the future and how 
to collaborate with development of these features. I found a thread on github 
about this ( [ https://github.com/galaxyproject/galaxy/issues/3299 | 
https://github.com/galaxyproject/galaxy/issues/3299 ] ) but don't know the 
resolutions taken about it (if any). 

Thanks in advance for the help. 
Ignacio 




Ignacio Eguinoa - Predoctoral fellow 
Applied Bioinformatics And Biostatistics 

VIB-UGent Center for Plant Systems Biology 
Ghent University 
Technologiepark 927 - 9052 Ghent - Belgium 
Tel. +32(0)9 331 36 95 
[ http://www.psb.ugent.be/ | www.psb.ugent.be ] 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Mapping LDAP groups to Galaxy groups/roles

[Ignacio EGUINOA]

Hi all, 

I'm currently looking for some help to manage groups/roles in Galaxy. 
The instance I'm working with uses LDAP authentication (all Galaxy registered 
users are in LDAP) and the datasets that are uploaded have permissions 
'patterns' that can be mapped to groups already available in the LDAP server. 
Current approach is to manually add roles and groups and assign the registered 
users to them. Since the amount of users and groups is considerable large and 
can also change, I was hoping to have a more automatic system such that, when a 
new dataset is added, the required group/roles are already available to be 
assigned. 
I guess that, since the LDAP system is only used for authentication purposes, 
there is no way to directly map all the groups and users to galaxy (and 
actually it would be an overkill). But, at least I would like to get to map the 
primary LDAP group in a somehow automatic way. 
The only approach I could come up with is to manage the group/roles creation 
through the API, assuming I have a list of the relevant group names at a 
certain point in time, and then update the link of users to the available 
groups/roles by using information from LDAP. 
When I first deployed the instance I added an extra check during user 
authentication to verify that the user belongs to a specific LDAP group which 
grants access to the Galaxy instance. To do this, a list of groups the user 
belongs to is obtained from the server. I was thinking about extending this 
step to also update the user's Galaxy profile, associating it with the groups 
that are already created in Galaxy. 
Since this will require some digging into Galaxy code, I first want ask if 
there is any alternative solution for this kind of situation or some work was 
already done.. 

Thanks in advance 



Ignacio EGUINOA - Predoctoral fellow 
Applied Bioinformatics And Biostatistics 

VIB Department of Plant Systems Biology 
Ghent University 
Technologiepark 927 - 9052 Ghent - Belgium 
Tel. +32(0)9 331 36 95 
www.psb.ugent.be 



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/