[galaxy-dev] Re: cyverse-ansible.singularity

[Keith Suderman]

Hi Luc,

The version used in our playbook actually references one of our pull requests 
[1] and it looks like it references the most recent version.  However, even the 
most recent playbook on GitHub uses the incorrect URL.  Something must have 
changed since January.

In the interim you could try editing the role locally.  In the 
roles/cyverse-ansible.singularity/tasks/singularity-post-3.yml fie (or, or 
around line 47) change:

url: 
"https://github.com/sylabs/singularity/releases/download/v{{singularity_version}}/singularity-{{singularity_version}}.tar.gz";

to:
url: https://github.com/sylabs/singularity/archive/refs/tags/v{{ 
singularity_version }}.tar.gz


I haven't had a chance to actually test that, but it should pull the correct 
tarball.

Cheers,
Keith

1. https://github.com/galaxyproject/training-material/pull/2269


> On May 7, 2021, at 9:24 AM, Luc Cornet  wrote:
> 
> 
>  External Email - Use Caution
> 
> 
> 
> Hello all,
> 
> I have a problem with singularity while installing galaxy with ansible.
> 
> ```
> ASK [cyverse-ansible.singularity : Ubuntu package installation for v3.0 or 
> newer] 
> *
> ok: [galaxy.inbios.uliege.be]
> 
> TASK [cyverse-ansible.singularity : CentOS package installation for v3.0 or 
> newer] 
> *
> skipping: [galaxy.inbios.uliege.be]
> 
> TASK [cyverse-ansible.singularity : check if singularity binary exists] 
> 
> ok: [galaxy.inbios.uliege.be]
> 
> TASK [cyverse-ansible.singularity : Obtain current binary version] 
> *
> skipping: [galaxy.inbios.uliege.be]
> 
> TASK [cyverse-ansible.singularity : download the singularity release] 
> **
> fatal: [galaxy.inbios.uliege.be]: FAILED! => {"changed": false, "dest": 
> "/tmp/singularity-3.7.0.tar.gz", "elapsed": 5, "msg": "Request failed", 
> "response": "HTTP Error 404: Not Found", "status_code": 404, "url": 
> "https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsylabs%2Fsingularity%2Freleases%2Fdownload%2Fv3.7.0%2Fsingularity-3.7.0.tar.gz&data=04%7C01%7Csuderman%40jhu.edu%7C43edc9785a1246f3b79a08d9116411fc%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637559943775184956%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=DZ5eYAoCSsZHRgHl7BnnGwyER%2Fcyv2RuNCn5KEZG8nw%3D&reserved=0"}
> ```
> 
> The URL for singularity has changed, Do you know how can i fix that ?
> 
> 
> Maybe a more recent version of :
> 
> ```
> - src: cyverse-ansible.singularity
> version: 048c4f178077d05c1e67ae8d9893809aac9ab3b7
> ```
> 
> Thanks for the help,
> Luc
> 
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  %(web_page_url)s
> 
> To search Galaxy mailing lists use the unified search at:
>  
> https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgalaxyproject.org%2Fsearch%2F&data=04%7C01%7Csuderman%40jhu.edu%7C43edc9785a1246f3b79a08d9116411fc%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637559943775184956%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6%2BNoAtq58508X9vNI4ijTiTsr%2BCCPEPk8vyVAL7K82Q%3D&reserved=0
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Ansible Galaxy using root user for role installation

[Keith Suderman]

Hi Sandra,

My initial hunch is that Ansible is trying to use root to create the 
galaxy_privsep_user.  Have you set galaxy_create_privsep_user to false in the 
group_vars?

Cheers,
Keith

On Feb 15, 2021, at 12:21 AM, Sandra Maksimovic 
mailto:sandra.maksimo...@mcri.edu.au>> wrote:

Hi there,

Would anyone happen to know why Ansible tries to connect to a remote host 
(CentOS 7) using 'root' during Galaxy role installation?

My playbook has been configured to connect to the remote host using a different 
user account via SSH, however, the role installation is not honoring this for 
some reason and continues to use root. The remote host has had SSH root login 
disabled for security reasons and setting various galaxy 
user/privsep/become/remote user variables in the group_vars has not had any 
effect.

TASK [galaxyproject.galaxy : Create galaxy_root] *
fatal: [server_name]: UNREACHABLE! => {"changed": false, "msg": 
"Invalid/incorrect password: Permission denied, please try again.\r\nPermission 
denied, please try again.\r\nroot@server_name: Permission denied 
(publickey,password).", "unreachable": true}

Thanks,
Sandra



This e-mail and any attachments to it (the "Communication") are, unless 
otherwise stated, confidential, may contain copyright material and is for the 
use only of the intended recipient. If you receive the Communication in error, 
please notify the sender immediately by return e-mail, delete the Communication 
and the return e-mail, and do not read, copy, retransmit or otherwise deal with 
it. Any views expressed in the Communication are those of the individual sender 
only, unless expressly stated to be those of Murdoch Children’s Research 
Institute (MCRI) ABN 21 006 566 972 or any of its related entities. MCRI does 
not accept liability in connection with the integrity of or errors in the 
Communication, computer virus, data corruption, interference or delay arising 
from or in respect of the Communication.

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: SAML Authentication

[Keith Suderman]

I now have the login portion of the SAML authentication working. I still have 
to implement the logout flow, but I wanted to get some feedback from the 
developers before proceeding too much further.  I have opened a [WIP] 
pull-request on GitHub [1] and I have an example running at 
https://idp.lappsgrid.org that uses https://samltest.id as the IdP.  It is only 
a two core VM with limited memory and will likely go away soon, so don't try to 
use it to do real work ;)

My original intent was to use the Social-Auth SAML backend and try to integrate 
with the OIDC classes, but that would require more changes to Galaxy than I am 
comfortable making right now.  So for now I use python3-saml library (what 
Social-Auth uses under the hood) and a single controller class to handle the 
login flow.  

The questions I have right now are:

1. The current controller piggy backs on (hijacks?) the /authnz url-space and 
adds several /authnz/saml urls (login, logout, metadata, acs).  Should I use 
something else or are these ok?

2. I think I have mostly figured out how to add fields to the galaxy.yml file. 
The python3-saml library requires a configuration directory that contains two 
JSON files and a 'certs" directory with certificates and keys used to 
sign/authenticate SAML requests.  Should this go into the config directory?  
That seem like the logical place to me.  However, I have tried to copy the code 
used to make the oidc_config_* relative to the config directory,  but when I 
access my saml_config_dir field at runtime the path is relative to the Galaxy 
directory and not the config directory.

3. I will need to maintain some information in a "session" of some sort to 
implement the logout correctly.  Should I just stick this in a cookie?  I don't 
think this is appropriate to go into the GalaxySession and I don't see any 
other "session-like" storage.

I'm sure to have more questions soon.

Cheers,
Keith

1. https://github.com/galaxyproject/galaxy/pull/9524

P.S. I will remember to squash my commits for the next PR ;)


> On Mar 12, 2020, at 8:39 PM, Keith Suderman  wrote:
> 
> I have the SAML login implementation 99% complete.  However, after 
> authenticating with an IdP and being redirected back to Galaxy the user is 
> still not "logged in"...  The relevant code is on GitHub [1], but basically 
> in the callback handler I do:
> 
> user = trans.get_or_create_remote_user(user_email)
> trans.handle_user_login(user)
> 
> Everything "works" in that there are no stack traces or errors in the logs 
> and the user is redirected correctly; they are just not logged in after going 
> through the login process.  If I do this on a new Galaxy install the user is 
> created so that part is working. What else do I have to do to get Galaxy to 
> recognize the user is logged in?  I have been pouring over the code for the 
> other authenticators and I don't see what I am missing.
> 
> Once I get the login sorted I can start preparing a proper PR.  
> 
> Cheers,
> Keith
> 
> 1.  
> https://github.com/lappsgrid-incubator/Galaxy/blob/bd220d6aefaf74316f1f859837aea9afeb536fba/lib/galaxy/webapps/galaxy/controllers/saml.py#L146
>  
> <https://github.com/lappsgrid-incubator/Galaxy/blob/bd220d6aefaf74316f1f859837aea9afeb536fba/lib/galaxy/webapps/galaxy/controllers/saml.py#L146>
> 
>> On Mar 7, 2020, at 5:54 PM, Youssef GHORBAL > <mailto:youssef.ghor...@pasteur.fr>> wrote:
>> 
>> Hi Keith,
>> 
>>  Last time I checked (before OIDC support in Galaxy) it was not possible 
>> to easly hack around the user/auth flow in Galaxy to introduce SAML 
>> semantics.
>>  As you said, since OIDC integration I hope that addding SAML would be 
>> more doable ("easy" and "SAML" in the same sentence are not permitted)
>>  At Pasteur we reverted back to Galaxy builtin auth even for internal 
>> users.
>> 
>>  In any case, if you got something working I'd be more than happy to 
>> test it.
>> 
>> Youssef Ghorbal
>> Institut Pasteur
>> ---
>>> On 7 Mar 2020, at 21:40, Keith Suderman >> <mailto:suder...@cs.vassar.edu>> wrote:
>>> 
>>> We are (finally) adding SAML authentication to our Galaxy instance so users 
>>> can authenticate with EduGain (EU) or InCommon (US).  While doing some 
>>> searching I found an article that says, "Galaxy supports LDAP, SAML and now 
>>> OIDC (see https://galaxyproject.org/authnz/ 
>>> <https://galaxyproject.org/authnz/> <https://galaxyproject.org/authnz/ 
>>> <https://galaxyproject.org/authnz/>>)." [1]  Looking at the release_20.01 
>>> code I see lots has ch

[galaxy-dev] Re: SAML Authentication

[Keith Suderman]

I have the SAML login implementation 99% complete.  However, after 
authenticating with an IdP and being redirected back to Galaxy the user is 
still not "logged in"...  The relevant code is on GitHub [1], but basically in 
the callback handler I do:

user = trans.get_or_create_remote_user(user_email)
trans.handle_user_login(user)

Everything "works" in that there are no stack traces or errors in the logs and 
the user is redirected correctly; they are just not logged in after going 
through the login process.  If I do this on a new Galaxy install the user is 
created so that part is working. What else do I have to do to get Galaxy to 
recognize the user is logged in?  I have been pouring over the code for the 
other authenticators and I don't see what I am missing.

Once I get the login sorted I can start preparing a proper PR.  

Cheers,
Keith

1.  
https://github.com/lappsgrid-incubator/Galaxy/blob/bd220d6aefaf74316f1f859837aea9afeb536fba/lib/galaxy/webapps/galaxy/controllers/saml.py#L146

> On Mar 7, 2020, at 5:54 PM, Youssef GHORBAL  
> wrote:
> 
> Hi Keith,
> 
>   Last time I checked (before OIDC support in Galaxy) it was not possible 
> to easly hack around the user/auth flow in Galaxy to introduce SAML semantics.
>   As you said, since OIDC integration I hope that addding SAML would be 
> more doable ("easy" and "SAML" in the same sentence are not permitted)
>   At Pasteur we reverted back to Galaxy builtin auth even for internal 
> users.
> 
>   In any case, if you got something working I'd be more than happy to 
> test it.
> 
> Youssef Ghorbal
> Institut Pasteur
> ---
>> On 7 Mar 2020, at 21:40, Keith Suderman  wrote:
>> 
>> We are (finally) adding SAML authentication to our Galaxy instance so users 
>> can authenticate with EduGain (EU) or InCommon (US).  While doing some 
>> searching I found an article that says, "Galaxy supports LDAP, SAML and now 
>> OIDC (see https://galaxyproject.org/authnz/ 
>> <https://galaxyproject.org/authnz/>)." [1]  Looking at the release_20.01 
>> code I see lots has changed in respect to authorization (we are still using 
>> 19.01 so I am glad I looked), but I don't see anything related to SAML.  I 
>> am not finding anything in the documentation either.  I am missing something 
>> or am I looking in the wrong place(s)? I was expecting (hoping) to see a 
>> saml.py in /lib/galaxy/auth/providers/ or something similar.  
>> 
>> I do have Shibboleth authentication working using Apache and mod_shib in 
>> front of Galaxy, but that requires everyone to be able to authenticate and 
>> we would still like to allow anonymous access. Ideally users that don't have 
>> InCommon/EduGain access would still be able to create local accounts as 
>> well.  
>> 
>> Before I start hacking around trying to create an authentication plugin for 
>> Galaxy I wanted to check to see if there was any other ongoing work in the 
>> same area.  There is an old issue on GitHub [2], but I can't find anything 
>> else related. I see the OIDC stuff uses social-auth, which does include a 
>> SAML backend. Social-auth just uses the python3-saml package and I already 
>> have a simple client that uses python3-saml for authentication so I am 
>> hoping a saml plugin will be "Relatively Easy" (TM).
>> 
>> Thanks,
>> Keith
>> 
>> 1. https://academic.oup.com/bioinformatics/article/36/1/1/5514039
>> 2. https://github.com/galaxyproject/galaxy/issues/3146
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> %(web_page_url)s
>> 
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] SAML Authentication

[Keith Suderman]

We are (finally) adding SAML authentication to our Galaxy instance so users can 
authenticate with EduGain (EU) or InCommon (US).  While doing some searching I 
found an article that says, "Galaxy supports LDAP, SAML and now OIDC (see 
https://galaxyproject.org/authnz/ )." [1]  
Looking at the release_20.01 code I see lots has changed in respect to 
authorization (we are still using 19.01 so I am glad I looked), but I don't see 
anything related to SAML.  I am not finding anything in the documentation 
either.  I am missing something or am I looking in the wrong place(s)? I was 
expecting (hoping) to see a saml.py in /lib/galaxy/auth/providers/ or something 
similar.  

I do have Shibboleth authentication working using Apache and mod_shib in front 
of Galaxy, but that requires everyone to be able to authenticate and we would 
still like to allow anonymous access. Ideally users that don't have 
InCommon/EduGain access would still be able to create local accounts as well.  

Before I start hacking around trying to create an authentication plugin for 
Galaxy I wanted to check to see if there was any other ongoing work in the same 
area.  There is an old issue on GitHub [2], but I can't find anything else 
related. I see the OIDC stuff uses social-auth, which does include a SAML 
backend. Social-auth just uses the python3-saml package and I already have a 
simple client that uses python3-saml for authentication so I am hoping a saml 
plugin will be "Relatively Easy" (TM).

Thanks,
Keith

1. https://academic.oup.com/bioinformatics/article/36/1/1/5514039
2. https://github.com/galaxyproject/galaxy/issues/3146

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Error

[Keith Suderman]

Hi Pietro,

I suspect the package is broken.  The anaconda.org server is available, but the 
libgcc-ng file is not where the package expects it to be.

I followed your suggestion and it worked... sort of. At least it exposed 
another error, actually the same problem but with numpy... So I applied the 
same solution and that worked... sort of.  Now the error is:

> Traceback (most recent call last):
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/_vendor/auxlib/entity.py",
>  line 411, in __get__
> val = instance.__dict__[self.name]
> KeyError: 'size'
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/exceptions.py",
>  line 1043, in __call__
> return func(*args, **kwargs)
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/cli/main.py",
>  line 84, in _main
> exit_code = do_call(args, p)
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/cli/conda_argparse.py",
>  line 82, in do_call
> exit_code = getattr(module, func_name)(args, parser)
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/cli/main_create.py",
>  line 37, in execute
> install(args, parser, 'create')
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/cli/install.py",
>  line 304, in install
> handle_txn(unlink_link_transaction, prefix, args, newenv)
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/cli/install.py",
>  line 320, in handle_txn
> unlink_link_transaction.print_transaction_summary()
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/core/link.py",
>  line 877, in print_transaction_summary
> change_report_str = self._change_report_str(change_report)
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/core/link.py",
>  line 926, in _change_report_str
> size = prec.size
>   File 
> "/home/galaxy/galaxy/database/dependencies/_conda/lib/python3.6/site-packages/conda/_vendor/auxlib/entity.py",
>  line 417, in __get__
> raise AttributeError("A value for {0} has not been 
> set".format(self.name))
> AttributeError: A value for size has not been set

The full Galaxy log is attached.

Cheers,
Keith

> On Jun 26, 2019, at 7:31 AM, Pietro Mandreoli 
>  wrote:
> 
> Hi,
> probably the package is broken see 
> https://anaconda.org/conda-forge/libgcc-ng/files .
> Try to install the previous version manually, activating conda and then the 
> conda environment of the tool.
> 
> commands:
> source _conda/bin/activate to activate CONDA
> cd _conda/envs 
> 
> source activate  
> 
> then conda install  -c 
> 
> Best regards, 
> 
> PM
> 
> Il 25/06/19 22:59, Nancy Ide   ha scritto: 
>> 
>> I get this error when trying to run "Plot confusion matrix, precision, 
>> recall and ROC and AUC curves":
>> 
>> 
>>> CondaError: CondaHTTPError: HTTP 404 NOT FOUND for url 
>>> >>  
>>> >
>> 
>> 
>> as the file libgcc-ng-7.3.0-hdf63c60_0.tar.bz2 does not exist in 
>> https://conda.anaconda.org/conda-forge/linux-64/ 
>> 
>> 
>> 
>>  
>> 
>> 
>> Please help!
>> 
>> Thanks
>> -
>> 
>> Nancy Ide
>> Professor of Computer Science
>> 
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, New York 12604-0520 
>> USA
>> 
>> tel: (+1 845) 437 5988
>> fax: (+1 845) 437 7498
>> email: i...@cs.vassar.edu
>> http://www.cs.vassar.edu/~ide
>> 
>> -
>> 
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> %(web_page_url)s
>> 
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/
>> 
>> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  %(web_page_url)s
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
Starting server in PID 14373.
serving on 0.0.0.0:8000 view at http://127.0.0.1:8000
73.16.113.171 - - [27/Jun/2

Re: [galaxy-dev] [External] Copying files to FTP upload directory [resolved]

[Keith Suderman]

It turns out I had fat-fingered the key name in the config and had specified 
ftp_upoload_dir, once I noticed the typo and changed it to ftp_upload_dir 
everything works as expected.  What threw me was that the "Choose FTP file" 
button appeared in the upload dialog even though the ftp_upload_dir was not 
specified.  Looking at the log files I see 403 errors for GET /api/remote_files 
and when I try that URL in a browser I get some JSON with the error message, 
"The configuration of this Galaxy instance does not allow upload from FTP 
directories."

I am not familiar enough with the Javascript client, but it looks like the 
logic used to determine whether or not to show the "Choose FTP file" button 
only checks if ftp_upload_site has been specified and not if ftp_upload_dir has 
been defined.

- Keith

> On Jan 25, 2019, at 7:29 PM, Keith Suderman  wrote:
> 
> Hi Martin,
> 
> Yes, Galaxy has permission to read the files.  I have only tested on my own 
> workstation, so Galaxy runs as me, all the files/directories are owned by me, 
> etc. but I will try on our dev server and see if it makes any difference.  
> Would using an admin account be a problem?  
> 
> And just to clarify, I am assuming I can do the following to get a file 
> available in Galaxy (where the ftp_upload_dir is /home/galaxy/database/ftp):
> 
> $> mkdir -p /home/galaxy/database/ftp/suderman@cs.vassar 
> <mailto:home/galaxy/database/ftp/suderman@cs.vassar>.edu
> $> echo "Hello world." > /home/galaxy/database/ftp/suder...@cs.vassar.edu 
> <mailto:home/galaxy/database/ftp/suder...@cs.vassar.edu>/Hello.txt
> $> chown -R galaxy:galaxy /home/galaxy/database/ftp
> 
> Thanks,
> Keith
> 
>> On Jan 25, 2019, at 5:07 PM, Martin Čech > <mailto:emulato...@gmail.com>> wrote:
>> 
>> Hi Keith,
>> 
>> your message is not bouncing, it shows here: 
>> http://dev.list.galaxyproject.org/ <http://dev.list.galaxyproject.org/>
>> Your setup seems correct to me, does Galaxy have permission to read files in 
>> the given folders?
>> 
>> Also Galaxy will auto-create folders if set the `user_library_import_dir` to 
>> be the same as ftp and use `#user_library_import_dir_auto_creation: false`
>> 
>> M.
>> 
>> On Fri, Jan 25, 2019 at 4:58 PM Keith Suderman > <mailto:suder...@cs.vassar.edu>> wrote:
>> I have been creating directories for the users, but putting files in (for 
>> example) /home/galaxy/database/ftp/u...@example.com 
>> <mailto:home/galaxy/database/ftp/u...@example.com> does not make them 
>> available for the u...@example.com <mailto:u...@example.com> user in Galaxy.
>> 
>> Also for the list admins... I never did see my message appear on the list 
>> and the message Cicada replied to appears to be from 
>> galaxy-dev-boun...@lists.galaxyproject.org 
>> <mailto:galaxy-dev-boun...@lists.galaxyproject.org>.  Any ideas why my 
>> message(s) appear to be bouncing.  I am pretty sure I am sending from the 
>> correct address.
>> 
>> Cheers,
>> Keith
>> 
>>> On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw >> <mailto:hbro...@iu.edu>> wrote:
>>> 
>>> I think you have to create the user's directories under 
>>> /home/galaxy/database/ftp. I don't think Galaxy will create them if they 
>>> are not there.
>>> Cicada
>>> 
>>> 
>>> From: galaxy-dev >> <mailto:galaxy-dev-boun...@lists.galaxyproject.org>> on behalf of Keith 
>>> Suderman mailto:suder...@cs.vassar.edu>>
>>> Sent: Friday, January 25, 2019 4:01 PM
>>> To: Galaxy Dev List
>>> Subject: [External] [galaxy-dev] Copying files to FTP upload directory
>> 
>>> 
>>> 
>>> Hello,
>>> 
>>> I am trying to (ab)use the FTP file upload functionality to get data from 
>>> an external service into Galaxy.  From my reading of the docs I simply need 
>>> to configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini 
>>> (haven't migrated to yaml yet) and Galaxy will expect to find 
>>> subdirectories named after the user's account (email) in the ftp_upload_dir.
>>> 
>>> I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our 
>>> case) and the "Choose FTP File" button is now available in the Upload 
>>> dialog, however regardless of where I put files, or what I name them, 
>>> Galaxy always reports that "Your FTP directory does not contain any files." 
>>>  For example, i

Re: [galaxy-dev] [External] Copying files to FTP upload directory

[Keith Suderman]

Hi Martin,

Yes, Galaxy has permission to read the files.  I have only tested on my own 
workstation, so Galaxy runs as me, all the files/directories are owned by me, 
etc. but I will try on our dev server and see if it makes any difference.  
Would using an admin account be a problem?  

And just to clarify, I am assuming I can do the following to get a file 
available in Galaxy (where the ftp_upload_dir is /home/galaxy/database/ftp):

$> mkdir -p /home/galaxy/database/ftp/suderman@cs.vassar 
<mailto:home/galaxy/database/ftp/suderman@cs.vassar>.edu
$> echo "Hello world." > /home/galaxy/database/ftp/suder...@cs.vassar.edu 
<mailto:home/galaxy/database/ftp/suder...@cs.vassar.edu>/Hello.txt
$> chown -R galaxy:galaxy /home/galaxy/database/ftp

Thanks,
Keith

> On Jan 25, 2019, at 5:07 PM, Martin Čech  wrote:
> 
> Hi Keith,
> 
> your message is not bouncing, it shows here: 
> http://dev.list.galaxyproject.org/ <http://dev.list.galaxyproject.org/>
> Your setup seems correct to me, does Galaxy have permission to read files in 
> the given folders?
> 
> Also Galaxy will auto-create folders if set the `user_library_import_dir` to 
> be the same as ftp and use `#user_library_import_dir_auto_creation: false`
> 
> M.
> 
> On Fri, Jan 25, 2019 at 4:58 PM Keith Suderman  <mailto:suder...@cs.vassar.edu>> wrote:
> I have been creating directories for the users, but putting files in (for 
> example) /home/galaxy/database/ftp/u...@example.com 
> <mailto:home/galaxy/database/ftp/u...@example.com> does not make them 
> available for the u...@example.com <mailto:u...@example.com> user in Galaxy.
> 
> Also for the list admins... I never did see my message appear on the list and 
> the message Cicada replied to appears to be from 
> galaxy-dev-boun...@lists.galaxyproject.org 
> <mailto:galaxy-dev-boun...@lists.galaxyproject.org>.  Any ideas why my 
> message(s) appear to be bouncing.  I am pretty sure I am sending from the 
> correct address.
> 
> Cheers,
> Keith
> 
>> On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw > <mailto:hbro...@iu.edu>> wrote:
>> 
>> I think you have to create the user's directories under 
>> /home/galaxy/database/ftp. I don't think Galaxy will create them if they are 
>> not there.
>> Cicada
>> 
>> 
>> From: galaxy-dev > <mailto:galaxy-dev-boun...@lists.galaxyproject.org>> on behalf of Keith 
>> Suderman mailto:suder...@cs.vassar.edu>>
>> Sent: Friday, January 25, 2019 4:01 PM
>> To: Galaxy Dev List
>> Subject: [External] [galaxy-dev] Copying files to FTP upload directory
> 
>> 
>> 
>> Hello,
>> 
>> I am trying to (ab)use the FTP file upload functionality to get data from an 
>> external service into Galaxy.  From my reading of the docs I simply need to 
>> configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't 
>> migrated to yaml yet) and Galaxy will expect to find subdirectories named 
>> after the user's account (email) in the ftp_upload_dir.
>> 
>> I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) 
>> and the "Choose FTP File" button is now available in the Upload dialog, 
>> however regardless of where I put files, or what I name them, Galaxy always 
>> reports that "Your FTP directory does not contain any files."  For example, 
>> if I have the user account u...@example.org <mailto:u...@example.org> I 
>> would expect to be able to place files in 
>> /home/galaxy/database/ftp/u...@example.org 
>> <mailto:home/galaxy/database/ftp/u...@example.org> and they will be 
>> available in the Upload dialog for the u...@example.org 
>> <mailto:u...@example.org> user.  However, this is not working; am I missing 
>> something?
>> 
>> Thanks,
>> Keith
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/ <http://galaxyproject.org/search/>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/ <http://galaxyproject.org/search/>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] [External] Copying files to FTP upload directory

[Keith Suderman]

I have been creating directories for the users, but putting files in (for 
example) /home/galaxy/database/ftp/u...@example.com 
<mailto:home/galaxy/database/ftp/u...@example.com> does not make them available 
for the u...@example.com <mailto:u...@example.com> user in Galaxy.

Also for the list admins... I never did see my message appear on the list and 
the message Cicada replied to appears to be from 
galaxy-dev-boun...@lists.galaxyproject.org 
<mailto:galaxy-dev-boun...@lists.galaxyproject.org>.  Any ideas why my 
message(s) appear to be bouncing.  I am pretty sure I am sending from the 
correct address.

Cheers,
Keith

> On Jan 25, 2019, at 4:36 PM, Dennis, H. E. Cicada Brokaw  
> wrote:
> 
> I think you have to create the user's directories under 
> /home/galaxy/database/ftp. I don't think Galaxy will create them if they are 
> not there.
> Cicada
> 
> ____
> From: galaxy-dev  on behalf of 
> Keith Suderman 
> Sent: Friday, January 25, 2019 4:01 PM
> To: Galaxy Dev List
> Subject: [External] [galaxy-dev] Copying files to FTP upload directory
> 
> Hello,
> 
> I am trying to (ab)use the FTP file upload functionality to get data from an 
> external service into Galaxy.  From my reading of the docs I simply need to 
> configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't 
> migrated to yaml yet) and Galaxy will expect to find subdirectories named 
> after the user's account (email) in the ftp_upload_dir.
> 
> I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) 
> and the "Choose FTP File" button is now available in the Upload dialog, 
> however regardless of where I put files, or what I name them, Galaxy always 
> reports that "Your FTP directory does not contain any files."  For example, 
> if I have the user account u...@example.org I would expect to be able to 
> place files in /home/galaxy/database/ftp/u...@example.org and they will be 
> available in the Upload dialog for the u...@example.org user.  However, this 
> is not working; am I missing something?
> 
> Thanks,
> Keith
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Copying files to FTP upload directory

[Keith Suderman]

Hello,

I am trying to (ab)use the FTP file upload functionality to get data from an 
external service into Galaxy.  From my reading of the docs I simply need to 
configure the ftp_upload_dir and ftp_upload_site in my galaxy.ini (haven't 
migrated to yaml yet) and Galaxy will expect to find subdirectories named after 
the user's account (email) in the ftp_upload_dir.

I have configured the ftp_upload_dir (/home/galaxy/database/ftp in our case) 
and the "Choose FTP File" button is now available in the Upload dialog, however 
regardless of where I put files, or what I name them, Galaxy always reports 
that "Your FTP directory does not contain any files."  For example, if I have 
the user account u...@example.org I would expect to be able to place files in 
/home/galaxy/database/ftp/u...@example.org and they will be available in the 
Upload dialog for the u...@example.org user.  However, this is not working; am 
I missing something?

Thanks,
Keith


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] good practice for Galaxy admin and updates ?

[Keith Suderman]

Hi Julie,

This is (almost) exactly what we do.  The only difference I see is due to 
modifications we have made to Galaxy itself we do not use the Galaxy master as 
our production server.  Instead we have our own separate production and 
development branches.  To upgrade we:

1. Pull the lastest master from upstream into the master branch
2. Merge upstream/master into our development branch
3. Test and fix until we are happy with everything.
4. Merge the development branch into our production branch.

 HTH,
Keith

> On Sep 3, 2018, at 7:19 AM, julie dubois  wrote:
> 
> Hi Galaxy Community,
> 
> I manage a Galaxy local instance on a standalone station since 6 years.
> Regarding the numerous troubleshoots I have encountered (and the very
> long time I spent) during these years with Galaxy updates trying to
> minimize impact for my users, trying in the same time to keep safe
> our customization of this instance. I'm wondering about the best solution
> to manage my instance and to test my own dev and also Galaxy updates
> before pushing them on
> my production server.
> 
> So I want to test a new way and I'd appreciate community feedback on this idea
> or let me know if other practices could be more appropriate in my situation.
> So my basic idea is:
> 
> First install a new Galaxy from scratch, not by cloning the stable branch of
> Galaxy but by forking it. And use the master branch of this fork as
> production server.
> Then to manage an update of the Galaxy stable, my idea is to make a
> test branch on my
> fork, then pull Galaxy update on this test branch and run this test instance 
> as
> a dev server in another station to be sure that update will not override our
> customization or any crucial tools/functionality for my users.
> Finally when I'll be sure this update is safe, I'll merge the test branch with
> the master branch of my forked Galaxy.
> 
> 
> Is anyone already acting in this way? Is it a good (the best) way to manage
> local instance of Galaxy? Have you other good practices?
> 
> Thanks a lot for your help.
> 
> Julie
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/

--
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu






signature.asc
Description: Message signed with OpenPGP
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Save data from visualization plugin

[Keith Suderman]

Hi Martin

Thank you!  I think:

>  formData.append('files_0|file_data', new Blob([cont], {type:'text/plain'}), 
> $('#editor').attr('file-name'));

was the line of code I was looking for.

Cheers,
Keith


> On Apr 23, 2018, at 5:49 PM, Martin Demko <325...@mail.muni.cz> wrote:
> 
> Dear Keith,
> 
> I don't know the best way but this work for us. It is used in cooperation with
> ACE editor (written in JS):
> 
>   
>var hist_id = null;
> 
>$(function() {
>var address = '${prefix}/history/current_history_json';
>$.get(address, function(resp) {
>if (resp && resp.id)
>hist_id = resp.id;
>});
>$('#save_btn').on('click', function() {
>var cont = ace.edit('editor').getValue();# or any other
> data, you need to save in new file, here, the data are textual
>var dInputs = {
>dbkey: '?',
>file_type: 'auto',
>'files_0|type': 'upload_dataset',
>'files_0|space_to_tab': null,
>'files_0|to_posix_lines': 'Yes'
>};
> 
>var formData = new FormData();
>formData.append('tool_id', 'upload1');
>formData.append('history_id', hist_id);
>formData.append('inputs', JSON.stringify(dInputs));
>formData.append('files_0|file_data', new Blob([cont], {type:
> 'text/plain'}), $('#editor').attr('file-name'));
> 
>$.ajax({
>url: '${prefix}/api/tools',
>data: formData,
>processData: false,
>contentType: false,
>type: 'POST',
>success: function (resp) {
># this part is there to refresh the history panel
> otherwise it seems not to turn green
>window.setTimeout(function() {
>
> window.parent.$('#history-refresh-button').trigger('click');
>}, 3000);
>}
>});
>});
>});
>   
> 
> I hope it helps you.
> 
> Best regards,
> Martin Demko
> 
> 
> Keith Suderman  wrote on Mon, 23 Apr 2018 15:38:19
> -0400:
>> My Google-Fu is failing me...
>> 
>> We have a visualization plugin that can also serve as an editor to modify
>> the data it is visualizing.  What is the best way to save the data from the
>> visualizer back to Galaxy?
>> 
>> I seem to recall reading (somewhere, sometime) that the REST API could not
>> do this and the "best" solution is to invoke the upload tool to upload data
>> into the current history.  I have used the upload tool from the Python and
>> Blend4J APIs, but I can not find a JavaScript example for invoking the
>> upload tool.
>> 
>> So two questions:
>> 
>> 1. Is the upload tool (still) the recommended way to put data into the
>> current history via API calls?
>> 2. How do I invoke the upload tool from JavaScript?
>> 
>> Thanks,
>> Keith
>> 
>> --
>> Keith Suderman
>> Research Associate
>> Department of Computer Science
>> Vassar College, Poughkeepsie NY
>> suder...@cs.vassar.edu
>> 
>> 
>> 
>> 

--
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu






signature.asc
Description: Message signed with OpenPGP
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Save data from visualization plugin

[Keith Suderman]

My Google-Fu is failing me...

We have a visualization plugin that can also serve as an editor to modify the 
data it is visualizing.  What is the best way to save the data from the 
visualizer back to Galaxy?

I seem to recall reading (somewhere, sometime) that the REST API could not do 
this and the "best" solution is to invoke the upload tool to upload data into 
the current history.  I have used the upload tool from the Python and Blend4J 
APIs, but I can not find a JavaScript example for invoking the upload tool.

So two questions:

1. Is the upload tool (still) the recommended way to put data into the current 
history via API calls?
2. How do I invoke the upload tool from JavaScript?

Thanks,
Keith

------
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu






signature.asc
Description: Message signed with OpenPGP
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Watching the tools directory and dynamic reloading

[Keith Suderman]

Thanks Marius, it was the release notes I had seen and what I was thinking 
about.

- Keith

> On Aug 20, 2017, at 2:48 AM, Marius van den Beek  
> wrote:
> 
> Hi Keith,
> 
> Galaxy has been able to load tools after installation for quite some time,
> however that did not work for production setups that are using multiple
> job handler threads.
> 
> We did indeed refine the tool loading to work for production servers starting 
> with release 16.10
> (it's mentioned in the release notes 
> https://docs.galaxyproject.org/en/master/releases/16.10_announce.html 
> <https://docs.galaxyproject.org/en/master/releases/16.10_announce.html>).
> Aspects of this have been refined up to release 17.05, which should be the 
> most reliable concerning this feature.
> 
> No special options need to be set to use this, it should hopefully just work.
> Technically we watch the various shed_tool_conf.xml / tool_conf.xml files as 
> well as the individual tool wrappers for changes
> and trigger a reload, which we speed up by caching the tool definitions.
> One thing we can't do at the moment is detect if a macro file that is 
> included in a tool has changes,
> so in this case you'd still need to restart the galaxy instance unfortunately.
> 
> There is another option (watch_tools, see 
> https://github.com/galaxyproject/galaxy/blob/release_17.05/config/galaxy.ini.sample#L257
>  
> <https://github.com/galaxyproject/galaxy/blob/release_17.05/config/galaxy.ini.sample#L257>)
>  in the galaxy.ini that allows you to watch a directory of tools 
> (i.e no installation from the toolshed or modification of a tool_conf,xml 
> file necessary) that you can activate.
> If you want to use this feature you need to indicate in your tool_conf.xml 
> file which directory you would like to monitor.
> The syntax is like this:
> 
> 
> 
> (The recursive="true" is not necessary if you want to monitor a flat 
> directory).
> 
> Best,
> Marius
> 
> 
> On 20 August 2017 at 04:13, Keith Suderman  <mailto:suder...@cs.vassar.edu>> wrote:
> I thought I had read that newer versions of Galaxy were able to watch the 
> tools directory and update the tools menu when any tool wrappers were added, 
> deleted, or modified; basically the "Admin -> Reload a tool's configuration" 
> without the administrator actually having to do anything.  However, my Google 
> searches are not turning up anything?  Am I over-remembering something I had 
> read or are my Google skills failing me?
> 
> This is for an upcoming course that will involve students developing tools 
> and pipelines. We want the teaching assistants to be able to update their 
> master Galaxy instance with new tools without a lot of button clicks or a 
> server restart.
> 
> Thanks,
> Keith
> 
> --
> Keith Suderman
> Research Associate
> Department of Computer Science
> Vassar College, Poughkeepsie NY
> suder...@cs.vassar.edu <mailto:suder...@cs.vassar.edu>
> 
> 
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/ <https://lists.galaxyproject.org/>
> 
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/ <http://galaxyproject.org/search/>
> 

--
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Watching the tools directory and dynamic reloading

[Keith Suderman]

I thought I had read that newer versions of Galaxy were able to watch the tools 
directory and update the tools menu when any tool wrappers were added, deleted, 
or modified; basically the "Admin -> Reload a tool's configuration" without the 
administrator actually having to do anything.  However, my Google searches are 
not turning up anything?  Am I over-remembering something I had read or are my 
Google skills failing me?

This is for an upcoming course that will involve students developing tools and 
pipelines. We want the teaching assistants to be able to update their master 
Galaxy instance with new tools without a lot of button clicks or a server 
restart.

Thanks,
Keith

------
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] SAML Authentication ala OpenID

[Keith Suderman]

Sadly, providing an authenticator in galaxy/auth/providers does not look like 
it will not work for SAML authentication since the authenticator is invoked 
after the user has already been prompted for a username and password.  Rather 
it looks like I will need to use the approach used for OpenID authentication 
and described at https://wiki.galaxyproject.org/Develop/Authentication

Does the Galaxy team have any plans to modularize third-party authenticators 
such as OpenID, SAML, OAuth2 etc?  If we do implement this it would be nice to 
do it in such a way that would help Galaxy move towards that goal.

- Keith

> On Feb 27, 2017, at 12:55 PM, Björn Grüning  wrote:
> 
> This all seems correct and I own you so many beers if you get this
> implemented! :)
> 
> Am 27.02.2017 um 18:34 schrieb Keith Suderman:
>> Is anyone using SAML for authentication?  There is a feature request for
>> SAML support on GitHub (
>> https://github.com/galaxyproject/galaxy/issues/3146), but it doesn't
>> look like any work has been done.
>> 
>> We need to add the ability for our users to authenticate with a SAML
>> identity provider (IdP), in particular with InCommon/EduRoam/EduCause et
>> al.  Looking at the documentation there appears to be "the old way"
>> (configuring Apache/Nginx to do the authentication) and "the new way"
>> where Galaxy handles the authentication.  Is it correct to assume that
>> to use the new way I should implement an external authenticator,
>> something like galaxy/auth/providers/saml.py and then some sort of magic
>> to get a config/auth_conf.xml working?  
>> 
>> I just want to make sure I am heading down the correct path before
>> investing too much time.
>> 
>> - Keith
>> 
>> Keith Suderman
>> Research Associate
>> Department of Computer Science
>> Vassar College, Poughkeepsie NY
>> suder...@cs.vassar.edu <mailto:suder...@cs.vassar.edu>
>> 
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 

--
Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] SAML Authentication

[Keith Suderman]

Is anyone using SAML for authentication?  There is a feature request for SAML 
support on GitHub ( https://github.com/galaxyproject/galaxy/issues/3146), but 
it doesn't look like any work has been done.

We need to add the ability for our users to authenticate with a SAML identity 
provider (IdP), in particular with InCommon/EduRoam/EduCause et al.  Looking at 
the documentation there appears to be "the old way" (configuring Apache/Nginx 
to do the authentication) and "the new way" where Galaxy handles the 
authentication.  Is it correct to assume that to use the new way I should 
implement an external authenticator, something like 
galaxy/auth/providers/saml.py and then some sort of magic to get a 
config/auth_conf.xml working?  

I just want to make sure I am heading down the correct path before investing 
too much time.

- Keith

Keith Suderman
Research Associate
Department of Computer Science
Vassar College, Poughkeepsie NY
suder...@cs.vassar.edu



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Galaxy Startup Failing

[Keith Suderman]

Greetings,

I am trying to upgrade our Galaxy instance to the latest stable code from 
master and now our instance won't start using my /etc/init.d/galaxy script.  
That is:

> sudo service galaxy start

no longer works. It says it does (i.e. prints "[OK]"), but Galaxy isn't running 
and nothing is written to the log. I am using the init.d script from simison 
[1], which is basically the same as the one posted to the Galaxy wiki [2], both 
of which are quite old.

Galaxy does start with:

> sudo su galaxy
> ./run.sh --daemon

so I have left Galaxy running in a screen session for now, but that is less 
than ideal.

I suspect something to do with the .venv as `./run.sh --skip-venv` also fails 
and I don't recall Galaxy being that finicky about a venv being active. Does 
anyone know how I can get the /etc/init.d/galaxy script (i.e. 
start-stop-daemon) to activate the venv for paster.py?

Cheers,
Keith

1. http://simison.com/brian/GalaxyAdmin_notes.html
2. 
https://wiki.galaxyproject.org/Events/GCC2012/TrainingDay/WS5?highlight=%28init.d%29

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Dataset Collections status

[Keith Suderman]

Greetings,

I started pulling Galaxy code from the dev branch a few months ago to take 
advantage of the (then just emerging) dataset collections feature.  However, it 
is not clear to me from the latest release notes if the data collections are 
now fully merged into master, or if I should continue to use the code in the 
dev branch to take advantage of bleeding edge code.  I would like to move back 
to the master branch as soon as feasible.

When running workflows over dataset collections I will frequently see errors 
like:

/bin/sh: 1: 
/home/galaxy/galaxy_old/database/job_working_directory/001/1216/galaxy_1216.sh: 
Text file busy

Which, from what I can tell, occurs when one process is trying to modify/delete 
a file open in another process.  While the error seems to be repeatable, it 
also seems random as the errors do not occur in the same places if I run the 
workflow multiple times.

Given that I am working from the dev branch I don't want to open/raise issues 
on features still in development.  But if this is unexpected then I can do some 
more investigating and file a proper bug report.

Cheers,
Keith

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Adding categories to Main

[Keith Suderman]

Awesome! 

One concern we have is lumping all NLP tools into a single category; are 
sub-categories (and sub-sub-categories) possible?  However, we can start with 
this and see what we need later.

Cheers,
Keith

On Aug 7, 2015, at 12:25 PM, Martin Čech  wrote:

> Dear Keith,
> 
> I have created 'NLP' category at https://testtoolshed.g2.bx.psu.edu for you 
> to test out the Tool Shed environment.
> 
> Unless your tools are proprietary I strongly recommend using the Main Tool 
> Shed at https://toolshed.g2.bx.psu.edu for their publishing and distribution. 
> Let me know when you are ready with the tools and I will create the category 
> for you in there.
> 
> We happily welcome every tool contribution to Galaxy! Thank you for it.
> 
> Martin, Galaxy Team
> 
> 
> On Fri, Aug 7, 2015 at 12:56 PM Björn Grüning  
> wrote:
> Hi Keith,
> 
> the Galaxy team can add new categories to the Tool Shed very easily and
> such a mail is exactly the way to go. As soon as the Galaxy teams has
> added the category you can use tools like planemo to upload all your
> tools at once.
> 
> https://github.com/galaxyproject/planemo
> 
> Looking forward to see NLP tools in Galaxy and the Main TS! Awesome!
> Bjoern
> 
> 
> > Dear Galaxy Team,
> >
> > A colleague would like to upload some NLP (Natural Language
> > Processing) tools to the Galaxy Test/Main tools sheds, but we are
> > unclear what "categories" to use for the tools.  I see that the
> > Main/Test tools sheds have a category for "Text Manipulation", but
> > that does not seem appropriate for NLP tools.  Is it possible to have
> > new categories added to the tool shed(s)?  If so, what is the
> > process?
> >
> > I am just starting to investigate setting up our own local tool shed
> > and I am coming across mentions of repository capsules and exporting
> > tool sheds.  Would it be preferable to install the NLP tools to a
> > local tool shed and then export a repository/capsule to be imported
> > to the Test/Main tools sheds?  What happens if our tool shed uses a
> > disjoint set of categories than Test/Main?
> >
> > Cheers, Keith
> >
> > -- Research Associate Department of
> > Computer Science Vassar College Poughkeepsie, NY
> >
> >
> > ___ Please
> > keep all replies on the list by using "reply all" in your mail
> > client.  To manage your subscriptions to this and other Galaxy lists,
> > please use the interface at: https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> > http://galaxyproject.org/search/mailinglists/
> >

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Adding categories to Main

[Keith Suderman]

Dear Galaxy Team,

A colleague would like to upload some NLP (Natural Language Processing) tools 
to the Galaxy Test/Main tools sheds, but we are unclear what "categories" to 
use for the tools.  I see that the Main/Test tools sheds have a category for 
"Text Manipulation", but that does not seem appropriate for NLP tools.  Is it 
possible to have new categories added to the tool shed(s)?  If so, what is the 
process?

I am just starting to investigate setting up our own local tool shed and I am 
coming across mentions of repository capsules and exporting tool sheds.  Would 
it be preferable to install the NLP tools to a local tool shed and then export 
a repository/capsule to be imported to the Test/Main tools sheds?  What happens 
if our tool shed uses a disjoint set of categories than Test/Main?

Cheers,
Keith

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Multi-level datatype conversions

[Keith Suderman]

Hi David,

On Jun 5, 2015, at 2:10 PM, David Kelly  wrote:

> Hi Keith,
> 
> I am able to manually convert from a->b and then separately b->c, but not 
> directly from a->c. I'm still thinking about how to approach this as well, 
> but I'll share my thoughts so far.

Does Galaxy automatically do the conversions for you in the workflow editor?  
For example, if you have a tool A that produces format 'a' and another tool B 
that consumes format 'b' you should be able to drag the "noodle" from A to B 
and Galaxy will do an implicit format conversion.  If this is working for you 
then someone from the Galaxy team will have to jump in.  If Galaxy is not doing 
implicit format conversions then it likely means your datatype(s) are not fully 
defined.  I didn't get the implicit format conversion working properly until I 
implemented the format sniffers.

> In our use case, we have a collection of climate tools we'd like to use. The 
> tools are written by different groups, so most use their own file formats. 
> Ideally we would like users to be able to upload their data in any of these 
> file formats and use all tools without having to explicitly convert their 
> data.
> 
> Let's say there are 10 climate datatypes. My understanding is that in order 
> to achieve this kind of interoperability, I'd have to write 90 converters.
> 
> If Galaxy could chain converters together, I could write 1 converter per 
> datatype and chain them together like this:
> a -> b -> c -> d -> e -> f -> g -> h -> i -> j -> a
> 
> Another scenario I've been thinking about is picking one data type to be the 
> 'standard'. Let's say it's 'a' in this example. You could write 18 converters:
> a->b, a->c, a->d,...
> b->a, c->a, d->a,...

This is what we do. We have a standard "pivot format" then converters from the 
other formats to/from the pivot format.

Cheers,
Keith

> 
> Then suppose you wanted to plug a 'd' file into a 'c' tool. You could first 
> manually convert d->a, then use the automatic conversion for a->c. Not fully 
> automatic, but not too bad.
> 
> On Thu, Jun 4, 2015 at 5:20 PM, Keith Suderman  wrote:
> Hi David,
> 
> We just got it working for the single conversions, that is, getting Galaxy to 
> automatically convert from a -> b when needed.  Are you able to do the 
> conversion, i.e a -> b and b -> c individually?
> 
> We are just coming to a similar use case and I'm definitely interested in how 
> you make out, even if I can't be much help.
> 
> Cheers,
> Keith
> 
> On Jun 3, 2015, at 4:15 PM, David Kelly  wrote:
> 
> > Hello,
> >
> > Suppose I have three custom datatypes defined: a, b, and c
> >
> > I have a datatype converter that converts from a -> b
> > I have a datatype converter that converts from b -> c
> >
> > Can I then send 'a' files as input into tools that accept type 'c'?
> >
> > This doesn't seem to work in my test case, but I thought it might be worth 
> > asking to see if anyone has tried this before. Thanks!
> >
> > Regards,
> > David
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >  https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >  http://galaxyproject.org/search/mailinglists/
> 
> --
> Research Associate
> Department of Computer Science
> Vassar College
> Poughkeepsie, NY
> 
> 
> 

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Problem in Json datatype?

[Keith Suderman]

Greetings,

I was adding datatype classes to galaxy.datatype.text.py for our custom JSON 
formats and I noticed what I think is a problem with the Json datatype class, 
in particular the _looks_like_json method.  The basic algorithm is (in pseudo 
code):

if the_file_is_small_enough:
try to load the the file as json
return True on success, False otherwise
else
while True
find first non-blank line
if line starts with ‘{‘ or ‘[‘:
return True
else
return False

However,  JSON is typically compressed by stripping whitespace (including 
newlines), particularly when it is fetched from a web service. This means that 
the first call to readLine is going to load the entire file, which defeats the 
purpose of checking the file size in the first place.

I would submit a pull request with our fix, but since I am not a Python 
programmer I thought I would simply post the code here for review.

class Json(Text):
# Unchanged bits of the class have been omitted.

# Add a function that reads a single character skipping over whitespace.
def read(self, fileHandle):
ch = fileHandle.read(1)
while ch.isspace():
ch = fileHandle.read(1)
return ch

# Only the “else-part” has changed
def _looks_like_json(self, filename):
# Pattern used by SequenceSplitLocations
if os.path.getsize(filename) < 5:
# If the file is small enough - don't guess just check.
try:
json.load(open(filename, "r"))
return True
except Exception:
return False
else:
with open(filename, "r") as fh:
ch = self.read(fh)
return ch == '{' or ch == '['

We then use the read() method to sniff out our JSON formats. Once we can sniff 
a header subclasses only need to specify the header.

class Lapps( Json ):
header = ‘’’{“discriminator”:”http://vocab.lappsgrid.org'''
def sniff(self, filename)
with open(filename) as fh:
for c in header:
if c != self.read(fh)
return False
return True
class Lif ( Lapps ):
header = 
‘’’{“discriminator”:”http://vocab.lappsgrid.org/ns/media/jsonld#lif”'''
class Gate( Lapps ):
header = 
‘’’{“discriminator”:”http://vocab.lappsgrid.org/ns/media/gate”'''
…

Regardless of how the JSON is rendered (pretty printed or not) we can match the 
“magic” strings used to identify our formats.

Being new to Python I don’t know how expensive it is to read a file one 
character at a time, but it has to be cheaper than potentially reading the 
entire file.

Cheers,
Keith

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Data Visualization

[Keith Suderman]

Greetings,

I am looking for information, tips, and pointers on writing a component for 
visualizing text data in Galaxy.  In particular I would like to use Brat 
(http://brat.nlplab.org/embed.html) which can be embedded in web pages/apps. 
The “Implementation Details” section on the Learn/Visualization page 
(https://wiki.galaxyproject.org/Learn/Visualization#Implementation_Details) 
hints at exactly what I want to do:
- use AJAX to fetch JSON data.
- use a Javascript client to draw the data

I can do both of the above steps, I am just not sure how to wire Brat into 
Galaxy. All of the documentation I can find about visualization covers 
installing existing Genome visualization tools and nothing about how to adapt 
an existing Javascript client to work in Galaxy.  I am hoping someone can 
sketch out the basic steps I need to accomplish before I spend too much time 
grepping through the Galaxy code base.

Cheers,
Keith

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Data visualization

[Keith Suderman]

Hi Björn,

Brilliant!  Thanks.  That is exactly what I was looking for.

Cheers,
Keith


On May 24, 2015, at 4:49 AM, Björn Grüning  wrote:

> Hi Keith,
> 
> I'm a fan of learning by eyample, so please find attached the BRAT VIS :)
> 
> Put the folder into your ./config/plugins/visualizations/ folder and
> restart Galaxy. You should see a new visualisation in every text
> datatype in your Galaxy history. (We should change this later to JSON).
> 
> Have a look at the nlp_brat.xml file in config and the nlp_brat.mako
> file. This is everything what you need to adopt or change if you want to
> create a VIS.
> 
> Hope this helps,
> Bjoern
> 
> Am 24.05.2015 um 02:45 schrieb Keith Suderman:
>> Greetings,
>> 
>> I am looking for information, tips, and pointers on writing a component for 
>> visualizing text data in Galaxy.  In particular I would like to use Brat 
>> (http://brat.nlplab.org/embed.html) which can be embedded in web pages/apps. 
>> The “Implementation Details” section on the Learn/Visualization page 
>> (https://wiki.galaxyproject.org/Learn/Visualization#Implementation_Details) 
>> hints at exactly what I want to do:
>> - use AJAX to fetch JSON data.
>> - use a Javascript client to draw the data
>> 
>> I can do both of the above steps, I am just not sure how to wire Brat into 
>> Galaxy. All of the documentation I can find about visualization covers 
>> installing existing Genome visualization tools and nothing about how to 
>> adapt an existing Javascript client to work in Galaxy.  I am hoping someone 
>> can sketch out the basic steps I need to accomplish before I spend too much 
>> time grepping through the Galaxy code base.
>> 
>> Cheers,
>> Keith
>> 
>> --
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>> 
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>> 
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>> 
> 

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Data visualization

[Keith Suderman]

Greetings,

I am looking for information, tips, and pointers on writing a component for 
visualizing text data in Galaxy.  In particular I would like to use Brat 
(http://brat.nlplab.org/embed.html) which can be embedded in web pages/apps. 
The “Implementation Details” section on the Learn/Visualization page 
(https://wiki.galaxyproject.org/Learn/Visualization#Implementation_Details) 
hints at exactly what I want to do:
- use AJAX to fetch JSON data.
- use a Javascript client to draw the data

I can do both of the above steps, I am just not sure how to wire Brat into 
Galaxy. All of the documentation I can find about visualization covers 
installing existing Genome visualization tools and nothing about how to adapt 
an existing Javascript client to work in Galaxy.  I am hoping someone can 
sketch out the basic steps I need to accomplish before I spend too much time 
grepping through the Galaxy code base.

Cheers,
Keith

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy for Natural Language Processing

[Keith Suderman]

 this, but as last resort probably an
>>> option.
>> 
>> Out of curiosity, what exactly should I avoid; making calls to the Galaxy 
>> REST/API from inside a tool, using blend4j, or populating a data library 
>> from inside a tool?  I can see myself doing all three in the near future.
> 
> * making calls to the Galaxy REST/API from inside a tool
> 
> Think big! Your tools will run in large cluster environments, one job 
> schedulers and Cloud-Infrstructures. You don't know if you job is allowed to 
> connect to your Galaxy instance - security wise. Also you need to 
> authenticate, more issues 
> 
> Ciao,
> Bjoern 
> 
>> Cheers,
>> Keith
>> 
>>> 
>>> Ciao,
>>> Bjoern
>>> 
>>>> Cheers,
>>>> Keith
>>>> 
>>>> REFERENCES
>>>> 
>>>> 1. https://wiki.galaxyproject.org/Learn/API#Collections
>>>> 2. 
>>>> https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_of_Output_datasets_cannot_be_determined_until_tool_run
>>>> 
>>>>> 
>>>>>>> Oh yes this is supported out of the box!
>>>>>>> See here for a small documentation:
>>>>>>> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox#supported-filetypes
>>>>>>> 
>>>>>>> Here is a example of how you can write your own datatypes:
>>>>>>> 
>>>>>>> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox/datatypes
>>>>>> 
>>>>>> I feel like I must be missing the obvious.  Here is the relevant section 
>>>>>> of my datatypes_conf.xml (you can see the full file at 
>>>>>> https://github.com/oanc/Galaxy/blob/master/config/datatypes_conf.xml)
>>>>>> 
>>>>>> >>>>> display_in_upload="True">
>>>>>> 
>>>>>>   >>>>> target_datatype="gate"/>
>>>>>> 
>>>>>> >>>>> mimetype="application/xml" display_in_upload="true">
>>>>>> 
>>>>>>   >>>>> target_datatype="lif"/>
>>>>>> 
>>>>>> 
>>>>>> Is there anything I need to do beyond defining the datatypes for 
>>>>>> implicit conversions to take place?
>>>>> 
>>>>> I guess you need to place your converters under
>>>>> https://github.com/oanc/Galaxy/tree/master/lib/galaxy/datatypes/converters/
>>>>> 
>>>>> And get rid of 'convert.' in your datatypes_conf.xml at least if you are
>>>>> not using the TS.
>>>>> 
>>>>> Hope this helps you a little bit more,
>>>>> Bjoern
>>>>> 
>>>>>> Thanks,
>>>>>> Keith Suderman
>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 4. OAuth 2.0 / OpenID Connect: 
>>>>>>>> 
>>>>>>>> I need to be able to fetch documents from data providers that require 
>>>>>>>> an OAuth 2.0 access token. Currently, I use a separate service to go 
>>>>>>>> through the OAuth authentication/authorization process and then have 
>>>>>>>> the user copy/paste their access token into a text field in Galaxy.   
>>>>>>>> Is there a way to perform the OAuth authentication dance required by 
>>>>>>>> the remote service inside Galaxy itself?   
>>>>>>> 
>>>>>>> I don't think so, but maybe someone else has an idea here.
>>>>>>> 
>>>>>>>> I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 
>>>>>>>> and OpenID Connect are on the radar, hopefully this use case is being 
>>>>>>>> considered as well.
>>>>>>>> 
>>>>>>>> I’m sure to have more questions after working through some 
>>>>>>>> visualization examples, but this should keep me busy for now.
>>>>>>> 
>>>>>>> Hope you are busy now :)
>>>>>>> Cheers and keep us up to date!
>>>>>>> Bjoern
>>>>>>> 
>>>>>>>> Sincerely,
>>>>>>>> Keith Suderman
>>>>>>>> 
>>>>>>>> REFERENCES
>>>>>>>> 
>>>>>>>> 1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Research Associate
>>>>>>>> Department of Computer Science
>>>>>>>> Vassar College
>>>>>>>> Poughkeepsie, NY
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ___
>>>>>>>> Please keep all replies on the list by using "reply all"
>>>>>>>> in your mail client.  To manage your subscriptions to this
>>>>>>>> and other Galaxy lists, please use the interface at:
>>>>>>>> https://lists.galaxyproject.org/
>>>>>>>> 
>>>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Research Associate
>>>>>> Department of Computer Science
>>>>>> Vassar College
>>>>>> Poughkeepsie, NY
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Research Associate
>>>> Department of Computer Science
>>>> Vassar College
>>>> Poughkeepsie, NY
>>>> 
>>>> 
>>> 
>> 
>> --
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>> 
>> 
> 

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy for Natural Language Processing

[Keith Suderman]

gt; Cheers,
>> Keith
>> 
>> REFERENCES
>> 
>> 1. https://wiki.galaxyproject.org/Learn/API#Collections
>> 2. 
>> https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files#Number_of_Output_datasets_cannot_be_determined_until_tool_run
>> 
>>> 
>>>>> Oh yes this is supported out of the box!
>>>>> See here for a small documentation:
>>>>> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox#supported-filetypes
>>>>> 
>>>>> Here is a example of how you can write your own datatypes:
>>>>> 
>>>>> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox/datatypes
>>>> 
>>>> I feel like I must be missing the obvious.  Here is the relevant section 
>>>> of my datatypes_conf.xml (you can see the full file at 
>>>> https://github.com/oanc/Galaxy/blob/master/config/datatypes_conf.xml)
>>>> 
>>>> >>> display_in_upload="True">
>>>>
>>>> 
>>>> >>> mimetype="application/xml" display_in_upload="true">
>>>>
>>>> 
>>>> 
>>>> Is there anything I need to do beyond defining the datatypes for implicit 
>>>> conversions to take place?
>>> 
>>> I guess you need to place your converters under
>>> https://github.com/oanc/Galaxy/tree/master/lib/galaxy/datatypes/converters/
>>> 
>>> And get rid of 'convert.' in your datatypes_conf.xml at least if you are
>>> not using the TS.
>>> 
>>> Hope this helps you a little bit more,
>>> Bjoern
>>> 
>>>> Thanks,
>>>> Keith Suderman
>>>> 
>>>>> 
>>>>>> 
>>>>>> 4. OAuth 2.0 / OpenID Connect: 
>>>>>> 
>>>>>> I need to be able to fetch documents from data providers that require an 
>>>>>> OAuth 2.0 access token. Currently, I use a separate service to go 
>>>>>> through the OAuth authentication/authorization process and then have the 
>>>>>> user copy/paste their access token into a text field in Galaxy.   
>>>>>> Is there a way to perform the OAuth authentication dance required by the 
>>>>>> remote service inside Galaxy itself?   
>>>>> 
>>>>> I don't think so, but maybe someone else has an idea here.
>>>>> 
>>>>>> I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 
>>>>>> and OpenID Connect are on the radar, hopefully this use case is being 
>>>>>> considered as well.
>>>>>> 
>>>>>> I’m sure to have more questions after working through some visualization 
>>>>>> examples, but this should keep me busy for now.
>>>>> 
>>>>> Hope you are busy now :)
>>>>> Cheers and keep us up to date!
>>>>> Bjoern
>>>>> 
>>>>>> Sincerely,
>>>>>> Keith Suderman
>>>>>> 
>>>>>> REFERENCES
>>>>>> 
>>>>>> 1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools
>>>>>> 
>>>>>> --
>>>>>> Research Associate
>>>>>> Department of Computer Science
>>>>>> Vassar College
>>>>>> Poughkeepsie, NY
>>>>>> 
>>>>>> 
>>>>>> ___
>>>>>> Please keep all replies on the list by using "reply all"
>>>>>> in your mail client.  To manage your subscriptions to this
>>>>>> and other Galaxy lists, please use the interface at:
>>>>>> https://lists.galaxyproject.org/
>>>>>> 
>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>> 
>>>>> 
>>>> 
>>>> --
>>>> Research Associate
>>>> Department of Computer Science
>>>> Vassar College
>>>> Poughkeepsie, NY
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> --
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>> 
>> 
> 

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy for Natural Language Processing

[Keith Suderman]

ou can write your own datatypes:
>>> 
>>> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox/datatypes
>> 
>> I feel like I must be missing the obvious.  Here is the relevant section of 
>> my datatypes_conf.xml (you can see the full file at 
>> https://github.com/oanc/Galaxy/blob/master/config/datatypes_conf.xml)
>> 
>> > display_in_upload="True">
>>  
>> 
>> > mimetype="application/xml" display_in_upload="true">
>>      
>> 
>> 
>> Is there anything I need to do beyond defining the datatypes for implicit 
>> conversions to take place?
> 
> I guess you need to place your converters under
> https://github.com/oanc/Galaxy/tree/master/lib/galaxy/datatypes/converters/
> 
> And get rid of 'convert.' in your datatypes_conf.xml at least if you are
> not using the TS.
> 
> Hope this helps you a little bit more,
> Bjoern
> 
>> Thanks,
>> Keith Suderman
>> 
>>> 
>>>> 
>>>> 4. OAuth 2.0 / OpenID Connect: 
>>>> 
>>>> I need to be able to fetch documents from data providers that require an 
>>>> OAuth 2.0 access token. Currently, I use a separate service to go 
>>>> through the OAuth authentication/authorization process and then have the 
>>>> user copy/paste their access token into a text field in Galaxy.   
>>>> Is there a way to perform the OAuth authentication dance required by the 
>>>> remote service inside Galaxy itself?   
>>> 
>>> I don't think so, but maybe someone else has an idea here.
>>> 
>>>> I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 and 
>>>> OpenID Connect are on the radar, hopefully this use case is being 
>>>> considered as well.
>>>> 
>>>> I’m sure to have more questions after working through some visualization 
>>>> examples, but this should keep me busy for now.
>>> 
>>> Hope you are busy now :)
>>> Cheers and keep us up to date!
>>> Bjoern
>>> 
>>>> Sincerely,
>>>> Keith Suderman
>>>> 
>>>> REFERENCES
>>>> 
>>>> 1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools
>>>> 
>>>> --
>>>> Research Associate
>>>> Department of Computer Science
>>>> Vassar College
>>>> Poughkeepsie, NY
>>>> 
>>>> 
>>>> ___
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>> https://lists.galaxyproject.org/
>>>> 
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/mailinglists/
>>>> 
>>> 
>> 
>> --
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>> 
>> 
>> 
> 

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy for Natural Language Processing

[Keith Suderman]

Hi Björn,

Thanks for the quick reply.

It might help to take a look at what I have so far. Our Galaxy server is 
running at http://grid.anc.org:8000  The front page contains a very basic 
tutorial (and I use the term loosely) with instructions for creating a simple 
workflow with our tools.  The "simple tutorial" assumes a good understanding of 
Galaxy.

It is also likely worthwhile pointing out that all of our "tools" invoke 
SOAP/REST web services running on other servers; nothing is "local".  I tend to 
use the terms "tool" and "service" interchangeably since all of our tools are 
web services.


> Can you tell me how your tool detects if it was processed before by an
> other tool? Metadata detection? Is this is different file type? If so
> you can define your own datatype(s). One of your tools can only consume
> the file types of an other tools output and so on.

We have two issues:
1. the format of the data
2. what the data contains.

Format:  This is easy; I have converters defined in the datatypes_conf.xml file 
and the workflow editor won't let me connect tools if the output/input formats 
don't match. 

Data Contents:  Just because Tool A produces format X and Tool B accepts format 
X does not mean the tools can be connected, I need to do deeper validation than 
simple format matching.  For example, if you drop several of the GATE tools 
into a workflow you can connect them in any order as they all accept and 
produce the same format,  however the tools must be connected in the correct 
order to produce something other than error messages.

There are two ways we do validation right now: each document contains metadata 
that describes the contents, and each service (tool) can produce metadata 
describing its input and output.  So we can check if two tools can be connected 
(the output of the first satisfies the input requirements of the second) and 
each tool checks at runtime that the input contains the necessary data.


>> A typical workflow might look something like:
>> 
>> a) Query Tool -> Server, find all documents that contain the word “cheese”
>> b) Server -> Here is the list of document IDs [ id1, id2, …, idn ]
>> c) WorkFlow -> for each id in the list do
>>  c1) Download document 
>>  c2 ) Work work work work…
>>  c3) Persist output
>> 
>> I can do all of the above except the most important bit; iterating…
> 
> Oh yes, this is simple. Just create one workflow that deals with one ID.
> This workflow you can run on multiple ids.

That is the question; how do I run the same workflow on multiple ids?  The 
server may return hundreds or thousands of id values so running the workflow 
manually for each id is not really an option.


> Oh yes this is supported out of the box!
> See here for a small documentation:
> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox#supported-filetypes
> 
> Here is a example of how you can write your own datatypes:
> 
> https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox/datatypes

I feel like I must be missing the obvious.  Here is the relevant section of my 
datatypes_conf.xml (you can see the full file at 
https://github.com/oanc/Galaxy/blob/master/config/datatypes_conf.xml)







 
Is there anything I need to do beyond defining the datatypes for implicit 
conversions to take place?

Thanks,
Keith Suderman

> 
>> 
>> 4. OAuth 2.0 / OpenID Connect: 
>> 
>> I need to be able to fetch documents from data providers that require an 
>> OAuth 2.0 access token. Currently, I use a separate service to go 
>> through the OAuth authentication/authorization process and then have the 
>> user copy/paste their access token into a text field in Galaxy.   
>> Is there a way to perform the OAuth authentication dance required by the 
>> remote service inside Galaxy itself?   
> 
> I don't think so, but maybe someone else has an idea here.
> 
>> I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 and 
>> OpenID Connect are on the radar, hopefully this use case is being considered 
>> as well.
>> 
>> I’m sure to have more questions after working through some visualization 
>> examples, but this should keep me busy for now.
> 
> Hope you are busy now :)
> Cheers and keep us up to date!
> Bjoern
> 
>> Sincerely,
>> Keith Suderman
>> 
>> REFERENCES
>> 
>> 1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools
>> 
>> --
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>> 
>> 
>> ___
>> Please

[galaxy-dev] Galaxy for Natural Language Processing

[Keith Suderman]

Hello,

Our group is investigating using Galaxy as a workflow engine for NLP (Natural 
Language Processing) tasks. I have installed a local Galaxy instance and 
created wrappers for the services we use and so far everything is working 
great.  I do have a few questions and they all fall under the “Advanced Topics” 
 section as defined at the end of the tutorial for creating a Histogram [1]

1. parameter validation: 

Many of our tools rely on additions made by previous tools in the workflow; for 
example, a tool that identifies noun phrases may require that the input has 
been run through a part of speech (POS) tagger, the POS tagger may require that 
the input has been run through a tokenizer, etc.  Our tools can do this 
validation, I am just looking for a way to wire this into Galaxy so a user can 
only connect tools in the workflow editor if this validation passes.

I have been looking looking at the code for 
lib/galaxy/tools/parameters/validation.py and I don’t see anything that I can 
(easily) bend to our use case.  What I was hoping for was something like:



validate.sh $input





I also see the tantalizing sentence, "Custom code execution at various time 
points of the workflow that allows a fine grained control over the execution 
process", but I can't find any examples of how this is done.


2. data repositories / data collections

I need to be able to process collections of data pulled from remote servers. I 
have been looking at DataManagers and data collections in Galaxy, but 
everything seems to assume the data is local to the server, or can be 
copied/uploaded to the server.  For practical and legal reasons beyond my pay 
grade this is not a solution in our case.  For example, an organization may be 
willing to allow our users to query their service for documents, run the 
documents through our workflow, and store the intermediate results; but they 
will not allow us to copy their data to another server verbatim.  There are 
possibilities for me to cache data, but the general use case is that I have to 
call an external service to fetch documents one at a time and then run the same 
workflow on each document.

Any suggestions on how to accomplish this in Galaxy?  I can do single 
documents, I just need to expand this to include collections of documents.  A 
typical workflow might look something like:

a) Query Tool -> Server, find all documents that contain the word “cheese”
b) Server -> Here is the list of document IDs [ id1, id2, …, idn ]
c) WorkFlow -> for each id in the list do
c1) Download document 
c2 ) Work work work work…
c3) Persist output

I can do all of the above except the most important bit; iterating…


3. format conversion:  

Is it possible for Galaxy to automatically convert between formats when 
designing a workflow?  I see the  tag, but that seems to change 
the output format of a tool based on the input (or some other condition) in the 
same tool; I need to be able to change the format based on the input 
requirements of the next tool in the workflow. For example, if Tool A produces 
format X, Tool B requires format Y,  and a converter from X to Y has been 
defined in the datatypes_conf.xml; I would like for Galaxy to implicitly insert 
the converter from X to Y when I drag the output noodle from Tool A to Tool B 
in the designer.  Is this possible? 


4. OAuth 2.0 / OpenID Connect: 

I need to be able to fetch documents from data providers that require an OAuth 
2.0 access token. Currently, I use a separate service to go through the OAuth 
authentication/authorization process and then have the user copy/paste their 
access token into a text field in Galaxy.   Is there a way to perform the OAuth 
authentication dance required by the remote service inside Galaxy itself?   
I’ve looked at the Trello site for Galaxy and see that both OAuth 2.0 and 
OpenID Connect are on the radar, hopefully this use case is being considered as 
well.


I’m sure to have more questions after working through some visualization 
examples, but this should keep me busy for now.

Sincerely,
Keith Suderman

REFERENCES

1. https://wiki.galaxyproject.org/Admin/Tools/AddingTools

--
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/