Re: [galaxy-dev] master_api_key currently doesn't benefit from a real admin's key?!

[Dooley, Damion]

Ah! Key idea is the ease of installation/maintenance of tools that need admin 
level api access.

I was hoping that the master api key enabled my program to do any of the api 
calls available (I'm using bioblend btw).  I am running workflows, uploading 
datasets from user history to data library, linking files into data library 
from server.  Alot of this work is done on behalf of a user - but I don't want 
to give such a user elevated access to the server.  So yes I can hard-code an 
admin api key into my galaxy tool for doing this work.

But I would much rather just count on a master/admin/uber api key that is 
defined in galaxy config for doing all this work.  This way, whenever a tool is 
installed from toolshed that needs admin functionality, it can just start 
working if the "uber" api key has been set up.  (No need to set up an admin 
user, copy key into config or wherever, reset galaxy etc).  That's what I was 
thinking the "master" api key did - I didn't realize it doesn't work with at 
least a handful of the API calls because it lacks an associated user object.  i 
had tried to put an admin api key into the master_api_key field but that failed 
on various needed api calls - it ignores the potential to use the admin user's 
user object.

By the way I do use the interacting user's non-privileged api key as well, it 
is convenient for delineating what that user does and does not have access to 
in terms of data libraries/datasets and workflows.  Its just the extra work I 
trigger on their behalf has to be admin level.

The tool itself is called the Data Versioning tool, and it is listing and 
caching/retrieving versioned fasta (or other types of) data and regenerating 
blast etc. databases as requested by users as an initial step to performing 
their other workflows.  It aims to quickly deal with datasets that are anywhere 
from megabytes to hundreds of gigabytes in size.

Hope that helps?

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: John Chilton [jmchil...@gmail.com]
Sent: Wednesday, November 19, 2014 6:39 AM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] master_api_key currently doesn't benefit from a real 
admin's key?!

If you already have an admin key in the database - that is more useful
than the master_api_key - I don't believe there is anything that the
master API key can do that the admin API key cannot (let me know if I
am wrong). I created master_api_key mostly just to bootstrap users and
API keys for new installations. So my question is what are you trying
to accomplish with a master API key if you already have an admin API
key? I would be happy to comment on the different ideas you mentioned
- but I want to understand what you are trying to accomplish.

-John

On Tue, Nov 18, 2014 at 8:35 PM, Dooley, Damion  wrote:
> I made the erroneous assumption that if I put my own admin user API key into 
> the galaxy configuration master_api_key field, it would accept that and run 
> all the api functions that needed a key connected to a user.  It took fair 
> bit of debugging to realize that the master_api_key field chops off all the 
> user info even if it is available (i.e. has no user object), thus yielding 
> numerous API errors for those things a user object is needed for.
>
> I can see a few dev solutions to this dilemma, and am wondering what people 
> think - and the result could get into a Trello feature card...
>
> a) allow master_api_key to be accompanied by a master_api_email; together 
> they trigger a user object to be associated that has the email address; and 
> this eliminates all the API errors one currently gets.  I like this solution 
> because it doesn't depend on the UI interface for managing user keys, i.e. 
> its rather permanent and secure.
>
> b) allow a api key called "admin_api_key" to be placed in the galaxy config 
> file.  This key has to be active as one user's api key (presumably power 
> user), so that all those api errors are avoided.
>
> c) have master_api_key just have a dummy user object included, with say 
> admin@localhost for an email address.
>
> Thoughts?
>
> Damion
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "repl

[galaxy-dev] master_api_key currently doesn't benefit from a real admin's key?!

[Dooley, Damion]

I made the erroneous assumption that if I put my own admin user API key into 
the galaxy configuration master_api_key field, it would accept that and run all 
the api functions that needed a key connected to a user.  It took fair bit of 
debugging to realize that the master_api_key field chops off all the user info 
even if it is available (i.e. has no user object), thus yielding numerous API 
errors for those things a user object is needed for.

I can see a few dev solutions to this dilemma, and am wondering what people 
think - and the result could get into a Trello feature card...

a) allow master_api_key to be accompanied by a master_api_email; together they 
trigger a user object to be associated that has the email address; and this 
eliminates all the API errors one currently gets.  I like this solution because 
it doesn't depend on the UI interface for managing user keys, i.e. its rather 
permanent and secure.

b) allow a api key called "admin_api_key" to be placed in the galaxy config 
file.  This key has to be active as one user's api key (presumably power user), 
so that all those api errors are avoided. 

c) have master_api_key just have a dummy user object included, with say 
admin@localhost for an email address.

Thoughts?

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Determining galaxy api path automatically from a tool via dynamic_options?

[Dooley, Damion]

Hi folks,

It was the universe_wsgi.ini 'prefix' parameter I was needing, though yes, 
http/https was an issue too.  Good to know that the 
trans.app.config.galaxy_infrastructure_url is in the works.

It turns out for now - thanks Eric Raschle - that the 
trans.request.application_url is the magic pill, it has the complete url.

self.api_url = trans.request.application_url

Cheers,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: John Chilton [jmchil...@gmail.com]
Sent: Wednesday, October 22, 2014 6:42 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Determining galaxy api path automatically from a tool 
via dynamic_options?

Well host is probably not enough - it doesn't seem like a good idea to
assume http right? There is the prefix and (trans.request.scheme) that
could be missing. Is trans.request.base what you want? Maybe
trans.request.path_url? It should be possible to recover that from
trans I guess - though I don't know exactly what parts of the request
you need to use.

So this isn't in stable yet - but there is going to be a sanctioned
way for Galaxy deployers to describe what the URL to their Galaxy
instance should be for internal consumption (stuff like docker
containers and Pulsar)
https://bitbucket.org/natefoo/galaxy-central/commits/4348618b4b8dd5f0c6d6cca595e1ff0affb6b3e1.
It will be settable in config/galaxy.ini and accessible via
trans.app.config.galaxy_infrastructure_url. There is a related
attribute trans.app.config.galaxy_infrastructure_url_set which if True
specifies that the deployer actually filled in a value and this is not
Galaxy's best guess at what URL should be used to access it. If it is
false - probably best to fallback to something like trans and the
request if you have it.

Hope this helps some.

-John
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Determining galaxy api path automatically from a tool via dynamic_options?

[Dooley, Damion]

After uploading my beta status tool to our test toolshed, I tried installing it 
on another of our galaxy servers but it wasn't able to find the api endpoint 
url.  Is there a way to automatically do this within tool forms?  I was hoping 
to avoid hardcoding this as part of the tool installation process.  I'd thought 
I had it automated:

In tool's form, a dynamic_options param 
 


was able to pass "trans" galaxy structure so I got as far as this python:
 
self.api_url = 'http://' + trans.request.host 

... but this yeilds 

http://acme.com/

rather than what's needed, the path to a particular installation:  

http://acme.com/galaxy/

I couldn't find a way to read the /galaxylab/ setting from the (slightly dated) 
universe_wsgi.ini config file - without say reading the file and parsing it.  
Is there an __app__.config.prefix or somesuch variable where this info is?

Thanks!

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Tool development: Is it possible to pass a user's id in dynamic_options() parameter?

[Dooley, Damion]

Hi Dan,

I'd like to get the user's galaxy user name directly - but the   that drives my dynamic_options select list has no access to any 
information in a galaxy's user session, right?  (Ultimately I'm trying to get 
the dynamic_options code to make galaxy api calls using the user's api key.)  I 
saw the GenomeSpace tool earlier - but got the impression that a user was 
entering their genome login in the tool form.  I was hoping just to get a 
user's galaxy id without any re-entry of it.

p.s. Its been really great getting advice from all of you; I hope to contribute 
likewise.

d.
 
Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: Daniel Blankenberg [d...@bx.psu.edu]
Sent: Friday, October 10, 2014 8:08 AM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Tool development: Is it possible to pass a user's id 
in dynamic_options() parameter?

Hi Damion,

Text fields are not currently able to be populated by dynamic_options, but in a 
dynamic_options select list you could access the user’s username directly. If 
you want the user to be able to override it, you could have a conditionally 
used empty text box that your tool defaults to using when it is populated.

An example that you might want to look at could be the GenomeSpace Export tool 
(tools/genomespace/genomespace_exporter.*), which queries the GenomeSpace API 
to get a list of available folders for export for a specific user using 
attributes of the current user’s preferences (GenomeSpace username and token). 
It generates a drill_down type select parameter, but the same can be done for a 
standard select list.


Thanks for using Galaxy,

Dan
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Tool development: Is it possible to pass a user's id in dynamic_options() parameter?

[Dooley, Damion]

This was asked a few years ago but no takers: Galaxy Tool Development: 
(http://dev.list.galaxyproject.org/Reading-user-id-before-the-execution-of-the-tool-td4490324.html)

Seems like this isn't possible, unless there's a way to plant user's name in a 
text field on the tool's form, so that a dynamic_options() call can refer to it?

Thx for insights,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Galaxy Bioblend option for importing dataset into a library?

[Dooley, Damion]

I see Galaxy API has a feature to import a history dataset into the library (in 
copy_hda_to_ldda() fn from GCC2013 training day course).  Is this available as 
well via Bioblend?  Latest docs don't seem to include this feature.  It would 
be the opposite of Bioblend's upload_dataset_from_library(history_id, 
lib_dataset_id) )

Objective is to get customized blast indexes into library that way for shared 
use.  Or have them actually exist outside galaxy, and linked in.
 
Regards,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Simple standard for API use of a global user/key that all loaded tools can draw upon?

[Dooley, Damion]

Ok, thanks for the on-target info and organizing the Trello card solution.

So the  construct lets us pass data that is available in the 
cheeta template context into a temporary file which the tool then has access 
to, thus avoiding exposing parameters in the command line execution which are 
visible when any user clicks on the "i" info link in history.  Sounds good for 
user-triggered tool case; I think this would work for my tool.

I left this note on the trello card... "Optional good.  A job specific key 
would be fine too, but it would need a powerful permission profile. John, I 
also see the security threat of passing the key to the command line un-encoded. 
 Is it possible for the command executable program to have access to one in the 
executable environment as long as its being executed by a 'galaxy' system 
privileged user?"  

I think that approach would detail the minimum stand-alone command line python 
(etc) code that could be run by system galaxy user to lookup the  
__app__.config.master_api_key , or to generate a tool-session key. (It would 
fail for other system users).  That would support API activity that isn't being 
called from the web interface and its cheeta templating system.  E.g use case 
of crontab scheduled events?

Cheers,

d.

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: John Chilton [jmchil...@gmail.com]
Sent: Thursday, October 02, 2014 9:42 AM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Simple standard for API use of a global user/key that 
all loaded tools can draw upon?

Okay - revisiting this because I was not nearly cautious enough with
my previous response. A couple large caveats -

...
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Simple standard for API use of a global user/key that all loaded tools can draw upon?

[Dooley, Damion]

Do any of you see the need to establish a way for tools to use a global 
(current site) API url/key combination?  From a security perspective I'm 
assuming this is basically ok since tools run under the galaxy user anyways 
(and can directly read and write files to their heart's content).  I understand 
that a particular user's API key could help limit what they have access to.  My 
tool however wants to do some prepatory form display and post-job work on 
behalf of the Galaxy system in addition to working with user-centric data; I 
presume others have similar use-case needs.

I think it would amount to having a standard GALAXY_TOOL_API_URL and 
GALAXY_TOOL_API_KEY managed in universe_wsgi.ini (left blank if not desired by 
sys admin).  The key would be generated by a galaxy admin using a real admin 
user id.  These would be available as python or shell environment variables 
when it came time to execute a tool's  etc. work. 
 

The wee other detail is that I'm using  
in my tool xml because there is API information I have to retrieve to set up 
the form with.  So the url and key need to be accessible in that execution 
environment too?!!

Am I missing the boat; alternately has this already been talked about?!

Thanks for advice ...

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

Well well, thanks very much for that reference!  I can see how your system to 
enable a workflow to process delta (diff) data (and merge the results back with 
a previous run's output) would greatly lighten the processing power for keeping 
results current.  Interesting choice of technologies too. 

Damion

Message: 5
Date: Fri, 5 Sep 2014 10:41:37 +
From: Pedersen Edvard 
To: "galaxy-dev@lists.bx.psu.edu" 
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data
Retrieval Tool

My PhD work may be of interest for this subject, although the primary focus has 
been on generating databases comprising the changes from a specific timeframe, 
and was not designed specifically for Galaxy. The similarities between my 
system and the system you are proposing are that it can generate a BLAST 
database from any date (that has been added to the system), as well as "diffs" 
between two dates, and supports FASTA, the Uniprot EMBL variant, full files 
(which does not give compression benefits) and several others. The system uses 
delta compression to make sure that non-updated fields do not take up extra 
space. It uses the Hadoop stack (HBase, HDFS and MapReduce) for parallelism in 
generating the databases (the blast database generation from FASTA files is not 
parallel).

You can find one of the publications here: 
 
http://bdps.cs.uit.no/papers/hibb13.pdf

I hope this can be of some use to you.

Regards,

Edvard Pedersen

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

Earlier on in the project analysis I was pursuing a Git solution because it 
seemed all its features would work with documents/code/files of any kind and so 
would be perfect for scientific reproducibility.  But its ability to 
efficiently archive non-documents is quite hit and miss, and the file size 
limitation becomes a major problem on top of that when it doesn't.
  
I will try to design the system so that handlers for different types of 
databases/files can be called into play to retrieve versioned content.

Its just that this fall I'll only have time to provide the handlers for fasta 
file archiving (the key-value database update approach enables fasta versioning 
and all the spinoff data from that.).
The next priority would be a handler for any type of file that needs to be 
replaced as a whole from version to version (one just needs hard drive space to 
accommodate this, since caching is pointless).
A git handler for well-behaved document content would also be a possibility.

Typo: I said yesterday "I wasn't going to leave that as just "fasta" datatype 
since it seems tools like makeblastdb don't allow anything else ..." - but I 
meant 'I WAS going to leave that as just "fasta"...'

d.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

Hi,

There have been a few comments about how general we could make the system for 
Galaxy use or just as a stand-alone command line driven tool.  So some notes 
below about what I could see it taking on.  Given the scale of the sequencing 
data problem, I'm sure the Galaxy community has important feedback on this.

I looked at git annex and it appears to me that though it promises to keep 
track of and synchronize network located files, it doesn't do versioning on 
them - am I wrong about that?

I also looked at https://code.google.com/p/leveldb/ , also a key value database 
which relies more heavily on indexes - but I see that though this is well-tuned 
to answering key queries, it isn't particularly good at storing and retrieving 
entire versions of a database that could be many gigabytes long, which is our 
mission.

It is relatively easy to generalize the simple keydb prototype I wrote so that 
it can handle any key-value database - including binary content and even binary 
key data, not just text (fasta sequences).  So a name change for the tool is a 
good idea. 

I want a versioning system that doesn't assume the incoming master file of 
key-value pairs is in the same order as it was on a previous import run.  I was 
afraid that any arbitrary change in the order of content on the source server 
could completely destroy the efficiency of a differential approach.  Git 
assumes its content is like a document - so it generates a slew of inserts and 
deletes, in fact provides no benefit, if the fasta entries are rearranged.  I 
tested helping git overcome this hurdle by converting the fasta content to 1 
line key/value fasta entries, and sorting them before git processing. That 
seemed to work for some smaller and larger nucleotide fasta files (tested 10m 
to 2gb) but failed when it came to processing protein fasta files; though 
possibly that was because of the fasta data line length.  That became another 
concern - thinking that git was failing because each line of the input file was 
many thousands of characters long.

So having done a "keydb" versioning engine that works and performs as well as 
git, I am definitely shying away from git now as unreliable on certain kinds of 
data.  The keydb approach is able to generate a version file at about the same 
speed that it takes to read the latest version of the same db, i.e. at 50mb/s 
on a standard hard drive.

An extension to keydb that enables it to take in just a list of adds or deletes 
or updates is desirable but that can come later.  More efficiency can be had by 
fine-tuning the updates so that one whole line of key-value doesn't have to 
replace the previous one but that's for later too.

A generalization note that the keydb approach works where the keys are a sparse 
array.  There's nothing stopping the keys from representing a 2D or 3D sparse 
array of data as long as the coordinates are coded uniquely into the one key 
list.

For those interested in versioning XML data there is an interesting summary of 
the challenges here:  
http://useless-factor.blogspot.ca/2008/01/matching-diffing-and-merging-xml.html 
.  It leaves me thinking that quick versioning of xml data could only be 
accomplished if it could somehow be converted into a key-value db, i.e. with 
each top level xml record identified by a unique key.

I could see breaking larger keydb databases up into smaller chunks for data 
retrieval and fast parallel processing - the usual approach being to separate 
the sorted key-value db out into files based on the first character or two in 
the key of each record.
  
Does this go along with people's expectations?

Cheers,

Damion 


From: Björn Grüning [bjoern.gruen...@gmail.com]
Sent: Monday, September 01, 2014 12:47 PM
To: Dooley, Damion; Björn Grüning; galaxy-dev@lists.bx.psu.edu
Cc: Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval 
Tool

Am 25.08.2014 um 18:05 schrieb Dooley, Damion:
> Ok, I'll be very happy to see what you've accomplished there.  I will read 
> through what you've done when I return from vacation in a week!
>
> A key need is to have whatever data comes in show up as linked data in one's 
> history to avoid server overhead;
a second objective was to not need to modify existing workflows - as
long as they could work of data in history that is typed appropriately.
  So your 'select type' solution sounds intreguing!
>
> And certainly interested in your use of git - I tried using git, using a 
> 1-line fasta data format, but git seemed to choke on protein fasta files?
> And did it run into performance problems with larger files?  That was my 
> experience.  I think I read its authors say that its upper limit was 15gb.

This is probably true for one large file. I'm storing the entire PDB in
git since a few years. One entry one file

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

About the datatype.  So you are thinking of a new datatype that applies to 
files that hold the versioned database contents (in this case a structured 
key-value fasta identifier/sequence pairs, right?)  Then the fasta archive 
versioning tool would take only files of that datatype for input. That sounds 
good.  

I was just going to have one folder (and its subfolders) in the data library 
that hold all the versioned databases to choose from.  So the versioned 
database tool would just populate its input list based on that subfolder tree.  
But ensuring that it lists only files of a certain datatype sounds beneficial.

Output in any case would be a fasta file that other tools are already 
expecting; I wasn't going to leave that as just "fasta" datatype since it seems 
tools like makeblastdb don't allow anything else as input from user history.

I'm hoping that a global (admin) API key can be used by the tool so that all 
users can get versioned data, but maybe that is a pipe dream.

Sure I'd like to see old patches!

d.

From: Bj?rn Gr?ning [bjoern.gruen...@gmail.com]
Sent: Saturday, August 23, 2014 12:17 AM
To: Dooley, Damion; galaxy-dev@lists.bx.psu.edu
Cc: Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval 
Tool

Hi Damion,

the idea sounds fantastic!
Can we go a step further and use a specific datatype that keeps entire
fasta files versioned and the user can choose which version he wants to
use, in any tool? Please have a look at my talk at GCC2012. Maybe you
are interested in the (old) patches. I would be very interested to
restart this old project.

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

I'll have a good look at the data manager approach - a quick glance tells me it 
is designed to fetch data and so be able to get fresh data on demand.  We'll 
need a 3rd party data import process that operates on its own pre-defined 
schedule so that each data file gets to have its own update schedule - 
weekly/monthly/yearly.  So it seems the galaxy API could accomplish this in 
talking with the data manager system?  I was going to look into the 
http://biomaj.genouest.org/ system as another possibility.

But is the data manager approach able to get data in such a way that it can 
just place it into a user's history as a linked file or set of files?  I.e. the 
solution we need has its master databases locally on the server, and will 
generate a cached version from the master file - without any networking off of 
the server.  

I'm back from vacation in a week so I'll get to check that out in more detail...

Cheers,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: Marius van den Beek [m.vandenb...@gmail.com]
Sent: Saturday, August 23, 2014 2:25 AM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu; Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval 
Tool

I also think that this a great idea, and as you described it I think it's 
feasible as a stand-alone galaxy tool.
Eventually you consider to implement this as a data manager 
(https://wiki.galaxyproject.org/Admin/Tools/DataManagers) ?


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

Ok, I'll be very happy to see what you've accomplished there.  I will read 
through what you've done when I return from vacation in a week!

A key need is to have whatever data comes in show up as linked data in one's 
history to avoid server overhead; a second objective was to not need to modify 
existing workflows - as long as they could work of data in history that is 
typed appropriately.  So your 'select type' solution sounds intreguing!

And certainly interested in your use of git - I tried using git, using a 1-line 
fasta data format, but git seemed to choke on protein fasta files?  And did it 
run into performance problems with larger files?  That was my experience.  I 
think I read its authors say that its upper limit was 15gb.  That was the 
motivation for writing a simple key-value master file diff system that seems to 
have the same I/O as git on smaller files, but more reliable for the fasta data 
case, and no problems with larger files - it outputs a new version in the same 
time it takes to read a master file.  It has drawbacks though - incoming data 
to compare master with must be sorted in 1 line fasta format first.

Thanks for your input; looking forward to your project writeup...

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: Björn Grüning [bjoern.gruen...@gmail.com]
Sent: Saturday, August 23, 2014 12:17 AM
To: Dooley, Damion; galaxy-dev@lists.bx.psu.edu
Cc: Hsiao, William
Subject: Re: [galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval 
Tool

Hi Damion,

the idea sounds fantastic!
Can we go a step further and use a specific datatype that keeps entire
fasta files versioned and the user can choose which version he wants to
use, in any tool? Please have a look at my talk at GCC2012. Maybe you
are interested in the (old) patches. I would be very interested to
restart this old project.

https://wiki.galaxyproject.org/Events/GCC2012/Abstracts#Keeping_Track_of_Life_Science_Data


Am 23.08.2014 um 03:24 schrieb Dooley, Damion:
> We are about to implement a fasta database (file) versioning system as a 
> Galaxy tool.  I wanted to get interested people's feedback first before we 
> roll ahead with the prototype implementation.  The versioning system aims to:
>
>- Simple makeblastdb or bowtie-build commands, or more specific workflows 
> that include dustmasker etc can be implemented.
>
> Does this sound attractive?

I think all of the use cases are covered by the old project mentioned
above. But I did not create a new tool I have created a new 'select
type' everyone can use in all tools. It was using git underneath (yeah,
I have the entire PDB in git and it is working fine :)) but we can
probably change git with a database if you like.

To answer your question: Yes, very attractive!

> We're hoping such a vision could handle Fasta databases from 12mb to e.g. 
> 200Gb (probably requires makeblastdb in parallel at that scale).
>
> Preliminary work suggests this project is doable via the Galaxy API without 
> galaxy customization - does that sound right?!

Yes, as long as the User has an API key.

Cheers,
Bjoern
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Concept for a Galaxy Versioned Fasta Data Retrieval Tool

[Dooley, Damion]

We are about to implement a fasta database (file) versioning system as a Galaxy 
tool.  I wanted to get interested people's feedback first before we roll ahead 
with the prototype implementation.  The versioning system aims to:

* Enable reproducible research: To recreate a search result at a certain point 
in time we need versioning so that search and mapping tools can look at 
sequence reference databases corresponding to a particular past date.  This 
recall can also explain the difference between what was known in the past vs. 
currently.

* Reduce hard drive space.  Some databases are too big to keep N copies around, 
e.g. 5 years of 16S, updated monthly, is say, 670Mb + 668Mb + 665Mb +   But 
occasionally we want to access past archives fairly quickly. 

* Integrate database versioning into Galaxy without adding a lot of complexity.

A bonus would be to enable the efficient sharing of version databases between 
computers/servers.

The solution we think would work centres around a "Versioned Data Retrieval" 
tool (draft image attached) that would work as follows: 

1) User selects from a list of databases provided by  "Shared Data > Data 
Libraries > Versioned Data". 
  - Each database has a master file that keeps its various versions as a list 
of time-stamped insert/delete transactions of key (fasta id) value (description 
& sequence) pairs.
  - Each master file is managed outside of galaxy via a triggered process on 
regular fasta file imports from data sources like NCBI or other niche sources.
  - We're expecting, due to the nature of fasta archived sequence updates, that 
our master file would only be about 1.1x the latest version in size 
(uncompressed).
2) User enters date / version id to retrieve (validated)
3) If a cached version of that database exists, it is linked into user's 
history.
4) Otherwise a new version of it is created, placed in cache, and linked into 
history.
  - The cached version itself then shows up as linked data under a Data Library 
> Versioned Data subfolder.
5) User can select preconfigured workflow(s) to execute on the selected 
retreived fasta file to regenerate any database products they need.
  - Workflow output data would also be cached in the same way the fasta data is 
- by linking the Galaxy Data Library to it. 
  - Workflow execution will be skipped if end data already exists in cache. 
  - Simple makeblastdb or bowtie-build commands, or more specific workflows 
that include dustmasker etc can be implemented.

Does this sound attractive?

We're hoping such a vision could handle Fasta databases from 12mb to e.g. 200Gb 
(probably requires makeblastdb in parallel at that scale).  

Preliminary work suggests this project is doable via the Galaxy API without 
galaxy customization - does that sound right?!

Feedback really appreciated!

Regards,

Damion Dooley  

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Galaxy Security Vulnerability

[Dooley, Damion]

Does this apply to all past galaxy installs?  I have an older galaxy site I've 
been wanting to phase out rather than upgrade. For now I'd like to use a patch 
but site version (parent: 7148:17d57db9a7c0 ) predates any of the tags.  I 
presume I'd have to just implement the patch by hand?

Regards,

Damion



Message: 7
Date: Thu, 31 Jul 2014 14:55:57 -0400
From: Nate Coraor 
To: Galaxy Development ,
galaxy-annou...@lists.bx.psu.edu
Subject: [galaxy-dev] Galaxy Security Vulnerability
Message-ID: 
Content-Type: text/plain; charset="us-ascii"

A security vulnerability was recently discovered by Inge Alexander Raknes that 
would allow a malicious person to execute arbitrary code on a Galaxy server. 
The vulnerability was in a method that uses Python "pickle" functionality to 
decode state information from tool forms. Because pickles can be used to 
instantiate arbitrary Python objects, tool states could be constructed to 
exploit this vulnerability.
...

 - pickle-2013.01.13.patch - This patch should apply cleanly (with offset/fuzz) 
to releases from 2013.01.13 up to 2013.08.12, and possibly older versions of 
Galaxy as well. Available at: 
https://depot.galaxyproject.org/patch/pickle-2013.01.13.patch


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Toolshed tool_dependency.xml git archive configure issue

[Dooley, Damion]

That was it! Yay!  I'll add a Trello card to have this info gem added to the 
docs.

Thanks,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

From: Aaron Petkau [aaron.pet...@gmail.com]
Sent: Friday, August 01, 2014 12:26 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Toolshed tool_dependency.xml git archive configure 
issue

Hmmm... you know what.  I tested this out and it looks like Galaxy expects the 
git repo to be named the same as the package name in your tool dependencies 
file ffp-phylogeny.  If you change the git command to:

git clone https://github.com/apetkau/ffp-3.19-custom.git ffp-phylogeny

I think it should work.

Aaron
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Toolshed tool_dependency.xml git archive configure issue

[Dooley, Damion]

A little more info:

ls -a 

#
ls -a
STDOUT
.
..
#

#
ls -a
STDERR

so I guess the .git archive is being unpackaged somewhere else!  After lunch 
I'll return to explore this mysterious territory.

d. 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Toolshed tool_dependency.xml git archive configure issue

[Dooley, Damion]

Sadly,

 ./configure --disable-gui 
--prefix=$INSTALL_DIR

yeilds install error:

 /bin/sh: ./configure: No such file or directory

But file must be there since previous git step is carried out ok:
...
#
git reset --hard 1819482db93bf0025c2588b5264cc0ec8c61029c
STDOUT
HEAD is now at 1819482 Added gitignore

#
git reset --hard 1819482db93bf0025c2588b5264cc0ec8c61029c
STDERR

#
./configure --disable-gui 
--prefix=/usr/local/galaxy/shared/tool_dependency/ffp-phylogeny/0.3.19_1819482db93bf0025c2588b5264cc0ec8c61029c/ddooley/ffp_phylogeny/4e213eee0fef
STDOUT

#
./configure --disable-gui 
--prefix=/usr/local/galaxy/shared/tool_dependency/ffp-phylogeny/0.3.19_1819482db93bf0025c2588b5264cc0ec8c61029c/ddooley/ffp_phylogeny/4e213eee0fef
STDERR
/bin/sh: ./configure: No such file or directory
#

argh!

d.

From: Aaron Petkau [aaron.pet...@gmail.com]
Sent: Friday, August 01, 2014 11:47 AM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Toolshed tool_dependency.xml git archive configure 
issue

Hey Damion,

You should be able to do ./configure --disable-gui ... to get it to work (that 
is, make sure the full path to the configure command is given).  At least I 
think that's probably what's going on.

Aaron


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Toolshed tool_dependency.xml git archive configure issue

[Dooley, Damion]

Hi folks,

Having a difficult time with coding the "configure" step in this new 
tool_dependency.xml file, namely that an error "/bin/sh: configure: command not 
found" results when I try to install my beta development tool from a local test 
toolshed (which is downloading and installing other tools just fine).  


git clone 
https://github.com/myuser/mygitstuff.git .
git reset --hard 
1819482db93bf0025c2588b5264cc0ec8c61029c 
configure --disable-gui 
--prefix=$INSTALL_DIR 


bin

$INSTALL_DIR/bin


$INSTALL_DIR/bin



Any hints about why this isn't being found or what the /bin/sh needs to get it 
going?
When I do a git reset --hard, the "configure" file is definitely there, with 
executable flag set.  Any other permission oddities?

I've also tried  , which yeilds same error.  
(Skipping that line and doing  at least 
finds and runs "make", though it errors out for lack of input data).

Regards,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Upload files to a data library - Invalid Paths - Solved

[Dooley, Damion]

Solved.  It was actually a file permissions thing on an ancestral folder.  I 
will add a trello card to suggest slightly better error reporting on this 
though.

Regards,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Upload files to a data library - Invalid Paths

[Dooley, Damion]

p.s. I do have "allow_library_path_paste = True" set in universe_wsgi.ini

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Upload files to a data library - Invalid Paths

[Dooley, Damion]

Hi folks,
In "Upload files to a data library" form, seems like whatever valid path I 
enter for linking to data on the server, even with "galaxy" as user and r 
permissions on the folder and files - I get the "invalid paths" message after 
submitting?  The basic form info:

   ***
   Invalid paths:
   /projects/gia/gia_analysis/blast/homologs_index_Spade_genomes/xml_reports/

   Upload option: Upload files from filesystem paths
   File Format: blastxml

   Paths to upload
   /projects/gia/gia_analysis/blast/homologs_index_Spade_genomes/xml_reports/

   Preserve directory structure?
   [yes or no]

   Copy data into Galaxy?
   Link to files without copying into Galaxy
   ***

Does anyone know a trick to this?  Would be nice to have more helpful error 
explanation here!

My galaxy: 5e605ed6069f+ (stable) release_2014.02.10 .  I think it is a pretty 
normal installation.

Thanks for any tips,

Regards,

Damion

Hsiao lab, BC Public Health Microbiology & Reference Laboratory, BC Centre for 
Disease Control
655 West 12th Avenue, Vancouver, British Columbia, V5Z 4R4 Canada
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Inform tool interface with data specific to selected dataset

[Dooley, Damion]

27; 
% self.lookup

fp_out.write("""\n\t\t\t%s""" % tdTags)
...

fp_out.write(HTML_REPORT_FOOTER_FILE)


Tool B:

To keep it simple this one just does a single output dataset but I can show a 
multiple output datset one, one for each set of query hits selected above if 
you want.  ' force_history_refresh="True" ' is supposed to refresh the history 
list after this executes all of its file writing but for some reason that 
doesn't seem to work on my galaxy.

 

select_subset.py $input $output1 $output1.id $__new_file_path__ 
$incl_excl $select




Include selection
Exclude selection













.. class:: infomark

**What it does**

This tool produces a tabular file with a subset of the lines in its input 
tabular file.



And the python:
'''
python select_subset.py $input $output $incl_excl $select
'''

def stop_err( msg ):
sys.stderr.write("%s\n" % msg)
sys.exit(1)

import sys

try:
input, output, incl_excl, select = sys.argv[1:]
except:
stop_err('you must provide the arguments input, output, incl_excl and 
select.')

lines = {}
try:
lines = dict([(int(num), '') for num in select.split(',')])
except:
stop_err('Did you remember to number the input dataset?')

include = bool(int(incl_excl))
if include:
print 'Including selected lines...'
else:
print 'Excluding selected lines...'

f_out = open(output, 'w')
with open(input) as f_in:
for line in f_in:
cols = line.split('\t')
try:
num = int(cols[-1])
except:
stop_err('Did you remember to number the input dataset?')
if include:
if num in lines:
f_out.write('\t'.join(cols[:-1])+'\n')
else:
if not num in lines:
    f_out.write('\t'.join(cols[:-1])+'\n')
f_in.close()
f_out.close()

print 'Done.'



From: Igor Topcin [igortop...@gmail.com]
Sent: Friday, May 09, 2014 1:05 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Inform tool interface with data specific to selected 
dataset

Hi Damion,
Would you mind sharing your approach with us all?
Thanks!
Igor

On May 9, 2014 1:51 PM, "Dooley, Damion" 
mailto:damion.doo...@bccdc.ca>> wrote:
Hello, Eric,

If the dynamic filters approach doesn't work out I can send you an approach 
that worked for me.  It involves creating a tool-generated html report that 
contains a form which provides selection choices; and the form is set to submit 
to a 2nd tool of your choice tool (it contains the necessary fields to prime 
the tool).  Not sure if it works on every breed of galaxy out there though.

d.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Inform tool interface with data specific to selected dataset

[Dooley, Damion]

Hello, Eric,

If the dynamic filters approach doesn't work out I can send you an approach 
that worked for me.  It involves creating a tool-generated html report that 
contains a form which provides selection choices; and the form is set to submit 
to a 2nd tool of your choice tool (it contains the necessary fields to prime 
the tool).  Not sure if it works on every breed of galaxy out there though.

d.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Restarting toolshed when "run_tool_Shed.sh --daemon stop" doesn't work

[Dooley, Damion]

I tried upgrading a local galaxy test site to the stable or default codebases.  
Most of galaxy seemed to work fine - and I can start and stop it.  But the 
local toolshed I had running under it seems now to be jammed in an error state:

   Internal Server Error

   Galaxy was unable to successfully complete your request

   URL: http:///toolshed/
   Module galaxy.web.framework.middleware.error:149 in __call__
   >>  app_iter = self.application(environ, sr_checker)
   Module paste.debug.prints:106 in __call__
   ...
   >>  
__M_writer(filters.html_escape(unicode(trans.app.shed_counter.unique_valid_tools
 )))
   AttributeError: 'ShedCounter' object has no attribute 'unique_valid_tools'


And there are a handful of admin tab links like 'Manage form definitions' that 
yield an error like:
  
   Error Traceback:
   View as:   Interactive  |  Text  |  XML (full)
   ⇝ AttributeError: 'FormsGrid' object has no attribute 'use_hide_message'
   URL: http://salk.bccdc.med.ubc.ca/galaxylab/forms/browse_form_definitions
   Module weberror.evalexception.middleware:364 in respond  view
   ...

Though I can stop/start the main galaxy site, I can't seem to stop/start the 
local toolshed site?  (I accidentally started galaxy (and probably the 
toolshed) in root in the past but I thought I'd recovered from that).

Any tips would be appreciated!

Regards,

Damion

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] FW: Correct toolshed tool config for a .loc file - RESOLVED

[Dooley, Damion]

Well Galaxians, I have to eat humble pie, kind of.  Mismatches between the 
tool's tool_data_table_conf.xml and the actual tabular data were responsible 
for the disappearing act, which log rather indirectly alluded to.  Something 
that didn't show itself on the dev server.  

I'll push for some greater reporting on this though, on the Trello board.  What 
I'd like to see is the "View data tables registry" report have its "Missing 
index file" column changed to something like "Status" and under that you get to 
see "Missing index file" or Bad file - Parse error, 23 lines" etc.

So my apologies to anyone who put some time into this.

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] FW: Correct toolshed tool config for a .loc file - addendum

[Dooley, Damion]

So I see this now in main.log:

galaxy.tools.data WARNING 2014-05-07 16:40:29,613 Line 3 in tool data table 
'blast_reporting_fields' is invalid (HINT: '' characters must be used to 
separate fields):
length  numeric int 1   1   1   
Alignment length

The thing is, tab characters ARE separating the fields.  Its just that some 
columns don't have values.  Now, this was working fine on the other galaxy 
install, updated to December 2013 changeset 11247 .  So has .loc processing 
changed since then?

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Correct toolshed tool config for a .loc file

[Dooley, Damion]

One further piece of the mystery I see now:

The "View Data tables Registry" shows my tool's loc files, but has odd stuff 
for "Tool data path" and "Missing index file" columns, for example:

   blast_reporting_fields [tab] 
/usr/local/galaxy/test/galaxy-dist/tool-data/salk.bccdc.med.ubc.ca/toolshed/repos/ddooley/blast_reporting/00d397a381f8/blast_reporting_fields.loc
 [tab] /usr/local/galaxy/test/galaxy-dist/tool-data

   ./database/tmp/tmp-toolshed-gmfcrDITxyK/blast_reporting_fields.loc [tab] 
./database/tmp/tmp-toolshed-gmfcrDITxyK [tab] missing

There is no galaxy-dist/database/tmp-toolshed-gmfcrDITxyK folder, if that's 
where its supposed to be? Is there some setting for database tmp folder cache 
for tools?

Thanks for tips ...

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Correct toolshed tool config for a .loc file

[Dooley, Damion]

I'll try to keep this short!  I'm wondering why on a fresh install of galaxy 
via a setup script into a python virtualenv, my custom toolshed tool's .loc 
files don't seem to be loading, i.e. the tool's tool_data_table_conf.xml file 
isn't loading.  All the other parts of the tool (and other toolshed tools) do 
run ok.  On the galaxy install where the tool was developed, it runs ok too, 
and I can't see any universe.wsgi.conf differences that would account for this 
though I'm sure the dev server has a number of other different config settings.

I thought I knew my way around this .loc stuff, but I'm flummoxed.  It is as 
though /usr/local/galaxy/test/galaxy-dist/tool-data/[toolshed domain etc 
etc]/00d397a381f8/tool_data_table_conf.xml isn't loading into Galaxy's tabular 
data list?!

The only other thing I can think of is that I stuffed the tool-data related loc 
files into a subfolder of the tool

   blast_reporting.py
   blast_reporting.xml
   tool_data_table_conf.xml.sample
   ...
   tool-data/
  blast_reporting_fields.loc.sample

But the resulting downloaded tool config in the galaxy-dist/tool-data/. 
/blast_reporting_fields.loc and tool_data_table_conf.xml files seem just fine!!!

Regards,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy-dev Digest, Vol 94, Issue 22

[Dooley, Damion]

Thanks for your input, John.  I see, so from what you say, anything installed 
via pip in the virtual environment then exists in the module/class include 
namespace within galaxy python command line tool code.  Good to know.

I'll note that this approach is compatible with toolshed tools (I'm testing it 
via one) because it relies on a galaxy admin managed .loc file to list the 
generic and customized reporting modules.  The (toolshed) reporting tool ships 
with a .loc file that doesn't reference any customized reports, e.g. 
blast_reporting_templates.loc.sample is

   #value   namedescription
   # Add list of html templates here 
   templates.html_reportStandard Report Provides report with sections 
and table sections

And then a galaxy admin adds an entry:
  # Customized templates listed below
  templates_custom.html_report  Customized report   Test customized report

Task for getting a customized report in place does require manual steps though: 
customized module, addition of the .pth file, and editing of the above .loc 
file.

Damion
--

Message: 4
Date: Wed, 16 Apr 2014 06:47:14 -0500
From: John Chilton 
To: "Dooley, Damion" 
Cc: "galaxy-dev@lists.bx.psu.edu" 
Subject: Re: [galaxy-dev] Can a tool upload a .loc file that is then
... RENAME: custom galaxy tool python modules
Message-ID:

Content-Type: text/plain; charset=UTF-8

If this is working it sounds reasonable. I think my approach might be
to setup Galaxy inside a virtualenv (as recommended here
https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer)
and just install your extra dependencies inside that virtualenv using
pip.

$ . /bin/activate
$ pip install dependency1
$ pip install dependency2
...

After that you should be able to just do a normal Python import.

I think you have another open question about the next steps of this -
but just to make it explicit - everything is going to work better in
"the Galaxy ecosystem" if you can avoid calling code from tools like
this and just use the sort of stock options. For instance - neither of
these approaches will work with the tool shed. Sometimes that is not
possible - I understand that - but I just thought I would put the
disclaimer out there.

-John
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Enabling an HTML report to prime a tool form (Dooley, Damion)

[Dooley, Damion]

For those who need it in the future, found the solution, so now the html 
report/form can trigger the given tool to run.  Turns out the tool_state value 
wasn't the problem.  It was the  value that was wrong.  Solution: the xml 
tool template that triggers rendering of the html form needs to pass the right 
dataset id to the form (which then passes it to the target tool) so one needs 
to have $selection_file.id passed:
 

blast_reporting.py $blastxml_file $tabular_file $html_file $out_format 
$selection_file $selection_file.id $html_template
...


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Enabling an HTML report to prime a tool form

[Dooley, Damion]

Hi Galaxians,

I've got a nice html report of blast results by query, and tabular subject 
result listing a handful of columns.  It was set up so that checkboxes by each 
row and a submit button for each subject sent form data to another tool we 
have.  I'd tricked the other tool's form into thinking it had all the right 
input for a refresh, including the tool_state string, borrowed from a previous 
view of that tool.  It worked great - it automatically submitted and generated 
a resulting dataset when the html report/form submitted to it - UNTIL ALAS it 
seems a change in history or session caused the tool form's tool_state string 
to fail and trigger an error on load (I guess it encodes some stuff that 
references the history panel?).  Looking at 
http://dev.list.galaxyproject.org/Feature-suggestion-quot-Re-Run-quot-button-td4132627.html
 it seemed like there might be a chance to encode a dummy tool_state?  

So, its a long shot, but is there a bare-bones tool_state code that doesn't 
care what's in the history for any given galaxy tool?  
Since this code is entirely in the html report rendering tool itself python 
doesn't seem to have access to galaxy system code.

Regards,

Damion


P.s. this is what the form html template looks like (borrowed mostly from the 
tool's html form itself):





   




...
gi|158343637|gb|EU057648.1|
99.55
442
2
0
...
etc.


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Can a tool upload a .loc file that is then ... RENAME: custom galaxy tool python modules

[Dooley, Damion]

Sorry about miss-named message.  I found a solution which makes me realize this 
might have been a bit of a python newbie 'duh' issue.

For those who might need something similar, solution is to add a .pth file to 
site-packages folder of python version that galaxy is running under, e.g. 
   /usr/lib/python2.6/site-packages/galaxy-custom-modules.pth
contains 1 line of text:
   /usr/local/galaxy/shared/python2.6_galaxy_custom_modules

And now if your code imports say:
   HTMLReportModule = __import__('custom_templates.html_report')

It will look in the python2.6_galaxy_custom_modules folder as well as the 
tool's folder.

Now is this the best solution?  I haven't seen any folder reserved for custom 
client python method code within a galaxy install, right?  I believe this 
approach lets any number of updates happen to the tool or to galaxy while 
preserving the customized code module(s).

Thanks for any input,

Damion

P.s If anyone wants an extra helpful detail on this dynamic replacement of a 
standard tool module with a custom one - this tip lets one work with any module 
folder depth "a.b.c.d.my_module".  If you skip the "fromlist" part then you 
have to bother with mentioning b.c.d.my_module expressly. E.g. for the 
HTMLReport class defined in html_report.py,
 
   # See http://stackoverflow.com/questions/769534
   HTMLReportModule = __import__('custom_templates.foo.bar.html_report', 
fromlist=['does not in fact matter what goes here!'])

lets one then simply say:

   htmlManager = HTMLReportModule.HTMLReport(...)

 > Message: 2
 > ...
 > Subject: Re: [galaxy-dev] Can a tool upload a .loc file that is then ...
 > Python code in galaxy tools can import all sorts of modules.  Is there a way 
 > to add an extra path to the sys.path so that I can include some other python 
 > modules that way? I.e. in some galaxy python config file?
 > 
 > I'd like to say in my tool code
 > 
 > HTMLReportModule = __import__(html_template)
 > ...
 > htmlManager = HTMLReportModule.HTMLReport(tagGroup, options, query_stats)
 > htmlManager.render(out_tabular_file, out_html_file)
 > 
 > and in this way provide a switch from a standard report template that my 
 > galaxy tool has in its folder, to a custom one that a client has fiddled 
 > with and is comfortably outside galaxy's codespace.  The html_template 
 > variable would have 'templates.html_report' for the stock template, but  
 > 'custom_templates.html_report' would pull it from wherever was set in the 
 > path.  Seems like a secure approach.  (I don't want to pass the paths via 
 > the browser).
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Can a tool upload a .loc file that is then reused by subsequent versions of the tool?

[Dooley, Damion]

Python code in galaxy tools can import all sorts of modules.  Is there a way to 
add an extra path to the sys.path so that I can include some other python 
modules that way? I.e. in some galaxy python config file?

I'd like to say in my tool code

HTMLReportModule = __import__(html_template)
...
htmlManager = HTMLReportModule.HTMLReport(tagGroup, options, query_stats)
htmlManager.render(out_tabular_file, out_html_file)

and in this way provide a switch from a standard report template that my galaxy 
tool has in its folder, to a custom one that a client has fiddled with and is 
comfortably outside galaxy's codespace.  The html_template variable would have 
'templates.html_report' for the stock template, but  
'custom_templates.html_report' would pull it from wherever was set in the path. 
 Seems like a secure approach.  (I don't want to pass the paths via the 
browser).

Thanks for your help!

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Can a tool upload a .loc file that is then reused by subsequent versions of the tool?

[Dooley, Damion]

Oh, great!  I'd assumed new tool versions would overwrite them so they could 
implement changes.  But what you say is simpler and more straightforward!

Thanks,

D.

From: Peter Cock [p.j.a.c...@googlemail.com]
Sent: Monday, April 07, 2014 2:23 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Can a tool upload a .loc file that is then reused by 
subsequent versions of the tool?

Isn't this how *.loc.sample files already work? If there is
no pre-existing *.loc file, the *.loc.sample file is copied
to become the initial *.loc file which the local Galaxy
administrator can update/fill in as needed.

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Can a tool upload a .loc file that is then reused by subsequent versions of the tool?

[Dooley, Damion]

Hey folks!  I'm trying to revise a tool that allows the user to select from an 
extendable list of template files for generating blast report output.  The idea 
is that if some reporting needs aren't covered by a generic template, then they 
can develop their own template which can then show up on the list.  And, 
wanting this to work with a tool that is versioned in the toolshed!  

So:
1) is there a way to include a .loc.sample in the tool that gets uploaded, and 
then can be manually revised over time?  A .loc file that subsequent versions 
of the tool would reuse (instead of using their own minimal one that comes with 
toolshed install)?

I can simply have the default template in the tools code folder, since it is 
apt to be upgraded over time.  Custom reports would have their own absolute 
file paths defined in the .loc file.  

Thanks for your input!

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ProFFTPd-1.3.4d no longer able to authenticate users from Postgresql database

[Dooley, Damion]

After following some of the links that this thread referenced (since it is 
totally related to the one I put in on friday), I found the "rollback" 
solution, to disable pbkdf2 authentication.  I did attempt pbkdf2 for a while, 
but openssl upgrade and pbkdf2 recompile issues have led me to drop that.  Can 
someone PLEASE document the pbkdf2 issues and this disable feature on the 
galaxy wiki page: https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP

   Message: 2 
   Date: Mon, 27 Jan 2014 17:08:24 +
   From: "Hazard, E. Starr" 
   Subject: Re: [galaxy-dev] ProFFTPd-1.3.4d no longer able to authenticate 
users from Postgresql database

   So I tried a new version of ProFTPd
   I was not able to locate lines in universe_wsgi.ini which disable PBKDF2


The answer here does switch the passwords back to SHA1 format: (See 
http://lists.bx.psu.edu/pipermail/galaxy-dev/2013-September/016749.html )

>>> Ahoj Honzo,
>>>
>>> Python is case-sensitive. Use 'use_pbkdf2 = False' and make sure you put
>>> it under the '[app:main]' section in the config file.
>>>
>>> Enjoy Galaxy!
>>>
>>> best
>>>
>>> Martin Cech
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Latest update on getting proftpd working with galaxy-dist

[Dooley, Damion]

I think this is a follow-up to last june's 
http://dev.list.galaxyproject.org/proftpd-setting-tt4660329.html question.

I was trying a new test install of galaxy (following instructions on galaxy 
website for a local install from galaxy-dist hg source), sitting on a server 
with two existing galaxy installs.  Got that going ok.  Now trying to get 
profptd 1.3.5rc3 to authenticate against users in the new galaxy install's 
postgresql database.  But it seems a change has been made in authentication, 
from sha1 to pbkdf2 .  Trouble is no docs seem to exist about what the new 
galaxy pbkdf2 parameters are - e.g. proftpd 1.3.5rc3 configuration seems to 
need the following for pbkdf2:

  SQLAuthTypes pbkdf2

  # Use the SHA1 digest algorithm, 1000 iterations, and expect an output
  # length of 20 bytes.
  SQLPasswordPBKDF2 sha1 1000 20
  SQLPasswordSaltFile /path/to/salt/file

now over to galaxy instructions where some detail exists about how to 
configure/compile proftpd but nothing on above parameters.
https://wiki.galaxyproject.org/Admin/Config/Upload%20via%20FTP

It seems like all the code is in place.  Anyone jumped the hurdle on this?

Regards,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Toolshed throwing error when reading tool's XML

[Dooley, Damion]

That advice did the trick, thanks Greg.  I only had .loc files in the upload 
set; didn't know about the .sample version or tool_data_table_conf.xml.sample.  
One thing, did I miss reading the documentation on this somewhere or is this 
workshop type knowledge?

From: Greg Von Kuster [g...@bx.psu.edu]
Sent: Monday, January 20, 2014 1:21 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Toolshed throwing error when reading tool's XML

Hello Damion,

Tool Shed repositories that contain tools that include dynamically generated 
select list parameters that refer to an entry in the tool_data_table_conf.xml 
file must contain a tool_data_table_conf.xml.sample file that contains the 
required entry for each dynamic parameter. Similarly, any index files (i.e., 
~/tool-data/xxx.loc files) to which the tool_data_table_conf.xml file entries 
refer must be defined in xxx.loc.sample files included in the tool shed 
repository along with the tools. If any of these tool_data_table_conf.xml 
entries or any of the required xxx.loc.sample files are missing from the tool 
shed repository, the tools will not properly load and metadata will not be 
generated for the repository. This means that the tools cannot be automatically 
installed into a Galaxy instance.

For those tools that include dynamically generated select list parameters that 
require a missing entry in the tool_data_table_conf.xml file, this file will be 
modified in real time (when it is installed into a Galaxy instance) by adding 
the entry from a tool_data_table_conf.xml.sample file contained in the tool 
shed repository.

Based on your problem description, it seems that your repository may not 
include a required xxx.loc.sample file.

Let me know if adding one does not solve the problem.  Is your repository in 
one of the Galaxy team;'s public tool sheds?

Greg Von Kuster

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Toolshed throwing error when reading tool's XML

[Dooley, Damion]

I'm stumped: Can tools that are destined for the toolshed use  ?

In a blast_reporting.xml tool script I have








But when this gets submitted as a tar file into a toolshed repository, toolshed 
comes back with cryptic:

 Metadata may have been defined for some items in revision '5bbb25b32de2'. 
Correct the following problems if necessary and reset metadata.
 blast_reporting.xml - invalid literal for int() with base 10: 'type'

I eliminate the  
line, but similar problem is still reported 

 blast_reporting.xml - invalid literal for int() with base 10: 'name'

This hints that maybe the toolshed, when it tests the code, it doesn't know 
what/where blast_reporting_fields is?  Is there some special location that the 
blast_reporting_fields.loc file should be located within the uploaded tar file? 
 (I tried putting it in various relative paths in the tar file, i.e. root, and 
under /tool-data/)

p.s. the tool runs file with menu functioning in galaxy itself.

Thanks for any pro tips you folks can come up with!

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] referring to tool_data_tables[] structure (John Chilton)

[Dooley, Damion]

Hi John,

Thansk for the feedback.  Righto, I saw abundant use of "trans" in core galaxy 
code; and problem is I'd wanted access from a tool wrapper's python code.  
Basically I'm trying to get more out of .loc files, for example this field 
specification for blast report data:

   #value   typesubtype sortfilter  default min max choose  
name
   # Remember to edit tool_data_table_conf.xml for column spec!
   length   numeric int 1   1   1   
Alignment length
   qstart   numeric int 1   1   1   
Alignment start in query
   qend numeric int 1   1   1   
Alignment end in query
   sstart   numeric int 1   1   1   
Alignment start in subject
   send numeric int 1   1   1   
Alignment end in subject
   qseq textatgc0   1   1   Aligned 
part of query sequence
   sseq textatgc0   1   1   Aligned 
part of subject sequence
   mseq textatgc0   1   1   
Alignment, matched part
   pident   numeric float   1   1   97  90  100 1   
Percentage of identical matches
   ...

This data is being accessed in our tool xml code via a too_data_table entry and 
is handily providing field lists for sorting, filtering, and searching input 
data.

In our python code I'd like to just say

blastfieldspec = app.tool_data_tables[ 'blast_report_fields' ]

And then go to town on sorting, filtering, validation etc. as desired in 
python, using this spec.  I don't want the python code to be specifying the 
.loc path directly (which I have to do now), I'd much rather take advantage of 
what tool_data_tables could provide.

Our second desired use of tool_data_tables info is a case where some tools 
would be making use of 3rd party datasets that another tool manages.  We want 
to set up a tool/system that manages 3rd party reference databases (e.g. for 
particular specialized gene "universal target" databases like Chaperonin cpn60 
or Legionella mip).  This system would periodically get and process fasta data 
online from sources listed in a .loc file.  We'd process and use these 
databases in pulldown menus, but each of these reference database will need 
management through a Galaxy interface via a plugin tool I guess.  Someone has 
done a prototype for this for database specific to NCBI.  We'd target other 
niche data sources.  I realise another hurdle is getting modified .loc file 
info refreshed back into galaxy without having to stop/restart the server.  
Hoping an extension to the tool_data_tables class could do this too.

Regards, 

Damion



------

Message: 1
Date: Fri, 10 Jan 2014 16:56:56 -0800
From: "Dooley, Damion" 
To: "galaxy-dev@lists.bx.psu.edu" 
Subject: [galaxy-dev] referring to tool_data_tables[] structure
Message-ID:
<7891813f3c8f424b97d8bf2e5600e51903301b88e...@vexccr02.phsabc.ehcnet.ca>

Content-Type: text/plain; charset="us-ascii"

I've seen

   $__app__.tool_data_tables[ 'all_fasta' ].get_fields() )[0][-1]

in tool xml templates.  Is there a way I can access the tool_data_tables 
structure from python code too?

I see all the initialization stuff happening in 
https://bitbucket.org/abrenner/galaxy-central/src/f3e736fe03df3a6dd5438c12ba35ea791a1eaca9/lib/galaxy/app.py?at=default
But not seeing where one can access any of these app variables?

Regards,

Damion


--

Message: 2
Date: Fri, 10 Jan 2014 20:42:50 -0600
From: John Chilton 
To: "Dooley, Damion" 
Cc: "galaxy-dev@lists.bx.psu.edu" 
Subject: Re: [galaxy-dev] referring to tool_data_tables[] structure
Message-ID:

Content-Type: text/plain; charset=ISO-8859-1

Just to clarify do you want to access them from Galaxy web server code
or from a Galaxy tool wrapper written in Python?

Nearly every part of the Galaxy source code has access to app and
everything inside of it, for instance all controller methods take in a
trans variable that contains a reference to app (trans.app).

I suspect you want to access it from a tool wrapper though? This is
not possible. If this is the case - what are you hoping to accomplish?
Do you want to access to the all_fasta data table information in the
example?

-John
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] referring to tool_data_tables[] structure

[Dooley, Damion]

I've seen 

   $__app__.tool_data_tables[ 'all_fasta' ].get_fields() )[0][-1]

in tool xml templates.  Is there a way I can access the tool_data_tables 
structure from python code too?

I see all the initialization stuff happening in 
https://bitbucket.org/abrenner/galaxy-central/src/f3e736fe03df3a6dd5438c12ba35ea791a1eaca9/lib/galaxy/app.py?at=default
But not seeing where one can access any of these app variables?

Regards,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Small icon bug in local galaxy install, followup

[Dooley, Damion]

Thanks Greg.  I'd followed the online local install instructions which I see 
leaves the install in the default branch.  I think I read somewhere that 
default gets the various updates on a monthly basis?  So a pull/update will get 
the fix sooner or later.

Damion

From: Greg Von Kuster [g...@bx.psu.edu]
Sent: Friday, January 03, 2014 3:48 PM
To: Dooley, Damion
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Small icon bug in local galaxy install

Hello Damion,

I believe this issue was corrected in

https://bitbucket.org/galaxy/galaxy-central/commits/3d8841746ce9b65e19a44028dea2eac73a4eddf8

It  looks like you are tracking the galaxy-central brach, so if you pull and 
update your repository you should get the fix.

Greg Von Kuster
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] Small icon bug in local galaxy install

[Dooley, Damion]

A few of our local galaxy installs have a relative folder path (i.e. in 
universe_wsgi.ini,  "prefix = /galaxylab") .  In the Admin tab, "Manage 
installed tool shed repositories", the icons in the legend below aren't showing 
because they have an absolute url:



I can see in the 
galaxy-central/lib/tool_shed/galaxy_install/grids/admin_toolshed_grids.py code:

return '' % 
latest_revision_tip_str

Our /galaxylab is on galaxy changeset:   11247:a9a0ac9c1afa

So I presume this code should be enhanced to get the prefix in there, or made a 
relative url?

Regards,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] galaxy-dev Digest, Vol 89, Issue 32

[Dooley, Damion]

Hi folks,

Has someone seen this behaviour?  I'm trying to do a once-over maintenance 
cleanup of our galaxy installs. The galaxy account is getting emailed nightly 
with an error which is happening when it gets to delete_datasets.sh cron task, 
basically:

 AttributeError: 'NoneType' object has no attribute 'dataset_id'

1) Now I found this thread: 
http://dev.list.galaxyproject.org/Error-when-running-cleanup-datasets-py-td4656392.html
 - and implemented that solution which removed some of the preceding log error 
info, but this related error remains:

 sh ... /production2/galaxy-dist/scripts/cleanup_datasets/delete_datasets.sh
 
 Traceback (most recent call last):
   File "./scripts/cleanup_datasets/cleanup_datasets.py", line 524, in 

 if __name__ == "__main__": main()
   File "./scripts/cleanup_datasets/cleanup_datasets.py", line 126, in main
 delete_datasets( app, cutoff_time, options.remove_from_disk, info_only 
= options.info_only, force_retry = options.force_retry )
   File "./scripts/cleanup_datasets/cleanup_datasets.py", line 301, in 
delete_datasets
 dataset_ids.append( ldda.dataset_id )
 AttributeError: 'NoneType' object has no attribute 'dataset_id'

In the delete_datasets.log it shows that this is last log line before error 
occurs:  

 # Processing LibraryDataset id: 14201


2) So to recap, in step 1 I found:

galaxydb_prod2=> select * from library_dataset where 
library_dataset_dataset_association_id is null;
   id   | library_dataset_dataset_association_id | folder_id | order_id |   
 create_time |update_time | name  | info | 
deleted | purged 
 
---++---+--+++---+--+-+
  14201 ||   544 |   18 | 
2012-11-01 17:26:23.877817 | 2012-11-01 17:26:35.787611 | NC_006625.fna |  
| f   | f
 (1 row)

And so I ran the recommended query, "update library_dataset set folder_id = 
null where id=14201;" and that seemed to have a positive effect.

For what its worth, 

 galaxydb_prod2=> select * from library_dataset_dataset_association where 
id = 14201
 galaxydb_prod2-> ;
   id   | library_dataset_id | dataset_id |create_time |
   update_time| copied_from_history_dataset_association_id | 
copied_from_library_dataset_dataset_association_id | name  | info | 
blurb | peek | extension | metadata | parent_id | designation | deleted 
| visible | user_id | message | state | tool_version 
 
---++++--+++---+--+---+--+---+--+---+-+-+-+-+-+---+--
  14201 |  14201 |  14706 | 2012-11-01 17:26:28.593081 | 
2012-11-01 17:26:30.8649 || 
   | NC_006625.fna |   |   
|  | auto  | {"dbkey": ["?"]} |   | | f   | t   
|   1 | |   | 
 (1 row)

The last thing I'd say is that this 'NC_006625.fna' item doesn't show up in the 
website front-end's sole Data Library list of folders and sub-items, nor does 
it show in the deleted list. It seems orphaned but I'd rather someone had 
confident advice on this one.  My novice attempt at surgery would probably lead 
to lots of blood on the table.

Help much appreciated!

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] NCBI Blast+ "BLAST XML to tabular" tool question - ADDENDUM

[Dooley, Damion]

Woops - I realize now findtext() must be unescaping all ">", so Peter was 
trying to address other non-splitting occurances of " >" as per his patch 
notes.  But perhaps a stop_err() isn't merrited in this case?

So ignore my test for ">" comment.

Regards,

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] NCBI Blast+ "BLAST XML to tabular" tool question

[Dooley, Damion]

I'm doing a 1 step generic reporting tool along the lines of the "BLAST XML to 
tabular" script by Peter.  I was just about to ask about this line, which 
looked pretty much like a bug: 

sallseqid = ";".join(name.split(None,1)[0] for name in hit_def.split(" 
>"))

Then I found the patch from Nov 7th 2013:


https://github.com/peterjc/galaxy_blast/blob/master/tools/ncbi_blast_plus/blastxml_to_tabular.py

try:
sallseqid = ";".join(name.split(None,1)[0] for name in 
hit_def.split(" >"))
except IndexError as e:
stop_err("Problem splitting multuple hits?\n%r\n--> %s" % 
(hit_def, e))

Yay!  But what I've seen in recent XML output reports is that the ">" content 
has been changed to ">" .  E.g. 

https://github.com/biopython/biopython/blob/master/Tests/Blast/mirna.xml


66
gi|195029385|ref|XR_047134.1|
Drosophila grimshawi miR-7-RA (Dgri\mir-7), ncRNA 
>gi|195336156|ref|XR_048470.1| Drosophila sechellia miR-7-RA (Dsec\mir-7), 
ncRNA >gi|195585143|ref|XR_050309.1| Drosophila simulans miR-7-RA 
(Dsim\mir-7), ncRNA
XR_047134
...

So perhaps a stop_err() could be avoided, if test is for ">" instead?  I 
assume that no variants of python ElementTree.iterparse() will unescape content 
when returned via the iterator?

Damion
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/