Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[James Taylor]

Clarity in this case, definitely needs the 1.

On Feb 19, 2013, at 3:38 PM, Peter Cock  wrote:

> On Tue, Feb 19, 2013 at 6:45 PM, Nicola Soranzo  wrote:
>> Il giorno mar, 19/02/2013 alle 14.15 +, Peter Cock ha scritto:
>>> On Tue, Feb 19, 2013 at 2:00 PM, James Taylor  wrote:
 On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock wrote:
> I think it could make sense to define generic 'asn1' and
> 'asn1-binary' formats in the Galaxy core (name suggestions
> welcome)
>> 
>> What about
>> 
>> extension="asn" type="galaxy.datatypes.data:GenericAsn1"
>> 
>> and
>> 
>> extension="asnb" type="galaxy.datatypes.binary:GenericAsn1Binary"
>> 
>> like GenericXml class?
> 
> Those seem sensible to me as the class names, although I'm
> not so sure about the format names (aka 'extensions' in Galaxy
> terms). I'd prefer to see the '1' in the name for clarity. My
> suggestions of 'asn1' and 'asn1-binary' were based on NCBI usage.
> 
> Perhaps the Galaxy team could comment on their views here
> for conciseness versus clarity in file format names for Galaxy?
> 
> Would a pull request implementing this be acceptable?
 
 Yes. My understanding is that ASN is a completely flexible metaformat,
 like XML, and so should be under either Text or Data, with appropriate
 subtypes defined for blast, et cetera.
>>> 
>>> Thank James,
>>> 
>>> Yes - very like XML, but with the subtlety that ASN.1 comes in text and
>>> binary favours (which I presume applies to all the variants, although
>>> the binary versions may not be as commonly used for the smaller files).
>>> 
>>> Nicola - do you want to make a pull request to galaxy-central defining
>>> ASN.1 text and binary formats (which we can then subclass for the
>>> NCBI BLAST+ wrappers)? Or should I?
>> 
>> If you mean a minimal implementation, I can surely do that. If something
>> more elaborated is needed, then probably you are more qualified than me!
> 
> The current minimal implementation you sent me for BLAST+
> would be an excellent start. Things like data type sniffers etc
> would be a nice to have feature, but can be added later I think.
> And the sooner this gets into the Galaxy core, the sooner we
> can use it in the BLAST+ wrappers :)
> 
>>> I think the mime-type for the base ASN.1 text format should probably
>>> be text/plain based on the NCBI usage patterns.
>> 
>> Ok.
>> 
>>> I'm not sure what the mime-type for the base ASN.1 binary format
>>> should be (but I don't think it should be chemical/ncbi-asn1-binary).
>> 
>> application/octet-stream ?
> 
> Probably OK - we can/should test this by uploading a test
> binary ASN.1 file into Galaxy and downloading it out again.
> 
> Thanks,
> 
> Peter

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Submitting jobs as a real user without using chown, please

[Thon de Boer]

Hi,

 

I am trying to setup my galaxy system to allow jobs to be submitted as the
real user, since people want to keep an eye on their job on the cluster
sometimes and they have no ideas which ones are theirs.

 

I tried the approach on the wiki here:

 

http://wiki.galaxyproject.org/Admin/Config/Performance/Cluster?highlight=%28
submit%29%7C%28jobs%29%7C%28as%29%7C%28user%29#Submitting_Jobs_as_the_Real_U
ser

 

but unfortunately, the CHOWN command is not allowed, not even as a sudo
user.

Probably has to do with the fact that we run our cluster from an isilon
system, which I assume is pretty typical.

 

The job was actually successfully submitted as the intended user, so that
part works, but if we can just get it to work without having to rely on
chown that would be awesome.

 

Can someone point me in the right direction?

 

Here's the error.

 

galaxy.jobs.runners.local DEBUG 2013-02-19 19:35:31,524 execution of
external set_meta for job 148 finished

galaxy.jobs DEBUG 2013-02-19 19:35:31,576 (148) Changing ownership of
working directory with: /usr/bin/sudo -E scripts/external_chown_script.py
/mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1
48 svcgalaxy 1

galaxy.jobs ERROR 2013-02-19 19:35:31,653 (148) Failed to change ownership
of
/mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1
48, failing

Traceback (most recent call last):

  File
"/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line
343, in finish

self.reclaim_ownership()

  File
"/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line
916, in reclaim_ownership

self._change_ownership( self.galaxy_system_pwent[0], str(
self.galaxy_system_pwent[3] ) )

  File
"/mnt/ngs/analysis/svcgalaxy/galaxy-test/lib/galaxy/jobs/__init__.py", line
902, in _change_ownership

assert p.returncode == 0

AssertionError

galaxy.jobs DEBUG 2013-02-19 19:35:31,722 fail(): Moved
/mnt/ngs/analysis/svcgalaxy/galaxy-test/database/job_working_directory/000/1
48/galaxy_dataset_332.dat to
/mnt/ngs/analysis/svcgalaxy/galaxy-test/database/files/000/dataset_332.dat

galaxy.datatypes.metadata DEBUG 2013-02-19 19:35:31,924 Cleaning up external
metadata files

 

Thanks

 

Thon

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Merge internal head to release_2012.02.08

[Derrick Lin]

Thanks guys,

I will give it a try.

Cheers,
D


On Wed, Feb 20, 2013 at 12:40 AM, Nate Coraor  wrote:

> Thanks Dannon.  One more tip:  If you commit your changes on the default
> branch, you will need to merge them with stable, i.e. `hg update
> release_2013.02.08` followed by `hg merge `
>
> --nate
>
> On Feb 19, 2013, at 7:27 AM, Dannon Baker wrote:
>
> > Derrick,
> >
> > The new update process outlined works with local changes the same way
> > as the 'hg pull -u' used previously.  The only difference is that it
> > updates to a particular tag, instead of whatever is at the galaxy-dist
> > tip.  'hg pull -u' that you've used before is actually just shorthand
> > for doing the two common commands at once, 'hg pull; hg update'.
> > 'update' will only discard local changes when executed with the '-C'
> > option, otherwise it'll try to merge.
> >
> > If you're concerned about persisting local changes, it's probably a
> > good idea to go ahead and commit them to your local repository or at
> > least create a patch that you could use to recreate the changes `hg
> > diff -p > our_local_changes.patch`.
> >
> > -Dannon
> >
> > On Mon, Feb 18, 2013 at 11:36 PM, Derrick Lin  wrote:
> >> Hi guys,
> >>
> >> We have checked in few internal changesets to our galaxy. Normally we
> did hg
> >> pull then hg merge for upgrading.
> >>
> >> With the 8 Feb release, it advice that we must do hg update
> >> release_2013.02.08 if we want to upgrade from stable branch.
> >>
> >> My understanding for hg update is, it will overwrite all local changes
> which
> >> we want to avoid.
> >>
> >> So in our case, what's the best way for us to upgrade?
> >>
> >> Cheers,
> >> Derrick
> >>
> >> ___
> >> Please keep all replies on the list by using "reply all"
> >> in your mail client.  To manage your subscriptions to this
> >> and other Galaxy lists, please use the interface at:
> >>
> >>  http://lists.bx.psu.edu/
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >
> >  http://lists.bx.psu.edu/
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] deleting and free space

[Hakeem Almabrazi]

Thank Alfonso,

Where is the Options menu though?  I do not see anything called Options.  

Thank you

-Original Message-
From: Alfonso Garrido-Lecca [mailto:alfonso.garrido-le...@colorado.edu] 
Sent: Tuesday, February 19, 2013 2:54 PM
To: Hakeem Almabrazi; Galaxy Dev
Subject: Re: [galaxy-dev] deleting and free space

you can go to options. then click on show deleted datasets. then this will make 
appear all datasets that have been "deleted" but are not yet removed from the 
disk. 
You will see something like this: "This dataset has been deleted. Click here to 
undelete or here to immediately remove it from disk". Then just click to 
immediatly 
remove from disk and will do the work. 
i hope this helps. 
alfonso 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Galaxy @ GMOD2013 & Biocuration 2013

[Dave Clements]

Hello all,

There are a couple of things going on with the GMOD
Project that
the Galaxy Community might be interested in. If you aren't familiar with
GMOD, it is a federation of open-source projects and tools that enable
visualizing, annotating, managing, and analyzing biological research data.
 GMOD includes Galaxy and several other widely used tools such as
BioMart
, InterMine , MAKER
, Chado  and GBrowse
 and JBrowse .

The first is that GMOD is running a community-wide
survey through
March 1. The survey "aims to find out how you are using GMOD, what you find
useful (or otherwise), what support you need, and some information on the
GMOD components you use."  The survey results will be published at
gmod.organd past surveys have been extremely useful to the project.
If that's not
enough motivation, once you are done with the survey, you can enter to win
a free genome profile from 23andMe.

   -

   → *C'mon, take the dang survey .*

The second is that registration is now open for the 2013 GMOD
Meeting being
held in Cambridge, UK, April 5 and 6.  There will be a Galaxy presence (at
least a project update, and I'm lobbying for a
CloudMan workshop
as well (and you could lobby too! )).

Finally, Biocuration 2013  is
immediately after the GMOD Meeting, and also in Cambridge.  Biocuration
will feature a GO Galaxy
Workshop
about
using Galaxy to reason with ontologies.

Thanks,

Dave Clements
-- 
http://galaxyproject.org/
http://getgalaxy.org/
http://usegalaxy.org/
http://wiki.galaxyproject.org/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] deleting and free space

[Alfonso Garrido-Lecca]

you can go to options. then click on show deleted datasets. then this will make 
appear all datasets that have been "deleted" but are not yet removed from the 
disk. 
You will see something like this: "This dataset has been deleted. Click here to 
undelete or here to immediately remove it from disk". Then just click to 
immediatly 
remove from disk and will do the work. 
i hope this helps. 
alfonso 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] deleting and free space

[Hakeem Almabrazi]

Hi,

I have my own local galaxy installed and things are great.  However, I am 
running of space, is there a better way to free space from the GUI?  I have 
been deleting sets/files from my history but it looks like the Usage still 
going up.  I manually run the following scripts but still nothing change
There are 5 scripts included in the Galaxy distribution that can be used to 
clean up unwanted histories, libraries and datasets. There are located in the 
/scripts/cleanup_datasets directory and are named:

  *   delete_userless_histories.sh
  *   purge_histories.sh
  *   purge_datasets.sh
  *   purge_folders.sh
  *   purge_libraries.sh
  *   delete_datasets.sh
Is there better ways to do this task?

Regards,




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[Peter Cock]

On Tue, Feb 19, 2013 at 6:45 PM, Nicola Soranzo  wrote:
> Il giorno mar, 19/02/2013 alle 14.15 +, Peter Cock ha scritto:
>> On Tue, Feb 19, 2013 at 2:00 PM, James Taylor  wrote:
>> > On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock wrote:
>> >>I think it could make sense to define generic 'asn1' and
>> >>'asn1-binary' formats in the Galaxy core (name suggestions
>> >>welcome)
>
> What about
>
> extension="asn" type="galaxy.datatypes.data:GenericAsn1"
>
> and
>
> extension="asnb" type="galaxy.datatypes.binary:GenericAsn1Binary"
>
> like GenericXml class?

Those seem sensible to me as the class names, although I'm
not so sure about the format names (aka 'extensions' in Galaxy
terms). I'd prefer to see the '1' in the name for clarity. My
suggestions of 'asn1' and 'asn1-binary' were based on NCBI usage.

Perhaps the Galaxy team could comment on their views here
for conciseness versus clarity in file format names for Galaxy?

>> >> Would a pull request implementing this be acceptable?
>> >
>> > Yes. My understanding is that ASN is a completely flexible metaformat,
>> > like XML, and so should be under either Text or Data, with appropriate
>> > subtypes defined for blast, et cetera.
>>
>> Thank James,
>>
>> Yes - very like XML, but with the subtlety that ASN.1 comes in text and
>> binary favours (which I presume applies to all the variants, although
>> the binary versions may not be as commonly used for the smaller files).
>>
>> Nicola - do you want to make a pull request to galaxy-central defining
>> ASN.1 text and binary formats (which we can then subclass for the
>> NCBI BLAST+ wrappers)? Or should I?
>
> If you mean a minimal implementation, I can surely do that. If something
> more elaborated is needed, then probably you are more qualified than me!

The current minimal implementation you sent me for BLAST+
would be an excellent start. Things like data type sniffers etc
would be a nice to have feature, but can be added later I think.
And the sooner this gets into the Galaxy core, the sooner we
can use it in the BLAST+ wrappers :)

>> I think the mime-type for the base ASN.1 text format should probably
>> be text/plain based on the NCBI usage patterns.
>
> Ok.
>
>> I'm not sure what the mime-type for the base ASN.1 binary format
>> should be (but I don't think it should be chemical/ncbi-asn1-binary).
>
> application/octet-stream ?
>

Probably OK - we can/should test this by uploading a test
binary ASN.1 file into Galaxy and downloading it out again.

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] can't run instance for CloudMan

[Dannon Baker]

Stefano,

Could you clarify what you did differently?  The public URL provided
by Amazon *should* be the primary interface to your cluster.

-Dannon

On Tue, Feb 19, 2013 at 12:55 PM, stefano cardinale
 wrote:
> Hello,
>  thanks, I managed to run the instances by using BioCloudCentral on the same 
> SecurityGroup and Keys I already set up, which were fine. For some reason 
> pointing the browser to the address provided by Amazon, as described in your 
> Wiki CloudMan thread, did not work, but finally I found some comments on 
> BioCloudCentral that helped. Suggestion: may be somebody should amend the 
> Wiki step-wise description on how to set up CloudMan?
> stefano
>
>
> On Feb 15, 2013, at 3:07 AM, Quang Trinh  wrote:
>
>> Hi Stefano,
>>  Can you try out what we put together:
>>
>> https://github.com/modENCODE-DCC/Galaxy
>>
>> Thanks,
>>
>> Q
>>
>> On Thu, Feb 14, 2013 at 8:28 PM, stefano cardinale
>>  wrote:
>>> Hi, I am having a problem with running galaxy on the Amazon cloud. I have 
>>> tried this several times over the past few days, I followed the list and it 
>>> seems my instance is up and running. However, when I type the DNS in any 
>>> browser it can't connect to it. I am kind of aiming in the dark now because 
>>> I don't know where the problem is, anyone can help?
>>> thanks!
>>> stefano
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>
>>>  http://lists.bx.psu.edu/
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>   http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] can't run instance for CloudMan

[stefano cardinale]

Hello,
 thanks, I managed to run the instances by using BioCloudCentral on the same 
SecurityGroup and Keys I already set up, which were fine. For some reason 
pointing the browser to the address provided by Amazon, as described in your 
Wiki CloudMan thread, did not work, but finally I found some comments on 
BioCloudCentral that helped. Suggestion: may be somebody should amend the Wiki 
step-wise description on how to set up CloudMan?
stefano


On Feb 15, 2013, at 3:07 AM, Quang Trinh  wrote:

> Hi Stefano,
>  Can you try out what we put together:
> 
> https://github.com/modENCODE-DCC/Galaxy
> 
> Thanks,
> 
> Q
> 
> On Thu, Feb 14, 2013 at 8:28 PM, stefano cardinale
>  wrote:
>> Hi, I am having a problem with running galaxy on the Amazon cloud. I have 
>> tried this several times over the past few days, I followed the list and it 
>> seems my instance is up and running. However, when I type the DNS in any 
>> browser it can't connect to it. I am kind of aiming in the dark now because 
>> I don't know where the problem is, anyone can help?
>> thanks!
>> stefano
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> 
>>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[Nicola Soranzo]

Il giorno mar, 19/02/2013 alle 14.15 +, Peter Cock ha scritto: 
> On Tue, Feb 19, 2013 at 2:00 PM, James Taylor  wrote:
> > On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock  
> > wrote:
> >>I think it could make sense to define generic 'asn1' and
> >>'asn1-binary' formats in the Galaxy core (name suggestions
> >>welcome)

What about 

extension="asn" type="galaxy.datatypes.data:GenericAsn1"

and

extension="asnb" type="galaxy.datatypes.binary:GenericAsn1Binary" 

like GenericXml class?

> >> Would a pull request implementing this be acceptable?
> >
> > Yes. My understanding is that ASN is a completely flexible metaformat,
> > like XML, and so should be under either Text or Data, with appropriate
> > subtypes defined for blast, et cetera.
> 
> Thank James,
> 
> Yes - very like XML, but with the subtlety that ASN.1 comes in text and
> binary favours (which I presume applies to all the variants, although
> the binary versions may not be as commonly used for the smaller files).
> 
> Nicola - do you want to make a pull request to galaxy-central defining
> ASN.1 text and binary formats (which we can then subclass for the
> NCBI BLAST+ wrappers)? Or should I?

If you mean a minimal implementation, I can surely do that. If something
more elaborated is needed, then probably you are more qualified than me!

> I think the mime-type for the base ASN.1 text format should probably
> be text/plain based on the NCBI usage patterns.

Ok.

> I'm not sure what the mime-type for the base ASN.1 binary format
> should be (but I don't think it should be chemical/ncbi-asn1-binary).

application/octet-stream ?

Best,
Nicola

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] KeyError ['tools'] on importing workflow from locally installed repository

[Dave Bouvier]

Jorrit,

Thank you for uncovering that issue, I've commited a fix in the stable 
branch of galaxy-central at 8887:d5896e8a60fa. To apply that changeset, 
you'll need to run the following commands from your Galaxy directory:


$ hg pull -b stable https://bitbucket.org/galaxy/galaxy-central/
$ hg update stable

   --Dave B.

On 2/18/13 21:57:18.000, jorrit poelen wrote:

Hey y'all,

I am testing how to best share a workflow through a tool shed.  I stumbled 
across a possible bug and am hoping that this is simply a configuration issue.

Steps to reproduce:

1. In local galaxy instance goto . .  admin > search and browse tool sheds
2. locate select "Galaxy test tool shed"
3. search for "obo"
4. preview and install repository "obo_test_workflows"
5. in next screen click "install to Galaxy"
6. in next screen click "install"
7. in next screen click "obo_test_workflows"
8. in next screen, under workflows, click "TestOboWorkflow1"
9. in svg graphics page > repository actions > import workflow to galaxy

Expected results:
Workflow is installed and made available in workflow section in local Galaxy 
instance.

Actual results:
URL: 
http://galaxy:7474/admin_toolshed/import_workflow?repository_id=2d9035b3fc152403&workflow_name=0c2d739e4f60c1bed9e8e46f2f317141702ceb26%3A546573744f626f576f726b666c6f7731
File 
'/home/ubuntu/galaxy-dist/eggs/WebError-0.8a-py2.7.egg/weberror/evalexception/middleware.py',
 line 364 in respond
   app_iter = self.application(environ, detect_start_response)
File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/debug/prints.py', 
line 98 in __call__
   environ, self.app)
File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/wsgilib.py', line 
539 in intercept_output
   app_iter = application(environ, replacement_start_response)
File '/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/recursive.py', 
line 80 in __call__
   return self.application(environ, start_response)
File 
'/home/ubuntu/galaxy-dist/eggs/Paste-1.6-py2.7.egg/paste/httpexceptions.py', 
line 632 in __call__
   return self.application(environ, start_response)
File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/base.py', line 132 in 
__call__
   return self.handle_request( environ, start_response )
File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/base.py', line 185 in 
handle_request
   body = method( trans, **kwargs )
File '/home/ubuntu/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 216 
in decorator
   return func( self, trans, *args, **kwargs )
File 
'/home/ubuntu/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
 line 744 in import_workflow
   tools_metadata = metadata[ 'tools' ]
KeyError: 'tools'

Thanks in advance for suggestions on how to solve / work around this issue.

-jorrit

PS Note that the workflow can be installed using a url pointing to the workflow 
from a hg repository on bitbucket (ie. 
https://bitbucket.org/jhpoelen/obo_workflows/raw/25d386a50562d02e03fa821d96a831cf75edcbae/Galaxy-Workflow-TestOboWorkflow1.ga).
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

   http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy hg repository problem "abort: path ... traverses symbolic link"

[Peter Cock]

On Tue, Feb 19, 2013 at 5:26 PM, Nate Coraor  wrote:
> On Feb 19, 2013, at 12:18 PM, Peter Cock wrote:
> Nate wrote:
>>> Hi Peter,
>>>
>>> This is due to the annoying way in which Mercurial handles symlinks.  We're
>>> going to graft the changesets that caused this up to stable shortly, but in 
>>> the
>>> meantime, you can do:
>>>
>>> % rm static/june_2007_style
>>> % hg update stable --clean
>>>
>>> This would remove any uncommitted local changes, if you had any, so it's
>>> not safe for all scenarios.  But it's fine here since you're working on a 
>>> fresh
>>> clone with no changes.
>>
>> Cheers Nate, that trick seems to work.
>>
>> I'm guessing this recent commit (11 days ago) is part of the issue:
>> https://bitbucket.org/galaxy/galaxy-central/commits/506484344db3a370f8ae24096041d38557d1967e
>
> Indeed, that, and the previous commit.
>
>> I hope this won't require any 'grafting' for those of us working from the
>> default branch? Will I be safe merging the 'default' into my own branch,
>> or should I wait while you guys sort out whatever you need to do first?
>
> It shouldn't require you to do anything.  The only complications that
> should arise will be if you have changes to files under 
> static/june_2007_style/

I've not touched that, so that's all good news - thank you :)

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy hg repository problem "abort: path ... traverses symbolic link"

[Nate Coraor]

On Feb 19, 2013, at 12:18 PM, Peter Cock wrote:

> On Tue, Feb 19, 2013 at 5:08 PM, Nate Coraor  wrote:
>> On Feb 19, 2013, at 11:23 AM, Peter Cock wrote:
>> 
>>> Hi all,
>>> 
>>> I've got an hg puzzle, can anyone on the Galaxy team explain the
>>> cause of this and what action I can take to avoid it?
>>> 
>>> Starting with a clean checkout,
>>> 
>>> $ hg clone https://bitbucket.org/galaxy/galaxy-central
>>> warning: bitbucket.org certificate with fingerprint
>>> 24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not
>>> verified (check hostfingerprints or web.cacerts config setting)
>>> destination directory: galaxy-central
>>> requesting all changes
>>> adding changesets
>>> adding manifests
>>> adding file changes
>>> added 8886 changesets with 33822 changes to 6914 files (+1 heads)
>>> updating to branch default
>>> 4019 files updated, 0 files merged, 0 files removed, 0 files unresolved
>>> 
>>> $ cd galaxy-central
>>> 
>>> $ hg branch
>>> default
>>> 
>>> $ hg branches
>>> default 8885:9852dd712f5c
>>> stable  8880:8da37d3985ab
>>> 
>>> $ hg checkout stable
>>> abort: path 'static/june_2007_style/Makefile' traverses symbolic link
>>> 'static/june_2007_style'
>>> 
>>> I was actually trying to checkout one of my own branches, having
>>> setup the .hg/hgrc to include a link to my own fork on bitbucket.
>> 
>> Hi Peter,
>> 
>> This is due to the annoying way in which Mercurial handles symlinks.  We're
>> going to graft the changesets that caused this up to stable shortly, but in 
>> the
>> meantime, you can do:
>> 
>> % rm static/june_2007_style
>> % hg update stable --clean
>> 
>> This would remove any uncommitted local changes, if you had any, so it's
>> not safe for all scenarios.  But it's fine here since you're working on a 
>> fresh
>> clone with no changes.
> 
> Cheers Nate, that trick seems to work.
> 
> I'm guessing this recent commit (11 days ago) is part of the issue:
> https://bitbucket.org/galaxy/galaxy-central/commits/506484344db3a370f8ae24096041d38557d1967e

Indeed, that, and the previous commit.

> I hope this won't require any 'grafting' for those of us working from the
> default branch? Will I be safe merging the 'default' into my own branch,
> or should I wait while you guys sort out whatever you need to do first?

It shouldn't require you to do anything.  The only complications that should 
arise will be if you have changes to files under static/june_2007_style/

--nate

> 
> Thanks,
> 
> Peter


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy hg repository problem "abort: path ... traverses symbolic link"

[Peter Cock]

On Tue, Feb 19, 2013 at 5:08 PM, Nate Coraor  wrote:
> On Feb 19, 2013, at 11:23 AM, Peter Cock wrote:
>
>> Hi all,
>>
>> I've got an hg puzzle, can anyone on the Galaxy team explain the
>> cause of this and what action I can take to avoid it?
>>
>> Starting with a clean checkout,
>>
>> $ hg clone https://bitbucket.org/galaxy/galaxy-central
>> warning: bitbucket.org certificate with fingerprint
>> 24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not
>> verified (check hostfingerprints or web.cacerts config setting)
>> destination directory: galaxy-central
>> requesting all changes
>> adding changesets
>> adding manifests
>> adding file changes
>> added 8886 changesets with 33822 changes to 6914 files (+1 heads)
>> updating to branch default
>> 4019 files updated, 0 files merged, 0 files removed, 0 files unresolved
>>
>> $ cd galaxy-central
>>
>> $ hg branch
>> default
>>
>> $ hg branches
>> default 8885:9852dd712f5c
>> stable  8880:8da37d3985ab
>>
>> $ hg checkout stable
>> abort: path 'static/june_2007_style/Makefile' traverses symbolic link
>> 'static/june_2007_style'
>>
>> I was actually trying to checkout one of my own branches, having
>> setup the .hg/hgrc to include a link to my own fork on bitbucket.
>
> Hi Peter,
>
> This is due to the annoying way in which Mercurial handles symlinks.  We're
> going to graft the changesets that caused this up to stable shortly, but in 
> the
> meantime, you can do:
>
> % rm static/june_2007_style
> % hg update stable --clean
>
> This would remove any uncommitted local changes, if you had any, so it's
> not safe for all scenarios.  But it's fine here since you're working on a 
> fresh
> clone with no changes.

Cheers Nate, that trick seems to work.

I'm guessing this recent commit (11 days ago) is part of the issue:
https://bitbucket.org/galaxy/galaxy-central/commits/506484344db3a370f8ae24096041d38557d1967e

I hope this won't require any 'grafting' for those of us working from the
default branch? Will I be safe merging the 'default' into my own branch,
or should I wait while you guys sort out whatever you need to do first?

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Changing job command line.

[Nate Coraor]

On Feb 14, 2013, at 7:26 AM, James wrote:

> Hi Nate,
> 
> Thanks for your reply,
> 
> I guess the idea is that the job is prepared in the interface which lets a 
> user select
> the grid. This is currently located in my galaxy instance as a if statement 
> in the
> default job dispatcher. Just before it sends it to the job runner and all the 
> preparation
> is done there.
> 
> But regarding the command line you are probably right, it would just mean 
> that every job runner
> on our instance will have to be patched to include this if statement as to 
> whether the command line
> needs to be reassigned.

Hi James,

In this case, either method seems fine.  You are going to have to modify the 
way that Galaxy prepares jobs regardless of the way you do it.

> Another simple question regarding adding UI options to toolform.mako. Could i 
> parse them
> to my module using javascript.
> 
> Although I feel the best approach would be to add these options to the
> job model and then they would remain tracked in the database.

There have been various requests in the past to allow for inclusion of a global 
parameter set in tool forms, for this exact purpose.  I thought there was a 
bitbucket issue or trello card for it, but I'm having trouble locating it now.

--nate

> 
> Cheers James.
> 
> On 2/14/2013 4:20 AM, Nate Coraor wrote:
>> On Feb 4, 2013, at 6:03 PM, James Boocock wrote:
>> 
>>> Hi All,
>>> 
>>> I am currently working on a clustering interface for galaxy that will
>>> hopefully enable users to pick the grid
>>> in the tool form.
>>> 
>>> With tools that need access to files within the galaxy directory, I
>>> have an idea to create a symlinked
>>> fake galaxy root with all the files the tools needs for galaxy. This
>>> is done currently in my own xml format
>>> for each grid.
>>> 
>>> Problem is when parsing a tool_wrapper variable I need to prepare the
>>> job and add my additional commands
>>> to the command line so that the fake galaxy dir is created with the
>>> symlinked databases.
>>> 
>>> Is it bad practice to disable preparation in each tool runner if the
>>> job has come from my clustering module
>>> or is there a better way to do this. Will the job runners
>>> wrapper.prepare() overwrite any of my changes to the command-line
>>> when the job makes its way to the runner say local.py.
>> Hi James,
>> 
>> You'll probably need to make the changes to the command line after the 
>> command line has been assembled in prepare().  Can you do this later in 
>> prepare()?
>> 
>> --nate
>> 
>>> 
>>> Cheers James.
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>> 
>>>  http://lists.bx.psu.edu/
> 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Galaxy hg repository problem "abort: path ... traverses symbolic link"

[Nate Coraor]

On Feb 19, 2013, at 11:23 AM, Peter Cock wrote:

> Hi all,
> 
> I've got an hg puzzle, can anyone on the Galaxy team explain the
> cause of this and what action I can take to avoid it?
> 
> Starting with a clean checkout,
> 
> $ hg clone https://bitbucket.org/galaxy/galaxy-central
> warning: bitbucket.org certificate with fingerprint
> 24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not
> verified (check hostfingerprints or web.cacerts config setting)
> destination directory: galaxy-central
> requesting all changes
> adding changesets
> adding manifests
> adding file changes
> added 8886 changesets with 33822 changes to 6914 files (+1 heads)
> updating to branch default
> 4019 files updated, 0 files merged, 0 files removed, 0 files unresolved
> 
> $ cd galaxy-central
> 
> $ hg branch
> default
> 
> $ hg branches
> default 8885:9852dd712f5c
> stable  8880:8da37d3985ab
> 
> $ hg checkout stable
> abort: path 'static/june_2007_style/Makefile' traverses symbolic link
> 'static/june_2007_style'
> 
> I was actually trying to checkout one of my own branches, having
> setup the .hg/hgrc to include a link to my own fork on bitbucket.

Hi Peter,

This is due to the annoying way in which Mercurial handles symlinks.  We're 
going to graft the changesets that caused this up to stable shortly, but in the 
meantime, you can do:

% rm static/june_2007_style
% hg update stable --clean

This would remove any uncommitted local changes, if you had any, so it's not 
safe for all scenarios.  But it's fine here since you're working on a fresh 
clone with no changes.

--nate

> 
> Thanks!
> 
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Datasets linked into Galaxy do not work

[Nate Coraor]

On Feb 14, 2013, at 4:38 AM, Sarah Diehl wrote:

> Hi Nate,
> 
> copied datasets do work and I run Galaxy from the server root.

Hi Sarah,

I'm having trouble replicating this issue.  Could you send along (privately) 
the relevant portions of your proxy server configuration, and universe_wsgi.ini?

Thanks,
--nate

> 
> Thanks,
> Sarah
> 
> 
> - Original Message -
> From: "Nate Coraor" 
> To: "Sarah Diehl" 
> Cc: "galaxy-dev@lists.bx.psu.edu List" 
> Sent: Wednesday, February 13, 2013 8:22:10 PM
> Subject: Re: [galaxy-dev] Datasets linked into Galaxy do not work
> 
> On Feb 12, 2013, at 11:17 AM, Sarah Diehl wrote:
> 
>> Hi all,
>> 
>> I have problems with datasets that were linked into Galaxy data libraries 
>> (as Admin: "Add datasets", select "Upload files from filesystem paths" and 
>> "Link to files without copying to Galaxy"). Those files cannot be looked at, 
>> downloaded or viewed in a genome browser.
>> 
>> Errors in the browser are:
>> The requested URL /datasets/126e6c4f4c2d468e/display/ was not found on this 
>> server.
>> The requested URL /library_common/download_dataset_from_folder was not found 
>> on this server.
>> An error occurred while accessing: 
>> http://galaxy.immunbio.mpg.de/display_application/7108f175b5be4900/igv_bam/local_default/d85d47a3ee6ecd54/data/galaxy_7108f175b5be4900.bam
>>  Read error; BinaryCodec in readmode; streamed file (filename not available)
>> 
>> I don't see any errors in the log files.
>> 
>> We do this kind of linking a lot and everything worked fine in the past. I 
>> need to use this feature and need it to work, due to hard disk space and 
>> data duplication.
> 
> Hi Sarah,
> 
> Do display applications work correctly for datasets that were uploaded to a 
> library and copied in to Galaxy rather than linked?  Do you run Galaxy from 
> the server root (http://galaxy.immunbio.mpg.de/) or some path underneath the 
> root?
> 
> Thanks,
> --nate
> 
>> 
>> Any help is appreciated.
>> 
>> Best regards,
>> Sarah
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> 
>> http://lists.bx.psu.edu/
> 


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Searching tool shed for file format definitions?

[Greg Von Kuster]

Hi Peter,

This is not yet available, but something I've been planning to do for quite a 
while.  I've opened the following Trello card for this.  I'm not exactly sure 
yet how this will be implemented as I haven't thought this completely though.  
Of course, ideas from the community are always welcome, and yours below will be 
incorporated in some capacity.  I'll try to have this worked out over the next 
few weeks.  My primary focus right now is Galaxy flavors, but I'll work this in 
as bandwidth allows.

https://trello.com/card/toolshed-add-support-for-search-tool-shed-repositories-for-custom-datatype-information/506338ce32ae458f6d15e4b3/649

Thanks Peter,

Greg Von Kuster


On Feb 19, 2013, at 11:18 AM, Peter Cock wrote:

> Hello all,
> 
> I was wondering how people keep track of the available file format
> definitions in the tool shed(s)? For core file types defined in Galaxy,
> I can just view the datatypes_conf.xml file and/or the classes in
> lib/galaxy/datatypes/ - but how can I do something similar on a
> Galaxy Tool Shed?
> 
> One solution might be automatic functional categories: tool shed
> repositories which define a format, or define a tool for the user,
> or define a dependency for use by other tools, etc
> 
> Thanks,
> 
> Peter
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Galaxy hg repository problem "abort: path ... traverses symbolic link"

[Peter Cock]

Hi all,

I've got an hg puzzle, can anyone on the Galaxy team explain the
cause of this and what action I can take to avoid it?

Starting with a clean checkout,

$ hg clone https://bitbucket.org/galaxy/galaxy-central
warning: bitbucket.org certificate with fingerprint
24:9c:45:8b:9c:aa:ba:55:4e:01:6d:58:ff:e4:28:7d:2a:14:ae:3b not
verified (check hostfingerprints or web.cacerts config setting)
destination directory: galaxy-central
requesting all changes
adding changesets
adding manifests
adding file changes
added 8886 changesets with 33822 changes to 6914 files (+1 heads)
updating to branch default
4019 files updated, 0 files merged, 0 files removed, 0 files unresolved

$ cd galaxy-central

$ hg branch
default

$ hg branches
default 8885:9852dd712f5c
stable  8880:8da37d3985ab

$ hg checkout stable
abort: path 'static/june_2007_style/Makefile' traverses symbolic link
'static/june_2007_style'

I was actually trying to checkout one of my own branches, having
setup the .hg/hgrc to include a link to my own fork on bitbucket.

Thanks!

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Searching tool shed for file format definitions?

[Peter Cock]

Hello all,

I was wondering how people keep track of the available file format
definitions in the tool shed(s)? For core file types defined in Galaxy,
I can just view the datatypes_conf.xml file and/or the classes in
lib/galaxy/datatypes/ - but how can I do something similar on a
Galaxy Tool Shed?

One solution might be automatic functional categories: tool shed
repositories which define a format, or define a tool for the user,
or define a dependency for use by other tools, etc

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] shall we need to download dataset when we install our local Galaxy?

[Dannon Baker]

Hi!

You'll probably find this page helpful in configuring your local
instance with tools and references:
http://wiki.galaxyproject.org/Admin/NGS%20Local%20Setup

In short, yes, to use these references locally you'll need to retrieve
them and configure your galaxy to use them.

-Dannon


On Mon, Feb 18, 2013 at 2:17 AM, 沈维燕  wrote:
> hi:
>When we running a local installation of Galaxy,there are some reference
> dataset ,such as Human Feb. 2009 (GRCh37/hg19) (hg19),A. gambiae Feb. 2003
> (IAGEC MOZ2/anoGam1) (anoGam1),and so on.Shall we download these datasets
> and save in our computer ?
>   This is my first use Galaxy,it is very good and iteresting.Thank you very
> much for replying.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>   http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] shall we need to download dataset when we install our local Galaxy?

[沈维燕]

hi:
   When we running a local installation of Galaxy,there are some reference
dataset ,such as Human Feb. 2009 (GRCh37/hg19) (hg19),A. gambiae Feb. 2003
(IAGEC MOZ2/anoGam1) (anoGam1),and so on.Shall we download these datasets
and save in our computer ?
  This is my first use Galaxy,it is very good and iteresting.Thank you very
much for replying.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Deploying LOC files for tool built-in data during a tool installation

[Greg Von Kuster]

Hello Jean-Frédéric,

Sorry for the delay in this response.  Please see my inline comments.

On Feb 8, 2013, at 10:33 AM, Jean-Frédéric Berthelot wrote:

> Hi list,
> 
> The tool I am currently wrapping has built-in data, which may be used by the 
> tool users (through a relevant  + .LOC file configuration).
> They are .fasta databases which are rather small and are thus bundled in the 
> tool distribution package.
> 
> Thanks to the tool_dependencies.xml file, said distribution package is 
> downloaded at install time, code is compiled, and since they are here, the 
> data files are copied to $INSTALL_DIR too, ready to be used.
> 
> After that, the user still has to edit tool-data/my_fancy_data_files.loc ; 
> but the thing is, during the install I know where these data files are (since 
> I copied those there), so I would like to save the user the trouble and set 
> up this file automagically.
> 
> I would have two questions:
> 
> 1/ Is it okay to have tool built-in data files in $INSTALL_DIR, or would it 
> be considered bad practice?


This is difficult to answer.  Generally, data files should be located in a 
shared location so that other tools can access them as well.  However, there 
are potentially exceptions to this that are acceptable.  The fact that the 
fasta data files are small and you are using a tool_dependencies.xml file to 
define a relationship to them for your tools is a good approach because it 
allows the data files to be used by other tools in separate repositories via a 
complex repository dependency definition in the remote repository.

If these fasta data files are available for download via a clone or a url, then 
in the near future the new Galaxy Data Manager (which uses a new, special 
category of Galaxy tools which are of type "data_manager") may be useful in 
this scenario.  Data Manager tools can be associated with tools in a repository 
like yours using repository dependency definitions, so they will be installed 
along with the selected repository.  These data manager tools allow for 
specified data to be installed into the Galaxy environment for use by tools.  
This new component is not yet released, but it is close.  In the meantime, your 
approach is the only way to make this work.  

If your files are not downloadable, then we might plan to allow simplified 
bootstrapping of .loc files in the tol-data directory with files included in 
the repository.  This would take some planning, and it's availability would not 
be in the short term


> 
> 2/ Is there a way to set up the tool-data/my_fancy_data_files.loc during the 
> install? Here are the options I though of:
> *shipping a “real” my_fancy_data_files.loc.sample with the good paths already 
> set-up, which is going to be copied as the .loc file (a rather ugly hack)

Assuming you use a file name that is not already in the Galaxy tool-data 
subdirectory, the above approach is probably the only way you can do this in a 
fully automated right now.  Again, when the new Data Manager is released, it 
will handle this kind of automated configuration.  But in the meantime, manual 
intervention is generally required to add the information to appropriate .loc 
files in the tool-data directory.


> *using more  during install to create 
> my_fancy_data_files.loc (but deploying this file it is not part of the tool 
> dependency install per se)

I advise against the above approach.  The "best practice" use of tool 
dependency definitions is to restrict movement of files to location within the 
defined $INSTALL_DIR (the installation directory of the tol dependency package) 
or $REPOSITORY_INSTALL_DIR (the installation directory of the repository), 
which is set at installation time.  Hard-coding file paths in  tags is 
fragile, and not recommeded.


> *variant of the previous : shipping my_fancy_data_files.loc as part of the 
> tool distribution package, and copy it through shell_command (same concern 
> than above).

The above approach is not recommended either - same issue as above.

> 
> Any thoughts?
> 
> Cheers,
> 
> -- 
> Jean-Frédéric
> Bonsai Bioinformatics group

Thanks very much Jean-Frédéric,

Greg Von Kuster

> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[Peter Cock]

On Tue, Feb 19, 2013 at 2:00 PM, James Taylor  wrote:
> On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock  wrote:
>> Would a pull request implementing this be acceptable?
>
> Yes. My understanding is that ASN is a completely flexible metaformat,
> like XML, and so should be under either Text or Data, with appropriate
> subtypes defined for blast, et cetera.

Thank James,

Yes - very like XML, but with the subtlety that ASN.1 comes in text and
binary favours (which I presume applies to all the variants, although
the binary versions may not be as commonly used for the smaller files).

Nicola - do you want to make a pull request to galaxy-central defining
ASN.1 text and binary formats (which we can then subclass for the
NCBI BLAST+ wrappers)? Or should I?

I think the mime-type for the base ASN.1 text format should probably
be text/plain based on the NCBI usage patterns.

I'm not sure what the mime-type for the base ASN.1 binary format
should be (but I don't think it should be chemical/ncbi-asn1-binary).

Regards,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Tool wrapper XSD

[John Chilton]

Hey Pierre,

If you are going to write one and want a starting point, here is the
schema I used for incorporating Galaxy tools into a Java web service
workflow framework called TINT:

https://github.com/jmchilton/TINT/blob/master/projects/TropixGalaxy/schema/galaxy.xsd

It is mostly a subset of actual valid tools (even more so because of
how dated this has become), but it has some extensions for storing
literal test results right in the test XML and literal configfiles
(i.e. non-cheetah) that you will probably want to rip out.

-John

On Tue, Feb 19, 2013 at 2:39 AM, Pierre Pericard
 wrote:
> Hi everyone,
>
> Is there a Galaxy XML tool wrapper XSD ?
>
> Thanks,
> Pierre
>
>
> --
> Pierre Pericard
> IE CDD - Projet Peptisan
> Service Informatique et Bio-informatique (SIB)
>
> Station Biologique de Roscoff
> CNRS - UPMC
> Place Georges Teissier
> CS 90074
> 29688 ROSCOFF CEDEX
> FRANCE
> Tel : (+33) 2 98 29 56 46
> http://abims.sb-roscoff.fr/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>  http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Tool wrapper XSD

[James Taylor]

On Tue, Feb 19, 2013 at 3:39 AM, Pierre Pericard
 wrote:
> Is there a Galaxy XML tool wrapper XSD ?


No, but if anyone would like to make such a schema we would be happy
to include it.

--
James Taylor, Assistant Professor, Biology/CS, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[James Taylor]

On Tue, Feb 19, 2013 at 6:32 AM, Peter Cock  wrote:
> Would a pull request implementing this be acceptable?


Yes. My understanding is that ASN is a completely flexible metaformat,
like XML, and so should be under either Text or Data, with appropriate
subtypes defined for blast, et cetera.

--
James Taylor, Assistant Professor, Biology/CS, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Merge internal head to release_2012.02.08

[Nate Coraor]

Thanks Dannon.  One more tip:  If you commit your changes on the default 
branch, you will need to merge them with stable, i.e. `hg update 
release_2013.02.08` followed by `hg merge `

--nate

On Feb 19, 2013, at 7:27 AM, Dannon Baker wrote:

> Derrick,
> 
> The new update process outlined works with local changes the same way
> as the 'hg pull -u' used previously.  The only difference is that it
> updates to a particular tag, instead of whatever is at the galaxy-dist
> tip.  'hg pull -u' that you've used before is actually just shorthand
> for doing the two common commands at once, 'hg pull; hg update'.
> 'update' will only discard local changes when executed with the '-C'
> option, otherwise it'll try to merge.
> 
> If you're concerned about persisting local changes, it's probably a
> good idea to go ahead and commit them to your local repository or at
> least create a patch that you could use to recreate the changes `hg
> diff -p > our_local_changes.patch`.
> 
> -Dannon
> 
> On Mon, Feb 18, 2013 at 11:36 PM, Derrick Lin  wrote:
>> Hi guys,
>> 
>> We have checked in few internal changesets to our galaxy. Normally we did hg
>> pull then hg merge for upgrading.
>> 
>> With the 8 Feb release, it advice that we must do hg update
>> release_2013.02.08 if we want to upgrade from stable branch.
>> 
>> My understanding for hg update is, it will overwrite all local changes which
>> we want to avoid.
>> 
>> So in our case, what's the best way for us to upgrade?
>> 
>> Cheers,
>> Derrick
>> 
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> 
>>  http://lists.bx.psu.edu/
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>  http://lists.bx.psu.edu/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Merge internal head to release_2012.02.08

[Dannon Baker]

Derrick,

The new update process outlined works with local changes the same way
as the 'hg pull -u' used previously.  The only difference is that it
updates to a particular tag, instead of whatever is at the galaxy-dist
tip.  'hg pull -u' that you've used before is actually just shorthand
for doing the two common commands at once, 'hg pull; hg update'.
'update' will only discard local changes when executed with the '-C'
option, otherwise it'll try to merge.

If you're concerned about persisting local changes, it's probably a
good idea to go ahead and commit them to your local repository or at
least create a patch that you could use to recreate the changes `hg
diff -p > our_local_changes.patch`.

-Dannon

On Mon, Feb 18, 2013 at 11:36 PM, Derrick Lin  wrote:
> Hi guys,
>
> We have checked in few internal changesets to our galaxy. Normally we did hg
> pull then hg merge for upgrading.
>
> With the 8 Feb release, it advice that we must do hg update
> release_2013.02.08 if we want to upgrade from stable branch.
>
> My understanding for hg update is, it will overwrite all local changes which
> we want to avoid.
>
> So in our case, what's the best way for us to upgrade?
>
> Cheers,
> Derrick
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>   http://lists.bx.psu.edu/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] environment variables and paths for toolshed tools

[Peter Cock]

On Tue, Feb 19, 2013 at 11:39 AM, Greg Von Kuster  wrote:
> On Feb 19, 2013, at 5:10 AM, Peter Cock wrote:
>
> Yes, the complete directory hierarchy and content of the original
> repository revision is preserved when it installed into Galaxy.
>
>> Based on a sample of one, it seems the installation process just
>> unpacks the tool shed files (well, probably using hg rather than
>> unpacking a tar-ball), and preserves their relative path structure.
>> If so, I can just use relative paths to find a data file from the tool
>> executable's own location on disk. (This is what I was hoping
>> would be the case - I'm looking for explicit confirmation).
>
> This is correct, hg clone provides this behavior.
>
>> i.e. For the example use case, if motif.py and motif.dat are in the
>> same folder in the Tool Shed upload, they will be in the same folder
>> as each other once installed. That way motif.py can easily locate the
>> data file motif.dat by looking in the same folder as it itself is located.
>> This is actually very simple (provided I can assume the local folder
>> structure is maintained).
>
> Yes, this is correct.

Excellent - that solves this use-case nicely.

>> (I think I was originally on the wrong track by assuming I should
>> be using the tool-data folder, which is complicated by not knowing
>> where that will existing on disk).
>
> I think your request is still valid though and we will add this
> enhancement for the Galaxy tool component to handle
> $REPOSITORY_INSTALL_DIR in tool configs.  In the meantime,
> it looks like you can work around this missing feature.  Please let
> me know if you bump into issues.

I can still see potential uses for $REPOSITORY_INSTALL_DIR
both in the tool XML language, and as an environment variable -
but I don't need that for the time being.

Thanks Greg, all sorted for now,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] environment variables and paths for toolshed tools

[Greg Von Kuster]

Hi Peter,  see below...

On Feb 19, 2013, at 5:10 AM, Peter Cock wrote:

> On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster  wrote:
>> Hi Peter,
>> 
>> Thanks for your thoughts on this.  I've created the following Trello card 
>> for this enhancement.
>> 
>> https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636
>> 
>> The priority for the tool shed in the area of tool dependencies like this 
>> has been
>> the implementation of features that enable sharing a single dependency across
>> multiple tools in separate repositories, so your scenario is not yet 
>> supported.
>> 
>> This enhancement would actually be as much (or perhaps more) on the Galaxy
>> end than the tool shed, I think, so there may be more resources available to 
>> get
>> to it sooner than I can right now.  Of course, someone wil get to it as soon 
>> as
>> possible.
>> 
>> Thanks again for all of your feedback and insight on this.
>> 
>> Greg Von Kuster
> 
> Hi Greg,
> 
> I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy
> which is targeted at Galaxy administrators rather than tools authors.
> It explains how a revision specific folder is assigned to each installed
> tool, but what (if anything) of the Tool's folder structure is preserved?

The installation process uses the mercurial API, so the process to install a 
specific repository changeset_revision ultimately includes the following 2 hg 
commands.

hg clone 
hg update -r changeset_revision

This results in the structure of the installed repository being exactly the 
same of the specific revision from which the repository was cloned.

> 
> e.g. One of the paths given is:
> 
> .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml
> 
> The prefix is the revision specific path for that tool for that Tool Shed,
> 
> .../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/
> 
> Within that, it seems to preserve the emboss_datatypes folders. i.e.
> goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes
> and click browse repository tip files from the top left actions menu,
> it has just one file: emboss_datatypes/datatypes_conf.xml

Yes, the complete directory hierarchy and content of the original repository 
revision is preserved when it installed into Galaxy.

> 
> Based on a sample of one, it seems the installation process just
> unpacks the tool shed files (well, probably using hg rather than
> unpacking a tar-ball), and preserves their relative path structure.
> If so, I can just use relative paths to find a data file from the tool
> executable's own location on disk. (This is what I was hoping
> would be the case - I'm looking for explicit confirmation).

This is correct, hg clone provides this behavior.

> 
> i.e. For the example use case, if motif.py and motif.dat are in the
> same folder in the Tool Shed upload, they will be in the same folder
> as each other once installed. That way motif.py can easily locate the
> data file motif.dat by looking in the same folder as it itself is located.
> This is actually very simple (provided I can assume the local folder
> structure is maintained).

Yes, this is correct.  

> 
> (I think I was originally on the wrong track by assuming I should
> be using the tool-data folder, which is complicated by not knowing
> where that will existing on disk).

I think your request is still valid though and we will add this enhancement for 
the Galaxy tool component to handle $REPOSITORY_INSTALL_DIR in tool configs.  
In the meantime, it looks like you can work around this missing feature.  
Please let me know if you bump into issues.

> 
> Thanks,
> 
> Peter


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] ASN.1 (text and binary) formats in Galaxy & Tool Shed

[Peter Cock]

Hello all,

Although they are these days also offering XML for many tools,
the NCBI still make heavy use of the older ASN.1 file format
(both as plain text and binary). This crops up in BLAST (e.g.
as the BLAST archive format, or as dustmasker output), in
the Entrez Utilities (e.g. for sequence data as an alternative
to GenBank for FASTA format etc, or pubmed, etc) and also
for 3D structures.

I think it could make sense to define generic 'asn1' and
'asn1-binary' formats in the Galaxy core (name suggestions
welcome), and even perhaps 'ncbi-asn1' and 'ncbi-asn1-binary'
too. Then ToolShed entries can define domain specific
subclasses. For instance, the BLAST+ wrapper could include
definitions for the dustmasker output, and perhaps the BLAST
archive format too. Separately anyone working with 3D
structures as ASN.1 could define another sub-format, etc.

I see this as a clear analogy to the assorted XML file formats
in existence - defined in Galaxy as subclasses of the core
XML format included with the Galaxy core.

Would a pull request implementing this be acceptable?

Peter

P.S. Does anyone know an authoritative source for the MIME
types used by the NCBI? Using the BLAST website they
offer plain text ASN.1 just as text/plain, likewise efetch also
seems to use text/plain for ASN.1 downloads. However I've
seen references to chemical/ncbi-asn1-ascii and
chemical/ncbi-asn1-binary mime-types mentioned, e.g.
http://www.ncbi.nlm.nih.gov/data_specs/asn/NCBI_all.asn

i.e. It appears that 3D structure NCBI ASN.1 files use
a well defined MIME type, while most NCBI ASN.1 text
files default to text/plain - which we can handle nicely in
Galaxy as subclasses.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] environment variables and paths for toolshed tools

[Peter Cock]

On Sat, Feb 16, 2013 at 9:41 PM, Greg Von Kuster  wrote:
> Hi Peter,
>
> Thanks for your thoughts on this.  I've created the following Trello card for 
> this enhancement.
>
> https://trello.com/card/galaxy-tool-enhancement-to-accommodate-repository-install-dir/506338ce32ae458f6d15e4b3/636
>
> The priority for the tool shed in the area of tool dependencies like this has 
> been
> the implementation of features that enable sharing a single dependency across
> multiple tools in separate repositories, so your scenario is not yet 
> supported.
>
> This enhancement would actually be as much (or perhaps more) on the Galaxy
> end than the tool shed, I think, so there may be more resources available to 
> get
> to it sooner than I can right now.  Of course, someone wil get to it as soon 
> as
> possible.
>
> Thanks again for all of your feedback and insight on this.
>
> Greg Von Kuster

Hi Greg,

I've read http://wiki.galaxyproject.org/InstallingRepositoriesToGalaxy
which is targeted at Galaxy administrators rather than tools authors.
It explains how a revision specific folder is assigned to each installed
tool, but what (if anything) of the Tool's folder structure is preserved?

e.g. One of the paths given is:

.../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/emboss_datatypes/datatypes_conf.xml

The prefix is the revision specific path for that tool for that Tool Shed,

.../shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/emboss_datatypes/a89163f31369/

Within that, it seems to preserve the emboss_datatypes folders. i.e.
goto http://toolshed.g2.bx.psu.edu/view/devteam/emboss_datatypes
and click browse repository tip files from the top left actions menu,
it has just one file: emboss_datatypes/datatypes_conf.xml

Based on a sample of one, it seems the installation process just
unpacks the tool shed files (well, probably using hg rather than
unpacking a tar-ball), and preserves their relative path structure.
If so, I can just use relative paths to find a data file from the tool
executable's own location on disk. (This is what I was hoping
would be the case - I'm looking for explicit confirmation).

i.e. For the example use case, if motif.py and motif.dat are in the
same folder in the Tool Shed upload, they will be in the same folder
as each other once installed. That way motif.py can easily locate the
data file motif.dat by looking in the same folder as it itself is located.
This is actually very simple (provided I can assume the local folder
structure is maintained).

(I think I was originally on the wrong track by assuming I should
be using the tool-data folder, which is complicated by not knowing
where that will existing on disk).

Thanks,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] all the data files in .loc files needed to download ?

[shenwiyn]

Hi all,
Are all the data files in .loc files needed to download and put to the right 
directory when we install our local galaxy?

thaks
shenwiyn




shenwiyn___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

[galaxy-dev] Tool wrapper XSD

[Pierre Pericard]

Hi everyone,

Is there a Galaxy XML tool wrapper XSD ?

Thanks,
Pierre


--
Pierre Pericard
IE CDD - Projet Peptisan
Service Informatique et Bio-informatique (SIB)

Station Biologique de Roscoff
CNRS - UPMC
Place Georges Teissier
CS 90074
29688 ROSCOFF CEDEX
FRANCE
Tel : (+33) 2 98 29 56 46
http://abims.sb-roscoff.fr/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] changing datasets file_path storage location

[Peter Cock]

On Tuesday, February 19, 2013, Derrick Lin wrote:

> Thanks Peter,
>
> Our data is backup and we are planning to shutdown the galaxy for moving
> data to the new place, change the file_path value in universe config, then
> start the galaxy again.
>
> My only concern is if the existing value of file_path get hard-coded in
> the database entries.
>
> Many thanks
> Derrick
>
>
That's what I was worried about - but it seems everything was using
relative paths :)

Good luck,

Peter
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/