[galaxy-dev] last hurdle

[Robert Day]

Hi, 
  Sorry to bother you guys with such a simple thing. I'm quite new to the 
terminal and command line etc. so downloading and getting Galaxy running on my 
PC was a voyage of discovery! Anyhow, I have it, it's alive.. just 
can't get any data in!

Have made myself an Admin User to create a data library, I point the importer 
at the files and it starts but never stops and remains at 0bytes. Have I 
forgotten to set something up?

Surely I can just dump all my files in a folder in the right place and go get 
it when I need it. Can't seem to find an idiots guide for setting this up for a 
local install (just me and my computer). Any pointers or helpful threads would 
be great.

Cheers Rob
Sent from my PC
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Re Acutal user code

[Chorny, Ilya]

One more thing there are some changes in lib/galaxy/tools/__init__.py and 
lib/galaxy/model/__init__.py that are not related to actual user. They look 
like they are toolshed related and may have been changed because to version of 
the file that I copied over was older then galaxy-central. Probably should have 
done a merge first. I can redo it if you like. Let me know.

Ilya


-Original Message-
From: Chorny, Ilya 
Sent: Friday, October 21, 2011 2:11 PM
To: Chorny, Ilya; Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: RE: Re Acutal user code

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, October 21, 2011 2:07 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
Sent: Friday, October 21, 2011 12:56 PM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: Re Acutal user code

Chorny, Ilya wrote:
> Nate,
> 
> I have been trying to figure out how to make the changes you suggested (i.e. 
> write output files to the job_working_directory) and then move them over to 
> files but I am having a difficult time navigating the code.  If you could 
> point me in the right direction, that would be helpful.  I know the 
> lib/galaxy/models/__init__.py initializes the output files but I don't think 
> that is the right place to make the change. This change probably needs to be 
> done in the Job_Wrapper but most of the code is very cryptic to me. Any 
> guidance would be greatly appreciated. These changes should not take very 
> long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the 
other code in your fork.  Could you create a fork with just this code?

Thanks,
--nate

> 
> Thanks,
> 
> Ilya
> 
> Ilya Chorny Ph.D.
> Bioinformatics Scientist I
> Illumina, Inc.
> 9885 Towne Centre Drive
> San Diego, CA 92121
> Work: 858.202.4582
> Email: icho...@illumina.com
> Website: www.illumina.com
> 
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this and other Galaxy 
lists, please use the interface at:

  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Re Acutal user code

[Chorny, Ilya]

Sorry, the link was wrong.

https://bitbucket.org/ichorny/galaxy-central-ilya2/

-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, October 21, 2011 2:07 PM
To: Nate Coraor (n...@bx.psu.edu)
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Re Acutal user code

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu]
Sent: Friday, October 21, 2011 12:56 PM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: Re Acutal user code

Chorny, Ilya wrote:
> Nate,
> 
> I have been trying to figure out how to make the changes you suggested (i.e. 
> write output files to the job_working_directory) and then move them over to 
> files but I am having a difficult time navigating the code.  If you could 
> point me in the right direction, that would be helpful.  I know the 
> lib/galaxy/models/__init__.py initializes the output files but I don't think 
> that is the right place to make the change. This change probably needs to be 
> done in the Job_Wrapper but most of the code is very cryptic to me. Any 
> guidance would be greatly appreciated. These changes should not take very 
> long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the 
other code in your fork.  Could you create a fork with just this code?

Thanks,
--nate

> 
> Thanks,
> 
> Ilya
> 
> Ilya Chorny Ph.D.
> Bioinformatics Scientist I
> Illumina, Inc.
> 9885 Towne Centre Drive
> San Diego, CA 92121
> Work: 858.202.4582
> Email: icho...@illumina.com
> Website: www.illumina.com
> 
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this and other Galaxy 
lists, please use the interface at:

  http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] update of cufflinks and tophat to use dbkey attribute of input files

[Chorny, Ilya]

Can you guys show me an example of how this is  used in the xml file?

Thanks,

Ilya


From: Jeremy Goecks [mailto:jeremy.goe...@emory.edu]
Sent: Thursday, October 20, 2011 7:03 PM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: update of cufflinks and tophat to use dbkey attribute of input 
files

I made some more modifications to the wrappers to allow for the tools to pull 
the genome information from the input files. The only drawback is that there is 
no way to validate whether the input has the dbkey attribute as a parameter 
(can you suggest a way to validate?). Attached are the diff's to the current 
version in galaxy-central.

See Dan's recent commit for validation via data tables:

https://bitbucket.org/galaxy/galaxy-central/changeset/150069a23aad

Thanks,
J.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Re Acutal user code

[Chorny, Ilya]

https://bitbucket.org/ichorny/galaxy-central-ilya2/pull-request/new

everything but the tophat/cufflinks wrapper.

Thanks,

Ilya


-Original Message-
From: Nate Coraor (n...@bx.psu.edu) [mailto:n...@bx.psu.edu] 
Sent: Friday, October 21, 2011 12:56 PM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: Re Acutal user code

Chorny, Ilya wrote:
> Nate,
> 
> I have been trying to figure out how to make the changes you suggested (i.e. 
> write output files to the job_working_directory) and then move them over to 
> files but I am having a difficult time navigating the code.  If you could 
> point me in the right direction, that would be helpful.  I know the 
> lib/galaxy/models/__init__.py initializes the output files but I don't think 
> that is the right place to make the change. This change probably needs to be 
> done in the Job_Wrapper but most of the code is very cryptic to me. Any 
> guidance would be greatly appreciated. These changes should not take very 
> long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from the 
other code in your fork.  Could you create a fork with just this code?

Thanks,
--nate

> 
> Thanks,
> 
> Ilya
> 
> Ilya Chorny Ph.D.
> Bioinformatics Scientist I
> Illumina, Inc.
> 9885 Towne Centre Drive
> San Diego, CA 92121
> Work: 858.202.4582
> Email: icho...@illumina.com
> Website: www.illumina.com
> 
> 

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Re Acutal user code

[Nate Coraor (n...@bx.psu.edu)]

Chorny, Ilya wrote:
> Nate,
> 
> I have been trying to figure out how to make the changes you suggested (i.e. 
> write output files to the job_working_directory) and then move them over to 
> files but I am having a difficult time navigating the code.  If you could 
> point me in the right direction, that would be helpful.  I know the 
> lib/galaxy/models/__init__.py initializes the output files but I don't think 
> that is the right place to make the change. This change probably needs to be 
> done in the Job_Wrapper but most of the code is very cryptic to me. Any 
> guidance would be greatly appreciated. These changes should not take very 
> long to implement once I get a handle on the code.

Hi Ilya,

I'm having a lot of difficulty separating out the actual user code from
the other code in your fork.  Could you create a fork with just this
code?

Thanks,
--nate

> 
> Thanks,
> 
> Ilya
> 
> Ilya Chorny Ph.D.
> Bioinformatics Scientist I
> Illumina, Inc.
> 9885 Towne Centre Drive
> San Diego, CA 92121
> Work: 858.202.4582
> Email: icho...@illumina.com
> Website: www.illumina.com
> 
> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Tool shed and datatypes

[Jim Johnson]

On 10/21/11 12:29 PM, James Taylor wrote:

Excerpts from Jim Johnson's message of 2011-10-21 17:13:02 +:

I put the gmap tool suite in the galaxy Tool Shed,  let me know if there is 
more I should do.

Awesome!


I added a requirement tag for the datatypes to the tool-configs:

 % grep 'requirement.*datatype' *.xml
 gmap_build.xml:gmapdb

Requirement tags for datatypes are an interesting idea, but I'm
wondering if this is something we should require? It seems like all this
information is implicit -- a tool requires a datatype if it has an input
or output parameter that references that type. Is there other
information that should go in the requirement tag?


That is certainly correct that the tag would be redundant, the tool config 
parser could identify the list of datatype formats.

I was just trying to think of some way to indicate that additional datatypes 
were required above those in the central distribution.
My goal would be to have the installation of tools from the Tool Shed also be 
able to install the extra datatypes that those tools require.

Having datatypes specified separately in the Tool Shed from tools would 
hopefully promote less redundancy of datatypes and better interoperability 
among developers tools.For example the metagenomics applications mothur and 
qiime have many specific formats that are internal to their tools, but also a 
few that might be used to migrate data between those applications.   We'd need 
a way to avoid name clashes, perhaps adopting a namespace pattern for the 
file_ext attribute.


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

 http://lists.bx.psu.edu/

Re: [galaxy-dev] Tool shed and datatypes

[Greg Von Kuster]

Thanks Jim,

This is on my development plan, but it may take a few days for me to get 
heavily into it.  I'll get back to you hopefully some time next week.

Greg

On Oct 21, 2011, at 1:13 PM, Jim Johnson wrote:

> Greg,
> 
> I put the gmap tool suite in the galaxy Tool Shed,  let me know if there is 
> more I should do.  
>   
> It has 5 galaxy tools:
> GMAP   -  Genomic Mapping and Alignment Program for mRNA and EST 
> sequences 
> GSNAP- Genomic Short-read Nucleotide Alignment Program   
> GMAP Build-  a database genome index for GMAP and GSNAP ( calls:  
> gmap_build, iit_store, snpindex, cmetindex, atoiindex ) 
> GMAP SNP Index- build index files for known SNPs 
> (calls:  iit_store, snpindex) 
> GMAP IIT- Create a map store for known genes or SNPs  
> (calls:  iit_store) 
> 
> It uses these added datatypes:
> % grep -E '(^class | file_ext)' lib/galaxy/datatypes/gmap.py 
> class GmapDB( Text ):
> file_ext = 'gmapdb'
> class GmapSnpIndex( Text ):
> file_ext = 'gmapsnpindex'
> class IntervalIndexTree( Text ):
> file_ext = 'iit'
> class SpliceSitesIntervalIndexTree( IntervalIndexTree ):
> file_ext = 'splicesites.iit'
> class IntronsIntervalIndexTree( IntervalIndexTree ):
> file_ext = 'introns.iit'
> class SNPsIntervalIndexTree( IntervalIndexTree ):
> file_ext = 'snps.iit'
> class IntervalAnnotation( Text ):
> file_ext = 'gmap_annotation'
> class SpliceSiteAnnotation(IntervalAnnotation):
> file_ext = 'gmap_splicesites'
> class IntronAnnotation(IntervalAnnotation):
> file_ext = 'gmap_introns'
> class SNPAnnotation(IntervalAnnotation):
> file_ext = 'gmap_snps'
> 
> I added a requirement tag for the datatypes to the tool-configs:
> % grep 'requirement.*datatype' *.xml
> gmap_build.xml:  gmapdb
> gmap_build.xml:  gmap_snps
> gmap.xml:gmapdb
> gmap.xml:gmap_annotation
> gmap.xml:gmap_splicesites
> gmap.xml:gmap_introns
> gmap.xml:gmap_snps
> gsnap.xml:  gmapdb
> gsnap.xml:  gmapsnpindex
> gsnap.xml:  splicesites.iit
> gsnap.xml:  introns.iit
> iit_store.xml:  gmap_annotation
> iit_store.xml:  gmap_snps
> iit_store.xml:  iit
> iit_store.xml:  splicesites.iit
> iit_store.xml:  introns.iit
> iit_store.xml:  snps.iit
> snpindex.xml:  gmapsnpindex
> snpindex.xml:  gmapdb
> snpindex.xml:  gmap_snps
> snpindex.xml:  snps.iit
> 
> Thanks,
> 
> JJ
> 
> 
> On 10/18/11 10:18 AM, Greg Von Kuster wrote:
>> 
>> Jim,
>> 
>> Sounds great - this will be very helpful!
>> 
>> Greg
>> 
>> On Oct 18, 2011, at 11:03 AM, Jim Johnson wrote:
>> 
>>> Greg,
>>> 
>>> The mothur_toolsuite in the ToolShed  contains a file with added datatypes 
>>> for metagenomics (used by mothur and some by qiime):
>>> mothur_toolsuite/mothur/lib/galaxy/datatypes/metagenomics.py
>>> The README has info on how I incorporated mothur into our local galaxy 
>>> server.
>>> 
>>> I'm also working on GMAP/GSNAP  (  http://research-pub.gene.com/gmap/ )
>>> So far I've created a GmapDB class,  analogous to the ngsindex.BowtieIndex 
>>> class, but with more metadata.
>>> I'm also adding a IntervalIndexTree class for indexing maps of splice 
>>> junctions, introns, and SNPs.
>>> I'll send you this as soon as I've got it working.
>>> 
>>> Thanks,
>>> 
>>> JJ
>>> 
>>> 
>>> On 10/17/11 1:06 PM, Greg Von Kuster wrote:
 We've digested this topic a bit here at Galaxy Central, and agree that at 
 some point ( maybe soon for very basic functionality ) we need to provide 
 support for new data types in tool shed repositories.  It would be very 
 helpful ( and significantly speed up the development process ) if the 
 community could provide at least 2 different tools that use data types not 
 included in the Galaxy distribution ( sending me a tarball that includes 
 all the tool dependencies, including the new data type class would be 
 ideal ).  When I get them I'll add this new feature set to my development 
 list.
 
 Thanks everyone for all the input on this!
 
 Greg Von Kuster
 
 
 On Oct 7, 2011, at 2:05 PM, Jim Johnson wrote:
 
> Greg,
> 
> It would be great if there were a way to expand upon the core datatypes 
> using the ToolShed.
> 
> Would it be possible to have a separate datatype repository within the 
> ToolShed?
> 
> Datatype
>  name=""
>  description=""
>  datatype_dependencies=[]
>  definition=
> 
> The tool config could be expanded to have requirement for datatypes.
>   ssmap
> 
> 
> 
> 
> Table datatype
>   Column|Type | Modifiers
> -+-+---
> id  | integer | not null default 
> nextval('datatype_id_seq'::regclass)
> name| c

Re: [galaxy-dev] Tool shed and datatypes

[James Taylor]

Excerpts from Jim Johnson's message of 2011-10-21 17:13:02 +:
> I put the gmap tool suite in the galaxy Tool Shed,  let me know if there is 
> more I should do.

Awesome!

> I added a requirement tag for the datatypes to the tool-configs:
> 
> % grep 'requirement.*datatype' *.xml
> gmap_build.xml: gmapdb

Requirement tags for datatypes are an interesting idea, but I'm
wondering if this is something we should require? It seems like all this
information is implicit -- a tool requires a datatype if it has an input
or output parameter that references that type. Is there other
information that should go in the requirement tag? 

-- 
James Taylor, Assistant Professor, Biology / Computer Science, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Tool shed and datatypes

[Jim Johnson]

Greg,

I put the gmap tool suite in the galaxy Tool Shed,  let me know if there is 
more I should do.

It has 5 galaxy tools:

GMAP   -  Genomic Mapping and Alignment Program for mRNA and EST 
sequences
GSNAP- Genomic Short-read Nucleotide Alignment Program
GMAP Build-  a database genome index for GMAP and GSNAP ( 
calls:  gmap_build, iit_store, snpindex, cmetindex, atoiindex )
GMAP SNP Index- build index files for known SNPs
 (calls:  iit_store, snpindex)
GMAP IIT- Create a map store for known genes or SNPs  
(calls:  iit_store)


It uses these added datatypes:

   % grep -E '(^class | file_ext)' lib/galaxy/datatypes/gmap.py
   class GmapDB( Text ):
file_ext = 'gmapdb'
   class GmapSnpIndex( Text ):
file_ext = 'gmapsnpindex'
   class IntervalIndexTree( Text ):
file_ext = 'iit'
   class SpliceSitesIntervalIndexTree( IntervalIndexTree ):
file_ext = 'splicesites.iit'
   class IntronsIntervalIndexTree( IntervalIndexTree ):
file_ext = 'introns.iit'
   class SNPsIntervalIndexTree( IntervalIndexTree ):
file_ext = 'snps.iit'
   class IntervalAnnotation( Text ):
file_ext = 'gmap_annotation'
   class SpliceSiteAnnotation(IntervalAnnotation):
file_ext = 'gmap_splicesites'
   class IntronAnnotation(IntervalAnnotation):
file_ext = 'gmap_introns'
   class SNPAnnotation(IntervalAnnotation):
file_ext = 'gmap_snps'


I added a requirement tag for the datatypes to the tool-configs:

   % grep 'requirement.*datatype' *.xml
   gmap_build.xml: gmapdb
   gmap_build.xml: gmap_snps
   gmap.xml: gmapdb
   gmap.xml: gmap_annotation
   gmap.xml: gmap_splicesites
   gmap.xml: gmap_introns
   gmap.xml: gmap_snps
   gsnap.xml: gmapdb
   gsnap.xml: gmapsnpindex
   gsnap.xml: splicesites.iit
   gsnap.xml: introns.iit
   iit_store.xml: gmap_annotation
   iit_store.xml: gmap_snps
   iit_store.xml: iit
   iit_store.xml: splicesites.iit
   iit_store.xml: introns.iit
   iit_store.xml: snps.iit
   snpindex.xml: gmapsnpindex
   snpindex.xml: gmapdb
   snpindex.xml: gmap_snps
   snpindex.xml: snps.iit


Thanks,

JJ


On 10/18/11 10:18 AM, Greg Von Kuster wrote:

Jim,

Sounds great - this will be very helpful!

Greg

On Oct 18, 2011, at 11:03 AM, Jim Johnson wrote:


Greg,

The mothur_toolsuite in the ToolShed  contains a file with added datatypes for 
metagenomics (used by mothur and some by qiime):
mothur_toolsuite/mothur/lib/galaxy/datatypes/metagenomics.py
The README has info on how I incorporated mothur into our local galaxy server.

I'm also working on GMAP/GSNAP  (  http://research-pub.gene.com/gmap/ )
So far I've created a GmapDB class,  analogous to the ngsindex.BowtieIndex 
class, but with more metadata.
I'm also adding a IntervalIndexTree class for indexing maps of splice 
junctions, introns, and SNPs.
I'll send you this as soon as I've got it working.

Thanks,

JJ


On 10/17/11 1:06 PM, Greg Von Kuster wrote:

We've digested this topic a bit here at Galaxy Central, and agree that at some 
point ( maybe soon for very basic functionality ) we need to provide support 
for new data types in tool shed repositories.  It would be very helpful ( and 
significantly speed up the development process ) if the community could provide 
at least 2 different tools that use data types not included in the Galaxy 
distribution ( sending me a tarball that includes all the tool dependencies, 
including the new data type class would be ideal ).  When I get them I'll add 
this new feature set to my development list.

Thanks everyone for all the input on this!

Greg Von Kuster


On Oct 7, 2011, at 2:05 PM, Jim Johnson wrote:


Greg,

It would be great if there were a way to expand upon the core datatypes using 
the ToolShed.

Would it be possible to have a separate datatype repository within the ToolShed?

Datatype
  name=""
  description=""
  datatype_dependencies=[]
  definition=

The tool config could be expanded to have requirement for datatypes.
   ssmap




Table datatype
   Column|Type | Modifiers
-+-+---
id  | integer | not null default 
nextval('datatype_id_seq'::regclass)
name| character varying(255)  |
version | character varying(40)   |
description | text|
definition  | text|
UNIQUE (name)

Table datatype_datatype_association
   Column|Type | Modifiers
-+-+---
id  | integer | not null default 
nextval('datatype_id_seq'::regclass)
datatype_id | integer |
requires_id | integer |
FOREIGN KEY (datatype_id) REFERENCES datatype(id)
FOREIGN KEY (requires_

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[James Taylor]

Excerpts from Chorny, Ilya's message of 2011-10-21 15:43:35 +:
> Also, I do not have access to trans in the Tools class in 
> lib/galaxy/tools/__init__.py.

'trans' is the state of a web transaction (a single request response
cycle). It is this only accessible for code that occurs within that
scope. For tools/__init__.py there are some methods that have access and
some that don't, it depends on how they are called (all the user
interface generation related code has access to trans for example). 


-- 
James Taylor, Assistant Professor, Biology / Computer Science, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[James Taylor]

Excerpts from Chorny, Ilya's message of 2011-10-21 01:51:03 +:
> I get the following error from routes
> 
> AttributeError: 'thread._local' object has no attribute 'mapper'

Okay, what I suspected. This is a product of the fact that job running
is decoupled from the web frontend. The job runner doesn't necessarily
live in the same process or even on the same machine as the web
interface. You could have multiple web interfaces serving different base
urls feeding one job runner. 

I don't have any simple answer to this. You could try to initialize
route's mapper in the job runner, but I think this would be a big
mistake. Instead you probably want to capture the URL somehow when the
job is created. I need to think about it some more.
-- 
James Taylor, Assistant Professor, Biology / Computer Science, Emory University
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[Greg Von Kuster]

Ilya,

Tools are not tightly coupled to the Galaxy web framework ( they can be 
executed on the command line outside of Galaxy ) so, as you've commented, you 
don't have access to the web framework from there.  It's unclear to me what you 
are trying to do.  Can you clarify a bit more?

Greg Von Kuster

On Oct 21, 2011, at 11:43 AM, Chorny, Ilya wrote:

> Also, I do not have access to trans in the Tools class in 
> lib/galaxy/tools/__init__.py.
> 
> Thanks,
> 
> Ilya
> 
> 
> -Original Message-
> From: galaxy-dev-boun...@lists.bx.psu.edu 
> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
> Sent: Friday, October 21, 2011 8:40 AM
> To: Greg Von Kuster
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output 
> files to my tool wrapper?
> 
> I need the base_url and the proxy. That is why I am using cookie_path. Any 
> thoughts on how to get both.
> 
> Thanks,
> 
> Ilya
> 
> 
> -Original Message-
> From: Greg Von Kuster [mailto:g...@bx.psu.edu]
> Sent: Friday, October 21, 2011 6:11 AM
> To: Chorny, Ilya
> Cc: galaxy-dev@lists.bx.psu.edu
> Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output 
> files to my tool wrapper?
> 
> Ilya, you can use trans.request.host to get the base url.
> 
> 
> On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:
> 
>> I thought I would add this to the list in case anyone in the galaxy :) knew 
>> the answer. I get a mapping error every time I invoke  url_for. What 
>> variable stores the base url for the galaxy instance?
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> -Original Message-
>> From: Chorny, Ilya
>> Sent: Wednesday, October 19, 2011 4:21 PM
>> To: 'James Taylor'
>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> Hi James,
>> 
>> I got the code working but I can't figure out how to set the URL. I have a 
>> hack right now (i.e. url = 'http://' + socket.getfqdn() + 
>> config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) + 
>> '/display/?preview=True' ).  I tried using url_for() but was unable to get 
>> that to work. Attached are the diff's for the files I made changes to.
>> 
>> If you have any suggestions on how to properly set the url please let me 
>> know.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> -Original Message-
>> From: James Taylor [mailto:ja...@jamestaylor.org]
>> Sent: Tuesday, October 18, 2011 11:38 AM
>> To: Chorny, Ilya
>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> Hi Ilya, I think any case where $__app__ is used should raise red flags, it 
>> is definitely breaking abstraction. In this case, you're passing a very 
>> specific implementation detail of the web layer all the way down to your 
>> tool.
>> 
>> I think the right way to do what you want is to add a 'web_display_url'
>> attribute to the class DatasetFilenameWrapper in tools/__init__.py.
>> Every dataset gets wrapped by this when put into the namespace for building 
>> command lines. If you added an attribute it would them be available as 
>> ${input.web_display_url} in the command line template. 
>> 
>> Historically tool wrappers were designed so that they have no knowledge of 
>> the UI, allowing non-web UIs to be added on top. At this point it seems 
>> unlikely that this will ever happen, but it is still a good idea to avoid 
>> this kind of coupling. Otherwise we can't change how the web interface does 
>> security without changing individual tools.
>> 
>> Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
>>> So we came up with a work around and I wouldn't mind getting your opinion 
>>> on it. Specifically with respect to passing cookie_path. I could not figure 
>>> out how to get the proxy prefix from the universe_wsgi.ini file via the 
>>> $__app__ call. Also you mention in the wiki the the $__app__ may get 
>>> deprecated.
>>> 
>>> Thanks,
>>> 
>>> Ilya
>>> 
>>> 
>>> We copied the encryption code (i.e)
>>> 
>>> def encode_id(dataset_id,id_secret):
>>>   id_cipher = Blowfish.new(id_secret)
>>>   s = dataset_id
>>>   s = ( "!" * ( 8 - len(s) % 8 ) ) + s
>>>   encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
>>>   return encrypted_id
>>> 
>>> and we passed to the too wrapperl:
>>> 
>>> -i input -i $input.dataset.id
>>> 
>>> as well as
>>> 
>>> --id_secret=$__app__.config.id_secret
>>> --cookie_path=$__app__.config.cookie_path
>>> 
>>> -Original Message-
>>> From: Chorny, Ilya
>>> Sent: Monday, October 17, 2011 11:10 AM
>>> To: Chorny, Ilya; James Taylor
>>> Cc: galaxy-dev
>>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>>> files to my tool wrapper?
>>> 
>>> Would it be difficult to add the url as an attribute of the input and 
>>> output parameters ($param.url)? If you point me to where these values are 
>>> set, I can try to make the changes to the code.
>>> 
>>> Thanks,
>>> 
>>> Ilya

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[Chorny, Ilya]

Also, I do not have access to trans in the Tools class in 
lib/galaxy/tools/__init__.py.

Thanks,

Ilya


-Original Message-
From: galaxy-dev-boun...@lists.bx.psu.edu 
[mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
Sent: Friday, October 21, 2011 8:40 AM
To: Greg Von Kuster
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output 
files to my tool wrapper?

I need the base_url and the proxy. That is why I am using cookie_path. Any 
thoughts on how to get both.

Thanks,

Ilya


-Original Message-
From: Greg Von Kuster [mailto:g...@bx.psu.edu]
Sent: Friday, October 21, 2011 6:11 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output 
files to my tool wrapper?

Ilya, you can use trans.request.host to get the base url.


On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:

> I thought I would add this to the list in case anyone in the galaxy :) knew 
> the answer. I get a mapping error every time I invoke  url_for. What variable 
> stores the base url for the galaxy instance?
> 
> Thanks,
> 
> Ilya
> 
> -Original Message-
> From: Chorny, Ilya
> Sent: Wednesday, October 19, 2011 4:21 PM
> To: 'James Taylor'
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi James,
> 
> I got the code working but I can't figure out how to set the URL. I have a 
> hack right now (i.e. url = 'http://' + socket.getfqdn() + 
> config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) + 
> '/display/?preview=True' ).  I tried using url_for() but was unable to get 
> that to work. Attached are the diff's for the files I made changes to.
> 
> If you have any suggestions on how to properly set the url please let me know.
> 
> Thanks,
> 
> Ilya
> 
> 
> -Original Message-
> From: James Taylor [mailto:ja...@jamestaylor.org]
> Sent: Tuesday, October 18, 2011 11:38 AM
> To: Chorny, Ilya
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi Ilya, I think any case where $__app__ is used should raise red flags, it 
> is definitely breaking abstraction. In this case, you're passing a very 
> specific implementation detail of the web layer all the way down to your tool.
> 
> I think the right way to do what you want is to add a 'web_display_url'
> attribute to the class DatasetFilenameWrapper in tools/__init__.py.
> Every dataset gets wrapped by this when put into the namespace for building 
> command lines. If you added an attribute it would them be available as 
> ${input.web_display_url} in the command line template. 
> 
> Historically tool wrappers were designed so that they have no knowledge of 
> the UI, allowing non-web UIs to be added on top. At this point it seems 
> unlikely that this will ever happen, but it is still a good idea to avoid 
> this kind of coupling. Otherwise we can't change how the web interface does 
> security without changing individual tools.
> 
> Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
>> So we came up with a work around and I wouldn't mind getting your opinion on 
>> it. Specifically with respect to passing cookie_path. I could not figure out 
>> how to get the proxy prefix from the universe_wsgi.ini file via the $__app__ 
>> call. Also you mention in the wiki the the $__app__ may get deprecated.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> We copied the encryption code (i.e)
>> 
>> def encode_id(dataset_id,id_secret):
>>id_cipher = Blowfish.new(id_secret)
>>s = dataset_id
>>s = ( "!" * ( 8 - len(s) % 8 ) ) + s
>>encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
>>return encrypted_id
>> 
>> and we passed to the too wrapperl:
>> 
>> -i input -i $input.dataset.id
>> 
>> as well as
>> 
>> --id_secret=$__app__.config.id_secret
>> --cookie_path=$__app__.config.cookie_path
>> 
>> -Original Message-
>> From: Chorny, Ilya
>> Sent: Monday, October 17, 2011 11:10 AM
>> To: Chorny, Ilya; James Taylor
>> Cc: galaxy-dev
>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> Would it be difficult to add the url as an attribute of the input and output 
>> parameters ($param.url)? If you point me to where these values are set, I 
>> can try to make the changes to the code.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> -Original Message-
>> From: galaxy-dev-boun...@lists.bx.psu.edu
>> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, 
>> Ilya
>> Sent: Sunday, October 16, 2011 7:36 AM
>> To: James Taylor
>> Cc: galaxy-dev
>> Subject: Re: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> PDF report.
>> 
>> Sent from my iPhone
>> 
>> On Oct 15, 2011, at 7:36 PM, "James Taylor"  wrote:
>> 
>>> Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
 Anyone

Re: [galaxy-dev] Database configuration

[Luciano Cosme]

Thanks Greg,
   I created the database instance and set the path at universe_wsgi.ini.
Then when I run sh run.sh I get the following error message:


root@aedes:/home/koala2/galaxy-dist# sh run.sh
python path is:
/home/koala2/galaxy-dist/eggs/numpy-1.6.0-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/Whoosh-0.3.18-py2.7.egg,
/home/koala2/galaxy-dist/eggs/pycrypto-2.0.1-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/python_lzo-1.08_2.03_static-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/bx_python-0.7.0_494c2d1d68b3_rebuild1-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/amqplib-0.6.1-py2.7.egg,
/home/koala2/galaxy-dist/eggs/pexpect-2.4-py2.7.egg,
/home/koala2/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg,
/home/koala2/galaxy-dist/eggs/Babel-0.9.4-py2.7.egg,
/home/koala2/galaxy-dist/eggs/MarkupSafe-0.12-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg,
/home/koala2/galaxy-dist/eggs/WebHelpers-0.2-py2.7.egg,
/home/koala2/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/wchartype-0.1-py2.7.egg,
/home/koala2/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.7.egg,
/home/koala2/galaxy-dist/eggs/docutils-0.7-py2.7.egg,
/home/koala2/galaxy-dist/eggs/WebOb-0.8.5-py2.7.egg,
/home/koala2/galaxy-dist/eggs/Routes-1.12.3-py2.7.egg,
/home/koala2/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg,
/home/koala2/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg,
/home/koala2/galaxy-dist/eggs/PasteScript-1.7.3-py2.7.egg,
/home/koala2/galaxy-dist/eggs/Paste-1.6-py2.7.egg,
/home/koala2/galaxy-dist/lib, /usr/lib/python2.7,
/usr/lib/python2.7/plat-linux2, /usr/lib/python2.7/lib-tk,
/usr/lib/python2.7/lib-old, /usr/lib/python2.7/lib-dynload,
/usr/local/lib/python2.7/dist-packages,
/usr/lib/python2.7/dist-packages/PIL,
/usr/lib/python2.7/dist-packages/gst-0.10,
/usr/lib/python2.7/dist-packages/gtk-2.0, /usr/lib/pymodules/python2.7,
/usr/lib/python2.7/dist-packages/ubuntu-sso-client,
/usr/lib/python2.7/dist-packages/ubuntuone-client,
/usr/lib/python2.7/dist-packages/ubuntuone-control-panel,
/usr/lib/python2.7/dist-packages/ubuntuone-couch,
/usr/lib/python2.7/dist-packages/ubuntuone-installer,
/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol,
/usr/lib/python2.7/dist-packages
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,646 Loading datatypes
from datatypes_conf.xml
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,648 Loaded display
application 'ucsc_bam' for datatype 'bam', inherit=False
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,649 Loaded display
application 'ensembl_bam' for datatype 'bam', inherit=False
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,649 Loaded display
application 'genetrack_interval' for datatype 'bed', inherit=False
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,651 Loaded display
application 'ucsc_bigbed' for datatype 'bigbed', inherit=False
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,653 Loaded display
application 'ucsc_bigwig' for datatype 'bigwig', inherit=False
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,656 Loaded display
application 'ensembl_gff' for datatype 'gff', inherit=True
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,657 Loaded display
application 'genetrack_interval' for datatype 'interval', inherit=True
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,660 Loaded display
application 'ensembl_interval' for datatype 'interval', inherit=True
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,661 Loaded display
application 'gbrowse_interval_as_bed' for datatype 'interval', inherit=True
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,664 Loaded display
application 'ucsc_maf_customtrack' for datatype 'mafcustomtrack',
inherit=False
galaxy.datatypes.assembly DEBUG 2011-10-21 00:28:30,664 Velvet log info  JJ
__init__
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.tabular:Vcf
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.binary:TwoBit
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.binary:Bam
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.binary:Sff
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.xml:BlastXml
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.sequence:Maf
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.sequence:Lav
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datatypes.sequence:csFasta
galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for
datatype: galaxy.datat

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[Chorny, Ilya]

I need the base_url and the proxy. That is why I am using cookie_path. Any 
thoughts on how to get both.

Thanks,

Ilya


-Original Message-
From: Greg Von Kuster [mailto:g...@bx.psu.edu] 
Sent: Friday, October 21, 2011 6:11 AM
To: Chorny, Ilya
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] FW: how do I pass the URL of the input and output 
files to my tool wrapper?

Ilya, you can use trans.request.host to get the base url.


On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:

> I thought I would add this to the list in case anyone in the galaxy :) knew 
> the answer. I get a mapping error every time I invoke  url_for. What variable 
> stores the base url for the galaxy instance?
> 
> Thanks,
> 
> Ilya
> 
> -Original Message-
> From: Chorny, Ilya
> Sent: Wednesday, October 19, 2011 4:21 PM
> To: 'James Taylor'
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi James,
> 
> I got the code working but I can't figure out how to set the URL. I have a 
> hack right now (i.e. url = 'http://' + socket.getfqdn() + 
> config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) + 
> '/display/?preview=True' ).  I tried using url_for() but was unable to get 
> that to work. Attached are the diff's for the files I made changes to.
> 
> If you have any suggestions on how to properly set the url please let me know.
> 
> Thanks,
> 
> Ilya
> 
> 
> -Original Message-
> From: James Taylor [mailto:ja...@jamestaylor.org]
> Sent: Tuesday, October 18, 2011 11:38 AM
> To: Chorny, Ilya
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi Ilya, I think any case where $__app__ is used should raise red flags, it 
> is definitely breaking abstraction. In this case, you're passing a very 
> specific implementation detail of the web layer all the way down to your tool.
> 
> I think the right way to do what you want is to add a 'web_display_url'
> attribute to the class DatasetFilenameWrapper in tools/__init__.py.
> Every dataset gets wrapped by this when put into the namespace for building 
> command lines. If you added an attribute it would them be available as 
> ${input.web_display_url} in the command line template. 
> 
> Historically tool wrappers were designed so that they have no knowledge of 
> the UI, allowing non-web UIs to be added on top. At this point it seems 
> unlikely that this will ever happen, but it is still a good idea to avoid 
> this kind of coupling. Otherwise we can't change how the web interface does 
> security without changing individual tools.
> 
> Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
>> So we came up with a work around and I wouldn't mind getting your opinion on 
>> it. Specifically with respect to passing cookie_path. I could not figure out 
>> how to get the proxy prefix from the universe_wsgi.ini file via the $__app__ 
>> call. Also you mention in the wiki the the $__app__ may get deprecated.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> We copied the encryption code (i.e)
>> 
>> def encode_id(dataset_id,id_secret):
>>id_cipher = Blowfish.new(id_secret)
>>s = dataset_id
>>s = ( "!" * ( 8 - len(s) % 8 ) ) + s
>>encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
>>return encrypted_id
>> 
>> and we passed to the too wrapperl:
>> 
>> -i input -i $input.dataset.id
>> 
>> as well as
>> 
>> --id_secret=$__app__.config.id_secret
>> --cookie_path=$__app__.config.cookie_path
>> 
>> -Original Message-
>> From: Chorny, Ilya
>> Sent: Monday, October 17, 2011 11:10 AM
>> To: Chorny, Ilya; James Taylor
>> Cc: galaxy-dev
>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> Would it be difficult to add the url as an attribute of the input and output 
>> parameters ($param.url)? If you point me to where these values are set, I 
>> can try to make the changes to the code.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> -Original Message-
>> From: galaxy-dev-boun...@lists.bx.psu.edu
>> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, 
>> Ilya
>> Sent: Sunday, October 16, 2011 7:36 AM
>> To: James Taylor
>> Cc: galaxy-dev
>> Subject: Re: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> PDF report.
>> 
>> Sent from my iPhone
>> 
>> On Oct 15, 2011, at 7:36 PM, "James Taylor"  wrote:
>> 
>>> Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
 Anyone know if it possible to pass the url (shown in the eye ball) to a 
 tool wrapper?
>>> 
>>> Ilya, this isn't currently supported because it breaks the 
>>> abstraction between the web interface and tools (tools shouldn't 
>>> know they are being invoked by a web application). What are you trying to 
>>> achieve with this?
>>> Is this for a tool generating an HTML report? 
>>> --
>>> James Taylor, Assistant Professor, Biology / Compute

Re: [galaxy-dev] Job output not returned from cluster

[Joseph Hargitai]

Nate, 

this error is intermittent. You resubmit the same job twice or three time and 
then it works.  Once we are over the midterm exams - which use galaxy - we will 
try to switch the filesystem from autofs to hard mount. We suspect this to be 
the issue.  

Could we suppress e and o SGE style to resolve this issue, or Galaxy wants the 
o? 


Do you have an idea about the url build for galaxy - ucsc page return when the 
url is :8080/galaxy and not just /galaxy? 

thanks,
joe


From: Nate Coraor [n...@bx.psu.edu]
Sent: Friday, October 21, 2011 10:26 AM
To: Joseph Hargitai
Cc: galaxy-dev@lists.bx.psu.edu
Subject: Re: [galaxy-dev] Job output not returned from cluster

Joseph Hargitai wrote:
>
> Hi,
>
> i was browsing through the list and found many entries for this issue but not 
> a definite answer.
>
> We are actually running into this error for simple file uploads from the 
> internal filesystem.

Hi Joe,

This error occurs when the job's standard output and error files are not
found where Galaxy expects them, namely:

/.o
/.e

Please check your queueing system to make sure it can correctly deliver
these back from the execution hosts to the specified filesystem.

--nate

>
> thanks,
> joe
>

> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>
>   http://lists.bx.psu.edu/




___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Job output not returned from cluster

[Nate Coraor]

Joseph Hargitai wrote:
> 
> Hi,
> 
> i was browsing through the list and found many entries for this issue but not 
> a definite answer.
> 
> We are actually running into this error for simple file uploads from the 
> internal filesystem.

Hi Joe,

This error occurs when the job's standard output and error files are not
found where Galaxy expects them, namely:

/.o
/.e

Please check your queueing system to make sure it can correctly deliver
these back from the execution hosts to the specified filesystem.

--nate

> 
> thanks,
> joe
> 

> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
> 
>   http://lists.bx.psu.edu/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:

  http://lists.bx.psu.edu/

Re: [galaxy-dev] Database configuration

[Greg Von Kuster]

It looks like you don't have you postgres database set up correctly to accept 
communication.  Check the postgres documentation.

On Oct 21, 2011, at 1:54 AM, Luciano Cosme wrote:

> Thanks Greg,
>I created the database instance and set the path at universe_wsgi.ini. 
> Then when I run sh run.sh I get the following error message:
> 
> 
> root@aedes:/home/koala2/galaxy-dist# sh run.sh
> python path is: 
> /home/koala2/galaxy-dist/eggs/numpy-1.6.0-py2.7-linux-x86_64-ucs4.egg, 
> /home/koala2/galaxy-dist/eggs/pysam-0.4.2_kanwei_b10f6e722e9a-py2.7-linux-x86_64-ucs4.egg,
>  /home/koala2/galaxy-dist/eggs/Whoosh-0.3.18-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/pycrypto-2.0.1-py2.7-linux-x86_64-ucs4.egg, 
> /home/koala2/galaxy-dist/eggs/python_lzo-1.08_2.03_static-py2.7-linux-x86_64-ucs4.egg,
>  
> /home/koala2/galaxy-dist/eggs/bx_python-0.7.0_494c2d1d68b3_rebuild1-py2.7-linux-x86_64-ucs4.egg,
>  /home/koala2/galaxy-dist/eggs/amqplib-0.6.1-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/pexpect-2.4-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/Babel-0.9.4-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/MarkupSafe-0.12-py2.7-linux-x86_64-ucs4.egg, 
> /home/koala2/galaxy-dist/eggs/Mako-0.4.1-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/WebHelpers-0.2-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/simplejson-2.1.1-py2.7-linux-x86_64-ucs4.egg, 
> /home/koala2/galaxy-dist/eggs/wchartype-0.1-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/docutils-0.7-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/WebOb-0.8.5-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/Routes-1.12.3-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/Cheetah-2.2.2-py2.7-linux-x86_64-ucs4.egg, 
> /home/koala2/galaxy-dist/eggs/PasteDeploy-1.3.3-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/PasteScript-1.7.3-py2.7.egg, 
> /home/koala2/galaxy-dist/eggs/Paste-1.6-py2.7.egg, 
> /home/koala2/galaxy-dist/lib, /usr/lib/python2.7, 
> /usr/lib/python2.7/plat-linux2, /usr/lib/python2.7/lib-tk, 
> /usr/lib/python2.7/lib-old, /usr/lib/python2.7/lib-dynload, 
> /usr/local/lib/python2.7/dist-packages, /usr/lib/python2.7/dist-packages/PIL, 
> /usr/lib/python2.7/dist-packages/gst-0.10, 
> /usr/lib/python2.7/dist-packages/gtk-2.0, /usr/lib/pymodules/python2.7, 
> /usr/lib/python2.7/dist-packages/ubuntu-sso-client, 
> /usr/lib/python2.7/dist-packages/ubuntuone-client, 
> /usr/lib/python2.7/dist-packages/ubuntuone-control-panel, 
> /usr/lib/python2.7/dist-packages/ubuntuone-couch, 
> /usr/lib/python2.7/dist-packages/ubuntuone-installer, 
> /usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol, 
> /usr/lib/python2.7/dist-packages
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,646 Loading datatypes 
> from datatypes_conf.xml
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,648 Loaded display 
> application 'ucsc_bam' for datatype 'bam', inherit=False
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,649 Loaded display 
> application 'ensembl_bam' for datatype 'bam', inherit=False
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,649 Loaded display 
> application 'genetrack_interval' for datatype 'bed', inherit=False
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,651 Loaded display 
> application 'ucsc_bigbed' for datatype 'bigbed', inherit=False
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,653 Loaded display 
> application 'ucsc_bigwig' for datatype 'bigwig', inherit=False
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,656 Loaded display 
> application 'ensembl_gff' for datatype 'gff', inherit=True
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,657 Loaded display 
> application 'genetrack_interval' for datatype 'interval', inherit=True
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,660 Loaded display 
> application 'ensembl_interval' for datatype 'interval', inherit=True
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,661 Loaded display 
> application 'gbrowse_interval_as_bed' for datatype 'interval', inherit=True
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,664 Loaded display 
> application 'ucsc_maf_customtrack' for datatype 'mafcustomtrack', 
> inherit=False
> galaxy.datatypes.assembly DEBUG 2011-10-21 00:28:30,664 Velvet log info  JJ 
> __init__
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for 
> datatype: galaxy.datatypes.tabular:Vcf
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for 
> datatype: galaxy.datatypes.binary:TwoBit
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for 
> datatype: galaxy.datatypes.binary:Bam
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for 
> datatype: galaxy.datatypes.binary:Sff
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:30,669 Loaded sniffer for 
> datatype: galaxy.datatypes.xml:BlastXml
> galaxy.datatypes.registry DEBUG 2011-10-21 00:28:

Re: [galaxy-dev] FW: how do I pass the URL of the input and output files to my tool wrapper?

[Greg Von Kuster]

Ilya, you can use trans.request.host to get the base url.


On Oct 20, 2011, at 11:52 AM, Chorny, Ilya wrote:

> I thought I would add this to the list in case anyone in the galaxy :) knew 
> the answer. I get a mapping error every time I invoke  url_for. What variable 
> stores the base url for the galaxy instance?
> 
> Thanks,
> 
> Ilya
> 
> -Original Message-
> From: Chorny, Ilya 
> Sent: Wednesday, October 19, 2011 4:21 PM
> To: 'James Taylor'
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi James,
> 
> I got the code working but I can't figure out how to set the URL. I have a 
> hack right now (i.e. url = 'http://' + socket.getfqdn() + 
> config_info.cookie_path + '/datasets/' + sec.encode_id(dataset.id) + 
> '/display/?preview=True' ).  I tried using url_for() but was unable to get 
> that to work. Attached are the diff's for the files I made changes to.
> 
> If you have any suggestions on how to properly set the url please let me know.
> 
> Thanks,
> 
> Ilya
> 
> 
> -Original Message-
> From: James Taylor [mailto:ja...@jamestaylor.org]
> Sent: Tuesday, October 18, 2011 11:38 AM
> To: Chorny, Ilya
> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output files 
> to my tool wrapper?
> 
> Hi Ilya, I think any case where $__app__ is used should raise red flags, it 
> is definitely breaking abstraction. In this case, you're passing a very 
> specific implementation detail of the web layer all the way down to your tool.
> 
> I think the right way to do what you want is to add a 'web_display_url'
> attribute to the class DatasetFilenameWrapper in tools/__init__.py.
> Every dataset gets wrapped by this when put into the namespace for building 
> command lines. If you added an attribute it would them be available as 
> ${input.web_display_url} in the command line template. 
> 
> Historically tool wrappers were designed so that they have no knowledge of 
> the UI, allowing non-web UIs to be added on top. At this point it seems 
> unlikely that this will ever happen, but it is still a good idea to avoid 
> this kind of coupling. Otherwise we can't change how the web interface does 
> security without changing individual tools.
> 
> Excerpts from Chorny, Ilya's message of 2011-10-17 22:56:40 +:
>> So we came up with a work around and I wouldn't mind getting your opinion on 
>> it. Specifically with respect to passing cookie_path. I could not figure out 
>> how to get the proxy prefix from the universe_wsgi.ini file via the $__app__ 
>> call. Also you mention in the wiki the the $__app__ may get deprecated.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> We copied the encryption code (i.e)
>> 
>> def encode_id(dataset_id,id_secret):
>>id_cipher = Blowfish.new(id_secret)
>>s = dataset_id
>>s = ( "!" * ( 8 - len(s) % 8 ) ) + s
>>encrypted_id = id_cipher.encrypt( s ).encode( 'hex' )
>>return encrypted_id
>> 
>> and we passed to the too wrapperl:
>> 
>> -i input -i $input.dataset.id
>> 
>> as well as
>> 
>> --id_secret=$__app__.config.id_secret
>> --cookie_path=$__app__.config.cookie_path
>> 
>> -Original Message-
>> From: Chorny, Ilya
>> Sent: Monday, October 17, 2011 11:10 AM
>> To: Chorny, Ilya; James Taylor
>> Cc: galaxy-dev
>> Subject: RE: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> Would it be difficult to add the url as an attribute of the input and output 
>> parameters ($param.url)? If you point me to where these values are set, I 
>> can try to make the changes to the code.
>> 
>> Thanks,
>> 
>> Ilya
>> 
>> 
>> -Original Message-
>> From: galaxy-dev-boun...@lists.bx.psu.edu
>> [mailto:galaxy-dev-boun...@lists.bx.psu.edu] On Behalf Of Chorny, Ilya
>> Sent: Sunday, October 16, 2011 7:36 AM
>> To: James Taylor
>> Cc: galaxy-dev
>> Subject: Re: [galaxy-dev] how do I pass the URL of the input and output 
>> files to my tool wrapper?
>> 
>> PDF report.
>> 
>> Sent from my iPhone
>> 
>> On Oct 15, 2011, at 7:36 PM, "James Taylor"  wrote:
>> 
>>> Excerpts from Chorny, Ilya's message of 2011-10-14 23:46:25 +:
 Anyone know if it possible to pass the url (shown in the eye ball) to a 
 tool wrapper?
>>> 
>>> Ilya, this isn't currently supported because it breaks the 
>>> abstraction between the web interface and tools (tools shouldn't 
>>> know they are being invoked by a web application). What are you trying to 
>>> achieve with this?
>>> Is this for a tool generating an HTML report? 
>>> --
>>> James Taylor, Assistant Professor, Biology / Computer Science, Emory 
>>> University 
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this and other 
>>> Galaxy lists, please use the interface at:
>>> 
>>> http://lists.bx.psu.edu/
>> 
>> ___
>> Please k