Hi Dan,
Thanks for your reply.
I don't handle the updates, but it was updated on 25 Jun 13. I'm pretty
sure I can rule out the version number as the BWA builds, created using
the Data Manager, work fine.
Here is the pertinent contents of shed_tool_data_table_conf.xml
<?xml version="1.0"?>
<tables>
<table comment_char="#" name="all_fasta">
<columns>value, dbkey, name, path</columns>
<file
path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_bwa_index_builder/fe6508204acc/all_fast
a.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep
ository_name>data_manager_bwa_index_builder</repository_name><repository_ow
ner>blankenberg</repository_owner><installed_changeset_revision>fe6508204ac
c</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="all_fasta">
<columns>value, dbkey, name, path</columns>
<file
path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/all_f
asta.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep
ository_name>data_manager_sam_fa_index_builder</repository_name><repository
_owner>blankenberg</repository_owner><installed_changeset_revision>926e5039
7b83</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="sam_fa_indexes">
<columns>line_type, value, path</columns>
<file
path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/sam_f
a_indices.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep
ository_name>data_manager_sam_fa_index_builder</repository_name><repository
_owner>blankenberg</repository_owner><installed_changeset_revision>926e5039
7b83</installed_changeset_revision></tool_shed_repository></table>
<table comment_char="#" name="all_fasta">
<columns>value, dbkey, name, path</columns>
<file
path="/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_fetch_genome_all_fasta/ca8b3709309e/all
_fasta.loc" />
<tool_shed_repository><tool_shed>testtoolshed.g2.bx.psu.edu</tool_shed><rep
ository_name>data_manager_fetch_genome_all_fasta</repository_name><reposito
ry_owner>blankenberg</repository_owner><installed_changeset_revision>ca8b37
09309e</installed_changeset_revision></tool_shed_repository></table>
</tables>
in ${GALAXY_HOME}/tool_data_table_conf.xml.sample the entry for sam_fa
reads as so:
<!-- Location of SAMTools indexes and other files -->
<table name="sam_fa_indexes" comment_char="#">
<columns>line_type, value, path</columns>
<file path="tool-data/sam_fa_indices.loc" />
</table>
The file tool-data/sam_fa_new_indices.loc (and .sample) does not exist.
If I keep the manually inserted builds listed in
tool-data/sam_fa_indices.loc and restart Galaxy, then I get the following
(abridged) entries in the paster.log:
galaxy.tools.data DEBUG 2013-07-15 09:51:11,109 Loaded tool data table
'all_fasta'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,115 Loaded tool data table
'bwa_indexes'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,116 Loaded tool data table
'bwa_indexes_color'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,167 Loaded tool data table
'sam_fa_indexes'
...
galaxy.tools.data DEBUG 2013-07-15 09:51:11,324 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,340 Loading another instance
of data table 'bwa_indexes', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,348 Loading another instance
of data table 'bwa_indexes_color', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,410 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,422 Loading another instance
of data table 'sam_fa_indexes', attempting to merge content.
galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields
(['index', 'cfraxinea_s1v1',
'/home/galaxy/software/galaxy-central/tool-data/cfraxinea_s1v1/sam_index/cf
raxinea_s1v1/c_fraxinea_s1v1.fa']) to data table 'sam_fa_indexes', but
this entry already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields
(['index', 'b_distachyon',
'/home/galaxy/software/galaxy-central/tool-data/b_distachyon/sam_index/b_di
stachyon/b_distachyon.fa']) to data table 'sam_fa_indexes', but this entry
already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields
(['index', 'n_sylvestris',
'/home/galaxy/software/galaxy-central/tool-data/n_sylvestris/sam_index/n_sy
lvestris/n_sylvestris.fa']) to data table 'sam_fa_indexes', but this entry
already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields
(['index', 'n_tomentosiformis',
'/home/galaxy/software/galaxy-central/tool-data/n_tomentosiformis/sam_index
/n_tomentosiformis/n_tomentosiformis.fa']) to data table 'sam_fa_indexes',
but this entry already exists and allow_duplicates is False.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,491 Loading another instance
of data table 'all_fasta', attempting to merge content.
The builds listed above with the 'entry already exists' error, are all the
ones which I attempted to load using the Data Manager.
If I comment out these in entries in tool-data/sam_fa_indices.loc and
restart Galaxy, the errors disappear, but when I try to use those builds
to SAM-to-BAM, I'm back to the "Sequences are not currently available for
the specified build." error.
paster.log entries after re-start:
galaxy.tools.data DEBUG 2013-07-15 10:02:56,484 Loaded tool data table
'all_fasta'
galaxy.tools.data DEBUG 2013-07-15 10:02:56,491 Loaded tool data table
'bwa_indexes'
galaxy.tools.data DEBUG 2013-07-15 10:02:56,492 Loaded tool data table
'bwa_indexes_color'
galaxy.tools.data DEBUG 2013-07-15 10:02:56,497 Loaded tool data table
'sam_fa_indexes'
...
galaxy.tools.data DEBUG 2013-07-15 10:02:56,508 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 10:02:56,509 Loading another instance
of data table 'bwa_indexes', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 10:02:56,510 Loading another instance
of data table 'bwa_indexes_color', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 10:02:56,512 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 10:02:56,512 Loading another instance
of data table 'sam_fa_indexes', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 10:02:56,514 Loading another instance
of data table 'all_fasta', attempting to merge content.
On both occasions the following lines are in the paster.log
galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,667 Loaded
Data Manager:
testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_index_builder
/data_manager/bwa_index_builder/0.0.1
galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,689 Loaded
Data Manager:
testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_index_builder
/data_manager/bwa_color_space_index_builder/0.0.1
galaxy.tools.data_manager.manager DEBUG 2013-07-15 09:51:47,755 Loaded
Data Manager:
testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_fa_index_buil
der/data_manager/sam_fa_index_builder/0.0.1
Any suggestions?
Cheers for now,
Graham
Dr. Graham Etherington
Bioinformatics Support Officer,
The Sainsbury Laboratory,
Norwich Research Park,
Norwich NR4 7UH.
UK
Tel: +44 (0)1603 450601
On 12/07/2013 15:45, "Daniel Blankenberg" <d...@bx.psu.edu> wrote:
>Hi Graham,
>
>Which revision of Galaxy are you currently using? Currently data managers
>require at least 9952:a28faa6ac188 on the default branch.
>
>When installed from the tool shed, data managers use the
>shed_tool_data_table_conf.xml file, could you check the contents of that
>file?
>
>Also can you check the paster.log during server start up for any errors,
>its most likely the case that this changeset
>https://bitbucket.org/galaxy/galaxy-central/commits/af20b15f7eda
>(10145:af20b15f7eda) creates a non-compatiible tool data table entry in
>the main file.
>
>
>Thanks for using Galaxy,
>
>Dan
>
>
>On Jul 12, 2013, at 8:14 AM, graham etherington (TSL) wrote:
>
>> Hi,
>> I've tried using the Data Manager (Admin > Data > Manage local data
>> (beta)) to install builds for BWA and Samtools on my local Galaxy
>>instance.
>> Previous to using the Data Manager, I used to add the build to
>> tool-data/shared/ucsc/builds.txt, create the .fai indexes (for samtools)
>> from the command line, add them to tool-data/sam_fa_indices.loc and
>> restart Galaxy (obviously doing a similar thing for BWA and adding the
>> build to bwa_index.loc).
>>
>> I thought I'd try using the Data Manager to add builds for BWA and
>> Samtools. The BWA builds work fine (I can map to the build), but when I
>> try to use SAM-to-BAM I get the error "Sequences are not currently
>> available for the specified build."
>>
>> Using the Data Manager creates the directory tool-data/n_sylvestris/
>>which
>> contains the sub-dirs 'seq', 'bwa_index' and 'sam_index'.
>> 'seq' contains a symlink to the n_sylvestris.fa sequence.
>> 'sam_index' and 'bwa_index' both contains the sub-directory
>> 'n_sylvestris', which contains a symlink to the symlink for
>> n_sylvestris.fa in 'seq' along with their respective n_sylvestris.fa.xxx
>> index files.
>>
>> OK - all goodŠ
>>
>> In tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ there are
>>three
>> subdirectories:
>> data_manager_bwa_index_builder, data_manager_sam_fa_index_builder and
>> data_manager_fetch_genome_all_fasta
>> All three directories contain all_fasta.loc, tool_data_table_conf.xml,
>> tool_data_table_conf.xml.sample and (for sam and bam dirs) their
>>pertinent
>> index.loc file.
>>
>> The data_manager_fetch_genome_all_fasta/all_fasta.loc file contains the
>> path to the fasta symlinks.
>>
>> The all_fasta.loc files in the sam and bwa data_manager_index_builder
>> directories don't contain any uncommented lines.
>>
>> The index.loc files in the sam and bwa data_manager_index_builder
>> directories point to:
>> tool-data/n_sylvestris/bwa_index/n_sylvestris/n_sylvestris.fa
>> tool-data/n_sylvestris/sam_index/n_sylvestris/n_sylvestris.fa
>>
>> As BWA runs fine, it's obviously reading the bwa_index.loc file from the
>> directory:
>>
>>tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_bwa_i
>>nd
>> ex_builder/fe6508204acc/bwa_index.loc
>>
>> ...but it's not reading the samtools indexes at:
>>
>>tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_f
>>a_
>> index_builder/926e50397b83/sam_fa_indices.loc
>>
>> For Galaxy to find the sam indexes, I have to go to the
>> tool-data/sam_fa_indices.loc file and manually insert into it the
>>contents
>> of:
>>
>>tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/data_manager_sam_f
>>a_
>> index_builder/926e50397b83/sam_fa_indices.loc
>>
>>
>> So, I guess my question is: other than inserting the genome builds into
>> builds.txt, should I be doing any other configuration to get Data
>>Manager
>> to write and configure Galaxy to read it's newly created builds. I find
>>it
>> strange that the BWA builds work OK, but the Samtools ones don't.
>>
>> I've done a few greps for mentions of .loc files in Galaxy and the only
>> difference between the bwa and sam .loc files is that there is a file
>> tool-data/tool_data_table_conf.xml (plus a .sample version) which
>>contains:
>>
>> <!-- Use the file tool_data_table_conf.xml.oldlocstyle if you don't want
>> to update your loc files as changed in revision 4550:535d276c92bc-->
>> <tables>
>> <!-- Location of SAMTools indexes and other files -->
>> <table name="sam_fa_indexes" comment_char="#">
>> <columns>line_type, value, path</columns>
>> <file path="tool-data/sam_fa_indices.loc" />
>> </table>
>> </tables>
>>
>>
>> Could Galaxy be reading this file and ignoring the one in
>> tool-data/testtoolshed.g2.bx.psu.edu/repos/blankenberg/ ??
>>
>>
>>
>> Best wishes,
>> Graham
>>
>>
>>
>>
>> Dr. Graham Etherington
>> Bioinformatics Support Officer,
>> The Sainsbury Laboratory,
>> Norwich Research Park,
>> Norwich NR4 7UH.
>> UK
>> Tel: +44 (0)1603 450601
>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client. To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>> http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>> http://galaxyproject.org/search/mailinglists/
>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
