Hi all,
> * For the tool FilterControl, our cluster is configured to kill jobs that use
> more than 4Go of memory.
> I don't achieve to modify qsub options in my Galaxy instance, so I'he changed
> this option -Xmx6g in "-Xmx4g". Maybe some treatment won't run by lake of
> memory ..
The memory that you specify with -Xmx6g is the memory used by Java and NOT the
memory used for the job launched by Galaxy on the cluster (see
http://www.auditmypc.com/java-memory-xmx512.asp).
So you have to specify in your java configuration the min and max memory
usable.
> If you have any idea on how to add option on Galaxy qsub, could you please
> help me ?
> I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G
Try by specifying in universe_wsgi.ini runner:///queue/-l
nodes=1:ppn=1,mem=6gb,h_vmem=8gb/
If it's not understood by Galaxy,To add new qsub options, you have to make some
modifications in the source code:
1. Identify your scheduler (pbs, drmaa, sge)
2. Edit the python script which creates jobs for your scheduler:
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/pbs.py
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
GALAXY_INSTALL_DIR/galaxy-dist/lib/galaxy/jobs/runners/sge.py
3. Search in the script the function which parse your scheduler options (ex:
for pbs.py, the function is named "def determine_pbs_options( self, url ):")
4. Modify the parsing step to make this function understand the h_vmem option
Hope it's help you,
++,
Alban
--
Alban Lermine
Unité 900: INSERM - Mines ParisTech - Institut Curie
" Bioinformatics and Computational Systems Biology of Cancer"
11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
Tel: +33 (0) 1 56 24 69 84
Le 5 févr. 2013 à 14:53, Sarah Maman <sarah.ma...@toulouse.inra.fr> a écrit :
> Hi Alban, Marie-Stephane and Bjoern,
>
>
> * For IntersectBed tool, running is OK. I've just deleted ";" in xml file and
> your tool runs. My cluster add one ";" so a double ";;" gives an error.
>
> I thing that this new eroor is due to files tested :
>
> Differing number of VCF fields encountered at line: 38. Exiting...
>
> *For RepeatMasker tool, thanks to Marie-Stephane (CONGRATULATIONS
> MARIE-STEPHANE !), we need to specify that bash commands are included iwith
> Python. So xml file have been modify in this way :
> - Specify "perl" before RepeatMasker command.
> - Specify RepeatMasker path
> - Include all bash comman in a Python script : #os.system("cp $gff_file
> $output_gff;") instead of cp $gff_file $output_gff;
> So the code is :
> <command>
> ## The command is a Cheetah template which allows some Python based syntax.
> ## Lines starting hash hash are comments. Galaxy will turn newlines into
> spaces
> ## create temp directory
> #import tempfile, os
> #set $dirname = os.path.abspath(tempfile.mkdtemp())
> #set $input_filename = os.path.split(str($query))[-1]
> #set $output_basename = os.path.join($dirname, $input_filename)
> perl /usr/local/bioinfo/bin/RepeatMasker -parallel 8 $nolow $noint $norna
> #if str($species)!="all":
> $species
> #end if
> -dir $dirname
> #if $adv_opts.adv_opts_selector=="advanced":
> #if str($adv_opts.gc)!="0":
> -gc $adv_opts.gc
> #end if
> $adv_opts.gccalc
> #set $output_files_list = str($adv_opts.output_files).split(',')
> #if "gff" in $output_files_list:
> -gff
> #end if
> #if "html" in $output_files_list:
> -html
> #end if
> $adv_opts.slow_search
> $adv_opts.quick_search
> $adv_opts.rush_search
> $adv_opts.only_alus
> $adv_opts.is_only
> #else:
> ## Set defaults
> -gff
> ## End of advanced options:
> #end if
> $query
> /dev/null 2> /dev/null;
> ## Copy the output files to galaxy
> #if $adv_opts.adv_opts_selector=="advanced":
> #if "summary" in $output_files_list:
> ## Write out the summary file (default)
> #set $summary_file = $output_basename + '.tbl'
> #os.system("cp $summary_file $output_summary;")
> #end if
> #if "gff" in $output_files_list:
> ## Write out the gff file (default)
> #set $gff_file = $output_basename + '.out.gff'
> #os.system("cp $gff_file $output_gff");
> #end if
> #if "html" in $output_files_list:
> ## Write out the html file
> #set $html_file = $output_basename + '.out.html'
> #os.system("cp $html_file $output_html;")
> #end if
> #else:
>
> ## Write out the summary file (default)
> #set $summary_file = $output_basename + '.tbl'
> #os.system("cp $summary_file $output_summary;")
> ## Write out the gff file (default)
> #set $gff_file = $output_basename + '.out.gff'
> #os.system("cp $gff_file $output_gff;")
> ## End of advanced options:
> #end if
> ## Write out mask sequence file
> #set $mask_sequence_file = $output_basename + '.masked'
> #os.system("cp $mask_sequence_file $output_mask;")
> ## Write out standard file (default)
> ## The default '.out' file from RepeatMasker has a 3-line header and spaces
> rather
> ## than tabs. Remove the header and replace the whitespaces with tab
> #set $standard_file = $output_basename + '.out'
> #os.system("tail -n +4 $standard_file | tr -s ' ' '\t' > $output_std;")
> ## Delete all temporary files
> #os.system("rm $dirname -r;"
>
> * For the tool FilterControl, our cluster is configured to kill jobs that use
> more than 4Go of memory.
> I don't achieve to modify qsub options in my Galaxy instance, so I'he changed
> this option -Xmx6g in "-Xmx4g". Maybe some treatment won't run by lake of
> memory ..
> If you have any idea on how to add option on Galaxy qsub, could you please
> help me ?
> I would like to add these options to qsub : qsub -l mem=6G -l h_vmem=8G
>
>
> * ChIPMunk :Sorry, but all ChIPMunk files (xml, pl; sh) didn't have
> execution rights... So I just do "chmod a+x" on these files and ChIPMunk tool
> is OK in my Galaxy instance....
>
>
> Thanks a lot for all your explanations, Alban, Marie-Stephane and Bjoern .
>
> Thanks in advance for qsub and IntersectBed tool (test file),
>
> Sarah
>
>
>
>
>
> alermine a écrit :
>>
>> Hi Sarah,
>>
>> I'll try to debug, point per point:
>>
>>>>> - Here is the error I get when running the tool FilterControl
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007fe56800ecd0 ***
>>
>> I think here you have a misconfiguration of the memory of your java install
>> (according to the need of the tool)
>>
>> If you look at the FilterControlPeaks.sh file, the java is called with the
>> option -Xmx6g. So your java install have to be allowed to use 6G as memory
>> (by default it's 1024M)
>>
>>>>> - Here is the error I get when running the tool IntersectBed
>>>>>
>>>>> /work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe
>>>>> près du symbole inattendu « ;; »
>>>>> /work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect
>>>>> -f 0.05
>>>>>
>>
>> I don't understand this one, The intersectBed tool is only composed of a xml
>> which simply call bedtools..
>>
>> Check the command by typing 'bedtools intersect' in a terminal, if it sends
>> you back the option list, the Galaxy tool should work.. If not, you have to
>> check your bedtools install
>>
>>>>> - Here is the error I get when running the tool ChIPmunk (thanks Alban
>>>>> for you help (ChIPMunk v2 is used and all librairies are found) but
>>>>> here is a new error, maybe a problem of "dirname" ?)
>>>>>
>>>>> mv: impossible d'évaluer
>>>>> «
>>>>> /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png
>>>>> »: Aucun fichier ou dossier de ce type
>>>>> mv: impossible d'évaluer « /work/gala
>>>>>
>> Here, I think that the error is due to the ChIPMunk execution which doesn't
>> work.
>>
>> You can try the ChIPMunk execution this way:
>>
>> Find the line containing:
>>
>> ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0
>> s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null
>>
>> Add a line with:
>>
>> echo "ruby $CHIPMUNK_BIN_DIR/run_chiphorde.rb $LOG $motifs $MODE yes 1.0
>> s:$inputfile 100 10 1 4 >> /dev/null 2>> /dev/null" > ~/ChIPMunk.log
>>
>> Then try to launch the command in ~/ChIPMunk.log in a terminal and look at
>> the output.
>>
>> If ChIPMunk has run, the error comes from next step (i.e. the
>> parseChipmunkOutput.pl script).
>>
>> Else, try to debug ChIPMunk, and then the tool will work.
>>>>> - Here is the error I get when running the tool PeaksToBed
>>>>> readline() on closed filehandle FILE at
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 124 (#1)
>>>>> (W closed) The filehandle you're reading from got itself closed
>>>>> sometime
>>>>> before now. Check your control flow.
>>>>>
>>>>> Use of uninitialized value $fields[1] in concatenation (.) or string at
>>>>>
>>
>> For this one, edit the PeaksToBed.pl script this way:
>>
>> The 3 lines:
>>
>> use strict;
>> use warnings;
>> use diagnostics;
>>
>> becomes:
>>
>> #use strict;
>> #use warnings;
>> #use diagnostics;
>>
>> Galaxy doesn't really love to have the return of a script (even if it's not
>> an error). Here the perl scripts gives you warnings that are understood as
>> errors by Galaxy.
>>
>>
>> Hope It will help you, don't hesitate if you need more..
>>
>> ++,
>>
>> Alban
>>
>>
>>
>> --
>> Alban Lermine
>> Unité 900: INSERM - Mines ParisTech - Institut Curie
>> " Bioinformatics and Computational Systems Biology of Cancer"
>> 11-13 rue Pierre et Marie Curie (1er étage) - 75005 Paris - France
>> Tel: +33 (0) 1 56 24 69 84
>>
>>
>>
>> Le 1 févr. 2013 à 14:41, Sarah Maman <sarah.ma...@toulouse.inra.fr> a écrit :
>>
>>> Thanks Bjoern,
>>>
>>> In your xml file, I 've specified the interpreter :
>>> <command> replace by <command interpreter="bash"> but the problem is not
>>> solved (";" is supported by our cluster) because the command is runned with
>>> an error ";;" (double ;) instead of ";" (simple ;) :
>>> `bash /my/path/to/RepeatMasker -parallel 8 -species fungi -dir
>>> /work/tmp/tmprRrJh2 -gff
>>> /my/path/to/galaxy/database/files/005/dataset_5979.dat > /dev/null 2>
>>> /dev/null;
>>> cp /my/path/to/work/tmp/tmprRrJh2/dataset_5979.dat.tbl
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat;
>>>
>>> cp /work/tmp/tmprRrJh2/dataset_5979.dat.out.gff None;
>>> cp /work/tmp/tmprRrJh2/dataset_5979.dat.masked None;
>>> tail -n +4 /work/tmp/tmprRrJh2/dataset_5979.dat.out | tr -s ' ' '\t' >
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat;
>>>
>>> rm /work/tmp/tmprRrJh2 -r;;
>>> cd /my/path/to/galaxy/galaxy-dist;
>>> /my/path/to/galaxy/galaxy-dist/set_metadata.sh
>>> /my/path/to/galaxy/database/files
>>> /my/path/to/galaxy/database/job_working_directory/004/4135 .
>>> /my/path/to/galaxy/galaxy-dist/universe_wsgi.ini
>>> /work/tmp/tmpi0u8TZ
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy.json
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6156_Lp_2X3,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6156_emJ62i,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6156_Dgw3la,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6156_aTCjOU,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5999.dat,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6156_I6enyB
>>>
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_in_HistoryDatasetAssociation_6155_LRm2gT,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_kwds_HistoryDatasetAssociation_6155_4xAztH,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_out_HistoryDatasetAssociation_6155_14AYvT,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_results_HistoryDatasetAssociation_6155_l_1gJ2,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/galaxy_dataset_5998.dat,
>>> /my/path/to/galaxy/database/job_working_directory/004/4135/metadata_override_HistoryDatasetAssociation_6155_NufEns'
>>>
>>> So the error is ";;"
>>> rm /work/tmp/tmprRrJh2 -r;;
>>> instead of ";"
>>> rm /work/tmp/tmprRrJh2 -r;
>>> I've tried to replace ";" by nothing but this generate others errors (files
>>> not generated) :
>>> cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.tbl »:
>>> Aucun fichier ou dossier de ce type
>>> cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.out.gff »:
>>> Aucun fichier ou dossier de ce type
>>> cp: impossible d'évaluer « /work/tmp/tmpga_QOo/dataset_5979.dat.masked »:
>>> Aucun fichier ou dossier de ce type
>>> tail: impossible d'ouvrir « /work/tmp/tmpga_QOo/dataset_5979.dat.out » en
>>> lecture: Aucun fichier ou dossier de ce type
>>> Do you have any idea ?
>>> Thanks in advance,
>>> Sarah
>>>
>>>
>>>
>>>
>>> Björn Grüning a écrit :
>>>>
>>>> Hi Sarah,
>>>>
>>>> i looks like something is wrong with the PBS system.
>>>> The errors for bedtools and repeatmasker are the same. I can't
>>>> understand french but i guess that ";" are not supported by PBS or in
>>>> your configuration? Repeatmasker and bedtools creating more than one
>>>> shell command and concatenating them with ';'.
>>>>
>>>> Hope that helps,
>>>> Bjoern
>>>>
>>>>
>>>>> Hello,
>>>>>
>>>>> I've installed NEBULA tools and "RepeatMasker" tool on our local
>>>>> Galaxy instance.
>>>>>
>>>>> For some tools (others work), I get the following errors:
>>>>> - Here is the error I get when running the tool FilterControl
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007fe56800ecd0 ***
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007fe56800ecd0 ***
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007fe56800ecd0 ***
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007fe56800ecd0 ***
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh:
>>>>> line 80: 61310 Abandon java -classpath $LOCAL_PATH/
>>>>> -Xmx6g FilterPeaks -f $inputfile -c $controlfile -t $minHeight -v
>>>>> $minRatio -o $output > /dev/null
>>>>> *** glibc detected *** java: double free or corruption (!prev):
>>>>> 0x00007f1f5800ecd0 ***
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/FilterControlPeaks/FilterControlPeaks.sh:
>>>>> line 81: 61351 Abandon
>>>>>
>>>>> - Here is the error I get when running the tool IntersectBed
>>>>>
>>>>> /work/galaxy/database/pbs/galaxy_4129.sh: line 13: Erreur de syntaxe
>>>>> près du symbole inattendu « ;; »
>>>>> /work/galaxy/database/pbs/galaxy_4129.sh: line 13: `bedtools intersect
>>>>> -f 0.05
>>>>>
>>>>> - Here is the error I get when running the tool ChIPmunk (thanks Alban
>>>>> for you help (ChIPMunk v2 is used and all librairies are found) but
>>>>> here is a new error, maybe a problem of "dirname" ?)
>>>>>
>>>>> mv: impossible d'évaluer
>>>>> «
>>>>> /work/galaxy/database/job_working_directory/004/4132/galaxy_dataset_5992.dat_0.xml.png
>>>>> »: Aucun fichier ou dossier de ce type
>>>>> mv: impossible d'évaluer « /work/gala
>>>>>
>>>>>
>>>>> - Here is the error I get when running the tool PeaksToBed
>>>>> readline() on closed filehandle FILE at
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 124 (#1)
>>>>> (W closed) The filehandle you're reading from got itself closed
>>>>> sometime
>>>>> before now. Check your control flow.
>>>>>
>>>>> Use of uninitialized value $fields[1] in concatenation (.) or string at
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 154, <FILE> line 1 (#2)
>>>>> (W uninitialized) An undefined value was used as if it were already
>>>>> defined. It was interpreted as a "" or a 0, but maybe it was a
>>>>> mistake.
>>>>> To suppress this warning assign a defined value to your variables.
>>>>>
>>>>> To help you figure out what was undefined, perl will try to tell you
>>>>> the
>>>>> name of the variable (if any) that was undefined. In some cases it
>>>>> cannot
>>>>> do this, so it also tells you what operation you used the undefined
>>>>> value
>>>>> in. Note, however, that perl optimizes your program and the operation
>>>>> displayed in the warning may not necessarily appear literally in your
>>>>> program. For example, "that $foo" is usually optimized into "that "
>>>>> . $foo, and the warning will refer to the concatenation (.) operator,
>>>>> even though there is no . in your program.
>>>>>
>>>>> Use of uninitialized value in concatenation (.) or string at
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 154, <FILE> line 1 (#2)
>>>>> Use of uninitialized value in numeric ge (>=) at
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 156, <FILE> line 1 (#2)
>>>>> ......
>>>>>
>>>>> /usr/local/bioinfo/src/galaxy/galaxy-dist/tools/Nebula/Peak2Bed/peak2bed.pl
>>>>> line 156, <FILE> line 63 (#2)
>>>>>
>>>>> Use of uninitialized value $fields[1] in conc
>>>>>
>>>>>
>>>>> I dowloaded tool "RepeatMasker" from Galaxy Tool Shed,and RepeatMasker
>>>>> is installed on our cluster.
>>>>> Here is the error I get when running the tool RepeatMasker:
>>>>>
>>>>> An error occurred running this job: Epilog : job finished at jeu.
>>>>> janv. 31 14:39:16 CET 2013
>>>>> /work/galaxy/database/pbs/galaxy_4134.sh: line 13: Erreur de syntaxe
>>>>> près du symbole inattendu « ;; »
>>>>> /work/galaxy/database/pbs/galaxy_4134.sh: line 13: `RepeatMasker
>>>>> -parallel 8 -speci
>>>>>
>>>>>
>>>>> Do you have any ideas ?
>>>>> Thank you in advance for your help,
>>>>> Sarah Maman
>>>>> --
>>>>> --*--
>>>>> Sarah Maman
>>>>> INRA - LGC - SIGENAE
>>>>> http://www.sigenae.org/
>>>>> Chemin de Borde-Rouge - Auzeville - BP 52627
>>>>> 31326 Castanet-Tolosan cedex - FRANCE
>>>>> Tel: +33(0)5.61.28.57.08
>>>>> Fax: +33(0)5.61.28.57.53
>>>>> --*--
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> --*--
>>> Sarah Maman
>>> INRA - LGC - SIGENAE
>>> http://www.sigenae.org/
>>> Chemin de Borde-Rouge - Auzeville - BP 52627
>>> 31326 Castanet-Tolosan cedex - FRANCE
>>> Tel: +33(0)5.61.28.57.08
>>> Fax: +33(0)5.61.28.57.53
>>> --*--
>>
>
>
> --
> --*--
> Sarah Maman
> INRA - LGC - SIGENAE
> http://www.sigenae.org/
> Chemin de Borde-Rouge - Auzeville - BP 52627
> 31326 Castanet-Tolosan cedex - FRANCE
> Tel: +33(0)5.61.28.57.08
> Fax: +33(0)5.61.28.57.53
> --*--
___________________________________________________________
The Galaxy User list should be used for the discussion of
Galaxy analysis and other features on the public server
at usegalaxy.org. Please keep all replies on the list by
using "reply all" in your mail client. For discussion of
local Galaxy instances and the Galaxy source code, please
use the Galaxy Development list:
http://lists.bx.psu.edu/listinfo/galaxy-dev
To manage your subscriptions to this and other Galaxy lists,
please use the interface at:
http://lists.bx.psu.edu/
