Re: [galaxy-dev] Implementing blend4j
Nice, this works! Maybe after some testing we can try to get the package in the central maven repositories. I think for now all the functionality I need is in your version of blend4j. But if I think of something new I'll try to participate. Thanks On 13 July 2013 08:58, John Chilton chil...@msi.umn.edu wrote: So you have packaged blend4j and all of its dependencies into one jar file? I don't have any experience doing that. I could imagine potentially some files conflicting and only one version being written and so jersey-json breaks? Is this a possibility, some of the jar metadata sort of files? I have used blend4j a few different ways (in a war, as a Java web start application, and as a standalone application) but always with applications where it is in its own jar file and all of its dependencies are in their own and original jar files. So I would recommend using it in that mode, if that is not possible however I would search the web for others having problems repackaging jersey-json. As for development, you can use my artifactory server to grab the canonical blend4j, by adding the following to your pom.xml file: repositories repository idmsi-artifactory/id nameMSI Artifactory/name urlhttp://artifactory.msi.umn.edu/libs-snapshot/url /repository /repositories dependencies dependency groupIdcom.github.jmchilton.blend4j/groupId artifactIdblend4j/artifactId version0.1-SNAPSHOT/version /dependency If instead you want to test your own changes to blend4j, I believe you can just do a mvn install or something like that from the blend4j fork directory on your development server to install a local copy into your maven cache. blend4j is the first project I have used maven with, so some of my terminology could be wrong there, but hopefully the idea makes sense. Also happy to look at pull requests if it helps to push changes upstream. Sorry if this response is not more helpful, let me know if there is anything else I can do or answer. -John On Thu, Jul 11, 2013 at 4:37 AM, Eric Kuyt eric.ku...@wur.nl wrote: Hello all, I'm trying to implement blend4j in a java application. for this I cloned the source, fetched the dependencies and added a maven assembly plugin to assemble a jar file with dependencies. This jar I put in my classpath. Now fetching histories works, but creating a new one fails. Online I see a lot of jersey users have this problem and it seems like jersey-json is not available, but jersey-json is packaged in the jar. Anyone else have this problem? @John maybe you have better practices? I would like to have blend4j in my maven dependencies, but not build my own nexus. Thanks, Eric -- Central Veterinary Institute of Wageningen UR (CVI) Department of Infection Biology PO box 65, 8200 AB Lelystad, NL Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad Tel: +31-(0)320-293391 Fax: +31-(0)320-238153 E-mail: eric.ku...@wur.nl Web: http://www.cvi.wur.nl -- Central Veterinary Institute of Wageningen UR (CVI) Department of Infection Biology PO box 65, 8200 AB Lelystad, NL Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad Tel: +31-(0)320-293391 Fax: +31-(0)320-238153 E-mail: eric.ku...@wur.nl Web: http://www.cvi.wur.nl ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] No samtools build after building index through Data Manager.
Hi Dan,
Thanks for your reply.
I don't handle the updates, but it was updated on 25 Jun 13. I'm pretty
sure I can rule out the version number as the BWA builds, created using
the Data Manager, work fine.
Here is the pertinent contents of shed_tool_data_table_conf.xml
?xml version=1.0?
tables
table comment_char=# name=all_fasta
columnsvalue, dbkey, name, path/columns
file
path=/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_bwa_index_builder/fe6508204acc/all_fast
a.loc /
tool_shed_repositorytool_shedtesttoolshed.g2.bx.psu.edu/tool_shedrep
ository_namedata_manager_bwa_index_builder/repository_namerepository_ow
nerblankenberg/repository_ownerinstalled_changeset_revisionfe6508204ac
c/installed_changeset_revision/tool_shed_repository/table
table comment_char=# name=all_fasta
columnsvalue, dbkey, name, path/columns
file
path=/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/all_f
asta.loc /
tool_shed_repositorytool_shedtesttoolshed.g2.bx.psu.edu/tool_shedrep
ository_namedata_manager_sam_fa_index_builder/repository_namerepository
_ownerblankenberg/repository_ownerinstalled_changeset_revision926e5039
7b83/installed_changeset_revision/tool_shed_repository/table
table comment_char=# name=sam_fa_indexes
columnsline_type, value, path/columns
file
path=/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_sam_fa_index_builder/926e50397b83/sam_f
a_indices.loc /
tool_shed_repositorytool_shedtesttoolshed.g2.bx.psu.edu/tool_shedrep
ository_namedata_manager_sam_fa_index_builder/repository_namerepository
_ownerblankenberg/repository_ownerinstalled_changeset_revision926e5039
7b83/installed_changeset_revision/tool_shed_repository/table
table comment_char=# name=all_fasta
columnsvalue, dbkey, name, path/columns
file
path=/home/galaxy/software/galaxy-central/tool-data/testtoolshed.g2.bx.psu
.edu/repos/blankenberg/data_manager_fetch_genome_all_fasta/ca8b3709309e/all
_fasta.loc /
tool_shed_repositorytool_shedtesttoolshed.g2.bx.psu.edu/tool_shedrep
ository_namedata_manager_fetch_genome_all_fasta/repository_namereposito
ry_ownerblankenberg/repository_ownerinstalled_changeset_revisionca8b37
09309e/installed_changeset_revision/tool_shed_repository/table
/tables
in ${GALAXY_HOME}/tool_data_table_conf.xml.sample the entry for sam_fa
reads as so:
!-- Location of SAMTools indexes and other files --
table name=sam_fa_indexes comment_char=#
columnsline_type, value, path/columns
file path=tool-data/sam_fa_indices.loc /
/table
The file tool-data/sam_fa_new_indices.loc (and .sample) does not exist.
If I keep the manually inserted builds listed in
tool-data/sam_fa_indices.loc and restart Galaxy, then I get the following
(abridged) entries in the paster.log:
galaxy.tools.data DEBUG 2013-07-15 09:51:11,109 Loaded tool data table
'all_fasta'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,115 Loaded tool data table
'bwa_indexes'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,116 Loaded tool data table
'bwa_indexes_color'
galaxy.tools.data DEBUG 2013-07-15 09:51:11,167 Loaded tool data table
'sam_fa_indexes'
...
galaxy.tools.data DEBUG 2013-07-15 09:51:11,324 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,340 Loading another instance
of data table 'bwa_indexes', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,348 Loading another instance
of data table 'bwa_indexes_color', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,410 Loading another instance
of data table 'all_fasta', attempting to merge content.
galaxy.tools.data DEBUG 2013-07-15 09:51:11,422 Loading another instance
of data table 'sam_fa_indexes', attempting to merge content.
galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields
(['index', 'cfraxinea_s1v1',
'/home/galaxy/software/galaxy-central/tool-data/cfraxinea_s1v1/sam_index/cf
raxinea_s1v1/c_fraxinea_s1v1.fa']) to data table 'sam_fa_indexes', but
this entry already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,422 Attempted to add fields
(['index', 'b_distachyon',
'/home/galaxy/software/galaxy-central/tool-data/b_distachyon/sam_index/b_di
stachyon/b_distachyon.fa']) to data table 'sam_fa_indexes', but this entry
already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields
(['index', 'n_sylvestris',
'/home/galaxy/software/galaxy-central/tool-data/n_sylvestris/sam_index/n_sy
lvestris/n_sylvestris.fa']) to data table 'sam_fa_indexes', but this entry
already exists and allow_duplicates is False.
galaxy.tools.data ERROR 2013-07-15 09:51:11,423 Attempted to add fields
(['index',
[galaxy-dev] Display data in browser
Hello list, I'm currently trying to create a new tool, which has an .xhtml file as its output. After finishing the job, I would like to be able to click on the eye icon aka Display data in browser and display the resulting .xhtml file in the frame to the left of my tool history. Unfortunately I have no idea yet how I could accomplish this. Any help is highly appreciated. Best regards, Gromobir ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] CloudMan Error
Thanks for getting back to me, Enis.
I went ahead and started a new cluster instance. I'll try to leave it
running today in case there's anything we want to check.
Here's my whole process:
Start Screen Setup:
--
http://snag.gy/DMIeC.jpg
Entering my share string:
http://snag.gy/wKFLy.jpg
Main Page Text and log:
Cluster name:
MSGGREG
Disk status:
0 / 0 (0%)
Worker status:
Idle: 0 Available: 0 Requested: 0
Service status:
Applications
Data
Cluster status log
14:08:32 - Master starting
14:08:34 - Completed the initial cluster startup process. This is a
new cluster; waiting to configure the type.
14:08:50 - Migration service prerequisites OK; starting the service
14:08:50 - SGE service prerequisites OK; starting the service
14:08:58 - Setting up SGE...
14:09:13 - HTCondor service prerequisites OK; starting the service
14:09:21 - Hadoop service prerequisites OK; starting the service
14:09:38 - Done adding Hadoop service; service running.
14:11:53 - Error creating volume from shared cluster's snapshot
'['snap-cfa775ba']': 'filesystems'
Admin Page CloudMan Log: (unfortunately nothing is jumping out at me?)
---
CloudMan from Galaxy
Admin | Report bugs | Wiki | Screencast
The entire log file (paster.log) is shown. Show latest | Back to admin view
Python version: (2, 7)
Image configuration suports: {'apps': ['cloudman', 'galaxy']}
2013-07-15 14:08:32,406 DEBUGapp:68 Initializing app
2013-07-15 14:08:32,407 DEBUGec2:121 Gathering instance
zone, attempt 0
2013-07-15 14:08:32,410 DEBUGec2:127 Instance zone is 'us-east-1d'
2013-07-15 14:08:32,410 DEBUGec2:45 Gathering instance
ami, attempt 0
2013-07-15 14:08:32,412 DEBUGapp:71 Running on 'ec2'
type of cloud in zone 'us-east-1d' using image 'ami-118bfc78'.
2013-07-15 14:08:32,412 DEBUGapp:89 Getting pd.yaml
2013-07-15 14:08:32,412 DEBUGec2:338 No S3 Connection,
creating a new one.
2013-07-15 14:08:32,413 DEBUGec2:342 Got boto S3 connection.
2013-07-15 14:08:32,452 DEBUG misc:212 Checking if bucket
'cm-0479bd75a331acc874033e98b2e1e03e' exists... it does not.
2013-07-15 14:08:32,452 DEBUG misc:583 Bucket
'cm-0479bd75a331acc874033e98b2e1e03e' does not exist, did not get
remote file 'persistent_data.yaml'
2013-07-15 14:08:32,452 DEBUGapp:96 Setting
deployment_version to 2
2013-07-15 14:08:32,453 INFO app:103 Master starting
2013-07-15 14:08:32,453 DEBUG master:55 Initializing console
manager - cluster start time: 2013-07-15 14:08:32.453182
2013-07-15 14:08:32,453 DEBUG comm:42 AMQP Connection
Failure: [Errno 111] Connection refused
2013-07-15 14:08:32,453 DEBUG master:791 Trying to discover
any worker instances associated with this cluster...
2013-07-15 14:08:32,454 DEBUGec2:317 Establishing boto
EC2 connection
2013-07-15 14:08:32,535 DEBUGec2:305 Got region as
'RegionInfo:us-east-1'
2013-07-15 14:08:32,777 DEBUGec2:326 Got boto EC2
connection for region 'us-east-1'
2013-07-15 14:08:33,022 DEBUG misc:574 Retrieved file
'snaps.yaml' from bucket 'cloudman' on host 's3.amazonaws.com' to
'cm_snaps.yaml'.
2013-07-15 14:08:33,035 DEBUGec2:286 Got region name as 'us-east-1'
2013-07-15 14:08:33,035 DEBUG master:226 Loaded default
snapshot data: [{'snap_id': 'snap-adad90fc', 'name': 'galaxy',
'roles': 'galaxyTools,galaxyData'}, {'snap_id': 'snap-5b030634',
'name': 'galaxyIndices', 'roles': 'galaxyIndices'}]
2013-07-15 14:08:33,035 DEBUGec2:81 Gathering instance
id, attempt 0
2013-07-15 14:08:33,037 DEBUGec2:87 Instance ID is 'i-5346d733'
2013-07-15 14:08:33,125 DEBUGec2:360 Adding tag
'clusterName:MSGGREG' to resource 'i-5346d733'
2013-07-15 14:08:33,307 DEBUGec2:360 Adding tag
'role:master' to resource 'i-5346d733'
2013-07-15 14:08:33,554 DEBUGec2:360 Adding tag
'Name:master: MSGGREG' to resource 'i-5346d733'
2013-07-15 14:08:33,744 DEBUG master:246 ud at manager start:
{'region_name': u'us-east-1', 'region_endpoint': u'ec2.amazonaws.com',
'ec2_port': None, 'deployment_version': 2, 'cloud_name': u'Amazon',
'boot_script_name': 'cm_boot.py', 'is_secure': True, 'password':
'', 'access_key': 'redacted!', 's3_port': None, 'cloud_type':
u'ec2', 'cloudman_home': '/mnt/cm', 'cluster_name': u'MSGGREG',
'freenxpass': u'', 'bucket_default': 'cloudman', 'role': 'master',
'bucket_cluster': 'cm-0479bd75a331acc874033e98b2e1e03e',
'boot_script_path': '/tmp/cm', 'secret_key': u'redacted!',
's3_conn_path': u'/', 's3_host': u's3.amazonaws.com', 'ec2_conn_path':
u'/'}
2013-07-15 14:08:33,744 DEBUG master:1858 Generating root
user's public key...
2013-07-15 14:08:33,763 DEBUG
[galaxy-dev] [GSoC2013] Week 4 Accomplishments and Week 5 Plans
Hi All, Week 4 blog post : http://galaxy-gsoc2013.blogspot.com/2013/07/week-4-accomplishments-and-week-5-plans.html Do drop in your comments , if any. Thanks Saket ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Error Uploading Directory of Files
On Jun 28, 2013, at 12:35 PM, Nicholas Kline wrote: Hi, Our lab recently installed a local version of Galaxy on a mid-2012 Mac Pro computer. We can access the Galaxy server and sign in as an administrator. Today we tried creating a Data Library, adding a dataset to it, and uploading a directory of files. We followed the Galaxy documentation at http://wiki.galaxyproject.org/Admin/DataLibraries/UploadingLibraryFiles?action=showredirect=Admin%2FData+Libraries%2FUploading+Library+Files to setup this feature: - Admin Data Library Add datasets Upload directory of files - file format was set to auto-detect - and we chose the option to link to files instead of copying them Galaxy confirmed that the files were successfully uploaded. However, in the data library, under the Message column, is a message in red saying Job error (click name for more info). Clicking on one of the uploaded files displays a page with this information: Date uploaded: 2013-06-28 File size: 7.5 GB Data type: auto Build: sacCer2 Miscellaneous information: Traceback (most recent call last): File /Users/administrator/galaxy-dist/tools/data_source/upload.py, line 386, in __main__() File /Users/administrator/galaxy-dist/tools/data_source/upload.py, line 357, in __main__ output_paths = Job Standard Error Traceback (most recent call last): File /Users/administrator/galaxy-dist/tools/data_source/upload.py, line 386, in __main__() File /Users/administrator/galaxy-dist/tools/data_source/upload.py, line 357, in __main__ output_paths = parse_outputs( sys.argv[4:] ) File /Users/administrator/galaxy-dist/tools/data_source/upload.py, line 64, in parse_outputs id, files_path, path = arg.split( ':', 2 ) ValueError: need more than 1 value to unpack error Database/Build: sacCer2 Number of data lines: None Disk file: /Volumes/G-SPEED Q/data/Person 2012 project/DCP2 mef.fastq Hi Nicholas, Could you try re-adding the data to the library with the spaces removed from all path components? I thought this had been fixed a long time ago but it's possible that the bug reappeared at some point. --nate Questions: 1. Should we be concerned about this error? 2. If so, what is the right way to fix it? 3. If not, how do we remove the red error message next to each file: Job error (click name for more info) ? Thank you ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy Server Questions
Hi Zain, The minimum requirements for Galaxy are pretty much any recent model computer. A basic install will load and run within a very short amount of time, following the instructions at: http://getgalaxy.org Advanced tools come from the Tool Shed. These are installed separately. No programming is needed, but some very basic unix skills are required, as they would be for ongoing server maintenance. The Tool Shed is one of the best documented parts of Galaxy, so you should be able to find all the answers here, or this is the right list to use for questions about local installs (galaxy-...@bx.psu.edu): http://wiki.galaxyproject.org/Tool%20Shed However, running a production instance that is intended to run compute intensive tools (such as Tophat2), and where you have some throughput goals, will require more substantial resources. This is always a difficult question to answer since so much depends on the tools used and the data volume. But in general, minimum requirements are about around the same as what those underlying tools would require on their own, if run line-command. So for Tophat or Tophat2, and really the entire Tuxedo RNA-seq tool package, you might be able to get by on 8G memory and 2 or 4 cores. But, it will probably be slow and if you are running replicates through Cuffdiff at the end, you might run out of memory if the files are large and the genome is large (such as human). And if you are are hosting a web Galaxy at the same time with visualizations and such, well, this is why systems are often set up with clusters. With all of these will be competing for the same resources, going low can work, but this will be something that will have to be managed/tested, and it will change through time as tools upgrade. You could test this out by setting up a cloud instance with the hardware you plan to use, loading some of your data, and running your workflow to see how this benchmarks. Have users on the instance while you are doing this - to judge performance. Cloud installations come with many of the advanced tools and data indexes already installed/configured, so this would be less investment than buying the hardware first and finding out later it was not not enough. http://usegalaxy.org/cloud And of course Slipstream Galaxy is an option. The whole intention here is to make a complete package with tools data already configured, in a system that has enough compute capability to do the work, for scientists/labs who do not want to deal with administrative tasks or work in the cloud (for whatever reason). http://bioteam.net/slipstream/galaxy-edition/ Good luck with your decision! Other are welcomed to comment about how they have set up their system. Jen Galaxy team On 7/15/13 7:14 AM, Zain A Alvi wrote: Hi Jen, I hope this reaches you well. I have a small question in regards to setting up a galaxy server. My mentors and I are looking into buying a server for doing NGS analysis through the use of Galaxy. We saw that slipstream's specifications for hardware to be the following: CPU: 2x Intel Xeon Processor E5-2690, 8 core (16 cores total) RAM Memory: 12x 32GB RDIMM (384 GB) with option to upgrade it to 512 GB Storage (Hard Drive space): 7x 3TB SAS 6 Gbps (16 TB usable) with 1 x 100GB Solid State Disk Power: Dual Pedundant Power Supplies Network: Dual Gigabit Network Adapter We are wondering what can be the minimum server hardware specifications that Galaxy be run on. Our second question is if we install Galaxy on the server, do all the tools currently available on Galaxy come pre-installed on it or do we have to program (Via Perl and Python) and install each of those tool sets ourselves. If we have to install those tools ourselves, is there a guide that we can do so? Lastly, how can we upgrade the tool sets such as Tophat 1.44 to Tophat 2 on this server. I was wondering about the last question as Tophat 1.44 is available on the main Galaxy server whereas Tophat 2 is available on the test Galaxy server. Sorry for so many questions. Thank you again for all the great help. Sincerely, Zain -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Connecting to new Galaxy Cloudman by FTP
On Jul 8, 2013, at 3:50 PM, Mohammad Heydarian wrote: Hi, I am having trouble setting up a FTP connection with the recently released version of Galaxy Cloudman (ami-118bfc78). I have instantiated the new version of Galaxy Cloudman with CloudLaunch and also through the AWS EC2 wizard (using the same security group settings as the previous versions) and neither instance will connect to my FTP connection. Has anyone else had this problem? Does anyone know what is preventing the FTP connection? Any help would be greatly appreciated. Hey Mo, This may be a case of the new password hashing algorithm's incompatibility with the provided ProFTPD config. Could you try the following: 1. Set 'use_pbkdf2 = False' in universe_wsgi.ini anywhere in the [app:main] section 2. Restart Galaxy 3. Reset your password in the Galaxy UI 4. Test FTP again --nate Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] plain text output format
Hi All I'm trying to format one of the three outputs of my galaxy tool as plain text. Thus far no matter how I specify my output format in the tool xml file I have the same problem. When I click the eye icon it opens a save dialog rather that displaying the text output in the central frame. The preview shows the text output and if I follow through and save the output and open with wordpad it is fine. Also if I edit the attributes of the output and change it to txt (which is one of the formats I had tried specifying in the xml) it will render in the central frame but I would rather not have to do that. Both of my other outputs (a vcf and a tab-delimited file are rendered in the central frame properly). What can I do to fix this? Thanks Mark This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Upstart script to manage a multi instance load balanced installation
After some work i've created an Upstart script which can manage a load
balanced galaxy configuration as described in the wiki. I thought that I
would put it on this list for other people to use. The script parses
universe_wsgi.ini just like run.sh and spawns all of the servers it finds.
It comes in two pieces galaxy.conf and galaxy-worker.conf. Once you place
them both in /etc/init and make the proper edits for the environment a
server can be started with sudo start galaxy. The configuration starts
the server at boot time and puts all of the instances under the management
of upstart which deals with pids, logging to syslog and respawning if an
instance crashes.
I have just gotten this working reasonably well but have done basically no
testing so there are bugs to be found. Any comments are welcome if anyone
knows a better way to do something here.
- Seth
*galaxy.conf*
author Seth Sims seth.s...@gmail.com
version 0.0.1 test
description galaxy master process. Fetches eggs and spawns all of the
servers it finds configured
start on started network-services
# make sure that any eggs we download are at least owned by the galaxy
group.
# we cannot use setuid in this script because only root can issue the
start galaxy-worker
# command. But this way the galaxy instances should still be able to use
their eggs.
setgid galaxy
# put galaxy root directory here
chdir /srv/galaxy-dist/
pre-start script
date
echo checking python version
python ./scripts/check_python.py
[ $? -ne 0 ] exit 1
echo pre-fetching tossing out expired eggs
python ./scripts/check_eggs.py -q
if [ $? -eq 0 ]; then
echo Some eggs are out of date, attempting to fetch...
python ./scripts/fetch_eggs.py
if [ $? -eq 0 ]; then
echo Fetch Successful.
else
echo Fetch failed.
fi
fi
echo starting servers
SERVERS=`sed -n 's/^\[server:\(.*\)\]/\1/ p' universe_wsgi.ini | xargs
echo`
for SERVER in ${SERVERS} ; do
echo starting server ${SERVER}
start galaxy-worker SERVER_NAME=${SERVER}
done
end script
post-stop script
SERVERS=`sed -n 's/^\[server:\(.*\)\]/\1/ p' universe_wsgi.ini | xargs
echo`
date
echo stopping galaxy servers.
for SERVER in ${SERVERS} ; do
echo stopping ${SERVER}
stop galaxy-worker SERVER_NAME=${SERVER}
done
end script
---
*galaxy-worker*
author Seth Sims seth.s...@gmail.com
version 0.0.1 test
description Starts a galaxy server instance. This is run from the
galaxy.conf file
instance $SERVER_NAME
#make sure we are running as the galaxy user
setuid galaxy
setgid galaxy
#put the galaxy root directory here
chdir /srv/galaxy-dist/
#having multiple instances of galaxy using the same egg directory was
causing a race
#condition that was stopping the instances from starting correctly. So give
each instance
#its own directory under /tmp
env PYTHON_EGG_CACHE=/tmp/${SERVER_NAME}_egg/
respawn
script
exec python ./scripts/paster.py serve universe_wsgi.ini
--server-name=${SERVER_NAME}
end script
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Implementing blend4j
Fantastic. blend4j definitely should be in a public repository, just don't know how and have not had the time. Any help in that process would be most welcome. -John On Jul 15, 2013 10:43 AM, Eric Kuyt eric.ku...@wur.nl wrote: Nice, this works! Maybe after some testing we can try to get the package in the central maven repositories. I think for now all the functionality I need is in your version of blend4j. But if I think of something new I'll try to participate. Thanks On 13 July 2013 08:58, John Chilton chil...@msi.umn.edu wrote: So you have packaged blend4j and all of its dependencies into one jar file? I don't have any experience doing that. I could imagine potentially some files conflicting and only one version being written and so jersey-json breaks? Is this a possibility, some of the jar metadata sort of files? I have used blend4j a few different ways (in a war, as a Java web start application, and as a standalone application) but always with applications where it is in its own jar file and all of its dependencies are in their own and original jar files. So I would recommend using it in that mode, if that is not possible however I would search the web for others having problems repackaging jersey-json. As for development, you can use my artifactory server to grab the canonical blend4j, by adding the following to your pom.xml file: repositories repository idmsi-artifactory/id nameMSI Artifactory/name urlhttp://artifactory.msi.umn.edu/libs-snapshot/url /repository /repositories dependencies dependency groupIdcom.github.jmchilton.blend4j/groupId artifactIdblend4j/artifactId version0.1-SNAPSHOT/version /dependency If instead you want to test your own changes to blend4j, I believe you can just do a mvn install or something like that from the blend4j fork directory on your development server to install a local copy into your maven cache. blend4j is the first project I have used maven with, so some of my terminology could be wrong there, but hopefully the idea makes sense. Also happy to look at pull requests if it helps to push changes upstream. Sorry if this response is not more helpful, let me know if there is anything else I can do or answer. -John On Thu, Jul 11, 2013 at 4:37 AM, Eric Kuyt eric.ku...@wur.nl wrote: Hello all, I'm trying to implement blend4j in a java application. for this I cloned the source, fetched the dependencies and added a maven assembly plugin to assemble a jar file with dependencies. This jar I put in my classpath. Now fetching histories works, but creating a new one fails. Online I see a lot of jersey users have this problem and it seems like jersey-json is not available, but jersey-json is packaged in the jar. Anyone else have this problem? @John maybe you have better practices? I would like to have blend4j in my maven dependencies, but not build my own nexus. Thanks, Eric -- Central Veterinary Institute of Wageningen UR (CVI) Department of Infection Biology PO box 65, 8200 AB Lelystad, NL Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad Tel: +31-(0)320-293391 Fax: +31-(0)320-238153 E-mail: eric.ku...@wur.nl Web: http://www.cvi.wur.nl -- Central Veterinary Institute of Wageningen UR (CVI) Department of Infection Biology PO box 65, 8200 AB Lelystad, NL Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad Tel: +31-(0)320-293391 Fax: +31-(0)320-238153 E-mail: eric.ku...@wur.nl Web: http://www.cvi.wur.nl ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] [galaxy-user] Connecting to new Galaxy Cloudman by FTP
Hey Nate, Thanks for the response and instructions. I understand the last three steps of your protocol, but the first step is difficult for me to understand. I'm guessing that, 1. Set 'use_pbkdf2 = False' in universe_wsgi.ini anywhere in the [app:main] section, is telling me to change a setting of Cloudman by command line. This is generally where we get stuck in using Cloudman, because we aren't familiar with command line (we get cold sweats and light palpitations) and are weary of making changes to the Cloudman code. I would ask our IT guys for help, but their expertise ends at updating Office tools. I would bother a programmer or bioinformatician, but most of them are so busy you need a formal collaboration to get on their radar. I would ask people who vaguely know command line for help, but most of the time their knowledge of command line is just higher than mine and we end up troubleshooting an issue neither of us can really grasp. So, is there a webcast, or video, or slideshow, that can show a newbie how to command line into Cloudman and make the changes you outline in step 1? Cheers, Mo Heydarian On Mon, Jul 15, 2013 at 4:22 PM, Nate Coraor n...@bx.psu.edu wrote: On Jul 8, 2013, at 3:50 PM, Mohammad Heydarian wrote: Hi, I am having trouble setting up a FTP connection with the recently released version of Galaxy Cloudman (ami-118bfc78). I have instantiated the new version of Galaxy Cloudman with CloudLaunch and also through the AWS EC2 wizard (using the same security group settings as the previous versions) and neither instance will connect to my FTP connection. Has anyone else had this problem? Does anyone know what is preventing the FTP connection? Any help would be greatly appreciated. Hey Mo, This may be a case of the new password hashing algorithm's incompatibility with the provided ProFTPD config. Could you try the following: 1. Set 'use_pbkdf2 = False' in universe_wsgi.ini anywhere in the [app:main] section 2. Restart Galaxy 3. Reset your password in the Galaxy UI 4. Test FTP again --nate Cheers, Mo Heydarian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Galaxy-less Tool Installing
One of my goals for the GCC was to sell the idea that tool shed repositories need to be installable without a database present. I talked with James Taylor and Enis Afgan about this idea briefly and they seemed to believe this was a good idea - I kept meaning to discuss it with Greg but I never got a good opportunity. Though in past Greg has made this sound potentially doable and has never objected to the goal overtly. I have two specific use cases in mind (CloudBioLinux and LWR), but perhaps the higher-level justification is something along the lines that a lot of effort from Greg and others (Dave, Bjorn, Peter, Nate) has gone into building a modular dependency system that could very easily be leveraged by applications other than Galaxy, so the extra steps that could be taken to make this possible should to make the codebase as broadly useful and to encourage adoption. The Galaxy community could benefit from other applications potentially utilizing and populating the tool shed and Galaxy tool developers would be further incentized to write good, modular dependencies and publish them to the tool shed. A high-level task decomposition would be something like this: 1. Rework installing tool shed repositories to not require a database. A kind of messy way to do this might be adding a use_database flag throughout. A cleaner way might be to use allow the core functionality to work with callbacks or plugins that performed the database interactions. 2. Separate the core functionality out of the Galaxy code base entirely into a reusable, stand-alone library. I would love buy in from the Galaxy team on item 2 above, but it is not strictly needed for my goals - I imagine I could write a script to pull it out Galaxy and build the library automatically or even just have the Galaxy codebase present when using Galaxy-less tool shed dependencies. Buy in on item 1 by the Galaxy team (specifically Greg and Dave B.) however is needed, are there any objections to this idea? Do you have any broad advice on how to approach this to ensure the changes make sense, work with your long term vision, and end up in Galaxy? Of all the things on my TODO list for the next year, this is probably the most potentially broadly interesting to this weeks BOSC codefest attendees, so I was going to attempt to sell this as something to work on. The sales pitch would include building a little tool shed version of the module command - http://linux.die.net/man/1/module to demonstrate this work and have something immediately useful produced. The idea would be to create a command-line tool for utilizing tool shed dependencies. # Unlike standard module, install procedure is available. Probably could # default to main tool shed and latest installable revision % tsmodule repo:install galaxyp/tint % tsmodule repo:install toolshed.g2.bx.psu.edu/galaxyp/tint/ab43b5ba7a4e # module lets you list packages, I guess tool shed version would need # repository and package listings: % tsmodule repo:list toolshed.g2.bx.psu.edu/galaxyp/tint/ab43b5ba7a4e % tsmodule package:list tint_proteomics_scripts/1.19.19/galaxyp/tint/ab43b5ba7a4e # Finally, a use command would source the env.sh script and make dependency # available in the command-line (might require starting new shell?): % tsmodule package:use tint_proteomics_scripts % tsmodule package:use tint_proteomics_scripts/1.19.19 % tsmodule package:use int_proteomics_scripts/1.19.19/galaxyp/tint/ab43b5ba7a4e # use apps that would be available to tools with valid requirements tags. % iQuantCLI This would be different from using the API scripts because there would be no API, Galaxy instance, or Galaxy database involved - just the Galaxy code. If this was able to split into its own Python library, one could imagine even allowing something like tsmodule to be installable right from pip and recursively fetch a toolshed_client library or something like that. ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy-less Tool Installing
This would be different from using the API scripts because there would be no API, Galaxy instance, or Galaxy database involved - just the Galaxy code. If this was able to split into its own Python library, one could imagine even allowing something like tsmodule to be installable right from pip and recursively fetch a toolshed_client library or something like that. I especially like this idea. I wasn't at GCC, but my boss was, so you may have seen some of the stuff I'm working on with Web Services. Right now, the transition to the tool shed it awesome, except for when it comes time to actually run the tools for adding web services. You see right now, the tool my colleagues and I have developed dynamically creates tool config files based on WSDLs and WADLs. The problem is, at the moment, our tool actually edits the tools_config.xml file directly in order to add the web service operation tools to galaxy. Your idea, if I understand it correctly, would mean that it might be possible for my tool to utilize this other method of installation in order to add/remove the generated tools to Galaxy. I'd definitely like to hear more about this. -- Sincerely, Michael E. Cotterell Ph.D. Student in Computer Science, University of Georgia Graduate RA TA, University of Georgia mepcotter...@gmail.com mepc...@uga.edu m...@cs.uga.edu http://michaelcotterell.com/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Galaxy-less Tool Installing
Hi John, It's really too bad that we didn't find time to discuss this in person at the GCC. Until now, I've not heard from anyone that installation from the tool shed without requiring a Galaxy database is important, so I'm lacking some context on this (I assume your statement without a database present refers to the Galaxy database). Some concerns that immediately pop into my mind are the following - this is not a complete list. 1) How do you ensure dependency relationships between installed repositories? 2) How do you manage locating installed repository contents? 3) How do you manage the current state of an installed repostiroy and determine if it can be updated? 4) How do you manage tool version lineage for installed repositories that contain tools (this plays an importnt role in ensuring reproducibility)? 5) How do you maintain the state of an installed repository, enabling it to be repaired if it or any of it's dependencies are in an error state? Since I've never considered re-engineering the tool shed installation process so that it would function in an environment without a Galaxy database, I'm not sure how much effort would need to go into doing so, or where to start. I'll have to think about this for a while. Greg Von Kuster On Jul 15, 2013, at 7:27 PM, John Chilton chil...@msi.umn.edu wrote: One of my goals for the GCC was to sell the idea that tool shed repositories need to be installable without a database present. I talked with James Taylor and Enis Afgan about this idea briefly and they seemed to believe this was a good idea - I kept meaning to discuss it with Greg but I never got a good opportunity. Though in past Greg has made this sound potentially doable and has never objected to the goal overtly. I have two specific use cases in mind (CloudBioLinux and LWR), but perhaps the higher-level justification is something along the lines that a lot of effort from Greg and others (Dave, Bjorn, Peter, Nate) has gone into building a modular dependency system that could very easily be leveraged by applications other than Galaxy, so the extra steps that could be taken to make this possible should to make the codebase as broadly useful and to encourage adoption. The Galaxy community could benefit from other applications potentially utilizing and populating the tool shed and Galaxy tool developers would be further incentized to write good, modular dependencies and publish them to the tool shed. A high-level task decomposition would be something like this: 1. Rework installing tool shed repositories to not require a database. A kind of messy way to do this might be adding a use_database flag throughout. A cleaner way might be to use allow the core functionality to work with callbacks or plugins that performed the database interactions. 2. Separate the core functionality out of the Galaxy code base entirely into a reusable, stand-alone library. I would love buy in from the Galaxy team on item 2 above, but it is not strictly needed for my goals - I imagine I could write a script to pull it out Galaxy and build the library automatically or even just have the Galaxy codebase present when using Galaxy-less tool shed dependencies. Buy in on item 1 by the Galaxy team (specifically Greg and Dave B.) however is needed, are there any objections to this idea? Do you have any broad advice on how to approach this to ensure the changes make sense, work with your long term vision, and end up in Galaxy? Of all the things on my TODO list for the next year, this is probably the most potentially broadly interesting to this weeks BOSC codefest attendees, so I was going to attempt to sell this as something to work on. The sales pitch would include building a little tool shed version of the module command - http://linux.die.net/man/1/module to demonstrate this work and have something immediately useful produced. The idea would be to create a command-line tool for utilizing tool shed dependencies. # Unlike standard module, install procedure is available. Probably could # default to main tool shed and latest installable revision % tsmodule repo:install galaxyp/tint % tsmodule repo:install toolshed.g2.bx.psu.edu/galaxyp/tint/ab43b5ba7a4e # module lets you list packages, I guess tool shed version would need # repository and package listings: % tsmodule repo:list toolshed.g2.bx.psu.edu/galaxyp/tint/ab43b5ba7a4e % tsmodule package:list tint_proteomics_scripts/1.19.19/galaxyp/tint/ab43b5ba7a4e # Finally, a use command would source the env.sh script and make dependency # available in the command-line (might require starting new shell?): % tsmodule package:use tint_proteomics_scripts % tsmodule package:use tint_proteomics_scripts/1.19.19 % tsmodule package:use int_proteomics_scripts/1.19.19/galaxyp/tint/ab43b5ba7a4e # use apps that would be available to tools with valid requirements tags. % iQuantCLI
