[galaxy-dev] example_watch_folder.py problem importing file
Hello, I am testing the example_watch_folder.py script on my local instance of Galaxy (Feb.10 distribution). I have set up a simple workflow with the Fasta-to-Tabular tool. It takes a single input fasta file. Works in Galaxy UI. I have created input and output folders and followed the steps here http://gmod.827538.n3.nabble.com/Trouble-Shooting-example-watch-folder-py-td4030355.html But when I run the script and put a simple fasta file in the input folder I get the following error in the logs: galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) command is: python galaxy_dist_dev/tools/data_source/upload.py galaxy_dist_dev galaxy_dist_dev/database/tmp/tmpK40iV3galaxy_dist_dev/database/tmp/tmpx_5QyU 3802:galaxy_dist_dev/database/job_working_directory/001/1899/dataset_3802_files:galaxy_dist_dev/database/files/003/dataset_3802.dat; return_code=$?; cd galaxy_dist_dev; galaxy_dist_dev/set_metadata.sh ./database/files galaxy_dist_dev/database/job_working_directory/001/1899 . galaxy_dist_dev/universe_wsgi.ini galaxy_dist_dev/database/tmp/tmpK40iV3 galaxy_dist_dev/database/job_working_directory/001/1899/galaxy.json; sh -c exit $return_code galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,127 (1899) native specification is: -V -q all.q -l hostname=hostname.ca galaxy.jobs.runners.drmaa INFO 2014-04-10 11:11:03,133 (1899) queued as 678390 galaxy.jobs DEBUG 2014-04-10 11:11:03,180 (1899) Persisting job destination (destination id: name_hwew) galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:03,579 (1899/678390) state change: job is running 10.1.1.111 - - [10/Apr/2014:11:11:07 -0400] POST /api/workflows?key=dd3916dfb37dffc08f070f1e3503015a HTTP/1.1 200 - - Python-urllib/2.6 galaxy.jobs.runners.drmaa DEBUG 2014-04-10 11:11:07,778 (1899/678390) state change: job finished normally galaxy.jobs DEBUG 2014-04-10 11:11:08,082 setting dataset state to ERROR galaxy.jobs DEBUG 2014-04-10 11:11:08,227 job 1899 ended galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,227 Cleaning up external metadata files galaxy.jobs DEBUG 2014-04-10 11:11:08,652 (1900) Working directory for job is: galaxy_dist_dev/database/job_working_directory/001/1900 galaxy.datatypes.metadata DEBUG 2014-04-10 11:11:08,770 Cleaning up external metadata files galaxy.jobs.handler INFO 2014-04-10 11:11:08,795 (1900) Job unable to run: one or more inputs in error state 10.202.22.186 - - [10/Apr/2014:11:11:31 -0400] GET /history/list HTTP/1.1 200 - http://galaxy.server.ca:8080/root; Mozilla/5.0 (Windows NT 6.1; WOW64; rv:11.0) Gecko/20100101 Firefox/11.0 And in the dataset itself in the resulting Galaxy history UI WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'asn1', using datatype from galaxy_dist_dev/database/tmp/tmpK40iV3. Yesterday, before I did the upgrade to Feb.10. The input files would import (as a green dataset in the History UI) put be empty and say 'no peak' Thanks in advance for any help, Liisa ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
It should be set on line 29 as the result of that submit function. Can you
add (between 29 and 30) a 'print library' to see what that call actually
results in?
Off the top of my head, this might be happening if your user account (that
the API key is associated with) isn't an admin account -- only admins can
create Data Libraries.
On Tue, Mar 12, 2013 at 6:35 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to
add the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the API
Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow )
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if it
doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries', lib_create_data,
return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0]['id']' is set or is in scope, since this seems to be the item in
the iterator above this section.
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerchttps://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Tue, Mar 12, 2013 at 6:14 PM, Dannon Baker dannon.ba...@gmail.comwrote:
The second argument there should point to the url of Galaxy's API --
in this case http://localhost:8080/api, I'm guessing.
-Dannon
On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm trying to get the example_watch_folder.py to work with a simple
example,
but I am running into an error (Caveat, python is not my native
language)
I call the following script, but get the subsequent error:
$ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
URL did not return JSON data
The preconditions I have in place are:
1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f
2. I created a new history on http://localhost:8080
3. I downloaded the the exons from hg19/chr22 from UCSC
3. On the dataset I performed a Text Manipulation Select first (20
lines)
4. I extracted a workflow from the history and recorded the workflow id
(2a56795cad3c7db3).
5. I created directories:
/Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/
My args may be malformed. Can someone provide me an example of a theirs?
Cheers,
Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
Hi Dannon,
Thanks. I've moved to a full size monitor, apparently my eyes are going!
They *key error: 0* seems to be the result of a bug in the code.
print(library)
results in:
{'url': '/api/libraries/4b187121143038ff', 'id': '4b187121143038ff',
'name': 'API Import11'}
So library[0]['id'] doesn't point to anything, resulting in the key error.
I changed the code on line 30
library_id = library['id']
This seems to fix the issue. I've now got to the section of code where I am
watching the in_folder, but there doesn't seem to be any subsequent
activity. I'll post my findings or questions as I make progress or lack
thereof.
Cheers,
Rob
On Wed, Mar 13, 2013 at 7:35 AM, Dannon Baker dannon.ba...@gmail.comwrote:
It should be set on line 29 as the result of that submit function. Can
you add (between 29 and 30) a 'print library' to see what that call
actually results in?
Off the top of my head, this might be happening if your user account (that
the API key is associated with) isn't an admin account -- only admins can
create Data Libraries.
On Tue, Mar 12, 2013 at 6:35 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to
add the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the
API Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow )
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if it
doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries', lib_create_data,
return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0]['id']' is set or is in scope, since this seems to be the item in
the iterator above this section.
Cheers,
Rob
On Tue, Mar 12, 2013 at 6:14 PM, Dannon Baker dannon.ba...@gmail.comwrote:
The second argument there should point to the url of Galaxy's API --
in this case http://localhost:8080/api, I'm guessing.
-Dannon
On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm trying to get the example_watch_folder.py to work with a simple
example,
but I am running into an error (Caveat, python is not my native
language)
I call the following script, but get the subsequent error:
$ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
URL did not return JSON data
The preconditions I have in place are:
1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f
2. I created a new history on http://localhost:8080
3. I downloaded the the exons from hg19/chr22 from UCSC
3. On the dataset I performed a Text Manipulation Select first (20
lines)
4. I extracted a workflow from the history and recorded the workflow id
(2a56795cad3c7db3).
5. I created directories:
/Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/
My args may be malformed. Can someone provide me an example of a
theirs?
Cheers,
Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
I was able to get the script to work. I've written down some notes for a
quickstart for anyone else who might be following in my footsteps.
1. Create input and output directores where you will drop and emit files.
In my example I used:
/Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output
2. Edit the galaxy-dist configuration script (universe_wsgi.ini) and set:
allow_library_path_paste = true
3. In example_watch_folder.py, comment out line 30 and replace it with:
#library_id = library[0]['id] This causes a key error : 0
library_id = library['id']
4. CREATE A SAMPLE Workflow Dataset:
1. Start Galaxy
2. Go to USCS and upload the Exons from chr22 to your clean history file
3. Use the Text Manipulation tool to select the first 5 lines of the
gene exon file
4. Extract the workflow
5. *RESTART Galaxy* (You'll get a 500 Internal Server Error if you try
to run this without restarting)
6. Download the exons file to local
5. In your browser go to: http://localhost:8080/api/workflows and identify
the id of the workflow you've just created. Let's assume it is
(4b187121143038ff) and that your api_key for admin is
(8c25bc83f6f9e4001dd21eb7b64f063f)
6. Run the script:
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-drop/input
/Users/Rob/Documents/galaxy-drop/output My API Import 4b187121143038ff
7. Copy the exons file to /Users/Rob/Documents/galaxy-drop/input when the
process is complete the file will be placed in
/Users/Rob/Documents/galaxy-drop/output
8. In your browser go to: http://localhost:8080/api/histories locate the
recently created history and then navigate to:
http://localhost:8080/api/histories/history_id/contents
where history_id is the id of the history that was created when you ran
the dataset on the last workflow.
This is my result. The second file is the output file.
[
{
id: 0d16186aaff7cbfd,
name: Galaxy1-[Exons_chr22].bed,
type: file,
url: /api/histories/2d9035b3fc152403/contents/0d16186aaff7cbfd
},
{
id: 81991b32733a4fc4,
name: top200,
type: file,
url: /api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
]
9. You can navigate to the JSON description of the file in your browser:
http://localhost:8080/api/histories/2d9035b3fc152403/contents/81991b32733a4fc4
and the output file can be downloaded here:
http:localhost:8080/datasets/81991b32733a4fc4/display?to_ext=data
Cheers!
Rob
On Wed, Mar 13, 2013 at 12:15 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon,
Thanks. I've moved to a full size monitor, apparently my eyes are going!
They *key error: 0* seems to be the result of a bug in the code.
print(library)
results in:
{'url': '/api/libraries/4b187121143038ff', 'id': '4b187121143038ff',
'name': 'API Import11'}
So library[0]['id'] doesn't point to anything, resulting in the key error.
I changed the code on line 30
library_id = library['id']
This seems to fix the issue. I've now got to the section of code where I
am watching the in_folder, but there doesn't seem to be any subsequent
activity. I'll post my findings or questions as I make progress or lack
thereof.
Cheers,
Rob
On Wed, Mar 13, 2013 at 7:35 AM, Dannon Baker dannon.ba...@gmail.comwrote:
It should be set on line 29 as the result of that submit function. Can
you add (between 29 and 30) a 'print library' to see what that call
actually results in?
Off the top of my head, this might be happening if your user account
(that the API key is associated with) isn't an admin account -- only admins
can create Data Libraries.
On Tue, Mar 12, 2013 at 6:35 PM, Rob Leclerc robert.lecl...@gmail.comwrote:
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to
add the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the
API Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow
)
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if
it doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries',
lib_create_data, return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
[galaxy-dev] example_watch_folder.py
If you're having problems with example_watch_folder.py you may need to increase the delay hack on line 55 to 15-20 seconds. Without the increased delay, it would upload a zerobyte file for me. It's not clear why, but there is a delay between the creation of a file name and the setting of the file to the dataset. Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
[galaxy-dev] example_watch_folder.py : URL did not return JSON data
I'm trying to get the example_watch_folder.py to work with a simple example, but I am running into an error (Caveat, python is not my native language) I call the following script, but get the subsequent error: $ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/ /Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3 * URL did not return JSON data* The preconditions I have in place are: 1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f 2. I created a new history on http://localhost:8080 3. I downloaded the the exons from hg19/chr22 from UCSC 3. On the dataset I performed a Text Manipulation Select first (20 lines) 4. I extracted a workflow from the history and recorded the workflow id (2a56795cad3c7db3). 5. I created directories: - /Users/Rob/Documents/galaxy-db/input/ - /Users/Rob/Documents/galaxy-db/output/ My args may be malformed. Can someone provide me an example of a theirs? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
The second argument there should point to the url of Galaxy's API -- in this case http://localhost:8080/api, I'm guessing. -Dannon On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com wrote: I'm trying to get the example_watch_folder.py to work with a simple example, but I am running into an error (Caveat, python is not my native language) I call the following script, but get the subsequent error: $ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/ /Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3 URL did not return JSON data The preconditions I have in place are: 1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f 2. I created a new history on http://localhost:8080 3. I downloaded the the exons from hg19/chr22 from UCSC 3. On the dataset I performed a Text Manipulation Select first (20 lines) 4. I extracted a workflow from the history and recorded the workflow id (2a56795cad3c7db3). 5. I created directories: /Users/Rob/Documents/galaxy-db/input/ /Users/Rob/Documents/galaxy-db/output/ My args may be malformed. Can someone provide me an example of a theirs? Cheers, Rob ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] example_watch_folder.py : URL did not return JSON data
Hi Dannon. Thanks! I had been experimenting with BioBlend and forgot to add
the /api/ on the url.
./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/api/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
However, I am now running into a new error when it tries to create the API
Imports folder
Traceback (most recent call last):
File ./example_watch_folder.py, line 87, in module
main(api_key, api_url, in_folder, out_folder, data_library, workflow )
File ./example_watch_folder.py, line 30, in main
library_id = library[0]['id']
KeyError: 0
(This is coming from the section that creates 'API Imports' library if it
doesn't already exist)
24: for library in libs:
25: if library['name'] == data_library:
26: library_id = library['id']
27: if not library_id:
28:lib_create_data = {'name':data_library}
29:library = submit(api_key, api_url + 'libraries',
lib_create_data, return_formatted=False)
30:library_id = library[0]['id']
As I mentioned, I'm not familiar with python, but I can't see where
'library[0]['id']' is set or is in scope, since this seems to be the
item in the iterator above this section.
Cheers,
Rob
Rob Leclerc, PhD
http://www.linkedin.com/in/robleclerc https://twitter.com/#!/robleclerc
P: (US) +1-(917)-873-3037
P: (Shanghai) +86-1-(861)-612-5469
Personal Email: rob.lecl...@aya.yale.edu
On Tue, Mar 12, 2013 at 6:14 PM, Dannon Baker dannon.ba...@gmail.comwrote:
The second argument there should point to the url of Galaxy's API --
in this case http://localhost:8080/api, I'm guessing.
-Dannon
On Tue, Mar 12, 2013 at 6:11 PM, Rob Leclerc robert.lecl...@gmail.com
wrote:
I'm trying to get the example_watch_folder.py to work with a simple
example,
but I am running into an error (Caveat, python is not my native language)
I call the following script, but get the subsequent error:
$ ./example_watch_folder.py 8c25bc83f6f9e4001dd21eb7b64f063f
http://localhost:8080/ /Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/ API Imports 2a56795cad3c7db3
URL did not return JSON data
The preconditions I have in place are:
1. I copied my api_key: 8c25bc83f6f9e4001dd21eb7b64f063f
2. I created a new history on http://localhost:8080
3. I downloaded the the exons from hg19/chr22 from UCSC
3. On the dataset I performed a Text Manipulation Select first (20
lines)
4. I extracted a workflow from the history and recorded the workflow id
(2a56795cad3c7db3).
5. I created directories:
/Users/Rob/Documents/galaxy-db/input/
/Users/Rob/Documents/galaxy-db/output/
My args may be malformed. Can someone provide me an example of a theirs?
Cheers,
Rob
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
___
Please keep all replies on the list by using reply all
in your mail client. To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
http://lists.bx.psu.edu/
