Re: concatenate function
Hi there, The problem has been solved. I 've decided to run the python script as argument of qsub instead of run qsub from inside of the script itself. I also use .wait() as suggest by colleagues above. Final code goes here: http://ompldr.org/vZDFiag Thank you very much for helping. -- View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4578408.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On Tue, Mar 13, 2012 at 1:35 PM, Robert Kern wrote:
> On 3/13/12 6:01 PM, ferreirafm wrote:
>> Robert Kern-2 wrote
>>> When you report a problem, you should copy-and-paste the output that you
>>> got and
>>> also state the output that you expected. I have no idea what you mean
>>> when
>>> you
>>> say "subprocess.Popen seems not accept to run "qsub" over a second
>>> program."
>>>
>>
>> Code goes here:
>> http://ompldr.org/vZDB5YQ
>>
>> stdout:
>> $ no_name.py --toplist top_percent.list
>> Traceback (most recent call last):
>> File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in
>> main()
>> File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main
>> comb_slt(toplist)
>> File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in
>> comb_slt
>> subprocess.Popen([cmd, options], env=qsub_env)
>> File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in
>> __init__
>> errread, errwrite)
>> File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in
>> _execute_child
>> raise child_exception
>> OSError: [Errno 13] Permission denied
>
>
> You need to use a command list like this:
>
> ['qsub', 'combine_silent.linuxgccrelease', '-database',
> '/home6/psloliveira/rosetta_database/', ...]
>
> The program to run ("qsub", not "qsub combine_silent.linuxgccrelease") and
> each individual argument must be a separate string in the list. You cannot
> combine them together with spaces. The reason you get a "Permission denied"
> error is that it tried to find an executable file named "qsub
> combine_silent.linuxgccrelease" and, obviously, could not.
See also the first "Note" box (and the description of "args" generally) under
http://docs.python.org/library/subprocess.html#popen-constructor
Cheers,
Chris
--
http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On 3/13/12 6:01 PM, ferreirafm wrote:
Robert Kern-2 wrote
When you report a problem, you should copy-and-paste the output that you
got and
also state the output that you expected. I have no idea what you mean when
you
say "subprocess.Popen seems not accept to run "qsub" over a second
program."
Code goes here:
http://ompldr.org/vZDB5YQ
stdout:
$ no_name.py --toplist top_percent.list
Traceback (most recent call last):
File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in
main()
File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main
comb_slt(toplist)
File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in comb_slt
subprocess.Popen([cmd, options], env=qsub_env)
File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in
__init__
errread, errwrite)
File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in
_execute_child
raise child_exception
OSError: [Errno 13] Permission denied
You need to use a command list like this:
['qsub', 'combine_silent.linuxgccrelease', '-database',
'/home6/psloliveira/rosetta_database/', ...]
The program to run ("qsub", not "qsub combine_silent.linuxgccrelease") and each
individual argument must be a separate string in the list. You cannot combine
them together with spaces. The reason you get a "Permission denied" error is
that it tried to find an executable file named "qsub
combine_silent.linuxgccrelease" and, obviously, could not.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
--
http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
Robert Kern-2 wrote > > When you report a problem, you should copy-and-paste the output that you > got and > also state the output that you expected. I have no idea what you mean when > you > say "subprocess.Popen seems not accept to run "qsub" over a second > program." > Code goes here: http://ompldr.org/vZDB5YQ stdout: $ no_name.py --toplist top_percent.list Traceback (most recent call last): File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 73, in main() File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 68, in main comb_slt(toplist) File "/home6/psloliveira/ferreirafm/bin/no_name.py", line 55, in comb_slt subprocess.Popen([cmd, options], env=qsub_env) File "/share/apps/python/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/share/apps/python/lib/python2.7/subprocess.py", line 1228, in _execute_child raise child_exception OSError: [Errno 13] Permission denied -- View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574967.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On 3/13/12 3:59 PM, ferreirafm wrote: Hi Robert, Thanks for you kind replay and I'm sorry for my semantic mistakes. Indeed, that's what I'm doing: qsub-ing different cshell scripts. Certainly, that's not the best approach and the only problem. It's not a problem to write out a script and have qsub run it. That's a perfectly fine thing to do. You need to read the documentation for your job queue to find out the right arguments to give to qsub to make it wait until the first job finishes before executing the second job. This is not a Python problem. You just need to find the right flags to give to qsub. Alternately, you could just make a single .qsub script running all three of your programs in a single job instead of making three separate .qsub scripts. I've unsuccessfully tried to set an os.environ and call qsub from it. However, subprocess.Popen seems not accept to run "qsub" over a second program. Do you have a over come to this issue? Code goes here: http://ompldr.org/vZDB5YQ When you report a problem, you should copy-and-paste the output that you got and also state the output that you expected. I have no idea what you mean when you say "subprocess.Popen seems not accept to run "qsub" over a second program." -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On 13/03/12 16:02, ferreirafm wrote: > Hi James, thank you for your replay. Indeed, the problem is qsub. And as > warned by Robert, I don't have functions properly, but just scripts. > > > -- > View this message in context: > http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574511.html > Sent from the Python - python-list mailing list archive at Nabble.com. It looks like you're not calling wait() on your subprocesses: you're effectively launching a bunch of processes, then not waiting for them to finish before you ask the next process to operate on the same file. If you haven't given it a good look-over already, the subprocess documentation [1] is worth taking a little time over. [1]: http://docs.python.org/library/subprocess.html#popen-objects James -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
Hi James, thank you for your replay. Indeed, the problem is qsub. And as warned by Robert, I don't have functions properly, but just scripts. -- View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574511.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
Hi Robert, Thanks for you kind replay and I'm sorry for my semantic mistakes. Indeed, that's what I'm doing: qsub-ing different cshell scripts. Certainly, that's not the best approach and the only problem. I've unsuccessfully tried to set an os.environ and call qsub from it. However, subprocess.Popen seems not accept to run "qsub" over a second program. Do you have a over come to this issue? Code goes here: http://ompldr.org/vZDB5YQ -- View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574496.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On 13/03/12 14:35, ferreirafm wrote: > Hi List, > I've coded three functions that I would like to concatenate. I mean, run > them one after another. The third function depends on the results of the > second function, which depends on the results of the first one. When I call > one function after another, python runs them at the same time causing > obvious errors messages. I've tried to call one of them from inside another > but no way. Any clues are appreciated. > Complete code goes here: > http://ompldr.org/vZDB4OQ Do you think you could provide a much shorter example to illustrate what you need? In general, when you want to run one function on the result of another, you can do something like: <<< def increment_all(l); ... return [i+1 for i in l] <<< increment_all(increment_all(range(3)) [2, 3, 4] Here we apply the function increment_all to the result of the function increment_all. If you are talking about the "results" of each function in terms of it mutating an object, and then the next function mutating the same object in a (possibly) different way, then calling the functions in order will do what you want. l = [0, 3, 5, 2] l.append(10)# [0, 3, 5, 2, 10] l.sort()# [0, 2, 3, 5, 10] l.append(3) # [0, 2, 3, 5, 10, 3] James > > > > -- > View this message in context: > http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574176.html > Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
Hi Ian,
That what I have:
> burst.py
Your job 46665 ("top_n_pdb.qsub") has been submitted
Your job 4 ("extr_pdb.qsub") has been submitted
Your job 46667 ("combine_top.qsub") has been submitted
The first job runs quite well. The second is still runing and the third
issue the following:
> more combine_top.qsub.e46667
ERROR: Cannot open PDB file "S_3MSEB_26_0032.pdb"
ERROR:: Exit from: src/core/import_pose/import_pose.cc line: 199
--
View this message in context:
http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574337.html
Sent from the Python - python-list mailing list archive at Nabble.com.
--
http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On 3/13/12 2:35 PM, ferreirafm wrote: Hi List, I've coded three functions that I would like to concatenate. I mean, run them one after another. The third function depends on the results of the second function, which depends on the results of the first one. When I call one function after another, python runs them at the same time causing obvious errors messages. I've tried to call one of them from inside another but no way. Any clues are appreciated. Complete code goes here: http://ompldr.org/vZDB4OQ Just to clarify, the Python functions are indeed running consecutively, not concurrently. Your Python functions write scripts and then use subprocess.call() to make qsub (an external program) to submit those scripts to a job queue. What you are calling a "function" in your post are these scripts. Please don't call them "functions". It's confusing. Python is not running these scripts concurrently. Your job queue is. subprocess.call() will wait until qsub returns. However, qsub just submits the script to the job queue; it does not wait until the job is completed. Most qsub-using job queues can be set up to make jobs depend on the completion of other jobs. You will need to read the documentation of your job queue to figure out how to do this. Once you figure out the right arguments to give to qsub, your Python code is already more or less correct. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: concatenate function
On Tue, Mar 13, 2012 at 8:35 AM, ferreirafm wrote: > Hi List, > I've coded three functions that I would like to concatenate. I mean, run > them one after another. The third function depends on the results of the > second function, which depends on the results of the first one. When I call > one function after another, python runs them at the same time causing > obvious errors messages. I've tried to call one of them from inside another > but no way. Any clues are appreciated. > Complete code goes here: > http://ompldr.org/vZDB4OQ They don't look to me like they would run at the same time -- subprocess.call is supposed to wait for the subprocess to finish. What error messages are you getting? -- http://mail.python.org/mailman/listinfo/python-list
concatenate function
Hi List, I've coded three functions that I would like to concatenate. I mean, run them one after another. The third function depends on the results of the second function, which depends on the results of the first one. When I call one function after another, python runs them at the same time causing obvious errors messages. I've tried to call one of them from inside another but no way. Any clues are appreciated. Complete code goes here: http://ompldr.org/vZDB4OQ -- View this message in context: http://python.6.n6.nabble.com/concatenate-function-tp4574176p4574176.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
