[Tutor] Question regarding Python loops
To whom this may concern: I am struggling on two questions: Problem 2. Suppose you are given a function leave(minute) which returns the number of students that leave an exam during its mth minute. Write a function already_left(t) that returns the number of students that left at or before minute t of the exam. Example code: # To simplify testing, you may wish to define a stand-in function for leave that # behaves predictably # Here's one possible stand-in: def leave(minute): if minute 0: return minute return 0 def already_left(t): # Your code goes here Example usage: print already_left(1) print already_left(4) Example output: 1 10 So far for this i have: def leave(minute): if minute 0: return minute elif minute=0: return 0 def already_left(i): for i in range(0,t): leave(i) print already_left(8) print already_left(4) Problem 3. Using your already_left function, write a function percent_left(t, l) that returns the percent of the class that has left at or before minute t, assuming the exam is l minutes long, and the entire class has left at or before minute l. Return your result as a float between 0 and 1. Example code: def percent_left(t, l): # Your code goes here Example usage: print percent_left(0, 4) print percent_left(1, 4) print percent_left(4, 4) print percent_left(30, 50) Example output: 0.0 0.1 1.0 0.364705882353___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] When to use classes
Thank you Alan, Dave and Cameron (and folks managing this email group)! Your replies were very helpful. Regards ni On Wed, Apr 9, 2014 at 4:25 AM, Cameron Simpson c...@zip.com.au wrote: On 08Apr2014 22:58, Ni hung niih...@gmail.com wrote: I am learning programming using python. I think of solving a problem using functions and for this reason all/most of my code consists of functions and no classes. I have some understanding of classes/Object Oriented Programming. I can write simple classes but I do not understand when to use classes. Loosely speaking, you can usefully make a class when there is a particular type of object on which you are make an assortedment of operations. For example, if you have several functions looking like this: def modify_thing1(obj1, ...): def add_something(obj1, ...): and so on, where obj1 is always the same kind of thing, representing some well define idea you have when writing the code. In that circumstance you might make a class looking like this: class ObjName(object): def __init__(self): # self here is an obj1 as used above # set whatever attributes an obj1 should have to start with # you can pass in some of those values as parameters to the __init__ function def modify(self, ...): # this is the same as modify_thing1(obj1, ...) earlier # using self as obj1 def add(self, ...): # this is the same as add_something(obj1, ...) earlier Then your code looks like: obj1 = ObjName() ... obj1.modify(5) obj1.add(6) or what have you. This has several advantages: - all the code that affects this kind of object is in one place - you can remove all the specifics of how things are done from the main code and keep them inside the class methods. This makes the main code more readable (if you pick good method names) and shorter (also more readable, usually). - later, if need be, you can change the inner workings of how ObjNames work without changing the main code, or at least not much - sometimes not at all Anyway, that's an ok rule of thumb for when to make a class: when you have a bunch of operations to do to one type of thing. Cheers, -- Cameron Simpson c...@zip.com.au ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Desktop Notifications on Linux
Hi all, I am trying to create a script that will go through the /var/log/secure file on a Linux system and provide desktop notifications for failed login attempts. Here is the code - http://pastebin.com/MXP8Yd91 And here's notification.py - http://pastebin.com/BhsSTP6H I am facing issue in the function new_attempts_from_last. I am not able to raise a desktop notification from this function. It always fails with this traceback - http://pastebin.com/cgHMScv3 I see this error only when I try to raise a notification from the aforementioned function. If I run test examples under /usr/share/doc/notify-python/examples, it works just fine. Also, if I try to raise a notification from under if __name__ == __main__:, it works without any issues. So I don't think there's any issue with OS's notification daemon. Running from python shell like below also works fine: In [1]: import notification In [2]: notification.notification(Hey) What am I missing or doing wrong here? If it matters, I am running Fedora 20, python 2.7 and Cinnamon desktop environment. For readability purposes, I have provided pastebin links. Let me know if this is not the correct way. Regards, Dharmit. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org Sent: Thu, 10 Apr 2014 2:00 Subject: Re: [Tutor] masking library files uga...@talktalk.net Wrote in message: A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. -- DaveA Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question regarding Python loops
Conner Crowe wrote: To whom this may concern: I am struggling on two questions: Problem 2. Suppose you are given a function leave(minute) which returns the number of students that leave an exam during its mth minute. Write a function already_left(t) that returns the number of students that left at or before minute t of the exam. Example code: # To simplify testing, you may wish to define a stand-in function for # leave that behaves predictably # Here's one possible stand-in: def leave(minute): if minute 0: return minute return 0 def already_left(t): # Your code goes here Example usage: print already_left(1) print already_left(4) Example output: 1 10 So far for this i have: def leave(minute): if minute 0: return minute elif minute=0: return 0 def already_left(i): for i in range(0,t): leave(i) Hint: You are throwing away the result of the leave() call. But what you want is to add up the results of all calls to leave() in the loop. Once you have that total you need already_left() to return it so that print already_left(8) print already_left(4) will print it instead of None (the default when you do not explicitly return anything from a function). Problem 3. How would you solve that with pen and paper? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Desktop Notifications on Linux
Dharmit Shah wrote: I am trying to create a script that will go through the /var/log/secure file on a Linux system and provide desktop notifications for failed login attempts. Here is the code - http://pastebin.com/MXP8Yd91 And here's notification.py - http://pastebin.com/BhsSTP6H I am facing issue in the function new_attempts_from_last. I am not able to raise a desktop notification from this function. It always fails with this traceback - http://pastebin.com/cgHMScv3 I see this error only when I try to raise a notification from the aforementioned function. If I run test examples under /usr/share/doc/notify-python/examples, it works just fine. Also, if I try to raise a notification from under if __name__ == __main__:, it works without any issues. So I don't think there's any issue with OS's notification daemon. Running from python shell like below also works fine: In [1]: import notification In [2]: notification.notification(Hey) What am I missing or doing wrong here? If it matters, I am running Fedora 20, python 2.7 and Cinnamon desktop environment. For readability purposes, I have provided pastebin links. Let me know if this is not the correct way. Maybe you are running the code as a user that has no desktop? Here's a strapped-down demo: $ cat notify_demo.py import pynotify import sys pynotify.init(Notification) n = pynotify.Notification(Notification, .join(sys.argv[1:])) n.show() $ python notify_demo.py hello world The notification appeared on my desktop. However, when trying again as root I got the error you are seeing: $ sudo su # python notify_demo.py hello again Traceback (most recent call last): File notify_demo.py, line 6, in module n.show() glib.GError: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Notifications was not provided by any .service files ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
-Original Message- From: Mark Lawrence breamore...@yahoo.co.uk To: tutor@python.org Sent: Thu, 10 Apr 2014 9:36 Subject: Re: [Tutor] masking library files On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org Sent: Thu, 10 Apr 2014 2:00 Subject: Re: [Tutor] masking library files uga...@talktalk.net Wrote in message: A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. -- DaveA Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor Like this? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
-Original Message- From: uga...@talktalk.net To: tutor@python.org Sent: Thu, 10 Apr 2014 10:17 Subject: Re: [Tutor] masking library files -Original Message- From: Mark Lawrence breamore...@yahoo.co.uk To: tutor@python.org Sent: Thu, 10 Apr 2014 9:36 Subject: Re: [Tutor] masking library files On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org Sent: Thu, 10 Apr 2014 2:00 Subject: Re: [Tutor] masking library files uga...@talktalk.net Wrote in message: A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. -- DaveA Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor Like this? ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor Sorry, but posting this way does not make sense. The header appears on top, the message at the foot, buried in multiple footers. It is a mess. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
On 10/04/14 10:30, uga...@talktalk.net wrote: On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. Mark Lawrence Like this? Yes, but delete all the junk such as extraneous headers/footers as I did above. It does actually make the conversation much easier to follow than top posting. I appreciate many modern mail tools do top posting as default (thanks Microsoft! :-( ) but for conversations its much easier to use inline comments. For a good, fairly short, balanced, explanation of how it helps, see this: http://en.wikipedia.org/wiki/Posting_style It helps if you have quoting turned on in your mail tool too - yours doesn't seem to be using quotes at present (Quoting means the bits in the message above - the number of characters tells you how far back in the conversation it was) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Question regarding Python loops
On 10/04/14 03:04, Conner Crowe wrote: *Problem 2.* # Here's one possible stand-in: def leave(minute): if minute 0: return minute return 0 So far for this i have: *def* *leave*(minute): *if* minute 0: *return* minute *elif* minute=0: *return* 0 Ignoring the messsed yup formatting, presumably due to HTML mail... You have introduced a bug into the example you were given. The elif line should read elif minute == 0: # use double = sign But as the example shows you don't need the explicit check the function returns zero even for negative numbers. Your version returns None for negatives which will likely cause your later code to break. *def**already_left*(i): *for* i *in* range(0,t) Where is t defined? leave(i) You calculate the result but then ignore it... You need to return a value from your function. *Problem 3.* Lets do one problem at a time. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Fwd: masking library files
But I must remember to cc tutor@python.org when replying to you. -Original Message- From: uga...@talktalk.net To: alan.ga...@btinternet.com Sent: Thu, 10 Apr 2014 13:15 Subject: Re: [Tutor] masking library files It is off, and I don't appear have an option to turn it on in plain text. I have this option, but but only in rich text/html, and as you see it top posts, which I quite like. Yes, but delete all the junk such as extraneous headers/footers as I did above. It does actually make the conversation much easier to follow than top posting. I appreciate many modern mail tools do top posting as default (thanks Microsoft! :-( ) but for conversations its much easier to use inline comments. For a good, fairly short, balanced, explanation of how it helps, see this: http://en.wikipedia.org/wiki/Posting_style It helps if you have quoting turned on in your mail tool too - yours doesn't seem to be using quotes at present (Quoting means the bits in the message above - the number of characters tells you how far back in the conversation it was) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos -Original Message- From: Alan Gauld alan.ga...@btinternet.com To: tutor@python.org Sent: Thu, 10 Apr 2014 12:23 Subject: Re: [Tutor] masking library files On 10/04/14 10:30, uga...@talktalk.net wrote: On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. Mark Lawrence Like this? Yes, but delete all the junk such as extraneous headers/footers as I did above. It does actually make the conversation much easier to follow than top posting. I appreciate many modern mail tools do top posting as default (thanks Microsoft! :-( ) but for conversations its much easier to use inline comments. For a good, fairly short, balanced, explanation of how it helps, see this: http://en.wikipedia.org/wiki/Posting_style It helps if you have quoting turned on in your mail tool too - yours doesn't seem to be using quotes at present (Quoting means the bits in the message above - the number of characters tells you how far back in the conversation it was) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Desktop Notifications on Linux
Hi Peter, On Thu, Apr 10, 2014 at 2:30 PM, Peter Otten __pete...@web.de wrote: Dharmit Shah wrote: I am trying to create a script that will go through the /var/log/secure file on a Linux system and provide desktop notifications for failed login attempts. Here is the code - http://pastebin.com/MXP8Yd91 And here's notification.py - http://pastebin.com/BhsSTP6H I am facing issue in the function new_attempts_from_last. I am not able to raise a desktop notification from this function. It always fails with this traceback - http://pastebin.com/cgHMScv3 I see this error only when I try to raise a notification from the aforementioned function. If I run test examples under /usr/share/doc/notify-python/examples, it works just fine. Also, if I try to raise a notification from under if __name__ == __main__:, it works without any issues. So I don't think there's any issue with OS's notification daemon. Running from python shell like below also works fine: In [1]: import notification In [2]: notification.notification(Hey) What am I missing or doing wrong here? If it matters, I am running Fedora 20, python 2.7 and Cinnamon desktop environment. For readability purposes, I have provided pastebin links. Let me know if this is not the correct way. Thanks for your prompt response. Maybe you are running the code as a user that has no desktop? Here's a strapped-down demo: $ cat notify_demo.py import pynotify import sys pynotify.init(Notification) n = pynotify.Notification(Notification, .join(sys.argv[1:])) n.show() $ python notify_demo.py hello world The notification appeared on my desktop. However, when trying again as root I got the error you are seeing: $ sudo su # python notify_demo.py hello again Traceback (most recent call last): File notify_demo.py, line 6, in module n.show() glib.GError: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Notifications was not provided by any .service files That does ring some bells. I am logged into my F20 system as non-root user but since reading /var/log/secure file requires superuser privileges, I am running it as sudo: sudo python secure.py That probably explains the issue I am facing. I will add the user to the root group and see if it helps. Regards, Dharmit. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] improving speed using and recalling C functions
Hi, I have a program that is reading near 60 elements from a file. For each element it performs 200 times a particular mathematical operation (a numerical interpolation of a function). Now these process takes near 8 hours. Creating a C function and calling it from the code could improve the speed? It could be useful that it contains both the mathematical operation and the for loop or only the mathematical operation? If it is a reasonable solution to try decreasing the computational time how can I implement this C function? Thanks Gabriele ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi there Gabriele, : I have a program that is reading near 60 elements from a : file. For each element it performs 200 times a particular : mathematical operation (a numerical interpolation of a function). : Now these process takes near 8 hours. Sounds fun! Here are some thoughts (though I may not have any solid answers, I hope these pointers are useful): Are you sure (from profiling) that the numerical interpolation is the bottleneck? What is the numerical operation? 1: Is the function implemented in Numpy? [0] 2: How about SciPy? [1] Is there a domain-specific Python library that already does the computation you want to achieve? If so, then you have only the format question--how to put your data into the format required by the library. If not, then on to the next question you ask ... : Creating a C function and calling it from the code could improve : the speed? It could be useful that it contains both the : mathematical operation and the for loop or only the mathematical : operation? : : If it is a reasonable solution to try decreasing the : computational time how can I implement this C function? It is certainly an option! In the event that you cannot find something that is fast enough in Python already and you cannot find any C library that has Python bindings, then, whether you have to write your own, or you have a third party C library, you have several options for writing bindings. You can write your Python bindings using: * ctypes: https://docs.python.org/2/library/ctypes.html * cffi: http://cffi.readthedocs.org/en/release-0.8/ * cython: http://www.cython.org/ * roll your own C! It might be beyond the scope of this list (certainly beyond my capabilities) to provide recommendations on which avenue to take--but if you are asking, you may already have the tools you need to assess for yourself. There is, however, quite a bit of experience loitering around this list, so perhaps somebody else will have a suggestion. -Martin [0] http://www.numpy.org/ [1] http://www.scipy.org/ -- Martin A. Brown http://linux-ip.net/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On 10/04/14 16:58, Gabriele Brambilla wrote: For each element it performs 200 times a particular mathematical operation (a numerical interpolation of a function). Now these process takes near 8 hours. The first thing to do in such cases is check that the time is going where you think it is. Run the cProfile tool on the code (with a smaller data set!) to see where the time is being spent. Creating a C function and calling it from the code could improve the speed? Yes, it could. But it may not be necessary. There are various add on tools that can drastically improve Python efficiency, especially for math type functions so you might want to check that nobody has already written what you need. If it is a reasonable solution to try decreasing the computational time how can I implement this C function? If its already written in Python the easiest way might be to use Cython to convert it into C. I'm no expert in Cython however, but others on this list can help. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi, 2014-04-10 13:05 GMT-04:00 Martin A. Brown mar...@linux-ip.net: Hi there Gabriele, : I have a program that is reading near 60 elements from a : file. For each element it performs 200 times a particular : mathematical operation (a numerical interpolation of a function). : Now these process takes near 8 hours. Sounds fun! Here are some thoughts (though I may not have any solid answers, I hope these pointers are useful): Are you sure (from profiling) that the numerical interpolation is the bottleneck? I see that the time needed by the code increase when I increase the number of these operation. It's the only thing I do in the code. (I'm sorry but I don't know very well what profiling is) What is the numerical operation? 1: Is the function implemented in Numpy? [0] 2: How about SciPy? [1] Is interp1d from scipy.interpolate. I think is the best one Is there a domain-specific Python library that already does the computation you want to achieve? If so, then you have only the format question--how to put your data into the format required by the library. If not, then on to the next question you ask ... No, interp1d do exactly what I want. : Creating a C function and calling it from the code could improve : the speed? It could be useful that it contains both the : mathematical operation and the for loop or only the mathematical : operation? : : If it is a reasonable solution to try decreasing the : computational time how can I implement this C function? It is certainly an option! In the event that you cannot find something that is fast enough in Python already and you cannot find any C library that has Python bindings, then, whether you have to write your own, or you have a third party C library, you have several options for writing bindings. You can write your Python bindings using: * ctypes: https://docs.python.org/2/library/ctypes.html * cffi: http://cffi.readthedocs.org/en/release-0.8/ * cython: http://www.cython.org/ * roll your own C! Do you know if does a C library exist with an already implemented function like interp1d? I use Anaconda, is there an already implemented mode to do it in Anaconda? Cython? It might be beyond the scope of this list (certainly beyond my capabilities) to provide recommendations on which avenue to take--but if you are asking, you may already have the tools you need to assess for yourself. There is, however, quite a bit of experience loitering around this list, so perhaps somebody else will have a suggestion. -Martin [0] http://www.numpy.org/ [1] http://www.scipy.org/ -- Martin A. Brown http://linux-ip.net/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
I do have a Reply All command, but as in this case, there is no recipient other than myself to be included. Also, hitting the Reply All command seems to generate an alert warning message, when recipients include a mail-list address, and some mail list replies go to the mail-list as recipient rather than the sender by default. I don't know which mail tool you are using but you should have a ReplyAll command that does that automatically. -Original Message- From: Alan Gauld alan.ga...@yahoo.co.uk To: uga...@talktalk.net Sent: Thu, 10 Apr 2014 16:41 Subject: Re: Fwd: masking library files On 10/04/14 13:17, uga...@talktalk.net wrote: But I must remember to cc tutor@python.org when replying to you. I don't know which mail tool you are using but you should have a ReplyAll command that does that automatically. If you are using a Desktop mail tool it may even have a mailing list mode that detects lists (or allows you to specify them) and auto uses replyall... Alan g. -Original Message- From: uga...@talktalk.net To: alan.ga...@btinternet.com Sent: Thu, 10 Apr 2014 13:15 Subject: Re: [Tutor] masking library files It is off, and I don't appear have an option to turn it on in plain text. I have this option, but but only in rich text/html, and as you see it top posts, which I quite like. Yes, but delete all the junk such as extraneous headers/footers as I did above. It does actually make the conversation much easier to follow than top posting. I appreciate many modern mail tools do top posting as default (thanks Microsoft! :-( ) but for conversations its much easier to use inline comments. For a good, fairly short, balanced, explanation of how it helps, see this: http://en.wikipedia.org/wiki/Posting_style It helps if you have quoting turned on in your mail tool too - yours doesn't seem to be using quotes at present (Quoting means the bits in the message above - the number of characters tells you how far back in the conversation it was) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos -Original Message- From: Alan Gauld alan.ga...@btinternet.com mailto:alan.ga...@btinternet.com To: tutor@python.org mailto:tutor@python.org Sent: Thu, 10 Apr 2014 12:23 Subject: Re: [Tutor] masking library files On 10/04/14 10:30, uga...@talktalk.net wrote: On 10/04/2014 02:20, uga...@talktalk.net wrote: Please write in plain English if you want to be understood. -Original Message- From: Dave Angel da...@davea.name To: tutor@python.org A message left in invisible ink. Please post in text form, not html, as html offers too many ways to mess up the message. Please don't top post on this list. And don't use language like that if you want answers, especially when replying to a highly respected member of this community such as Dave. Mark Lawrence Like this? Yes, but delete all the junk such as extraneous headers/footers as I did above. It does actually make the conversation much easier to follow than top posting. I appreciate many modern mail tools do top posting as default (thanks Microsoft! :-( ) but for conversations its much easier to use inline comments. For a good, fairly short, balanced, explanation of how it helps, see this: http://en.wikipedia.org/wiki/Posting_style It helps if you have quoting turned on in your mail tool too - yours doesn't seem to be using quotes at present (Quoting means the bits in the message above - the number of characters tells you how far back in the conversation it was) -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor -- Alan G Author of the Learn To Program web site http://www.alan-g.me.uk http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
We should get back to the topic. Did you have a Python question? Meta-discussion on mailing list etiquette must not dominate discussion on helping people learning to program with Python. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
On Wed, Apr 9, 2014 at 7:17 AM, uga...@talktalk.net wrote: Is it common for files saved to a working directory to 'mask' library files located in the Python framework? Hi ugajin, To come back to your original question: yes, unfortunately this happens. I think it's a flaw in the language. There are steps to mitigate the problem, some by convention, some by using newer features in Python. Certain style guidelines dictate that module import never use relative paths, but rather use one rooted at the toplevel package directory. As a concrete example, see Google Python style guidelines on how to do imports: http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Packages This is odd at first, but it has the intent of avoiding module collision. The basic idea is to have all your code packaged, and always use the unique package as a namespace to avoid collision with the standard library. So if you have modules m1, m2 in package p, and m1 wants to import m2, then we say: import p.m2 Other mitigation strategies include using explicit-relative imports: https://docs.python.org/2/tutorial/modules.html#intra-package-references but I think the convention approach is a little easier to handle. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On 10/04/2014 18:29, Gabriele Brambilla wrote: (I'm sorry but I don't know very well what profiling is) Take a look at these for some tips http://www.huyng.com/posts/python-performance-analysis/ and https://wiki.python.org/moin/PythonSpeed/PerformanceTips -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
On 10/04/14 19:09, uga...@talktalk.net wrote: I do have a Reply All command, but as in this case, there is no recipient other than myself to be included. That's because I (deliberately) replied only to you. :-) commandseems to generate an alert warning message, when recipients include a mail-list address, and some mail list replies go to the mail-list as recipient rather than the sender by default. Which tool are you using? Others on the list may be using the same tool and can help with settings... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
Apologies for the multiple emails. I'm still paging in from memory how Python imports are working these days. :P If you have the freedom to do so, you may also turn on the absolute import system: https://docs.python.org/2/whatsnew/2.5.html#pep-328-absolute-and-relative-imports so that you can avoid the issue altogether, at least within your own code. This is not the default behavior in Python 2, so you've got to opt-into it. I believe it is the default in Python 3, so it should not be an issue in Python 3 code. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Gabriele, Have you profiled your program? Please look at: https://docs.python.org/2/library/profile.html If you can, avoid guessing what is causing performance to drop. Rather, use the tools in the profiling libraries to perform measurements. It may be that your program is taking a long time because of something obvious, but perhaps there is some other factor that's contributing. Please do this. More details would be helpful. Are you using any libraries such as Numpy? Just writing something in C doesn't magically make it go faster. CPython is written in C, for example, and yet people do not say that Python itself is very fast. :P It may be the case that writing the computations in C will allow you to specify enough type information so that the computer can effectively run your computations quickly. But if you're using libraries like Numpy to do vector parallel operations, I would not be surprised if that would outperform native non-parallel C code. See: http://technicaldiscovery.blogspot.com/2011/06/speeding-up-python-numpy-cython-and.html which demonstrates that effective use of NumPy may speed up computations by a significant order of magnitude, if you use the library to take advantage of its vectorizing compuations. More domain-specific details may help folks on the tutor list to give good advice here. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] difference between expressions and statements
Caveat: I began this before there were any other responses. So this may be overkill - but I ike to be thorough. On 4/9/2014 12:49 PM, Jared Nielsen wrote: Hi Pythons, Could someone explain the difference between expressions and statements? I know that expressions are statements that produce a value. No. Expressions are not statements. These are mutually exclusive. Expressions do produce values. An attempt at a thorough answer: In the language reference glossary under expression you will find: A piece of syntax which can be evaluated to some value. In other words, an expression is an accumulation of expression elements like literals, names, attribute access, operators or function calls which all return a value There are also statements which cannot be used as expressions, such as if. Assignments are also statements, not expressions. Tthe above is a quote; I don't like some of the grammar. In your examples print is a function. So all calls to print are expressions. In the language reference you will also find: 7. Simple statements 7.1. Expression statements 7.2. Assignment statements 7.3. The assert statement 7.4. The pass statement 7.5. The del statement 7.6. The return statement 7.7. The yield statement 7.8. The raise statement 7.9. The break statement 7.10. The continue statement 7.11. The import statement 7.12. The global statement 7.13. The nonlocal statement 8. Compound statements 8.1. The if statement 8.2. The while statement 8.3. The for statement 8.4. The try statement 8.5. The with statement 8.6. Function definitions 8.7. Class definitions With the exception of - 7.1. Expression statements - all of the above are either start with a keyword except 7.2 assignment which is indicated by an equal sign (=) . - all of the above cause something to happen (except pass), and do not return a value. 7.1. Expression statement is either one expression or several separated by commas. Used interactively to display value(s). Used anywhere to make a function call. I'm unclear on functions and especially strings. Are any of the following expressions? print(42) print(spam) spam = 42 print(spam) Is the first example producing a value or simply displaying an integer? All function calls return a value. In the case of print the return value is always None. spam = 42 is a statement. (indicated by the = sign. 42 is a value. Does a string count as a value? Yes - however I suspect you are limiting string to something within quotes. Those are string literals. Is a variable assignment considered a value? No If I print a variable is that considered production of a value? See above comment on print. Long but comprehensive answer. Feel free to ask questions. Note there are various subtleties here -some keywords may be used to start a statement or in an expression - e.g. if, else, for yield. This also raises the fact that else (inter ala) is neither an expression or a statement; rather it is part of a compound statement. Nothing is simple. Oh there is more but I may never hit send ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
I'm trying to profile it adding this code:
import cProfile
import re
import pstats
cProfile.run('re.compile(foo|bar)', 'restats')
p = pstats.Stats('restats')
p.strip_dirs().sort_stats('name')
p.sort_stats('time').print_stats(10)
but where I have to add this in my code?
because I obtain
Thu Apr 10 15:23:17 2014restats
194 function calls (189 primitive calls) in 0.001 seconds
Ordered by: internal time
List reduced from 34 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
3/10.0000.0000.0000.000 sre_compile.py:33(_compile)
10.0000.0000.0000.000
sre_compile.py:208(_optimize_chars
et)
3/10.0000.0000.0000.000 sre_parse.py:141(getwidth)
10.0000.0000.0000.000
sre_compile.py:362(_compile_info)
20.0000.0000.0000.000 sre_parse.py:380(_parse)
10.0000.0000.0000.000 sre_parse.py:302(_parse_sub)
10.0000.0000.0010.001 re.py:226(_compile)
100.0000.0000.0000.000 sre_parse.py:183(__next)
10.0000.0000.0010.001 sre_compile.py:496(compile)
150.0000.0000.0000.000 {isinstance}
but my program take more than 0.001 seconds!
I think it's not working as I want.
Gabriele
2014-04-10 15:09 GMT-04:00 Danny Yoo d...@hashcollision.org:
Hi Gabriele,
Have you profiled your program? Please look at:
https://docs.python.org/2/library/profile.html
If you can, avoid guessing what is causing performance to drop.
Rather, use the tools in the profiling libraries to perform
measurements.
It may be that your program is taking a long time because of something
obvious, but perhaps there is some other factor that's contributing.
Please do this. More details would be helpful. Are you using any
libraries such as Numpy? Just writing something in C doesn't
magically make it go faster. CPython is written in C, for example,
and yet people do not say that Python itself is very fast. :P
It may be the case that writing the computations in C will allow you
to specify enough type information so that the computer can
effectively run your computations quickly. But if you're using
libraries like Numpy to do vector parallel operations, I would not be
surprised if that would outperform native non-parallel C code.
See:
http://technicaldiscovery.blogspot.com/2011/06/speeding-up-python-numpy-cython-and.html
which demonstrates that effective use of NumPy may speed up
computations by a significant order of magnitude, if you use the
library to take advantage of its vectorizing compuations.
More domain-specific details may help folks on the tutor list to give
good advice here.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Gabriele,
I should probably have pointed you to:
https://docs.python.org/2/library/profile.html#instant-user-s-manual
instead.
Here is an example that uses the cProfile module. Let's say that I'm
trying to pinpoint where something is going slow in some_program():
#
import cProfile
def slow_string_mul(w, n):
s =
for x in xrange(n):
s = slow_append(s, w)
return s
def slow_append(s, w):
return s + w
def fast_string_mul(w, n):
return w * n
def some_program(w, n):
print slow_string_mul(w, n) == fast_string_mul(w, n)
## Try running the operation, and let cProfile report stats.
cProfile.run('some_program(testing, 5)', None, 'time')
#
We tell cProfile.run to execute some_program(), sorting its
measurements by time. You might save the collected statistics to a
file, but here I have not, and have cProfile.run() just report the
results after the program finishes.
Here's what comes back from cProfile's report:
#
$ python test_profile.py
True
50005 function calls in 1.422 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
51.2250.0001.2250.000 test_profile.py:9(slow_append)
10.1970.1971.4221.422 test_profile.py:3(slow_string_mul)
10.0000.0001.4221.422 test_profile.py:15(some_program)
10.0000.0000.0000.000
test_profile.py:12(fast_string_mul)
10.0000.0001.4221.422 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
#
This is telling me that slow_append is being called 5 times, which
in hindsight sounds right. It's taking the majority of the time of
this program, which is intuitively true, as I wrote it in a way to do
a very naive string-appending operation, which gets more expensive the
longer the string grows.
The point of examples is to show simple uses of the libraries. But
you will have to adapt the examples to work for your context.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] masking library files
uga...@talktalk.net Wrote in message: Please write in plain English if you want to be understood Like the op, you post in html. There are a bunch of things that can wrong when you do that, so you should use text mail instead. In the case of the op, he apparently changed the text color to the same as the background. I called it invisible ink, because I couldn't read a word. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi,
I get this result:
Thu Apr 10 17:35:53 2014restats
21071736 function calls in 199.883 seconds
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
330445.4700.0006.4440.000
interpolate.py:394(_call_linear)
232.2720.000 21.2790.000 instruments.py:10(kappa)
2313282.1200.0002.1200.000 {numpy.core.multiarray.array}
330441.7190.0003.8360.000
interpolate.py:454(_check_bounds)
660881.6110.0001.6110.000 {method 'reduce' of
'numpy.ufunc'
objects}
330441.1460.000 11.6230.000 interpolate.py:443(_evaluate)
330441.1200.0005.5420.000 interpolate.py:330(__init__)
330440.6590.0002.3290.000 polyint.py:82(_set_yi)
the major time is required by mymain that is the whole program.
the write on file is an operation that I do in the end but if I increase
the number of data it doesn't increase (I tried doubing the sample of
values and I know why it's behaving in this way)
the third are interpolate and kappa: the two functions (one called inside
the other one)
So they are the ones that are taking time.
Now what can I do?
thanks
Gabriele
2014-04-10 17:14 GMT-04:00 Danny Yoo d...@hashcollision.org:
Hi Gabriele,
I should probably have pointed you to:
https://docs.python.org/2/library/profile.html#instant-user-s-manual
instead.
Here is an example that uses the cProfile module. Let's say that I'm
trying to pinpoint where something is going slow in some_program():
#
import cProfile
def slow_string_mul(w, n):
s =
for x in xrange(n):
s = slow_append(s, w)
return s
def slow_append(s, w):
return s + w
def fast_string_mul(w, n):
return w * n
def some_program(w, n):
print slow_string_mul(w, n) == fast_string_mul(w, n)
## Try running the operation, and let cProfile report stats.
cProfile.run('some_program(testing, 5)', None, 'time')
#
We tell cProfile.run to execute some_program(), sorting its
measurements by time. You might save the collected statistics to a
file, but here I have not, and have cProfile.run() just report the
results after the program finishes.
Here's what comes back from cProfile's report:
#
$ python test_profile.py
True
50005 function calls in 1.422 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
51.2250.0001.2250.000
test_profile.py:9(slow_append)
10.1970.1971.4221.422
test_profile.py:3(slow_string_mul)
10.0000.0001.4221.422
test_profile.py:15(some_program)
10.0000.0000.0000.000
test_profile.py:12(fast_string_mul)
10.0000.0001.4221.422 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
#
This is telling me that slow_append is being called 5 times, which
in hindsight sounds right. It's taking the majority of the time of
this program, which is intuitively true, as I wrote it in a way to do
a very naive string-appending operation, which gets more expensive the
longer the string grows.
The point of examples is to show simple uses of the libraries. But
you will have to adapt the examples to work for your context.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] difference between expressions and statements
Thanks for the thorough answer, Bob. I now understand the difference. On Apr 10, 2014 2:11 PM, bob gailer bgai...@gmail.com wrote: Caveat: I began this before there were any other responses. So this may be overkill - but I ike to be thorough. On 4/9/2014 12:49 PM, Jared Nielsen wrote: Hi Pythons, Could someone explain the difference between expressions and statements? I know that expressions are statements that produce a value. No. Expressions are not statements. These are mutually exclusive. Expressions do produce values. An attempt at a thorough answer: In the language reference glossary under expression you will find: A piece of syntax which can be evaluated to some value. In other words, an expression is an accumulation of expression elements like literals, names, attribute access, operators or function calls which all return a value There are also statements which cannot be used as expressions, such as if. Assignments are also statements, not expressions. Tthe above is a quote; I don't like some of the grammar. In your examples print is a function. So all calls to print are expressions. In the language reference you will also find: 7. Simple statements 7.1. Expression statements 7.2. Assignment statements 7.3. The assert statement 7.4. The pass statement 7.5. The del statement 7.6. The return statement 7.7. The yield statement 7.8. The raise statement 7.9. The break statement 7.10. The continue statement 7.11. The import statement 7.12. The global statement 7.13. The nonlocal statement 8. Compound statements 8.1. The if statement 8.2. The while statement 8.3. The for statement 8.4. The try statement 8.5. The with statement 8.6. Function definitions 8.7. Class definitions With the exception of - 7.1. Expression statements - all of the above are either start with a keyword except 7.2 assignment which is indicated by an equal sign (=) . - all of the above cause something to happen (except pass), and do not return a value. 7.1. Expression statement is either one expression or several separated by commas. Used interactively to display value(s). Used anywhere to make a function call. I'm unclear on functions and especially strings. Are any of the following expressions? print(42) print(spam) spam = 42 print(spam) Is the first example producing a value or simply displaying an integer? All function calls return a value. In the case of print the return value is always None. spam = 42 is a statement. (indicated by the = sign. 42 is a value. Does a string count as a value? Yes - however I suspect you are limiting string to something within quotes. Those are string literals. Is a variable assignment considered a value? No If I print a variable is that considered production of a value? See above comment on print. Long but comprehensive answer. Feel free to ask questions. Note there are various subtleties here -some keywords may be used to start a statement or in an expression - e.g. if, else, for yield. This also raises the fact that else (inter ala) is neither an expression or a statement; rather it is part of a compound statement. Nothing is simple. Oh there is more but I may never hit send ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Refining Code
My task is :A food vending machine accepts 10p, 20p, 50p and £1 coins. One or more coins are inserted and the current credit is calculated and displayed. A product is selected from those available. The system checks to see if there is enough credit to purchase the product chosen. If there is not enough credit the system displays an error message. If there is enough credit it dispenses the product, updates the credit available and displays the remaining credit. Further selections can be made if there is enough credit. The vending machine simulation should have five products and prices. Design, code, test and evaluate a program for this simulation. I have designed the following code, but would like to know how to make it more efficient without making it too complex as I am a beginner or is this fine? Also, how do I add a loop to this so that once one product has been dispensed the program asks the user if they would like to continue and purchase another product? Code: print Welcome to Snack Attack snack1 = 0.40snack2 = 0.75snack3 = 1.20snack4 = 0.99snack5 = 0.50insert = 0 change = 0currentCredit = 0.00A = 0.10B = 0.20C = 0.50D = 1.00a = 0.10b = 0.20c = 0.50d = 1.00 print Menuprint Snack 1: Snickers - £0.40print Snack 2: Doritos - £0.75 print Snack 3: J20 - £1.20print Snack 4: Oreos - £0.99print Snack 5: MM's - £0.50 print Exit?- how do I make this a Boolean expression, so the user can respond with either yes or no? choice = input(Select your snack: ) if choice==1: printprint You have selected Snickers, which cost £0.40 print Please insert £0.40 while currentCredit snack1: print Please select which of these coins to insert; A:10p,B:20p,C:50p and D:£1 insert_coins = input(Insert coins: ) currentCredit = insert_coins + currentCredit print Your current credit is £,currentCredit else: change_given=currentCredit-snack1 printprint Your change is £,change_given print Your Snickers have been dispensed...Enjoy! elif choice==2: print You have selected Doritos, which cost £0.75 print Please insert £0.75 while currentCreditsnack2: print Please select which of these coins to insert; A:10p,B:20p,C:50p and D:£1 insert_coins = input(Enter coins: ) currentCredit = insert_coins + currentCredit print Your current credit is £,currentCredit else: change_given=currentCredit-snack2 printprint Your change is £,change_given print Your Doritos have been dispensed...Enjoy! elif choice==3: print You have selected J20, which costs £1.20 print Please insert £1.20 while currentCreditsnack3: print Please select which of these coins to insert; A:10p,B:20p,C:50p and D:£1 insert_coins = input(Enter coins: ) currentCredit = insert_coins + currentCredit print Your current credit is £,currentCredit else: change_given=currentCredit-snack3 printprint Your change is £,change_given print Your J2O has been dispensed...Enjoy! elif choice==4: print You have selcetd Oreos, which cost £0.99 print Please insert £0.99 while currentCreditsnack4: print Please select which of these coins to insert; A:10p,B:20p,C:50p and D:£1 insert_coins = input(Enter coins: ) currentCredit = insert_coins + currentCredit print Your current credit is £,currentCredit else: change_given=currentCredit-snack4 printprint Your change is £,change_given print Your Oreos have been dispensed...Enjoy! elif choice==5: print You have selected MM's, which cost £0.50 print Please insert £0.50 while currentCreditsnack5: print Please select which of these coins to insert; A:10p,B:20p,C:50p and D:£1 insert_coins = input(Enter coins: ) currentCredit = insert_coins + currentCredit print Your current credit is £,currentCredit else: change_given=currentCredit-snack5 printprint Your change is £,change_given print Your MM's have been dispensed...Enjoy! elif choice==Exit:print Thank You and Have a Nice Day else:print print Invalid choice Thanks,Saba ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Gabriele,
21071736 function calls in 199.883 seconds
The 21 million function calls isn't really a surprise to me, given
18 million calls to file.write(). Given that the majority of the
time is still spent in skymaps5.py, I think you'll need to
instrument that a bit more to figure out where the hotspot is.
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
the major time is required by mymain that is the whole program.
the write on file is an operation that I do in the end but if I
increase the number of data it doesn't increase (I tried doubing
the sample of values and I know why it's behaving in this way) the
third are interpolate and kappa: the two functions (one called
inside the other one)
This is a good finding, in fact. Now, you know which module
contains the bottleneck. Is your CPU pegged when you run that
skymap5.py code? Can you run the profiler on the skymaps5.py code,
too, to locate the specific hotspot? Great to see that you are
using the profiler, effectively, too!
So they are the ones that are taking time.
Now what can I do?
I think you now need to profile skymaps5.py code? Specifically the
stuff in main().
OK, so I have not myself used scipy.interpolate.interp1d before, but
I went to have a look at it. So, you feed interp1d() an x and a y
(e.g. a plot line on a diagram), and it essentially produces its
best guess of a function which will fit that curve, correct?
Well, this is how it performs with random input and an identity
function:
element count 100, duration 0.000
element count 1000, duration 0.001
element count 1, duration 0.005
element count 10, duration 0.055
element count 100, duration 0.858
element count 1000, duration 30.404
So, with 10 million inputs on an admittedly brain-dead function,
there's not a performance bottleneck. If you can find the parts of
your skymaps5.py code that are the bottleneck, then you could post
it here.
I hadn't thought of using interpolate, myself, as I didn't even know
it existed.
Thanks and good luck,
-Martin
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
330445.4700.0006.4440.000
interpolate.py:394(_call_linear)
232.2720.000 21.2790.000 instruments.py:10(kappa)
2313282.1200.0002.1200.000
{numpy.core.multiarray.array}
330441.7190.0003.8360.000
interpolate.py:454(_check_bounds)
660881.6110.0001.6110.000 {method 'reduce' of
'numpy.ufunc'
objects}
330441.1460.000 11.6230.000
interpolate.py:443(_evaluate)
330441.1200.0005.5420.000 interpolate.py:330(__init__)
330440.6590.0002.3290.000 polyint.py:82(_set_yi)
the major time is required by mymain that is the whole program.
Good! Profiles like this allow us to pinpoint issues.
Wait... ?!
The profiler is saying that the majority of time is in my main, but _not_
in auxiliary functions. That's surprising. Am I misreading the profile?
Can you show what mymain is doing?
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
but main is the program that contains everything.
I used the profile in this way:
import cProfile
import pstats
def mymain():
#all the code
#end of main indentation
cProfile.run('mymain()', 'restats', 'time')
p = pstats.Stats('restats')
p.strip_dirs().sort_stats('name')
p.sort_stats('time').print_stats(10)
So all the function I used are contained in main(), so even all the others
that are appeared.
Gabriele
2014-04-10 19:41 GMT-04:00 Martin A. Brown mar...@linux-ip.net:
Gabriele,
21071736 function calls in 199.883 seconds
The 21 million function calls isn't really a surprise to me, given 18
million calls to file.write(). Given that the majority of the time is
still spent in skymaps5.py, I think you'll need to instrument that a bit
more to figure out where the hotspot is.
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
the major time is required by mymain that is the whole program. the write
on file is an operation that I do in the end but if I increase the number
of data it doesn't increase (I tried doubing the sample of values and I
know why it's behaving in this way) the third are interpolate and kappa:
the two functions (one called inside the other one)
This is a good finding, in fact. Now, you know which module contains the
bottleneck. Is your CPU pegged when you run that skymap5.py code? Can you
run the profiler on the skymaps5.py code, too, to locate the specific
hotspot? Great to see that you are using the profiler, effectively, too!
So they are the ones that are taking time.
Now what can I do?
I think you now need to profile skymaps5.py code? Specifically the stuff
in main().
OK, so I have not myself used scipy.interpolate.interp1d before, but I
went to have a look at it. So, you feed interp1d() an x and a y (e.g. a
plot line on a diagram), and it essentially produces its best guess of a
function which will fit that curve, correct?
Well, this is how it performs with random input and an identity function:
element count 100, duration 0.000
element count 1000, duration 0.001
element count 1, duration 0.005
element count 10, duration 0.055
element count 100, duration 0.858
element count 1000, duration 30.404
So, with 10 million inputs on an admittedly brain-dead function, there's
not a performance bottleneck. If you can find the parts of your
skymaps5.py code that are the bottleneck, then you could post it here.
I hadn't thought of using interpolate, myself, as I didn't even know it
existed.
Thanks and good luck,
-Martin
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
sure.
def mymain():
def LEstep(n):
Emin=10**6
Emax=5*(10**10)
Lemin=log10(Emin)
Lemax=log10(Emax)
stepE=(Lemax-Lemin)/n
return (stepE, n, Lemin, Lemax)
if __name__ == __main__:
import sys
if len(sys.argv)=1:
stepENE, nex, Lemin, Lemax = LEstep(200)
elif len(sys.argv)=2:
stepENE, nex, Lemin, Lemax =
LEstep(int(sys.argv[1]))
else:
stepENE, nex, Lemin, Lemax =
LEstep(int(sys.argv[1]))
freq=float(sys.argv[2])
eel = list(range(nex))
eels = np.logspace(Lemin, Lemax, num=nex, endpoint=False)
indpha = list(range(npha))
indobs = list(range(nobs))
rlc = c/(2*pi*freq)
MYMAP1 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP2 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP3 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP4 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP5 = np.zeros([npha, nobs, nex], dtype=float)
count=0
omegacliston = []
alpha = '60_'
for my_line in open('datasm0_60_5s.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gammar1 = my_parts[7]
gammar2 = my_parts[8]
gammar3 = my_parts[9]
gammar4 = my_parts[10]
gammar5 = my_parts[11]
gmlis.append(gammar1)
gmlis.append(gammar2)
gmlis.append(gammar3)
gmlis.append(gammar4)
gmlis.append(gammar5)
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar in gmlis:
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
omegacliston.append(omC)
for w in eel[:]:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
P = Iom*(c/(rho*rlc))/(2*pi)
phps = P/(hcut*omega )
www = phps/(stepPHA*sin(zobs)*stepOB)
myinternet.append(www)
count = count + 1
oo = 0
for k in eel[:]:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
BIGMAPS = [MYMAP1, MYMAP2, MYMAP3, MYMAP4, MYMAP5]
sigmas = [1, 3, 5, 10, 30]
fiq1 = plt.figure()
fiq2 = plt.figure()
fiq3 = plt.figure()
fiq4 = plt.figure()
fiq5 = plt.figure()
fiqqs = [fiq1, fiq2, fiq3, fiq4, fiq5]
multis = zip(sigmas, BIGMAPS, fiqqs)
for sigma, MYMAP, fiq in multis:
filename=alpha+'_'+str(sigma)+'_'+str(npha)+'_'+str(phamin)+'_'+str(phamax)+'_'+str(nobs)+'_'+str(obamin)+'_'+str(obamax)+'_'+str(nex)+'_'+str(Lemin)+'_'+str(Lemax)+'_.dat'
MYfile = open(filename, 'a')
for k in eel[:]:
for j in indobs[:]:
for i in indpha[:]:
A=MYMAP[i, j, k]
stringa = str(A) + ','
MYfile.write(stringa)
accapo = '\n'
MYfile.write(accapo)
MYfile.close()
2014-04-10 19:55 GMT-04:00 Danny Yoo d...@hashcollision.org:
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
Re: [Tutor] difference between expressions and statements
On 4/10/2014 5:48 PM, Jared Nielsen wrote: Thanks for the thorough answer, Bob. I now understand the difference. Thanks for the ACK. It helps me remember I have something to contribute. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Refining Code
On 10/04/14 23:26, Saba Usmani wrote:
My task is :
A food vending machine accepts 10p, 20p, 50p and £1 coins
*/I have designed the following code, but would like to know how to make
it more efficient without making it too complex as I am a beginner
Have you covered functions yet?
If so you can use functions to remove a lot of duplication from the code
which will make it clearer. But if you haven't covered functions then
what you have is not too bad.
Thee is one bad habit you should really avoid.
You should not use input() in Python 2. It is a security risk and
although your program is not going to be used in earnest anywhere its a
bad habit to get into. Better to use raw_input() and then convert to a
number using int() - or float if thats what you need.
[In v3 raw_input has been renamed as input and the v2 input removed.]
this fine? Also, how do I add a loop to this so that once one product
has been dispensed the program asks the user if they would like to
continue and purchase another product? /*
You should probably use a while loop.
You could use this pattern:
while True: # means loop forever
display menu
get choice
if choice is Exit:
break# drop out of the loop
elif choice ==
The final thing is that your coin counting code could
be made more concise by using a dictionary to store
the mapping of menu choice(A-D) to value:
coins = {'A': 0.1, 'B':0.2...}
Rather than all the individual variables.
Then the input code becomes:
print Please select which
choice = raw_input(Enter coin: )
insert_coins = coins[choice]
Its a trade off of more code to define the data
or more code to read the input.
hth
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Gabriele,
but main is the program that contains everything.
And, that is precisely the point of profiling the thing that
contains 'everything'. Because the bottleneck is almost always
somewher inside of 'everything'. But, you have to keep digging
until you find it.
I saw that you replied to Danny Yoo with your code, and I have to
say that this is rather domain-specific, so it may be quite
difficult for somebody to glance at it and figure out where the
hotspot is. It is for this reason that we were asking about
profiling.
Some follow-on questions:
Code: for my_line in open('datasm0_60_5s.dat')
Q: How big is datasm0_60_5s.dat? Unless there's a whitespace
pasting issue, it looks like you are reading that file for each
run through mymain(). Has this file changed in size recently?
Code: kap = instruments.kappa(x)
Q: What is instruments? A module? Is the performance hit there?
Code:
for k in eel[:]:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP,[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
...
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
That should cut down a bit of looping time. Especially as the eel
grows longer.
Another suggestion, that is more along the lines of how do I figure
out what's broken this time in my code. I almost always add the
logging module to any program larger than a few lines. Why?
Because then, I can simply add logger lines and see what's going on.
Since I'm a perfect programmer and, like you, I don't make mistakes,
I never need this, but I do it anyway to look good around my
colleagues (best practices and all).
In seriousness, using logging [0] is not at all tricky for
standalone scripts. (It does get a bit more involved when you are
importing modules and libraries), but,) Consider the following:
import sys
import logging
logformat='%(asctime)s %(name)s %(levelname)s %(message)s'
logging.basicConfig(format=logformat, stream=sys.stderr, level=logging.INFO)
logger = logging.getLogger({ '__main__': None }.get(__name__, __name__))
With that setup at the top of the program, now you can sprinkle
lines like this throughout your code with impunity.
import os
# -- calling logger.info() will print stuff to STDERR
logger.info(silly example %r, os.environ)
# -- calling logger.debug() will not print to STDERR
#using, above config
logger.debug(debug example %d, 1)
# -- Ok, set it so anything that is set to logging.DEBUG (or
#higher) is shown
logger.setLevel(logging.DEBUG)
logger.debug(debug example %d, 2)
# -- and restore the prior pattern; restting so newer .debug lines
#are not shown
logger.setLevel(logging.INFO)
logger.debug(debug example %d, 3)
OK, so why is this useful? Well, timestamps in log lines is one
reason. Another reason is the typical diagnostic technique
What is the value of variable x, y, z, oo, text_contens
Good luck tracking down your peformance issue!
-Martin
[0] https://docs.python.org/2/library/logging.html
https://docs.python.org/3/library/logging.html
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Thu, Apr 10, 2014 at 11:58:30AM -0400, Gabriele Brambilla wrote: Hi, I have a program that is reading near 60 elements from a file. For each element it performs 200 times a particular mathematical operation (a numerical interpolation of a function). Now these process takes near 8 hours. Why are you repeating each operation 200 times? Surely you don't mean something like this? for element in elements_from_file(): for i in range(200): result = function(element) Before spending your time re-writing the function in C, it may help checking that there are no inefficencies in the code. Calculating the function may not be what is slowing your code down. It might help if you show us your code. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Fri, Apr 11, 2014 at 10:59:05AM +1000, Steven D'Aprano wrote: It might help if you show us your code. Oops, never mind, I see you have done so. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Ok, good. There's a few things you'll want to fix in your mymain() in order for the profiler to work more effectively in pinpointing issues. 1. Move functionality outside of if __name__ == '__main__': At the moment, you've put the entire functionality of your program in the body of that if statement within mymain. That structure is probably not right. I see that this block is computing values for stepENE, nex, Lemin, Lemax, and, conditionally, freq. This should be lifted out into its own function. You should also note that, because 'freq' is computed conditionally, there are certain code paths in which your mymain() will fail. This is most likely a bad thing. Recommendation: have mymain() take in parameters. Move LEstep() toplevel. Restructure to: import sys def mymain(stepENE, nex, Lemin, Lemax, freq): ## everything starting after eel = list(range(nex))... ## if __name__ == '__main__': if len(sys.argv)=1: stepENE, nex, Lemin, Lemax = LEstep(200) elif len(sys.argv) = 2: stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1])) else: stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1])) freq=float(sys.argv[2]) mymain(stepENE, nex, Lemin, Lemax, freq) 2. Do not slice lists unless you really mean to do so. I see a lot of slices that do not look right. Examples like: for k in eel[:]: # code cut ... Just loop over eel. No copy necessary. This is doing a lot more memory copying over and over again. Instead: for k in eel: # code cut ... Do this everywhere you're creating slices with [:], unless you really need to copy. This is happening in multiple places in the code. 3. Be sure to remove dead variables. There are variables here that are not used. omegacliston is dead code, for example. You're appending to it, but doing nothing with its value. 4. Watch for repetitive code. There's something about the use of MYMAP1, MYMAP2, etc. that looks very suspicious. That is, the block here: oo = 0 for k in eel: MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo] oo = oo + 1 feels repetitive and strange. Martin Brown identifies this problem as well, so at least we're on the same page. Solving this problem takes a little bit of work. If you have five maps, with some kind of relationship, represent and use that. Ah. You're already doing this by representing BIGMAPS at reporting time. Then move the definition of an array of maps to the front of the code. Use a container holding those five maps in a single variable. Call it MYMAPS. MYMAPS = [np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float)] Wen you're tempted to say MYMAP1, use MYMAPS[0]. MYMAP2 -- MYMAPS[1], and so on. This will allow you to dissolve a lot of complexity out of the code. We'll see this in a moment. 5. Change the computational structure. The computation of MYMAPS being done after the processing of gmlis is not right. It's the whole reason why there's this awkward intermediate myinternet structure that's used just to fill in each MYMAP later. Do the processing as part of your earlier loop. We know that gmlis is exactly five elements long. Just say so: gmlis = [my_parts[7], my_parts[8], my_parts[9], my_parts[10], my_parts[11]] Once you have this, and once you have a definition of MYMAPS, then you can kill a lot of the code by doing a zip loop across them both: for gammar, MYMAP in zip(gmlis, MYMAPS):
Re: [Tutor] improving speed using and recalling C functions
Comment: You are looping over your sliced eel five times. Do you need to? I like eel salad a great deal, as well, but, how about: for k in eel: MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo] MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo] MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo] MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo] MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo] oo = oo + 1 Hi Gabriele, Also note that, when Martin looked at this part of the code, he unfortunately misinterpreted its effect; Martin's proposed rewrite here does not preserve the meaning of the original code. But rather than wag my finger at how Martin interpreted the code, I'd rather make the observation that this is a warning sign that the original code here was not easy to understand. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Refining Code
Hi Saba, Do you see any similarities between each of the snack choices? Do you see any differences? (Did you happen to use copy-and-paste at any time when you wrote the program?) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Danny,
I followed your suggestion.
Tomorrow morning I will run this new version of the code.
Now using a sample of 81 elements (instead of 60) the profile returns:
Thu Apr 10 23:25:59 2014restats
18101188 function calls in 1218.626 seconds
Ordered by: internal time
List reduced from 13 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 1015.803 1015.803 1218.334 1218.334 skymaps5.py:44(mymain)
18101000 202.4900.000 202.4900.000 {method 'write' of 'file'
objects}
10.2920.292 1218.626 1218.626 string:1(module)
60.0290.0050.0290.005 {open}
50.0100.0020.0100.002 {method 'close' of 'file'
objects}
810.0020.0000.0020.000 {method 'split' of 'str'
objects}
820.0010.0000.0010.000 {zip}
10.0000.0000.0000.000 function_base.py:8(linspace)
10.0000.0000.0000.000 function_base.py:93(logspace)
50.0000.0000.0000.000 {numpy.core.multiarray.zeros}
Anyway I would like to try to speed it up using C functions (and maybe
comparing the resuts of the two profile in the end)
How can I do it now? Can I use Cython?
Thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
Hi Gabriele,
Also note that, when Martin looked at this part of the code, he
unfortunately misinterpreted its effect; Martin's proposed rewrite
here does not preserve the meaning of the original code. But rather
than wag my finger at how Martin interpreted the code, I'd rather make
the observation that this is a warning sign that the original code
here was not easy to understand.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
