Re: [Tutor] finally without try or except
On Jul 31, 2012, at 12:26 PM, tutor-requ...@python.org wrote: > Message: 2 > Date: Tue, 31 Jul 2012 10:44:29 -0400 > From: Tino Dai > To: "Steven D'Aprano" > Cc: "tutor@python.org" > Subject: Re: [Tutor] finally without try or except > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Mon, Jul 30, 2012 at 9:01 PM, Steven D'Aprano wrote: > >> If you want to be robust, it is best not to try to beat the database. That >> means you should write to the database as soon as you can, as often as you >> need to, and let the database do what it does best: reliable transaction >> storage. Any decent database will guarantee ACID compliance (atomicity, >> consistency, isolation, durability). >> > >> But sometimes you need to compromise on robustness for speed or >> convenience. >> >> >> [...] >> >> This is how the program was originally structured, but we found >>> performance >>> problems with in. >>> It's using Django and an Oracle DB, which is notoriously bad for single >>> record read/writes. So, I >>> figured why don't we append them all to an array, and write them out at >>> the >>> end of the program. >>> >> >> I suggest you periodically check the item log, and if there are more than >> (say) 20 items, you write them to the database and clear the temporary >> list. That way, the number of item logs should never get too large, and the >> performance shouldn't suffer too greatly. You will need to tweak that >> number to be more or less depending on how poorly the Oracle DB performs. >> >> Then, at the very end of the program, you write whatever items are left in >> a finally clause, or atexit. That way, the finally clause should be nice >> and speedy and the user is unlikely to hit Ctrl-C a second time. >> >> > I think that is the process I'm going to take. It doesn't seem as elegant, > but I think it's a good compromise. Another option might be to write each item to a different database which has better performance for single record writes—and, I'm betting someone on this list will have an excellent recommendation for such a DB. Then, you could batch the updates to the Oracle DB independently of the main program (the one related to your question). Take care, Don___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] ImportError
On Tue, Jul 31, 2012 at 10:32 AM, Tino Dai wrote: > > File "/home/tdai/ProjectOne-TNT/leg_apps/etl/transfers/__init__.py", line > 8, in > from api import models > File "/home/tdai/ProjectOne-TNT/leg_apps/api/models.py", line 20, in > > from etl.transfers import eastern > ImportError: cannot import name eastern It looks to me like you have a circular import problem. Before the etl.transfers package defines eastern it attempts to import api.models, but executing the latter tries and fails to import (the as yet undefined) eastern from etl.transfers. If this is the problem, you could refactor the cyclic dependency into its own module, or move the import to after eastern is defined, or do the import in functions/methods that run after the modules have been loaded, etc. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] finally without try or except
On Tue, Jul 31, 2012 at 5:23 PM, Prasad, Ramit wrote: > [snip] > > > >> This will be the most robust as it will > > >> also work for cases where the program is terminated without the use of > > >> the keyboard (i.e. kill -9, task manager, computer reboot, etc.) but > > > > That unfortunately is not so. kill -9 does *not* send a signal or raise > an > > exception. It just kills the process dead, instantly and without warning. > > Does a plain kill send a signal or do you need to use an option > like SIGHUP? > > kill is supposed to always send a signal. Without options it will send the process SIGTERM, which indicates to the process it should terminate. kill -9 sends SIGKILL, which is kind of special since it is a signal that never really arrives at the process itself. When the kernel notices the SIGKILL it will just kill the process. Hugo ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] finally without try or except
[snip] > >> This will be the most robust as it will > >> also work for cases where the program is terminated without the use of > >> the keyboard (i.e. kill -9, task manager, computer reboot, etc.) but > > That unfortunately is not so. kill -9 does *not* send a signal or raise an > exception. It just kills the process dead, instantly and without warning. Does a plain kill send a signal or do you need to use an option like SIGHUP? [snip] > > >> it might also slow the program down. You could also try writing the > >> item log to a local file using pickle or csv. > > Better is to use the logging module rather than to reinvent the wheel. > I was under the impression it was not logging in the traditional sense of the word, but more like storable "status" so that reprocessing can occur. I could be wrong though. Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] finally without try or except
On Mon, Jul 30, 2012 at 9:01 PM, Steven D'Aprano wrote: > If you want to be robust, it is best not to try to beat the database. That > means you should write to the database as soon as you can, as often as you > need to, and let the database do what it does best: reliable transaction > storage. Any decent database will guarantee ACID compliance (atomicity, > consistency, isolation, durability). > > But sometimes you need to compromise on robustness for speed or > convenience. > > > [...] > > This is how the program was originally structured, but we found >> performance >> problems with in. >> It's using Django and an Oracle DB, which is notoriously bad for single >> record read/writes. So, I >> figured why don't we append them all to an array, and write them out at >> the >> end of the program. >> > > I suggest you periodically check the item log, and if there are more than > (say) 20 items, you write them to the database and clear the temporary > list. That way, the number of item logs should never get too large, and the > performance shouldn't suffer too greatly. You will need to tweak that > number to be more or less depending on how poorly the Oracle DB performs. > > Then, at the very end of the program, you write whatever items are left in > a finally clause, or atexit. That way, the finally clause should be nice > and speedy and the user is unlikely to hit Ctrl-C a second time. > > I think that is the process I'm going to take. It doesn't seem as elegant, but I think it's a good compromise. > > This will be the most robust as it will >>> also work for cases where the program is terminated without the use of >>> the keyboard (i.e. kill -9, task manager, computer reboot, etc.) but >>> >> > That unfortunately is not so. kill -9 does *not* send a signal or raise an > exception. It just kills the process dead, instantly and without warning. > > Likewise for cases when somebody trips over the power cord and disconnects > the server from the power. (Really paranoid sys admins insist on having two > power supplies connected to two different UPSes for their servers.) > > > > > it might also slow the program down. You could also try writing the >>> item log to a local file using pickle or csv. >>> >> > Better is to use the logging module rather than to reinvent the wheel. > > Do your item logs actually need to go in the database? Perhaps you could > just write them to a local log file and leave the database for more > critical records. > > Yes for reprocessing. :( -Tino ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] ImportError
On 31/07/2012 15:32, Tino Dai wrote: > Hi All, > >I have been banging my head against a wall trying to figure it > out. I'm getting a ImportError on a > class that I know exists. I'm wondering if there is some dark corner of > the import mechanism that Try running python with the -v parameter: and then "from etl.transfers import eastern". That will at least confirm the file which Python is reading for the import. TJG ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] ImportError
Hi All,
I have been banging my head against a wall trying to figure it out.
I'm getting a ImportError on a
class that I know exists. I'm wondering if there is some dark corner of the
import mechanism that
I don't understand.
-Tino
Traceback (most recent call last):
File "manage.py", line 14, in
execute_manager(settings)
File
"/usr/local/lib/python2.6/dist-packages/Django-1.4-py2.6.egg/django/core/management/__init__.py",
line 459, in execute_manager
utility.execute()
File
"/usr/local/lib/python2.6/dist-packages/Django-1.4-py2.6.egg/django/core/management/__init__.py",
line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File
"/usr/local/lib/python2.6/dist-packages/Django-1.4-py2.6.egg/django/core/management/__init__.py",
line 261, in fetch_command
klass = load_command_class(app_name, subcommand)
File
"/usr/local/lib/python2.6/dist-packages/Django-1.4-py2.6.egg/django/core/management/__init__.py",
line 69, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File
"/usr/local/lib/python2.6/dist-packages/Django-1.4-py2.6.egg/django/utils/importlib.py",
line 35, in import_module
__import__(name)
File
"/home/tdai/ProjectOne-TNT/leg_apps/../leg_apps/etl/management/commands/driver.py",
line 22, in
from etl.transfers.bill_subject import Bill_Subject
File "/home/tdai/ProjectOne-TNT/leg_apps/etl/transfers/__init__.py", line
8, in
from api import models
File "/home/tdai/ProjectOne-TNT/leg_apps/api/models.py", line 20, in
from etl.transfers import eastern
ImportError: cannot import name eastern
< etl/transfers/__init__.py >
existing_pending_items = {}
new_pending_items = []
class eastern(object):
@staticmethod
def localize(theDate):
if theDate:
return easternTz.localize(theDate)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pydev + Remote System Explorer (RSE) : problems with local imports when working localy
hmmm ... menu "window">"Reset perspective" and "close all perspectives" seems to have fix the problem. Sorry for unnecessary mail. On Tue, Jul 31, 2012 at 2:02 PM, learner404 wrote: > Hello List, > > I'm very happy with PyDev plugin for Eclipse as a Python IDE (I'm on > Eclipse 3.7, WinXP with latest Pydev). > Recently I've installed the Remote System Explorer plugin for Eclipse > which adds the ability to edit remote files on a server when I need it > (awesome). > > Unfortunately it's messing my ability to work on some Python > files locally on my PC (those with local imports): > File "C:\Documents and > Settings\myaccount\workspace\RemoteSystemsTempFiles\LOCALHOST\c\Documents > and Settings\myaccount\workspace\myprojet\myfile.py", line 50, in ? > ImportError: No module named mymodule > (it works only if I "python myfile.py" in the command shell) > > I know some people of the list are using Eclipse and maybe they use RSE > too. > > Do you know if I can easily deactivate RSE so I can run a python script > normally (without the *localhost* inserting in the path) ? > Is there a configuration to do to keep both ? > > I've tried to switch form perspective "remote systems" to perspective > "pydev" but I have the same problem. > > Thanks. > > francois > > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pydev + Remote System Explorer (RSE) : problems with local imports when working localy
Hello learner404, Tuesday, July 31, 2012, 8:02:26 AM, you wrote: > Hello List, > I'm very happy with PyDev plugin for Eclipse as a Python IDE (I'm > on Eclipse 3.7, WinXP with latest Pydev). > File "C:\Documents and > Settings\myaccount\workspace\RemoteSystemsTempFiles\LOCALHOST\c\Documents > and Settings\myaccount\workspace\myprojet\myfile.py", line 50, in ? > ImportError: No module named mymodule Did you trying using forward slashes / rather than backslashes \ ? Alan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Pydev + Remote System Explorer (RSE) : problems with local imports when working localy
Hello List, I'm very happy with PyDev plugin for Eclipse as a Python IDE (I'm on Eclipse 3.7, WinXP with latest Pydev). Recently I've installed the Remote System Explorer plugin for Eclipse which adds the ability to edit remote files on a server when I need it (awesome). Unfortunately it's messing my ability to work on some Python files locally on my PC (those with local imports): File "C:\Documents and Settings\myaccount\workspace\RemoteSystemsTempFiles\LOCALHOST\c\Documents and Settings\myaccount\workspace\myprojet\myfile.py", line 50, in ? ImportError: No module named mymodule (it works only if I "python myfile.py" in the command shell) I know some people of the list are using Eclipse and maybe they use RSE too. Do you know if I can easily deactivate RSE so I can run a python script normally (without the *localhost* inserting in the path) ? Is there a configuration to do to keep both ? I've tried to switch form perspective "remote systems" to perspective "pydev" but I have the same problem. Thanks. francois ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help please!
>> Ex: >> > No! >> print "Hi %s! You like %s and %s" (user_name, x, y) >> > Yes! > print "Hi %s! You like %s and %s" % (user_name, x, y) >> >> >>> Forgot the interpolation operator(%). Thank you very much Joel for correcting me. Regards, ttmticdi. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help please!
On Tue, Jul 31, 2012 at 5:52 AM, ttmticdi . wrote: >> > print "Mum is in a %s mood" % (mum_mood) >> > print "Dad is in a %s mood" % (dad_mood) >> >> > > > Hi Victoria! > > Since you have only one format character in the strings above there is > no need to surround the variables mum_mood and dad_mood with > parenthesis. > You only do that when you have multiple formats in your string that > print multiple variables. > > Ex: > No! > print "Hi %s! You like %s and %s" (user_name, x, y) > Yes! print "Hi %s! You like %s and %s" % (user_name, x, y) > > >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> http://mail.python.org/mailman/listinfo/tutor > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor -- Joel Goldstick ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help please!
> > print "Mum is in a %s mood" % (mum_mood) > > print "Dad is in a %s mood" % (dad_mood) > > Hi Victoria! Since you have only one format character in the strings above there is no need to surround the variables mum_mood and dad_mood with parenthesis. You only do that when you have multiple formats in your string that print multiple variables. Ex: print "Hi %s! You like %s and %s" (user_name, x, y) > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
