MemoryError on insert_query

[thanos]

os: win32
db: oracle 10
django: 1.3
python: 2.7


While inserting around 750K of records I get the following error

trades done 661000 in  4007.3089
trades done 662000 in  4011.9012
Traceback (most recent call last):
  File "manage.py", line 14, in 
execute_manager(settings)
  File "c:\python27\lib\site-packages\django\core\management
\__init__.py", line 438, in execute_manager
utility.execute()
  File "c:\python27\lib\site-packages\django\core\management
\__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File "c:\python27\lib\site-packages\django\core\management\base.py",
line 191, in run_from_argv
self.execute(*args, **options.__dict__)
  File "c:\python27\lib\site-packages\django\core\management\base.py",
line 220, in execute
output = self.handle(*args, **options)
  File "F:\work\control\src\control\migration\management\commands
\convert.py", line 33, in handle
print conversion.execute()
  File "F:\work\control\src\control\..\control\migration\models.py",
line 341, in execute
return PROCESSOR(self, **kwa)
  File "F:\work\control\src\control\staging\models.py", line 96, in
extract
trade.save()
  File "c:\python27\lib\site-packages\django\db\models\base.py", line
460, in save
self.save_base(using=using, force_insert=force_insert,
force_update=force_update)
  File "c:\python27\lib\site-packages\django\db\models\base.py", line
553, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
  File "c:\python27\lib\site-packages\django\db\models\manager.py",
line 195, in _insert
return insert_query(self.model, values, **kwargs)
  File "c:\python27\lib\site-packages\django\db\models\query.py", line
1436, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
  File "c:\python27\lib\site-packages\django\db\models\sql
\compiler.py", line 791, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "c:\python27\lib\site-packages\django\db\models\sql
\compiler.py", line 735, in execute_sql
cursor.execute(sql, params)
  File "c:\python27\lib\site-packages\django\db\backends\util.py",
line 43, in execute
logger.debug('(%.3f) %s; args=%s' % (duration, sql, params),
MemoryError

In my code I've been careful to only use iterators, etc and should
only be using enough memory for one record (around 1k). While doing
inserts is there any accumulation of objects going on ? For instance
is commit only occurring after n saves.


here is a stripped down version.


@classmethod
def extract(cls, conversion,  migration_run_id = 6711):
"""
   This code is used to migrate trades from one trading system to
another.
   """
tick = time.time()
run = Job.objects.get(name=cls.JOB_NAME).start()
if migration_run_id is None:
migration_run_id = run.pk

_, src_cursor = conversion.source_sql.execute() # getting a
cursor on the source database (using sqlachemy against mssql here )
count = 0
for record in src_cursor:
count +=1
fields = conversion.source_sql.source.to_dict(src_cursor,
record) # justs converts a record to a dict.
fields['run'] = run
fields['migration_run'] = migration_run_id
trade = cls(**fields)  # creating a django model
trade.save() # here we run out of memory after about
650,000 inserts. All records where committed before crash.
if count % 1000 == 0:
print 'trades done', count, 'in ', time.time() - tick
print 'trades done', count, 'in ', time.time() - tick
return migration_run_id, run.pk, count

Thanks




-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: MemoryError on insert_query

[thanos]

Thanks Tom
settings.DEBUG was True, now False and all well

Thanks again

 Thanos



trades done 75 in  3640.7334
trades done 751000 in  3645.2171
trades done 751812 in  3649.1547
(6711, 6732, 751812)



On Jul 19, 11:56 am, Tom Evans  wrote:
> On Tue, Jul 19, 2011 at 4:33 PM, thanos  wrote:
> > os: win32
> > db: oracle 10
> > django: 1.3
> > python: 2.7
>
> > While inserting around 750K of records I get the following error
>
> settings.DEBUG is True or False?
>
> If True, it will store a log of every query and statistics about that
> query in django.db.connection.queries. IE, with settings.DEBUG, you
> will have a (deliberate) memory leak until the connection is
> destroyed.
>
> Cheers
>
> Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Slow page load

[thanos]

Why are you calling os.path.join 27636 times ? These are the questions
you could be asking yourself ...

On Nov 18, 10:55 am, Dre  wrote:
> I thought my code was faulty, but the admin interface is also very
> slow.
> It takes 8096 msec for a model with 1000 database rows to be opened
> using the admin interface with debug-toolbar enabled. It is paginated
> to
> 100 entries per page, and SQL makes 339 queries in 128.77 msec.
> I have two foreign keys on this model to models with < 20 entries
> each.
>
> After turning on additional profiling using hotshot/debug toolbar, it
> takes
> 61 sec (additional profiling code being run). SQL makes 675 queries in
> 297.63 msec.
>
> Profiling result:
>
>          1320805 function calls (1285639 primitive calls) in 12.122
> CPU seconds
>
>    Ordered by: internal time, call count
>    List reduced from 916 to 100 due to restriction <100>
>
>    ncalls  tottime  percall  cumtime  percall
> filename:lineno(function)
>    219094    2.790    0.000    3.576    0.000 /usr/lib/python2.7/
> posixpath.py:130(islink)
>    206072    2.248    0.000    2.248    0.000 /usr/lib/python2.7/
> posixpath.py:60(join)
> 27636/14223    1.751    0.000    8.587    0.001 /usr/lib/python2.7/
> posixpath.py:355(realpath)
>    219094    0.545    0.000    0.786    0.000 /usr/lib/python2.7/
> stat.py:55(S_ISLNK)
>     28294    0.494    0.000    0.494    0.000 /usr/lib/python2.7/
> posixpath.py:312(normpath)
>     13413    0.343    0.000    1.108    0.000 /usr/lib/python2.7/
> posixpath.py:378(_resolve_link)
>     14562    0.324    0.000    0.324    0.000 /usr/lib/python2.7/
> linecache.py:43(checkcache)
>       339    0.253    0.001    8.910    0.026 /usr/lib/pymodules/
> python2.7/debug_toolbar/panels/sql.py:31(tidy_stacktrace)
>    219098    0.240    0.000    0.240    0.000 /usr/lib/python2.7/
> stat.py:24(S_IFMT)
> 18480/4037    0.202    0.000    0.502    0.000 /usr/lib/python2.7/
> copy.py:145(deepcopy)
>       339    0.181    0.001    0.598    0.002 /usr/lib/python2.7/
> traceback.py:280(extract_stack)
>     55930    0.132    0.000    0.132    0.000 /usr/lib/python2.7/
> posixpath.py:51(isabs)
>       301    0.094    0.000    0.293    0.001 /usr/lib/pymodules/
> python2.7/debug_toolbar/panels/sql.py:49(get_template_info)
>     15393    0.075    0.000    0.075    0.000 /usr/lib/pymodules/
> python2.7/django/utils/functional.py:274(__getattr__)
>     18138    0.075    0.000    0.075    0.000 /usr/lib/python2.7/
> copy.py:267(_keep_alive)
>     31585    0.072    0.000    0.072    0.000 /usr/lib/pymodules/
> python2.7/django/views/debug.py:18(linebreak_iter)
>       339    0.072    0.000   10.087    0.030 /usr/lib/pymodules/
> python2.7/debug_toolbar/panels/sql.py:88(execute)
>     14225    0.070    0.000    0.338    0.000 /usr/lib/python2.7/
> posixpath.py:341(abspath)
> 2353/2018    0.069    0.000    0.279    0.000 /usr/lib/pymodules/
> python2.7/django/utils/tree.py:55(__deepcopy__)
> 311    0.064    0.000    0.128    0.000 /usr/lib/pymodules/python2.7/
> django/template/loaders/app_directories.py:52(load_template_source)
>      1009    0.060    0.000    0.585    0.001 /usr/lib/pymodules/
> python2.7/django/db/models/sql/query.py:229(clone)
>       968    0.057    0.000    0.111    0.000 /usr/lib/pymodules/
> python2.7/django/db/models/base.py:273(__init__)
>     14562    0.054    0.000    0.093    0.000 /usr/lib/python2.7/
> linecache.py:13(getline)
>       339    0.052    0.000    0.052    0.000 /usr/lib/pymodules/
> python2.7/django/db/backends/mysql/base.py:263(_valid_connection)
> 9376/8557    0.050    0.000    4.411    0.001 /usr/lib/pymodules/
> python2.7/django/utils/encoding.py:54(force_unicode)
>       339    0.050    0.000    0.070    0.000 /usr/lib/pymodules/
> python2.7/MySQLdb/cursors.py:273(_do_query)
> 2689/1344    0.050    0.000    0.233    0.000 /usr/lib/python2.7/
> copy.py:234(_deepcopy_tuple)
>      6365    0.044    0.000    0.044    0.000 /usr/lib/python2.7/
> encodings/utf_8.py:15(decode)
>      1200    0.041    0.000   10.759    0.009 /usr/lib/pymodules/
> python2.7/django/contrib/admin/templatetags/admin_list.py:
> 129(items_for_result)
>       932    0.037    0.000   11.203    0.012 /usr/lib/pymodules/
> python2.7/django/db/models/fields/related.py:286(__get__)
>   700/100    0.033    0.000    0.108    0.001 /usr/lib/pymodules/
> python2.7/django/db/models/query.py:1144(get_cached_row)
>      2713    0.030    0.000    0.032    0.000 /usr/lib/pymodules/
> python2.7/django/utils/datastructures.py:89(__init__)
>       468    0.027    0.000    0.033    0.000 /usr/lib/pymodules/
> python2.7/django/db/backends/mysql/compiler.py:4(resolve_columns)
>       600    0.026    0.000    0.044    0.000 /usr/lib/pymodules/
> python2.7/django/contrib/admin/util.py:321(display_for_field)
>      3231    0.025    0.000    0.025    0.000 /usr/lib/pymodules/
> python2.7/django/utils/safestring.py:89(mark_safe)
>     14562    0.024    0.000    0.039    0.000 /usr/lib/python2.7/
> linecache.py:

Re: Python Database Framework

[thanos]

Django is great and I've writing quite a lot with it but my projects
need dynamic multiple database connections. Now with 1.2 I might do
things differently. But I would miss elixir.ext.versioned !

thanos

On May 14, 11:07 am, Jirka Vejrazka  wrote:
> > Do anyone knows a Python Database framework, which resembles Django's
> > approach to database programming? I need it to use outside of Django, for
> > other Python scripts.
>
> Any reason why you wouldn't Django itself? I do use the ORM to
> multiple projects that have no relation to web. Works very well.
>
>    Cheers
>
>      Jirka
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: "Pro Django" book still up to date

[thanos]

It's useless. The "The definitive guide to Django" by Holovaty and
Kaplan-moss is okay, but the Pro version is really just a waste of
paper and time.

Thanos Vassilakis

On Jan 27, 11:55 am, Achim Domma  wrote:
> Hi,
>
> I'm interested in buying the "Pro Django" book. I could not find any
> hint about what version it's written for. As it covers mostly
> internals and the ideas behind Django, it should not be outdated as
> quickly as other books. Has anybody read that book and can give some
> feedback about how useful it is for the current version of Django?
>
> cheers,
> Achim

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What's the going rate for a Django developer?

[thanos]

In new York I know a few consultants at $1100/day - and that rate is
confirmed.

On Jan 29, 9:16 am, Dougal Matthews  wrote:
> I think the answer to that is probably "it depends".
>
>    1. Permanent or contractor?
>    2. What country?
>    3. Often it even matters what city?
>    4. How much experience do they have?
>
> It's probably best for you to look at job adverts and see what they are
> offering. Some listings onhttp://djangogigs.cominclude the salary/budget.
>
> Cheers,
> Dougal
>
> On 29 January 2010 12:00, Astley Le Jasper wrote:
>
>
>
> > Any ideas?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > django-users+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Images, videos and thumbnails.

[thanos]

I would take a different tack. Store my videos is a service such as
Vimeo and then link to their generated thumbnails.
I'm doing that using django-picasa (http://pypi.python.org/pypi/django-
picasa/1.2). The code might give you an idea now how do it. I'd also
serve up my videos from them and save my bandwidth.

t.



On Feb 11, 12:17 pm, Nasp  wrote:
> Hey guys, I just did the big switch from PHP and I'm wondering if you
> can help me solve this issue the django way.
>
> The big picture is, we've got news associated with images and videos.
> Those "medias" can be ordered and must provide thumbnails for the
> client side.
>
> So far i came up with inline ordering in the admin and thumbnail
> generations via sorl.
> I found out ffmpeg could easily extract thumbnails from videos so i
> was about to try to build a wrapper around it to handle video files
> the way sorl does with images via PIL. Do you know any other way to
> generate video thumbnails?
>
> In order to structure the project i built a django app called "media"
> and defined a VideoField(forms.FileField) class, this way we extract .
> I'm planing to use an intermediate field called MediaField which will
> handle the abstraction with a dictionary association based on the
> model type attribute.
>
> class NewsMedia(models.Model):
>     """
>     News Media
>     """
>     MEDIA_CHOICES = (
>         ('image', _('image')),
>         ('video', _('video')),
>     )
>     news             = models.ForeignKey(News)
>     type             = models.CharField(max_length=1,
> choices=MEDIA_CHOICES)
>     media            = models.MediaField(medias={
>                                                  'images':
> models.ImageField(upload_to='uploads/images/'),
>                                                  'video' :
> models.VideoField(upload_to='uploads/videos/'),
>                                                  },
>                                          type_field='type');
>     order            = models.IntegerField(null=True, blank=True)
>
>     class Meta:
>         ordering     = ('order',)
>
> Also i'm having problems trying to figure out whats the exact
> difference between django.form.FileField and
> django.db.models.FileField.
>
> I'd greatly appreciate If you have any idea how to implement those
> features or know an already existing solution.
>
> thanks,
> Simon

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Getting uploaded images to show

[thanos]

If your images are uploaded_to='media'

I do in my settings.py
STATIC_DOC_ROOT = os.path.join(ROOT,'media')


ADMIN_MEDIA_PREFIX = '/media/admin/'
MEDIA_ROOT  = STATIC_DOC_ROOT
MEDIA_URL  = '/media/'


And my urls.py includes this:
urlpatterns = patterns('',
.
.
.
(r'^admin/media/(?P.*)$', 'django.views.static.serve',
{'document_root': settings.STATIC_DOC_ROOT}),
(r'^admin/', include(admin.site.urls)),
)

I hope that kind of helps. Currently  I use django-picasa (
http://code.google.com/p/django-googledata/ ) to store my uploaded
images and show them. It will even show you an thumbnail in your admin
view !



Thanos



On Feb 13, 5:14 am, holger  wrote:
> I am new to django and I am trying to get an image upload to work.
>
> I am using the admin interface with a ImageField in the model.
>
> imagefilename = models.ImageField(upload_to = 'uploads/')
>
> My media url is
> MEDIA_URL = 'http://127.0.0.1:8000/media/'
>
> and my media_root is
> MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media')
>
> where project_root points to the root folder for the project
>
> So I want the images to be uploaded tohttp://127.0.0.1:8000/media/uploads/
>
> I can see the images being uploaded to the directory but I can´t
> access the file through the url in the template.
>
> What am I missing?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Need help understanding a custom upload example

[thanos]

I would use Django's current behavior. This defaults to using  64K
chunks when a file is bigger that FILE_UPLOAD_MAX_MEMORY_SIZE.
Since 64k chunks  should be small enough for Webfaction. You just need
to set  FILE_UPLOAD_MAX_MEMORY_SIZE  in the settings.py to something
low such as 102500 (100K).

I'm also using WebFaction and that's the strategy I'm using to handle
very large images (> 100 Meg).

Yours

Thanos.



On Feb 16, 10:07 am, Brandon Taylor  wrote:
> Hi Everyone,
>
> I'm confused about creating a custom upload handler and how it ties
> into my model. From the docs, this is an example of a custom upload
> handler, which I'm assuming goes in my view, or is accessed from my
> view:
>
> def handle_uploaded_file(f):
>     destination = open('some/file/name.txt', 'wb+')
>     for chunk in f.chunks():
>         destination.write(chunk)
>     destination.close()
>
> Let's say my model is:
>
> class MyModel(models.Model):
>     my_file = models.FileField(upload_to='uploads/')
>
> How do the FileField "my_file" and the handler "handled_uploaded_file"
> work together? In my project, I need to allow users to upload large-
> ish files. Say 10 - 20 MB. What's the best way to handle this
> functionality? I would sincerely appreciate an example if someone has
> one.
>
> Kindest Regards,
> Brandon

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: what is the name of your super class?

[thanos]

How about:

>>> class MySuperClass: pass
...
>>> class MySubClass(MySuperClass): pass
...
>>> c = MySubClass
>>>
>>> c.__bases__[0].__name__
'MySuperClass'
>>>

What I always ask develoeprs hwo join my team is "Please read the
Language Reference!"  Its a great read, I'm not joking and your answer
is  there some where under Data Model.


Thanos





On Feb 19, 5:51 pm, Joel Stransky  wrote:
> Bit of a python question here.
> Say I have a class named MySuperClass and an extension of that class called
> MySubClass. When referring to MySubClass (the class, not an instance of it),
> how would I retrieve the class name of its super class.
>
> For instance if I had:
> c = MySubClass
>
> I'd like to know if there is a method that would work to the effect of:
> b = superclassname(c)
> which would return MySuperClass
>
> Thanks.
>
> --
> --Joel Stransky
> stranskydesign.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Dev environnement, Eclipse, pydev, Windows, remote developement

[thanos]

How about running your linux in  vmware or virtualbox on your windows
machine.
Its the right way to go. You can then develop in you production
environment. BTW do you use virtualenv. It can save you a lot of pain!

Thanos


On Feb 25, 9:59 am, Eric Brockmann 
wrote:
> Hi all !
>
> Just begining to set up my work environnement...
> Is there a way to develop using Eclipse/Pydev from my workstation on  
> Windows (i know, i know...can't choose )  to create a Linux Django site. ?
> The remote Linux machine is very basic and i can't install any dev
> environnement on it.
> Is Remote editing posssible with Eclipse  ?
> Any advice very welcomed
> Thanks .

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: ImageField and built-in filesystem storage

[thanos]

You take a look at how I did things in

http://code.google.com/p/django-googledata/

Thanos

On Mar 2, 4:44 am, Xavier Ordoquy  wrote:
> Hi,
>
> I'm currently working on Django for a website on a legacy database but I'm 
> facing an issue on getting ImageField working with it in the admin.
>
> In the database images are stored as a simple file name, because they have 
> several size precomputed on different directories.
> So basically, I need to:
>  - save the filename - and only the filename - the database
>  - save the file to a given location and resize it to different size
>
> I tried different ways to get this working but none did work in order to get 
> the correct filename in the database.
>
> First I edited the ImageField.name when model is saved but the name gets 
> overwritten when saving the model.
>
> Then I gave the build-in filesystem storage but it didn't work 
> better.http://docs.djangoproject.com/en/1.1/topics/files/#the-built-in-files...doesn't
>  mention you need upload_to
> However, whenever upload_to is set, it overrides the storage location which 
> seems to contradict ticket #8918.
>
> Has someone already solved a similar issue ?
>
> Xavier.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: What validation tests are applied to ImageField?

[thanos]

DJango's ImageFields uses the PIL verify to check the image.

trial_image = Image.open(file)
trial_image.verify()


See: http://www.pythonware.com/library/pil/handbook/image.htm


On Mar 15, 10:56 pm, john2095  wrote:
> The documentation states about the ImageField
>
> "Like FileField, but validates that the uploaded object is a valid
> image. Has two extra optional arguments:"
>
> http://docs.djangoproject.com/en/dev/ref/models/fields/#imagefield
>
> I went cruising the source to try and work out what tests are
> implemented to define "a valid image" but didn't get much
> satisfaction. There seemed to be a fair bit of talk about dimensions
> but nothing about much else.  I was thinking/hoping to find something
> which inspected the binary stream to ensure the .jpg/.gif/.png/.bmp
> fit its applicable formats.
>
> Question is "What is the definition of a 'valid image'" in this
> context?"
> and, optionally, where is the source code for that?
>
> t.i.a.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Generate brochure pdf

[thanos]

I've been doing generating complex catalogs and books from django by
using TeX  or LaTex. Generating PDF from scratch is a pig. Its not
much different from programming in postscript. Latex or Tex make life
very easy and are very expressive. They are available on all
platforms. I've been using TeX/Latex since the 80s to generate very
complex reports from c/c++ code. trying to do it from the code would
have been just to expensive. latex handles hyphenation, heading,
footings  floats, end notes, margin notes, text flow, etc etc. Doing
that using a programming api is very difficult. The code is very
stable and well documented.

I template out the page design and then just generate the TeX markup.
>From a .tex file its easy  to generate to span off the Tex interpreter
to generate PDF, postscript, HTML or what ever else you might need.

Remember all the python documentation before 2.6  was done in Latex,
so you can just look at the Python build to get some good ideas.


http://www.latex-project.org/








On Apr 21, 10:13 am, cootetom  wrote:
> You are right though. In my experience, using Pisa to do HTML to PDF
> conversion isn't pixel perfect.
>
> On Apr 21, 2:29 pm, derek  wrote:
>
>
>
> > On Apr 21, 1:14 pm, Alessandro Ronchi 
> > wrote:
>
> > > 2010/4/20 cootetom :
>
> > > > django and and pisa are great for creating PDF's because you can
> > > > combine django's templating engine with pisa's HTML to PDF conversion.
> > > > Pisa uses the reportlab tool kit.http://pypi.python.org/pypi/pisa/
>
> > > I think I can't use PISA because I need the pdf to be precise as
> > > pixels, and HTML is not the best solution for that.
>
> > > --
> > > Alessandro Ronchi
>
> > >http://www.soasi.com
> > > SOASI - Sviluppo Software e Sistemi Open Source
>
> > > Hobby & Giochi, l'e-commerce del 
> > > divertimentohttp://hobbygiochi.comhttp://www.facebook.com/pages/Forli/Hobby-Gioch...
>
> > Well, ReportLab will give you sub-mm control; even more accurate than
> > pixels (which are really only useful for on-screen 
> > displays):http://www.reportlab.com/software/opensource/rl-toolkit/faq/#2.1.2
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/django-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Working for a startup.

[thanos]

Until 1998 NYSE only allowed applications developed in C++ on their
trading floor. Both the client and server side technology was HP Unix.
They switched in around 2004 to Linux. They use a custom built
database and messaging system. In 1998 we launched a new trading
system based on Python. It did not fail but was a great success and is
still in use to day. The speed in which Python apps could be developed
and deployed impressed management and it became the second De Facto
development language of NYSE.
 In 2001 there was a very large scale Java project that failed and had
to be pulled off the floor, but this is was not due to the language
but due to the poor design (The project had been outsourced to a big
company). A year later the re-factored (in house) system was launched
with success.

NYSE floor is linux based, so C# doesn't come into the picture. Why
unix and windows: Four really good reasons:

1. It scales.
2. It's a lot cheaper.
3. Its much cleaner and easier to program for. Also its easier to
deploy on. (Remember all the DLL issues C# developer  have).
5. Power and Energy- ??? - Yes the NYSE floor has a limit to its
energy supply. With Linux you just  get a lot more bang per Wat.

The standard of software developer for the NYSE floor are very high.
In the years I worked their I remember only a handful of program
failure in production and they were all fixed in minutes. Downtime of
a system was just to accepted, and NYSE's operation was more stringent
than NASA or the USAF. In fact i remember generals coming on tours to
see how we did it.
Of course all this control and extensive design documentation  was
stiffing, but using languages like Python helped.

Finally just recently SEC announced that they might require regulatory
files to be submitted in Python and not in legal English.
Their rational is that Python is less ambigious and the English and
more clean, readable  and structured than XML. (http://www.sec.gov/
rules/proposed/2010/33-9117.pdf).

Thanos










On Apr 22, 1:40 am, "Joe Goldthwaite"  wrote:
> I've been working for a startup for a month or so.  The main guy is
> enthusiastic about technology.  He probably runs more applications on his
> computer than anyone I know.  He also switched from a Windows background to
> a Mac notebook a year ago.  He doesn't know a lot about web development or
> how it all works.  That's what I've been helping with.
>
> He's interviewed a number of companies trying to find a good one to develop
> version1 of the application.  I've been pushing him towards
> Linux/Python/Django and thought I had him convinced.  The company he's
> thinking about going with is a .net shop.  They say they can do anything but
> the bulk of their developers are .net.  Still, he was thinking about going
> with this company but still having them develop in Django.
>
> This morning we had a conference call with one of the references the
> development company provided.  It turns out they're a startup that's working
> on a different application in the same industry as us so there was an
> affinity there.  The guy mentioned that they originally started with a
> different company using Ruby on Rails.  He said that after the other company
> got bogged down and wasn't making progress, he switched to the new company
> that we're considering.  He said that they got the project back on track and
> organized but ended up running into lots of problems with Rails.  They
> talked it over and decided to rewrite everything in .net.  Now he's happy
> with the progress and feels like .net is a superior platform and is allowing
> him to develop his program at a lower total cost.
>
> This has my boss worried.  Most of the people he's been talking to are from
> large companies.  One of them was even a high end Microsoft person.  I think
> he's starting to think that the Microsoft technology stack might be the
> safer choice. He wants to get to the point of having millions of users and
> is worried that Django wont cut it. I'm also worried that if he does stick
> with Django, the development company might blame any setbacks on Django and
> push for a rewrite in .net.
>
> I'd like examples of large systems written in Django or other open source
> frameworks.  I'd also like stories about companies who tried .net switched
> over to open source.  It seems like there was a big Microsoft project on
> some stock exchange program that failed but I can't find any references to
> it.  I tried searching Google but it seems like it doesn't differentiate
> ".net" with the internet so trying to find .net failures only turned up
> stories about internet downtime.
>
> Any help would be greatly appreciated.
>
> --
> You received this message because 

Re: Decode a JSON object?

[thanos]

Thomas

Try:

>>> import simplejson
>>> simplejson.loads('{"totalspace":243862.672,"freespace":94053.564}')
{'freespace': 94053.5638, 'totalspace': 243862.671}
>>>

Although cjson is a lot faster, you should have simplejson if you are
using Django.

Thanos


On May 3, 12:33 pm, Thomas Allen  wrote:
> How can I parse a simple JSON object in Django?
>
> I would like to parse the response: {"totalspace":
> 243862.672,"freespace":94053.564}
>
> django.core.serializers.deserialize('json', packet) seems to have
> trouble parsing such a simple string.
>
> Thomas
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Python Database Framework

[thanos]

Elixir gets my vote.

I now always use http://www.sqlalchemy.org/ for any db work. But
Elixir is cool. I used it with
http://elixir.ematia.de/apidocs07/elixir.ext.versioned.html
and it saved my bacon !



On May 14, 10:51 am, Nabil Servais  wrote:
> Hello
> Le 14 mai 2010 à 16:47, Ozgur Yılmaz a écrit :
>
> > Hi everybody,
>
> > Do anyone knows a Python Database framework, which resembles Django's 
> > approach to database programming?
>
> Yes, Elixir :http://elixir.ematia.de/trac/wiki.
>
> > I need it to use outside of Django, for other Python scripts.
>
> > Best regards,
>
> > E.Ozgur Yilmaz
> > Lead Technical Director
> >www.ozgurfx.com
>
> > --
> > You received this message because you are subscribed to the Google Groups 
> > "Django users" group.
> > To post to this group, send email to django-us...@googlegroups.com.
> > To unsubscribe from this group, send email to 
> > django-users+unsubscr...@googlegroups.com.
> > For more options, visit this group 
> > athttp://groups.google.com/group/django-users?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django users" group.
> To post to this group, send email to django-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group 
> athttp://groups.google.com/group/django-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Django performance vs others

[thanos]

For real performance I suggest taking a big breath and looking at
ChicagoBoss. It's a move I've done that has saved me a lot of money on AWS.

On Wed, Oct 10, 2012 at 9:07 AM, Amirouche Boubekki <
amirouche.boube...@gmail.com> wrote:

> 2012/10/10 Moonlight 
>
>> I think this  one
>> explains how it works.
>
>
> A quick read make me think it's a bit complex.
>
> Check out the routing benchmark too
> http://mindref.blogspot.fr/2012/10/python-web-routing-benchmark.html
>
> If this is legit tests, they are impressive.
>
> Would you mind at least using the django-users@googlegroups.com ML...
>
> Thanks,
>
> Amirouche
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django-users@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: django templates and tex/latex

[thanos]

On Thu, Mar 14, 2013 at 9:58 AM, Bill Freeman  wrote:

> Note that you don't have to use the Django template engine to render
> things.
>
> For one thing, if you found a less problematic templating engine, using it
> as
> well doesn't prevent you from using Django's template engine for other
> pages.
>
> You could also roll your own by coming up with, for example, a TeX friendly
> syntax for variable substitution (by which I mean, doesn't collide with
> syntax
> that you will be using - best would be coming up with something that is
> actually illegal in TeX), and process things with a big regex or had coded
> parser..  You could parse this directly into the output, or this could be a
> loading stage that converts the file to a Django template (or engine of
> your
> choice) file, where the TeX/LaTeX stuff that looks like template specials
> is
> converted to a quoted structure that renders as the original.
>
> A final approach would be to use TeX definitions to perform your variable
> substitutions, prepending a stanza that makes the definitions at render
> time.  (Though making this work where you want template loops is much
> harder.)
>
>
> On Wed, Mar 13, 2013 at 2:15 PM, Ken  wrote:
>
>> Actually, that's exactly what I'm trying to do.  I have database tables
>> whose
>> columns contain TeX fragments.  I am trying to generate various file
>> outputs
>> from database searches - TeX, pdf, etc.
>>
>> I could construct the tex file on the fly with python strings as you
>> suggest but
>> that's rather cumbersome.  It's simpler with a template.
>>
>> I've also considered using the xml.element api to construct my own
>> template
>> but that would set me back several weeks.  I'm trying to finish. :/
>>
>> I have something that almost resembles a template.  It is not the most
>> flexible
>> or elegant but it works.  I have a single replacement string between
>> \begin{document} and \end{document} and I simply replace this one token
>> with a
>> string.replace()
>>
>>
>>
>>
>> On Wednesday, March 13, 2013 11:39:18 AM UTC-5, Drew Ferguson wrote:
>>
>>> On Wed, 13 Mar 2013 07:01:45 -0700 (PDT)
>>> Ken  wrote:
>>>
>>> > I would like to write a tex/latex file with django tags in it.  Load
>>> it
>>> > with the template loader and render it with a context.  The problem is
>>> > that my tex/latex file has quite a few '{%' in them.   They are
>>> > conventional in TeX for writing readable macros and are used to escape
>>> > the newline.  I could try rewriting them but before I do, I thought
>>> I'd
>>> > ask if this is even worth tackling.  TeX uses curly braces as grouping
>>> > and percent signs as a comment.
>>> >
>>> > The TeX file would look rather confusing but the more important
>>> question
>>> > is what the loader does when it sees braces and brace comments.  Is it
>>> > possible to reassign the block and variable tags?
>>> >
>>> > Thanks
>>> >
>>> >
>>>
>>> I have done this in the past (with Plone rather than django) by calling
>>> a
>>> python wrapper script to generate a file which latex can process;
>>> essentially construct the Latex source as python strings
>>> and have the script assemble the components you want. The objective was
>>> to
>>> generate a PDF file by Latex with variables fed from SQL & a web page.
>>> But
>>> this may not be what you are interested in.
>>>
>>> Are you trying to render a Latex source document as a web page?
>>>
>>> --
>>> Drew
>>>
>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Django users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-users@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-users?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
&

Re: fastapi skills

[thanos]

i konw it well but will only work remote.

Thanos

On Thu, Feb 18, 2021 at 12:17 PM suresh natarajan 
wrote:

> does anyone know fastapi well ? there is an opening with a startup.
> please msg
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/CALxxTe32vongg6%2BWYH0TT%2B_TPjS%3DRvSc9%3DDp_jHLUK%3D0x0rjKQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-users/CALxxTe32vongg6%2BWYH0TT%2B_TPjS%3DRvSc9%3DDp_jHLUK%3D0x0rjKQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CAKFaRRUUkpZmPJ0XmgCJXTJBWSkf%3Dxb2yQxaaQRy%3DH8fdkuwAg%40mail.gmail.com.

Re: Anyone want to take over maintaining Instant Django?

[Thanos Vassilakis]

I would be quite interested. I use instant django so might work out  
for me.


Thanos

On Jul 14, 2010, at 2:48 PM, cjl  wrote:


Thanks for the replies.

To be clear, I'm not looking for hosting, I'm looking for someone to
take it completely off my hands. This means doing whatever you want
with it, but I would hope it would at least mean keeping it up to
date. This only takes and hour or two whenever a new version of Django
is released, but I no longer have any interest in doing it.

@Andy - The "build" script is a windows '.bat' file. Basically, you
put all the components that comprise Instant Django into a folder, run
the '.bat' file, and in builds the Instant Django download. This got a
little harder to do after the Python dev's broke the MSI installer.
Prior to Python 2.6 you could install Python from the command line
into a self-contained temporary directory, without touching the
underlying host system. This is no longer possible, and I couldn't
convince anyone that mattered that it was a problem.

Anyway, I'll email the people who have expressed an interest directly
with further details.

-CJL

--
You received this message because you are subscribed to the Google  
Groups "Django users" group.

To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com 
.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en 
.




--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To post to this group, send email to django-us...@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Caching and Vary: Cookie header

[Thanos Diacakis]

I have spent several hours researching this, and I'm stumped, so
perhaps I could tap into some collective wisdom:

I have a website whose page template includes a "Sign-In or Register"
link at the top.   This is shown on pretty much every page.  If the
user is already signed in, this is replaced with "Hello _username_ |
My Account".  This is done by passing request.user to the template and
checking is_authenticated().

The problem is, that as soon as I touch request.user, the Vary: Cookie
header is tagged on, and that destroys caching.  Every anonymous user
gets their own version of the page cached, which is almost pointless.

I'm trying to figure out a way that I can fully utilize the cache for
anonymous users (i.e. one instance of the rendered page in the cache)
and I'm OK for caching with Vary: Cookie for logged in users.

So, is there a way to figure out if the user is logged in or not,
without tripping the Vary: Cookie header, or is there a better way of
doing this altogether?

BTW, it seems that it is significantly faster (15x CPU time) to cache
a whole view, that to cache a fragment, so I'd like to avoid that if
possible.

Thanks

Thanos

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Re: Caching and Vary: Cookie header

[Thanos Diacakis]

Doh - thanks.

I guess to keep the structure simple, I can set the page up for
anonymous users, fully cached and all, and then use AJAX to figure out
if they are logged in and tweak my header.

Thanos

On Mar 7, 9:43 am, Tom Evans  wrote:
> On Mon, Mar 7, 2011 at 4:23 PM, Thanos Diacakis  wrote:
> > I have spent several hours researching this, and I'm stumped, so
> > perhaps I could tap into some collective wisdom:
>
> > I have a website whose page template includes a "Sign-In or Register"
> > link at the top.   This is shown on pretty much every page.  If the
> > user is already signed in, this is replaced with "Hello _username_ |
> > My Account".  This is done by passing request.user to the template and
> > checking is_authenticated().
>
> > The problem is, that as soon as I touch request.user, the Vary: Cookie
> > header is tagged on, and that destroys caching.  Every anonymous user
> > gets their own version of the page cached, which is almost pointless.
>
> > I'm trying to figure out a way that I can fully utilize the cache for
> > anonymous users (i.e. one instance of the rendered page in the cache)
> > and I'm OK for caching with Vary: Cookie for logged in users.
>
> > So, is there a way to figure out if the user is logged in or not,
> > without tripping the Vary: Cookie header, or is there a better way of
> > doing this altogether?
>
> > BTW, it seems that it is significantly faster (15x CPU time) to cache
> > a whole view, that to cache a fragment, so I'd like to avoid that if
> > possible.
>
> > Thanks
>
> > Thanos
>
> If it is the same URL being visited by the anonymous users and the
> authenticated users, then how is the upstream cache to determine which
> version to send?
>
> Or put it like this, you want to have anonymous users viewing the page
> to not generate a 'Vary: cookie' header. This leads to the following
> situation:
>
> Anonymous user visits /contact-us/
> Django generates the anonymous version of the page and delivers it,
> with appropriate caching headers.
> Upstream cache stores anonymous version in cache
> Logged in user visits /contact-us/
> Upstream cache intercepts the request, looks in its cache for an
> appropriate hit, and delivers the anonymous page.
>
> The basic thing here is that you cannot have your cake and eat it as
> well. Either the page contains dynamic, user-specific content, in
> which case vary on cookie, or it does not, in which case cache it.
>
> If you want to deliver truly cache-able anonymous content, deliver it
> on a different URL to your dynamic content.
>
> Cheers
>
> Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to 
django-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-users?hl=en.

Raw query with "where-in" clause

[Thanos Diacakis]

Hey all,

I have the following query:

cursor.execute( "select * from MyTable where Foo=%s or Bar in %s", 
[some_id, some_array] )

This seems to work fine if len(some_array) > 1 but it blows up if there are 
zero or one items in the array (invalid SQL syntax).

Is there a better way to do this?  The Python DB-API docs seems to be very 
light.  Could not find anything relevant.

Thanks

Thanos


-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.