When I have arrays of 100,000+ the performance of lists gets unacceptable in standard c python. That's why arrays exist in python!
After doing research and testing, I have decided to rebuild the project in java with embedded jython. For some bizarre reason jython performance increases with large datasets in memory where standard c python plunges (From what I hear its that the GC strategy doesn't scale). I also can't use django at having to read and convert everything in memory etc is going to be too bad a hit. Shame, as I like django but it won't do for this project as there are many assumptions that don't fit my requirements - I would have to hack it to death to get what I want and that is a waste of time. Everything is going to have to be stream based (conversions happening on the fly) and use efficient use of memory internally for arrays etc. I now will be using jetty, servlets, jython as I can optimise everything easily for huge data sets. There is a maximum permissible time lag for each request. On 25 Aug, 21:38, Peter Bengtsson <pete...@gmail.com> wrote: > what's wrong with turning it into a list? If you gzip it it won't be > that big. > > On Aug 25, 5:16 pm, John Baker <j...@jdiligence.com> wrote: > > > I need to json serialize some very large objects which include large > > arrays. How can I do this in django? The arrays will be very big and > > heavily processed before so need to use efficient array storage. > > > Testing with arrays I get.. > > > TypeError at /zeros/ > > > array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) is not JSON > > serializable > > > What would you recommend I do to support arrays? > > > Thanks in advance, > > John --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
