Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread Lie Ryan
On 23/06/14 19:05, smur...@gmail.com wrote: On Monday, June 23, 2014 5:54:38 PM UTC+2, Lie Ryan wrote: If you don't want each thread to have their own copy of the object, Don't use thread-scoped session. Use explicit scope instead. How would that work when multiple threads traverse the in-me

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread smurfix
On Monday, June 23, 2014 5:54:38 PM UTC+2, Lie Ryan wrote: > If you don't want each thread to have their own copy of the object, > > Don't use thread-scoped session. Use explicit scope instead. How would that work when multiple threads traverse the in-memory object structure and cause relation

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread smurfix
memcache (or redis or ...) would be an option. However, I'm not going to go through the network plus deserialization for every object, that'd be too slow - thus I'd still need a local cache - which needs to be invalidated. -- https://mail.python.org/mailman/listinfo/python-list

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread Matthias Urlichs
Hi, William Ray Wing: > Are you sure it won’t fit in memory? Default server memory configs these > days tend to start at 128 Gig, and scale to 256 or 384 Gig. > I am not going to buy a new server. I can justify writing a lot of custom code for that kind of money. Besides, the time to actually

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread Lie Ryan
On 22/06/14 10:46, smur...@gmail.com wrote: I've been doing this with a "classic" session-based SQLAlchemy ORM, approach, but that ends up way too slow and memory intense, as each thread gets its own copy of every object it needs. I don't want that. If you don't want each thread to have thei

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread Roy Smith
In article , William Ray Wing wrote: > On Jun 23, 2014, at 12:26 AM, smur...@gmail.com wrote: > > > On Sunday, June 22, 2014 3:49:53 PM UTC+2, Roy Smith wrote: > > > >> Can you give us some more quantitative idea of your requirements? How > >> many objects? How much total data is being stor

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-23 Thread William Ray Wing
On Jun 23, 2014, at 12:26 AM, smur...@gmail.com wrote: > On Sunday, June 22, 2014 3:49:53 PM UTC+2, Roy Smith wrote: > >> Can you give us some more quantitative idea of your requirements? How >> many objects? How much total data is being stored? How many queries >> per second, and what is th

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-22 Thread smurfix
On Sunday, June 22, 2014 3:49:53 PM UTC+2, Roy Smith wrote: > Can you give us some more quantitative idea of your requirements? How > many objects? How much total data is being stored? How many queries > per second, and what is the acceptable latency for a query? Not yet, A whole lot, More t

Re: Python ORM library for distributed mostly-read-only objects?

2014-06-22 Thread Roy Smith
In article <85659fdd-511b-4aea-9c4b-17a4bbb88...@googlegroups.com>, smur...@gmail.com wrote: > My problem: I have a large database of interconnected objects which I need to > process with a combination of short- and long-lived workers. These objects > are mostly read-only (i.e. any of them can

Python ORM library for distributed mostly-read-only objects?

2014-06-22 Thread smurfix
My problem: I have a large database of interconnected objects which I need to process with a combination of short- and long-lived workers. These objects are mostly read-only (i.e. any of them can be changed/marked-as-deleted, but that happens infrequently). The workers may or may not be within o