Re: [ZODB-Dev] storage record_iternext() question

2014-12-09 Thread Jim Fulton
rtainly reflects my intent when I added that. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Iterate through all objects from ZODB

2014-09-23 Thread Jim Fulton
with each object, you'll probably want to call cacheGC on the connection to free unneeded memory. ... > We are using RelStorage (MySQL) and ZEO (4 Dev) I don't know if RelStorage implements IStorageCurrentRecordIteration. Jim -- Jim Fulton http:/

Re: [ZODB-Dev] Race condition in RelStorage history-free pack?

2014-07-18 Thread Jim Fulton
; growth of the database is an issue, and they pack more frequently. > > —Ian > > ___ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev > -- Jim Fulton h

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
7395884631 (2014-07-01 00:10:04.686741) > Transactions: 1275366 So, you can pass a time.time in future, or just wait a few hours :) > Thanks for the valuable help of everybody, I learned a lot. You're welcome. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
On Mon, Jun 30, 2014 at 10:44 AM, Alessandro Pisa wrote: > On 30 June 2014 16:02, Alessandro Pisa wrote: >> On 30 June 2014 15:42, Jim Fulton wrote: >>> On Mon, Jun 30, 2014 at 8:24 AM, Alessandro Pisa >>> >>> I suggest using the file-store iterator to lo

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
On Mon, Jun 30, 2014 at 8:24 AM, Alessandro Pisa wrote: > On 30 June 2014 12:43, Jim Fulton wrote: >> On Mon, Jun 30, 2014 at 3:30 AM, Alessandro Pisa >> wrote: >>> Hello everybody :) >> >> As the comment suggests, if you continued packing, the new file

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
ode in a product > before using it? Sorry the documentation isn't thorough enough. > It's posts like this that make me not want to try to help others anymore. If I can keep an unqualified helper from causing harm, I've accomplished something. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
On Mon, Jun 30, 2014 at 7:21 AM, Simone Deponti wrote: > Hi Jim, > > On Mon, Jun 30, 2014 at 12:43 PM, Jim Fulton wrote: > >> As the comment suggests, if you continued packing, the new file >> would be as large as the old one, because no records would be >> r

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
ords (as of the pack time). This is properly called "packing". 2. Removing objects that are no longer reachable (from any records, current or non-current) from the root object. If packing doesn't remove any objects, neither will packing. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
this version. If I read the code correctly, > this would force (at least trying) a pack in any case, assuming > pack-gc=true. Really? You're suggesting modifying the code. > > I'm not guaranteeing anything and I'm just a zodb user though ;) And yet you suggest modifying code

Re: [ZODB-Dev] Pack problem

2014-06-30 Thread Jim Fulton
ime before, or b) None of the objects written up to the pack time have been written after the pack time and this there are no old records to be removed. > What I am trying to do: > - comment the Data.fs.pack removing. Or, you could leave off pack-keep-old=false setting. Jim -- Jim Fulton h

Re: [ZODB-Dev] Massive LockError: Couldn't lock....check_size.lock messages

2014-04-15 Thread Jim Fulton
Xkr9VwEGWKWQaODS26n2IachD5fs8AC5 > a38Jo3EcR3cTIYiZady62M7muJqkJO4= > =UF30 > -END PGP SIGNATURE- > _______ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.z

[ZODB-Dev] Sprucing up zodb.org and pull request

2014-02-22 Thread Jim Fulton
n pull request. BTW, I'd like non-trivial changes to be made via pull request. This last week, I finally got around to some overdue work on the tutorial. It would be great is someone would review my changes: https://github.com/zopefoundation/zodbdocs/pull/1 Jim -- Jim Fulton http://www.

[ZODB-Dev] New ZODB Mailing list

2014-02-22 Thread Jim Fulton
I'm 91% sure one of *us* must have set it up at some point.) In a week or so, I'll disable posts to this list, with a notice to move to the google group. In the mean time, I'll post to both places. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___

Re: [ZODB-Dev] Does ZODB pipeline load requests?

2014-02-20 Thread Jim Fulton
bject()) Well, this would simply be another custom iterator wrapper. for ob in main_iterator(myiter): ... Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Does ZODB pipeline load requests?

2014-02-19 Thread Jim Fulton
On Wed, Feb 19, 2014 at 9:57 AM, Dylan Jay wrote: > On 19 Feb 2014, at 10:44 pm, Jim Fulton wrote: ... > yeah I figured it might be the case thats its hard to predict. In this case > it's catalog indexing so I was wondering if something could be done with > __iter__ o

Re: [ZODB-Dev] Does ZODB pipeline load requests?

2014-02-19 Thread Jim Fulton
under-motivated to address the first 2 problems. We really should address problems 1 and 2 to make it possible for people to experiment with approaches to problem 3. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Did someone create a zodb-dev google group?

2014-02-19 Thread Jim Fulton
I just tried creating one, but it was already taken and is not public. :( Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https

Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit

2014-02-05 Thread Jim Fulton
On Wed, Feb 5, 2014 at 9:23 AM, Godefroid Chapelle wrote: > Le 05/02/14 13:25, Jim Fulton a écrit : > >>>>> Acquisition is added as a test dependency. Any hint how to replicate >>>>> >> >the bug without acquisition is welcome. >>>> >

Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit

2014-02-05 Thread Jim Fulton
ior anyway, and we > *definitely* don't want broken sdists on PyPI.) I couldn't agree more. One of the advantages of moving to git was circumventing setuptool's misguided magic. I've no idea what Tres was referring to wrt "instsall-from-a-github-generated-tarball", but I use Manifest.in files in all my modern projects. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Unpickler.noload, zc.zodbdgc and multi-database reference bug (IndexError)?

2014-01-30 Thread Jim Fulton
handling of references. I wish I'd stored them outside the pickle so they could be analyzed without unpicking (or at least without unpickling the application data). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Unpickler.noload, zc.zodbdgc and multi-database reference bug (IndexError)?

2014-01-30 Thread Jim Fulton
x the IndexError by checking the size of the list first, we miss all > the cross-db references, meaning that a GC is going to be too aggressive. But > using `load` is slower and requires access to all of the classes referenced. > If anyone has run into this before or has oth

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
or what "fast enough" is. We use it on a database with ~200 million objects. > - so I skipped this alternative. Good luck. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
On Mon, Nov 18, 2013 at 8:43 AM, Jan-Wijbrand Kolman wrote: > On 11/18/13 12:19 PM, Jim Fulton wrote: >> >> On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein >> wrote: >>> >>> I started a new packing script for Relstorage (history free, postgresql). &g

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
t's been in production for a few years now at ZC. Not sure if it would need to be updated for relstorage. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-De

Re: [ZODB-Dev] Relstorage and over growing database.

2013-11-11 Thread Jim Fulton
formation in the relational database? > > Any hints or comments are welcome. Definately look at zodbdgc. It doesn't traverse the graph. It essentially does reference counting and is able to iterate over the database, which for FileStorage, is relatively quick. Jim -- Jim Fulton http://ww

Re: [ZODB-Dev] ZODB.FileStorage.format: TxnHeader cannot handle Unicode 'descr'

2013-10-07 Thread Jim Fulton
ical raisins", but if we mean "bytes", we should say so. > > Note that the implementation's use of an unadorned string literal to join > the values means that in Py3k, it really *is* text, and not bytes. If we > want the application to do the encoding, then we shou

Re: [ZODB-Dev] ZODB.FileStorage.format: TxnHeader cannot handle Unicode 'descr'

2013-10-05 Thread Jim Fulton
Alternatively, Pyramid could store the path in transaction extended info, which accepts any picklable type. Of course, we could revisit this. If we did, I'd deprecate the transaction user and description attributes and only support meta data via the extended info mechanism, which I'd

[ZODB-Dev] Backward-incompatible change in TimeStamp repr in ZODB 4.

2013-09-29 Thread Jim Fulton
x27;t pin versions. If someone wants to mix ZODB3 and ZODB 4, they can explicitly require ZODB3 3.11a3. Thoughts? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] https://github.com/zopefoundation/zc.beforestorage

2013-09-27 Thread Jim Fulton
imited sequence of transactions to provide a limited form of time travel and, most importantly, to provide a temporary snapshot of a database that's being written, mainly for use with DemoStorage. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton __

[ZODB-Dev] ZODB 4.0.0 and ZEO 4.0.0 Released

2013-09-18 Thread Jim Fulton
Hopefully, we can increase our development tempo a bit now that we have this base to build on. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev

Re: [ZODB-Dev] polite advice request

2013-08-18 Thread Jim Fulton
't know what he meant by prescription. :) Does it matter? Might it as easily be foos and bars? Christian, Are you saying that you might need to access items from an old database that aren't in the current snapshot? Jim > > On 08/18/2013 06:34 PM, Jim Fulton wrote: >>

Re: [ZODB-Dev] polite advice request

2013-08-18 Thread Jim Fulton
pdates? Or schema updates too? > We need to cope with millions of recipes that come from certain dates > and therefore need to inquire different versions of the database. I don't understand this. What's a "recipe"? Why do you need to consider old versions of the

Re: [ZODB-Dev] polite advice request

2013-08-18 Thread Jim Fulton
t seem very large in terms of storage, at least not for ZODB. Beyond that there are lots of dimensions of scale that ZODB doesn't handle well (e.g. large transaction rates, very high availability). It's really hard to make specific recommendations without knowing more about the problem.

Re: [ZODB-Dev] when to use connection close/sync

2013-08-16 Thread Jim Fulton
you answer. > > One bit from my original question though: if I have a single threaded > client, why would I ever want to close the connection? There might not be any reason. It depends on your application. We typically do so in web apps, because it makes the architecture c

Re: [ZODB-Dev] when to use connection close/sync

2013-08-16 Thread Jim Fulton
connections)?" The answer to that question is: To see database changes, you need to start a new transaction. You always see data as of the time the current transaction started. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more informa

Re: [ZODB-Dev] RFC: database ids

2013-08-15 Thread Jim Fulton
On Wed, Aug 14, 2013 at 6:49 PM, Vincent Pelletier wrote: > Le 15 août 2013 00:09, "Jim Fulton" a écrit : > > Comments? > > Please make database ID reachable where _p_oid is reachable (maybe on > _p_jar, I don't mind a few attribute lookup levels/trivial calls)

Re: [ZODB-Dev] RFC: database ids

2013-08-14 Thread Jim Fulton
On Wed, Aug 14, 2013 at 6:08 PM, Jim Fulton wrote: > When using a database server (ZEO, relstorage), you can make a > configuration error that causes you to connect to the wrong database. > This can be especially painful in a situation where you get > disconnected from the server an

[ZODB-Dev] RFC: database ids

2013-08-14 Thread Jim Fulton
be configured to discard cache if it (re)connects to a server with a last transaction lower than the last one the client has seen as long as the database ID matches. - ZRS secondaries will also check database ids when (re)connecting to primaries. Comments? Jim -- Jim Fulton http://www.

Re: [ZODB-Dev] Ackward PersistentList read Performance

2013-08-13 Thread Jim Fulton
ficiency of update, depending on access patterns. > or am I missing something? Possibly > I am using ZODB3-3.10.5. The whole setup (incl. results) is at > https://github.com/jhb/zodbtime tl;dr Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton __

Re: [ZODB-Dev] BTrees package problems

2013-07-24 Thread Jim Fulton
mplementation isn't informed by user provided attributes. It would be better if it was. For example. I'd like bucket and internal node sizes to be controllable via instance attributes (typically defined in a subclass. Jim -- Jim Fulton http://

Re: [ZODB-Dev] BTrees package problems

2013-07-23 Thread Jim Fulton
ng that depends on writing this attribute. I verified this by adding a fix and running the tests (in 3.10). For what you're trying to do, I suspect you want to fork BTrees, or start over. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For

Re: [ZODB-Dev] BTrees and ZODB simplicity

2013-07-22 Thread Jim Fulton
On Mon, Jul 22, 2013 at 8:11 AM, Christian Tismer wrote: > On 22.07.13 13:13, Jim Fulton wrote: >> >> On Sat, Jul 20, 2013 at 11:43 PM, Christian Tismer >> wrote: >>> >>> Third rant, dear Zope-Friends (and I mean it as friends!). >>> >>

Re: [ZODB-Dev] make ZODB as small and compact as expected

2013-07-22 Thread Jim Fulton
distribution for a long time until recently. It makes sense for it to be optional, as it's of no interest to people who use relstorage. More importantly, it's more complicated than any other part of ZODB and it makes a lot of sense for ZODB development to be unburdened of it. Ji

Re: [ZODB-Dev] zc.zlibstorage missing from zodb package

2013-07-22 Thread Jim Fulton
grate". I suggest, if you want to make it simpler is to provide new ZConfig tags or Python factories that make configuration simpler the way you'd like, but that do so by assembling layers under the hood. ... > Meant in a friendly, collaborative sense -- Chris Much appreciate

Re: [ZODB-Dev] make ZODB as small and compact as expected

2013-07-22 Thread Jim Fulton
pretty happy with the layered storage architecture. - With modern package installation tools like buildout and pip, having lots of dependencies shouldn't be a problem. ZODB uses lots of packages that have uses outside of ZODB. I consider this a strength, not a weakness. Honestly, I h

Re: [ZODB-Dev] BTrees and ZODB simplicity

2013-07-22 Thread Jim Fulton
; > - or a real stand-alone package with some way of adding persistence as >an option. I don't agree that because a package depends on ZODB it should be in ZODB. There are lots of packages that depend on ZODB. I agree with your sentiments about namespace pollution. You and I

Re: [ZODB-Dev] BTrees package problems

2013-07-22 Thread Jim Fulton
't understand this. > > * this was my second little rant about ZODB. Not finished as it seems. > > please, see this again as my kraut way of showing interest in improving > very good things. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects

2013-06-07 Thread Jim Fulton
> >> Also, requiring protocol 3 for Python 3 sounds correct as well. > > How about also bumping the magic number of Data.fs files so ZODB on > Python 2 would reject it early? +1 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___

Re: [ZODB-Dev] Replication for the Zope Object Database

2013-05-28 Thread Jim Fulton
On Mon, May 27, 2013 at 8:42 PM, Carlos de la Guardia wrote: > Hey Jim, > > great news! > > pypi link is wrong, should be: https://pypi.python.org/pypi/zc.zrs Right you are. Thanks. > > Year for changes in pypi page is shown as 2015. Oops. Fixed on PyPI and and future re

[ZODB-Dev] Replication for the Zope Object Database

2013-05-27 Thread Jim Fulton
14276/Zope-Replication-Services and: https://pypi.python.org/zc.zrs Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailma

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-05-10 Thread Jim Fulton
ackward-compatibility >at the data level with Python2. A future version of ZODB may add >such support. > >Applications which need migrate data from Python2 to Python3 should >plan to script this migration using separte databases, e.g. via a > &qu

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Mon, Apr 29, 2013 at 1:15 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 12:44 PM, Jim Fulton wrote: >> Yes. There are lots of features I'd like to add to ZODB. I tend to >> work on them when I have time (infrequently)

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Mon, Apr 29, 2013 at 12:25 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 11:00 AM, Jim Fulton wrote: >> Let's keep master stable. Maybe someone will want to add features >> before the Python 3 support is stable. I don&

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Mon, Apr 29, 2013 at 10:54 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 10:51 AM, Jim Fulton wrote: > >> Right. As I suggested, let's get to a point where we can get a stable >> ZODB 4.0 release for Python 2. As

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Mon, Apr 29, 2013 at 10:20 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 04/29/2013 09:48 AM, Jim Fulton wrote: >> On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter >> wrote: >>> On Sunday, April 28, 2013 07:23:12 PM Jim Fult

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Mon, Apr 29, 2013 at 10:24 AM, Stephan Richter wrote: > On Monday, April 29, 2013 09:48:05 AM Jim Fulton wrote: >> I'd like there to a stable 4.0 release **soon** >> that doesn't use zodbpickle for Python 2. > > I would like to agree. But on the other hand, t

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-29 Thread Jim Fulton
On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter wrote: > On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: >> Can ZODB 4 be used now without zodbpickle? > > No, unfortunately for Py2 we need the custom cPickle and for Py3 `noload()` > support (as Tres mentioned). This

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-28 Thread Jim Fulton
en up to an add-on package. This way people can > experiment with different approaches and we do not have to nail the conversion > problem with ZODB 4.0. I've lost track of where we are with ZODB 4. Can ZODB 4 be used now without zodbpickle? Jim -- Jim Fulton http://www.linkedin.com/in/jim

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-28 Thread Jim Fulton
convert", for instance (the downtime required to do the conversion > would be prohibitive, I believe). > >> But which strategy would be useful for a large Plone site for example? >> I think we should focus on that and provide one good way to do it. > > Plone has hi

Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility

2013-04-28 Thread Jim Fulton
of a storage (supporting convert_storages_). > > - - ``zodbpickle`` should provide a new ``binary`` type which Python2 > applications can begin using to signal that attributes should be > unpickled in Py3k as ``bytes``. See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - - ``zodbpickle`` should provide a pickler/unpickler for use by > Python2 clients who operate against converted storages > (replace_py2_cpickle_). See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - - ``zodbpickle`` should provide a pickler/unpickler for use by > Py3k clients who operate against unconverted storages > (replace_py3k_pickle_). See: > https://github.com/zopefoundation/zodbpickle > > - - ``zodbpickle`` might need to provide a wrapper storage supporting > straddle_no_convert_. > > > Comments? Thanks for taking the time to work all of this out. It sounds rather complex. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Towards ZODB on Python 3

2013-03-10 Thread Jim Fulton
On Sun, Mar 10, 2013 at 12:13 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 03/10/2013 11:55 AM, Jim Fulton wrote: >> On Sun, Mar 10, 2013 at 11:25 AM, Tres Seaver >> wrote: >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 &g

Re: [ZODB-Dev] Towards ZODB on Python 3

2013-03-10 Thread Jim Fulton
On Sun, Mar 10, 2013 at 11:25 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 03/10/2013 09:19 AM, Jim Fulton wrote: ... >> I think the fix is pretty straightforward. >> >> In the default __setstate__ provided by Persistent, and

Re: [ZODB-Dev] Towards ZODB on Python 3

2013-03-10 Thread Jim Fulton
, and when loading non-persistent instances: - On Python 2, ASCII encode unicode attribute names. - On Python 3, ASCII decode byte attribute names. The same transformation is necessary when looking up global names. This will cover the vast majority of cases where the default __setstate__ is u

Re: [ZODB-Dev] Cache warm up time

2013-03-09 Thread Jim Fulton
On Sat, Mar 9, 2013 at 9:02 AM, Jim Fulton wrote: ... > I think a simple method on a storage that gives a hint that a set of > object ids will be loaded is enough. A network storage could then > issue a pipelined request for those oids. The application can then > proceed as usual.

Re: [ZODB-Dev] Cache warm up time

2013-03-09 Thread Jim Fulton
s a sort of anti-pattern than only makes sense for small catalogs. It would likely make more sense to have a dedicated catalog server that returned oids and possibly object records in response to queries (or whimper, use solr ). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is

Re: [ZODB-Dev] transaction: synchronizer newTransaction() behavior

2013-03-09 Thread Jim Fulton
I'm retaining a > reference to the synch as a global that I never use. Just seems a bit > contrived so thought I'd mention that as well, in case there's anything that > can be done about that. This is to prevent memory leaks. Normally, the synchronizier is associated wi

Re: [ZODB-Dev] zodb conversion questions

2013-02-07 Thread Jim Fulton
and powerful. This API just hides that. The second part, replaying old transactions is a bit more subtle, but it's still worth it for people to be aware of it. If I were doing this today, I'd make this documentation rather than API. But then, documentation ... whimper. > Anybody

[ZODB-Dev] #zodb on freenode

2013-02-01 Thread Jim Fulton
I hang there, fwiw :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-28 Thread Jim Fulton
On Wed, Jan 2, 2013 at 9:37 AM, Jim Fulton wrote: > On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: >> I plan to do ZEO shortly. > > Well, that didn't go well. svn git fetch spent several days and didn't > finish. It seemed to be really thrashing trying to f

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-19 Thread Jim Fulton
On Sat, Jan 19, 2013 at 10:06 AM, Leonardo Santagada wrote: > > On Sat, Jan 19, 2013 at 12:51 PM, Jim Fulton wrote: >> >> - solr is so fast it almost makes me want to cry. At ZC, we're >> increasingly using solr instead of the catalog. As the original >>

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-19 Thread Jim Fulton
I'll have to preload upon every zeopack. If it's not that, > then I'm not sure what it could be. I think you have some basic application design problem(s). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm __

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-19 Thread Jim Fulton
e, puting most data in catalog-like data structures and querying to get most of your data. The strength of a OODB is that you don't have to query to get data from a well-designed object model. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! ht

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-18 Thread Jim Fulton
~10 seconds on the 350k-sized index > trees vs. ~60-90 seconds. This seems to indicate that less things end up > being pre-loaded... I guess I was too subtle before. Preloading is a waste of time. Just use a persistent ZEO cache of adequate size and be done with it.

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Jim Fulton
On Tue, Jan 15, 2013 at 2:45 PM, Claudiu Saftoiu wrote: > On Tue, Jan 15, 2013 at 2:40 PM, Jim Fulton wrote: >> >> So, first, a concise partial answer to a previous question: >> >> ZODB provides an in-memory object cache. This is non-persistent. >> If you resta

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Jim Fulton
DB doesn't provide any pre-warming facility. This would be application dependent. You're probably better off using a persistent ZEO cache and letting the cache fill with objects you actually use. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http:

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Jim Fulton
On Tue, Jan 15, 2013 at 2:08 PM, Claudiu Saftoiu wrote: > On Tue, Jan 15, 2013 at 2:07 PM, Leonardo Santagada > wrote: >> >> >> >> >> On Tue, Jan 15, 2013 at 3:10 PM, Jim Fulton wrote: >>> >>> On Tue, Jan 15, 2013 at 12:00 PM, Claudiu Saftoi

Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?

2013-01-15 Thread Jim Fulton
x27;t a big enough win, but this likely depends on your use cases. We (ZC) took a different approach. If there is a reasonable way to classify your corpus by URL (or other request parameter), then check out zc.resumelb. This fit our use cases well. Jim -- Jim Fulton http://www.linkedin.c

Re: [ZODB-Dev] API question

2013-01-15 Thread Jim Fulton
lestorage # if needed We shouldn't design an API where we expected people to grab underware. Aside from not liking from imports and the _filestorage nit, +1 > For extra bonus fun, we could rename 'ZODB' to 'zodb' :) In that case, we might switch to a nam

Re: [ZODB-Dev] API question

2013-01-15 Thread Jim Fulton
grown some additional shortcuts that I think are also useful. Among them: ZODB.DB(filename) -> DB with a file storage ZODB.DB(None) -> DB with a mapping storage ZODB.connection(filename) -> connection to DB with file storage ZODB.connection(None) -> connection to DB

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Jim Fulton
e more objects through garbage collection. The first step is to find out if the record is really damaged. Maybe make a back up of your database if you can and then pack without garbage collection to free up space while you figure out what's wrong with your data. Jim -- Jim Fulton http://w

Re: [ZODB-Dev] zeopack error in zrpc.connection

2013-01-07 Thread Jim Fulton
t;, line > 630, in referencesf > u.noload() > TypeError: 'NoneType' object does not support item assignment > > > I'm afraid this doesn't seem to help me figure out what's wrong... I suspect your database is corrupted. You'd probably want to l

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-05 Thread Jim Fulton
On Fri, Jan 4, 2013 at 1:19 PM, Jim Fulton wrote: >>>> I'll do and BTrees, persistent and transaction in about a week. ... > This weekend, I'll update svn for these packages and ZODB to indicate > that development is taking place in github. So ZODB, persistent, BTre

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-04 Thread Jim Fulton
On Fri, Jan 4, 2013 at 1:05 PM, Jim Fulton wrote: > On Wed, Jan 2, 2013 at 9:37 AM, Jim Fulton wrote: >> On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: >>> I plan to do ZEO shortly. >> >> Well, that didn't go well. svn git fetch spent several days and

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-04 Thread Jim Fulton
On Wed, Jan 2, 2013 at 9:37 AM, Jim Fulton wrote: > On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: >> I plan to do ZEO shortly. > > Well, that didn't go well. svn git fetch spent several days and didn't > finish. It seemed to be really thrashing trying to f

Re: [ZODB-Dev] repoze.catalog.query very slow

2013-01-04 Thread Jim Fulton
nd again 1 second each subsequent time. >> >> What makes it run so slow? Is it that the catalog isn't fully in memory? If >> so, is there any way I can guarantee the catalog will be in memory given >> that my entire database doesn't fit in memory all at once? >>

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-02 Thread Jim Fulton
On Wed, Jan 2, 2013 at 1:04 PM, Leonardo Santagada wrote: > > > On Wed, Jan 2, 2013 at 12:37 PM, Jim Fulton wrote: >> >> On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: >> > I plan to do ZEO shortly. >> >> Well, that didn't go well. svn git fet

Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2013-01-02 Thread Jim Fulton
On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: > I plan to do ZEO shortly. Well, that didn't go well. svn git fetch spent several days and didn't finish. It seemed to be really thrashing trying to follow tags. I'm probably going to have to convert just the trunk. I&#x

Re: [ZODB-Dev] Blobstorage shrinks only after the second pack operation - bug or feature?

2012-12-30 Thread Jim Fulton
On Sun, Dec 30, 2012 at 12:22 PM, Andreas Jung wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > > > Jim Fulton wrote: >> On Sun, Dec 30, 2012 at 3:50 AM, Andreas Jung >> wrote: >>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 >>>

Re: [ZODB-Dev] Blobstorage shrinks only after the second pack operation - bug or feature?

2012-12-30 Thread Jim Fulton
> > So why do I have to pack two times in order to get a minimized blob > storage? Without knowing more details, it's impossible to know. What did you have pack-keep-old set to? > Bug or feature? I doubt either. Jim -- Jim Fulton http://www.linkedin.com/in/jimf

[ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github

2012-12-28 Thread Jim Fulton
I plan to do ZEO shortly. I'll do and BTrees, persistent and transaction in about a week. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zod

Re: [ZODB-Dev] Moving ZODB projects to github

2012-12-28 Thread Jim Fulton
more and maybe/probably incorporated the github/oauth goodness you pointed me to. But here's a snapshot. Let me know if it works for you (or doesn't). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm svn2git.py Description: Binary dat

Re: [ZODB-Dev] Moving ZODB projects to github

2012-12-28 Thread Jim Fulton
On Wed, Dec 26, 2012 at 1:42 PM, Jim Fulton wrote: > I'd like to move ZODB-related projects to github nowish. > (I'm having problems converting ZODB though :() OK, I think I worked out the problems I was having with svn2git (It's broken. I wrote a simple Python script th

Re: [ZODB-Dev] Please clean up your unneeded dev branches in the ZODB svn project

2012-12-26 Thread Jim Fulton
On Wed, Dec 26, 2012 at 7:25 PM, Jim Fulton wrote: > Some of the branches (especially some tseaver_...), are giving svn2git fits. G. The branches that bothered svn2git were already deleted. Whimper. Sorry. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than ba

[ZODB-Dev] Please clean up your unneeded dev branches in the ZODB svn project

2012-12-26 Thread Jim Fulton
Some of the branches (especially some tseaver_...), are giving svn2git fits. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev

[ZODB-Dev] Moving ZODB projects to github

2012-12-26 Thread Jim Fulton
I'd like to move ZODB-related projects to github nowish. (I'm having problems converting ZODB though :() If you're currently working on something, let me know so I don't leave your work behind. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bac

Re: [ZODB-Dev] shared cache when no write?

2012-12-13 Thread Jim Fulton
abase. If you break that, then you're injecting concurrency issues into the app or in some pretty magical layer. > You perhaps could have a single read only db connection which is > shared? But even if the database data was only read, objects have other state that may be mutated. You'd

Re: [ZODB-Dev] shared cache when no write?

2012-12-13 Thread Jim Fulton
pend a lot of time blocked waiting on some external service, why not close the database connection and reopen it when you need it? Then you could have a lot more threads than database connections. It's possible that ZODB could help at the savepoint level. For example, maybe you could somehow

Re: [ZODB-Dev] ZEO invalidation message transaction-inbound or outbound

2012-12-03 Thread Jim Fulton
vent this from happening. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/ma

[ZODB-Dev] Need Windows Volunteers -- Losing interest in windows

2012-12-02 Thread Jim Fulton
seem to work with Python 3. . I'll keep using it as long as it works, but ... We need people who care about Windows to step up. It's necessary, but not sufficient to run tests on windows. We need people willing to debug and fix windows issues. Jim -- Jim Fulton http://www.linkedin.co

  1   2   3   4   5   6   7   8   9   10   >