Re: [Zope-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB

2012-08-28 Thread Vincent Pelletier
On Mon, 27 Aug 2012 14:37:37 +0200,
Vincent Pelletier vinc...@nexedi.com wrote :
 Under the hood, it relies on simple features of SQL databases

To make things maybe a bit clearer, from the feedback I get:
You can forget about SQL presence. NEO usage of SQL is as a relational
as a handful of python dicts is. Except there is no way to load only
part of a pickled dict, or do range searches (ZODB's BTrees are much
better in this regard), or writable to disk atomically without having to
implement this level of atomicity ourselves.

Ideally, NEO would use something like libhail, or maybe even simpler
like kyotocabinet (except that we need composed keys, and kyotocabinet
b-trees have AFAIK no such notion).
SQL as a data definition language was simply too convenient during
development (need a new column ? easy, even if you have a 40GB table),
and it stuck - and we have yet to find a significant drawback to
implement a new storage backend.

As a side effect, SQL allows gathering some statistics over the data
contained in a database very efficiently. Number of current objects,
number of revisions per object, number of transactions, when
transactions occured in base history, average object size, largest
object, you name it.

-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB

2012-08-28 Thread Vincent Pelletier
On Tue, 28 Aug 2012 16:31:20 +0200,
Martijn Pieters m...@zopatista.com wrote :
 Anything else different? Did you make any performance comparisons
 between RelStorage and NEO?

I believe the main difference compared to all other ZODB Storage
implementation is the finer-grained locking scheme: in all storage
implementations I know, there is a database-level lock during the
entire second phase of 2PC, whereas in NEO transactions are serialised
only when they alter a common set of objects.
This removes an argument in favour of splitting databases (ie, using
mountpoints): evading the tpc_vote..tpc_finish database-level locking.

Also, NEO distributes objects over several servers (aka, some or all
servers might not contain the whole database), for load balancing/
parallelism purposes. This is not possible if one relies on relational
database replication alone.

I forgot in the original mail to mention that NEO does all conflict
resolutions on client side rather than server side. The same happens in
relStorage, but this is different from ZEO. Packing on client side
makes it easier to get the setup right: with ZEO you will get more
conflicts than normal if it cannot load some class which implements
conflict resolution, and this might go unnoticed until someone worries
about a performance drop or so. With client-side resolution, if you
don't see Broken Objects, conflict resolution for those classes works.

Some comments on some points you mentioned:
 * NEO supports MySQL and sqlite, RelStorage MySQL, PostgreSQL and
 Oracle.

It should be rather easy to adapt to more back-ends.
We (Nexedi) are not interested in proprietary software, so we will
probably not implement Oracle support ourselves. For PostgreSQL, it's
just that we do not have a setup at hand and the experience to
implement a client properly. I expect that it would not take more than a
week to get PostgreSQL implemented by someone used to it and knowing
python, but new to NEO.

Just to demonstrate that NEO really does not rely on fancy features of
SQL servers, you may dig in older revisions in NEO's git repository. You
can find a btree.py[1] test storage, which is based on ZODB.BTree
class. It was just a toy, without persistence support (I initially
intended to provide it, but never finished it) and hence limited by
the available amount of RAM. But it was otherwise a fully functional NEO
storage backend. I think it took me a week-end to put it together,
while discovering ZODB.Btree API and adapting NEO's storage backend
API along the way (this was the first non-MySQL backend ever
implemented, so API was a bit too ad-hoc at that time).

sqlite was chosen as a way to get rid of the need to setup a
stand-alone SQL server in addition to NEO storage process. We are not
sure yet of how well our database schema holds when there are several
(10+) GB of data in each storage node.

 * RelStorage can act as a BlobStorage, NEO can not.

I would like to stress that this has nothing to do with design, rather
it's just not implemented. We do not wish to rely on filesystem-level
sharing, so we consider something along the lines of providing a
FUSE-based to share blob storage, which then can abstract the blobs
being distributed over several servers. This is just the general idea,
we don't have much experience with blob handling ourselves (which is
why we preferred to leave it asides rather than providing an
unrealistic - and hence unusable - implementation).

[1]http://git.erp5.org/gitweb/neoppod.git/blob/75d83690bd4a34cfe5ed83c949e4a32c7dec7c82:/neo/storage/database/btree.py

Regards,
-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB

2012-08-27 Thread Vincent Pelletier
Hi,

We've just tagged the 1.0 NEO release.

NEO aims at being a replacement for use-cases where ZEO is used, but
with better scalability (by allowing data of a single database to be
distributed over several machines, and by removing database-level
locking), with failure resilience (by mirroring database content among
machines). Under the hood, it relies on simple features of SQL
databases (safe on-disk data structure, efficient memory usage,
efficient indexes).

Release highlights:
- production-ready !
- asynchronous replication across clusters, for inter-datacenter
  redundancy
- there will be no further expensive data schema changes within the 1.x
  branch as there were in 0.x branch
- replication performance is significantly increased
- general implementation performance improved
- several bugfixes

What you need to know if you are used to ZODB features:
- Blob API is not implemented yet.
- pack's GC phase will not be implemented in NEO, it relies on
  zc.zodbdgc for this: http://pypi.python.org/pypi/zc.zodbdgc

For more details, look at README and CHANGES included with the sources:
  http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/README
  http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/CHANGES

NEO is published on pypi as neoppod:
  http://pypi.python.org/pypi/neoppod

Regards,
-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Zope 4 sprint 27-31 August, Lille (FR)

2012-07-30 Thread Vincent Pelletier
On Mon, 16 Jul 2012 19:12:16 +0200,
Vincent Pelletier vinc...@nexedi.com wrote :
 A wiki page[1] has been set up with a list of subscribers, a topic
 list, and general information about transport  stay accommodation.

Something I forgot: to subscribe, you should either update the wiki
page or mail me directly.

-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Zope 4 sprint 27-31 August, Lille (FR)

2012-07-16 Thread Vincent Pelletier
Hi.

Nexedi would like to host a Zope 4 sprint in Lille, France, from the
27th of august to 31st.

A wiki page[1] has been set up with a list of subscribers, a topic list,
and general information about transport  stay accommodation.

A mailing list[2] is available to help coordinate participants
(deciding on meeting points...).

If you would like to attend the sprint but cannot afford the trip to
Lille, please let me know.

[1] http://www.erp5.org/Zope4Sprint2012
[2] https://mail.tiolive.com/mailman/listinfo/zope4sprint2012
-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] [ZODB-Dev] Bug (?) in zope/publisher/publish.py:unwrapMethod

2011-01-26 Thread Vincent Pelletier
Le mardi 25 janvier 2011 19:08:11, Tres Seaver a écrit :
 The Zope2-specific version of 'mapply()' (in 'ZPublisher.mapply') is the
 right place to fix this issue, if it is to be fixed:

Thanks for the info.

 P.S. This issue is off-topic for the ZODB list:  I have cross-posted
  to 'zope-dev':  please follow up there.

Woops, lazy typing and wrong mail client completion. I indeed intended it for 
zope-dev.
For some reason, I didn't see your mail on zope-dev (I checked the archives 
too, but they might be lagging).

-- 
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] [ZODB-Dev] Bug (?) in zope/publisher/publish.py:unwrapMethod

2011-01-26 Thread Vincent Pelletier
Le mercredi 26 janvier 2011 08:54:02, Vincent Pelletier a écrit :
 For some reason, I didn't see your mail on zope-dev

As this mail reached the list, I think Tres' mail got caught by some filter. 
Original mail was:

In publish.py[1], unwrapMethod tried to find what can be used to publish an 
object. In a site, I had someone create a very-badly-named func_code 
external method in a place accessible by acquisition from every page on the 
site (this bad by itself, and I corrected it already). This caused 
unwrapMethod to think it can use any object directly for publishing, because 
of:

elif getattr(unwrapped, 'func_code', None) is not None:
break

and unwrapped is still in an acquisition context.

Shouldn't the checks be done on unwrapped (from acquisition context) objects 
instead, to prevent such stupid mistake to have such a wide impact.

I have the intuition that this could even be a security problem, allowing an 
unexpected object to be called instead of another, but I cannot come out with 
an example.

Do you think there is anything to fix in zope.publisher[2] ? If so, I'll open 
a bug.

[1] 
http://svn.zope.org/zope.publisher/trunk/src/zope/publisher/publish.py?view=markup
[2] following Tres' answer, make this Zope2's mapply

Regards,
-- 
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] NEO High Performance Distributed Fault Tolerant ZODB Storage

2010-04-07 Thread Vincent Pelletier
 if a node starts flapping, and could be made 
configurable.

Storage comes back
When a storage comes back to life (after a power failure or whatever), it asks 
other storage nodes for the transactions it missed, and replicate them.

 That's unfortunate.  Why not a less restrictive license?

Because we have several FSF members in Nexedi, so we use the GPL for all our 
products (with very few exceptions, notably monkey-patches reusing code under 
different licenses).

 These seem to be very high level.
 You provide a link to the source, which is rather low level.
 Anything in between?

You mean, a presentation of NEO (maybe what I wrote above would fit) ?
Or on the write scalable Zope application topic ?

Regards,
-- 
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] NEO High Performance Distributed Fault Tolerant ZODB Storage

2010-04-01 Thread Vincent Pelletier
Le mercredi 31 mars 2010 18:32:31, vous avez écrit :
 A few questions that you may want to add in a FAQ.

We started that page and will publish it very soon, based on most points you 
raised. Other pages are also being worked on, such as an overview of a simple 
NEO cluster.

   - Why not include ZODB 3.10/Python 2.6 as a goal of the project?
 - I understand *today* the technologies use python 2.4 but
   ZODB 3.10/Plone 4/Zope 2.12 use python 2.6

We indeed aim at supporting more recent versions of python and Zope.

Actually, your remark made us realise that our functional tests are currently 
(accidentally) running in a mixed 2.4/2.5 python environment: test process is 
started explicitly with 2.4, and forked processes (masters, storage and admin 
nodes) are running on default python, which is 2.5 (as of Debian stable).

The standard Zope version in Nexedi is 2.8, which explains why we want to 
support it. We will switch to 2.12, as we have ERP5 unit tests running on 2.12 
for some weeks[1] now. NEO will move to 2.12 at the same time or earlier.

   - Maybe explain the goal of the project clearer:
 
 NEO provides distributed, redundant and transactional storage designed
 for petabytes of persistent (python?) objects.
 

Thanks, updated.

   - A buildout for NEO would lower bar for evaluation

This is on our roadmap (...to be published along with the FAQ), but the 
priority currently goes to 2 developments which might/will break 
compatibility: pack support (required an undo rework which was recently 
integrated, pack itself needs more unit testing prior to integration) and 
multi-export support (aka ZODB mountpoints, also in need for more testing 
before integration).

[1] http://mail.nexedi.com/pipermail/erp5-report/ (_z212 in subject)
-- 
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] NEO High Performance Distributed Fault Tolerant ZODB Storage

2010-03-31 Thread Vincent Pelletier
Hi,

I would like to present you the NEOPPOD project, aiming at improving ZODB 
Storage scalability. The implementation is in a rather good shape, although it
fails at a few ZODB tests at the moment (they are currently being worked on).
Scalability is achieved by distributing data over multiple servers
(replication and load balancing) with the ability to extend/reduce cluster
on-line.

Its code is available under the GPL, more information can be found on the 
project website[1].

One nice aspect is that the underlying protocol is being analysed with 
model-checking tools based on Petri Nets by a team of post-doc, PhD 
students and researchers. An article should appear in PETRI NETS 2010 [2].

We hope that NEO will be usable for production systems in 12 months from 
now, and will notify the community the day we think it is, ie. after 
using it ourselves. Meanwhile, it can be interesting for research and 
fun. Contributions are very welcome (extend portability beyond Linux 2.6, 
replace MySQL daemon dependency with a lighter  embeddable transactional 
storage, etc). For now, it has been manually confirmed to run Plone and ERP5.

Of course, to get the best out of NEO, a Zope application (or ZODB-based
application) needs to be designed in a way taking advantage of back-end
parallelism (much in the same way that a single-process application cannot
take advantage of SMP). We wrote a presentation[3] out of our experience with 
ERP5 scalability testing  improvement, which might be an interesting read for 
people developing on the Zope framework. It describes what was the most common
mistakes we corrected, and tools we developed to further extend scalability at 
various levels (NEO being the latest).

And for most productions system, ZEO is really great. We have for 
example used ZEO for now more than 2 years to operate a Central Bank ERP 
with 300 concurrent users and ZEO never crashed.

[1] http://www.neoppod.org/
[2] http://acsd-petrinets2010.di.uminho.pt/
[3] http://www.myerp5.com/kb/enterprise-High.Performance.Zope/view

Note: If this mail would fit another list better, please advise.

Regards,
--
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Dangerous shutdown procedure behaviour

2008-07-09 Thread Vincent Pelletier
Hi.

I think I discovered a dangerous code behaviour at zope shutdown.

I've had a strange problem on a site where persistent objects are created from 
data inserted in an SQL table. Upon object creation, SQL table is updated to 
mark the line as imported.
Such import got triggered just before a shutdown. After restart and another 
import, documents were created twice.

What I believe happened (but I could not find any hard evidence of it) is that 
Zope blindly exited while the working thread was runing, and in the worst 
possible method: tpc_finish.
ZODB was already commited, but mysql was not. So mysql did a rollback on 
changes, and the lines were in a ready to import state. And imported again 
at next import attemp.

Reading shutdown code, I discovered 2 distinct timeout mechanism (note: having 
just one is enough to trigger the problem):
 - Lifetime.py: iterating through asyncore sockets, it alerts
   servers that it will shut down soon. If they take the veto for too long,
   the veto is ignored and shutdown continues.
   Default timeout is 20 seconds, meaning there is at most one minute from the
   first shutdown notice to the effective process exit (taking all runing
   threads down).
   When invoking zopectl stop, it's runing a fast shutdown, which means
   the timeout is shortened to 1 second, so total maximum sutdown time is 3
   seconds.
   This timeout can be worked around by just writing blocking shutdown
   methods and not using the veto system.
 - zdaemon/zdrun.py: if the instance being shut down still
   responds after 10 seconds, it will be sent a SIGKILL.
   This cannot be worked around without changing code in zdrun.py or not
   executing it at all (no idea if there is any alternative).

I could easily reproduce the problem by writing a simple connection mamager 
which calls time.wait(3600) in _finish method and defining a sortKey method 
to make it commit after another connection manager.

I could not find a trace of any mechanism preventing commit from happening 
when a shutdown is in progress, and I don't think there should be any: 
considering that some storages might be accessed through a network, latency 
can become a problem, so tpc_finish can take time to complete, so just 
checking that there is no pending shutdown before entering this function 
would not solve the problem.

I suggest removing all those timeouts. If a user wants a Zope to shutdown for 
a reason serious enough to send it a SIGKILL or causing immediate python 
thread termination, it's his responsibility.
But I think regular shutdown mechanism must not do that.

Also, the same problem can happen with zopectl fg since Zope does not go 
through any shutdown sequence as far as I can tell (it just dies).

-- 
Vincent Pelletier
___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )