Re: [ZODB-Dev] [Plone-developers] ZODB on a DHT for P2P Plone
Hi, after looking in to Plone/ZODB we've found that it will not be an appropriate technology for P2P use. Firstly Plone would need to be modified greatly to ensure that it works in a way where it maintains indexes rather than performing real-time queries due to the high latency and intermittent availability of peers. From the ZODB aspect the closest storage layer to working in P2P is NEO, but Vincent from the NEO developer team pointed out that it won't be ready for P2P for some time yet: As one of the NEO team, what are your thoughts on the practicality of running Plone in a P2P environment with the latencies experienced in standard DHT (such as for example those based on Kademlia) implemtations? First, I must say that we have not run any benchmark on NEO outside LAN conditions yet, because there are some issues which need attention before we can increase test hostility. To name a few blockers, there is a need for peaceful deadlock resolution/avoidance when the same set of objects gets modified concurrently, and an important from scratch replication performance issue. Another show-stopper for NEO production-readiness is the lack of backup tools, as NEO currently relies on storage back-end tools (eg. mysqldump) and on a replication scheme which is not implemented (useful in a all-nodes-in-datacenter setup, not if nodes are to be scattered around the globe). On the bright side, we have found a technology which looks very promising which is Squeak (open source Smalltalk) and the Seaside web application framework, so we'll be pursuing this route more deeply for a while. Thanks all for your advice and feedback :-) Cheers, Aran On 08/01/11 05:04, Carol Ganz wrote: Hello Aran, This project looks very interesting and we would be happy to enter into discussions with you on the feasibility of the proposed architecture. Six Feet Up excels in helping clients with complex requirements deliver easy-to-use products to their users. Our proven project management skills are invaluable to our partners through the cost savings they provide (http://www.sixfeetup.com/blog/how-to-save-40-percent-on-development-costs). Well-defined development processes, backed up by in-house quality assurance and Hudson testing, ensure the high quality of our code and that the results we produce are exceptional. Six Feet Up develops long-term partnerships with our clients through transparency in every step of the process and involving decision makers from day one. Is your team to the point of entering into a discovery phase to define the project and to begin prototyping? I would like to schedule a meeting to discuss this project at your convenience. Thank you, Carol Ganz Account Manager ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Plone in P2P using Zope over DHT
Hi guys, It's looking like Plone/ZODB would need too much modification and testing at the current time to move in to P2P. The most promising environment in our research so far are frameworks built on the Squeak language which is used by the OpenCobalt project (a P2P collaborative 3D world). Thanks for all your feedback and advice, and happy new year :-) Aran ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Plone in P2P using Zope over DHT
Thanks for the feedback Vincent :-) it sounds like NEO is pretty close to being SQL-free. As one of the NEO team, what are your thoughts on the practicality of running Plone in a P2P environment with the latencies experienced in standard DHT (such as for example those based on Kademlia) implemtations? On 04/01/11 22:27, Vincent Pelletier wrote: Hi. Le mardi 4 janvier 2011 07:18:34, Aran Dunkley a écrit : The problem is that it uses SQL for its indexing queries (they quote NoSQL as meaning Not only SQL). SQL cannot work in P2P space, but can be made to work on server-clusters. Yes, we use MySQL, and it bites us on both worlds actually: - in relational world, we irritate developers as we ask questions like why does InnoDB load a whole row when we just select primary key columns, which ends up with don't store blobs in mysql - in key-value world, because NoSQL using MySQL doesn't look consistent So, why do we use MySQL in NEO ? We use InnoDB as an efficient BTree implementation, which handles persistence. We use MySQL as a handy data definition language (NEO is still evolving, we need an easy way to tweak table structure when a new feature requires it), but we don't need any transactional isolation (each MySQL process used for NEO is accessed by only one process through one connection). We want to stop using MySQL InnoDB in favour of leaner-and-meaner back-ends. I would especially like to try kyoto cabinet[1] in on-disk BTree mode, but it requires more work than the existing MySQL adaptor and there are more urgent tasks in NEO. Just as a proof-of-concept, NEO can use a Python BTree implementation as an alternative (RAM-only) storage back-end. We use ZODB's BTree implementation, which might look surprising as it's designed to be stored in a ZODB... But they work just as well in-RAM, and that's all I needed for such proof-of- concept. [1] http://fallabs.com/kyotocabinet/ Regards, ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Plone in P2P using Zope over DHT
Hi, I'm part of a development team who are helping an organisation to architect a CMS based project that they want to work in a P2P network rather than using a centralised web-server. We'd prefer to use an existing popular CMS as a starting point so that it is mature, has a large development community and a wide range of extensions/modules available. From our initial research it seems that Plone should be more capable of moving in to the P2P space due to it using ZODB rather than SQL and that ZODB seems able to be connected to a variety of storage mechanisms. I'm wondering what you guys, the core developers, think of the practicalities of Plone in P2P, for example could ZODB use a DHT as its storage layer? what kind of querying is required on the DHT? We have a good budget available for this and will be developing it as a completely free open source component, so we'd also like to hear from developers who may be interested in working on the project too. Thanks, Aran ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Plone in P2P using Zope over DHT
I mean P2P as in using a DHT (distributed hash table) for its storage layer so that it can run in a network composed only of the client users with no web-server, the same way that the file-sharing programs such as azzureus and aMule work. We can develop a basic wiki/blog in this way, but would prefer a mature CMS like Plone. On 04/01/11 18:34, Andreas Jung wrote: Aran Dunkley wrote: I'm wondering what you guys, the core developers, think of the practicalities of Plone in P2P, for example could ZODB use a DHT as its storage layer? what kind of querying is required on the DHT? No idea what P2P means to you in particular. The ZODB is a mandatory requirement for Plone as storage layer. You may work on storage layer implementation for the ZODB fulfilling your requirements (similar to Relstorage implementing a RDBMS backend for the ZODB). I have strong doubts that a P2P or (cloud-ish) backend is feasible as the ZODB is ACID compatible and ACID is basically something you can not achieve in the cloud or better spoken in a highly distributed environment. -aj ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Plone in P2P using Zope over DHT
I have looked at NEO which is the closest thing I've found to the answer, in fact NEO is why I felt Plone was the best choice of CMS to inquire further about The problem is that it uses SQL for its indexing queries (they quote NoSQL as meaning Not only SQL). SQL cannot work in P2P space, but can be made to work on server-clusters. We intend not to have any machines in our network other than the users computers running the P2P application. So we would need to know exactly what kinds of querying ZODB expects to be available in its interface to the storage layer. DHT's can be slow for the first read but cache locally after that. On 04/01/11 20:06, Andreas Jung wrote: Marc Villemade wrote: Hey Andreas, I think it makes sense if the storage backend is hosted on-premise (hence private cloud), and i think that's what Aran is thinking about. They could have Plone-ZODB-Storage backend in the same datacenter, hence no latency problem and good network throughput, provided that the cloud storage used is capable of offering high performance. First you should bring up arguments why the existing backends like ZEO, Relstorage or NEO are not good enough in your case. Looking at the development history of Relstorage or NEO: implementing an enterprise-level storage for the ZODB seems to be hard and time-consuming (and expensive). -aj ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
