On Mon, 10 Aug 2020 at 22:10, Daniel Gruno <[email protected]> wrote: > > On 10/08/2020 22.56, sebb wrote: > > On Mon, 10 Aug 2020 at 21:49, Daniel Gruno <[email protected]> wrote: > >> > >> On 10/08/2020 22.46, sebb wrote: > >>> On Mon, 10 Aug 2020 at 18:51, Daniel Gruno <[email protected]> wrote: > >>>> > >>>> Hi folks, > >>>> with the incoming new UI and with ElasticSearch moving far away from > >>>> what Pony Mail supports currently, I think it's also time to get started > >>>> on a "next generation" of Pony Mail, that supports the new structures > >>>> laid out in ES 7.x and above, and while we're at it....I think we should > >>>> ditch Lua and use a pure python implementation instead, to increase the > >>>> number of potential contributors (and make use of Python's excellent > >>>> libraries). > >>>> > >>>> SO, with all that said, I'll be setting up a new repository for this > >>>> 'next generation' of Pony Mail, so as to not start mixing the old and > >>>> the new too much. I think this is justified, as it's a very major shift > >>>> from the old code-base, and the two would be internally incompatible > >>>> (except for the JSON API, I believe that should remain as is.) > >>>> > >>>> My plan is to: > >>>> > >>>> - create a new repository > >>>> - import the UIX code-base into this > >>>> - import the tools we have from pony v1, tweak those later on > >>>> - get started on a pure python back-end for the UI (I have some > >>>> semblance of a prototype that I'm hacking on currently) > >>>> - finally, write a migration script for migrating from the old DBs to > >>>> the new format. > >>> > >>> Before one can even think about migrating a database, there needs to > >>> be a full test suite. > >>> In particular, there need to be exhaustive tests to show that the same > >>> Permalinks will be generated. > >> > >> I think perhaps you misunderstand me here :) > >> The database *contents* would be migrated verbatim, just with the new ES > >> structure, where each doctype in the old setup is its own index with > >> doctype _doc. The IDs stay the same as they were before. > > > > I realise that. > > > > However, I don't think this can be done without changing the backend > > code which is responsible for generating the IDs going forward. > > There have already been unplanned changes to the generators which > > affect the output. > > Such changes need to be caught before they cause compatibility issues. > > There have not been changes to what's allowed as a document ID in ES. > > 'generating the IDs' does not play into this at all, I'm not talking > about re-importing, but copying the database content including document > IDs verbatim, just to a new "directory structure". If something has ID > 1234foo in the old database, it would still be 1234foo in the new one, > but it would be present in the index ponymail-mbox instead of ponymail.
I think you are missing the point. The move to a different database structure will necessarily involve changes to the backend Python code. It is vital any changes don't affect the Permalinks, so there needs to be a proper test suite. > > > >> thus, the ponymail database (or whatever you have called it) would get > >> split into several databases instead: > >> - ponymail-mbox > >> - ponymail-attachments > >> - ponymail-sources > >> - ponymail-sessions > >> etc etc > >> > >> With regards, > >> Daniel. > >> > >>> > >>>> Comments/feedback is always welcome as usual :) > >>>> > >>>> With regards, > >>>> Daniel. > >> >
