[Citadel Development] Re: Master: citserver coredump on Shutdown
yay
[Citadel Development] Re: Master: citserver coredump on Shutdown
>When I build from master now, when I shutdown citserver, it core dumps right >after closing the 0d database. Since I had an issue where webcit was core >dumping and was not re-creatable by anyone else, I am wondering if this is >just me. Remember that discussion, from early July? That's part of what started this all ... and made me basically spend the whole summer overhauling the database layer instead of working on WebCit-NG. Well guess what ... it's fixed. I'm sure it had something to do with the "dbenv" and whether it was disk backed or memory backed or both, or something like that. Now that we're using what I believe is mmap'ed disk to back the environment (that's the new __db.00x files you'll see in the data directory) it no longer segfaults when closing the environment. And did I mention that since this change, it hasn't corrupted the test database even once, no matter how badly I abuse it?
[Citadel Development] Re: Master: citserver coredump on Shutdown
All good, take as much time as you need on the IMAP flags stuff. We'll make sure we do it right. As you've undoubtedly noticed if you follow the commit logs, over the last couple of weeks I've written "ctdldump" and "ctdlload" utilities that can be used to dump a Citadel database into a flat file and then load it back on another system. This can be the first step towards a transition period between databases, of course, since we can build compatible dump/load utilities for each database that share a common export format. To make things easier: the export format is not intended to be backwards compatible. As with its predecessor, we expect the dumping and loading systems to be running the same version of Citadel Server. Uncensored is still running on 32-bit i386. One of the first things I want to do is use these utilities to finally get my own system onto 64-bit. I tried it a few months ago with ctdlmigrate and it did *not* go well. Aside from taking many hours to run, the resulting system was unstable and had data missing. I guess all that conversion back and forth to XML on running servers just wasn't a great idea, which is why ctdlmigrate is being discontinued. Sqlite is pretty cool, and I'd be interested in hearing your schema idea. I am somewhat turned off by the idea of having a schema layer and not really using it. I wonder if it's possible to access Sqlite's B-Tree back end directly?
[Citadel Development] Re: Master: citserver coredump on Shutdown
After I finish up the IMAP Flags, I should have some time to contribute to the database. I was initially thinking of SqlLite. The trick is what database schema we want to support. I was looking at keeping with the key/value concept and just replace database.c. I have an idea of the schema for the database that will be efficient on database size.
[Citadel Development] Re: Master: citserver coredump on Shutdown
To get things started, I have begun work on `ctdldump` and `ctdlload` frameworks. I've also improved the Makefile for Citadel Server so that it doesn't require compiling the entire server every time even one file is touched. I got lazy when I removed the GNU Autotools and just had it compile *.c in one shot, and I'm finally coming back to that now. It needs to be tested on FreeBSD to make sure I didn't do anything Linux-specific in there.
[Citadel Development] Re: Master: citserver coredump on Shutdown
Potentially, yes. I'm thinking more in terms of how to refactor the code. The database access stuff is all in `database.c` because there *was* a somewhat pluggable architecture in the past -- from flat files to gdbm and later to Berkeley DB. Since we've been using Berkeley DB exclusively for 20+ years, some tight couplings may have snuck in, so we'll have to identify them and properly abstract them. The pluggable database layer was, at the time, intended to be used to switch between blob stores, not SQL databases. I'm not against going SQL but it would greatly impact how we build it -- and that's going to depend on who is doing the work. The way I see it, a Citadel Server binary would be linked against one (and only one) database back end. We've played around with linkable modules in the past, and it ended up being a support nightmare. Maybe that's different now because we only really have to support Linux and FreeBSD, and both of those are more stable targets. So if you have support for multiple database types, each one would have a `citserver` binary, a `dump` binary, and a `load` binary. Some of these would be unnecessary if some of the back ends are SQL based and the dump format is a standard SQL dump. I see two different development paths. One is a march towards LMDB and one is a march towards SQL: (A) Blob stores only: 1. Make sure the `database.c` interface is 100% decoupled from the rest of the system. 2. Refactor `ctdl3264` into a dump-and-load program. 3. Develop drivers for LMDB, Cassandra, MongoDB, or whatever. (B) Blob *and* SQL stores: 1. The new abstraction becomes the interfaces such as Ctdl[Get|Put]User, Ctdl[Get|Put]Room, ForEachUser, etc. Refactor the code so these interfaces find their way into the pluggable side of things. 2. Add new pluggable stores for SQLite, MariaDB, etc. 3. Refactor `ctdl3264` into a program that can dump the blob stores into a standard SQL dump. (Dump and load on the SQL based back ends would use their native utilities.) I'm interested in hearing any and all good ideas! But no one gets a vote unless they commit to helping :)
[Citadel Development] Re: Master: citserver coredump on Shutdown
MariaDB and/or MySQL Sun Jul 09 2023 09:46:26 EDT from IGnatius T Foobar Subject: Re: Master: citserver coredump on Shutdown Grrr. Unrecoverable error on my database this morning. I think maybe it is time to put Berkeley DB out to pasture after all. Let's think about what we want the next to look like.
[Citadel Development] Re: Master: citserver coredump on Shutdown
Grrr. Unrecoverable error on my database this morning. I think maybe it is time to put Berkeley DB out to pasture after all. Let's think about what we want the next to look like.
[Citadel Development] Re: Master: citserver coredump on Shutdown
>Pretty consistent for me. I put a check in the Berkley code to work around, >but not sure of the root cause either. Something must be getting double freed, but I can't figure out what.
[Citadel Development] Re: Master: citserver coredump on Shutdown
Pretty consistent for me. I put a check in the Berkley code to work around, but not sure of the root cause either.
[Citadel Development] Re: Master: citserver coredump on Shutdown
>It looks like the Berkley code is passing a NULL to a memory free routine >that does not check for NULL. Anyone else having a problem? If not, I >will look into how I build the server. It happens to me too, some of the time, and I have been busting my butt trying to figure it out.
