>Hardcode Visit legacy record sizes so will not change if defines change
Oh my ... if we ever changed SIZ to something other than 4096 there would
probably be so many side effects it would collapse the entire space-time
continuum.
:)
But yes, putting not one but two of those in a dat
Well, I've just upgraded Uncensored to the code in git master as of right
now, so we're now running in guinea pig mode. If it's going to do anything
wonky, I'd rather see it go wonky here than somewhere downstream. Uncensored
is configured to save all of its core dumps so if anything happens I'll
Back when the IMAP server was in initial development it was tested with a lot of different clients --- Pine, Mutt, Outlook, Eudora, Evolution, all of the "oh, I remember that!" ones you don't think about anymore. Nowadays I just test with Thunderbird and whatever is on my phone. Eventually
Most of what you're suggesting there would simply relate to how the server
passes data around itself internally and wouldn't even affect the data formats
on disk. So we can definitely look at those things.
Remember that Citadel is nearly 43 years old, and this implementation of
it is about 37
Ok, there is a test copy of Uncensored that has been upgraded to the
"development
only release" 1003
It is at test.citadel.org
Please log in and check it out! We can consider this a User Acceptance Testing
build.
Ignore the "1003 release" verbiage; we aren't actually doing a 1003 release.
This is just so I can bump the version number and do some test tarballs to
run against a test copy of my production database.
Also ... my apologies for the coding style cleanup; I probably forgot to
even mention
>Fix Visit IMAP not to return error on empty flags specification
Oops, I didn't realize you were still in there tuning it up. I hope my
style cleanup patches aren't interfering with your development.
Ooooh, I just noticed this in your commit:
Changed CtdlUserGoto() to return structure of info instead of passing
pointers. Now we only compute
the info if caller needs it. Saves having to scan all the messages
each time we call it.
That is great; thanks for doing it. I do
Ok, the fix was easy. If I am guessing correctly, you derived the meaning
of "new messages" in the output of the GOTO command to mean the same thing
as the IMAP "Recent" flag. In reality it means "Not Seen". The above patch
appears to restore the correct behavior and it appears to be working
UPDATE: upon further inspection, I am seeing something. What is actually
broken is the reporting of HOW MANY new messages are in a room. So when I
make a trip through the rooms with a test user, it actually takes me to the
rooms with new messages, and it says "0 new of xxx messages", but when
Confirmed that what is being delivered incorrectly is the second parameter
output by the result of a GOTO server command (see
https://www.citadel.org/protocol.html
for details). When this command completes successfully, the first parameter
output is the name of the room, the second is the num
Ok, here are my first results.
* Before upgrading, I ran loadtest to put a bunch of new messages in some
test rooms, then logged in as a test user and made a mental note of how many
new messages there were in various rooms.
* All code built cleanly, no errors or warnings.
* Ran ctdld
Dammit. A lot of files in there were not owned by the "git" group. I believe
I have fixed that, so try again.
Oh wait ... I'm an idiot. The upstream path is different for committers because
we have to use ssh.
ssh://harlowsoluti...@code.citadel.org:/var/www/git/citadel.git
You can use a git command to change it or you can just edit .git/config
If you're not on a branch, all you have to do is "git commit -a" followed
by "git push"
Ok, your login should be ready to go. I am out of town this week (in fact,
I'm writing this from an airport) so there is little chance of me writing
conflicting changes. Actually I might get bored and start coding ... but
I'll stick to webcit-ng :)
>I implemented everything on the v1001 tag. I have pulled the latest in
>git and am creating a local branch with my updates. Should I commit
>the code to master after I finish testing?
I have this annoying habit of going through code and doing syntax and style
cleanup when I am at
Welcome back! People come and go so I figured you'd lost interest. Very
happy to see that you're still here.
Now would be a decent time to work in your changes, since we are not in the
middle of a release cycle. Nearly all of my effort is going into WebCit-NG,
and I'm immersed in the ca
UFarx: thank you for bumping those issues. I'm about to apply some of your
patches, please watch the progress in this room and test the build from git
if you are willing.
What C compiler are you using?
Ok there we go. I know only a few people read this room but I like the idea
that someone is reading my commit messages because I pour my heart and soul
into them.
The way I just spent a week doing a major overhaul to the html_to_ascii
converter
will probably be missed by almost everyone, bec
I added my goostapo mail account as a mailing list recipient for this room.
If I get the message, it ought to have the "One Click Unsubscribe" button available, because we generate those headers now, and they are DKIM-signed.
In general, Uncensored seems to be out of the "reject completely" filte
So this is happening ...
[ https://code.citadel.org/?p=ig-dkim.git ]
I found a nice tidy little DKIM-signing library, just a few hundred lines
of code, one source module, abandoned for about twelve years after the project
that built it ceased to exist. Big thanks to them for leaving
I'm really pleased with how libical does the heavy lifting for me here. It
has improved greatly since the old days when we were its caretakers. Comparing
two timestamps that are in different time zones actually works. I don't know
how it works, but it works. Just to make sure it wasn't just usi
Well that's a fine howdy-do, I tagged the release without also mentioning why I pushed it at this time. This version has the config option to advertise STARTTLS on the public SMTP server.
All righty then. Pulling in the RSS feed from cgit, since OinkLab is no longer
in use. There will be some dupes. Here it comes.
For those of you who are pulling updates from the Citadel git repository,
please take note: the address of the repository has changed (again).
It is now: https://code.citadel.org/git/citadel.git
There is also a GitWeb instance at that address, for the benefit of anyone
who wants to do
Me too, which is why it took me so long to get out of that headspace. The
truth is, however, Citadel is extremely popular with the operators of small
autonomous sites, not so much the "Exchange killer" I once thought it could
become. That crap all moved to the cloud anyway.
Even now as I am
>That is a hard one. But keeping them separate might help with
>scalability?Â
That was true for WebCit-classic which contains a server-side user interface.
And until a few years ago they could run on separate machines, so you could
have multiple WebCit instances running with a load bal
I've been thinking lately that it might begin to make sense to put the web
server (really, the web API) directly into Citadel Server. Since it's now
a web API and no longer a user interface, it could be an improvement.
*shrug*
I'm starting to be quite happy with this. WebCit-NG is starting to take
the shape of a very competent webmail client. Very clean, too, since it doesn't
need to push a bunch of other garbage in your face the way commercial offerings
do: "Here's a feature we really want you to use and we won't
Whenever I have some time to kill (boring meeting, waiting for something
to finish, etc) sometimes I will start up a test environment in the clown.
I have a subscription to a training site that allows free time-limited
sandboxes
in the three big hyperscale clowns. And I'll run a Citadel Serv
>git push from an airplane? showoff
Yup. :) Unfortunately I was en route to World Of Crap (AWS re:Invent) while
on the plane.
As a t-mobile customer I get a lot of free wifi benefits. I think they provide
the ground links or something.
Ok, so it has to stay, then. Does sendmail not support LMTP?
*sigh*
Another day, another person who won't listen.
Quick poll for fellow Citerati out there. Does anyone think we still need
the `citmail` utility? As a refresher, this program is an MDA, analagous
to `procmail` or `maildrop`, that files incoming mail into Citadel mailboxes
after
So here's what is on my mind regarding this. Suggestions are welcome.
`/var/run` is a good place to put the PID file, which as previously mentioned
is also the lock file. Mounting the same database from two different instances
of `citserver` would be A Bad Thing. (Even though I'm still
Whoa, what font is that ?!!
I am aware that `/var/run` is the conventional location, and may yet move
it there. Doing so would, however, prevent multiple instances of Citadel
Server from running on the same host. There were definitely people doing
that in the past, but I wonder if nowada
Quick update regarding GitLab and git itself:
I've removed SSH access to the repository. All access should be through
HTTPS.
This is much more secure. And yes, you can store your creds. The recommended
procedure if you plan to commit is to grab yourself a Personal Access Token
using
your
Uncensored has been updated from git master as of right now (Saturday morning).
This includes a fix for a security vulnerability that some weirdo expected
me to be able to read in Japanese. :(
Here's what worked for me:
Going through my history... here's what I've got. (This is with `bash`, alter
accordingly for your favorite shell.)
pkg install gmake gcc autoconf automake binutils gsed libtool intltool openssl
curl expat libical libiconv icu bash git
curl https://easyinstall.c
Geez. I can't believe how much more stable Citadel Server is with this new improved backend. it was worth spending most of the summer working on this.
Kitty -- I hope you're having good results on FreeBSD -- please let me know if it works for you.
I've got a test running right now with the load
The above commit has to do with the experience a few of us had with rooms
suddenly containing all new messages.
There was no database corruption. THE DREADED AUTO-PURGER did this, and
it appears to have believed that it was supposed to do that. When we purge
`visit` records, we first bui
Phooey. This was a bit of a brown-paper-bug. I have to do a full release
of the software to correct it.
Dumping and loading a test system was fine, but when I tried to dump a clone
of Uncensored it showed the memory leak.
No pressure at all! Just looking out to see if you are ok and still with
us. But after a break you'll notice that there is likely some tweaking to
do. Specifically, if we are to have multiple versions of the visit record
(and I do think that's a good way to go) we'll need to tweak `ctdldump` to
>The rc system fully detaches it from the entire call stack, in the
>end it just looks like init ran the daemon. The rc system is really
>just a series of shell scripts, nothing too fancy. That's why PID
>files are so important, the rc system uses those to figure out which
>PID to
All righty then. The current code in git master is running on FreeBSD.
Here's the deal:
1. You need ldap client library (`pkg install openldap26-client`) even if
you aren't using LDAP
2. `gmake` and `gcc` are required. It won't build with whatever FreeBSD
is using natively.
3. The c
UUU---
Drive 0 on my desktop is fux0red.
This was my main workstation and also where I had my FreeBSD vm installed.
Time to rebuild.
nd just like that, my main machine at home bit the dust. *grumble*
>So FreeBSD's rc system sends a SIGTERM when you do "doas service
>citserver stop". This should run the same code that cleanly shuts
>down citserver when you do a ".ATN y" from the text client. All the
Perfect. That's exactly the desired behavior, and it will be "even more
that wa
>citserver should exit cleanly when it receives a SIGINT, SIGQUIT, or
>SIGKILL.
>
>This may be why using FreeBSD's rc system caused badness on my
>system.
And it so happens that FreeBSD is the reason I pulled that section of code
out.
This is an opportunity to remove all o
>When I build from master now, when I shutdown citserver, it core dumps right
>after closing the 0d database. Since I had an issue where webcit was core
>dumping and was not re-creatable by anyone else, I am wondering if this
is
>just me.
Remember that discussion, from early Jul
Yet another status update!
With some sleep under my belt and a thankfully slow day at work I've been
able to poke around a little more.
Uncensored crashed a couple of times last night but I think it was because
of a null pointer bug I somehow introduced. Turning off the indexer modul
Yeesh. It's nearly 2:00am and I still haven't figured out the deadlock
problem.
But I can make it appear pretty reliably by doing these things together:
1. loadtest, with 3 threads
2. while true ; do ./sendcommand tdap ; sleep 1 ; done
I also have a room on the test system that p
Yecch. The new code has a deadlock problem. The DB is rock solid though.
On it.
Oh, and in case I didn't make it abundantly clear: the remaining backend
decoupling work is complete, and work on more modern backends can begin soon.
LMDB? MariaDB? SeattleCloud Hosted DB? You name it, we'll be able to do
it.
Ok, I've got my FreeBSD system installed and have begun making changes to
fix any problems that came up.
I am also now running Uncensored on 'git master' (commit
6960933711b8353ad5cddcc17994db670911a1dc)
as of right now. When it comes to dealing with database corruption, in the
words of
Ok, I've got my FreeBSD VM set up and will give the Citadel build a shot today.
\
I'm having *really* good results with this latest commit. I haven't gotten
it to corrupt the db no matter how badly I abuse it. This is the one.
The only issue so far is that under multiple very heavy load tests, I have
gotten it to deadlock. But I don't know whether that's a new proble
All right everyone, this is the one to test. I think I've fixed all of the
conditions that would make it corrupt the database. PLEASE test this version
if you have the ability to. (The version currently in git master, not the
release version on the web site.)
The funny thing is, in the
Geez. The more I look into the Berkeley DB back end the more fucked up I
find it is. Disregard the previous comment about the locking subsystem.
I changed dbenv->open() flags again, this time
DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN |
DB_RECOVER
| DB_THREAD
Some interesting developments here.
I disabled Berkeley DB's locking subsystem, because Citadel handles locking
and concurrency on its own. This seems to have fixed the segfault when it
closes the database environment. I also have not managed to corrupt the
database
no matter how badly
Clarification of the above commit -- because it's important.
When traversing an entire table (as opposed to just fetching one row for
which you already know the key) the backend API now returns both a key and
a value. This is done by returning a "struct cdbkeyval" which simply contains
a
HarlowSolutions: I noticed that the IMAP Flags branch is no longer in the
repo. Are you refactoring it? Do you need anything from me to move forward?
I've been heads down on this stupid database thing lately.
Putting HTML directly into the editor does work if you push the "HTML" button
:)
(...and someday soon, when this whole war on berkeley db thing is finished,
we'll get back to webcit-ng...)
UFarx: the formatter messed up that patch, I think.
just remove the br tags
I ended up having to apply your changes manually. Thank you for the patch, but in the future please either post it as an attachment (good) or upload it to our gitlab as a merge request (better).
And thank you for
UFarx: the formatter messed up that patch, I think. Can you put it in as
a merge request? I'll pm you a login.
>if that's welcome, i'm happy to beef things up a little without
>sacrificing any compatibility.
I'd be pleased to see that. Maybe just do one component first so we can
see what it will look like.
We try to make the build work cleanly on FreeBSD and Linux. There is no
interest in t
All right, that worked much better. The database backend interface has been
reverted to what it was before. The backend for Berkeley DB still needs
thread-specific
data, but now it maintains that data locally to the module instead of tainting
the server with its own needs.
Sensibly, when so
> Therefore I am changing the API so that cdb_rewind() returns a cursor,
>and cdb_next_item() must be passed that cursor with every call. As
Well that was a disaster, and with more people involved now I ought to test
this stuff before announcing an API change.
I think I'm going to
>as for creating a patch, i would but if you wanna switch to cmake
>anyway it might not be necessary anymore. a word of caution though
>regarding cmake:
CMake is interesting and I'm happy to learn about it, but we've got too much
going on right now to think about switching to a new bu
>I already did the account creation thingy.
Ok, I approved the account. I guess I need to set up more alerts on that
sort of thing.
Absolutely! Just go to https://code.citadel.org and create an account. You
will need to use your @uncensored.citadel.org address, at least initially.
I'll approve it in short order.
>If it were up to me, the entire build system would be using CMake.Â
That does look nice and clean.
My objective wasn't to stop using Make, but to stop using the GNU Autotools
stuff. Those tools are built to handle all sorts of obscure edge cases that
don't exist anymore. There just
More notes on the database abstraction layer:
I did point out that the API isn't totally frozen :)
For database cursors, we were keeping cursor variables around
per-thread-per-table,
which requires putting Berkeley DB specific stuff in server/threads.h , and
that's got to go.
T
UFarx: your suggestions for the build system are most welcome.
I don't suppose we could ask you to submit a patch? We have a GitLab instance
at [ https://code.citadel.org ] which accepts merge requests to the repository.
(You'll need to sign up for it using uf...@uncensored.citadel.org a
Work on the abstraction layer for the storage backend continues.
There is a new directory ./server/backends/ that is similar in design to
./server/modules/ for the rest of the server modules. I chose to open a
different
hierarchy because, as previously mentioned, the storage backend gets
All righty, folks ... anyone looking to build a database backend of their
own, please check out the latest batch of changes in the git repository.
It basically works like this:
* database.c has been renamed to database_bdb.c
* database.h now basically declares an API for storage backend
Ok, here's what I'm starting to do. You can totally follow the commit logs
to stay up to date.
database.c has been renamed to database_bdb.c for the obvious reason. (We've
been here before, but it was about 25 years ago when we switched from GDBM
to Berkeley DB.) The database interface
The only thing that's "all the rage with the kids" is *calling* it "NoSQL".
Berkeley DB has been a simple key/value data store for 30 years. And until
recently it's worked fine for us.
To be honest I'm leaning towards LMDB, and for the people running on Raspberry
Pi we will have to say "look, you're just going to have to install the 64-bit
OS before you upgrade"
Read through this presentation on LMDB. It's practically designed for our
use case:
[ http://schd.ws/hos
The options currently under consideration are SQLite, MariaDB, and LMDB.
SQLite: well known, well respected, embeddable. To do it right, however,
would require adding a schema layer. The idea of just throwing away the SQL
semantics and storing all records as blobs kind of bothers me from
Ok, I think I found it.
[ https://docs.oracle.com/cd/E17276_01/html/installation/build_unix_small.html
]
Easy Install now uses the "--enable-smallbuild" flag, which disables verify.
I am going to change it to
--with-cryptography=no --disable-hash --disable-h
I changed your project role from Developer to Maintainer, so hopefully you
can do merges now. I'm also going to try to get GitLab to send more "news"
to this room.
Ok, here is the DB installation:
$MAKE $MAKEOPTS || die
$MAKE install_lib || die
$MAKE install_include || die
$MAKE install_utilities || die
So they *should* be there. Are your tools not on the system, or did it throw
an error message when you tried
Did your db get corrupted because you were playing around with a development
system, or did it happen on its own on a production system?
I've just about had it with Berkeley DB, which is why I've spent a large
number of hours on ctdldump/ctdlload. This is our escape hatch once we have
an alte
Crap. Ok, I'll do the merge, but I really need to figure out what the heck
is happening.
Sorry about that, it just got away from me. I approved it. Looks good.
I agree. The developer who introduced StrBuf and ConstBuf was helpful in
giving us an elastic string class, but I wish he hadn't gone back through
old code and converted stuff without a really deep inspection of how it was
used.
>Not sure why the parameter was empty, but a lot of code accessed ConstStr
>pointers without checking for zero length and pointer was null and could
>coredump, so went through and added checks.
*sigh*
The IMAP server was written completely with straight C pointers and string
buffe
>I am changing to updating the records on they fly rather than upgrading,
so
>the version number is really not that important. New records use a version
>signature and don't compare to the citserver version so no conflict.
Sounds good. We are officially changed over from ctdlmigrate
That is more than sufficient, thanks!
I just pushed an update out that's going to do something ... that will either
win big or lose big.
It's possible that the occasional database corruption has something to do
with aborting so quickly when an occasional segfault happens (for whatever
reason) and it just stops without closing
I would certainly welcome having a chance to test the build on that target
and see how it goes.
Do you have a live system you can use as a test of source data? If so, the
most excellent test would be (after I get ctdldump and ctdlload into the build)
to dump your system, load the dump on another system, and go through it
carefully
to see if the source system was reproduced with 100% fidelit
Oh, and I pushed the release button a few times, so when the visit merge is
done we'll have to adjust the version numbers being applied.
> If not then I'll merge and then troubleshoot. I think when we tell it
>not to squash the commits, it brings in your entire commit history
>followed by a merge commit.
Ok, I went ahead and tried it. It did exactly that. The commits that make
up the merge are written to the log an
F*g hell. I had to restore from a backup this afternoon because my
database
bit the dust.
Looks like it's time to accelerate the replacement of Berkeley DB.
ctdldump/ctdlload is functionally complete, and we can start testing it.
I'm also thinking that maybe we want to split th
I had the same problem, and then I realized I was not logged in. Any chance
that's happening to you?
If not then I'll merge and then troubleshoot. I think when we tell it not
to squash the commits, it brings in your entire commit history followed by
a merge commit.
Geez. Nice find. Approved, of course.
All good, take as much time as you need on the IMAP flags stuff. We'll make
sure we do it right.
As you've undoubtedly noticed if you follow the commit logs, over the last
couple of weeks I've written "ctdldump" and "ctdlload" utilities that can
be used to dump a Citadel database into a flat
To get things started, I have begun work on `ctdldump` and `ctdlload`
frameworks.
I've also improved the Makefile for Citadel Server so that it doesn't require
compiling the entire server every time even one file is touched. I got lazy
when I removed the GNU Autotools and just had it com
1 - 100 of 2223 matches
Mail list logo
