The process is finished, but I don't see any improvement in space.
Sorry for the storm of messages I send; where I write the results that
showed the progress.
In the last messages, I document how it marks a memory error:
May 17 13:28:38 em2 citserver[25444]: db: compacting database 7
>I'll let it run another +4 hours, which is about how long it took to do
>the cleanup process.
Did it finish? The warning about the housekeeping loop is normal if you're
doing a big purge, because the purge runs inside the housekeeping loop so
it won't start another one.
It seems that the 4GB of RAM in the VM was insufficient?
May 17 13:28:37 em2 citserver[25444]: housekeeping: WARNING:
housekeeping loop has not run for 42 minutes. Is something stuck?
*May 17 13:28:37 em2 citserver[25444]: db: BDB3017 unable to
allocate space from the buffer cache*
I'll let it run another +4 hours, which is about how long it took to do
the cleanup process.
Or is really something stuck? I just hope the syslog file doesn't grow
too large.
Done. Now I will wait for the auto-purger process, I already set it up
at 12pm.
One more question:
Is it possible to get a listing with the size of each mailbox/account?
It would be very useful to know this information.
On 5/15/21 2:33 PM, IGnatius T Foobar wrote:
>*Question #1*. How can I
I'm wondering if the process of moving from an already fully allocated VM to one on a local thinpool, if you'll be able to step that VM back from full allocation to dynamic.
Sun May 16 2021 15:45:01 EDT from s3cr3to Subject: Re: citadel-1620778956-x86_64.appimage -- looks like it's working?
On 5/15/21 2:39 PM, IGnatius T Foobar wrote:
You *could* use database_cleanup.sh to recover disk space on an older server.
I seem to remember a post where you mentioned that it was not possible
to do this in my current version.
I'm not sure what your VM storage looks like, but on my VMs
According to Proxmox, just because you've returned freed blocks back to the host system doesn't mean you'll see an increase in your available space in your thinpool. I think it has to do with "largest contiguous blocks" of free space. But I think the bottom line is that you may free space, but
>*Question #2*: Will the reduced DB still be compatible with my current
>8.17 version in production? It would be great to have the downsized DB
>for future migration.
No. Citadel databases are never backward compatible. Once you start up
thew new version of the server with the old
>*Question #1*. How can I configure that option using the AppImage?
>I'm very curious to see what size the DB will be at when I do a big
>purge on some mailboxes that I'm sure are no longer checked.
You can telnet to port 504 and log in as an administrator.
The command you want
On 5/15/21 12:03 PM, IGnatius T Foobar wrote:
But right now we are looking for problems specific to the AppImage
distribution. Can
you confirm that the AppImage is running properly, and it isn't showing any
problems that aren't problems with Citadel in general?
Like these two things?
Thank you Art
you can set the hidden server configuration key called "c_shrink_db_files"
to any value other than 0. This will cause the nightly auto-purger run to
shrink the database files on disk after it has completed its work.
*Question #1*. How can I configure that option using the
>I remembered database_cleanup and wanted to try it after copying my
>backup to the corresponding directory.
Are you running database_cleanup because your db is corrupted? Or are you
running it in an attempt to recover unused disk space?
Please read this knowledge base entry:
> * In the alias section, I don't think I can place all my aliases, it
>seems to be limited to 512 characters, currently I have 779 and
>increasing, of course, I need to debug some aliases.
If I am reading this correctly, you need more space for an account's aliases
than the
As I commented in private messages.
Everything seems to indicate that the amount of characters in the
aliases causes my password to fail.
And it seems that using too many characters when recording the aliases
causes the App to crash.
Regards
I remembered database_cleanup and wanted to try it after copying my
backup to the corresponding directory.
(This is before testing with the "run" parameter)
I did what warning #3 says to do first.
WARNING #3:
Please try "cd /usr/local/citadel/data;
>Yes! it's working: Sending and receiving external mail.
>
>Can I delete the /usr/local/citadel folder to copy my backup and test
>the migration?
Awesome! Yes, go ahead and delete anything you want. I am finished testing
on your system. Thank you for making it available.
Art
Yes! it's working: Sending and receiving external mail.
Can I delete the /usr/local/citadel folder to copy my backup and test
the migration?
I guess I will have to re-edit my production domain and in the
configuration to be able to do tests
I guess, after restoring I will have to
I'm away on business travel this week, so there won't be a lot going on with
Citadel for a few days. Keep that VM running if you are able, and I will
poke in and test things when I can.
We have determined that the main problem, as suspected, is that the server
crashes whenever it makes a
Oh ... I should have thought of that. Instead of sending the VM, you can
just let me log in to it remotely. Why didn't I think of that?
That will be fine, and I will make an effort to give it a try this weekend.
I will probably do something like this: mount the AppImage by hand, jump
Hi Art
In a following private mail, I will send you the data of this VM so you
can try it (of course, it is with "run") for sure if you try to send
mails from it it will fail.
If you let me know, I could leave it installed (with install) for
further testing.
I managed to get the DMZ working
Thank you Paranoid!
Actually my problem is that I made the VM very large (500Gb); and
although I cloned it I expected it to reduce in size when cloning, it
did not.
I have previously pulled the VM from Proxmox as an emergency backup.
I will have to make the VM again with a smaller size and
This is pretty trivial to do... I was able to clone the whole VM to a NAS, then copy it to a different PVE node - and I'm sure I'm not as good at this as you are. I can figure out exactly how I did it, if you run into problems, and document it here.
I need to learn how to clone the VM the current snapshot and see if I
can get it out of storage (NAS).
I guess it would work for you with the selected Format.
I would also have to try to upload it where you indicate at night.
Or devise some way to do so, due to certain restrictions on my side.
Dammit. It's showing the backtrace of an idle thread, not the thread that
crashed.
I don't suppose you have enough bandwidth to simply upload the entire VM?
I don't know if it is useful:
# ls -l /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 May 1 2019
/lib/x86_64-linux-gnu/libc.so.6 -> libc-2.28.so
On 5/5/21 8:58 AM, s3cr3to wrote:
*citserver-backtrace.487*:
/lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f88a4c77840]
*citserver-backtrace.487*:
/lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f88a4c77840]
/lib/x86_64-linux-gnu/libc.so.6(nanosleep+0x40) [0x7f88a4d06720]
/lib/x86_64-linux-gnu/libc.so.6(usleep+0x44) [0x7f88a4d31874]
citserver(go_threading+0x3a) [0x43267a]
citserver(main+0x500)
Ok, good. Now look in /tmp and see if there are any files in /tmp that begin
with "citserver-backtrace". Please post their contents or upload them
somewhere.
After a reboot it can't run
As root:
# uptime
12:25:29 up 0 min, 2 users, load average: 0.00, 0.00, 0.00
# ls -l
total 10392
-rwxr-xr-x 1 root root 10641384 May 2 15:05
citadel-1619988875-x86_64.appimage
# ./citadel-1619988875-x86_64.appimage run
ctdlvisor:
When trying to send an external mail the app crashes.; then I try to run
the app again and it does not start.
./citadel-1619988875-x86_64.appimage run
ctdlvisor: Welcome to the Citadel System, brought to you using AppImage.
ctdlvisor: LD_LIBRARY_PATH = /tmp/.mount_citadee2euCe/usr/lib
Thank you for the heads up.
I am also setting up my virtualization environment so I can do full testing.
On 4/30/21 8:41 AM, IGnatius T Foobar wrote:
Just a heads up to mention that crash reporting (particularly in AppImage
builds) is still the #1 priority. The previous bunch of commits
Just a heads up to mention that crash reporting (particularly in AppImage
builds) is still the #1 priority. The previous bunch of commits were just
some brainless sweeping of the floor to keep me occupied while multitasking.
All right, exit code 139 means the server exited on signal 11, which means
that the AppImage supervisor did the wrong thing. It should have restarted
the server instead of exiting.
I will correct the restart-on-crash problem, but since we also need to find
out *where* yours is crashing, we
Good day
This is the network configuration of the VM, I need to give myself time
to install a second interface (dmz):
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
allow-hotplug ens18
>And yes, when trying to send messages it fails; I can send messages to
>the "admin" account itself.
Damn. I did a fresh install of Debian 10, minimal with no extra software
added, on the exact virtual hardware you described in your last post. I still
can't get it to fail.
Can
Hi Art.
The results I reported a couple of days ago are with the DB itself that
creates the test.
At this time I have not backed up the DB in production for use. I will
do it these days in the evening, as I have to stop my server to be able
to copy it.
# uname -a
Linux em2
Ok, so you're using stock Debian 10 (64-bit x86, I assume) and the AppImage
declares that it is compatible. You are attaching it to a copy of an existing
database. Local mail works fine, but any attempt to deliver Internet mail
results in a server crash.
Do I have it clear now?
The appimage is compatible
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
# ./citadel-1617821022-x64.appimage test
/tmp/.mount_citadeSJ4RAB/usr/local/citadel/citserver: *binary
>And although I discovered my mistake and now this VM works as I know it
>should, in my test it still fails to use the "test" of the AppImage, if
>I try to send some mail it just closes the App.
That's progress. :)
So let me see if I understand correctly. If you run the
I misread the message about missing commands in Debian (that was during
installation, not after installation).
And although I discovered my mistake and now this VM works as I know it
should, in my test it still fails to use the "test" of the AppImage, if
I try to send some mail it just closes
Thank you. This got me sorted out on that. As described in Linux> I'm still having some issues... but, that is to be expected, I suppose. :)
Wed Apr 14 2021 17:59:12 EDT from s3cr3to Subject: Re: Citadel app image install
If you try
./citadel-nn-xxx.appimage --help
As an unrecognize
If you try
./citadel-/nn-xxx/.appimage --help
As an unrecognize option, it will show you the supported parameters and
command: run, test, install, database_cleanup
To install you need:
./citadel-/nn-xxx/.appimage install
Greetings Art
The results of your tests continue to baffle me.
I agree with you, this VM is really weird.
I have reinstalled the VM with debian-10.7.0-amd64-DVD.iso and the
problems are still occurring, I think I found a post (I didn't save it)
where people report similar problems where
Thanks Art.
I'm going to have to reinstall this VM, for some reason I don't know;
it's showing strange behavior; things that should work don't: I don't
have fdisks, lvm2 commands, and things that a fresh install should have.
Regards
On 4/13/21 12:08 PM, IGnatius T Foobar wrote:
>So, it
>So, it won't help if I add another virtual disk with more storage if I
>can't change the /tmp dump path. Rigth?
Correct. I've removed the misleading guidance from the help text. The
intermediate
dumps must be on /tmp and this cannot be changed.
If you add another virtual disk
So, it won't help if I add another virtual disk with more storage if I
can't change the /tmp dump path. Rigth?
On 4/12/21 2:55 PM, IGnatius T Foobar wrote:
>For the latter, how should I run the database_cleanup to get it to take
>the working path?
database_cleanup always stores
>For the latter, how should I run the database_cleanup to get it to take
>the working path?
database_cleanup always stores its intermediate dumps to /tmp.
I really should change the help text which suggests otherwise.
That's fine with me, let me finish with a good workload that has piled up.
I will restore the "clean" snapshot (with the old data backup intact
from the production server.), update it and try to put an extra disk in it:
For the latter, how should I run the database_cleanup to get it to take
The results of your tests continue to baffle me. But I am happy to see it,
because if I can fix it for you, that means there are others who would have
had the same result, and we are preventing all those support requests from
happening.
If you have the patience, I would be VERY grateful
Hi Art. I answer each point.
1. I need to add another disk because I don't have enough space to run
the database_cleanup. *But how I should specify a location*? The usage
lacks of that info:
*usage*: /home/sys1/citadel-1617821022-x64.appimage [-h
data_directory] [-p http_port] [-s
>1. What effect would it have if I run *database_cleanup* with my
>backup+data in the DB?
Yes, I'd try database_cleanup just to see what it does.
Also, I have to ask: is the copy of your database actually in
/usr/local/citadel
or is it somewhere else?
You should also try
Greetings Art
I manage to run the AppImage, but I can't login
* Using my correct username and password, it does not show any message
and does not change the page.
* If I try to use invalid credentials, an error message tells me that
the user does not exist or the password is invalid.
I'm happy to hear your feedback. One of the reasons I post these internal
updates is to give you (and others who are following the conversation) an
idea of the progress being made. This is how you know what is being worked
on and how close to completion each element has progressed.
Building
Hi Art
I started reading daily since the promise of the container in Docker,
and now with the AppImage, I check daily especially your updates; but I
almost don't comment so as not to invade the thread of your messages.
About the SSH connection freezing, an anecdote:
If I try to work
Finally ... got a working clone of Uncensored. It took eight hours to copy.
When I do it for real I will have to schedule some actual downtime. :(
I am going through the clone system and so far it appears to be a clean replica
of its source (but running in 64-bit of course). New
For anyone following the drama:
No matter what I tried, the SSH connection would eventually choke and die.
Small data, big data, keepalives, whatever I tried it would still die during
the replication. So I removed SSH from the data path. Trying it now using
a direct connection to the
Geez. It's amazing how badly broken the migration tool was. Thankfully
I am running it against a clone of Uncensored and not the real Uncensored
(btrfs is teh r0x0r) because you folks would be miserable getting on here
:)
Right now I'm dealing with a problem where both ends get a
This sucks and blows. I know people hate it when we deprecate the export
format, but as the commit says, it wasn't working anyway.
I've been procrastinating for years on upgrading Uncensored to 64-bit. It
needs to be done, because I want to use the AppImage on my production system.
This
If you have to install any dependencies, then the AppImage has failed to live up to its promise.
Hang tight for now. For the past couple of months I've been getting occasional crashes on my production system (this one) and a stack trace shows that it's happening inside of OpenSSL, which is
Hi
Maybe I need to install some package or library beforehand.
I'm tempted to reinstall from scratch the VM, just to be sure it's not a
problem on my side.
This new version of Debian has a lot of changes that even I haven't got
to know yet; it is very different from Debian 6 which was
Ever since moving my hosting operation back home, about once a day I would enter my office and find my computer fans screaming like a jet engine, with WebCit on the production system being the culprit. I finally tracked down the problem. Someone "optimized" a function some number of years ago
I'm less concerned with the hostname stuff because that's largely just configuration and has less to do with the AppImage.
The service crashing on outbound email is more concerning. Are there core files left behind? We need to figure out where the crash is happening. I may have to just try to
Hi Art.
Using Chrome/Linux Manjaro and the "Feb 25 22:21
citadel-1614316873-x64.appimage" with install.
* I change the password of the admin user, and the email address (it
works)
* I made a new user, assigned him an email and an alias.
* If I check the "Global Address Book", it shows
Interesting. It sounds like we're making progress. I haven't tried it on the public Internet yet, but I have a couple of extra public IP addresses so I'll give that a try next.
Have you considered that the AppImage can update the DB version of my current server?
Can the installation with
Good day Art.
Note: I don't have this VM connected directly to the Internet, it is
using a gateway with static IP to test only the mail sending. Will it be
enough for testing?
With this version. Things that worked:
* login in webcit
* I changed the admin password and it worked
* create a
>You are connected to a Citadel server running Citadel 930.00.
>In order to run this version of WebCit you must also have Citadel 931.00
or newer.
Ok, I believe I have this fixed, but I definitely need to get my builds in
order.
For 64-bit AMD/Intel, try this one:
wtf? How did it build 930 from a version 931 repo?
Time to build a few more test machines :(
I got this message in Firefox
You are connected to a Citadel server running Citadel 930.00.
In order to run this version of WebCit you must also have Citadel 931.00 or
newer.
Process are:
# ps -eaf|grep cit
root 611 1 2 08:50 ? 00:00:00 ./citadel-1613953314-x64.appimage
run
> I'm looking at the complaints in the Citadel Support room from some
>users who are noting the wrong addresses appearing in the to/from
>headers on Internet mail under certain conditions. That looks like
>something that needs to be addressed sooner than later, so I'll be
(heart)
Fri Feb 12 2021 19:59:45 EST from IGnatius T Foobar Subject: Re: Yak shaving!
Yeah. It isn't glamorous but time spent on stability and consistency is time well spent.
Yeah. It isn't glamorous but time spent on stability and consistency is time
well spent.
Glad it is happening. I haven't done much testing other than the initial install of it. Had other fish frying myself. But I've been checking in and watching.
Wed Feb 10 2021 10:06:00 EST from IGnatius T Foobar Subject: Yak shaving!
So ... lots of stuff going on, as you can see in the
>Regarding item 4. What will be the mechanism to stop the service using
>the AppImage, just Ctrl-C?
>
>I wonder if it will be with another switch:
># AppImage *stop*
If you're running the AppImage in the foreground, Ctrl-C will still be the
correct way to terminate it.
@Art
Regarding item 4. What will be the mechanism to stop the service using
the AppImage, just Ctrl-C?
I wonder if it will be with another switch:
# AppImage *stop*
or a separate App/command that generates a 'stop flag file' inside the
directory and when the service detects it, it stops
>ctdlvisor: pid=449 exited, status=26880, exitcode=105
Hmm. Exit code 105 means it couldn't open the database.
What version of Berkeley DB is running on the production system you cloned?
If it is an Easy Install system, it's probably 6.2.32? That could be the
problem. Your
Snapshot restored, running as root: it starts, but it ends soon.
# md5sum Citadel-x86_64.AppImage
51d5fe3e8274b9b004d846fa1a473b34 Citadel-x86_64.AppImage
# ./Citadel-x86_64.AppImage run -h/usr/local/citadel
ctdlvisor: Welcome to the Citadel System, brought to you using AppImage.
The correct way to run it would be
./Citadel-86_64.AppImage run -h/usr/local/citadel
Do not specify "/data". That will be added by the program.
wrong way, running whith the directory (/usr/local/citadel/data),
re-creates the folders inside like in /usr/local/citadel (restoring VM
snapshot):
-rw--- 1 root root 380760064 Jan 25 19:00 cdb.00
-rw--- 1 root root 176128 Feb 2 10:55 cdb.01
-rw--- 1 root root
I try this and seems to run, but I can login, with my user & password,
If this is the right way to run it, how can I test it from a terminal?
# ./Citadel-x86_64.AppImage *run -h /usr/local/citadel/data*
ctdlvisor: Welcome to the Citadel System, brought to you using AppImage.
ctdlvisor:
This is what it shows the 3 times I tried to run it, the services stop.
This is a fresh snapshot, with the DBs intact in the previously used
folder, you can see that it process the citadel.* files since they are
gone, but it always leaves a citadel.lock file in their place
#
Ugh. I opened up a pandora's box with this anti-LFHS thing. Lots of code
to rewrite. Uggghh
>It seems to me that the App is not updated, I download it and it is
>dated *January 15th*. And if I run it I get errors that had already been
>solved.
Weird. I must have only done the ARM version. But I'm not going to waste
your time because I found some problems related to the
It seems to me that the App is not updated, I download it and it is
dated *January 15th*. And if I run it I get errors that had already been
solved.
The md5sum is the same on both files downloaded (on jan 24 & 27):
$ md5sum Citadel-x86_64.AppImage20210115
587e56bbd11511fb4cfe7eae34e7f5d8
>I'm going to restore a snapshot of my VM so I can try again; when I try
>to log in with http; and the "citadel" process just crashes.
Yes, definitely use a new snapshot. If you ran the previous version of the
appimage, you can pretty much count on any database it touched being
Thanks!
I'm going to restore a snapshot of my VM so I can try again; when I try
to log in with http; and the "citadel" process just crashes.
webcit[2802]: POST /ajax_login_username_password HTTP/1.1
citserver[2801]: [(not logged in)(0)] USER ***
citserver[2801]: user_ops:
I don't see that anyone ever responded to this. If the AppImage install is creating a self-signed cert at startup - if that cert is a 1024 bit key - it isn't going to work and modern browsers will reject it. You need at least a 2048 bit key - otherwise, you'll get this error when you try to
AppImage development is teaching me something about the rest of the server:
we spend way too much time "cleaning up" when the server is exiting. I am
now removing thousands of lines of "exit cleanup hook" code that doesn't really
need to be there. Kudos to whoever wrote it, they had the
No, keep using whatever you were using. We want the AppImage to be tested on as many different systems as possible. If everyone is using what I'm using, that misses the point.
I'm working on this today and I'm going to try to figure out why WebCit keeps going catatonic on us (insert your
On 1/22/21 8:59 AM, IGnatius T Foobar wrote:
That's kind of odd. Ctrl-C works for me on both x64 and ARM.
At first, it did work. It seems that when the citadel process crashes
only webcit is running and Ctrl-C stops working.
I'm concerned about the "buffer overflow detected" error.
That's kind of odd. Ctrl-C works for me on both x64 and ARM.
I'm concerned about the "buffer overflow detected" error. That's a new one
for me. What operating system are you trying it on?
¿How can I test from a terminal and stop the AppImage?
After the following error I report, I can't stop the App using
Control-C. Maybe I should just kill the processes? So far I have only
restarted the VM during these errors.
> ps -eaf|grep cit
> root 435 1 0 10:27
On 1/21/21 9:51 AM, IGnatius T Foobar wrote:
Excellent. Happy to hear that it works.
...
* Then, I will only need to redirect the traffic according to this
VM (from my router/fortigate)
The answer is yes, that will work and it is done that way all the time,
but you're getting
> * Will it be possible to recover space from the DB by compressing
>what is deleted? (because I see that the DB is still +90GB)
In the version of Citadel you are running in the AppImage, you should now
be able to set the hidden "c_shrink_db_files" configuration option, which
will run a
Excellent. Happy to hear that it works.
For better performance, do I need to install a local DNS like BIND?
That's up to you and not related to Citadel. If you want a caching DNS on your local host, I would recommend dnsmasq over BIND, but again, Citadel doesn't care.
If my current
I forgot to ask this:
Will it be possible to recover space from the DB by compressing what is deleted? (because I see that the DB is still +90GB)
Kudos
** I have a few questions at the end, thanks for answering.
New backup, now (before running):
# ls -la /usr/local/citadel/ && ls -la /usr/local/citadel/data/total 56 drwxr-xr-x 3 root root 4096 Jan 20 20:27 . drwxr-xr-x 11 root root 4096 Jan 13 14:11 .. -rw--- 1 root root 3256 Jan 20
>Citadel Server found BOTH legacy and new configurations present.
>Exiting to prevent data corruption.
There's your problem. If you ran Citadel server without citadel.config and
citadel.control present, it would have generated new configurations. Now
it sees two conflicting
Now te application behaved differently, I still can't connect to port
504; citserver is the process that listens on that port, right?
Maybe I should(?) delete all the files and copy a new backup.
citserver[5657]: libcitadel(unnumbered)
citserver[5657]: main: creating lockfile
Definitely move citadel.config, citadel.control, and refcount_adjustments.dat down into /usr/local/citadel/ (not in data/). We want the database directory to contain only the database.
Thank you!
Both files exists in the same path as the database and copied on the new
server. This system is used only for email.
# ls /var/lib/citadel/data/ -lh
total 91G
-rw--- 1 citadel citadel 363M Jan 19 14:46 cdb.00
-rw--- 1 citadel citadel 48K Jan 19 14:48 cdb.01
-rw--- 1
201 - 300 of 1530 matches
Mail list logo
