Launchpad has imported 38 comments from the remote bug at
https://bugzilla.mozilla.org/show_bug.cgi?id=1878541.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2024-02-04T09:38:58+00:00 Omry Yadan wrote:

Created attachment 9378154
Screenshot_9.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0

Steps to reproduce:

I am not sure how to trigger it.
I get system warnings about the disk being full and then I see a nstmp file 
using 1TB.
I deleted it, and hours later it happened again.
This have happened a few times for me in the past few weeks, but it seems to be 
happening more and more.

Additional context:
I am using imap. The entire mailbox on the server is 9.9GB.

Thunderbird 123.0b2 (64-bit)

OS:
Edition Windows 11 Pro
Version 22H2
Installed on    ‎22/‎01/‎2023
OS build        22621.3007
Experience      Windows Feature Experience Pack 1000.22681.1000.0

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/0

------------------------------------------------------------------------
On 2024-02-05T10:34:18+00:00 Mkmelin+mozilla wrote:

That's only used for compacting folders. 
https://searchfox.org/comm-central/rev/1452d8f1e1582cc44529f638e1eba1c7b3804fe4/mailnews/base/src/nsMsgFolderCompactor.cpp#393
The sent folder is also quite large, 50GB.

Is auto compact enabled, or do you do compaction only when it asks/or
manually?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/1

------------------------------------------------------------------------
On 2024-02-05T11:19:45+00:00 Omry Yadan wrote:

> Is auto compact enabled, or do you do compaction only when it asks/or
manually?

I didn't change the setting, so I think it's probably enabled by default.
I dug through the settings now and was not able to locate it.

> The sent folder is also quite large, 50GB.
I noticed that too. I deleted it and it. The reconstructed file is 16GB.

> That's only used for compacting folders. https://searchfox.org/comm-
central/rev/1452d8f1e1582cc44529f638e1eba1c7b3804fe4/mailnews/base/src/nsMsgFolderCompactor.cpp#393

It looks like this function is not always cleaning up after itself.
For example, if an exception is thrown here,  CleanupTempFilesAfterError is 
never called:
https://searchfox.org/comm-central/rev/1452d8f1e1582cc44529f638e1eba1c7b3804fe4/mailnews/base/src/nsMsgFolderCompactor.cpp#411

Why not wrap it with a big "try finally" to always delete the file if it
exists when the function returns?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/2

------------------------------------------------------------------------
On 2024-02-05T11:20:48+00:00 Omry Yadan wrote:

There is also the underlying question of why on earth the function is
creating a file that is 1TB in size. It sounds like there is a quadratic
usage of disk space in the compaction algorithm.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/3

------------------------------------------------------------------------
On 2024-02-05T11:30:25+00:00 Omry Yadan wrote:

There is an attempt to clean up the file if it's still there in the destructor:
https://searchfox.org/comm-central/source/mailnews/base/src/nsMsgFolderCompactor.cpp#171-177

However, it only kicks in if NV_FAILED returns true.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/4

------------------------------------------------------------------------
On 2024-02-05T13:11:17+00:00 Anjeyelf wrote:

Sounds like the Sent-1 file is so big it's not possible to do compacting
maybe because either Thunderbird is interrupted during the compacting
eg: shutdown, crash or receiving mail into the folder being compacted,
maybe another program is scanning the file or it's possible that you
might not be able to compact a large folder because you don't have
enough free disk space or memory or there is some type of corruption in
the file.  50GB is very large for a single text file and it looks like
it's been failing to compact for a while, so is full of old deleted
mail.

Does it say anything in the Error console ?
Open Error console (Ctrl + Shift + J') or 'Tools' > 'Developer Tools' > 'Error 
Console'
Clear Error console - top left bin icon

right click on 'Sent' folder and select 'compact'
If compacting does not work - Sent-1 is still 50Gb and  nsmtp appears.
What do you see in Error Console - please post an image.

Either way you need a solution to overcome the current issue.

Suggestion:
- Create a new folder called 'Sent Store'.
- Move all the wanted sent emails into 'Sent Store' as a precaution to make 
sure you do not lose them.

So now Sent appears empty.

First try to fix Sent msf file:
- Right click on Sent folder and select 'Properties' - click on 'Repair' folder 
and click on OK.

then try to compact Sent folder:
- Go into 'Offline' mode to ensure nothing gets received into 'Sent' and then 
right click on 'Sent' to see if it manages to compact the Sent folder.

If it fails to compact and produces a new nsmtp:

- Exit Thunderbird
- Access profile
- Delete the nsmtp file and also the Sent-1 mbox and Sent-1.msf files.
- Empty the computer Recycle Bin.

Restart Thunderbird and the Sent files should get created.

It might be useful to keep the 'Sent Store' folder for all the older
sent mail or Archive it or if you really want to you can move the emails
back into the 'Sent' folder.

Please report on whether this fixed the problem.

> auto compacting - I dug through the settings now and was not able to
locate it.

- Settings > General
- scroll down to 'Network & Disk Space'
- Under 'Disk Space'
- Is this checkbox selected 'Compact all folders when it will save over' ?
- What is the number of MB set ?
- Is this checkbox selected 'Ask every time before compacting' ?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/5

------------------------------------------------------------------------
On 2024-02-05T13:29:49+00:00 Omry Yadan wrote:

Thanks for the detailed answer.

> Sounds like the Sent-1 file is so big it's not possible to do
compacting maybe because either Thunderbird is interrupted during the
compacting eg: shutdown, crash or receiving mail into the folder being
compacted, maybe another program is scanning the file or it's possible
that you might not be able to compact a large folder because you don't
have enough free disk space or memory or there is some type of
corruption in the file. 50GB is very large for a single text file and it
looks like it's been failing to compact for a while, so is full of old
deleted mail.

I have 1 TB of free storage so it's not out of disk space (except when 
Thunderbird fills it all up with nstmp).
Normally, I never delete things from sent items (I have messages as old as 2005 
there).
I already deleted the Sent-1 file, thinking Thunderbird will rebuild it from 
the server but it looks like it's satisfied with Sent-1.msf and did not 
recreate Sent-1.
I did not run into the problem since I opened the bug report.

> Does it say anything in the Error console?

Manually compacting `Sent` works now and there is nothing printed to the
Error Console.

> Is this checkbox selected 'Compact all folders when it will save over' ?
Yes.

> What is the number of MB set ?
20 MB

> Is this checkbox selected 'Ask every time before compacting'?
No.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/6

------------------------------------------------------------------------
On 2024-02-05T15:42:40+00:00 Anjeyelf wrote:

> I have 1 TB of free storage so it's not out of disk space

It's not about the actual size of total free disk space, but maybe due
to not being able to open such a large file in the current folder OR RAM
not being able to handle such a large file.

> I already deleted the Sent-1 file, thinking Thunderbird will rebuild
it from the server but it looks like it's satisfied with Sent-1.msf and
did not recreate Sent-1.

That sounds like you are not downloading full copies of emails into that 
particular folder, only headers are being downloaded, hence you only see a .msf 
file.   That Sent.msf file may be much smaller as well. If you were intending 
to create a backup of profile then you would need to ensure you are downloading 
full copies of emails first.
- 'Account Settings' > 'Synchronisation & Storage'
- click on 'Advanced' button and select checkboxes for all folders which you 
want to download emails.

> Is this checkbox selected 'Ask every time before compacting'?
> No.

I would suggest you change that option to Yes by selecting that
checkbox. Why? because then you will be aware that compacting is
occuring meaning you must not exit Thunderbird or eg: send email when
it's compacting the Sent folder.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/7

------------------------------------------------------------------------
On 2024-02-06T05:58:12+00:00 Omry Yadan wrote:

Created attachment 9378449
Screenshot_1.png

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/8

------------------------------------------------------------------------
On 2024-02-06T05:58:26+00:00 Omry Yadan wrote:

Created attachment 9378450
Screenshot_2.png

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/9

------------------------------------------------------------------------
On 2024-02-06T05:59:17+00:00 Omry Yadan wrote:

> It's not about the actual size of total free disk space, but maybe due
to not being able to open such a large file in the current folder OR RAM
not being able to handle such a large file.

I understand that there could be many reasons a failure, but regardless of the 
reason - the temp file should be deleted afterwards.
In fact, it should probably be cleared on startup as well if it's there.

I ran into a problem where the indexing process kept getting stuck on message 
25 of many thousands in some folder.
I backed up the profile and cleared the imap mail directory and allowed 
Thunderbird to download everything again.
Manual compacting finishes for Sent items without an error (I was never able to 
reproduce it manually).
However, indexing still gets stuck on the same folder. The Error Console does 
not show anything that is obviously related (see attachments).

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/10

------------------------------------------------------------------------
On 2024-02-06T09:27:57+00:00 Vseerror-i wrote:

Thanks for adding more details.
I see nothing interesting in the error console. 

Can't stress enough that you should attempt to trim your sent folder size.  
Suggest target to reduce by half.
1. Add the message size column, sort on it, and delete as many of the larger 
messages as possible.  Messages which contain attachments, for example images, 
are twice as big as their attachment sizes and so very inefficient -  if their 
contents have been saved elsewhere you should delete the message or delete the 
attachments from the message.
2. Sort on message date and delete as many of the older messages as possible.
3. Archive remaining messages by year.

Some additional thought is needed to resolve the compact issue.  One
possibility is antivirus interferring.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/11

------------------------------------------------------------------------
On 2024-02-06T11:25:57+00:00 Omry Yadan wrote:

> Can't stress enough that you should attempt to trim your sent folder
size. Suggest target to reduce by half.

I think you are still reacting to the inexplicit 50GB from the first screen 
shot.
My current size (after fully downloading the Sent folder) is 3GB.

Also, to clarify: I did not reproduce the compact issue when it creates a 
massive file since I deleted my entire IMapMail dir and reconstructed it.
At this point I think there was some corruption.
In general, I think at this day and age it would be better to use sqlite to 
store the messages instead of creating a custom flat file implementation. That 
would be much more robust and reliable if a bit slower.

Re virus scanner:
I disabled the realtime windows virus protection and it doesn't make a 
difference for the indexing.
I am also running a virus scanner on the entire mail directory in the mail 
server now to be sure there are no mines there that can trigger the realtime 
virus protection.

When I disable the indexing for the 2009 folder, Thunderbird is able to 
download and index everything.
At this point I can either leave it excluded, or try to hunt down the 
problematic message.
Is there a way to enable verbose logging for the indexer?
If not, I can probably do a binary search by moving messages around in that 
folder until I find the offending message.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/12

------------------------------------------------------------------------
On 2024-02-06T15:49:48+00:00 Vseerror-i wrote:

Thanks for setting me straight.

> In general, I think at this day and age it would be better to use
sqlite to store the messages instead of creating a custom flat file
implementation. That would be much more robust and reliable if a bit
slower.

It is headed in this direction - work is in progress.

> I disabled the realtime windows virus protection and it doesn't make a
difference for the indexing.

Good to know.

> When I disable the indexing for the 2009 folder, Thunderbird is able to 
> download and index everything.
> At this point I can either leave it excluded, or try to hunt down the 
> problematic message.
> Is there a way to enable verbose logging for the indexer?

I don't foresee us attempting to twiddle old code that is soon to be
replaced - cost/benefit - especially considering AV is somehow
implicated.  Unless there is a clear smoking gun or the code change out
gets delayed.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/13

------------------------------------------------------------------------
On 2024-02-06T16:09:54+00:00 Omry Yadan wrote:

> It is headed in this direction - work is in progress.

Good to hear.

> I don't foresee us attempting to twiddle old code that is soon to be
replaced - cost/benefit - especially considering AV is somehow
implicated. Unless there is a clear smoking gun or the code change out
gets delayed.

Is the indexer being rewritten as well?
It feels like it hits a bad message and either gets stuck processing it or 
maybe an error is causing endless retries.
I was hoping to be able to pinpoint the offending msg via verbose logging of 
the indexer (that hopefully logs details about each msg as it's indexing it) 
and provide the offending msg to a new bug in the indexer (if the problem is of 
relevance to other people).

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/14

------------------------------------------------------------------------
On 2024-02-08T19:48:50+00:00 Omry Yadan wrote:

I am able to reproduce by manually compacting my Sent items.
The Sent file grew back to 47GB, even though the sent items folder on the 
server is only 1.5GB.
When compacting, I can see nstmp growing beyond the size of the Sent file which 
I believe should never happen while "compacting".
As a side note, exiting TB with alt-f4 while it's compacting does not delete 
nstmp.

A few additional observations:
I deleted Sent and Sent.msf and redownloaded the the folder fully.
As mentioned, the size on the server is around 1.5GB.
The local size on disk after downloading is matching that.
One strange thing, toward the end of the download, Thunderbird got stuck with 
the last few messages.
After restarting it it wouldn't download any new messages and things seemed ok.

Manually compacting does not do anything at this stage (finishes
immediately, and report 0KB saved).

I should still be able to reproduce by unpacking the 50GB Sent and
corresponding msf.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/15

------------------------------------------------------------------------
On 2024-02-08T19:49:16+00:00 Omry Yadan wrote:

Created attachment 9379077
Large Sent and larger nstmp.png

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/16

------------------------------------------------------------------------
On 2024-02-08T19:50:00+00:00 Omry Yadan wrote:

Created attachment 9379078
Sent reconstructed from server.png

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/17

------------------------------------------------------------------------
On 2024-02-09T17:08:09+00:00 Anjeyelf wrote:

> As a side note, exiting TB with alt-f4 while it's compacting does not
delete nstmp.

If  the compacting process is terminated by eg: exiting Thunderbird then it 
will create and leave the nstmp file - not delete it. Hence why you should not 
interupt the compacting process.
You see nstmp because the compacting process was interupted or could not 
complete for some reason. Maybe the Sent file was corrupted or contained a 
corrupted email.

In topicbox beta forum there has been a report of large nstmp file.  The 
compacting process was not able to complete hence nstmp file of enormouse size.
Users reporting the issue did have some emails that were missing headers and 
garbled text.
After deleting those 'bad' messages, the compacting process worked ok.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/18

------------------------------------------------------------------------
On 2024-02-09T17:59:34+00:00 Omry Yadan wrote:

Thanks Anje,

> If the compacting process is terminated by eg: exiting Thunderbird
then it will create and leave the nstmp file - not delete it. Hence why
you should not interupt the compacting process.

I would call this is a bug.
1. The tmp file should not be left there on orderly exit. (e.g. not kill -9 or 
a power outage).
2. The tmp file should probably be cleaned up on startup.

> You see nstmp because the compacting process was interupted or could
not complete for some reason. Maybe the Sent file was corrupted or
contained a corrupted email.

Everything is possible of course.
As a user I find it hard to debug though.  I have 26k mails in my Sent items.
Whatever corruption causes it to grow from 1.5GB to 50GB, it happened at least 
twice.

> In topicbox beta forum there has been a report of large nstmp file.
The compacting process was not able to complete hence nstmp file of
enormouse size.

Can you point to that message?
I did not read the code of the compacting process, but I would guess it's 
simply iterating on the messages and writing each non-deleted message to nstmp, 
and then swaping nstmp with the flat file.
If this is the case, the only situation where the compacting process would get 
stuck is if the iteration on the mails in the folder is infinite.
It should be possible to detect such a case (simply by observing that we 
iterated more emails than exist in the folder) and automatically repair the 
folder by redownloading it from the server.

> Users reporting the issue did have some emails that were missing headers and 
> garbled text.
After deleting those 'bad' messages, the compacting process worked ok.

Any help in detecting such 'bad' messages would be appreciated.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/19

------------------------------------------------------------------------
On 2024-02-12T16:42:24+00:00 W-alessandro wrote:

A few people experienced this so with can confirm it.
Ben, can you please investigate this and give it top priority?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/20

------------------------------------------------------------------------
On 2024-02-15T18:06:13+00:00 Mkmelin+mozilla wrote:

Sorry, that was meant for bug 1872849

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/21

------------------------------------------------------------------------
On 2024-03-10T07:18:58+00:00 Traderwin wrote:

I have this exact issue, dozens of useless nstmp files wasting 150 GB
for an inbox weighing only 15 GB... This is a serious bug. Why not
always clean up orphan files upon startup? What to do in the meantime?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/22

------------------------------------------------------------------------
On 2024-04-02T12:09:06+00:00 Vseerror-i wrote:

(In reply to Erwin from comment #23)
> I have this exact issue, dozens of useless nstmp files wasting 150 GB for an 
> inbox weighing only 15 GB... This is a serious bug. Why not always clean up 
> orphan files upon startup? What to do in the meantime?

Compact overhaul is nearing completion.  Then we can give better advice.

Do you have automatic compact disabled?

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/23

------------------------------------------------------------------------
On 2024-04-02T12:38:52+00:00 Traderwin wrote:

(In reply to Wayne Mery (:wsmwk) from comment #24)
> (In reply to Erwin from comment #23)
> > I have this exact issue, dozens of useless nstmp files wasting 150 GB for 
> > an inbox weighing only 15 GB... This is a serious bug. Why not always clean 
> > up orphan files upon startup? What to do in the meantime?
> 
> Compact overhaul is nearing completion.  Then we can give better advice.
> 
> Do you have automatic compact disabled?

Automatic compact is enabled (provided it'll save at least 200 MB).

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/24

------------------------------------------------------------------------
On 2024-04-04T11:49:23+00:00 Vseerror-i wrote:

> Automatic compact is enabled (provided it'll save at least 200 MB).

For now, suggest you disable automatic compact.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/25

------------------------------------------------------------------------
On 2024-04-05T04:30:38+00:00 Vseerror-i wrote:

omry, Are you still seeing this problem with 125.0b1 or newer?

The patch for bug 1872849 is in 125.0b1 (available 2024-03-20/21) and
newer.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/26

------------------------------------------------------------------------
On 2024-04-05T13:22:27+00:00 X-dan-o wrote:

(In reply to Wayne Mery (:wsmwk) from comment #27)
> omry, Are you still seeing this problem with 125.0b1 or newer?
> 
> The patch for bug 1872849 is in 125.0b1 (available 2024-03-20/21) and newer.

I was just on a version of 125.0bx (< beta 4, but i was on 125) and had
this happen. just reported yesterday [bug
1889755](https://bugzilla.mozilla.org/show_bug.cgi?id=1889755)

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/27

------------------------------------------------------------------------
On 2024-04-05T14:13:26+00:00 Mkmelin+mozilla wrote:

*** Bug 1889755 has been marked as a duplicate of this bug. ***

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/28

------------------------------------------------------------------------
On 2024-04-05T14:15:58+00:00 Omry Yadan wrote:

I did not see it for a while, but I did have my auto-compaction disabled per 
troubleshooting done here.
I re-enabled it and report here if I see it again.
I just restarted my thunderbird for an update, my current version is 125.0b4.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/29

------------------------------------------------------------------------
On 2024-04-06T11:45:37+00:00 Ezh wrote:

Using 125b4 - still see nstmp fill the whole disk.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/30

------------------------------------------------------------------------
On 2024-04-06T12:16:00+00:00 Omry Yadan wrote:

Clearing needinfo under the assumption that it still exists, I will
still report in I run into it myself with 125b4 or newer.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/31

------------------------------------------------------------------------
On 2024-04-07T01:07:41+00:00 Benc-q wrote:

I _think_ this might the same issue as Bug 1890135.
If so, the problem occurs when you compact folders which have had _all_ their 
messages deleted.
Can anyone confirm this?

If it's not the case, then I have actually re-written all message
compaction code, and will be aiming to get that reviewed to land in the
next week or so. It's much simpler and robust than the old compaction
code, and much better at cleaning up after errors!

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/32

------------------------------------------------------------------------
On 2024-04-07T02:41:53+00:00 Omry Yadan wrote:

No, the problem here is not related to compacting a folder after deleting all 
the messages.
Good to hear you rewrote that code. Can you point to your diff?

A few points/ideas that would make things more robust (some mentioned above):
1. Diligent error handling while writing the tmp file.
2. Making sure that multiple processes/threads would not write to the same tmp 
file concurrently.
3. Always cleaning up compaction tmp files on error.
4. Clean up compaction tmp files on startup on case of an unclean shutdown 
during compaction.
5. Ensure that the number of written messages while compacting does not exceed 
the number of messages in the folder being compacted.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/33

------------------------------------------------------------------------
On 2024-04-10T01:53:00+00:00 Benc-q wrote:

(In reply to omry from comment #34)
> No, the problem here is not related to compacting a folder after deleting all 
> the messages.
> Good to hear you rewrote that code. Can you point to your diff?

See Bug 1890448. There's a patch in phabricator attached there.

> A few points/ideas that would make things more robust (some mentioned above):
> 1. Diligent error handling while writing the tmp file.

Indeed.

> 2. Making sure that multiple processes/threads would not write to the
same tmp file concurrently.

There is some folder locking, but nothing at the FS level. All folder
stuff is on the main thread anyway (so the JS frontend can use it).

> 3. Always cleaning up compaction tmp files on error.

Yup. Using an existing file class which deletes itself if an explicit
commit is not issued.

> 4. Clean up compaction tmp files on startup on case of an unclean
shutdown during compaction.

Nope... but that's mostly covered by 3. A hard power-down or extreme crash 
would probably leave something behind, I think...
Potentially could add a startup check, but there are some corner-case issues 
there too.

> 5. Ensure that the number of written messages while compacting does
not exceed the number of messages in the folder being compacted.

Not a problem with the new code. The store iterates through the existing
messages as it writes out the new mbox, and never revisits ones it's
already written or skipped.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/35

------------------------------------------------------------------------
On 2024-04-10T03:02:47+00:00 Omry Yadan wrote:

> See Bug 1890448. There's a patch in phabricator attached there.
Thanks.

> > 2. Making sure that multiple processes/threads would not write to the same 
> > tmp file concurrently.
> 
> There is some folder locking, but nothing at the FS level. All folder stuff 
> is on the main thread anyway (so the JS frontend can use it).
Is it? There is a progress bar on compaction and the UI is not locked as far as 
I remember.

As for multiple processes:
It looks like attempting to open TB a second time focuses on the first window 
instead of opening a new instance so this is probably something safe to ignore 
and in unlikely to be related anyway.
 
> > 4. Clean up compaction tmp files on startup on case of an unclean shutdown 
> > during compaction.
> 
> Nope... but that's mostly covered by 3. A hard power-down or extreme crash 
> would probably leave something behind, I think...
> Potentially could add a startup check, but there are some corner-case issues 
> there too.

What I have done before in cases like this was to drop a pid file in the 
folder, and check it on startup.
If the PID inside the pid file is different than my process pid and if the file 
was not touched recently (per file system metadata) I assume it's a stale file 
and clean it up (along with the old pid file).
 
> > 5. Ensure that the number of written messages while compacting does not 
> > exceed the number of messages in the folder being compacted.
> 
> Not a problem with the new code. The store iterates through the existing 
> messages as it writes out the new mbox, and never revisits ones it's already 
> written or skipped.

I suggested it as a simple fallback protection mechanism. If the file grows 
indefinitely the problem is likely either an infinite stream of messages coming 
while iterating (another possibility is very large messages being written due 
to some bug).
We can punt this for now as a total rewrite like yours have a good chance of 
working out the underlying bug anyway.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/36

------------------------------------------------------------------------
On 2024-04-10T12:11:47+00:00 Mkmelin+mozilla wrote:

*** Bug 1890290 has been marked as a duplicate of this bug. ***

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/37

------------------------------------------------------------------------
On 2024-04-20T11:08:30+00:00 Mkmelin+mozilla wrote:

*** Bug 1892566 has been marked as a duplicate of this bug. ***

Reply at:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/2060534/comments/39


** Changed in: thunderbird
       Status: Unknown => Confirmed

** Bug watch added: Mozilla Bugzilla #1889755
   https://bugzilla.mozilla.org/show_bug.cgi?id=1889755

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2060534

Title:
  Thunderbird doesn't always clean nstmp cache files

To manage notifications about this bug go to:
https://bugs.launchpad.net/thunderbird/+bug/2060534/+subscriptions


-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to