(our last 2 messages crossed).
"So, if assp tries (and retries for ever) to use a BerkeleyDB - but the
folder not extsts - and BerkeleyDB.pm does for what ever reason not close
files/filehandles/... - this will lead in to a 'too many opened files'
error."
YES! YES! YES! Explains exactly how I was crashing. I'm going to add that
comment on the 21280 thread in case anyone else is following there.
On Mon, Oct 11, 2021 at 9:28 AM Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> The count of handles seems not to be the problem!
> But, there is a limit for available filedescrptors per process on windows
> (msvcrt.dll) - 2048 !
>
> I made a test with the following code
>
> #####
> use Tie::File;
> our %k;
> for (0...10000) {
> $k{$_} = [];
> }
>
> for (sort {$a <=> $b} keys %k) {
> tie @{$k{$_}}, 'Tie::File', "c:/tmp/test/$_";
> push @{$k{$_}}, "$_";
> print "$_\n";
> }
> ####
>
>
> this creates 2044 files (not the expected 10001). 2044 + STDOUT + STDIN +
> STDERR + script = 2048
>
> So, if assp tries (and retries for ever) to use a BerkeleyDB - but the
> folder not extsts - and BerkeleyDB.pm does for what ever reason not close
> files/filehandles/... - this will lead in to a 'too many opened files'
> error.
>
>
>
>
> Thomas
>
>
>
>
>
> Von: "Thomas Eckardt" <thomas.ecka...@thockar.com>
> An: "ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum: 11.10.2021 09:58
> Betreff: Re: [Assp-test] Multiple topics: Griplist vs GripList,
> stuck open handles to dbd-error on windows, more.
> ------------------------------
>
>
>
> 1) - this will be fixed
>
> 2) - not default settings needs to be checked by the admin
>
> 3) - BDB-error.txt files are used to record BerkeleyDB errors - a handle
> is keeped opened by each thread for each BDB-environment (db) as long as
> the database-env is opened by the thread
> most times such handles are opened until the thread is stopped (assp ends)
>
> 4)
> a) yes
>
> b) yes
>
> c)
> Scoring is done using gripValencePB and the resulting message/ip - score
> is calculated as follows:
>
> if the grip value is < 0.3 : -int(((0.3 - grip value) / 0.3) *
> gripValencePB)
> if the grip value is > 0.7 : int(((grip value - 0.7) / 0.3) *
> gripValencePB)
> grip values between 0.3 and 0.7 are ignored.
>
>
> windows handles (IMHO):
>
> system-max handles : ~ 2**24 (>16.000.000)
> process/thread max handles : ~ 10.000 (configurable in the registry -
> hex(2710))
>
> use Testlimit.exe / Testlimit64.exe -h
>
> How ever - the c-library used by the process sets the handle (and other)
> limits! If a valid but not-default-system c-lib is found in the PATH, it
> will be used (with there internal limit settings).
>
> assp never uses more than 2000 handles (typical less than 1000) on any
> system watched by me
> most handles (~ 1.000.000) are used permanently by mysqld on windows
>
>
> Thomas
>
>
>
> Von: "K Post" <nntp.p...@gmail.com>
> An: "ASSP development mailing list" <
> assp-test@lists.sourceforge.net>
> Datum: 09.10.2021 16:21
> Betreff: [Assp-test] Multiple topics: Griplist vs GripList, stuck
> open handles to dbd-error on windows, more.
> ------------------------------
>
>
>
> Several related items here:
>
> 1. Bug? Rebuild process still uploading griplist, even if disabled,
> due possibly to case error in code.
> 2. ASSP not checking for valid griplist, if an invalid folder name is
> entered
> 3. On windows, Rebuild process leaving \tmpDB\rebuildDB\BDB-error.txt
> handle open (wider issue with DBD-error.txt files getting stuck open on
> Windows?)
> 4. Griplist clarification request
>
>
> 1) I've got noGriplistUpload and noGriplistDownload both checked in the
> GUI. But I noticed that ,at the end of my rebuild log, it's still doing
> the upload
>
> Uploading Griplist via Direct Connection
>
>
> *rebuildspamdb.pm* <http://rebuildspamdb.pm/> has
> return if $main::noGripListUpload;
> before the upload happens, but I think there's a case mistake there.
>
> I believe it should be Griplist, with a lower case L.
>
>
> 2) In trying to figure out my ASSP on Windows crashes due to too many open
> files, I've started using Sysinternals Handle utility to look at open
> handles. Yesterday I saw 1300+ handles open to
> d:\assp\tmpDB\Griplist\BDB-error.txt
>
> I believe that had something to do with me having an invalid griplist
> database name, "d/griplist" but the d folder didn't exist on my new server
> config!! My fault there for sure, not sure how d/ prefixed it, but it
> appears ASSP didn't check to see that the folder existed. Shame on me for
> missing my config file, but it might be good for ASSP to warn or create the
> folder if it doesn't exist.
>
> I cleared out the griplist database file to test and restarted. I don't
> see any more open griplist\dbd-error.txt handles, even though I'm getting
> many many bad SMTP servers connecting with early TLS. Good. I'll probably
> put the griplist database back (obviously with a valid filename!!) See
> #3....
>
>
> 3) Windows (both 2012 and 2019) might not be closing Berkely error files
> correctly in general.
>
> When the new version started logging the new SSL errors, I >believe<
> that's when it started trying to access the non-existent "d/" folder config
> error in m assp.cfg. Every time an early TLS line was caught, BerkeleyDB
> would keep the error file open. Strangely, the error file was 0kb.
>
> Now that I've cleared out the griplist entry, I don't get those
> /tmpdb/griplist/dbd-error.txt open handles. However, after my rebuild over
> night, I see and open handle to:
> tmpDB\rebuildDB\BDB-error.txt
> about 9+ hours after rebuild.
>
> Is that normal? Might Windows not be closing handles correctly in general?
>
> 4) While I'm at it, could some clarification be provided as to the
> function of the griplist? (and please correct me if anything I say here is
> incorrect!!) I've searched like crazy over the last couple of days, but
> can't find the answers I'm looking for.
>
> The griplist is an ip scoring database correct? I know it was originally
> called the grey-ip-list, 15+ years ago, but then greylisting became common
> language for delaying, so the original grayiplist started being called the
> "griplist" to avoid confusion.
>
> Note: the gui still says:
> GreyIPlist Database (griplist)
> The file with the current Grey-IP-List database -- make this blank if you
> don't use it.
> If I'm understanding griplist correctly, I think the gui should be
> reworded. Maybe some explanation added too??
>
> There's also the optional upload and download concept of the griplist.
> This appears to send the local griplist to sourceforge, it gets processed
> by whatever you've got running on the backend, and then I can (also
> optionally) download another griplist which is based on all ASSP user
> data. If you don't upload, you can't download, and that's fair - you need
> to contribute to benefit from the group.
>
> The charity that I work for has a pretty poorly thought out privacy policy
> that requires me to jump through all kinds of hoops when sharing >any<
> information. It's frustrating for sure. That means I cannot upload our
> griplist without petitioning an internal committee. I'm thinking I want to
> do that, but need to fully understand the griplist first.
>
> - If we don't upload/download, the griplist is stlil maintained
> locally, just only with my data right?
> - If I get approval to share the ip data and download the griplist,
> that downloaded griplist is merged with my local griplist?
> - And in the end, how is a match on the griplist scored? Let's say an
> IP is on the griplist as being a really bad IP. What score does a message
> get? Is that configurable?
>
>
> As always, thank you
> Ken
>
>
>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
