Noel Jones put forth on 10/31/2009 1:12 AM: > Each lookup table requires overhead. 30 separate tables requires > considerably more overhead than one table. The size of the dataset > doesn't change, it's the overhead that gets smaller. The more > concurrent smtpd processes running, the more it matters.
I guess I was kinda looking for the technical definition of 'overhead' here. If the overhead per file is a straight percentage, then it shouldn't matter if it's one big file or 30 smaller files. What is the actual memory footprint overhead that's being added by Posfix for each map file? And is the amount of overhead the same for all map types, or different/variable depending on map type? Is it something like 3 bytes of overhead added for every 10 bytes in a map file, for example? I can understand that having everything in one map file, or few map files, compared to many, can increase lookup performance due to indexing efficiency. What I don't understand is how having fewer larger map files decreases memory footprint. > It probably doesn't matter in your environment, but posting examples > suitable for only a very small environment without labeled as such > doesn't help the wider user base. Agreed. That's why I described my environment. It's likely much smaller scale than the average Postfix environment. > And postmap doesn't compress files, but the index allows postfix to find > a particular key quickly -- lookup time and system overhead for hash: > and cdb: is essentially independent of the size of the table, so it > makes sense to use a single table as big as necessary. In particular, > cdb: scales well up to millions of entries, hash: tables easily to > hundreds of thousands. Agreed. Indexed files are always much faster to search than flat text files (unless we're talking really tiny flat files). And it's quicker to search fewer indexes than more indexes. This thread began with the premise being stated that many small map files consume more memory in Postfix than fewer larger map files. Indexes are typically proportional in size to the file being indexed, unless the file contains many duplicate entries. I freely admit I'm no indexing or database expert. However, that said, I don't understand how map file consolidation decreases memory footprint. Maybe Wieste can chime in with a brief explanation. Also, given that cidr files aren't postmap'd, is it safe to assume they aren't indexed? If they aren't indexed, how does putting the contents in a single file speed up searching? If they are indexed when Postfix starts or reloads, then why wouldn't Postfix created a single index _after_ reading in all the data from all the cidr files, creating one internal program table? -- Stan
