-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 John,
On 7/6/20 11:48, john.e.gr...@wellsfargo.com.INVALID wrote: > Chris, > > >> -----Original Message----- From: Christopher Schultz >> <ch...@christopherschultz.net> Sent: Monday, July 06, 2020 10:21 >> AM To: Tomcat Users List <users@tomcat.apache.org> Subject: [OT] >> Trying to determine the minimum heap required for an operation >> > All, > > Definitely off-topic, but it's the kind of weird thing someone here > might have experience with. > > I have an offline operation I'm considering bringing "inside" my > web-based application. My only concern is memory usage: it requires > that a bunch of data be loaded from a db into memory and then > analyzed. It doesn't take long to execute -- maybe 10 seconds or > so, so the memory can be released back to the rest of the > application. > > I've instrumented the command-line process with a thread which runs > every .5sec and captures the used-memory, maintaining a high-water > mark and reporting it after the whole operation is done. The first > time I ran it (with no specific JVM memory-related settings), it > reported that the high-water mark was ~450MiB. > > I figured that was higher than necessary, and probably just > represented a lazy GC with loads of memory, so I constrained the > process using -Xmx64M. That resulted in a 16MiB high-water mark. I > tried again with -Xmx8M and the high-water mark became 5MiB. > > Is there a particularly good way to force the GC to be as > aggressive as possible to see how low I can go, or should I just > keep playing-around with the -Xmx setting. > > Another option is to serialize my in-memory structure to the disk > to get a sense of the size in-memory, though it's really not the > same -- it will at least get me in the right ballpark. > > Any suggestions? > > Thanks, -chris >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > I guess I’m that person with the weird experiences. > > Is memory or CPU in short supply? If not, I don't think you'll > have a problem. This isn't 1997 anymore. I do think you should run a realistic load test, however. No specific problem exists, but this is a multi-user web application. Usually somewhere around 500 - 1000 users logged-in at once. Session size is typically quite low -- only a handful of small objects present with lots of sharing of "large" objects and structures. Heap size is set to max 1GiB on each server and memory usage shows a beautiful sawtooth pattern hovering around ~400MiB for days at a time. I will certainly limit the number of these operations that can occur at once, and they should be relatively rare. My test example was using a small data set, but the size of the data-set varies wildly with the client, so I have to be careful for the larger ones. Busting the heap isn't something I'd like to have happen. > To me the most important GC metric is time spent per minute/hour/etc. The next most important metric is individual pause durations. Through testing you'll see what those numbers are. I work with some large apps that have multi-GB heaps and it's rare to see GC time being over 1-2%. IOW, 600-1200ms per minute. Often it's a fraction of a percent. With those small numbers you're talking about, I don't think you'll have any trouble in this area unless the server is very heavily loaded. Actually, I'm not super concerned about performance of the GC itself. I was just wondering if there was a way to ask the JVM "if you *had* to accomplish this task with the smallest possible heap, what would it be?" > Be sure to enable verbose GC. In java 8, it's something like > this: > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps - -Xloggc:/path/to/gc.log > > Run tests with and without the changes. You can analyze the GC > output with tools like GCEasy and GCViewer. Sure. Again, I'm more concerned with the overhead that will be required for a particular operation, so I can predict when running such an operation might end up endangering the application server's heap -- and therefore the logged-in users. Theoretically, if the thread hits a heap-full error, the thread will experience an OOME, release it's (temporary) large object tree, and the GC will be able to recover, but after an OOME it's never a great plan to trust the JVM for very long. - -chris -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP6U8 pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1MhfOl66zXx4upJcIpPXgBuIkigbe 9d9y/jFnRCyHsFodSEsjtT/q2CxD7k30DIAwRrTaGxzrz60QlD/+t8l3getT9xot s0bAxvlpjZTvvhTtpAAv9hkSwJuMxxECksbqmYXaO/rtoBu/N9R8MCjPz4cihTaB dLZZ32Ibhg7tn0VBhwaJYz8AlYK3qJLUfISBU8h3WXXpmrbw48wDmkMYtdwUPcoL aWw9UbOolLj0EZiCh4QjCXb404pLbqTe0Hbuy7FBNv1rB8RDOFj8vWo8eotshqiJ c8fVY5jxHV9dqc0S2A6cCjoTrwsvUPimiykhCnrvCUpnif1I90H2b181sCtEyIyV XTADe/ore5K07DJGhFSenAYBvrU19TuiRL7NGH980DndYGns8woQ9BvR4WHEBDuf 15qkHdwBtjD6o5D6X5Mmyf6x6WbafhI6gb3hBPUGMEtxAhgvHNmIGX7+QVQHdtat 9+QSuu2viArftOKHfFpM85O25QuyfkPi4nGz1S86vcgd6q+N1Lk/qVTyJo2VJSmg Jq6DEEko/YaLIujgjU6rQdPoF8oeRAtvWXsnQ3Yw/x7Wv3rRH4l3NMhwv139nCPb 2Jt/LYAC4v48O1COt0huuCAu0dWIJqky3VIeIcI9lOpAZ1Jm3Do= =svYj -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
