It seems that it's currently very difficult to use the javadoc tool in a long-running VM because there is a severe memory leak.
This appears to be related to the class jdk.javadoc.internal.doclets.toolkit.WorkArounds, specifically the field serializedForms. It looks like elements are never removed from this map. Furthermore, even if compiling the same sources over and over again, existing entries are not reused. Does this field have to be static? What's the purpose of this construct, anyway? Is it more than just a cache?