Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[Benedict]

I disagree with the first claim, as the process has all the information it chooses to utilise about which resources it’s using and what it’s using those resources for. The inability to isolate GC domains is something we cannot address, but also probably not a problem if we were doing everything with memory management as well as we could be. But, not worth detailing this thread for. Today we do very little well on this front within the process, and a separate process is well justified given the state of play. On 28 Mar 2023, at 16:38, Derek Chen-Becker <de...@chen-becker.org> wrote:  On Tue, Mar 28, 2023 at 9:03 AM Joseph Lynch <joe.e.ly...@gmail.com> wrote: ... I think we might be underselling how valuable JVM isolation is, especially for analytics queries that are going to pass the entire dataset through heap somewhat constantly. Big +1 here. The JVM simply does not have significant granularity of control for resource utilization, but this is explicitly a feature of separate processes. Add in being able to separate GC domains and you can avoid a lot of noisy neighbor in-VM behavior for the disparate workloads. Cheers, Derek -- +---------------------------------------------------------------+ | Derek Chen-Becker                                             | | GPG Key available at https://keybase.io/dchenbecker and       | | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  | +---------------------------------------------------------------+