Re: Questions for platform to choose

2019-08-22 Thread Liam Clarke-Hutchinson
Hi Eliza, As I mentioned to you in the Kafka mailing list when you asked this there, there are pros and cons to all of the technologies you've mentioned, and you really need to sit down and try each solution to see what suits your needs best. Kind regards, Liam Clarke On Wed, Aug 21, 2019 at

RDD size in memory - Array[String] vs. case classes

2014-10-10 Thread Liam Clarke-Hutchinson
Hi all, I'm playing with Spark currently as a possible solution at work, and I've been recently working out a rough correlation between our input data size and RAM needed to cache an RDD that will be used multiple times in a job. As part of this I've been trialling different methods of