Display a warning in EMR welcome screen

2024-05-11 Thread Abhishek Basu
Hi Team, for Elastic Map Reduce (EMR) cluster, it would be great if there is a warning message that Logging should be handled carefully and INFO or DEBUG should be enabled only when required. This logging thing took my whole day and lastly I discovered that it’s exploding the hdfs and not

Re: [spark-graphframes]: Generating incorrect edges

2024-05-11 Thread Nijland, J.G.W. (Jelle, Student M-CS)
Hi all, The issue is solved. I conducted a lot more testing and built checkers to verify at which size it's going wrong. When checking for specific edges, I could construct successful graphs up to 261k records. When verifying all edges created, is breaks somewhere in the 200-250k records. I