Hi All, I am seeing a huge variation on spark Task Deserialization Time for my collect and reduce operations. while most tasks complete within 100ms a few take mote than a couple of seconds which slows the entire program down. I have attached a screen shot of the web ui where you can see the variation
As you can see the Task Deserialization Time time has a Max of 7s and 75th percentile at 0.3 seconds. Does anyone know the reasons that may cause these kind of numbers. Any help would be greatly appreciated. Best Regards, -- Pulasthi S. Wickramasinghe Graduate Student | Research Assistant School of Informatics and Computing | Digital Science Center Indiana University, Bloomington cell: 224-386-9035
