I"m trying to debug query results inside spark-shell, but finding it cumbersome to save to file and then use file system utils to explore the results, and .foreach(print) tends to interleave the results among the myriad log messages. Take() and collect() truncate.
Is there a simple way to present the contents of an RDD/SchemaRDD on the screen in a formatted way? For example, say I want to take() the first 30 lines/rows in an *RDD and present them in a readable way on the screen so that I can see what's missing or invalid. Obviously, I'm just trying to sample the results in a readable way, not download everything to the driver. Thank you
