Can you identify a specific file that fails? There might be a real bug here, but I have found gzip to be reliable. Every time I have run into a "bad header" error with gzip, I had a non-gzip file with the wrong extension for whatever reason.
----- Madhu https://www.linkedin.com/in/msiddalingaiah -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/count-ing-gz-files-gives-java-io-IOException-incorrect-header-check-tp5768p6169.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
