Re: Spark MySQL Invalid DateTime value killing job

2019-06-05 Thread Anthony May
Murphy's Law striking after asking the question, I just discovered the solution: The jdbc url should set the zeroDateTimeBehavior option. https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

Spark MySQL Invalid DateTime value killing job

2019-06-05 Thread Anthony May
Hi, We have a legacy process of scraping a MySQL Database. The Spark job uses the DataFrame API and MySQL JDBC driver to read the tables and save them as JSON files. One table has DateTime columns that contain values invalid for java.sql.Timestamp so it's throwing the exception: