Support for storing date time fields as TIMESTAMP_MILLIS(INT64)

2017-12-05 Thread Rahul Raj
Hi, I believe spark writes datetime fields as INT96. What are the implications of https://issues.apache.org/jira/browse/SPARK-10364(Support Parquet logical type TIMESTAMP_MILLIS) which is part of 2.2.0? I am having issues while reading spark generated parquet dates using Apache Drill (Drill

Fwd: SparkLauncher does not return State/ID on a standalone cluster

2016-12-17 Thread Rahul Raj
I am unable to retrieve the state and Id of a submitted application on a Standalone cluster. The job gets executed successfully on the cluster. The state was checked using: while(!handle.getState().isFinal()){ //print handle.getState() } When run as local, state gets reported correctly.