Support for storing date time fields as TIMESTAMP_MILLIS(INT64)

[Rahul Raj]

Hi,

I believe spark writes datetime fields as INT96. What are the implications
of https://issues.apache.org/jira/browse/SPARK-10364(Support Parquet
logical type TIMESTAMP_MILLIS) which is part of 2.2.0?

I am having issues while reading spark generated parquet dates using Apache
Drill (Drill supports INT64 default). Will the change allow me to store
datetime fields as INT64?

Regards,
Rahul

-- 
 This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom it is 
addressed. If you are not the named addressee then you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately and delete this e-mail from your system.

Fwd: SparkLauncher does not return State/ID on a standalone cluster

[Rahul Raj]

I am unable to retrieve the state and Id of a submitted application on a
Standalone cluster. The job gets executed successfully on the cluster.

The state was checked using:

while(!handle.getState().isFinal()){
   //print handle.getState()
}

When run as local, state gets reported correctly.

Regards,
Rahul

-- 
 This email and any files transmitted with it are confidential and 
intended solely for the use of the individual or entity to whom it is 
addressed. If you are not the named addressee then you should not 
disseminate, distribute or copy this e-mail. Please notify the sender 
immediately and delete this e-mail from your system.