wyp created LIVY-601:
------------------------
Summary: Add support for user defined date format and timezone
when transformspark result rows to json
Key: LIVY-601
URL: https://issues.apache.org/jira/browse/LIVY-601
Project: Livy
Issue Type: Improvement
Components: REPL
Affects Versions: 0.6.0
Reporter: wyp
When Livy get the row data from Spark DataFrame and transform those rows to
json, we use DefaultFormats to transform Spark timestamp to scala timestamp
which use default timezone.
{code:java}
private implicit def formats = DefaultFormats
val result = spark.sql(code)
val schema = parse(result.schema.json)
// Get the row data
val rows = result.take(maxResult)
.map {
_.toSeq.map {
// Convert java BigDecimal type to Scala BigDecimal, because current
version of
// Json4s doesn't support java BigDecimal as a primitive type (LIVY-455).
case i: java.math.BigDecimal => BigDecimal(i)
case e => e
}
}
val jRows = Extraction.decompose(rows)
// DefaultFormats defined in org.json4s.DefaultFormats
/** Default date format is UTC time.
*/
object DefaultFormats extends DefaultFormats {
val UTC = TimeZone.getTimeZone("UTC")
val losslessDate = {
def createSdf = {
val f = new java.text.SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
f.setTimeZone(UTC)
f
}
new ThreadLocal(createSdf)
}
}{code}
It would be useful if we can customize the time zone and time format
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
