GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/8400
[SPARK-10177] [SQL] fix reading Timestamp in parquet from Hive
We misunderstood the Julian days and nanoseconds of the day in parquet (as
TimestampType) from Hive/Impala, they are overlapped, so can't be added
together directly.
In order to avoid the confusing rounding when do the converting, we use
`2440588` as the Julian Day of epoch of unix timestamp (which should be
2440587.5).
This PR includes #8392 to have an regression test. (will rebase once it's
merged).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark timestamp_parquet
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8400.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8400
----
commit 808ae3bb284bae163634d708ee699aebb671c9f7
Author: Cheng Lian <l...@databricks.com>
Date: 2015-08-24T07:32:35Z
Refactors ParquetHiveCompatibilitySuite and adds more test cases
commit 809e164d9c4da77ff3a2617321ca3d3928ec561f
Author: Davies Liu <dav...@databricks.com>
Date: 2015-08-24T19:00:30Z
read timestamp in parquet generated from Hive
commit 3491f2f784cf2df7107266ccbff53470476fe05b
Author: Davies Liu <dav...@databricks.com>
Date: 2015-08-24T19:09:27Z
Merge branch 'parquet_tests'
commit e96f92f390e4f82e085427e963122f06ba88a74d
Author: Davies Liu <dav...@databricks.com>
Date: 2015-08-24T19:12:02Z
enable regression test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
