GitHub user davies opened a pull request:

    https://github.com/apache/spark/pull/8400

    [SPARK-10177] [SQL] fix reading Timestamp in parquet from Hive

    We misunderstood the Julian days and nanoseconds of the day in parquet (as 
TimestampType) from Hive/Impala, they are overlapped, so can't be added 
together directly.
    
    In order to avoid the confusing rounding when do the converting, we use 
`2440588` as the Julian Day of epoch of unix timestamp (which should be 
2440587.5).
    
    This PR includes #8392 to have an regression test. (will rebase once it's 
merged).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davies/spark timestamp_parquet

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/8400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #8400
    
----
commit 808ae3bb284bae163634d708ee699aebb671c9f7
Author: Cheng Lian <l...@databricks.com>
Date:   2015-08-24T07:32:35Z

    Refactors ParquetHiveCompatibilitySuite and adds more test cases

commit 809e164d9c4da77ff3a2617321ca3d3928ec561f
Author: Davies Liu <dav...@databricks.com>
Date:   2015-08-24T19:00:30Z

    read timestamp in parquet generated from Hive

commit 3491f2f784cf2df7107266ccbff53470476fe05b
Author: Davies Liu <dav...@databricks.com>
Date:   2015-08-24T19:09:27Z

    Merge branch 'parquet_tests'

commit e96f92f390e4f82e085427e963122f06ba88a74d
Author: Davies Liu <dav...@databricks.com>
Date:   2015-08-24T19:12:02Z

    enable regression test

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to