[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6759 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114315491 The last failure is not related, I'd merge this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114313226 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114313203 [Test build #35489 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35489/console) for PR 6759 at commit [`849e301`](https://github.com/apache/spark/commit/849e3016d79813379037c72c4c937d837760c55f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114292465 [Test build #35489 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35489/consoleFull) for PR 6759 at commit [`849e301`](https://github.com/apache/spark/commit/849e3016d79813379037c72c4c937d837760c55f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114292440 @davies, yeah, I think this should be good to go; looks like all of my comments have been addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114292053 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114292039 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114291488 @JoshRosen Is this ready to go? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114290446 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114290357 [Test build #35481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35481/console) for PR 6759 at commit [`b0e4cad`](https://github.com/apache/spark/commit/b0e4cadfa16effbabdbcc57089121b488a8c33ea). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114245215 [Test build #35481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35481/consoleFull) for PR 6759 at commit [`b0e4cad`](https://github.com/apache/spark/commit/b0e4cadfa16effbabdbcc57089121b488a8c33ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114243271 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-114243198 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113845945 [Test build #948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/948/console) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113838754 [Test build #948 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/948/consoleFull) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113838592 [Test build #945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/945/console) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113838256 [Test build #945 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/945/consoleFull) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113702346 [Test build #942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/942/console) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113696057 [Test build #941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/941/console) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113690706 [Test build #942 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/942/consoleFull) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113687010 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113687001 [Test build #35332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35332/console) for PR 6759 at commit [`634b9f5`](https://github.com/apache/spark/commit/634b9f5540b8045b24c20c7296d8cd73193c). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113673700 [Test build #941 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/941/consoleFull) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113673423 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113673416 [Test build #35338 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35338/console) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113670942 [Test build #35338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35338/consoleFull) for PR 6759 at commit [`8e2d56f`](https://github.com/apache/spark/commit/8e2d56fffc0560f0e9b915a705d92d70ae4676e9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113670659 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113670648 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113659586 [Test build #35332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35332/consoleFull) for PR 6759 at commit [`634b9f5`](https://github.com/apache/spark/commit/634b9f5540b8045b24c20c7296d8cd73193c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113658676 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113658717 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32872749
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
---
@@ -21,19 +21,31 @@ import java.sql.Timestamp
import org.apache.spark.SparkFunSuite
-class DateUtilsSuite extends SparkFunSuite {
+class DateTimeUtilsSuite extends SparkFunSuite {
- test("timestamp") {
+ test("timestamp and 100ns") {
val now = new Timestamp(System.currentTimeMillis())
now.setNanos(100)
-val ns = DateUtils.fromJavaTimestamp(now)
+val ns = DateTimeUtils.fromJavaTimestamp(now)
assert(ns % 1000L == 1)
-assert(DateUtils.toJavaTimestamp(ns) == now)
+assert(DateTimeUtils.toJavaTimestamp(ns) == now)
List(-L, -1L, 0, 1L, L).foreach { t =>
- val ts = DateUtils.toJavaTimestamp(t)
- assert(DateUtils.fromJavaTimestamp(ts) == t)
- assert(DateUtils.toJavaTimestamp(DateUtils.fromJavaTimestamp(ts)) ==
ts)
+ val ts = DateTimeUtils.toJavaTimestamp(t)
+ assert(DateTimeUtils.fromJavaTimestamp(ts) == t)
+
assert(DateTimeUtils.toJavaTimestamp(DateTimeUtils.fromJavaTimestamp(ts)) == ts)
}
}
+
+ test("100ns and julian day") {
+val (d, ns) = DateTimeUtils.toJulianDay(0)
+assert(d == 2440587)
--- End diff --
Could use `===` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32872412
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
Yeah, let's just leave this as-is.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32872290
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
---
@@ -21,19 +21,31 @@ import java.sql.Timestamp
import org.apache.spark.SparkFunSuite
-class DateUtilsSuite extends SparkFunSuite {
+class DateTimeUtilsSuite extends SparkFunSuite {
- test("timestamp") {
+ test("timestamp and 100ns") {
val now = new Timestamp(System.currentTimeMillis())
now.setNanos(100)
-val ns = DateUtils.fromJavaTimestamp(now)
+val ns = DateTimeUtils.fromJavaTimestamp(now)
assert(ns % 1000L == 1)
-assert(DateUtils.toJavaTimestamp(ns) == now)
+assert(DateTimeUtils.toJavaTimestamp(ns) == now)
List(-L, -1L, 0, 1L, L).foreach { t =>
- val ts = DateUtils.toJavaTimestamp(t)
- assert(DateUtils.fromJavaTimestamp(ts) == t)
- assert(DateUtils.toJavaTimestamp(DateUtils.fromJavaTimestamp(ts)) ==
ts)
+ val ts = DateTimeUtils.toJavaTimestamp(t)
+ assert(DateTimeUtils.fromJavaTimestamp(ts) == t)
+
assert(DateTimeUtils.toJavaTimestamp(DateTimeUtils.fromJavaTimestamp(ts)) == ts)
}
}
+
+ test("100ns and julian day") {
+val (d, ns) = DateTimeUtils.toJulianDay(0)
+assert(d == 2440587)
--- End diff --
You could replace this constant with `DateTimeUtils.JULIAN_DAY_OF_EPOCH`
for clarity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32872239
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
---
@@ -21,19 +21,31 @@ import java.sql.Timestamp
import org.apache.spark.SparkFunSuite
-class DateUtilsSuite extends SparkFunSuite {
+class DateTimeUtilsSuite extends SparkFunSuite {
- test("timestamp") {
+ test("timestamp and 100ns") {
val now = new Timestamp(System.currentTimeMillis())
now.setNanos(100)
-val ns = DateUtils.fromJavaTimestamp(now)
+val ns = DateTimeUtils.fromJavaTimestamp(now)
assert(ns % 1000L == 1)
-assert(DateUtils.toJavaTimestamp(ns) == now)
+assert(DateTimeUtils.toJavaTimestamp(ns) == now)
List(-L, -1L, 0, 1L, L).foreach { t =>
- val ts = DateUtils.toJavaTimestamp(t)
- assert(DateUtils.fromJavaTimestamp(ts) == t)
- assert(DateUtils.toJavaTimestamp(DateUtils.fromJavaTimestamp(ts)) ==
ts)
+ val ts = DateTimeUtils.toJavaTimestamp(t)
+ assert(DateTimeUtils.fromJavaTimestamp(ts) == t)
+
assert(DateTimeUtils.toJavaTimestamp(DateTimeUtils.fromJavaTimestamp(ts)) == ts)
}
}
+
+ test("100ns and julian day") {
--- End diff --
Are there any other inputs that are worth testing here? It wouldn't be
super hard to fuzz this using the invariant that some of these methods should
be inverses.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32872039
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
Also, comparing to the speed of writing into disk or network file system,
the performance of writing pass is not that critical.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32871889
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
I think it's fine to have small and short-term allocations, and also don't
try to optimize everything.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32871700
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
Actually, this is probably tricky given that we need to do the byte-order
handling too. Maybe we shouldn't optimize this now if it's going to be really
hard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32871422
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -360,7 +367,7 @@ private[parquet] class MutableRowWriteSupport extends
RowWriteSupport {
case FloatType => writer.addFloat(record.getFloat(index))
case BooleanType => writer.addBoolean(record.getBoolean(index))
case DateType => writer.addInteger(record.getInt(index))
- case TimestampType =>
writeTimestamp(record(index).asInstanceOf[Long])
+ case TimestampType => writeTimestamp(record.getLong(index))
--- End diff --
Nice catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32871383
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
Actually, do you know if there are any static methods that we could call
that would just write put the longs and ints directly into the byte array at
given offsets?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/6759#discussion_r32871309 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala --- @@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends WriteSupport[InternalRow] with Lo writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes)) } + // array used to write Timestamp as Int96 (fixed-length binary) + private val int96buf = new Array[Byte](12) --- End diff -- Actually, I guess this is fine since we use the same pattern with `scratchBytes` for `writeDecimal` in the preceding lines. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32871228
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala
---
@@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends
WriteSupport[InternalRow] with Lo
writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes))
}
+ // array used to write Timestamp as Int96 (fixed-length binary)
+ private val int96buf = new Array[Byte](12)
+
private[parquet] def writeTimestamp(ts: Long): Unit = {
-val binaryNanoTime = CatalystTimestampConverter.convertFromTimestamp(
- DateUtils.toJavaTimestamp(ts))
-writer.addBinary(binaryNanoTime)
+val (julianDay, timeOfDayNanos) = DateTimeUtils.toJulianDay(ts)
+val buf = ByteBuffer.wrap(int96buf)
--- End diff --
If you wanted to perform even less allocation, you might be able to re-use
the wrapped byte buffer, too. On the other hand, maybe optimizing this is
premature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/6759#discussion_r32871144 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableSupport.scala --- @@ -313,10 +314,16 @@ private[parquet] class RowWriteSupport extends WriteSupport[InternalRow] with Lo writer.addBinary(Binary.fromByteArray(scratchBytes, 0, numBytes)) } + // array used to write Timestamp as Int96 (fixed-length binary) + private val int96buf = new Array[Byte](12) --- End diff -- This assumes that `RowWriteSupport` instances are thread-safe; just want to check that that assumption is true. This could also be `private[this]`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113647862 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113647849 [Test build #35321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35321/console) for PR 6759 at commit [`4891efb`](https://github.com/apache/spark/commit/4891efbb6b5f277082c06ea56400c83bc4678f35). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113642670 [Test build #35321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35321/consoleFull) for PR 6759 at commit [`4891efb`](https://github.com/apache/spark/commit/4891efbb6b5f277082c06ea56400c83bc4678f35). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113642575 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113642553 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113632305 At a high level, this looks like it's in really good shape. I might have more comments in a bit; have to step out to handle some calls now, but I'll loop back later in the afternoon to finish looking this over. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32863060
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala ---
@@ -499,69 +497,23 @@ private[parquet] object CatalystArrayConverter {
}
private[parquet] object CatalystTimestampConverter {
- // TODO most part of this comes from Hive-0.14
- // Hive code might have some issues, so we need to keep an eye on it.
- // Also we use NanoTime and Int96Values from parquet-examples.
- // We utilize jodd to convert between NanoTime and Timestamp
- val parquetTsCalendar = new ThreadLocal[Calendar]
- def getCalendar: Calendar = {
-// this is a cache for the calendar instance.
-if (parquetTsCalendar.get == null) {
-
parquetTsCalendar.set(Calendar.getInstance(TimeZone.getTimeZone("GMT")))
-}
-parquetTsCalendar.get
- }
- val NANOS_PER_SECOND: Long = 10
- val SECONDS_PER_MINUTE: Long = 60
- val MINUTES_PER_HOUR: Long = 60
- val NANOS_PER_MILLI: Long = 100
-
- def convertToTimestamp(value: Binary): Timestamp = {
-val nt = NanoTime.fromBinary(value)
-val timeOfDayNanos = nt.getTimeOfDayNanos
-val julianDay = nt.getJulianDay
-val jDateTime = new JDateTime(julianDay.toDouble)
-val calendar = getCalendar
-calendar.set(Calendar.YEAR, jDateTime.getYear)
-calendar.set(Calendar.MONTH, jDateTime.getMonth - 1)
-calendar.set(Calendar.DAY_OF_MONTH, jDateTime.getDay)
-
-// written in command style
-var remainder = timeOfDayNanos
-calendar.set(
- Calendar.HOUR_OF_DAY,
- (remainder / (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)
-calendar.set(
- Calendar.MINUTE, (remainder / (NANOS_PER_SECOND *
SECONDS_PER_MINUTE)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE)
-calendar.set(Calendar.SECOND, (remainder / NANOS_PER_SECOND).toInt)
-val nanos = remainder % NANOS_PER_SECOND
-val ts = new Timestamp(calendar.getTimeInMillis)
-ts.setNanos(nanos.toInt)
-ts
- }
-
- def convertFromTimestamp(ts: Timestamp): Binary = {
-val calendar = getCalendar
-calendar.setTime(ts)
-val jDateTime = new JDateTime(calendar.get(Calendar.YEAR),
- calendar.get(Calendar.MONTH) + 1,
calendar.get(Calendar.DAY_OF_MONTH))
-// Hive-0.14 didn't set hour before get day number, while the day
number should
-// has something to do with hour, since julian day number grows at 12h
GMT
-// here we just follow what hive does.
-val julianDay = jDateTime.getJulianDayNumber
-
-val hour = calendar.get(Calendar.HOUR_OF_DAY)
-val minute = calendar.get(Calendar.MINUTE)
-val second = calendar.get(Calendar.SECOND)
-val nanos = ts.getNanos
-// Hive-0.14 would use hours directly, that might be wrong, since the
day starts
-// from 12h in Julian. here we just follow what hive does.
-val nanosOfDay = nanos + second * NANOS_PER_SECOND +
- minute * NANOS_PER_SECOND * SECONDS_PER_MINUTE +
- hour * NANOS_PER_SECOND * SECONDS_PER_MINUTE * MINUTES_PER_HOUR
-NanoTime(julianDay, nanosOfDay).toBinary
+ def convertToTimestamp(value: Binary): Long = {
+Preconditions.checkArgument(value.length() == 12, "Must be 12 bytes")
+val buf = value.toByteBuffer
--- End diff --
Does this end up allocating a new byte buffer? If so, it might be kind of
expensive.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32862797
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala ---
@@ -499,69 +497,23 @@ private[parquet] object CatalystArrayConverter {
}
private[parquet] object CatalystTimestampConverter {
- // TODO most part of this comes from Hive-0.14
- // Hive code might have some issues, so we need to keep an eye on it.
- // Also we use NanoTime and Int96Values from parquet-examples.
- // We utilize jodd to convert between NanoTime and Timestamp
- val parquetTsCalendar = new ThreadLocal[Calendar]
- def getCalendar: Calendar = {
-// this is a cache for the calendar instance.
-if (parquetTsCalendar.get == null) {
-
parquetTsCalendar.set(Calendar.getInstance(TimeZone.getTimeZone("GMT")))
-}
-parquetTsCalendar.get
- }
- val NANOS_PER_SECOND: Long = 10
- val SECONDS_PER_MINUTE: Long = 60
- val MINUTES_PER_HOUR: Long = 60
- val NANOS_PER_MILLI: Long = 100
-
- def convertToTimestamp(value: Binary): Timestamp = {
-val nt = NanoTime.fromBinary(value)
-val timeOfDayNanos = nt.getTimeOfDayNanos
-val julianDay = nt.getJulianDay
-val jDateTime = new JDateTime(julianDay.toDouble)
-val calendar = getCalendar
-calendar.set(Calendar.YEAR, jDateTime.getYear)
-calendar.set(Calendar.MONTH, jDateTime.getMonth - 1)
-calendar.set(Calendar.DAY_OF_MONTH, jDateTime.getDay)
-
-// written in command style
-var remainder = timeOfDayNanos
-calendar.set(
- Calendar.HOUR_OF_DAY,
- (remainder / (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)
-calendar.set(
- Calendar.MINUTE, (remainder / (NANOS_PER_SECOND *
SECONDS_PER_MINUTE)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE)
-calendar.set(Calendar.SECOND, (remainder / NANOS_PER_SECOND).toInt)
-val nanos = remainder % NANOS_PER_SECOND
-val ts = new Timestamp(calendar.getTimeInMillis)
-ts.setNanos(nanos.toInt)
-ts
- }
-
- def convertFromTimestamp(ts: Timestamp): Binary = {
-val calendar = getCalendar
-calendar.setTime(ts)
-val jDateTime = new JDateTime(calendar.get(Calendar.YEAR),
- calendar.get(Calendar.MONTH) + 1,
calendar.get(Calendar.DAY_OF_MONTH))
-// Hive-0.14 didn't set hour before get day number, while the day
number should
-// has something to do with hour, since julian day number grows at 12h
GMT
-// here we just follow what hive does.
-val julianDay = jDateTime.getJulianDayNumber
-
-val hour = calendar.get(Calendar.HOUR_OF_DAY)
-val minute = calendar.get(Calendar.MINUTE)
-val second = calendar.get(Calendar.SECOND)
-val nanos = ts.getNanos
-// Hive-0.14 would use hours directly, that might be wrong, since the
day starts
-// from 12h in Julian. here we just follow what hive does.
-val nanosOfDay = nanos + second * NANOS_PER_SECOND +
- minute * NANOS_PER_SECOND * SECONDS_PER_MINUTE +
- hour * NANOS_PER_SECOND * SECONDS_PER_MINUTE * MINUTES_PER_HOUR
-NanoTime(julianDay, nanosOfDay).toBinary
+ def convertToTimestamp(value: Binary): Long = {
+Preconditions.checkArgument(value.length() == 12, "Must be 12 bytes")
+val buf = value.toByteBuffer
+buf.order(ByteOrder.LITTLE_ENDIAN)
+val timeOfDayNanos = buf.getLong
+val julianDay = buf.getInt
+DateTimeUtils.fromJulianDay(julianDay, timeOfDayNanos)
+ }
+
+ def convertFromTimestamp(ts: Long): Binary = {
--- End diff --
Similarly, I think we should add a comment here as well (it can be the
inverse of my other suggested Scaladoc comment).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113614492 [Test build #35296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35296/console) for PR 6759 at commit [`bfc437c`](https://github.com/apache/spark/commit/bfc437c0c3a52a0e7afa8fdfe4a5d545104a65ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113614535 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32859328
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala ---
@@ -499,69 +497,23 @@ private[parquet] object CatalystArrayConverter {
}
private[parquet] object CatalystTimestampConverter {
- // TODO most part of this comes from Hive-0.14
- // Hive code might have some issues, so we need to keep an eye on it.
- // Also we use NanoTime and Int96Values from parquet-examples.
- // We utilize jodd to convert between NanoTime and Timestamp
- val parquetTsCalendar = new ThreadLocal[Calendar]
- def getCalendar: Calendar = {
-// this is a cache for the calendar instance.
-if (parquetTsCalendar.get == null) {
-
parquetTsCalendar.set(Calendar.getInstance(TimeZone.getTimeZone("GMT")))
-}
-parquetTsCalendar.get
- }
- val NANOS_PER_SECOND: Long = 10
- val SECONDS_PER_MINUTE: Long = 60
- val MINUTES_PER_HOUR: Long = 60
- val NANOS_PER_MILLI: Long = 100
-
- def convertToTimestamp(value: Binary): Timestamp = {
-val nt = NanoTime.fromBinary(value)
-val timeOfDayNanos = nt.getTimeOfDayNanos
-val julianDay = nt.getJulianDay
-val jDateTime = new JDateTime(julianDay.toDouble)
-val calendar = getCalendar
-calendar.set(Calendar.YEAR, jDateTime.getYear)
-calendar.set(Calendar.MONTH, jDateTime.getMonth - 1)
-calendar.set(Calendar.DAY_OF_MONTH, jDateTime.getDay)
-
-// written in command style
-var remainder = timeOfDayNanos
-calendar.set(
- Calendar.HOUR_OF_DAY,
- (remainder / (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE *
MINUTES_PER_HOUR)
-calendar.set(
- Calendar.MINUTE, (remainder / (NANOS_PER_SECOND *
SECONDS_PER_MINUTE)).toInt)
-remainder = remainder % (NANOS_PER_SECOND * SECONDS_PER_MINUTE)
-calendar.set(Calendar.SECOND, (remainder / NANOS_PER_SECOND).toInt)
-val nanos = remainder % NANOS_PER_SECOND
-val ts = new Timestamp(calendar.getTimeInMillis)
-ts.setNanos(nanos.toInt)
-ts
- }
-
- def convertFromTimestamp(ts: Timestamp): Binary = {
-val calendar = getCalendar
-calendar.setTime(ts)
-val jDateTime = new JDateTime(calendar.get(Calendar.YEAR),
- calendar.get(Calendar.MONTH) + 1,
calendar.get(Calendar.DAY_OF_MONTH))
-// Hive-0.14 didn't set hour before get day number, while the day
number should
-// has something to do with hour, since julian day number grows at 12h
GMT
-// here we just follow what hive does.
-val julianDay = jDateTime.getJulianDayNumber
-
-val hour = calendar.get(Calendar.HOUR_OF_DAY)
-val minute = calendar.get(Calendar.MINUTE)
-val second = calendar.get(Calendar.SECOND)
-val nanos = ts.getNanos
-// Hive-0.14 would use hours directly, that might be wrong, since the
day starts
-// from 12h in Julian. here we just follow what hive does.
-val nanosOfDay = nanos + second * NANOS_PER_SECOND +
- minute * NANOS_PER_SECOND * SECONDS_PER_MINUTE +
- hour * NANOS_PER_SECOND * SECONDS_PER_MINUTE * MINUTES_PER_HOUR
-NanoTime(julianDay, nanosOfDay).toBinary
+ def convertToTimestamp(value: Binary): Long = {
--- End diff --
I think that this method should have Scaladoc. Based on the Scaladoc for
`readTimestamp` further up in this file, I think that we should say something
here to make it clear that we're converting input stored in int96 format, e.g.
```scala
/**
* Read a Timestamp value from a Parquet Int96Value; returns the number of
100ns since [... fill this in ...]
*/
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32857375
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -117,4 +124,27 @@ object DateUtils {
0L
}
}
+
+ /**
+ * Return the number of 100ns (hundred of nanoseconds) since epoch from
julian day
+ * and nanoseconds in a day
+ */
+ def fromJulianDay(day: Int, nanoseconds: Long): Long = {
+// use integer to avoid rounding errors
--- End diff --
It looks like we're using longs here to avoid the rounding errors, not
integers, so maybe we should update this comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32857204
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -17,18 +17,25 @@
package org.apache.spark.sql.catalyst.util
-import java.sql.{Timestamp, Date}
+import java.sql.{Date, Timestamp}
import java.text.SimpleDateFormat
import java.util.{Calendar, TimeZone}
import org.apache.spark.sql.catalyst.expressions.Cast
/**
- * Helper function to convert between Int value of days since 1970-01-01
and java.sql.Date
+ * Helper function to convert between Int value of days since epoch and
java.sql.Date,
--- End diff --
I think that we should re-word this slightly. I'd say something like this
(provided it's accurate):
> Helper functions for converting between internal and external date and
time representations.
>
> Dates are exposed externally as java.sql.Date and are represented
internally as the number of dates since the Unix epoch (1970-01-01).
Timestamps are exposed externally as java.sql.Timestamp and are stored
internally as longs, which are capable of storing timestamps with 100
nanosecond precision.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32856580
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -117,4 +124,27 @@ object DateUtils {
0L
}
}
+
+ /**
+ * Return the number of 100ns (hundred of nanoseconds) since epoch from
julian day
+ * and nanoseconds in a day
+ */
+ def fromJulianDay(day: Int, nanoseconds: Long): Long = {
+// use integer to avoid rounding errors
+val seconds = (day - JULIAN_DAY_OF_EPOCH).toLong * SECONDS_PER_DAY -
SECONDS_PER_DAY / 2
+seconds * HUNDRED_NANOS_PER_SECOND + nanoseconds / 100L
+ }
+
+ /**
+ * Return julian day and nanoseconds in a day from the number of 100ns
(hundred of nanoseconds)
--- End diff --
Minor nit, but I think Julian should be capitalized in Julian day.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32856533
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -117,4 +124,27 @@ object DateUtils {
0L
}
}
+
+ /**
+ * Return the number of 100ns (hundred of nanoseconds) since epoch from
julian day
+ * and nanoseconds in a day
+ */
+ def fromJulianDay(day: Int, nanoseconds: Long): Long = {
+// use integer to avoid rounding errors
+val seconds = (day - JULIAN_DAY_OF_EPOCH).toLong * SECONDS_PER_DAY -
SECONDS_PER_DAY / 2
+seconds * HUNDRED_NANOS_PER_SECOND + nanoseconds / 100L
+ }
+
+ /**
+ * Return julian day and nanoseconds in a day from the number of 100ns
(hundred of nanoseconds)
+ * @param num100ns
--- End diff --
If we're not going to document these Scaladoc parameters, then I think we
should leave them out, since it looks a bit messy in my IDE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113586589 [Test build #35296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35296/consoleFull) for PR 6759 at commit [`bfc437c`](https://github.com/apache/spark/commit/bfc437c0c3a52a0e7afa8fdfe4a5d545104a65ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113585740 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113585763 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113584267 [Test build #35294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35294/console) for PR 6759 at commit [`ae5979c`](https://github.com/apache/spark/commit/ae5979c7ea07b11903309ce1b141e958cd29982d). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113584270 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113582942 [Test build #35294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35294/consoleFull) for PR 6759 at commit [`ae5979c`](https://github.com/apache/spark/commit/ae5979c7ea07b11903309ce1b141e958cd29982d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113582798 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-113582763 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-112692322 @JoshRosen can you review this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-112653561 @rxin This is ready to review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111912647 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111912600 [Test build #34899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34899/console) for PR 6759 at commit [`602b969`](https://github.com/apache/spark/commit/602b969a643f0eb9f5bd09e286481e69d8857ba9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/6759#discussion_r32387873
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetConverter.scala ---
@@ -498,69 +493,21 @@ private[parquet] object CatalystArrayConverter {
}
private[parquet] object CatalystTimestampConverter {
- // TODO most part of this comes from Hive-0.14
- // Hive code might have some issues, so we need to keep an eye on it.
- // Also we use NanoTime and Int96Values from parquet-examples.
- // We utilize jodd to convert between NanoTime and Timestamp
- val parquetTsCalendar = new ThreadLocal[Calendar]
- def getCalendar: Calendar = {
-// this is a cache for the calendar instance.
-if (parquetTsCalendar.get == null) {
-
parquetTsCalendar.set(Calendar.getInstance(TimeZone.getTimeZone("GMT")))
-}
-parquetTsCalendar.get
- }
- val NANOS_PER_SECOND: Long = 10
- val SECONDS_PER_MINUTE: Long = 60
- val MINUTES_PER_HOUR: Long = 60
- val NANOS_PER_MILLI: Long = 100
+ // see
http://stackoverflow.com/questions/466321/convert-unix-timestamp-to-julian
+ val JULIAN_DAY_OF_EPOCH = 2440587.5
--- End diff --
I had verified this using the sample parquet file in SPARK-4768, it can
read by exact the same value back (with timzone difference).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111896333 [Test build #34899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34899/consoleFull) for PR 6759 at commit [`602b969`](https://github.com/apache/spark/commit/602b969a643f0eb9f5bd09e286481e69d8857ba9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111896174 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111896167 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111800160 [Test build #912 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/912/console) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111800273 [Test build #913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/913/console) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111800040 [Test build #911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/911/console) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111787259 [Test build #913 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/913/consoleFull) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111787249 [Test build #911 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/911/consoleFull) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111787252 [Test build #912 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/912/consoleFull) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111768698 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111768662 [Test build #34858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34858/console) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111767120 [Test build #34858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34858/consoleFull) for PR 6759 at commit [`2f2e48c`](https://github.com/apache/spark/commit/2f2e48cce9587350ad3ea79a1ea81a8f4217824c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111766707 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111766704 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111764798 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111764796 [Test build #34849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34849/console) for PR 6759 at commit [`8ace611`](https://github.com/apache/spark/commit/8ace6112b3ad7df37e43b9df006c5bba504a5e53). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111758286 [Test build #34849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34849/consoleFull) for PR 6759 at commit [`8ace611`](https://github.com/apache/spark/commit/8ace6112b3ad7df37e43b9df006c5bba504a5e53). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111758248 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111758242 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111754541 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111754539 [Test build #34842 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34842/console) for PR 6759 at commit [`212143b`](https://github.com/apache/spark/commit/212143b35e61aa2100d29b732c5c618c35225454). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111754146 [Test build #34841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34841/console) for PR 6759 at commit [`c834108`](https://github.com/apache/spark/commit/c834108c3499a134357cec511e2dfefcc02aada5). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111754149 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111753854 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111753873 [Test build #34842 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34842/consoleFull) for PR 6759 at commit [`212143b`](https://github.com/apache/spark/commit/212143b35e61aa2100d29b732c5c618c35225454). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111753849 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111753724 [Test build #34841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34841/consoleFull) for PR 6759 at commit [`c834108`](https://github.com/apache/spark/commit/c834108c3499a134357cec511e2dfefcc02aada5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8307] [SQL] improve timestamp from parq...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6759#issuecomment-111753702 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
