[GitHub] spark pull request: [SPARK-9186][SQL] make deterministic describin...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7525#issuecomment-122782123 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9153][SQL] codegen StirngLPad/StringRPa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7527#issuecomment-122781995 [Test build #1117 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1117/consoleFull) for PR 7527 at commit [`3840c6b`](https://github.com/apache/spark/commit/3840c6bbd4dbdbf4cdfb2913122eaa1dbda0d7e0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4598][WebUI]Task table pagination for t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7399#issuecomment-122781934 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9187][WebUI] Timeline view may show neg...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7526#issuecomment-122781928 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9153][SQL] codegen StirngLPad/StringRPa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7527#issuecomment-122781903 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9153][SQL] codegen StirngLPad/StringRPa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7527#issuecomment-122781817 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9026] Refactor SimpleFutureAction.onCom...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/7385#discussion_r34971118
--- Diff: core/src/main/scala/org/apache/spark/scheduler/JobWaiter.scala ---
@@ -50,28 +72,23 @@ private[spark] class JobWaiter[T](
}
override def taskSucceeded(index: Int, result: Any): Unit = synchronized
{
-if (_jobFinished) {
+if (isCompleted) {
throw new UnsupportedOperationException("taskSucceeded() called on a
finished JobWaiter")
}
resultHandler(index, result.asInstanceOf[T])
finishedTasks += 1
if (finishedTasks == totalTasks) {
- _jobFinished = true
- jobResult = JobSucceeded
- this.notifyAll()
+ promise.success()
}
}
override def jobFailed(exception: Exception): Unit = synchronized {
-_jobFinished = true
-jobResult = JobFailed(exception)
-this.notifyAll()
- }
-
- def awaitResult(): JobResult = synchronized {
-while (!_jobFinished) {
- this.wait()
+// There are certain situations where jobFailed can be called multiple
times for the same
+// job. We guard against this by making this method idempotent.
+if (!isCompleted) {
+ promise.failure(exception)
--- End diff --
Looks `tryFailure` would be simpler.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9153][SQL] codegen StirngLPad/StringRPa...
GitHub user tarekauel opened a pull request: https://github.com/apache/spark/pull/7527 [SPARK-9153][SQL] codegen StirngLPad/StringRPad Jira: https://issues.apache.org/jira/browse/SPARK-9153 You can merge this pull request into a Git repository by running: $ git pull https://github.com/tarekauel/spark SPARK-9153 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7527.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7527 commit 92b6a5d5d89c909ae408bc5fb58542225f1f915c Author: Tarek Auel Date: 2015-07-20T06:50:30Z [SPARK-9153] codegen lpad/rpad --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9091][STREAMING]Add the CompressionCode...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7442#issuecomment-122778962 [Test build #37813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37813/consoleFull) for PR 7442 at commit [`5bbf66c`](https://github.com/apache/spark/commit/5bbf66c0abd4a26500b1e591d606f1e438935eb7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9067][SQL] Close reader in NewHadoopRDD...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7424#issuecomment-122778668 cc @zsxwing for review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9091][STREAMING]Add the CompressionCode...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7442#issuecomment-122778516 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9091][STREAMING]Add the CompressionCode...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7442#issuecomment-122778570 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9067][SQL] Close reader in NewHadoopRDD...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7424#discussion_r34970834
--- Diff: core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala ---
@@ -141,6 +141,11 @@ class NewHadoopRDD[K, V](
override def hasNext: Boolean = {
if (!finished && !havePair) {
finished = !reader.nextKeyValue
+ if (finished) {
--- End diff --
would it make sense if we just call close here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122777314 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122777013 [Test build #37804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37804/console) for PR 7458 at commit [`3013579`](https://github.com/apache/spark/commit/301357979f9111e3c7106d86c111c275429178e6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-122776730 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-122776686 [Test build #37805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37805/console) for PR 7259 at commit [`48ee4b3`](https://github.com/apache/spark/commit/48ee4b3a4745fb206e84cb0c69a9e48b6456a332). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class ExpressionInfo ` * `case class DescribeFunction(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4598][WebUI]Task table pagination for t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7399#issuecomment-122776608 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9187][WebUI] Timeline view may show neg...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7526#issuecomment-122776531 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7456 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8756][SQL] Keep cached information and ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7154#issuecomment-122776765 [Test build #37812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37812/consoleFull) for PR 7154 at commit [`12a0ed9`](https://github.com/apache/spark/commit/12a0ed92a9ed310166b9fb50bc6b971fa18ad4ac). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9187][WebUI] Timeline view may show neg...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7526#issuecomment-122776566 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8756][SQL] Keep cached information and ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7154#issuecomment-122776552 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8756][SQL] Keep cached information and ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7154#issuecomment-122776612 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4598][WebUI]Task table pagination for t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7399#issuecomment-122776540 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7456#issuecomment-122776235 Looks pretty good. I'm going to merge it. Please submit a followup pr to address some of the comments on documentation and choice of buffer size. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9187][WebUI] Timeline view may show neg...
GitHub user carsonwang opened a pull request: https://github.com/apache/spark/pull/7526 [SPARK-9187][WebUI] Timeline view may show negative value for running tasks For running tasks, the executorRunTime metrics is 0 which causes negative executorComputingTime in the timeline. It also causes an incorrect SchedulerDelay time.  You can merge this pull request into a Git repository by running: $ git pull https://github.com/carsonwang/spark timeline-negValue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7526 commit 7b17db203702178900cba089d59522c186777ddf Author: Carson Wang Date: 2015-07-20T05:30:50Z Fix negative value in timeline view --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4598][WebUI]Task table pagination for t...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/7399#issuecomment-122775702 @andrewor14 Here is the new screenshot for the error message: https://cloud.githubusercontent.com/assets/1000778/8770962/0ac43d8c-2eed-11e5-90e2-2fef6949d9a4.png";> And if the user updates the page size, the page number will be reset to 1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7456#discussion_r34970583
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.io.{DataInputStream, DataOutputStream, OutputStream,
InputStream}
+import java.nio.ByteBuffer
+
+import scala.reflect.ClassTag
+
+import com.google.common.io.ByteStreams
+
+import org.apache.spark.serializer.{SerializationStream,
DeserializationStream, SerializerInstance, Serializer}
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.unsafe.PlatformDependent
+
+/**
+ * Serializer for serializing [[UnsafeRow]]s during shuffle. Since
UnsafeRows are already stored as
+ * bytes, this serializer simply copies those bytes to the underlying
output stream. When
+ * deserializing a stream of rows, instances of this serializer mutate and
return a single UnsafeRow
+ * instance that is backed by an on-heap byte array.
+ *
+ * Note that this serializer implements only the [[Serializer]] methods
that are used during
+ * shuffle, so certain [[SerializerInstance]] methods will throw
UnsupportedOperationException.
+ *
+ * This serializer does not support UnsafeRows that use
+ * [[org.apache.spark.sql.catalyst.util.ObjectPool]].
+ *
+ * @param numFields the number of fields in the row being serialized.
+ */
+private[sql] class UnsafeRowSerializer(numFields: Int) extends Serializer
with Serializable {
+ override def newInstance(): SerializerInstance = new
UnsafeRowSerializerInstance(numFields)
+ override private[spark] def supportsRelocationOfSerializedObjects:
Boolean = true
+}
+
+private class UnsafeRowSerializerInstance(numFields: Int) extends
SerializerInstance {
+
+ private[this] val EOF: Int = -1
--- End diff --
it'd be great to briefly explain the encoding format, i.e. length followed
by data, and if length < 0, eof.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7456#discussion_r34970552
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.io.{DataInputStream, DataOutputStream, OutputStream,
InputStream}
+import java.nio.ByteBuffer
+
+import scala.reflect.ClassTag
+
+import com.google.common.io.ByteStreams
+
+import org.apache.spark.serializer.{SerializationStream,
DeserializationStream, SerializerInstance, Serializer}
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.unsafe.PlatformDependent
+
+/**
+ * Serializer for serializing [[UnsafeRow]]s during shuffle. Since
UnsafeRows are already stored as
+ * bytes, this serializer simply copies those bytes to the underlying
output stream. When
+ * deserializing a stream of rows, instances of this serializer mutate and
return a single UnsafeRow
+ * instance that is backed by an on-heap byte array.
+ *
+ * Note that this serializer implements only the [[Serializer]] methods
that are used during
+ * shuffle, so certain [[SerializerInstance]] methods will throw
UnsupportedOperationException.
+ *
+ * This serializer does not support UnsafeRows that use
+ * [[org.apache.spark.sql.catalyst.util.ObjectPool]].
+ *
+ * @param numFields the number of fields in the row being serialized.
+ */
+private[sql] class UnsafeRowSerializer(numFields: Int) extends Serializer
with Serializable {
+ override def newInstance(): SerializerInstance = new
UnsafeRowSerializerInstance(numFields)
+ override private[spark] def supportsRelocationOfSerializedObjects:
Boolean = true
+}
+
+private class UnsafeRowSerializerInstance(numFields: Int) extends
SerializerInstance {
+
+ private[this] val EOF: Int = -1
+
+ override def serializeStream(out: OutputStream): SerializationStream =
new SerializationStream {
+private[this] var writeBuffer: Array[Byte] = new Array[Byte](4096)
+private[this] val dOut: DataOutputStream = new DataOutputStream(out)
+
+override def writeValue[T: ClassTag](value: T): SerializationStream = {
+ val row = value.asInstanceOf[UnsafeRow]
+ assert(row.getPool == null, "UnsafeRowSerializer does not support
ObjectPool")
+ dOut.writeInt(row.getSizeInBytes)
+ var dataRemaining: Int = row.getSizeInBytes
+ val baseObject = row.getBaseObject
+ var rowReadPosition: Long = row.getBaseOffset
+ while (dataRemaining > 0) {
+val toTransfer: Int = Math.min(writeBuffer.length, dataRemaining)
+PlatformDependent.copyMemory(
+ baseObject,
+ rowReadPosition,
+ writeBuffer,
+ PlatformDependent.BYTE_ARRAY_OFFSET,
+ toTransfer)
+out.write(writeBuffer, 0, toTransfer)
+rowReadPosition += toTransfer
+dataRemaining -= toTransfer
+ }
+ this
+}
+override def writeKey[T: ClassTag](key: T): SerializationStream = {
+ assert(key.isInstanceOf[Int])
+ this
+}
+override def writeAll[T: ClassTag](iter: Iterator[T]):
SerializationStream =
+ throw new UnsupportedOperationException
+override def writeObject[T: ClassTag](t: T): SerializationStream =
+ throw new UnsupportedOperationException
+override def flush(): Unit = dOut.flush()
+override def close(): Unit = {
+ writeBuffer = null
+ dOut.writeInt(EOF)
+ dOut.close()
+}
+ }
+
+ override def deserializeStream(in: InputStream): DeserializationStream =
{
+new DeserializationStream {
+ private[this] val dIn: DataInputStream = new DataInputStream(in)
+ private[this] var rowBuffer: Array[Byte] = new Array[Byte](1024)
--- End diff --
[GitHub] spark pull request: [SPARK-8756][SQL] Keep cached information and ...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/7154#issuecomment-122775568 @liancheng I've added the check you suggested. Please take a look when you have time. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7456#discussion_r34970544
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.io.{DataInputStream, DataOutputStream, OutputStream,
InputStream}
+import java.nio.ByteBuffer
+
+import scala.reflect.ClassTag
+
+import com.google.common.io.ByteStreams
+
+import org.apache.spark.serializer.{SerializationStream,
DeserializationStream, SerializerInstance, Serializer}
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.unsafe.PlatformDependent
+
+/**
+ * Serializer for serializing [[UnsafeRow]]s during shuffle. Since
UnsafeRows are already stored as
+ * bytes, this serializer simply copies those bytes to the underlying
output stream. When
+ * deserializing a stream of rows, instances of this serializer mutate and
return a single UnsafeRow
+ * instance that is backed by an on-heap byte array.
+ *
+ * Note that this serializer implements only the [[Serializer]] methods
that are used during
+ * shuffle, so certain [[SerializerInstance]] methods will throw
UnsupportedOperationException.
+ *
+ * This serializer does not support UnsafeRows that use
+ * [[org.apache.spark.sql.catalyst.util.ObjectPool]].
+ *
+ * @param numFields the number of fields in the row being serialized.
+ */
+private[sql] class UnsafeRowSerializer(numFields: Int) extends Serializer
with Serializable {
+ override def newInstance(): SerializerInstance = new
UnsafeRowSerializerInstance(numFields)
+ override private[spark] def supportsRelocationOfSerializedObjects:
Boolean = true
+}
+
+private class UnsafeRowSerializerInstance(numFields: Int) extends
SerializerInstance {
+
+ private[this] val EOF: Int = -1
+
+ override def serializeStream(out: OutputStream): SerializationStream =
new SerializationStream {
+private[this] var writeBuffer: Array[Byte] = new Array[Byte](4096)
+private[this] val dOut: DataOutputStream = new DataOutputStream(out)
+
+override def writeValue[T: ClassTag](value: T): SerializationStream = {
+ val row = value.asInstanceOf[UnsafeRow]
+ assert(row.getPool == null, "UnsafeRowSerializer does not support
ObjectPool")
+ dOut.writeInt(row.getSizeInBytes)
+ var dataRemaining: Int = row.getSizeInBytes
+ val baseObject = row.getBaseObject
+ var rowReadPosition: Long = row.getBaseOffset
+ while (dataRemaining > 0) {
--- End diff --
probably doesn't matter in the MVP, but if we know the UnsafeRow is backed
by a byte array, we don't need to do this copying, do we?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user tarekauel commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774605 Sure. I am going to solve some of them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9023] [SQL] Efficiency improvements for...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7456#discussion_r34970391
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala
---
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.io.{DataInputStream, DataOutputStream, OutputStream,
InputStream}
+import java.nio.ByteBuffer
+
+import scala.reflect.ClassTag
+
+import com.google.common.io.ByteStreams
+
+import org.apache.spark.serializer.{SerializationStream,
DeserializationStream, SerializerInstance, Serializer}
+import org.apache.spark.sql.catalyst.expressions.UnsafeRow
+import org.apache.spark.unsafe.PlatformDependent
+
+/**
+ * Serializer for serializing [[UnsafeRow]]s during shuffle. Since
UnsafeRows are already stored as
+ * bytes, this serializer simply copies those bytes to the underlying
output stream. When
+ * deserializing a stream of rows, instances of this serializer mutate and
return a single UnsafeRow
+ * instance that is backed by an on-heap byte array.
+ *
+ * Note that this serializer implements only the [[Serializer]] methods
that are used during
+ * shuffle, so certain [[SerializerInstance]] methods will throw
UnsupportedOperationException.
+ *
+ * This serializer does not support UnsafeRows that use
+ * [[org.apache.spark.sql.catalyst.util.ObjectPool]].
+ *
+ * @param numFields the number of fields in the row being serialized.
+ */
+private[sql] class UnsafeRowSerializer(numFields: Int) extends Serializer
with Serializable {
+ override def newInstance(): SerializerInstance = new
UnsafeRowSerializerInstance(numFields)
+ override private[spark] def supportsRelocationOfSerializedObjects:
Boolean = true
+}
+
+private class UnsafeRowSerializerInstance(numFields: Int) extends
SerializerInstance {
+
+ private[this] val EOF: Int = -1
+
+ override def serializeStream(out: OutputStream): SerializationStream =
new SerializationStream {
+private[this] var writeBuffer: Array[Byte] = new Array[Byte](4096)
+private[this] val dOut: DataOutputStream = new DataOutputStream(out)
+
+override def writeValue[T: ClassTag](value: T): SerializationStream = {
+ val row = value.asInstanceOf[UnsafeRow]
+ assert(row.getPool == null, "UnsafeRowSerializer does not support
ObjectPool")
+ dOut.writeInt(row.getSizeInBytes)
+ var dataRemaining: Int = row.getSizeInBytes
+ val baseObject = row.getBaseObject
+ var rowReadPosition: Long = row.getBaseOffset
+ while (dataRemaining > 0) {
+val toTransfer: Int = Math.min(writeBuffer.length, dataRemaining)
+PlatformDependent.copyMemory(
+ baseObject,
+ rowReadPosition,
+ writeBuffer,
+ PlatformDependent.BYTE_ARRAY_OFFSET,
+ toTransfer)
+out.write(writeBuffer, 0, toTransfer)
+rowReadPosition += toTransfer
+dataRemaining -= toTransfer
+ }
+ this
+}
+override def writeKey[T: ClassTag](key: T): SerializationStream = {
+ assert(key.isInstanceOf[Int])
--- End diff --
you need to add some comment explaining why we are not doing anything when
writing keys.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774568 [Test build #37809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37809/consoleFull) for PR 7516 at commit [`0c1313a`](https://github.com/apache/spark/commit/0c1313af67cadf75aaab49a08dd77cb4fd6db19e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774362 I triggered it finally. Meantime, if you have time, do you want to pick up some other issues here? https://issues.apache.org/jira/browse/SPARK-8829 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774336 [Test build #1116 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1116/consoleFull) for PR 7516 at commit [`0c1313a`](https://github.com/apache/spark/commit/0c1313af67cadf75aaab49a08dd77cb4fd6db19e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774312 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774305 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122774249 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9186][SQL] make deterministic describin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7525#issuecomment-122774240 [Test build #37808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37808/consoleFull) for PR 7525 at commit [`4189bfa`](https://github.com/apache/spark/commit/4189bfa26ba29967bfc2cf10c98a9bac8d7adaa2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user tarekauel commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122773919 @rxin Jenkins still doesn't like me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9186][SQL] make deterministic describin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7525#issuecomment-122773807 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9186][SQL] make deterministic describin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7525#issuecomment-122773813 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9186][SQL] make deterministic describin...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/7525 [SPARK-9186][SQL] make deterministic describing the tree rather than the expression You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark deterministic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7525.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7525 commit 4189bfa26ba29967bfc2cf10c98a9bac8d7adaa2 Author: Wenchen Fan Date: 2015-07-20T06:20:35Z make deterministic describing the tree rather than the expression --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-122773507 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7524 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-122773454 [Test build #37800 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37800/console) for PR 7365 at commit [`fd7eba4`](https://github.com/apache/spark/commit/fd7eba4d812ff43307ae766953a33afdfa6f8038). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7524#issuecomment-122773396 Thanks - merging this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/spark/pull/7524#discussion_r34970068
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -111,7 +111,7 @@ trait SchemaRelationProvider {
*
* The difference between a [[RelationProvider]] and a
[[HadoopFsRelationProvider]] is
* that users need to provide a schema and a (possibly empty) list of
partition columns when
- * using a SchemaRelationProvider. A relation provider can inherits both
[[RelationProvider]],
+ * using a HadoopFsRelationProvider. A relation provider can inherits both
[[RelationProvider]],
--- End diff --
Sure. Modified the same in SchemaRelationProvider
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9066][SQL] Improve cartesian performanc...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7417#issuecomment-122773389 [Test build #37807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37807/consoleFull) for PR 7417 at commit [`a168900`](https://github.com/apache/spark/commit/a1689000c21497eadddc594a922395236e62347e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9066][SQL] Improve cartesian performanc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7417#issuecomment-122773299 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9066][SQL] Improve cartesian performanc...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7417#issuecomment-122773289 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7523#discussion_r34969987
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala
---
@@ -198,14 +203,39 @@ case class AtLeastNNonNulls(n: Int, children:
Seq[Expression]) extends Predicate
val nonnull = ctx.freshName("nonnull")
val code = children.map { e =>
val eval = e.gen(ctx)
- s"""
-if ($nonnull < $n) {
- ${eval.code}
- if (!${eval.isNull}) {
-$nonnull += 1;
- }
-}
- """
+ e.dataType match {
+case DoubleType =>
+ s"""
+if ($nonnull < $n) {
+ ${eval.code}
+ if (!${eval.isNull}) {
+if (!Double.isNaN(${eval.primitive})) {
--- End diff --
let's leave that for future prs. don't think it is that big of a deal if
they are already in the codebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7524#issuecomment-122773205 [Test build #37806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37806/consoleFull) for PR 7524 at commit [`e10e211`](https://github.com/apache/spark/commit/e10e211da02a3a2cadc03fe5f7eb316f45395364). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user yjshen commented on a diff in the pull request:
https://github.com/apache/spark/pull/7523#discussion_r34969931
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala
---
@@ -198,14 +203,39 @@ case class AtLeastNNonNulls(n: Int, children:
Seq[Expression]) extends Predicate
val nonnull = ctx.freshName("nonnull")
val code = children.map { e =>
val eval = e.gen(ctx)
- s"""
-if ($nonnull < $n) {
- ${eval.code}
- if (!${eval.isNull}) {
-$nonnull += 1;
- }
-}
- """
+ e.dataType match {
+case DoubleType =>
+ s"""
+if ($nonnull < $n) {
+ ${eval.code}
+ if (!${eval.isNull}) {
+if (!Double.isNaN(${eval.primitive})) {
--- End diff --
yes, actually there's a bunch of code could just be checked with
`Double.isNaN`, but use Float.isNaN and Double.isNaN separately, should I
simplify them all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7524#issuecomment-122772974 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][DOC] Minor document fix in HadoopFsRelat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7524#issuecomment-122772976 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9177][SQL] Reuse of calendar object in ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7516#issuecomment-122772915 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Minor document fix in HadoopFsRelationPr...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7524#discussion_r34969879
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -111,7 +111,7 @@ trait SchemaRelationProvider {
*
* The difference between a [[RelationProvider]] and a
[[HadoopFsRelationProvider]] is
* that users need to provide a schema and a (possibly empty) list of
partition columns when
- * using a SchemaRelationProvider. A relation provider can inherits both
[[RelationProvider]],
+ * using a HadoopFsRelationProvider. A relation provider can inherits both
[[RelationProvider]],
--- End diff --
can you put [[ ]] around this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7523#issuecomment-122772598 While you are at this, can you add some unit test for the AtLeastNNonNulls expression to NullFunctionsSuite? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-122772380 [Test build #37805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37805/consoleFull) for PR 7259 at commit [`48ee4b3`](https://github.com/apache/spark/commit/48ee4b3a4745fb206e84cb0c69a9e48b6456a332). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9178][SQL] Add an empty string constant...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7509#issuecomment-122772372 There is a failure ``` [error] Test org.apache.spark.unsafe.types.UTF8StringSuite.concatTest failed: java.lang.NullPointerException: null [error] at org.apache.spark.unsafe.types.UTF8String.concat(UTF8String.java:407) [error] at org.apache.spark.unsafe.types.UTF8StringSuite.concatTest(UTF8StringSuite.java:100) [error] ... ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7523#issuecomment-122772382 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-122772276 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7259#issuecomment-122772268 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9178][SQL] Add an empty string constant...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7509#issuecomment-122772218 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7523#issuecomment-122771394 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7523#discussion_r34969510
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullFunctions.scala
---
@@ -113,7 +113,7 @@ case class IsNaN(child: Expression) extends
UnaryExpression
boolean ${ev.isNull} = false;
${ctx.javaType(dataType)} ${ev.primitive} =
${ctx.defaultValue(dataType)};
if (${eval.isNull}) {
--- End diff --
can you change this to
```
${ev. primitive} = !${ev.isNull} && Float.isNaN(${eval.primitive});
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9081][SQL][WIP] fillna/dropna should al...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7523#issuecomment-122771343 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9185][SQL] improve code gen for mutable...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7521 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9185][SQL] improve code gen for mutable...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7521#issuecomment-122769537 I'm going to merge this so it unblocks @tarekauel. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9185][SQL] improve code gen for mutable...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7521#issuecomment-122768840 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122767504 4. How to run scalastyle properly configured to pass jenkins check ./dev/scalastyle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7259#discussion_r34968904
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -300,3 +300,72 @@ case class ShowTablesCommand(databaseName:
Option[String]) extends RunnableComma
rows
}
}
+
+/**
+ * A command for users to list all of the registered functions.
+ * The syntax of using this command in SQL is:
+ * {{{
+ *SHOW FUNCTIONS
+ * }}}
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
--- End diff --
If we do that we should create a shim that we can stabilize, rather than
exposing expression directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9172][SQL] Make DecimalPrecision suppor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7511#issuecomment-122766137 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9172][SQL] Make DecimalPrecision suppor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7511#issuecomment-122765993 [Test build #37795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37795/console) for PR 7511 at commit [`4d29d10`](https://github.com/apache/spark/commit/4d29d10565d09f2a2bad593b9eaa8eede5165538). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122764883 [Test build #37804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37804/consoleFull) for PR 7458 at commit [`3013579`](https://github.com/apache/spark/commit/301357979f9111e3c7106d86c111c275429178e6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122764387 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-12276 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122764195 [Test build #37802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37802/consoleFull) for PR 7462 at commit [`710780c`](https://github.com/apache/spark/commit/710780c42768f6a27b29f707366f0f476cbc0eac). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user tarekauel commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122764175 @EntilZha 1. `eval` and `nullSafeEval` `eval` will be invoked to evaluate the expression. Most expressions should return `null` if one of there arguments is `null`. In order to avoid that every expression has to check if `left` or `right` is `null`, `nullSafeEval` has been added. `eval` does the null check and calls `nullSafeEval`, see. https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L289-L313 You should override `eval` if you don't want to return `null`, if one the arguments is `null`. Most of the times you will use `nullSafeEval`. 2. `UnaryExpression`: Expression has one parameter (like `size(x)`) `BinaryExpression`: Expression has two parameters (like `contains(a, b)`) `ExpectsInputTypes`: Allows to automatically check if the argument type is correct, see https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExpectsInputTypes.scala#L42-L57. You specify the allowed types by overriding `inputTypes`. `ImplicitCastInputs`: The difference to `ExpectsInputTypes` is that this tries to cast the value. Most string operations are implemented with a byte array as input. A string can be "casted" to a byte array by calling `.getBytes`. `ImplicitCastInputs` allows to call `contains(s: String, s2: String)` and `contains(s: Array[Byte], s2: Array[Byte])`. Typically you use this if a cast is reasonable. Cast from anything else to string is most of the times reasonable, but casting a string (automatically == implicit) to an integer value is most of the time not helpful. Users could still invoke the `cast` function. 3. I don't know 4. Intellij allows to run most suites from the IDE. And have a look at https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-RunningIndividualTests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122764105 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122764106 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122764095 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-122764101 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user EntilZha commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122764068 Fixed tests using feedback from @chenghao-intel --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/7462#discussion_r34968615
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala
---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types._
+
+
+class CollectionFunctionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
+
+ test("Array and Map Size") {
+val a0 = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+val a1 = Literal.create(Array[Integer](), ArrayType(IntegerType))
+val a2 = Literal.create(Array(1, 2), ArrayType(IntegerType))
+
+checkEvaluation(Size(a0), 3)
--- End diff --
Is this spark representation of types or scala representation of types?
Fixed that, fixing another error from codegen. When I get that figured out (or
stuck) will push that up
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122762289 [Test build #37801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37801/consoleFull) for PR 7462 at commit [`4d0e29d`](https://github.com/apache/spark/commit/4d0e29d90dd092f6da4e9cffed21d10407c79664). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/7462#discussion_r34968571
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionFunctionsSuite.scala
---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.types._
+
+
+class CollectionFunctionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
+
+ test("Array and Map Size") {
+val a0 = Literal.create(Array(1, 2, 3), ArrayType(IntegerType))
+val a1 = Literal.create(Array[Integer](), ArrayType(IntegerType))
+val a2 = Literal.create(Array(1, 2), ArrayType(IntegerType))
+
+checkEvaluation(Size(a0), 3)
--- End diff --
`[I` means the type of `Array of Integer`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122762012 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [WIP] Aggregation Improveme...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7458#discussion_r34968498
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate2/Aggregate2Sort.scala
---
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.aggregate2
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.errors._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.aggregate2._
+import org.apache.spark.sql.catalyst.plans.physical.{AllTuples,
ClusteredDistribution, Distribution, UnspecifiedDistribution}
+import org.apache.spark.sql.execution.{SparkPlan, UnaryNode}
+import org.apache.spark.sql.types.NullType
+
+import scala.collection.mutable.ArrayBuffer
+
+case class Aggregate2Sort(
+groupingExpressions: Seq[NamedExpression],
+aggregateExpressions: Seq[AggregateExpression2],
+aggregateAttributes: Seq[Attribute],
+resultExpressions: Seq[NamedExpression],
+child: SparkPlan)
+ extends UnaryNode {
+
+ /** Indicates if this operator is for partial aggregations. */
+ val partialAggregation: Boolean = {
+aggregateExpressions.map(_.mode).distinct.toList match {
+ case Partial :: Nil => true
+ case Final :: Nil => false
+ case other =>
+sys.error(
+ s"Could not evaluate ${aggregateExpressions} because we do not
support evaluate " +
+ s"modes $other in this operator.")
+}
+ }
+
+ override def requiredChildDistribution: List[Distribution] = {
+if (partialAggregation) {
+ UnspecifiedDistribution :: Nil
+} else {
+ if (groupingExpressions == Nil) {
+AllTuples :: Nil
+ } else {
+ClusteredDistribution(groupingExpressions) :: Nil
+ }
+}
+ }
+
+ override def requiredChildOrdering: Seq[Seq[SortOrder]] =
+groupingExpressions.map(SortOrder(_, Ascending)) :: Nil
+
+ override def output: Seq[Attribute] =
resultExpressions.map(_.toAttribute)
+
+ protected override def doExecute(): RDD[InternalRow] = attachTree(this,
"execute") {
+child.execute().mapPartitions { iter =>
+
+ new Iterator[InternalRow] {
--- End diff --
@chenghao-intel Yeah, that's a good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-122760458 [Test build #37800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37800/consoleFull) for PR 7365 at commit [`fd7eba4`](https://github.com/apache/spark/commit/fd7eba4d812ff43307ae766953a33afdfa6f8038). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8007][SPARK-8003][SQL] Added virtual co...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7478#discussion_r34968303
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
---
@@ -90,4 +90,11 @@ class HiveTableScanSuite extends HiveComparisonTest {
assert(sql("select CaseSensitiveColName from spark_4959_2").head() ===
Row("hi"))
assert(sql("select casesensitivecolname from spark_4959_2").head() ===
Row("hi"))
}
+
+ test("SPARK-8003 Resolve spark__partition__id in Hive queries") {
+TestHive.sql("CREATE TABLE tb (KEY INT, VALUE STRING)")
--- End diff --
OK. Then, `org.apache.spark.sql.hive.execution.SQLQuerySuite` will be a
better place for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8867][SQL][WIP] Support list / describe...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/7259#discussion_r34968226
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -300,3 +300,72 @@ case class ShowTablesCommand(databaseName:
Option[String]) extends RunnableComma
rows
}
}
+
+/**
+ * A command for users to list all of the registered functions.
+ * The syntax of using this command in SQL is:
+ * {{{
+ *SHOW FUNCTIONS
+ * }}}
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
--- End diff --
Ok, I will remove it for now.
But do we have plan to expose the Expression as API for advanced developers
in the future?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user EntilZha commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122759346 I pushed updates to the code based on the comments above. I wrote an expressions test, but can't figure out why its failing. I get the following stacktrace: ``` [info] CollectionFunctionsSuite: [info] - Array and Map Size *** FAILED *** (958 milliseconds) [info] Exception evaluating size([I@7e14fccd) (ExpressionEvalHelper.scala:69) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:496) [info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555) [info] at org.scalatest.Assertions$class.fail(Assertions.scala:1348) [info] at org.scalatest.FunSuite.fail(FunSuite.scala:1555) [info] at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:69) [info] at org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluationWithoutCodegen(CollectionFunctionsSuite.scala:24) [info] at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluation(ExpressionEvalHelper.scala:43) [info] at org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluation(CollectionFunctionsSuite.scala:24) [info] at org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply$mcV$sp(CollectionFunctionsSuite.scala:31) [info] at org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26) [info] at org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26) [info] at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42) [info] at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) [info] at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) [info] at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) [info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175) [info] at org.scalatest.FunSuite.runTest(FunSuite.scala:1555) [info] at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208) [info] at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208) [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413) [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401) [info] at scala.collection.immutable.List.foreach(List.scala:318) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) [info] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396) [info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483) [info] at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208) [info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1555) [info] at org.scalatest.Suite$class.run(Suite.scala:1424) [info] at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555) [info] at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212) [info] at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212) [info] at org.scalatest.SuperEngine.runImpl(Engine.scala:545) [info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212) [info] at org.scalatest.FunSuite.run(FunSuite.scala:1555) [info] at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) [info] at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:294) [info] at sbt.ForkMain$Run$2.call(ForkMain.java:284) [info] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [info] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [info] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [info] at java.lang.Thread.run(Thread.java:745) [info] Cause: java.lang.ClassCastException: [
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-122758917 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122758959 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-122758958 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8230][SQL] Add array/map size method
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7462#issuecomment-122758899 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
