[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45579679 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2049] [sql] Avg function may cause over...
Github user egraldlo commented on the pull request: https://github.com/apache/spark/pull/978#issuecomment-45579617 yes, maybe I don't make the version of test branch to be up-to-date at that time. I will close the issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2049] [sql] Avg function may cause over...
Github user egraldlo closed the pull request at: https://github.com/apache/spark/pull/978 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45579557 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1917: fix PySpark import of scipy.specia...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/866#issuecomment-45579547 FYI we've also been using skipIf for a while in python/pyspark/mllib/tests.py. I guess you're trying Jenkins on 2.6 now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1026#discussion_r13579821
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala ---
@@ -533,7 +538,7 @@ class ALS private (
// Add regularization
var i = 0
while (i < rank) {
-fullXtX.data(i * rank + i) += lambda
+fullXtX.data(i * rank + i) += numRatings(index) * lambda
--- End diff --
minor: you can move `numRatings(index) * lambda` outside the while loop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45579331 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1917: fix PySpark import of scipy.specia...
Github user laserson commented on the pull request: https://github.com/apache/spark/pull/866#issuecomment-45579062 I guess that would work, though I've never seen anyone do that. `unittest2` was created to backport all of the new functionality in `unittest` to py2.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1917: fix PySpark import of scipy.specia...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/866#issuecomment-45578792 @laserson is it possible to just wrap the test in an "if statement"? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1917: fix PySpark import of scipy.specia...
Github user laserson commented on the pull request: https://github.com/apache/spark/pull/866#issuecomment-45578676 Yes, that's annoying about python 2.6 Note that I just copied the same pattern that was present in `spark/python/pyspark/mllib/tests.py`. One option would be to import `unittest2` if the interpreter is 2.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1917: fix PySpark import of scipy.specia...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/866#issuecomment-45578368 @mateiz @laserson This patched introduced a change which breaks the unit tests on Python 2.6. The `skipIf` feature is not supported in Python 2.7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1516]Throw exception in yarn client ins...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/490#issuecomment-45578114 @codeboyyong It is not mergable now. Do you mind merging the master branch and also create a separate PR for branch-0.9? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45577910 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45577911 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15604/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Spark 1384 - Fix spark-shell on yarn access to...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/287#issuecomment-45576980 LGTM. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45576556 LGTM. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45576278 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45576281 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15603/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45575878 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45575870 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1026#discussion_r13578603 --- Diff: spark --- @@ -0,0 +1 @@ +Subproject commit 91a64eba90032729873675c6f9c0e53b76b5cf56 --- End diff -- Remove this line. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2085: [MLlib] Apply user-specific regula...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1026#issuecomment-45575615 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1025#discussion_r13578489 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)]) } /** + * Return a subset of this RDD sampled by key (via stratified sampling). + * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith --- End diff -- Also, it would be good to put a comment saying that this method requires two passes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/1025#discussion_r13578453 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -156,6 +161,182 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)]) } /** + * Return a subset of this RDD sampled by key (via stratified sampling). + * We guarantee a sample size = math.ceil(fraction * S_i), where S_i is the size of the ith --- End diff -- + "with high probability" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578424
--- Diff:
core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution,
NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
+
+ def getLambda1(s: Double): Double = {
--- End diff --
`getLowerBound`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578415
--- Diff:
core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution,
NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
--- End diff --
`phi` is not used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578427
--- Diff:
core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution,
NormalDistribution}
+
+private[spark] object PoissonBounds {
+
+ val delta = 1e-4 / 3.0
+ val phi = new NormalDistribution().cumulativeProbability(1.0 - delta)
+
+ def getLambda1(s: Double): Double = {
+var lb = math.max(0.0, s - math.sqrt(s / delta)) // Chebyshev's
inequality
+var ub = s
+while (lb < ub - 1.0) {
+ val m = (lb + ub) / 2.0
+ val poisson = new PoissonDistribution(m, 1e-15)
+ val y = poisson.inverseCumulativeProbability(1 - delta)
+ if (y > s) ub = m else lb = m
+}
+lb
+ }
+
+ def getMinCount(lmbd: Double): Double = {
+if(lmbd == 0) return 0
+val poisson = new PoissonDistribution(lmbd, 1e-15)
+poisson.inverseCumulativeProbability(delta)
+ }
+
+ def getLambda2(s: Double): Double = {
--- End diff --
`getUpperBound`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1025#discussion_r13578391
--- Diff:
core/src/main/scala/org/apache/spark/util/random/SamplingUtils.scala ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.random
+
+import org.apache.commons.math3.distribution.{PoissonDistribution,
NormalDistribution}
+
+private[spark] object PoissonBounds {
--- End diff --
Though private, this object needs some documentation. I understand this is
a little unfair ... :p
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-45575097 @rxin I changed this to only get the position twice (at the beginning and end), and also added metrics to NewHadoopRDD, which I'd forgotten about previously. I also surrounded the calls to reader.getPos() with try/catch since they can throw exceptions. This should be good to go now (and I verified it works for both Hadoop APIs on EC2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2088] fix NPE in toString
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1028#issuecomment-45575003 @dorx Could you add a unit test for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574939 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574946 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574899 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/455 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1416: PySpark support for SequenceFile a...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/455#issuecomment-45574901 Thanks Nick for the latest changes and Patrick for the comments! I've merged this in now. This will be an awesome feature to have. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574852 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574847 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574848 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15602/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1027 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574719 LGTM. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1978] In some cases, spark-yarn does no...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/921#issuecomment-45574679 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15593/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1978] In some cases, spark-yarn does no...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/921#issuecomment-45574678 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574621 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574622 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15592/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574559 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574555 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574461 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574359 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15601/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574347 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574339 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574352 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15600/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574350 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574358 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/990#issuecomment-45574336 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15597/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45574353 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45574349 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574343 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15598/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574330 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45574322 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574348 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15599/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45574344 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45574325 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15595/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574342 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/990#issuecomment-45574334 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/990#issuecomment-45574335 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2088] fix NPE in toString
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1028#issuecomment-45574319 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574331 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15596/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2088] fix NPE in toString
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1028#issuecomment-45574321 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15594/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574326 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Moved hiveOperators.scala to the right package...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1029#issuecomment-45574323 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574240 Hey @liancheng - I think this refactoring will solve this particular ticket (i.e. queries w/ sorts will not be eagerly executed anymore). However, I don't see why we don't *need* `processCmd` and `eagerlyProcess` anymore; for instance, for set commands this PR's changes will eagerly call `toRdd`, but we still need some logic in the two `toRdd` to handle the set. Are you proposing to just take that piece of logic and put them into SchemaRDDLike? That could work. And what about `processCmd` then? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45573695 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.
Github user concretevitamin commented on a diff in the pull request:
https://github.com/apache/spark/pull/956#discussion_r13577888
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -269,6 +271,22 @@ class SQLContext(@transient val sparkContext:
SparkContext)
protected abstract class QueryExecution {
def logical: LogicalPlan
+def eagerlyProcess(plan: LogicalPlan): RDD[Row] = plan match {
+ case SetCommand(key, value) =>
--- End diff --
Hey @liancheng -- so this function `eagerlyProcess` is supposed to be
triggered as soon as `hql("set ...")` is typed in / is being evaluated. The
actual physical operator `SetCommandPhysical` handles evaluating the *actual
contents of the SchemaRDD*. These contents are not (and need not be, in the
spirit of RDDs) eagerly processed at this particular place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2049] [sql] Avg function may cause over...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/978#issuecomment-45573486 I think the test case you added already passes because of the rules [here](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala#L262). Am I missing something? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45573475 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1013#discussion_r13577643
--- Diff:
yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -507,12 +508,18 @@ object Client {
Apps.addToEnvironment(env, Environment.CLASSPATH.name,
Environment.PWD.$() +
Path.SEPARATOR + LOG4J_PROP)
}
+
+val cachedSecondaryJarLinks =
+
sparkConf.getOption(CONF_SPARK_YARN_SECONDARY_JARS).getOrElse("").split(",").filter(_.nonEmpty)
--- End diff --
line too wide?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1978] In some cases, spark-yarn does no...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/921#issuecomment-45572939 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45572938 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15591/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45572937 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Make sure that empty string is filtered out wh...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1027#issuecomment-45572901 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45572897 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45572899 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15590/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1940] Enabling rolling of executor logs...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/895#issuecomment-45572646 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/990#discussion_r13577271
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
case left :: Literal(null, _) :: Nil => Literal(null, e.dataType)
case _ => e
}
+ case e: CaseConversionExpression => e.children match {
--- End diff --
Good catch! Can you double check that constant folding works as expected
and then remove the unneeded rules?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1025#issuecomment-45571805 Build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45571809 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2082] stratified sampling in PairRDDFun...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1025#issuecomment-45571806 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15589/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Made deployment with --jars work ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45571772 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/999#issuecomment-45571770 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/999#issuecomment-45571771 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15588/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1714. Take advantage of AMRMClient APIs ...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/655#issuecomment-45571300 I tried testing this manually today and found that the locality stuff doesn't work even without this patch - https://issues.apache.org/jira/browse/SPARK-2089. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/956#discussion_r13576810
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -269,6 +271,22 @@ class SQLContext(@transient val sparkContext:
SparkContext)
protected abstract class QueryExecution {
def logical: LogicalPlan
+def eagerlyProcess(plan: LogicalPlan): RDD[Row] = plan match {
+ case SetCommand(key, value) =>
--- End diff --
I'm a bit confused that the actual logic for the `set` command is not
handled in `SetCommandPhysical` but here as a special case. I think we can
remove `eagerlyProcess` and `processCmd` in this file with the help of changes
made in PR #948, so that we can move the actual processing logic of `set`
command back to `SetCommandPhysical`. @rxin @marmbrus What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/990#discussion_r13576722
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
case left :: Literal(null, _) :: Nil => Literal(null, e.dataType)
case _ => e
}
+ case e: CaseConversionExpression => e.children match {
--- End diff --
Ok, I do.
BTW, I noticed that there are some Expressions in the same situation like
`UnaryMinus`, `Cast`, or `Not`, ( or `BinaryArithmetic`, `BinaryComparison`,
`StringRegexExpression`, too ? )
I think they also can be handled by constant folding.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45570350 @marmbrus @concretevitamin While working on the cache table SQL command and reviewing PR #956, I think this PR may make both changes cleaner, so I tried to finish this one first. For example, we won't need [`processCmd`](https://github.com/apache/spark/pull/956/files#diff-ff50aea397a607b79df9bec6f2a841dbR255) and [`eagerlyProcess`](https://github.com/apache/spark/pull/956/files#diff-131c27c6a1f59770d738b11f2a4755ecR274) anymore since all commands and insertions are eagerly executed in a unified way. Instead of `QueryExecution`, I factored the duplicated code into `SchemaRDDLike` to make the change cleaner and more straightforward (after some experiments, it turned out that moving code to `QueryExecution` complicates things a lot). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/990#discussion_r13576069
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
case left :: Literal(null, _) :: Nil => Literal(null, e.dataType)
case _ => e
}
+ case e: CaseConversionExpression => e.children match {
--- End diff --
Yes please.
On Jun 9, 2014 8:16 PM, "Takuya UESHIN" wrote:
> In
>
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
>
> > @@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
> > case left :: Literal(null, _) :: Nil => Literal(null,
e.dataType)
> > case _ => e
> >}
> > + case e: CaseConversionExpression => e.children match {
>
> Fixed the two points.
> Should I remove the mentioned lines?
>
> â
> Reply to this email directly or view it on GitHub
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/990#discussion_r13576056
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -154,6 +155,10 @@ object NullPropagation extends Rule[LogicalPlan] {
case left :: Literal(null, _) :: Nil => Literal(null, e.dataType)
case _ => e
}
+ case e: CaseConversionExpression => e.children match {
--- End diff --
Fixed the two points.
Should I remove the mentioned lines?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45569480 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2052] [SQL] Add optimization for CaseCo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/990#issuecomment-45569478 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [Spark 2060][SQL] Querying JSON Datasets with ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/999#issuecomment-45568509 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/956#issuecomment-45568506 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15586/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
