[GitHub] spark issue #14417: [SPARK-16812] Open up SparkILoop.getAddedJars
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14417 **[Test build #63036 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63036/consoleFull)** for PR 14417 at commit [`237a3f4`](https://github.com/apache/spark/commit/237a3f4ab1e8ef1a68a0bcdfe8d2624d0bd7d9c6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14417: [SPARK-16812] Open up SparkILoop.getAddedJars
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14417 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14417: [SPARK-16812] Open up SparkILoop.getAddedJars
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14417 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63036/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet co...
GitHub user holdenk opened a pull request: https://github.com/apache/spark/pull/14419 [SPARK-16814][SQL][WIP] Fix deprecated parquet constructor usage ## What changes were proposed in this pull request? Replace deprecated ParquetWriter with the new builders ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/holdenk/spark SPARK-16814-fix-deprecated-parquet-constructor-usage Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14419.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14419 commit 9fa8c8f1f3d2e519541ae988e009116615375b34 Author: Holden Karau Date: 2016-07-29T20:46:33Z start refactoring to builder commit b41a057c7fab71fd0f0b7564a158de568d919f35 Author: Holden Karau Date: 2016-07-30T01:33:13Z Merge branch 'master' into fix-deprecated-parquet-constructor-usage commit 1ba76430eaf64804fe983ed50d967dbc58c5f8da Author: Holden Karau Date: 2016-07-30T06:04:52Z Use the builder commit e10c147db6fa9250bbaf16faee944dd22d741ada Author: Holden Karau Date: 2016-07-30T07:01:31Z Cleanup some other deprecated parquet writer constructors --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14419 **[Test build #63039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63039/consoleFull)** for PR 14419 at commit [`e10c147`](https://github.com/apache/spark/commit/e10c147db6fa9250bbaf16faee944dd22d741ada). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14418: [SPARK-16813][SQL] Remove private[sql] and private[spark...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14418 **[Test build #63037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63037/consoleFull)** for PR 14418 at commit [`5fa81a9`](https://github.com/apache/spark/commit/5fa81a954948ce815528f04e82267a24e8cc4594). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class MonotonicallyIncreasingID() extends LeafExpression with Nondeterministic ` * `case class SparkPartitionID() extends LeafExpression with Nondeterministic ` * `case class AggregateExpression(` * `case class CurrentDatabase() extends LeafExpression with Unevaluable ` * `class GenericInternalRow(val values: Array[Any]) extends BaseGenericInternalRow ` * `class AbstractScalaRowIterator[T] extends Iterator[T] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14418: [SPARK-16813][SQL] Remove private[sql] and private[spark...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14418 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63037/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14418: [SPARK-16813][SQL] Remove private[sql] and private[spark...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14418 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14407: [SPARK-16779][TRIVIAL][WIP] Avoid using postfix operator...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14407 **[Test build #63040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63040/consoleFull)** for PR 14407 at commit [`df67cde`](https://github.com/apache/spark/commit/df67cdec42d52f87c55a52b2d1ceb785ee0bf443). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14416 **[Test build #63038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63038/consoleFull)** for PR 14416 at commit [`b7a53f2`](https://github.com/apache/spark/commit/b7a53f2902f5de0c4bca1dbaa76e4c70bfd92b86). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14416 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63038/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14419 **[Test build #63039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63039/consoleFull)** for PR 14419 at commit [`e10c147`](https://github.com/apache/spark/commit/e10c147db6fa9250bbaf16faee944dd22d741ada). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ParquetWriterBuilder() extends` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14419 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14419 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63039/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14407: [SPARK-16779][TRIVIAL][WIP] Avoid using postfix operator...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14407 **[Test build #63040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63040/consoleFull)** for PR 14407 at commit [`df67cde`](https://github.com/apache/spark/commit/df67cdec42d52f87c55a52b2d1ceb785ee0bf443). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14407: [SPARK-16779][TRIVIAL][WIP] Avoid using postfix operator...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63040/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14407: [SPARK-16779][TRIVIAL][WIP] Avoid using postfix operator...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14407 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14419 Do we know if this works across versions of Parquet that are still in play for Spark builds? I know we've got some YARN and Scala warnings that we can't adjust for that reason. Just double checking. @HyukjinKwon did you look at this too or am I imagining it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14416: [SPARK-16776][STREAMING] Replace deprecated API i...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14416#discussion_r72886131
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
---
@@ -178,8 +178,9 @@ private[kafka010] class KafkaTestUtils extends Logging {
/** Send the array of messages to the Kafka broker */
def sendMessages(topic: String, messages: Array[String]): Unit = {
-producer = new Producer[String, String](new
ProducerConfig(producerConfiguration))
-producer.send(messages.map { new KeyedMessage[String, String](topic, _
) }: _*)
+producer = new KafkaProducer[String, String](producerConfiguration)
+val records = messages.map { new ProducerRecord[String, String](topic,
_) }
+records.map(producer.send)
--- End diff --
This type of thing is even essential for correctness in some cases, like
https://issues.apache.org/jira/browse/SPARK-16664
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14332: [SPARK-16694] [CORE] Use for/foreach rather than map for...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14332 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14380: [SPARK-16485][DOC][ML] Remove useless latex in scaladoc ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14380 @lins05 can you update with a fix for the additional occurrences? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14405 Jenkins add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14405 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14332: [SPARK-16694] [CORE] Use for/foreach rather than ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14332 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14405 **[Test build #63041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63041/consoleFull)** for PR 14405 at commit [`d3059d5`](https://github.com/apache/spark/commit/d3059d517e97e4c3605995d02b344d77076e9a55). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14402#discussion_r72886476
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
---
@@ -52,11 +46,16 @@ public JavaDoubleRDD union(JavaDoubleRDD... rdds) {
if (rdds.length == 0) {
throw new IllegalArgumentException("Union called on empty list");
}
-List> rest = new ArrayList<>(rdds.length - 1);
+rest = populateRDDList(rdds);
+return union(rdds[0], rest);
+ }
+
+ public static final List populateRDDList(T rdds) {
--- End diff --
I agree with making the method "private".But rdds is not any varargs so why
do we need ...??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14405 **[Test build #63041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63041/consoleFull)** for PR 14405 at commit [`d3059d5`](https://github.com/apache/spark/commit/d3059d517e97e4c3605995d02b344d77076e9a55). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63041/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14405 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14402: Update JavaSparkContextVarargsWorkaround.java
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14402#discussion_r72886519
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
---
@@ -52,11 +46,16 @@ public JavaDoubleRDD union(JavaDoubleRDD... rdds) {
if (rdds.length == 0) {
throw new IllegalArgumentException("Union called on empty list");
}
-List> rest = new ArrayList<>(rdds.length - 1);
+rest = populateRDDList(rdds);
+return union(rdds[0], rest);
+ }
+
+ public static final List populateRDDList(T rdds) {
--- End diff --
This doesn't build. You're using `T` like an array type but it isn't. It
has to be an array type as far as the method is concerned. `T[]` would be fine
as the type.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14402#discussion_r72886550
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
---
@@ -52,11 +46,16 @@ public JavaDoubleRDD union(JavaDoubleRDD... rdds) {
if (rdds.length == 0) {
throw new IllegalArgumentException("Union called on empty list");
}
-List> rest = new ArrayList<>(rdds.length - 1);
+rest = populateRDDList(rdds);
+return union(rdds[0], rest);
+ }
+
+ public static final List populateRDDList(T rdds) {
--- End diff --
I created a build with T...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14402#discussion_r72886590
--- Diff:
core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
---
@@ -52,11 +46,16 @@ public JavaDoubleRDD union(JavaDoubleRDD... rdds) {
if (rdds.length == 0) {
throw new IllegalArgumentException("Union called on empty list");
}
-List> rest = new ArrayList<>(rdds.length - 1);
+rest = populateRDDList(rdds);
+return union(rdds[0], rest);
+ }
+
+ public static final List populateRDDList(T rdds) {
--- End diff --
will this work ??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on the issue: https://github.com/apache/spark/pull/14402 Any update on the recent change ?? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14402 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14402 **[Test build #63042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63042/consoleFull)** for PR 14402 at commit [`3e831e5`](https://github.com/apache/spark/commit/3e831e5538ba7960119a1db426072d0f22c85331). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14402 **[Test build #63042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63042/consoleFull)** for PR 14402 at commit [`3e831e5`](https://github.com/apache/spark/commit/3e831e5538ba7960119a1db426072d0f22c85331). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14402 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63042/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14402 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14402 Yeah, still doesn't compile. Are you compiling this locally? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14420: [SPARK-14204] [SQL] register driverClass rather t...
GitHub user mchalek opened a pull request: https://github.com/apache/spark/pull/14420 [SPARK-14204] [SQL] register driverClass rather than user-specified class This is a pull request that was originally merged against branch-1.6 as #12000, now being merged into master as well. @srowen @zzcclp @JoshRosen This pull request fixes an issue in which cluster-mode executors fail to properly register a JDBC driver when the driver is provided in a jar by the user, but the driver class name is derived from a JDBC URL (rather than specified by the user). The consequence of this is that all JDBC accesses under the described circumstances fail with an IllegalStateException. I reported the issue here: https://issues.apache.org/jira/browse/SPARK-14204 My proposed solution is to have the executors register the JDBC driver class under all circumstances, not only when the driver is specified by the user. This patch was tested manually. I built an assembly jar, deployed it to a cluster, and confirmed that the problem was fixed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mchalek/spark mchalek-jdbc_driver_registration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14420.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14420 commit c0bd08ef36b8b4ed073fcdc2f88927389593e47f Author: Kevin McHale Date: 2016-03-23T17:11:54Z register driverClass rather than user class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14420 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14420 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14420 **[Test build #63043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63043/consoleFull)** for PR 14420 at commit [`c0bd08e`](https://github.com/apache/spark/commit/c0bd08ef36b8b4ed073fcdc2f88927389593e47f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14416: [SPARK-16776][STREAMING] Replace deprecated API i...
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/14416#discussion_r72888085
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
---
@@ -178,8 +178,9 @@ private[kafka010] class KafkaTestUtils extends Logging {
/** Send the array of messages to the Kafka broker */
def sendMessages(topic: String, messages: Array[String]): Unit = {
-producer = new Producer[String, String](new
ProducerConfig(producerConfiguration))
-producer.send(messages.map { new KeyedMessage[String, String](topic, _
) }: _*)
+producer = new KafkaProducer[String, String](producerConfiguration)
+val records = messages.map { new ProducerRecord[String, String](topic,
_) }
+records.foreach(producer.send)
--- End diff --
I think the point here was that you don't need to map at all;
messages.foreach should suffice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/14416 LGTM, minor comments about map notwithstanding. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [Spark-16916] Add api to get JavaSparkContext fro...
GitHub user phalodi opened a pull request: https://github.com/apache/spark/pull/14421 [Spark-16916] Add api to get JavaSparkContext from SparkSession ## What changes were proposed in this pull request? Add api to get JavaSparkContext from SparkSession ## How was this patch tested? Run all test cases of spark You can merge this pull request into a Git repository by running: $ git pull https://github.com/phalodi/spark SPARK-16816 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14421.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14421 commit b9134baa3ab8b4154e09f24ea04600395e44a928 Author: sandy Date: 2016-07-30T13:57:02Z [Spark-16916] Add api to get JavaSparkContext from SparkSession --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14421: [Spark-16916] Add api to get JavaSparkContext from Spark...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14421 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14416 **[Test build #63044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63044/consoleFull)** for PR 14416 at commit [`a1ef311`](https://github.com/apache/spark/commit/a1ef31129faa3c571bfd58c196ce3757eec890b8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [SPARK-16816] Add api to get JavaSparkContext fro...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/14421#discussion_r72888442 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -122,6 +122,14 @@ class SparkSession private( val sqlContext: SQLContext = new SQLContext(this) /** + * This is the interface through which the user can get [[JavaSparkContext]] + * + * @since 2.0.0 + */ + @transient + val javaSparkContext: JavaSparkContext = new JavaSparkContext(sparkContext) --- End diff -- This always make a JavaSparkContext. Why not just have the caller do this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14419 Yes, actually I took a look for this as well and it seems okay (I was thinking doing this with other issues). If my understanding is correct, this would not affect actual writing in Spark but just tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [SPARK-16816] Add api to get JavaSparkContext fro...
Github user phalodi commented on a diff in the pull request: https://github.com/apache/spark/pull/14421#discussion_r72888602 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -122,6 +122,14 @@ class SparkSession private( val sqlContext: SQLContext = new SQLContext(this) /** + * This is the interface through which the user can get [[JavaSparkContext]] + * + * @since 2.0.0 + */ + @transient + val javaSparkContext: JavaSparkContext = new JavaSparkContext(sparkContext) --- End diff -- But when user write code in java and he/she wants the JavaSparkConext, then need to get the sparkContext and create the java context with JavaSparkContext method fromSparkContext(sc) so its more good to provide directly JavaSparkContext from spark session to the user. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [SPARK-16816] Add api to get JavaSparkContext fro...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/14421#discussion_r72888618 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -122,6 +122,14 @@ class SparkSession private( val sqlContext: SQLContext = new SQLContext(this) /** + * This is the interface through which the user can get [[JavaSparkContext]] + * + * @since 2.0.0 + */ + @transient + val javaSparkContext: JavaSparkContext = new JavaSparkContext(sparkContext) --- End diff -- Yeah but this exposes a Java-specific API and always creates the object for all callers. It's minor either way, but that seems even less nice. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [SPARK-16816] Add api to get JavaSparkContext fro...
Github user phalodi commented on a diff in the pull request: https://github.com/apache/spark/pull/14421#discussion_r72888636 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -122,6 +122,14 @@ class SparkSession private( val sqlContext: SQLContext = new SQLContext(this) /** + * This is the interface through which the user can get [[JavaSparkContext]] + * + * @since 2.0.0 + */ + @transient + val javaSparkContext: JavaSparkContext = new JavaSparkContext(sparkContext) --- End diff -- or we can add this code in spark.javaSparkContext so its more good for user to use it because context is primary thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14421: [SPARK-16816] Add api to get JavaSparkContext fro...
Github user phalodi commented on a diff in the pull request: https://github.com/apache/spark/pull/14421#discussion_r72888685 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala --- @@ -122,6 +122,14 @@ class SparkSession private( val sqlContext: SQLContext = new SQLContext(this) /** + * This is the interface through which the user can get [[JavaSparkContext]] + * + * @since 2.0.0 + */ + @transient + val javaSparkContext: JavaSparkContext = new JavaSparkContext(sparkContext) --- End diff -- @srowen Yeah you are right but for java developers its pain when they get sparkContext from spark session and call parallelize method on it and its give error that its take argument type of any scala seq so i think for java developer or user its more easy to get api. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12000: [SPARK-14204] [SQL] register driverClass rather than use...
Github user zzcclp commented on the issue: https://github.com/apache/spark/pull/12000 thanks, @mchalek --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14333: [SPARK-16696][ML][MLLib] destroy KMeans bcNewCenters whe...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/14333 @srowen I check code again, the problem I mentioned above `But now I found another problem in BisectKMeans: in line 191 there is a iteration it also need this pattern âpersist current step RDD, unpersist previousâ` is NOT A PROBLEM, I misread code. And I check other modifications in this PR again and it seems no problems. So I think the PR is OK now. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14333: [SPARK-16696][ML][MLLib] destroy KMeans bcNewCenters whe...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14333 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples that fail ...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14405 Merged to master/2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14333: [SPARK-16696][ML][MLLib] destroy KMeans bcNewCent...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14333 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14405: [SPARK-16800][EXAMPLES][ML] Fix Java examples tha...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14405 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14420 **[Test build #63043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63043/consoleFull)** for PR 14420 at commit [`c0bd08e`](https://github.com/apache/spark/commit/c0bd08ef36b8b4ed073fcdc2f88927389593e47f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14400: [Spark-16791] [SQL] cast struct with timestamp fi...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14400#discussion_r72889243
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
---
@@ -727,6 +727,16 @@ class CastSuite extends SparkFunSuite with
ExpressionEvalHelper {
}
}
+ test("cast struct with a timestamp field") {
+val originalSchema = new StructType().add( "tsField", TimestampType,
nullable = false )
--- End diff --
please remove the space in `add( xxx )`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14420 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14420: [SPARK-14204] [SQL] register driverClass rather than use...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14420 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63043/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14363 **[Test build #63045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63045/consoleFull)** for PR 14363 at commit [`80d2f50`](https://github.com/apache/spark/commit/80d2f505974f6f94549ccbc5d5014ab83e46a99a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions
GitHub user xubo245 opened a pull request: https://github.com/apache/spark/pull/14422 Add rand(numRows: Int, numCols: Int) functions ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) add rand(numRows: Int, numCols: Int) functions to DenseMatrix object,like breeze.linalg.DenseMatrix.rand() You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/spark patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14422.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14422 commit a7a1261f52112a3bca375dd0bed1c1bc0a2e0ed8 Author: å¾æ³¢ <601450...@qq.com> Date: 2016-07-30T15:43:36Z Add rand(numRows: Int, numCols: Int) functions add rand(numRows: Int, numCols: Int) functions to DenseMatrix object,like breeze.linalg.DenseMatrix.rand() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions
Github user xubo245 commented on the issue: https://github.com/apache/spark/pull/14422 we can use it to replacebreeze.linalg.DenseMatrix.rand(numRows: Int, numCols: Int) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14422 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14231: [SPARK-16586] Change the way the exit code of launcher i...
Github user zasdfgbnm commented on the issue: https://github.com/apache/spark/pull/14231 Yes this patch looks clearer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user drcrallen commented on the issue: https://github.com/apache/spark/pull/13713 OoooOOO master updated to 1.0.0 Fixed merge conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user drcrallen commented on the issue: https://github.com/apache/spark/pull/13713 @tnachen this test is a little larger than I originally anticipated. Let me see if I can add some unit tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13713 **[Test build #63046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63046/consoleFull)** for PR 13713 at commit [`bad9d69`](https://github.com/apache/spark/commit/bad9d695fd9d24d5043182c00a1537953ceb7bd4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14422#discussion_r72889871
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
---
@@ -497,6 +497,20 @@ object DenseMatrix {
}
/**
+* Generate a `DenseMatrix` consisting of `i.i.d.` uniform random
numbers.
+*
+* @param numRows number of rows of the matrix
+* @param numCols number of columns of the matrix
+* @return DenseMatrix` with size `numRows` x `numCols` and values in
U(0, 1)
+*/
+ @Since("2.0.0")
+ def rand(numRows: Int, numCols: Int): DenseMatrix = {
+require(numRows.toLong * numCols <= Int.MaxValue,
+ s"$numRows x $numCols dense matrix is too large to allocate")
+new DenseMatrix(numRows, numCols, Array.fill(numRows * numCols)((new
Random).nextDouble()))
--- End diff --
This makes a new RNG for every element, which isn't great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on the issue: https://github.com/apache/spark/pull/14402 Added the required fix.Can you please start the build ?? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14419 Could cc @liancheng who has more insight about thie please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14416 **[Test build #63044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63044/consoleFull)** for PR 14416 at commit [`a1ef311`](https://github.com/apache/spark/commit/a1ef31129faa3c571bfd58c196ce3757eec890b8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14416 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14416: [SPARK-16776][STREAMING] Replace deprecated API in Kafka...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63044/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14402 @sakky11 you have not even addressed the compilation error. This is wasting everyone's time. Please _compile_ this locally and fix it before pinging again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on the issue: https://github.com/apache/spark/pull/14402 Hey sorry @srowen ,Actually I have fixed the compilation error but missed it in adding in a pull request.Sorry. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14423: Add zeros(size: Int) function
GitHub user xubo245 opened a pull request: https://github.com/apache/spark/pull/14423 Add zeros(size: Int) function ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Generate a `DenseVector` consisting of zeros. It can replace breeze.linalg.DenseVector#zeros[Double] You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/spark patch-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14423.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14423 commit b111cdb9236c87af18e1ea773cea72b73fc68561 Author: å¾æ³¢ <601450...@qq.com> Date: 2016-07-30T16:30:47Z Add zeros(size: Int) function Generate a `DenseVector` consisting of zeros. It can replace breeze.linalg.DenseVector#zeros[Double] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14423: Add zeros(size: Int) function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14423 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14422: Add rand(numRows: Int, numCols: Int) functions
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14422#discussion_r72890434
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
---
@@ -497,6 +497,20 @@ object DenseMatrix {
}
/**
+* Generate a `DenseMatrix` consisting of `i.i.d.` uniform random
numbers.
+*
+* @param numRows number of rows of the matrix
+* @param numCols number of columns of the matrix
+* @return DenseMatrix` with size `numRows` x `numCols` and values in
U(0, 1)
+*/
+ @Since("2.0.0")
+ def rand(numRows: Int, numCols: Int): DenseMatrix = {
+require(numRows.toLong * numCols <= Int.MaxValue,
+ s"$numRows x $numCols dense matrix is too large to allocate")
+new DenseMatrix(numRows, numCols, Array.fill(numRows * numCols)((new
Random).nextDouble()))
--- End diff --
Can fix RNG, This makes a new RNG for all element :
val rng = new Random()
new DenseMatrix(numRows, numCols, Array.fill(numRows *
numCols)(rng.nextDouble()))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14380: [SPARK-16485][DOC][ML] Remove useless latex in scaladoc ...
Github user lins05 commented on the issue: https://github.com/apache/spark/pull/14380 @srowen I have looked about those files, but for complex equations, (e.g. [this one](https://github.com/apache/spark/blob/v2.0.0/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala#L778-L782)) I found it difficult to strip the latex symbols without affecting the readability of the javadoc: With the latex symbols there, the users can copy the equation and feed it to a latex viewer like [mathjax online previewer](https://cdn.mathjax.org/mathjax/latest/test/sample-dynamic.html), but it would make it impossible if we remove latex symbols from it. Meanwhile I find it's possible to use MathJax to render the latex in the scaladoc, as long as we add the necessary javascript to load the MathJax in the scaladoc javascript template. I played with it a bit and you can see the screenshot: https://cloud.githubusercontent.com/assets/717363/17271812/d9c40b00-56b8-11e6-9c17-affac4706153.png";> Do you think it's a good idea to do it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14422 Doesn't that need a JIRA, test and PR descriptio maybe? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14363 **[Test build #63045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63045/consoleFull)** for PR 14363 at commit [`80d2f50`](https://github.com/apache/spark/commit/80d2f505974f6f94549ccbc5d5014ab83e46a99a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14363 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63045/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14363: [SPARK-16731][SQL] use StructType in CatalogTable and re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14363 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14402: Update JavaSparkContextVarargsWorkaround.java
Github user sakky11 commented on the issue: https://github.com/apache/spark/pull/14402 I think the issue should be fixed now.Sorry for trouble. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13713 **[Test build #63046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63046/consoleFull)** for PR 13713 at commit [`bad9d69`](https://github.com/apache/spark/commit/bad9d695fd9d24d5043182c00a1537953ceb7bd4). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging with Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63046/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13713: [SPARK-15994] [MESOS] Allow enabling Mesos fetch cache i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13713 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14424: Add test:DenseMatrix.rand with no rng
GitHub user xubo245 opened a pull request: https://github.com/apache/spark/pull/14424 Add test:DenseMatrix.rand with no rng ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Add test:DenseMatrix.rand with no rng You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/spark patch-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14424.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14424 commit 2e945479af45fdadd8abb4529173db04226de64e Author: å¾æ³¢ <601450...@qq.com> Date: 2016-07-30T18:00:01Z Add test:DenseMatrix.rand with no rng Add test:DenseMatrix.rand with no rng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14422: Add rand(numRows: Int, numCols: Int) functions
Github user xubo245 commented on the issue: https://github.com/apache/spark/pull/14422 I add test : https://github.com/apache/spark/pull/14424 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14424: Add test:DenseMatrix.rand with no rng
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14424 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14424: Add test:DenseMatrix.rand with no rng
Github user xubo245 commented on the issue: https://github.com/apache/spark/pull/14424 It add for https://github.com/apache/spark/pull/14422 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14419 So yes as @HyukjinKwon says its just in the tests. I figured this would be ok across the hadoop versions we target because I looked at the build deps files and they all listed parquet-hadoop-1.8.1 for the different hadoop profiles we build with (`grep -r parquet-hadoop ./dev/deps/`) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14419: [SPARK-16814][SQL][WIP] Fix deprecated parquet construct...
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/14419 @HyukjinKwon do you have a PR for this? If so I can go ahead and close this one (and lets maybe sync some on the deprecation stuff we are both cleaning up :)) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14406: [SPARK-16778][SQL][TRIVIAL] Fix deprecation warni...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/14406#discussion_r72892893
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala ---
@@ -28,7 +28,7 @@ class SQLConfSuite extends QueryTest with
SharedSQLContext {
test("propagate from spark conf") {
// We create a new context here to avoid order dependence with other
tests that might call
// clear().
-val newContext = new SQLContext(sparkContext)
+val newContext = new
SQLContext(SparkSession.builder().sparkContext(sc).getOrCreate())
--- End diff --
oops thought this built locally but I guess I was on the wrong branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
