[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14160 **[Test build #62188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62188/consoleFull)** for PR 14160 at commit [`7557e1c`](https://github.com/apache/spark/commit/7557e1cf5916331255bebca9d7e4682d8ca4021c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14162 Correct, MR throws an exception, but again a lot of that is legacy from hadoop 1.x when it was the only thing running. The auxiliary services were added just for that originally. You could very well argue its a bug there too. So you are seeing users hit the max number of executor failures then? Without blacklisting I can definitely see this as an issue. The other question is why is it failing to start on the NM? is it port conflict like you mentioned or is it just something bad happened on one node? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Queries
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14132 I'm back. I'll resolve them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14162 To answer my own question: the MR service throws exceptions when there's an error during initialization. (See `ShuffleHandler.java`, there's no swallowing of exceptions during `serviceInit` and `serviceStart`.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14162 > What is the case you are seeing this issue? I'm ok with changing it if we have a good reason. Well, I guess "good reason" is in the eye of the beholder. :-) My argument for the change is to avoid users complaining when their app fails because of this. Instead, they'd have a clear warning that something is not right on a certain NM, and the rest of the world would keep going, using other NMs. I can see the argument for not affecting other services; but how does the MR service behave? Also, the current behavior leads not only to app failures, but also to a lot of noise in the NM's logs. So at least that part could be fixed, if people really don't want the behavior change, so that the shuffle service is really not running instead of running in a broken state. Finally, this is a single error spot that is being filtered... e.g., if there's a port conflict, the Spark shuffle service will throw an exception and prevent the NM from starting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Qu...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/14132#discussion_r70525213
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -86,6 +86,7 @@ class Analyzer(
WindowsSubstitution,
EliminateUnions),
Batch("Resolution", fixedPoint,
+ ResolveHint ::
--- End diff --
Sure!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14162 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14162 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62190/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14162 **[Test build #62190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62190/consoleFull)** for PR 14162 at commit [`61a40b1`](https://github.com/apache/spark/commit/61a40b16a028605a68f9b0ae1da5470b1d2b52df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Support basic INFORMATION_SCHEMA
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14116 Actually, this PR makes that test more robust generally. But, at the final commit of this PR, if this is not required, I will revert the change on R side in order to make PR smaller. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Support basic INFORMATION_SCHEMA
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/14116 Spark temporary views (tables) are included, too. When you specify any databases, `show tables` also returns the tables of the database and the Spark temporary table. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14162 So we had specifically decided to have this behavior when this was first written. The reason is that an issue with the spark shuffle services shouldn't stop other services from running fine on the NM. ie the mapreduce shuffle services. The node still works fine for MR even if there is bug in spark shuffle service. This was definitely a concern when we first released this. That isn't as much of an issue now. We had talked about this again recently and again decided to leave this behavior, the reason is that it should fail fast, ie as soon as it registers the executor would fail and there wouldn't be any wasted work. I guess this could cause the job to fail if it kept trying to launch on some bad node. Or is it not really killing the executor? What is the case you are seeing this issue? I'm ok with changing it if we have a good reason. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14113: [SPARK-16457] [SQL] Fix Wrong Messages when CTAS with a ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14113 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in partiti...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/13756 cc @liancheng @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14162 **[Test build #62190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62190/consoleFull)** for PR 14162 at commit [`61a40b1`](https://github.com/apache/spark/commit/61a40b16a028605a68f9b0ae1da5470b1d2b52df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14079: [SPARK-8425][CORE] New Blacklist Mechanism
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r70519430
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler
+
+import java.util.concurrent.atomic.AtomicReference
+
+import scala.collection.mutable.{HashMap, HashSet}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config
+import org.apache.spark.util.Clock
+import org.apache.spark.util.SystemClock
+import org.apache.spark.util.Utils
+
+/**
+ * BlacklistTracker is designed to track problematic executors and nodes.
It supports blacklisting
+ * specific (executor, task) pairs within a stage, blacklisting entire
executors and nodes for a
+ * stage, and blacklisting executors and nodes across an entire
application (with a periodic
+ * expiry).
+ *
+ * The tracker needs to deal with a variety of workloads, eg.: bad user
code, which may lead to many
+ * task failures, but that should not count against individual executors;
many small stages, which
+ * may prevent a bad executor for having many failures within one stage,
but still many failures
+ * over the entire application; "flaky" executors, that don't fail every
task, but are still
+ * faulty; etc.
+ *
+ * THREADING: As with most helpers of TaskSchedulerImpl, this is not
thread-safe. Though it is
+ * called by multiple threads, callers must already have a lock on the
TaskSchedulerImpl. The
--- End diff --
nit spacing off.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14079: [SPARK-8425][CORE] New Blacklist Mechanism
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r70519286
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -97,6 +97,49 @@ package object config {
.toSequence
.createWithDefault(Nil)
+ // Blacklist confs
+ private[spark] val BLACKLIST_ENABLED =
+ConfigBuilder("spark.scheduler.blacklist.enabled")
+.booleanConf
+.createOptional
+
+ private[spark] val MAX_TASK_FAILURES_PER_NODE =
+ConfigBuilder("spark.blacklist.maxTaskFailuresPerNode")
--- End diff --
spark.blacklist.maxFailedTasksPerNode to match naming of others?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14161: [MINOR] Fix Java style errors and remove unused imports
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14161 **[Test build #62189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62189/consoleFull)** for PR 14161 at commit [`e2b2edf`](https://github.com/apache/spark/commit/e2b2edfbbbd0445623bf7b86a28969b0e36435c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14161: [MINOR] Fix Java style errors and remove unused imports
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14161 LGTM if it passes tests, yes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Propagate error during shuffle servi...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/14162 @tgravescs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14162: [SPARK-16505][yarn] Propagate error during shuffl...
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/14162 [SPARK-16505][yarn] Propagate error during shuffle service startup. This prevents the NM from starting when something is wrong, which would lead to later errors which are confusing and harder to debug. Added a unit test to verify startup fails if something is wrong. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark SPARK-16505 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14162 commit 61a40b16a028605a68f9b0ae1da5470b1d2b52df Author: Marcelo Vanzin Date: 2016-07-12T20:43:51Z [SPARK-16505][yarn] Propagate error during shuffle service startup. This prevents the NM from starting when something is wrong, which would lead to later errors which are confusing and harder to debug. Added a unit test to verify startup fails if something is wrong. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14161: [MINOR] Fix Java style errors and remove unused i...
GitHub user keypointt opened a pull request: https://github.com/apache/spark/pull/14161 [MINOR] Fix Java style errors and remove unused imports ## What changes were proposed in this pull request? Fix Java style errors and remove unused imports, which are randomly found ## How was this patch tested? Tested on my local machine. You can merge this pull request into a Git repository by running: $ git pull https://github.com/keypointt/spark SPARK-16437 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14161 commit 7695ef67dea632e6c9bd3cbf5f83337542f56fd5 Author: Xin Ren Date: 2016-07-12T17:54:27Z [SPARK-16437] very minor change... commit ee64d572de60309005a5cd29d7ad7b05b1c46a80 Author: Xin Ren Date: 2016-07-12T20:38:11Z [SPARK-16437] remove redundant modifier commit e2b2edfbbbd0445623bf7b86a28969b0e36435c8 Author: Xin Ren Date: 2016-07-12T20:39:42Z [SPARK-16437] remove unused import --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user keypointt commented on the issue: https://github.com/apache/spark/pull/14160 I'll fix these styles, thanks a lot Sean --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13894 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62187/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13894 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14160 Yeah, the problem is we don't have anything that runs dev/lint-java regularly. If you find it currently shows errors in master, go ahead and fix them in a [MINOR] PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13894 **[Test build #62187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62187/consoleFull)** for PR 13894 at commit [`f9725cc`](https://github.com/apache/spark/commit/f9725ccf09268aa82beb57ce89ce04ff3be456e3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user keypointt commented on the issue: https://github.com/apache/spark/pull/14160 Hi @srowen I just found it's a duplicate so I closed it. But, when I ran `./dev/lint-java` I got these style problems. I'm just wondering for these random tiny style fixes, what should I do, just submit a [MINOR] PR? ``` Using `mvn` from path: /usr/local/bin/mvn Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Checkstyle checks failed at following occurrences: [ERROR] src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java:[27,1] (imports) RedundantImport: Redundant import from the same package - org.apache.spark.unsafe.memory.MemoryAllocator. [ERROR] src/main/java/org/apache/spark/unsafe/memory/MemoryAllocator.java:[26,3] (modifier) RedundantModifier: Redundant 'public' modifier. [ERROR] src/main/java/org/apache/spark/unsafe/memory/MemoryAllocator.java:[30,3] (modifier) RedundantModifier: Redundant 'public' modifier. [ERROR] src/main/java/org/apache/spark/unsafe/memory/MemoryAllocator.java:[31,3] (modifier) RedundantModifier: Redundant 'public' modifier. [ERROR] src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java:[143] (sizes) LineLength: Line is longer than 100 characters (found 113). [ERROR] src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java:[208] (sizes) LineLength: Line is longer than 100 characters (found 106). [ERROR] Failed to execute goal on project spark-streaming-kafka-0-10-assembly_2.11: Could not resolve dependencies for project org.apache.spark:spark-streaming-kafka-0-10-assembly_2.11:jar:2.0.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-streaming-kafka-0-10_2.11:jar:2.0.0-SNAPSHOT in apache.snapshots (http://repository.apache.org/snapshots) -> [Help 1] ``` https://github.com/keypointt/spark/blob/7557e1cf5916331255bebca9d7e4682d8ca4021c/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L143 https://github.com/keypointt/spark/blob/7557e1cf5916331255bebca9d7e4682d8ca4021c/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L208 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14160 I don't know that there's actually a problem, just saying that this is the kind of thing we should be careful about. There are a number of build warnings like this. Your change could be fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user keypointt commented on the issue: https://github.com/apache/spark/pull/14160 oh I see...sorry didn't notice it's for backward compatibility I'll close it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14160: [SPARK-16502] update depreciated ParquetFileReade...
Github user keypointt closed the pull request at: https://github.com/apache/spark/pull/14160 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14159: [SQL][PARQUET] Fix for Vectorized Parquet filter pushdow...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14159 https://github.com/apache/spark/pull/14160 solves the same thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14160 https://github.com/apache/spark/pull/14159 solves the same thing... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13670: [SPARK-15951] Change Executors Page to use datatables to...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/13670 Made a pass through and some mostly minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13894 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62185/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13894 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13894 **[Test build #62185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62185/consoleFull)** for PR 13894 at commit [`013a1db`](https://github.com/apache/spark/commit/013a1dbe06c2babf68b597b4b025a156a9723b03). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14160 The thing we have to be careful of in general is making some change that isn't compatible with older versions of things like Hadoop (and their Parquet dependencies). This change could break older Hadoop builds. No idea, but, just suggesting that this is the thing you'd have to test, and the PR builder won't necessarily test vs, say, Hadoop 2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14116: [SPARK-16452][SQL][WIP] Support basic INFORMATION_SCHEMA
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/14116 maybe I'm missing something, shouldn't `show tables` only returns tables from the select database? if I didn't `use information_schema` then `show tables` shouldn't include `tables`, `schemata` etc. right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14116: [SPARK-16452][SQL][WIP] Support basic INFORMATION...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/14116#discussion_r70511249
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -502,20 +502,21 @@ test_that("jsonRDD() on a RDD with json string", {
})
test_that("test tableNames and tables", {
+ previous_count <- length(tableNames())
df <- read.json(jsonPath)
createOrReplaceTempView(df, "table1")
- expect_equal(length(tableNames()), 1)
+ expect_true(length(tableNames()) == 1 + previous_count)
--- End diff --
ah cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14138: [SPARK-16284][SQL] Implement reflect SQL function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14138 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62183/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14160: [SPARK-16502] update depreciated ParquetFileReader()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14160 **[Test build #62188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62188/consoleFull)** for PR 14160 at commit [`7557e1c`](https://github.com/apache/spark/commit/7557e1cf5916331255bebca9d7e4682d8ca4021c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14138: [SPARK-16284][SQL] Implement reflect SQL function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14138 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14138: [SPARK-16284][SQL] Implement reflect SQL function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14138 **[Test build #62183 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62183/consoleFull)** for PR 14138 at commit [`b574c5d`](https://github.com/apache/spark/commit/b574c5d8abf93fcb34e7a2f88af14fdeb7d1bc66). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/13670#discussion_r70510298 --- Diff: docs/monitoring.md --- @@ -288,7 +288,11 @@ where `[base-app-id]` is the YARN application ID. /applications/[app-id]/executors -A list of all executors for the given application. +A list of all active executors for the given application. + + +/applications/[app-id]/executors --- End diff -- yep, I think you meant allexecutors --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14160: [SPARK-16502] update depreciated ParquetFileReade...
GitHub user keypointt opened a pull request:
https://github.com/apache/spark/pull/14160
[SPARK-16502] update depreciated ParquetFileReader()
https://issues.apache.org/jira/browse/SPARK-16502
## What changes were proposed in this pull request?
During code compile, got below depreciation message.
```
/Users/quickmobile/workspace/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
Warning:(448, 28) method listType in object ConversionPatterns is
deprecated: see corresponding Javadoc for more information.
ConversionPatterns.listType(
^
Warning:(464, 28) method listType in object ConversionPatterns is
deprecated: see corresponding Javadoc for more information.
ConversionPatterns.listType(
^
```
Here just updated the method invocation.
from
```
/** @deprecated */
public ParquetFileReader(Configuration configuration, Path filePath,
List blocks, List columns) throws IOException {
this(configuration, (FileMetaData)null, filePath, blocks, columns);
}
```
to:
```
public ParquetFileReader(Configuration configuration, FileMetaData
fileMetaData, Path filePath, List blocks, List
columns) throws IOException {
```
## How was this patch tested?
Tested on local machine, running unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/keypointt/spark SPARK-16502
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14160.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14160
commit 7557e1cf5916331255bebca9d7e4682d8ca4021c
Author: Xin Ren
Date: 2016-07-12T19:02:22Z
[SPARK-16502] update depreciated ParquetFileReader()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70509851
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/AllExecutorListResource.scala
---
@@ -0,0 +1,43 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+*http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.spark.status.api.v1
+
+import javax.ws.rs.{GET, PathParam, Produces}
+import javax.ws.rs.core.MediaType
+
+import org.apache.spark.ui.SparkUI
+import org.apache.spark.ui.exec.ExecutorsPage
+
+@Produces(Array(MediaType.APPLICATION_JSON))
+private[v1] class AllExecutorListResource(ui: SparkUI) {
+
+ @GET
+ def executorList(): Seq[ExecutorSummary] = {
+val listener = ui.executorsListener
+listener.synchronized {
+ // The follow codes should be protected by `listener` to make sure
no executors will be
+ // removed before we query their status. See SPARK-12784.
+ val storageStatusList = listener.activeStorageStatusList
--- End diff --
can we just change this to be activeNumExecutors =
listener.activeStorageStatusList.size since we don't ever use this list other
then to get the size. same with below for dead ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14159: [PARQUET] Fix for Parquet filter pushdown
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14159 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13831: [SPARK-16119][sql] Support PURGE option to drop t...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13831 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70508838
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/AllExecutorListResource.scala
---
@@ -0,0 +1,43 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+*http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+package org.apache.spark.status.api.v1
+
+import javax.ws.rs.{GET, PathParam, Produces}
--- End diff --
PathParam doesn't appear to be used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14159: [PARQUET] Fix for Parquet filter pushdown
GitHub user andreweduffy opened a pull request: https://github.com/apache/spark/pull/14159 [PARQUET] Fix for Parquet filter pushdown ## What changes were proposed in this pull request? Fix parquet filter pushdown from not reaching all the way down to the file level Use of previous deprecated constructor defaults to null metadata, which prevents pushdown from reaching the Parquet level. ## How was this patch tested? Looking at output of collects from SparkShell, before were printing warnings about CorruptStatistics, preventing pushing down filters to individual parquet files. Now able to use the metadata in each file to pushdown. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) You can merge this pull request into a Git repository by running: $ git pull https://github.com/andreweduffy/spark bugfix/pushdown Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14159 commit f825ad709cdc3c89d0cc7e41d0410998e6cc7541 Author: Andrew Duffy Date: 2016-07-12T19:41:22Z Fix for Parquet filter pushdown Use of previous deprecated constructor defaults to null metadata, which prevents pushdown from reaching the Parquet level. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13831: [SPARK-16119][sql] Support PURGE option to drop table / ...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/13831 Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70508419
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
--- End diff --
sorry I'm assuming I'm wrong here because we have things like:
+{data: 'totalInputBytes', render: formatBytes },
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14113: [SPARK-16457] [SQL] Fix Wrong Messages when CTAS with a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14113 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14113: [SPARK-16457] [SQL] Fix Wrong Messages when CTAS with a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14113 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62181/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70507431
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark issue #14113: [SPARK-16457] [SQL] Fix Wrong Messages when CTAS with a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14113 **[Test build #62181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62181/consoleFull)** for PR 14113 at commit [`26bddfd`](https://github.com/apache/spark/commit/26bddfd78697fc1354d3f8ed6a26a0e52b7d9cb4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/13670#discussion_r70507279 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html --- @@ -0,0 +1,102 @@ + + +
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/13670#discussion_r70507194 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html --- @@ -0,0 +1,102 @@ + + +
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13894 **[Test build #62187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62187/consoleFull)** for PR 13894 at commit [`f9725cc`](https://github.com/apache/spark/commit/f9725ccf09268aa82beb57ce89ce04ff3be456e3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70506931
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70506844
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70506386
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70506185
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark issue #14147: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, Developer...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14147 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14147: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, Developer...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14147 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62184/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14147: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, Developer...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14147 **[Test build #62184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62184/consoleFull)** for PR 14147 at commit [`5bd0d38`](https://github.com/apache/spark/commit/5bd0d384f27674eafcf908f7cc05c08c469b1aae). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12004 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62186/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12004 **[Test build #62186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62186/consoleFull)** for PR 12004 at commit [`d609126`](https://github.com/apache/spark/commit/d609126dbd4da75d6001cf931b08927c7113a889). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70505486
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark issue #12004: [SPARK-7481][build] [WIP] Add Hadoop 2.6+ spark-cloud mo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12004 **[Test build #62186 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62186/consoleFull)** for PR 12004 at commit [`d609126`](https://github.com/apache/spark/commit/d609126dbd4da75d6001cf931b08927c7113a889). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14112: [SPARK-16240][ML] Model loading backward compatibility f...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/14112 @hhbyyh I agree about not changing the other models' loading code unless it becomes necessary. I hope we can design a better long-term solution during 2.1. We should definitely cover the other LDA classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14112: [SPARK-16240][ML] Model loading backward compatib...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/14112#discussion_r70504864
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala ---
@@ -566,26 +565,52 @@ object LocalLDAModel extends
MLReadable[LocalLDAModel] {
}
}
+ private case class Data(
+ vocabSize: Int,
+ topicsMatrix: Matrix,
+ docConcentration: Vector,
+ topicConcentration: Double,
+ gammaShape: Double)
+
private class LocalLDAModelReader extends MLReader[LocalLDAModel] {
private val className = classOf[LocalLDAModel].getName
override def load(path: String): LocalLDAModel = {
+ // Import implicits for Dataset Encoder
+ val sparkSession = super.sparkSession
+ import sparkSession.implicits._
+
val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
val dataPath = new Path(path, "data").toString
val data = sparkSession.read.parquet(dataPath)
-.select("vocabSize", "topicsMatrix", "docConcentration",
"topicConcentration",
- "gammaShape")
-.head()
- val vocabSize = data.getAs[Int](0)
- val topicsMatrix = data.getAs[Matrix](1)
- val docConcentration = data.getAs[Vector](2)
- val topicConcentration = data.getAs[Double](3)
- val gammaShape = data.getAs[Double](4)
+ val vectorConverted = MLUtils.convertVectorColumnsToML(data,
"docConcentration")
+ val Row(vocabSize: Int, topicsMatrix: Matrix, docConcentration:
Vector,
+topicConcentration: Double, gammaShape: Double) =
MLUtils.convertMatrixColumnsToML(
+vectorConverted, "topicsMatrix").as[Data]
val oldModel = new OldLocalLDAModel(topicsMatrix, docConcentration,
topicConcentration,
gammaShape)
val model = new LocalLDAModel(metadata.uid, vocabSize, oldModel,
sparkSession)
- DefaultParamsReader.getAndSetParams(model, metadata)
+
+ metadata.sparkVersion match {
+case "1.6" =>
+ implicit val format = DefaultFormats
+ metadata.params match {
+ case JObject(pairs) =>
+ pairs.foreach { case (paramName, jsonValue) =>
+ val origParam =
+ if (paramName == "topicDistribution")
"topicDistributionCol" else paramName
+ val param = model.getParam(origParam)
+ val value = param.jsonDecode(compact(render(jsonValue)))
+ model.set(param, value)
+ }
+ case _ =>
+ throw new IllegalArgumentException(
+ s"Cannot recognize JSON metadata:
${metadata.metadataJson}.")
+ }
+case "2.x" =>
--- End diff --
I don't think you can match on "2.x" You'll need a more robust comparison
of versions. (Check out what sparkSession.version returns; it could be things
like "2.0.1" or "2.0.1-SNAPSHOT".)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in partiti...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13756 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user krishnakalyan3 commented on the issue: https://github.com/apache/spark/pull/13894 cc @holdenk @MLnick @jkbradley. Does the current state look good?. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in partiti...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13756 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62182/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14147: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, Developer...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14147 **[Test build #62184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62184/consoleFull)** for PR 14147 at commit [`5bd0d38`](https://github.com/apache/spark/commit/5bd0d384f27674eafcf908f7cc05c08c469b1aae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13894: [SPARK-15254][DOC] Improve ML pipeline Cross Validation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13894 **[Test build #62185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62185/consoleFull)** for PR 13894 at commit [`013a1db`](https://github.com/apache/spark/commit/013a1dbe06c2babf68b597b4b025a156a9723b03). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13756: [SPARK-16041][SQL] Disallow Duplicate Columns in partiti...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13756 **[Test build #62182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62182/consoleFull)** for PR 13756 at commit [`08b5374`](https://github.com/apache/spark/commit/08b5374e827f6680b4e4a00ed700ef689dce22ff). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14112: [SPARK-16240][ML] Model loading backward compatib...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/14112#discussion_r70504461
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala ---
@@ -566,26 +565,52 @@ object LocalLDAModel extends
MLReadable[LocalLDAModel] {
}
}
+ private case class Data(
+ vocabSize: Int,
+ topicsMatrix: Matrix,
+ docConcentration: Vector,
+ topicConcentration: Double,
+ gammaShape: Double)
+
private class LocalLDAModelReader extends MLReader[LocalLDAModel] {
private val className = classOf[LocalLDAModel].getName
override def load(path: String): LocalLDAModel = {
+ // Import implicits for Dataset Encoder
+ val sparkSession = super.sparkSession
+ import sparkSession.implicits._
+
val metadata = DefaultParamsReader.loadMetadata(path, sc, className)
val dataPath = new Path(path, "data").toString
val data = sparkSession.read.parquet(dataPath)
-.select("vocabSize", "topicsMatrix", "docConcentration",
"topicConcentration",
- "gammaShape")
-.head()
- val vocabSize = data.getAs[Int](0)
- val topicsMatrix = data.getAs[Matrix](1)
- val docConcentration = data.getAs[Vector](2)
- val topicConcentration = data.getAs[Double](3)
- val gammaShape = data.getAs[Double](4)
+ val vectorConverted = MLUtils.convertVectorColumnsToML(data,
"docConcentration")
+ val Row(vocabSize: Int, topicsMatrix: Matrix, docConcentration:
Vector,
--- End diff --
Does this work? You should need to take the first row
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70504183
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark pull request #14112: [SPARK-16240][ML] Model loading backward compatib...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/14112#discussion_r70504191
--- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala ---
@@ -566,26 +565,52 @@ object LocalLDAModel extends
MLReadable[LocalLDAModel] {
}
}
+ private case class Data(
+ vocabSize: Int,
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14112: [SPARK-16240][ML] Model loading backward compatib...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/14112#discussion_r70504176 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala --- @@ -121,7 +126,8 @@ private[clustering] trait LDAParams extends Params with HasFeaturesCol with HasM * - Value should be >= 0 * - default = (1.0 / k), following the implementation from * [[https://github.com/Blei-Lab/onlineldavb]]. - * @group param +* --- End diff -- ditto --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14112: [SPARK-16240][ML] Model loading backward compatib...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/14112#discussion_r70504163 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala --- @@ -80,7 +84,8 @@ private[clustering] trait LDAParams extends Params with HasFeaturesCol with HasM * - Values should be >= 0 * - default = uniformly (1.0 / k), following the implementation from * [[https://github.com/Blei-Lab/onlineldavb]]. - * @group param +* --- End diff -- fix indentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70504077
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70504020
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var newBaseURI = words.slice(0, ind + 2).join('/');
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors"
+} else {
+ind = words.indexOf("history");
+var appId = words[ind + 1];
+var attemptId = words[ind + 2];
+var newBaseURI = words.slice(0, ind).join('/');
+if(isNaN(attemptId) ) {
+return newBaseURI + "/api/v1/applications/" + appId +
"/allexecutors";
+} else {
+return newBaseURI + "/api/v1/applications/" + appId + "/" +
attemptId + "/allexecutors";
+}
+}
+}
+
+function formatLogsCells(execLogs, type) {
+if(type !== 'display') return Object.keys(execLogs);
+if(!execLogs) return;
+var result = '';
+$.each(execLogs, function(logName, logUrl) {
+result += '' + logName + ''
+});
+return result;
+}
+
+// Determine Color Opacity from 0.5-1
+// activeTasks range from 0 to
[GitHub] spark issue #14147: [SPARK-14812][ML][MLLIB][PYTHON] Experimental, Developer...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/14147 CC: @MLnick @yanboliang If you have a chance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70503658
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -54,78 +53,16 @@ private[ui] class ExecutorsPage(
// When GCTimePercent is edited change ToolTips.TASK_TIME to match
private val GCTimePercent = 0.1
- // a safe String to Int for sorting ids (converts non-numeric Strings to
-1)
- private def idStrToInt(str: String) : Int = Try(str.toInt).getOrElse(-1)
-
def render(request: HttpServletRequest): Seq[Node] = {
-val (activeExecutorInfo, deadExecutorInfo) = listener.synchronized {
- // The follow codes should be protected by `listener` to make sure
no executors will be
- // removed before we query their status. See SPARK-12784.
- val _activeExecutorInfo = {
-for (statusId <- 0 until listener.activeStorageStatusList.size)
- yield ExecutorsPage.getExecInfo(listener, statusId, isActive =
true)
- }
- val _deadExecutorInfo = {
-for (statusId <- 0 until listener.deadStorageStatusList.size)
- yield ExecutorsPage.getExecInfo(listener, statusId, isActive =
false)
- }
- (_activeExecutorInfo, _deadExecutorInfo)
-}
-
-val execInfo = activeExecutorInfo ++ deadExecutorInfo
-implicit val idOrder = Ordering[Int].on((s: String) =>
idStrToInt(s)).reverse
-val execInfoSorted = execInfo.sortBy(_.id)
-val logsExist = execInfo.filter(_.executorLogs.nonEmpty).nonEmpty
-
-val execTable = {
-
-
- Executor ID
- Address
- Status
- RDD Blocks
- Storage Memory
- Disk Used
- Cores
- Active Tasks
- Failed Tasks
- Complete Tasks
- Total Tasks
- Task
Time (GC Time)
- Input
- Shuffle Read
-
-
-
- Shuffle Write
-
-
- {if (logsExist) Logs else
Seq.empty}
- {if (threadDumpEnabled) Thread
Dump else Seq.empty}
-
-
- {execInfoSorted.map(execRow(_, logsExist))}
-
-
-}
-
val content =
-
-
- Summary
- {execSummary(activeExecutorInfo, deadExecutorInfo)}
-
-
-
-
- Executors
- {execTable}
-
+
+{
+ ++
+
+}
;
-UIUtils.headerSparkPage("Executors", content, parent)
+UIUtils.headerSparkPage("Executors", content, parent, useDataTables =
true)
}
/** Render an HTML row representing an executor */
--- End diff --
it seems like there is a bunch of other things in here that were used by
render that can be removed. execRow, execSummaryRow, taskData, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70502730
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
+var appId = words[ind + 1];
+var baseURI = words.slice(0, ind + 1).join('/') + '/' + appId +
'/static/executorspage-template.html';
+return baseURI;
+} else {
+ind = words.indexOf("history");
+var baseURI = words.slice(0, ind).join('/') +
'/static/executorspage-template.html';
+return baseURI;
+}
+}
+
+function createRESTEndPoint() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
--- End diff --
nit space after if, note this is throughout file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70502282
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
+if(type !== 'display') return bytes;
+if(bytes == 0) return '0 B';
+var k = 1000;
+var dm = 3;
+var sizes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB', 'EB', 'ZB', 'YB'];
+var i = Math.floor(Math.log(bytes) / Math.log(k));
+return parseFloat((bytes / Math.pow(k, i)).toFixed(dm)) + ' ' +
sizes[i];
+}
+
+
+jQuery.extend( jQuery.fn.dataTableExt.oSort, {
+"title-numeric-pre": function ( a ) {
+var x = a.match(/title="*(-?[0-9\.]+)/)[1];
+return parseFloat( x );
+},
+
+"title-numeric-asc": function ( a, b ) {
+return ((a < b) ? -1 : ((a > b) ? 1 : 0));
+},
+
+"title-numeric-desc": function ( a, b ) {
+return ((a < b) ? 1 : ((a > b) ? -1 : 0));
+}
+} );
+
+$(document).ajaxStop($.unblockUI);
+$(document).ajaxStart(function(){
+$.blockUI({ message: 'Loading Executors Page...'});
+});
+
+function createTemplateURI() {
+var parser = document.createElement('a');
+var words = parser.baseURI.split('/');
+var ind = words.indexOf("proxy");
+if(ind > 0) {
--- End diff --
nit, space between if and (
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14158: [SPARK-13547] [SQL] [WEBUI] Add SQL query in web ...
Github user ajbozarth commented on a diff in the pull request:
https://github.com/apache/spark/pull/14158#discussion_r70501467
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala
---
@@ -103,6 +104,9 @@ private[ui] abstract class ExecutionTable(
{descriptionCell(executionUIData)}
+
+{executionUIData.sqlText.getOrElse("-")}
--- End diff --
Is there a reason to insert a "-" rather than leave this blank in the case
it's not available?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70500868
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
+ if(type !== 'display') return milliseconds;
+ if (milliseconds < 100) {
+return milliseconds + " ms";
+ }
+ var seconds = milliseconds * 1.0 / 1000;
+ if (seconds < 1) {
+return seconds.toFixed(1) + " s";
+ }
+ if (seconds < 60) {
+return seconds.toFixed(0) + " s";
+ }
+ var minutes = seconds / 60;
+ if (minutes < 10) {
+return minutes.toFixed(1) + " min";
+ } else if (minutes < 60) {
+return minutes.toFixed(0) + " min";
+ }
+ var hours = minutes / 60;
+ return hours.toFixed(1) + " h";
+}
+
+function formatStatus(status, type) {
+if(type !== 'display') return status;
+if(status) {
+return "Active"
+} else {
+return "Dead"
+}
+}
+
+function formatBytes(bytes,type) {
--- End diff --
similar here, type isn't really used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/14139 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/13670#discussion_r70500759
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// this function works exactly the same as UIUtils.formatDuration
+function formatDuration(milliseconds, type) {
--- End diff --
this is the pretty much the same with historypage.js could we easily make a
utils.js and include in both.
I also don't really see the type being used here, it seems everywhere its
called it already has check for type === display
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14080: [SPARK-16405] Add metrics and source for external shuffl...
Github user lovexi commented on the issue: https://github.com/apache/spark/pull/14080 @rxin Thank you for mentioning that. I even didn't notice that. Haha. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13670: [SPARK-15951] Change Executors Page to use datata...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/13670#discussion_r70499206 --- Diff: core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html --- @@ -0,0 +1,102 @@ + + +
[GitHub] spark issue #14158: [SPARK-13547] [SQL] [WEBUI] Add SQL query in web UI's SQ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14158 **[Test build #62180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62180/consoleFull)** for PR 14158 at commit [`626f3f7`](https://github.com/apache/spark/commit/626f3f74fb3fa09dd781c4e11d7412776fe06970). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14158: [SPARK-13547] [SQL] [WEBUI] Add SQL query in web UI's SQ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14158 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62180/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
