[spark] branch master updated: [SPARK-34123][WEB UI] optimize spark history summary page loading
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ebd8bc9 [SPARK-34123][WEB UI] optimize spark history summary page
loading
ebd8bc9 is described below
commit ebd8bc934de9d6aec53beb4ab60c998052038fad
Author: mohan3d
AuthorDate: Sun Jan 17 14:37:28 2021 -0600
[SPARK-34123][WEB UI] optimize spark history summary page loading
### What changes were proposed in this pull request?
Display history server entries using datatables instead of Mustache +
Datatables which proved to be faster and non-blocking for the webpage while
searching (using search bar in the page)
### Why are the changes needed?
Small changes in the attempts (entries) and removed part of HTML (Mustache
template).
### Does this PR introduce _any_ user-facing change?
Not very sure, but it's not supposed to change the way the page looks
rather it changes how entries are displayed.
### How was this patch tested?
Running test, since it's not adding new functionality.
Closes #31191 from mohan3d/feat/history-server-ui-optimization.
Lead-authored-by: mohan3d
Co-authored-by: Author: mohan3d
Signed-off-by: Sean Owen
---
.../spark/ui/static/historypage-template.html | 20
.../org/apache/spark/ui/static/historypage.js | 54 --
2 files changed, 41 insertions(+), 33 deletions(-)
diff --git
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
index 7e9927d..5427125 100644
---
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
+++
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
@@ -75,26 +75,6 @@
- {{#applications}}
-
- {{#attempts}}
- {{version}}
- {{id}}
- {{name}}
- {{#hasMultipleAttempts}}
- {{attemptId}}
- {{/hasMultipleAttempts}}
- {{startTime}}
- {{#showCompletedColumns}}
- {{endTime}}
- {{duration}}
- {{/showCompletedColumns}}
- {{sparkUser}}
- {{lastUpdated}}
- Download
- {{/attempts}}
-
- {{/applications}}
diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
index 3a4c815..aa542a7 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
@@ -140,9 +140,13 @@ $(document).ready(function() {
(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "logs";
attempt["durationMillisec"] = attempt["duration"];
attempt["duration"] = formatDuration(attempt["duration"]);
- var hasAttemptId = attempt.hasOwnProperty("attemptId");
- var app_clone = {"id" : id, "name" : name, "version": version,
"hasAttemptId" : hasAttemptId, "attempts" : [attempt]};
- array.push(app_clone);
+ attempt["id"] = id;
+ attempt["name"] = name;
+ attempt["version"] = version;
+ attempt["attemptUrl"] = uiRoot + "/history/" + id + "/" +
+(attempt.hasOwnProperty("attemptId") ? attempt["attemptId"] + "/"
: "") + "jobs/";
+
+ array.push(attempt);
}
}
if(array.length < 20) {
@@ -165,17 +169,41 @@ $(document).ready(function() {
var completedColumnName = 'completed';
var durationColumnName = 'duration';
var conf = {
+ "data": array,
"columns": [
-{name: 'version'},
-{name: 'appId', type: "appid-numeric"},
-{name: 'appName'},
-{name: attemptIdColumnName},
-{name: startedColumnName},
-{name: completedColumnName},
-{name: durationColumnName, type: "title-numeric"},
-{name: 'user'},
-{name: 'lastUpdated'},
-{name: 'eventLog'},
+{name: 'version', data: 'version' },
+{
+ name: 'appId',
+ type: "appid-numeric",
+ data: 'id',
+ render: (id, type, row) => `${id}`
+},
+{name: 'appName', data: 'name' },
+{
+ name: attemptIdColumnName,
+ data: 'attemptId',
+ render: (attemptId, type, row) => (attemptId ? `${attemptId}` : '')
+},
+{name: startedColumnName, data: 'startTime' },
+{name: completedColumnName, data: 'endTime' },
+{name: durationColumnName, type: "title-numeric", data: 'duration'
},
+{name: 'user', data: 'sparkUser' },
+{name: 'lastUpdated', data: 'lastUpdated' },
[spark] branch branch-2.4 updated (7ae6c8d -> e0e1e21)
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 7ae6c8d [SPARK-34118][CORE][SQL][2.4] Replaces filter and check for emptiness with exists or forall add e0e1e21 [SPARK-34125][CORE][2.4] Make EventLoggingListener.codecMap thread-safe No new revisions were added by this update. Summary of changes: .../org/apache/spark/scheduler/EventLoggingListener.scala | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-33730][PYTHON] Standardize warning types
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 098f226 [SPARK-33730][PYTHON] Standardize warning types
098f226 is described below
commit 098f2268e4ad43dd9453ada91161ea428dd57d16
Author: zero323
AuthorDate: Mon Jan 18 09:32:55 2021 +0900
[SPARK-33730][PYTHON] Standardize warning types
### What changes were proposed in this pull request?
This PR:
- Adds as small hierarchy of warnings to be used in PySpark applications.
These extend built-in classes and top level `PySparkWarning`.
- Replaces `DeprecationWarnings` (intended for developers) with PySpark
specific subclasses of `FutureWarning` (intended for end users).
### Why are the changes needed?
- To be more precise and add users additional control (in addition to
standard module level filters) over PySpark warnings handling.
- Correct semantics (at the moment we use `DeprecationWarning` in
user-facing API, but it is intended "for warnings about deprecated features
when those warnings are intended for other Python developers").
### Does this PR introduce _any_ user-facing change?
Yes. Code can raise different type of warning than before.
### How was this patch tested?
Existing tests.
Closes #30985 from zero323/SPARK-33730.
Authored-by: zero323
Signed-off-by: HyukjinKwon
---
python/pyspark/ml/clustering.py| 2 +-
python/pyspark/mllib/classification.py | 2 +-
python/pyspark/mllib/regression.py | 7 ---
python/pyspark/rdd.py | 10 ++
python/pyspark/sql/catalog.py | 6 --
python/pyspark/sql/column.py | 6 --
python/pyspark/sql/context.py | 15 ++-
python/pyspark/sql/dataframe.py| 4 +++-
python/pyspark/sql/functions.py| 6 +++---
python/pyspark/worker.py | 10 --
10 files changed, 44 insertions(+), 24 deletions(-)
diff --git a/python/pyspark/ml/clustering.py b/python/pyspark/ml/clustering.py
index 54c1a43..60726cb 100644
--- a/python/pyspark/ml/clustering.py
+++ b/python/pyspark/ml/clustering.py
@@ -821,7 +821,7 @@ class BisectingKMeansModel(JavaModel,
_BisectingKMeansParams, JavaMLWritable, Ja
"""
warnings.warn("Deprecated in 3.0.0. It will be removed in future
versions. Use "
"ClusteringEvaluator instead. You can also get the cost
on the training "
- "dataset in the summary.", DeprecationWarning)
+ "dataset in the summary.", FutureWarning)
return self._call_java("computeCost", dataset)
@property
diff --git a/python/pyspark/mllib/classification.py
b/python/pyspark/mllib/classification.py
index bd43e91..5705401 100644
--- a/python/pyspark/mllib/classification.py
+++ b/python/pyspark/mllib/classification.py
@@ -324,7 +324,7 @@ class LogisticRegressionWithSGD(object):
"""
warnings.warn(
"Deprecated in 2.0.0. Use ml.classification.LogisticRegression or "
-"LogisticRegressionWithLBFGS.", DeprecationWarning)
+"LogisticRegressionWithLBFGS.", FutureWarning)
def train(rdd, i):
return callMLlibFunc("trainLogisticRegressionModelWithSGD", rdd,
int(iterations),
diff --git a/python/pyspark/mllib/regression.py
b/python/pyspark/mllib/regression.py
index c224e38..3908e4a 100644
--- a/python/pyspark/mllib/regression.py
+++ b/python/pyspark/mllib/regression.py
@@ -299,7 +299,7 @@ class LinearRegressionWithSGD(object):
(default: 0.001)
"""
warnings.warn(
-"Deprecated in 2.0.0. Use ml.regression.LinearRegression.",
DeprecationWarning)
+"Deprecated in 2.0.0. Use ml.regression.LinearRegression.",
FutureWarning)
def train(rdd, i):
return callMLlibFunc("trainLinearRegressionModelWithSGD", rdd,
int(iterations),
@@ -453,7 +453,8 @@ class LassoWithSGD(object):
warnings.warn(
"Deprecated in 2.0.0. Use ml.regression.LinearRegression with
elasticNetParam = 1.0. "
"Note the default regParam is 0.01 for LassoWithSGD, but is 0.0
for LinearRegression.",
-DeprecationWarning)
+FutureWarning
+)
def train(rdd, i):
return callMLlibFunc("trainLassoModelWithSGD", rdd,
int(iterations), float(step),
@@ -607,7 +608,7 @@ class RidgeRegressionWithSGD(object):
warnings.warn(
"Deprecated in 2.0.0. Use ml.regression.LinearRegression with
elasticNetParam = 0.0. "
"Note the default regParam is 0.01 for RidgeRegressionWithSGD, but
is 0.0 for "
-"LinearRegression.", DeprecationWarning)
+"LinearRegression.", FutureWarning)
def tra
[spark] branch master updated (098f226 -> 415506c)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 098f226 [SPARK-33730][PYTHON] Standardize warning types add 415506c [SPARK-34142][CORE] Support Fallback Storage Cleanup during stopping SparkContext No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/SparkContext.scala | 1 + .../org/apache/spark/internal/config/package.scala | 7 ++ .../org/apache/spark/storage/FallbackStorage.scala | 25 -- .../spark/storage/FallbackStorageSuite.scala | 21 +- 4 files changed, 51 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][DOCS] Fix broken python doc links
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8847b7f [MINOR][DOCS] Fix broken python doc links
8847b7f is described below
commit 8847b7fa6d9713646257c6640017aab7e0c22e5a
Author: Huaxin Gao
AuthorDate: Mon Jan 18 10:06:45 2021 +0900
[MINOR][DOCS] Fix broken python doc links
### What changes were proposed in this pull request?
Fix broken python links
### Why are the changes needed?
links broken.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually checked
Closes #31220 from huaxingao/docs.
Authored-by: Huaxin Gao
Signed-off-by: HyukjinKwon
---
docs/ml-classification-regression.md | 40 -
docs/ml-clustering.md | 10 ++---
docs/ml-collaborative-filtering.md | 2 +-
docs/ml-features.md| 80 +-
docs/ml-frequent-pattern-mining.md | 4 +-
docs/ml-pipeline.md| 8 ++--
docs/ml-statistics.md | 6 +--
docs/ml-tuning.md | 4 +-
docs/mllib-clustering.md | 18
docs/mllib-collaborative-filtering.md | 2 +-
docs/mllib-data-types.md | 44 +--
docs/mllib-decision-tree.md| 4 +-
docs/mllib-dimensionality-reduction.md | 4 +-
docs/mllib-ensembles.md| 8 ++--
docs/mllib-evaluation-metrics.md | 8 ++--
docs/mllib-feature-extraction.md | 14 +++---
docs/mllib-frequent-pattern-mining.md | 6 +--
docs/mllib-isotonic-regression.md | 2 +-
docs/mllib-linear-methods.md | 4 +-
docs/mllib-naive-bayes.md | 8 ++--
docs/mllib-statistics.md | 28 ++--
21 files changed, 152 insertions(+), 152 deletions(-)
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index 247989d..bad74cb 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -85,7 +85,7 @@ More details on parameters can be found in the [Java API
documentation](api/java
-More details on parameters can be found in the [Python API
documentation](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegression).
+More details on parameters can be found in the [Python API
documentation](api/python/reference/api/pyspark.ml.classification.LogisticRegression.html).
{% include_example python/ml/logistic_regression_with_elastic_net.py %}
@@ -135,11 +135,11 @@ Continuing the earlier example:
-[`LogisticRegressionTrainingSummary`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionSummary)
+[`LogisticRegressionTrainingSummary`](api/python/reference/api/pyspark.ml.classification.LogisticRegressionSummary.html)
provides a summary for a
-[`LogisticRegressionModel`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionModel).
+[`LogisticRegressionModel`](api/python/reference/api/pyspark.ml.classification.LogisticRegressionModel.html).
In the case of binary classification, certain additional metrics are
-available, e.g. ROC curve. See
[`BinaryLogisticRegressionTrainingSummary`](api/python/pyspark.ml.html#pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary).
+available, e.g. ROC curve. See
[`BinaryLogisticRegressionTrainingSummary`](api/python/reference/api/pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary.html).
Continuing the earlier example:
@@ -232,7 +232,7 @@ More details on parameters can be found in the [Java API
documentation](api/java
-More details on parameters can be found in the [Python API
documentation](api/python/pyspark.ml.html#pyspark.ml.classification.DecisionTreeClassifier).
+More details on parameters can be found in the [Python API
documentation](api/python/reference/api/pyspark.ml.classification.DecisionTreeClassifier.html).
{% include_example python/ml/decision_tree_classification_example.py %}
@@ -275,7 +275,7 @@ Refer to the [Java API
docs](api/java/org/apache/spark/ml/classification/RandomF
-Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classification.RandomForestClassifier)
for more details.
+Refer to the [Python API
docs](api/python/reference/api/pyspark.ml.classification.RandomForestClassifier.html)
for more details.
{% include_example python/ml/random_forest_classifier_example.py %}
@@ -316,7 +316,7 @@ Refer to the [Java API
docs](api/java/org/apache/spark/ml/classif
[spark] branch branch-3.1 updated: [MINOR][DOCS] Fix broken python doc links
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new a88cbbe [MINOR][DOCS] Fix broken python doc links
a88cbbe is described below
commit a88cbbe9d84abfb5139d5deab7077d2f6a76dc51
Author: Huaxin Gao
AuthorDate: Mon Jan 18 10:06:45 2021 +0900
[MINOR][DOCS] Fix broken python doc links
### What changes were proposed in this pull request?
Fix broken python links
### Why are the changes needed?
links broken.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually checked
Closes #31220 from huaxingao/docs.
Authored-by: Huaxin Gao
Signed-off-by: HyukjinKwon
(cherry picked from commit 8847b7fa6d9713646257c6640017aab7e0c22e5a)
Signed-off-by: HyukjinKwon
---
docs/ml-classification-regression.md | 40 -
docs/ml-clustering.md | 10 ++---
docs/ml-collaborative-filtering.md | 2 +-
docs/ml-features.md| 80 +-
docs/ml-frequent-pattern-mining.md | 4 +-
docs/ml-pipeline.md| 8 ++--
docs/ml-statistics.md | 6 +--
docs/ml-tuning.md | 4 +-
docs/mllib-clustering.md | 18
docs/mllib-collaborative-filtering.md | 2 +-
docs/mllib-data-types.md | 44 +--
docs/mllib-decision-tree.md| 4 +-
docs/mllib-dimensionality-reduction.md | 4 +-
docs/mllib-ensembles.md| 8 ++--
docs/mllib-evaluation-metrics.md | 8 ++--
docs/mllib-feature-extraction.md | 14 +++---
docs/mllib-frequent-pattern-mining.md | 6 +--
docs/mllib-isotonic-regression.md | 2 +-
docs/mllib-linear-methods.md | 4 +-
docs/mllib-naive-bayes.md | 8 ++--
docs/mllib-statistics.md | 28 ++--
21 files changed, 152 insertions(+), 152 deletions(-)
diff --git a/docs/ml-classification-regression.md
b/docs/ml-classification-regression.md
index 247989d..bad74cb 100644
--- a/docs/ml-classification-regression.md
+++ b/docs/ml-classification-regression.md
@@ -85,7 +85,7 @@ More details on parameters can be found in the [Java API
documentation](api/java
-More details on parameters can be found in the [Python API
documentation](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegression).
+More details on parameters can be found in the [Python API
documentation](api/python/reference/api/pyspark.ml.classification.LogisticRegression.html).
{% include_example python/ml/logistic_regression_with_elastic_net.py %}
@@ -135,11 +135,11 @@ Continuing the earlier example:
-[`LogisticRegressionTrainingSummary`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionSummary)
+[`LogisticRegressionTrainingSummary`](api/python/reference/api/pyspark.ml.classification.LogisticRegressionSummary.html)
provides a summary for a
-[`LogisticRegressionModel`](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegressionModel).
+[`LogisticRegressionModel`](api/python/reference/api/pyspark.ml.classification.LogisticRegressionModel.html).
In the case of binary classification, certain additional metrics are
-available, e.g. ROC curve. See
[`BinaryLogisticRegressionTrainingSummary`](api/python/pyspark.ml.html#pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary).
+available, e.g. ROC curve. See
[`BinaryLogisticRegressionTrainingSummary`](api/python/reference/api/pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary.html).
Continuing the earlier example:
@@ -232,7 +232,7 @@ More details on parameters can be found in the [Java API
documentation](api/java
-More details on parameters can be found in the [Python API
documentation](api/python/pyspark.ml.html#pyspark.ml.classification.DecisionTreeClassifier).
+More details on parameters can be found in the [Python API
documentation](api/python/reference/api/pyspark.ml.classification.DecisionTreeClassifier.html).
{% include_example python/ml/decision_tree_classification_example.py %}
@@ -275,7 +275,7 @@ Refer to the [Java API
docs](api/java/org/apache/spark/ml/classification/RandomF
-Refer to the [Python API
docs](api/python/pyspark.ml.html#pyspark.ml.classification.RandomForestClassifier)
for more details.
+Refer to the [Python API
docs](api/python/reference/api/pyspark.ml.classification.RandomForestClassifier.html)
for more details.
{% include_example python/ml/random_forest_
[spark] branch master updated: [MINOR][DOCS] Fix typos in sql-ref-datatypes.md
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 536a725 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md 536a725 is described below commit 536a7258a829299a13035eb3550e6ce6f7632677 Author: Mitsuru Kariya AuthorDate: Mon Jan 18 13:18:03 2021 +0900 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md ### What changes were proposed in this pull request? Fixing typos in the docs sql-ref-datatypes.md. ### Why are the changes needed? To display '' correctly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually run jekyll. before this fix  after this fix  Closes #31221 from kariya-mitsuru/fix-typo. Authored-by: Mitsuru Kariya Signed-off-by: HyukjinKwon --- docs/sql-ref-datatypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md index 0087867..fe1090e 100644 --- a/docs/sql-ref-datatypes.md +++ b/docs/sql-ref-datatypes.md @@ -193,7 +193,7 @@ The following table shows the type names as well as aliases used in Spark SQL pa |**BinaryType**|BINARY| |**DecimalType**|DECIMAL, DEC, NUMERIC| |**CalendarIntervalType**|INTERVAL| -|**ArrayType**|ARRAY| +|**ArrayType**|ARRAY\| |**StructType**|STRUCT| |**MapType**|MAP| - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [MINOR][DOCS] Fix typos in sql-ref-datatypes.md
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new 67ee28c [MINOR][DOCS] Fix typos in sql-ref-datatypes.md 67ee28c is described below commit 67ee28c1b50c95c09472de21a85ae753b64d1e03 Author: Mitsuru Kariya AuthorDate: Mon Jan 18 13:18:03 2021 +0900 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md ### What changes were proposed in this pull request? Fixing typos in the docs sql-ref-datatypes.md. ### Why are the changes needed? To display '' correctly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually run jekyll. before this fix  after this fix  Closes #31221 from kariya-mitsuru/fix-typo. Authored-by: Mitsuru Kariya Signed-off-by: HyukjinKwon (cherry picked from commit 536a7258a829299a13035eb3550e6ce6f7632677) Signed-off-by: HyukjinKwon --- docs/sql-ref-datatypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md index 0087867..fe1090e 100644 --- a/docs/sql-ref-datatypes.md +++ b/docs/sql-ref-datatypes.md @@ -193,7 +193,7 @@ The following table shows the type names as well as aliases used in Spark SQL pa |**BinaryType**|BINARY| |**DecimalType**|DECIMAL, DEC, NUMERIC| |**CalendarIntervalType**|INTERVAL| -|**ArrayType**|ARRAY| +|**ArrayType**|ARRAY\| |**StructType**|STRUCT| |**MapType**|MAP| - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [MINOR][DOCS] Fix typos in sql-ref-datatypes.md
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new d8ce224 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md d8ce224 is described below commit d8ce2249d0fb62dce161f5caa4a9143295949f93 Author: Mitsuru Kariya AuthorDate: Mon Jan 18 13:18:03 2021 +0900 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md ### What changes were proposed in this pull request? Fixing typos in the docs sql-ref-datatypes.md. ### Why are the changes needed? To display '' correctly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually run jekyll. before this fix  after this fix  Closes #31221 from kariya-mitsuru/fix-typo. Authored-by: Mitsuru Kariya Signed-off-by: HyukjinKwon (cherry picked from commit 536a7258a829299a13035eb3550e6ce6f7632677) Signed-off-by: HyukjinKwon --- docs/sql-ref-datatypes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md index f27f1a0..144594b 100644 --- a/docs/sql-ref-datatypes.md +++ b/docs/sql-ref-datatypes.md @@ -191,7 +191,7 @@ The following table shows the type names as well as aliases used in Spark SQL pa |**BinaryType**|BINARY| |**DecimalType**|DECIMAL, DEC, NUMERIC| |**CalendarIntervalType**|INTERVAL| -|**ArrayType**|ARRAY| +|**ArrayType**|ARRAY\| |**StructType**|STRUCT| |**MapType**|MAP| - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (536a725 -> ac322a1)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 536a725 [MINOR][DOCS] Fix typos in sql-ref-datatypes.md add ac322a1 [SPARK-34080][ML][PYTHON][FOLLOWUP] Add UnivariateFeatureSelector - make methods private No new revisions were added by this update. Summary of changes: .../ml/feature/UnivariateFeatureSelector.scala | 74 -- 1 file changed, 27 insertions(+), 47 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated: [SPARK-34080][ML][PYTHON][FOLLOWUP] Add UnivariateFeatureSelector - make methods private
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 56f93e5 [SPARK-34080][ML][PYTHON][FOLLOWUP] Add
UnivariateFeatureSelector - make methods private
56f93e5 is described below
commit 56f93e56ab731be27a05a299fcbe0ef529f280ba
Author: Ruifeng Zheng
AuthorDate: Mon Jan 18 13:19:59 2021 +0900
[SPARK-34080][ML][PYTHON][FOLLOWUP] Add UnivariateFeatureSelector - make
methods private
### What changes were proposed in this pull request?
1, make `getTopIndices`/`selectIndicesFromPValues` private;
2, avoid setting `selectionThreshold` in `fit`
3, move param checking to `transformSchema`
### Why are the changes needed?
`getTopIndices`/`selectIndicesFromPValues` should not be exposed to end
users;
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existing testsuites
Closes #31222 from zhengruifeng/selector_clean_up.
Authored-by: Ruifeng Zheng
Signed-off-by: HyukjinKwon
(cherry picked from commit ac322a1ac3be79b5e514f0119275f53b3a40c923)
Signed-off-by: HyukjinKwon
---
.../ml/feature/UnivariateFeatureSelector.scala | 74 --
1 file changed, 27 insertions(+), 47 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
index 6d5f09e..bfe1d5f 100644
---
a/mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
+++
b/mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala
@@ -76,8 +76,7 @@ private[feature] trait UnivariateFeatureSelectorParams
extends Params
@Since("3.1.1")
final val selectionMode = new Param[String](this, "selectionMode",
"The selection mode. Supported options: numTopFeatures, percentile, fpr,
fdr, fwe",
-ParamValidators.inArray(Array("numTopFeatures", "percentile", "fpr", "fdr",
- "fwe")))
+ParamValidators.inArray(Array("numTopFeatures", "percentile", "fpr",
"fdr", "fwe")))
/** @group getParam */
@Since("3.1.1")
@@ -161,48 +160,17 @@ final class UnivariateFeatureSelector
@Since("3.1.1")(@Since("3.1.1") override v
transformSchema(dataset.schema, logging = true)
val numFeatures = MetadataUtils.getNumFeatures(dataset, $(featuresCol))
-$(selectionMode) match {
- case ("numTopFeatures") =>
-if (!isSet(selectionThreshold)) {
- set(selectionThreshold, 50.0)
-} else {
- require($(selectionThreshold) > 0 && $(selectionThreshold).toInt ==
$(selectionThreshold),
-"selectionThreshold needs to be a positive Integer for selection
mode numTopFeatures")
-}
- case ("percentile") =>
-if (!isSet(selectionThreshold)) {
- set(selectionThreshold, 0.1)
-} else {
- require($(selectionThreshold) >= 0 && $(selectionThreshold) <= 1,
-"selectionThreshold needs to be in the range of 0 to 1 for
selection mode percentile")
-}
- case ("fpr") =>
-if (!isSet(selectionThreshold)) {
- set(selectionThreshold, 0.05)
-} else {
- require($(selectionThreshold) >= 0 && $(selectionThreshold) <= 1,
-"selectionThreshold needs to be in the range of 0 to 1 for
selection mode fpr")
-}
- case ("fdr") =>
-if (!isSet(selectionThreshold)) {
- set(selectionThreshold, 0.05)
-} else {
- require($(selectionThreshold) >= 0 && $(selectionThreshold) <= 1,
-"selectionThreshold needs to be in the range of 0 to 1 for
selection mode fdr")
-}
- case ("fwe") =>
-if (!isSet(selectionThreshold)) {
- set(selectionThreshold, 0.05)
-} else {
- require($(selectionThreshold) >= 0 && $(selectionThreshold) <= 1,
-"selectionThreshold needs to be in the range of 0 to 1 for
selection mode fwe")
-}
- case _ =>
-throw new IllegalArgumentException(s"Unsupported selection mode:" +
- s" selectionMode=${$(selectionMode)}")
+var threshold = Double.NaN
+if (isSet(selectionThreshold)) {
+ threshold = $(selectionThreshold)
+} else {
+ $(selectionMode) match {
+case "numTopFeatures" => threshold = 50
+case "percentile" => threshold = 0.1
+case "fpr" | "fdr" | "fwe" => threshold = 0.05
+ }
}
-require(isSet(featureType) && isSet(labelType), "featureType and labelType
need to be set")
val resultDF = ($(featureType), $(labelType)) match {
case ("categorical", "categorical") =>
ChiSquareTest.test(dataset.toDF, getFeaturesCol, getLabelCol, true)
@@ -215,14 +183,12 @@ final class UnivariateFeatureS
[spark] branch master updated (ac322a1 -> b5bdbf2)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac322a1 [SPARK-34080][ML][PYTHON][FOLLOWUP] Add UnivariateFeatureSelector - make methods private add b5bdbf2 [SPARK-30682][R][SQL][FOLLOW-UP] Keep the name similar with Scala side in higher order functions No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 8 R/pkg/R/generics.R | 3 ++- 2 files changed, 6 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated (56f93e5 -> 8758773)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git. from 56f93e5 [SPARK-34080][ML][PYTHON][FOLLOWUP] Add UnivariateFeatureSelector - make methods private add 8758773 [SPARK-30682][R][SQL][FOLLOW-UP] Keep the name similar with Scala side in higher order functions No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 8 R/pkg/R/generics.R | 3 ++- 2 files changed, 6 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated (8758773 -> e1ad275)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git. from 8758773 [SPARK-30682][R][SQL][FOLLOW-UP] Keep the name similar with Scala side in higher order functions add e1ad275 [SPARK-33819][CORE][FOLLOWUP][3.1] Restore the constructor of SingleFileEventLogFileReader to remove Mima exclusion No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/deploy/history/EventLogFileReaders.scala | 4 +++- project/MimaExcludes.scala | 5 + 2 files changed, 4 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33819][CORE][FOLLOWUP][3.0] Restore the constructor of SingleFileEventLogFileReader to remove Mima exclusion
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 70c0bc9 [SPARK-33819][CORE][FOLLOWUP][3.0] Restore the constructor of
SingleFileEventLogFileReader to remove Mima exclusion
70c0bc9 is described below
commit 70c0bc9a3358fc36b967cc28ae9232fe7437ab6a
Author: Dongjoon Hyun
AuthorDate: Mon Jan 18 14:31:17 2021 +0900
[SPARK-33819][CORE][FOLLOWUP][3.0] Restore the constructor of
SingleFileEventLogFileReader to remove Mima exclusion
### What changes were proposed in this pull request?
This PR proposes to remove Mima exclusion via restoring the old constructor
of SingleFileEventLogFileReader. This partially adopts the remaining parts of
#30814 which was excluded while porting back.
### Why are the changes needed?
To remove unnecessary Mima exclusion.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass CIs.
Closes #31225 from HeartSaVioR/SPARK-33819-followup-branch-3.0.
Authored-by: Dongjoon Hyun
Signed-off-by: HyukjinKwon
---
.../scala/org/apache/spark/deploy/history/EventLogFileReaders.scala | 4 +++-
project/MimaExcludes.scala | 5 +
2 files changed, 4 insertions(+), 5 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
index 6fe3a7b..b4771c8 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/EventLogFileReaders.scala
@@ -167,9 +167,11 @@ object EventLogFileReader {
private[history] class SingleFileEventLogFileReader(
fs: FileSystem,
path: Path,
-maybeStatus: Option[FileStatus] = None) extends EventLogFileReader(fs,
path) {
+maybeStatus: Option[FileStatus]) extends EventLogFileReader(fs, path) {
private lazy val status =
maybeStatus.getOrElse(fileSystem.getFileStatus(rootPath))
+ def this(fs: FileSystem, path: Path) = this(fs, path, None)
+
override def lastIndex: Option[Long] = None
override def fileSizeForLastIndex: Long = status.getLen
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index b4e36dc..b799624 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -461,10 +461,7 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.streaming.StreamingQueryListener#QueryStartedEvent.this"),
// [SPARK-30667][CORE] Add allGather method to BarrierTaskContext
-
ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.RequestToSync"),
-
-// [SPARK-33790][CORE] Reduce the rpc call of getFileStatus in
SingleFileEventLogFileReader
-
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.deploy.history.SingleFileEventLogFileReader.this")
+
ProblemFilters.exclude[IncompatibleTemplateDefProblem]("org.apache.spark.RequestToSync")
)
// Exclude rules for 2.4.x
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b5bdbf2 -> c87b008)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b5bdbf2 [SPARK-30682][R][SQL][FOLLOW-UP] Keep the name similar with Scala side in higher order functions add c87b008 [SPARK-33696][BUILD][SQL] Upgrade built-in Hive to 2.3.8 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-2.3| 26 +++--- dev/deps/spark-deps-hadoop-3.2-hive-2.3| 26 +++--- docs/building-spark.md | 4 ++-- docs/sql-data-sources-hive-tables.md | 8 +++ docs/sql-migration-guide.md| 2 +- pom.xml| 20 +++-- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 11 - .../thriftserver/HiveThriftServer2Suites.scala | 4 ++-- .../org/apache/spark/sql/hive/HiveUtils.scala | 2 +- .../sql/hive/client/IsolatedClientLoader.scala | 4 ++-- .../org/apache/spark/sql/hive/client/package.scala | 10 - .../hive/HiveExternalCatalogVersionsSuite.scala| 2 +- .../spark/sql/hive/execution/HiveQuerySuite.scala | 7 +++--- 13 files changed, 72 insertions(+), 54 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c87b008 -> 78893b8)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c87b008 [SPARK-33696][BUILD][SQL] Upgrade built-in Hive to 2.3.8 add 78893b8 [SPARK-34139][SQL] UnresolvedRelation should retain SQL text position for DDL commands No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/CheckAnalysis.scala | 6 +-- .../spark/sql/catalyst/parser/AstBuilder.scala | 43 +- .../analysis/AnalysisExceptionPositionSuite.scala | 17 + .../sql-tests/results/postgreSQL/with.sql.out | 2 +- 4 files changed, 46 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
