[spark] branch master updated (780555b -> 373c2c3)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 780555b [MINOR][CORE] Make EventLogger codec be consistent between EventLogFileWriter and SparkContext add 373c2c3 [SPARK-29862][SQL] CREATE (OR REPLACE) ... VIEW should look up catalog/table like v2 commands No new revisions were added by this update. Summary of changes: .../apache/spark/sql/catalyst/parser/SqlBase.g4| 2 +- .../apache/spark/sql/catalyst/analysis/view.scala | 35 ++- .../spark/sql/catalyst/parser/AstBuilder.scala | 51 +++ .../sql/catalyst/plans/logical/statements.scala| 17 +++- .../spark/sql/catalyst/parser/DDLParserSuite.scala | 101 - .../catalyst/analysis/ResolveSessionCatalog.scala | 16 .../spark/sql/execution/SparkSqlParser.scala | 52 --- .../apache/spark/sql/execution/command/views.scala | 37 +--- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 8 ++ .../sql/execution/command/DDLParserSuite.scala | 51 --- 10 files changed, 227 insertions(+), 143 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8b0121b -> 780555b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8b0121b [MINOR][DOC] Fix the CalendarIntervalType description add 780555b [MINOR][CORE] Make EventLogger codec be consistent between EventLogFileWriter and SparkContext No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (53e19f3 -> 8b0121b)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 53e19f3 [SPARK-30032][BUILD] Upgrade to ORC 1.5.8 add 8b0121b [MINOR][DOC] Fix the CalendarIntervalType description No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/types/CalendarIntervalType.scala | 11 +++ 1 file changed, 7 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2a28c73 -> 53e19f3)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2a28c73 [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive` module add 53e19f3 [SPARK-30032][BUILD] Upgrade to ORC 1.5.8 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 6 +++--- dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 6 +++--- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 6 +++--- pom.xml | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2a28c73 -> 53e19f3)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2a28c73 [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive` module add 53e19f3 [SPARK-30032][BUILD] Upgrade to ORC 1.5.8 No new revisions were added by this update. Summary of changes: dev/deps/spark-deps-hadoop-2.7-hive-1.2 | 6 +++--- dev/deps/spark-deps-hadoop-2.7-hive-2.3 | 6 +++--- dev/deps/spark-deps-hadoop-3.2-hive-2.3 | 6 +++--- pom.xml | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (38240a7 -> 2a28c73)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 38240a7 [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.` add 2a28c73 [SPARK-30031][BUILD][SQL] Remove `hive-2.3` profile from `sql/hive` module No new revisions were added by this update. Summary of changes: pom.xml | 8 sql/hive/pom.xml | 58 2 files changed, 29 insertions(+), 37 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.`
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 0dc22df [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.` 0dc22df is described below commit 0dc22df84f2fa53d60b418e8dca42bd959980083 Author: Dongjoon Hyun AuthorDate: Mon Nov 25 12:03:15 2019 -0800 [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.` This PR replace `TokenChecker` with `RegexChecker` in `scalastyle` and fixes the missed instances. This will remove the old `comons-lang2` dependency from `core` module **BEFORE** ``` $ dev/scalastyle Scalastyle checks failed at following occurrences: [error] /Users/dongjoon/PRS/SPARK-SerializationUtils/core/src/test/scala/org/apache/spark/util/PropertiesCloneBenchmark.scala:23:7: Use Commons Lang 3 classes (package org.apache.commons.lang3.*) instead [error] of Commons Lang 2 (package org.apache.commons.lang.*) [error] Total time: 23 s, completed Nov 25, 2019 11:47:44 AM ``` **AFTER** ``` $ dev/scalastyle Scalastyle checks passed. ``` No. Pass the GitHub Action linter. Closes #2 from dongjoon-hyun/SPARK-29081-2. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun (cherry picked from commit 38240a74dc047796e9f239e44d9bc0bbc66e1f7f) Signed-off-by: Dongjoon Hyun --- scalastyle-config.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scalastyle-config.xml b/scalastyle-config.xml index da5c3f2..866e938 100644 --- a/scalastyle-config.xml +++ b/scalastyle-config.xml @@ -234,7 +234,7 @@ This file is divided into 3 sections: scala.collection.JavaConverters._ and use .asScala / .asJava methods - + org\.apache\.commons\.lang\. Use Commons Lang 3 classes (package org.apache.commons.lang3.*) instead of Commons Lang 2 (package org.apache.commons.lang.*) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1466863 -> 38240a7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1466863 [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` add 38240a7 [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to check `org.apache.commons.lang.` No new revisions were added by this update. Summary of changes: .../src/test/scala/org/apache/spark/util/PropertiesCloneBenchmark.scala | 2 +- scalastyle-config.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (bec2068 -> 1466863)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from bec2068 [SPARK-26260][CORE] For disk store tasks summary table should show only successful tasks summary add 1466863 [SPARK-30015][BUILD] Move hive-storage-api dependency from `hive-2.3` to `sql/core` No new revisions were added by this update. Summary of changes: core/pom.xml | 6 +- pom.xml | 22 ++ sql/core/pom.xml | 4 3 files changed, 23 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (6880ccd -> 94ddc2a)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 6880ccd [MINOR][INFRA] Use GitHub Action Cache for `build` add 94ddc2a [SPARK-29971][CORE][2.4] Fix buffer leaks in `TransportFrameDecoder/TransportCipher` No new revisions were added by this update. Summary of changes: .../spark/network/crypto/TransportCipher.java | 50 +++- .../network/util/ByteArrayReadableChannel.java | 24 +++--- .../spark/network/util/TransportFrameDecoder.java | 18 - .../spark/network/crypto/TransportCipherSuite.java | 91 ++ 4 files changed, 148 insertions(+), 35 deletions(-) create mode 100644 common/network-common/src/test/java/org/apache/spark/network/crypto/TransportCipherSuite.java - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (29ebd93 -> bec2068)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 29ebd93 [SPARK-29979][SQL] Add basic/reserved property key constants in TableCatalog and SupportsNamespaces add bec2068 [SPARK-26260][CORE] For disk store tasks summary table should show only successful tasks summary No new revisions were added by this update. Summary of changes: .../org/apache/spark/status/AppStatusStore.scala | 82 +-- .../scala/org/apache/spark/status/LiveEntity.scala | 102 ++- .../scala/org/apache/spark/status/storeTypes.scala | 76 -- .../apache/spark/status/AppStatusStoreSuite.scala | 112 - 4 files changed, 234 insertions(+), 138 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f09c1a3 -> 29ebd93)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f09c1a3 [SPARK-29890][SQL] DataFrameNaFunctions.fill should handle duplicate columns add 29ebd93 [SPARK-29979][SQL] Add basic/reserved property key constants in TableCatalog and SupportsNamespaces No new revisions were added by this update. Summary of changes: .../sql/connector/catalog/SupportsNamespaces.java| 19 +++ .../spark/sql/connector/catalog/TableCatalog.java| 12 .../sql/catalyst/analysis/ResolveCatalogs.scala | 8 .../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++-- .../sql/catalyst/plans/logical/statements.scala | 5 - .../spark/sql/connector/catalog/CatalogV2Util.scala | 13 +++-- .../scala/org/apache/spark/sql/DataFrameWriter.scala | 2 +- .../catalyst/analysis/ResolveSessionCatalog.scala| 14 +++--- .../datasources/v2/DescribeNamespaceExec.scala | 11 +-- .../execution/datasources/v2/V2SessionCatalog.scala | 20 +--- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 3 +-- .../datasources/v2/V2SessionCatalogSuite.scala | 6 +++--- 12 files changed, 71 insertions(+), 47 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f09c1a3 -> 29ebd93)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f09c1a3 [SPARK-29890][SQL] DataFrameNaFunctions.fill should handle duplicate columns add 29ebd93 [SPARK-29979][SQL] Add basic/reserved property key constants in TableCatalog and SupportsNamespaces No new revisions were added by this update. Summary of changes: .../sql/connector/catalog/SupportsNamespaces.java| 19 +++ .../spark/sql/connector/catalog/TableCatalog.java| 12 .../sql/catalyst/analysis/ResolveCatalogs.scala | 8 .../spark/sql/catalyst/parser/AstBuilder.scala | 5 +++-- .../sql/catalyst/plans/logical/statements.scala | 5 - .../spark/sql/connector/catalog/CatalogV2Util.scala | 13 +++-- .../scala/org/apache/spark/sql/DataFrameWriter.scala | 2 +- .../catalyst/analysis/ResolveSessionCatalog.scala| 14 +++--- .../datasources/v2/DescribeNamespaceExec.scala | 11 +-- .../execution/datasources/v2/V2SessionCatalog.scala | 20 +--- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 3 +-- .../datasources/v2/V2SessionCatalogSuite.scala | 6 +++--- 12 files changed, 71 insertions(+), 47 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2d5de25 -> f09c1a3)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2d5de25 [SPARK-29415][CORE] Stage Level Sched: Add base ResourceProfile and Request classes add f09c1a3 [SPARK-29890][SQL] DataFrameNaFunctions.fill should handle duplicate columns No new revisions were added by this update. Summary of changes: .../apache/spark/sql/DataFrameNaFunctions.scala| 64 +++--- .../spark/sql/DataFrameNaFunctionsSuite.scala | 45 +++ 2 files changed, 88 insertions(+), 21 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-29415][CORE] Stage Level Sched: Add base ResourceProfile and Request classes
This is an automated email from the ASF dual-hosted git repository.
tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2d5de25 [SPARK-29415][CORE] Stage Level Sched: Add base
ResourceProfile and Request classes
2d5de25 is described below
commit 2d5de25a999e0e5580cf4024937b61e6c9265672
Author: Thomas Graves
AuthorDate: Mon Nov 25 09:36:39 2019 -0600
[SPARK-29415][CORE] Stage Level Sched: Add base ResourceProfile and Request
classes
### What changes were proposed in this pull request?
This PR is adding the base classes needed for Stage level scheduling. Its
adding a ResourceProfile and the executor and task resource request classes.
These are made private for now until we get all the parts implemented, at which
point this will become public interfaces. I am adding them first as all the
other subtasks for this feature require these classes. If people have better
ideas on breaking this feature up please let me know.
See https://issues.apache.org/jira/browse/SPARK-29415 for more detailed
design.
### Why are the changes needed?
New API for stage level scheduling. Its easier to add these first because
the other jira for this features will all use them.
### Does this PR introduce any user-facing change?
Yes adds API to create a ResourceProfile with executor/task resources, see
the spip jira https://issues.apache.org/jira/browse/SPARK-27495
Example of the api:
val rp = new ResourceProfile()
rp.require(new ExecutorResourceRequest("cores", 2))
rp.require(new ExecutorResourceRequest("gpu", 1,
Some("/opt/gpuScripts/getGpus")))
rp.require(new TaskResourceRequest("gpu", 1))
### How was this patch tested?
Tested using Unit tests added with this PR.
Closes #26284 from tgravescs/SPARK-29415.
Authored-by: Thomas Graves
Signed-off-by: Thomas Graves
---
.../spark/resource/ExecutorResourceRequest.scala | 77 ++
.../spark/resource/ExecutorResourceRequests.scala | 122 +++
.../apache/spark/resource/ResourceProfile.scala| 147 ++
.../org/apache/spark/resource/ResourceUtils.scala | 7 +-
.../spark/resource/TaskResourceRequest.scala | 43 ++
.../spark/resource/TaskResourceRequests.scala | 75 ++
.../spark/resource/JavaResourceProfileSuite.java | 66
.../spark/resource/ResourceProfileSuite.scala | 166 +
8 files changed, 701 insertions(+), 2 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequest.scala
b/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequest.scala
new file mode 100644
index 000..88ceaad
--- /dev/null
+++
b/core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequest.scala
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.resource
+
+import scala.collection.mutable
+
+import org.apache.spark.resource.ResourceUtils.RESOURCE_DOT
+
+/**
+ * An Executor resource request. This is used in conjunction with the
ResourceProfile to
+ * programmatically specify the resources needed for an RDD that will be
applied at the
+ * stage level.
+ *
+ * This is used to specify what the resource requirements are for an Executor
and how
+ * Spark can find out specific details about those resources. Not all the
parameters are
+ * required for every resource type. The resources names supported
+ * correspond to the regular Spark configs with the prefix removed. For
instance overhead
+ * memory in this api is memoryOverhead, which is
spark.executor.memoryOverhead with
+ * spark.executor removed. Resources like GPUs are resource.gpu
+ * (spark configs spark.executor.resource.gpu.*). The amount, discoveryScript,
and vendor
+ * parameters for resources are all the same parameters a user would specify
through the
+ * configs: spark.executor.resource.{resourceName}.{amount, discoveryScript,
vendor}.
+ *
+ * For instance, a user wants to allocate an Executor with GPU resources on
[spark] branch master updated (de21f28 -> bd9ce83)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from de21f28 [SPARK-29986][SQL] casting string to date/timestamp/interval should trim all whitespaces add bd9ce83 [SPARK-29975][SQL][FOLLOWUP] document --CONFIG_DIM No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/SQLQueryTestSuite.scala | 22 -- .../thriftserver/ThriftServerQueryTestSuite.scala | 8 +++- 2 files changed, 19 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
