[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782582371 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39880/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
AngersZh commented on a change in pull request #30421: URL: https://github.com/apache/spark/pull/30421#discussion_r579618193 ## File path: docs/sql-migration-guide.md ## @@ -101,6 +101,8 @@ license: | - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - In Spark 3.1, we support using corresponding typed literal of partition column value type as partition column value in SQL, such as if we have a partition table with partition column of date type, we can use typed date literal `date '2020-01-01'` as partition spec `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`. Review comment: > Should be `In Spark 3.2`? Hmm, yea, updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
MaxGekk commented on a change in pull request #30421: URL: https://github.com/apache/spark/pull/30421#discussion_r579617798 ## File path: docs/sql-migration-guide.md ## @@ -101,6 +101,8 @@ license: | - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - In Spark 3.1, we support using corresponding typed literal of partition column value type as partition column value in SQL, such as if we have a partition table with partition column of date type, we can use typed date literal `date '2020-01-01'` as partition spec `PARTITION (dt = date '2020-01-01')`, it will be treated as partition column value `2020-01-01`. In Spark 3.0 the partition value will be treated as string value `date '2020-01-01'` and it's a illegal date type string value and will been converted to `__HIVE_DEFAULT_PARTITION__`. Review comment: Should be `In Spark 3.2`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yuchenhuo opened a new pull request #31599: [SPARK-34481] Refactor dataframe reader/writer optionsWithPath logic
yuchenhuo opened a new pull request #31599: URL: https://github.com/apache/spark/pull/31599 ### What changes were proposed in this pull request? Extract optionsWithPath logic into their own function. ### Why are the changes needed? Reduce the code duplication and improve modularity. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Just some refactoring. Existing tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782578304 **[Test build #135299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135299/testReport)** for PR 31598 at commit [`4db4c05`](https://github.com/apache/spark/commit/4db4c0599f1db93bbf87969f449ffe69f8dda4c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782578463 **[Test build #135299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135299/testReport)** for PR 31598 at commit [`4db4c05`](https://github.com/apache/spark/commit/4db4c0599f1db93bbf87969f449ffe69f8dda4c1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782578304 **[Test build #135299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135299/testReport)** for PR 31598 at commit [`4db4c05`](https://github.com/apache/spark/commit/4db4c0599f1db93bbf87969f449ffe69f8dda4c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
SparkQA commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782578112 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39877/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
mridulm commented on a change in pull request #31496:
URL: https://github.com/apache/spark/pull/31496#discussion_r579614384
##
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##
@@ -242,17 +252,48 @@ class ResourceProfile(
object ResourceProfile extends Logging {
// task resources
+ /**
+ * built-in task resource: cpus
+ */
val CPUS = "cpus"
// Executor resources
// Make sure add new executor resource in below allSupportedExecutorResources
+ /**
+ * built-in executor resource: cores
+ */
val CORES = "cores"
+ /**
+ * built-in executor resource: cores
+ */
val MEMORY = "memory"
+ /**
+ * built-in executor resource: offHeap
+ */
val OFFHEAP_MEM = "offHeap"
+ /**
+ * built-in executor resource: memoryOverhead
+ */
val OVERHEAD_MEM = "memoryOverhead"
+ /**
+ * built-in executor resource: pyspark.memory
+ */
val PYSPARK_MEM = "pyspark.memory"
- // all supported spark executor resources (minus the custom resources like
GPUs/FPGAs)
- val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM, OFFHEAP_MEM)
+ private val _allSupportedExecutorResources =
+Seq(CORES, MEMORY, OVERHEAD_MEM, PYSPARK_MEM, OFFHEAP_MEM)
+
+ /**
+ * Return all supported Spark built-in executor resources, custom resources
like GPUs/FPGAs
+ * are excluded.
+ */
+ def allSupportedExecutorResources: Seq[String] =
_allSupportedExecutorResources
Review comment:
This looks fine to me ... hope you are ok with it @tgravescs
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bozhang2820 commented on pull request #31590: [SPARK-34471][SS][DOCS] Document Streaming Table APIs in Structured Streaming Programming Guide
bozhang2820 commented on pull request #31590: URL: https://github.com/apache/spark/pull/31590#issuecomment-782577421 Thank you @HeartSaVioR and everyone who reviewed this change! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
AngersZh commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782577362 Gentle ping @maropu @cloud-fan @MaxGekk Since #30538 is Done, I think we can restart this pr. Any more suggestion? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AngersZh commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579613517
##
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
this
}
+// These submit configuration only effect when config before submit app.
+private val SUBMIT_LAUNCHER_CONFIG =
+ Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS,
DRIVER_EXTRA_LIBRARY_PATH,
+PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL,
CHILD_PROCESS_LOGGER_NAME,
+CHILD_CONNECTION_TIMEOUT)
+
+// These configuration can effect when SparkContext is not started.
+private val EXECUTOR_LAUNCHER_CONFIG =
+ Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH,
EXECUTOR_DEFAULT_JAVA_OPTIONS,
+EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH,
EXECUTOR_CORES)
+
+def checkAndSetConfig(key: String, value: String): Unit = {
+ if (SparkContext.getActive.isEmpty) {
+if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+ logWarning(s"Since spark has been started, configuration ${key}
won't work" +
+s" when set it here")
Review comment:
> Can we have a directional guide instead of `xxx won't work when set it
here`?
Change code to check these in `getOrCreate` to check all case together,
avoid to log too much information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AngersZh commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579613517
##
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
this
}
+// These submit configuration only effect when config before submit app.
+private val SUBMIT_LAUNCHER_CONFIG =
+ Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS,
DRIVER_EXTRA_LIBRARY_PATH,
+PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL,
CHILD_PROCESS_LOGGER_NAME,
+CHILD_CONNECTION_TIMEOUT)
+
+// These configuration can effect when SparkContext is not started.
+private val EXECUTOR_LAUNCHER_CONFIG =
+ Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH,
EXECUTOR_DEFAULT_JAVA_OPTIONS,
+EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH,
EXECUTOR_CORES)
+
+def checkAndSetConfig(key: String, value: String): Unit = {
+ if (SparkContext.getActive.isEmpty) {
+if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+ logWarning(s"Since spark has been started, configuration ${key}
won't work" +
+s" when set it here")
Review comment:
> Can we have a directional guide instead of `xxx won't work when set it
here`?
Change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
SparkQA commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782576564 **[Test build #135300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135300/testReport)** for PR 31496 at commit [`96d0760`](https://github.com/apache/spark/commit/96d07604b79c52bba4ff3aa4b892f363e73b2faa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shardulm94 commented on pull request #31597: [SPARK-34477][CORE] Register KryoSerializers for Avro GenericData classes
shardulm94 commented on pull request #31597: URL: https://github.com/apache/spark/pull/31597#issuecomment-782576508 The test which failed above `org.apache.spark.ml.source.image.ImageFileFormatSuite` seems unrelated to this change. I also ran the test locally and it seems to be working fine. `build/mvn test -Dtest=none -Dsuites="org.apache.spark.ml.source.image.ImageFileFormatSuite" -pl mllib` ``` Run starting. Expected test count is: 7 ImageFileFormatSuite: - Smoke test: create basic ImageSchema dataframe - image datasource count test - image datasource test: read jpg image - image datasource test: read png image - image datasource test: read non image - image datasource partition test - readImages pixel values test Run completed in 6 seconds, 939 milliseconds. Total number of tests run: 7 Suites: completed 1, aborted 0 Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
AmplabJenkins removed a comment on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782576167 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
AmplabJenkins removed a comment on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782576173 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135290/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
AmplabJenkins removed a comment on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782576165 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
AmplabJenkins removed a comment on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782576166 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39876/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31548: [SPARK-34127][SQL] Support table valued command
AmplabJenkins removed a comment on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782576171 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135287/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AmplabJenkins removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782576170 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135288/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AmplabJenkins commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782576170 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135288/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
AmplabJenkins commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782576172 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
AmplabJenkins commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782576168 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
AmplabJenkins commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782576166 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39876/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31548: [SPARK-34127][SQL] Support table valued command
AmplabJenkins commented on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782576171 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135287/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
AmplabJenkins commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782576173 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135290/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31548: [SPARK-34127][SQL] Support table valued command
SparkQA removed a comment on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782547075 **[Test build #135287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135287/testReport)** for PR 31548 at commit [`2eeb9d7`](https://github.com/apache/spark/commit/2eeb9d7dd0aa23cfddb0babe710dfb3352b16383). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
SparkQA removed a comment on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782557549 **[Test build #135291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135291/testReport)** for PR 31480 at commit [`8a8aadd`](https://github.com/apache/spark/commit/8a8aadd0816a77af40feb51264bfaf643458e072). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31548: [SPARK-34127][SQL] Support table valued command
SparkQA commented on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782575302 **[Test build #135287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135287/testReport)** for PR 31548 at commit [`2eeb9d7`](https://github.com/apache/spark/commit/2eeb9d7dd0aa23cfddb0babe710dfb3352b16383). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
SparkQA commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782575180 **[Test build #135291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135291/testReport)** for PR 31480 at commit [`8a8aadd`](https://github.com/apache/spark/commit/8a8aadd0816a77af40feb51264bfaf643458e072). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
SparkQA commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782574829 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39870/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31590: [SPARK-34471][SS][DOCS] Document Streaming Table APIs in Structured Streaming Programming Guide
HeartSaVioR commented on pull request #31590: URL: https://github.com/apache/spark/pull/31590#issuecomment-782574714 Thanks @bozhang2820 for the contribution, and thanks all for reviewing this quickly! I merged this in master/branch-3.1 branches. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #31590: [SPARK-34471][SS][DOCS] Document Streaming Table APIs in Structured Streaming Programming Guide
HeartSaVioR closed pull request #31590: URL: https://github.com/apache/spark/pull/31590 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
SparkQA commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782574525 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39877/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
SparkQA removed a comment on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782557663 **[Test build #135290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135290/testReport)** for PR 31496 at commit [`5ce6fbe`](https://github.com/apache/spark/commit/5ce6fbe62563f025bfae27f6c89043bad3d23b2a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #31590: [SPARK-34471][SS][DOCS] Document Streaming Table APIs in Structured Streaming Programming Guide
HeartSaVioR commented on pull request #31590: URL: https://github.com/apache/spark/pull/31590#issuecomment-782574463 OK I think there're enough number of reviewers and all reviewers already approved this so "cross-checked". Let's go merging now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
SparkQA commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782573293 **[Test build #135290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135290/testReport)** for PR 31496 at commit [`5ce6fbe`](https://github.com/apache/spark/commit/5ce6fbe62563f025bfae27f6c89043bad3d23b2a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
SparkQA removed a comment on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782526850 **[Test build #135285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135285/testReport)** for PR 31595 at commit [`8a3be92`](https://github.com/apache/spark/commit/8a3be928a37a6f62534561061c4872fae73db11c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782572462 **[Test build #135285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135285/testReport)** for PR 31595 at commit [`8a3be92`](https://github.com/apache/spark/commit/8a3be928a37a6f62534561061c4872fae73db11c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782557550 **[Test build #135288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135288/testReport)** for PR 31598 at commit [`675d35f`](https://github.com/apache/spark/commit/675d35f1ce978930c760d173e04c7c38a9e77526). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782571943 **[Test build #135288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135288/testReport)** for PR 31598 at commit [`675d35f`](https://github.com/apache/spark/commit/675d35f1ce978930c760d173e04c7c38a9e77526). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
SparkQA commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782571631 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39876/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
Ngone51 commented on a change in pull request #31496:
URL: https://github.com/apache/spark/pull/31496#discussion_r579605235
##
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##
@@ -242,17 +252,48 @@ class ResourceProfile(
object ResourceProfile extends Logging {
// task resources
+ /**
+ * built-in task resource: cpus
+ */
val CPUS = "cpus"
// Executor resources
// Make sure add new executor resource in below allSupportedExecutorResources
+ /**
+ * built-in executor resource: cores
+ */
val CORES = "cores"
+ /**
+ * built-in executor resource: cores
+ */
val MEMORY = "memory"
+ /**
+ * built-in executor resource: offHeap
+ */
val OFFHEAP_MEM = "offHeap"
+ /**
+ * built-in executor resource: memoryOverhead
+ */
val OVERHEAD_MEM = "memoryOverhead"
+ /**
+ * built-in executor resource: pyspark.memory
+ */
val PYSPARK_MEM = "pyspark.memory"
- // all supported spark executor resources (minus the custom resources like
GPUs/FPGAs)
- val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM, OFFHEAP_MEM)
+ private val _allSupportedExecutorResources =
+Seq(CORES, MEMORY, OVERHEAD_MEM, PYSPARK_MEM, OFFHEAP_MEM)
+
+ /**
+ * Return all supported Spark built-in executor resources, custom resources
like GPUs/FPGAs
+ * are excluded.
+ */
+ def allSupportedExecutorResources: Seq[String] =
_allSupportedExecutorResources
Review comment:
I see, this also looks ok to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
Ngone51 commented on a change in pull request #31496:
URL: https://github.com/apache/spark/pull/31496#discussion_r579605207
##
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##
@@ -242,17 +252,48 @@ class ResourceProfile(
object ResourceProfile extends Logging {
// task resources
+ /**
+ * built-in task resource: cpus
+ */
val CPUS = "cpus"
// Executor resources
// Make sure add new executor resource in below allSupportedExecutorResources
+ /**
+ * built-in executor resource: cores
+ */
val CORES = "cores"
+ /**
+ * built-in executor resource: cores
+ */
val MEMORY = "memory"
+ /**
+ * built-in executor resource: offHeap
+ */
val OFFHEAP_MEM = "offHeap"
+ /**
+ * built-in executor resource: memoryOverhead
+ */
val OVERHEAD_MEM = "memoryOverhead"
+ /**
+ * built-in executor resource: pyspark.memory
+ */
val PYSPARK_MEM = "pyspark.memory"
- // all supported spark executor resources (minus the custom resources like
GPUs/FPGAs)
- val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM, OFFHEAP_MEM)
+ private val _allSupportedExecutorResources =
+Seq(CORES, MEMORY, OVERHEAD_MEM, PYSPARK_MEM, OFFHEAP_MEM)
+
+ /**
+ * Return all supported Spark built-in executor resources, custom resources
like GPUs/FPGAs
+ * are excluded.
+ */
+ def allSupportedExecutorResources: Seq[String] =
_allSupportedExecutorResources
Review comment:
I see, this also looks ok to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782570709 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39875/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #31569: [SPARK-34443][CORE] Replace symbol literals with Symbol constructor invocations to comply with Scala 2.13
sarutak edited a comment on pull request #31569: URL: https://github.com/apache/spark/pull/31569#issuecomment-782570546 > It might be better to show the current blessed syntax in all our code, examples and docs no? Under the `examples`, only `SimpleTypedAggregator.scala` uses `'` syntax, which is already in the scope of this PR. Under the `doc`, the following docs seems to use the syntax. * `sql-data-sources-avro.md` * `sql-ref-syntax-qry-explain.md` I'll fix them too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
Ngone51 commented on a change in pull request #31496:
URL: https://github.com/apache/spark/pull/31496#discussion_r579605057
##
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##
@@ -242,17 +252,48 @@ class ResourceProfile(
object ResourceProfile extends Logging {
// task resources
+ /**
+ * built-in task resource: cpus
+ */
val CPUS = "cpus"
// Executor resources
// Make sure add new executor resource in below allSupportedExecutorResources
+ /**
+ * built-in executor resource: cores
+ */
val CORES = "cores"
+ /**
+ * built-in executor resource: cores
+ */
val MEMORY = "memory"
+ /**
+ * built-in executor resource: offHeap
+ */
val OFFHEAP_MEM = "offHeap"
+ /**
+ * built-in executor resource: memoryOverhead
+ */
val OVERHEAD_MEM = "memoryOverhead"
+ /**
+ * built-in executor resource: pyspark.memory
+ */
val PYSPARK_MEM = "pyspark.memory"
- // all supported spark executor resources (minus the custom resources like
GPUs/FPGAs)
- val allSupportedExecutorResources = Seq(CORES, MEMORY, OVERHEAD_MEM,
PYSPARK_MEM, OFFHEAP_MEM)
+ private val _allSupportedExecutorResources =
+Seq(CORES, MEMORY, OVERHEAD_MEM, PYSPARK_MEM, OFFHEAP_MEM)
+
+ /**
+ * Return all supported Spark built-in executor resources, custom resources
like GPUs/FPGAs
+ * are excluded.
+ */
+ def allSupportedExecutorResources: Seq[String] =
_allSupportedExecutorResources
Review comment:
Array is mutable, as @mridulm mentioned too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #31569: [SPARK-34443][CORE] Replace symbol literals with Symbol constructor invocations to comply with Scala 2.13
sarutak commented on pull request #31569: URL: https://github.com/apache/spark/pull/31569#issuecomment-782570546 > It might be better to show the current blessed syntax in all our code, examples and docs no? Under the `examples`, only `SimpleTypedAggregator.scala` uses `'` syntax, which is already in the scope of this PR. Under the `doc`, the following docs seems to use the syntax. * `sql-data-sources-avro.md` * `sql-ref-syntax-qry-explain.md` I'll fix them too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
AmplabJenkins removed a comment on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782569209 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135297/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
AmplabJenkins commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782569209 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135297/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
SparkQA removed a comment on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782562113 **[Test build #135297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135297/testReport)** for PR 31588 at commit [`f9a0465`](https://github.com/apache/spark/commit/f9a0465d27e85df9a5ba24321ac7ae6fb3bfb572). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
SparkQA commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782569104 **[Test build #135297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135297/testReport)** for PR 31588 at commit [`f9a0465`](https://github.com/apache/spark/commit/f9a0465d27e85df9a5ba24321ac7ae6fb3bfb572). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AngersZh commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579603890
##
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
this
}
+// These submit configuration only effect when config before submit app.
+private val SUBMIT_LAUNCHER_CONFIG =
+ Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS,
DRIVER_EXTRA_LIBRARY_PATH,
+PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL,
CHILD_PROCESS_LOGGER_NAME,
+CHILD_CONNECTION_TIMEOUT)
+
+// These configuration can effect when SparkContext is not started.
+private val EXECUTOR_LAUNCHER_CONFIG =
+ Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH,
EXECUTOR_DEFAULT_JAVA_OPTIONS,
+EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH,
EXECUTOR_CORES)
+
+def checkAndSetConfig(key: String, value: String): Unit = {
+ if (SparkContext.getActive.isEmpty) {
+if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+ logWarning(s"Since spark has been started, configuration ${key}
won't work" +
+s" when set it here")
Review comment:
> Can we have a directional guide instead of `xxx won't work when set it
here`?
Yea, I will make it more clear according to the doc you refer to.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
SparkQA commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782568973 **[Test build #135298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135298/testReport)** for PR 30421 at commit [`b68ee81`](https://github.com/apache/spark/commit/b68ee8196f072cedb151ffc623a7c4e8c7755d1d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AngersZh commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782568954 > Although [we had a similar document since Apache Spark 2.3.0](https://github.com/apache/spark/commit/457dc9ccbf8404fef6c1ebf8f82e59e4ba480a0e), I agree with @AngersZh 's suggestion because these are very frequent Q To be honest, not all user will read all usage document and they even not know too much about the whole process of spark, then they don't understand what is saying in the document . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
AngersZh commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782568625 ping @tgravescs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase
AmplabJenkins removed a comment on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-782568360 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39872/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase
AmplabJenkins commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-782568360 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39872/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782568250 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39875/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #31597: [SPARK-34477][CORE] Register KryoSerializers for Avro GenericData classes
dongjoon-hyun commented on pull request #31597: URL: https://github.com/apache/spark/pull/31597#issuecomment-782567917 cc @gengliangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
dongjoon-hyun commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579602687
##
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
this
}
+// These submit configuration only effect when config before submit app.
+private val SUBMIT_LAUNCHER_CONFIG =
+ Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS,
DRIVER_EXTRA_LIBRARY_PATH,
+PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL,
CHILD_PROCESS_LOGGER_NAME,
+CHILD_CONNECTION_TIMEOUT)
+
+// These configuration can effect when SparkContext is not started.
+private val EXECUTOR_LAUNCHER_CONFIG =
+ Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH,
EXECUTOR_DEFAULT_JAVA_OPTIONS,
+EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH,
EXECUTOR_CORES)
+
+def checkAndSetConfig(key: String, value: String): Unit = {
+ if (SparkContext.getActive.isEmpty) {
+if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+ logWarning(s"Since spark has been started, configuration ${key}
won't work" +
+s" when set it here")
Review comment:
Can we have a directional guide instead of `xxx won't work when set it
here`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31588: [SPARK-34470][ML] VectorSlicer utilize ordering if possible
SparkQA commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782567428 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39876/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31573: [SPARK-34444][SQL] Pushdown scalar-subquery filter to FileSourceScan
wangyum commented on pull request #31573: URL: https://github.com/apache/spark/pull/31573#issuecomment-782565340 Thank you all. I need to fix the failed test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31563: [SPARK-34436][SQL] DPP support LIKE ANY/ALL expression
wangyum commented on pull request #31563: URL: https://github.com/apache/spark/pull/31563#issuecomment-782564922 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct like operators
SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-782564410 **[Test build #135296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135296/testReport)** for PR 31595 at commit [`164706e`](https://github.com/apache/spark/commit/164706ef3ce4e43676b19513a7deaa444204811f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase
SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-782564125 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39872/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
AmplabJenkins removed a comment on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782563594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39871/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
AmplabJenkins commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782563594 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39871/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
SparkQA commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782563590 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39871/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.
AmplabJenkins removed a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-782563505 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39868/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.
AmplabJenkins commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-782563505 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39868/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-782563499 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39868/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
SparkQA commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782563403 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39869/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
AmplabJenkins removed a comment on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782563410 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39869/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
AmplabJenkins commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782563410 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39869/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31480: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner
SparkQA commented on pull request #31480: URL: https://github.com/apache/spark/pull/31480#issuecomment-782563371 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39870/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #31479: [SPARK-34373][SQL] HiveThriftServer2 startWithContext may hang with a race issue
yaooqinn commented on pull request #31479: URL: https://github.com/apache/spark/pull/31479#issuecomment-782562741 gentle ping @HyukjinKwon @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31588: [SPARK-34470][ML] VectorSlicer use ordering if possible
SparkQA commented on pull request #31588: URL: https://github.com/apache/spark/pull/31588#issuecomment-782562113 **[Test build #135297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135297/testReport)** for PR 31588 at commit [`f9a0465`](https://github.com/apache/spark/commit/f9a0465d27e85df9a5ba24321ac7ae6fb3bfb572). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31196: [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString
AmplabJenkins removed a comment on pull request #31196: URL: https://github.com/apache/spark/pull/31196#issuecomment-782561779 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135284/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
AmplabJenkins removed a comment on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782561704 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31548: [SPARK-34127][SQL] Support table valued command
AmplabJenkins removed a comment on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782561705 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39866/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AmplabJenkins removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782561703 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39867/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
AmplabJenkins removed a comment on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782561702 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135292/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31196: [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString
AmplabJenkins commented on pull request #31196: URL: https://github.com/apache/spark/pull/31196#issuecomment-782561779 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135284/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
AmplabJenkins commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-782561704 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31548: [SPARK-34127][SQL] Support table valued command
AmplabJenkins commented on pull request #31548: URL: https://github.com/apache/spark/pull/31548#issuecomment-782561705 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39866/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
AmplabJenkins commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782561702 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135292/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
AmplabJenkins commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782561703 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39867/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31196: [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString
SparkQA removed a comment on pull request #31196: URL: https://github.com/apache/spark/pull/31196#issuecomment-782478032 **[Test build #135284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135284/testReport)** for PR 31196 at commit [`e95280f`](https://github.com/apache/spark/commit/e95280f3b0da8c8465361e7703c1e691ddc4925a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31196: [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString
SparkQA commented on pull request #31196: URL: https://github.com/apache/spark/pull/31196#issuecomment-782561537 **[Test build #135284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135284/testReport)** for PR 31196 at commit [`e95280f`](https://github.com/apache/spark/commit/e95280f3b0da8c8465361e7703c1e691ddc4925a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782560941 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39867/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase
SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-782560710 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39872/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys
SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-782560441 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39867/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
SparkQA removed a comment on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782557538 **[Test build #135292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135292/testReport)** for PR 31437 at commit [`db37f5e`](https://github.com/apache/spark/commit/db37f5e791315263d4c83f7ee93d2d4a1b356b16). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
SparkQA commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782560299 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39871/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.
SparkQA commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-782560240 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39868/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31496: [SPARK-34384][CORE] Add missing docs for ResourceProfile APIs
SparkQA commented on pull request #31496: URL: https://github.com/apache/spark/pull/31496#issuecomment-782560231 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39869/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31437: [SPARK-34329][YARN] When hit ApplicationAttemptNotFoundException, we can't just stop app for all case
SparkQA commented on pull request #31437: URL: https://github.com/apache/spark/pull/31437#issuecomment-782559847 **[Test build #135292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135292/testReport)** for PR 31437 at commit [`db37f5e`](https://github.com/apache/spark/commit/db37f5e791315263d4c83f7ee93d2d4a1b356b16). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31196: [SPARK-34129][SQL] Add table name to LogicalRelation.simpleString
wangyum commented on a change in pull request #31196:
URL: https://github.com/apache/spark/pull/31196#discussion_r575671102
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala
##
@@ -64,7 +64,8 @@ case class LogicalRelation(
}
override def simpleString(maxFields: Int): String = {
-s"Relation[${truncatedString(output, ",", maxFields)}] $relation"
+s"Relation ${catalogTable.map(_.identifier.unquotedString).getOrElse("")}"
+
+ s"[${truncatedString(output, ",", maxFields)}] $relation"
Review comment:
Yes. It seems v2 already show the table name. for example:
`ParquetTable.name()` is table name.
https://github.com/apache/spark/blob/e857e06452c2cf478beb31367f76d6950b660ebb/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L67
https://github.com/apache/spark/blob/e857e06452c2cf478beb31367f76d6950b660ebb/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala#L123
https://github.com/apache/spark/blob/3d1dce75d96373130e27b3809c73d3796b5b77be/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetDataSourceV2.scala#L34-L36
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
