[jira] [Updated] (SPARK-38661) [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests
[ https://issues.apache.org/jira/browse/SPARK-38661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Tzvetanov Grigorov updated SPARK-38661: -- Description: This ticket is a follow up of SPARK-38351. When building with Scala 2.13 many test classes produce warnings like: {code:java} [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead [warn] 'd.cast("string"), [warn] ^ [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead [warn] 'e.cast("string")).collect()) {code} For easier migration to Scala 3.x later it would be good to fix this warnings! Also as suggested by [https://github.com/HeartSaVioR] it would be good to use Spark's $"abc" syntax for columns. was: When building with Scala 2.13 many test classes produce warnings like: {code:java} [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead [warn] 'd.cast("string"), [warn] ^ [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead [warn] 'e.cast("string")).collect()) {code} For easier migration to Scala 3.x later it would be good to fix this warnings! > [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests > - > > Key: SPARK-38661 > URL: https://issues.apache.org/jira/browse/SPARK-38661 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.2.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Fix For: 3.3.0 > > > This ticket is a follow up of SPARK-38351. > > When building with Scala 2.13 many test classes produce warnings like: > {code:java} > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("d") instead > [warn] 'd.cast("string"), > [warn] ^ > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("e") instead > [warn] 'e.cast("string")).collect()) > {code} > For easier migration to Scala 3.x later it would be good to fix this warnings! > > Also as suggested by [https://github.com/HeartSaVioR] it would be good to use > Spark's $"abc" syntax for columns. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38661) [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests
Martin Tzvetanov Grigorov created SPARK-38661: - Summary: [TESTS] Replace 'abc & Symbol("abc") symbols with $"abc" in tests Key: SPARK-38661 URL: https://issues.apache.org/jira/browse/SPARK-38661 Project: Spark Issue Type: Improvement Components: Tests Affects Versions: 3.2.1 Reporter: Martin Tzvetanov Grigorov Assignee: Martin Tzvetanov Grigorov Fix For: 3.3.0 When building with Scala 2.13 many test classes produce warnings like: {code:java} [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead [warn] 'd.cast("string"), [warn] ^ [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead [warn] 'e.cast("string")).collect()) {code} For easier migration to Scala 3.x later it would be good to fix this warnings! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38351) [TESTS] Replace 'abc symbols with Symbol("abc") in tests
[ https://issues.apache.org/jira/browse/SPARK-38351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500812#comment-17500812 ] Martin Tzvetanov Grigorov commented on SPARK-38351: --- Yes, there are more ! Related to [https://github.com/apache/spark/pull/35560#issuecomment-1044535505] - I wanted the PR to be not too big for two reasons: 1) the review will be too long and tedious; 2) bigger chance for merge conflicts. I expect the more active developers in [SQL] to have some issues now. > [TESTS] Replace 'abc symbols with Symbol("abc") in tests > > > Key: SPARK-38351 > URL: https://issues.apache.org/jira/browse/SPARK-38351 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.2.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Fix For: 3.3.0 > > > When building with Scala 2.13 many test classes produce warnings like: > {code:java} > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("d") instead > [warn] 'd.cast("string"), > [warn] ^ > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("e") instead > [warn] 'e.cast("string")).collect()) > {code} > For easier migration to Scala 3.x later it would be good to fix this warnings! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38351) [TESTS] Replace 'abc symbols with Symbol("abc") in tests
[ https://issues.apache.org/jira/browse/SPARK-38351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500802#comment-17500802 ] Martin Tzvetanov Grigorov commented on SPARK-38351: --- [~srowen] Should I open a new Jira ticket for the next batch of Symbol() changes ? In my previous comment I've asked to keep this ticket open but maybe Spark project prefers separate tickets for each PR ? > [TESTS] Replace 'abc symbols with Symbol("abc") in tests > > > Key: SPARK-38351 > URL: https://issues.apache.org/jira/browse/SPARK-38351 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.2.1 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Fix For: 3.3.0 > > > When building with Scala 2.13 many test classes produce warnings like: > {code:java} > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("d") instead > [warn] 'd.cast("string"), > [warn] ^ > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("e") instead > [warn] 'e.cast("string")).collect()) > {code} > For easier migration to Scala 3.x later it would be good to fix this warnings! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38392) [K8S] Improve the names used for K8S namespace and driver pod in integration tests
Martin Tzvetanov Grigorov created SPARK-38392: - Summary: [K8S] Improve the names used for K8S namespace and driver pod in integration tests Key: SPARK-38392 URL: https://issues.apache.org/jira/browse/SPARK-38392 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.3.0 Reporter: Martin Tzvetanov Grigorov Currently when there is no configured K8S namespace by the user the Kubernetes IT tests use UUID.toString() without the '-'es: {code:java} val namespace = namespaceOption.getOrElse(UUID.randomUUID().toString.replaceAll("-", "")) {code} Proposal: prefix the temporary namespace with "spark-" or "spark-test-". The name of the driver pod is: {code:java} driverPodName = "spark-test-app-" + UUID.randomUUID().toString.replaceAll("-", "") {code} i.e. it does not mention that it is the driver. For non-test it uses names like `spark-on-k8s-app-f6bfc57f4a938abe-driver`, i.e. the value of "–name" config + hash + "-driver". In both IT tests and non-test the executor pods always mention "-exec-", so they are clear. Proposal: unify non-test and IT test name and use "spark-test-app-"+ hash + "-driver" for the IT tests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-38362) Move eclipse.m2e Maven plugin config in its own profile
[ https://issues.apache.org/jira/browse/SPARK-38362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Tzvetanov Grigorov updated SPARK-38362: -- Description: Today I had a weird issue with org.eclipse.m2e:lifecycle-mapping Maven fake-plugin: {code:java} WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available [WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced {code} It was weird because I didn't do any changes to my setup since yesterday when the maven build was working fine. {*}T{*}he actual problem was that ./dev/make-distribution was failing to read the version from pom.xml. The warnings above was the only thing printed by "mvn help:evaluate -Dexpression=project.version" so I thought it was related and spent time investigating it. There is no need other developers to waste time on Eclipse M2E warnings! org.eclipse.m2e:lifecycle-mapping is a hack that is used by Eclipse to map Maven plugins' lifecycle with Eclipse lifecycle. It does not affect plain Maven usage on the command line! There is no Maven artifact at [https://repo.maven.apache.org/maven2/org/eclipse/m2e] ! As explained at [https://stackoverflow.com/a/23707050/497381] the best way to setup Maven+m2e is by using a custom Maven profile that is auto-activated only by Eclipse when M2E plugin is being used: {code:java} only-eclipse m2e.version org.eclipse.m2e lifecycle-mapping 1.0.0 ... {code} was: Today I had a weird issue with org.eclipse.m2e:lifecycle-mapping Maven fake-plugin: {code:java} WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available [WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced {code} It was weird because I didn't do any changes to my setup since yesterday when the maven build was working fine. org.eclipse.m2e:lifecycle-mapping is a hack that is used by Eclipse to map Maven plugins' lifecycle with Eclipse lifecycle. It does not affect plain Maven usage on the command line! There is no Maven artifact at [https://repo.maven.apache.org/maven2/org/eclipse/m2e] ! As explained at [https://stackoverflow.com/a/23707050/497381] the best way to setup Maven+m2e is by using a custom Maven profile that is auto-activated only by Eclipse when M2E plugin is being used: {code:java} only-eclipse m2e.version org.eclipse.m2e lifecycle-mapping 1.0.0 ... {code} > Move eclipse.m2e Maven plugin config in its own profile > --- > > Key: SPARK-38362 > URL: https://issues.apache.org/jira/browse/SPARK-38362 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.0 >Reporter: Martin Tzvetanov Grigorov >Priority: Minor > > Today I had a weird issue with org.eclipse.m2e:lifecycle-mapping Maven > fake-plugin: > {code:java} > WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, > no dependency information available > [WARNING] Failed to retrieve plugin descriptor for > org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin > org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not > be resolved: org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 was not found in > https://maven-central.storage-download.googleapis.com/maven2/ during a > previous attempt. This failure was cached in the local repository and > resolution is not reattempted until the update interval of > gcs-maven-central-mirror has elapsed or updates are forced {code} > > It was weird because I didn't do any changes to my setup since yesterday when > the maven build
[jira] [Created] (SPARK-38362) Move eclipse.m2e Maven plugin config in its own profile
Martin Tzvetanov Grigorov created SPARK-38362: - Summary: Move eclipse.m2e Maven plugin config in its own profile Key: SPARK-38362 URL: https://issues.apache.org/jira/browse/SPARK-38362 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.3.0 Reporter: Martin Tzvetanov Grigorov Today I had a weird issue with org.eclipse.m2e:lifecycle-mapping Maven fake-plugin: {code:java} WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available [WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 was not found in https://maven-central.storage-download.googleapis.com/maven2/ during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of gcs-maven-central-mirror has elapsed or updates are forced {code} It was weird because I didn't do any changes to my setup since yesterday when the maven build was working fine. org.eclipse.m2e:lifecycle-mapping is a hack that is used by Eclipse to map Maven plugins' lifecycle with Eclipse lifecycle. It does not affect plain Maven usage on the command line! There is no Maven artifact at [https://repo.maven.apache.org/maven2/org/eclipse/m2e] ! As explained at [https://stackoverflow.com/a/23707050/497381] the best way to setup Maven+m2e is by using a custom Maven profile that is auto-activated only by Eclipse when M2E plugin is being used: {code:java} only-eclipse m2e.version org.eclipse.m2e lifecycle-mapping 1.0.0 ... {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38351) [TESTS] Replace 'abc symbols with Symbol("abc") in tests
[ https://issues.apache.org/jira/browse/SPARK-38351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17499373#comment-17499373 ] Martin Tzvetanov Grigorov commented on SPARK-38351: --- Please do not close this ticket when [https://github.com/apache/spark/pull/35560] is merged! There will be few more PRs after it! The task will be split in several PRs so that it is easier to review them. > [TESTS] Replace 'abc symbols with Symbol("abc") in tests > > > Key: SPARK-38351 > URL: https://issues.apache.org/jira/browse/SPARK-38351 > Project: Spark > Issue Type: Improvement > Components: Tests >Affects Versions: 3.2.1 >Reporter: Martin Tzvetanov Grigorov >Priority: Minor > > When building with Scala 2.13 many test classes produce warnings like: > {code:java} > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("d") instead > [warn] 'd.cast("string"), > [warn] ^ > [warn] > /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: > [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; > use Symbol("e") instead > [warn] 'e.cast("string")).collect()) > {code} > For easier migration to Scala 3.x later it would be good to fix this warnings! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38351) [TESTS] Replace 'abc symbols with Symbol("abc") in tests
Martin Tzvetanov Grigorov created SPARK-38351: - Summary: [TESTS] Replace 'abc symbols with Symbol("abc") in tests Key: SPARK-38351 URL: https://issues.apache.org/jira/browse/SPARK-38351 Project: Spark Issue Type: Improvement Components: Tests Affects Versions: 3.2.1 Reporter: Martin Tzvetanov Grigorov When building with Scala 2.13 many test classes produce warnings like: {code:java} [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:562:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("d") instead [warn] 'd.cast("string"), [warn] ^ [warn] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala:563:11: [deprecation @ | origin= | version=2.13.0] symbol literal is deprecated; use Symbol("e") instead [warn] 'e.cast("string")).collect()) {code} For easier migration to Scala 3.x later it would be good to fix this warnings! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38242) Sort the SparkSubmit debug output
Martin Tzvetanov Grigorov created SPARK-38242: - Summary: Sort the SparkSubmit debug output Key: SPARK-38242 URL: https://issues.apache.org/jira/browse/SPARK-38242 Project: Spark Issue Type: Wish Components: Spark Submit Affects Versions: 3.2.1 Reporter: Martin Tzvetanov Grigorov When '-verbose' is passed to SparkSubmit it prints some useful debug information: Main class, Argument and Spark config. I find it a bit hard to find information there because the arguments/configs are shuffled. I suggest to sort those before printing them. {code:java} Main class: org.apache.spark.deploy.k8s.submit.KubernetesClientApplication Arguments: --main-class --primary-java-resource local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar org.apache.spark.examples.SparkPi Spark config: (spark.app.name,spark-on-k8s-app) (spark.app.submitTime,1645106476125) (spark.driver.cores,1) (spark.driver.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true) (spark.driver.memory,2048m) (spark.dynamicAllocation.enabled,true) (spark.dynamicAllocation.shuffleTracking.enabled,true) (spark.executor.cores,2) (spark.executor.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true) (spark.executor.instances,3) (spark.executor.memory,2048m) (spark.jars,local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar) (spark.kubernetes.allocation.batch.delay,1) (spark.kubernetes.allocation.batch.size,3) (spark.kubernetes.authenticate.driver.serviceAccountName,spark-account-name) (spark.kubernetes.driver.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim) (spark.kubernetes.executor.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim) (spark.kubernetes.namespace,spark-on-k8s) (spark.master,k8s://https://192.168.49.2:8443) (spark.network.timeout,300) (spark.submit.deployMode,cluster) (spark.submit.pyFiles,) Classpath elements:{code} The "Parsed arguments:" order is hardcoded at org.apache.spark.deploy.SparkSubmitArguments#toString, so they are still shuffled. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38241) Close KubernetesClient in K8S integrations tests
Martin Tzvetanov Grigorov created SPARK-38241: - Summary: Close KubernetesClient in K8S integrations tests Key: SPARK-38241 URL: https://issues.apache.org/jira/browse/SPARK-38241 Project: Spark Issue Type: Task Components: Kubernetes, Tests Affects Versions: 3.2.1 Reporter: Martin Tzvetanov Grigorov The implementations of org.apache.spark.deploy.k8s.integrationtest.backend.IntegrationTestBackend should close their KubernetesClient instance in #cleanUp() method. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38096) [BUILD] Update sbt to 1.6.2
Martin Tzvetanov Grigorov created SPARK-38096: - Summary: [BUILD] Update sbt to 1.6.2 Key: SPARK-38096 URL: https://issues.apache.org/jira/browse/SPARK-38096 Project: Spark Issue Type: Task Components: Build Affects Versions: 3.2.1 Reporter: Martin Tzvetanov Grigorov sbt 1.6.2 has been released few days ago - https://eed3si9n.com/sbt-1.6.2 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org