[jira] [Commented] (SPARK-38265) Update comments of ExecutorAllocationClient
[ https://issues.apache.org/jira/browse/SPARK-38265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495269#comment-17495269 ] Shockang commented on SPARK-38265: -- Working on this. > Update comments of ExecutorAllocationClient > --- > > Key: SPARK-38265 > URL: https://issues.apache.org/jira/browse/SPARK-38265 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.1 >Reporter: Shockang >Priority: Trivial > Fix For: 3.3.0 > > > The class comment of ExecutorAllocationClient is out of date. > {code:java} > This is currently supported only in YARN mode. {code} > Nowadays, this is supported in the following modes: Spark's Standalone, > YARN-Client, YARN-Cluster, Mesos, Kubernetes. > > In my opinion, this comment should be updated. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-38265) Update comments of ExecutorAllocationClient
Shockang created SPARK-38265: Summary: Update comments of ExecutorAllocationClient Key: SPARK-38265 URL: https://issues.apache.org/jira/browse/SPARK-38265 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.2.1 Reporter: Shockang Fix For: 3.3.0 The class comment of ExecutorAllocationClient is out of date. {code:java} This is currently supported only in YARN mode. {code} Nowadays, this is supported in the following modes: Spark's Standalone, YARN-Client, YARN-Cluster, Mesos, Kubernetes. In my opinion, this comment should be updated. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-37030) Maven build failed in windows!
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang resolved SPARK-37030. -- Resolution: Done > Maven build failed in windows! > -- > > Key: SPARK-37030 > URL: https://issues.apache.org/jira/browse/SPARK-37030 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.0 > Environment: OS: Windows 10 Professional > OS Version: 21H1 > Maven Version: 3.6.3 > >Reporter: Shockang >Priority: Minor > Fix For: 3.2.0 > > Attachments: image-2021-10-17-22-18-16-616.png > > > I pulled the latest Spark master code on my local windows 10 computer and > executed the following command: > {code:java} > mvn -DskipTests clean install{code} > Build failed! > !image-2021-10-17-22-18-16-616.png! > {code:java} > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run > (default) on project spark-core_2.12: An Ant BuildException has occured: > Execute failed: java.io.IOException: Cannot run program "bash" (in directory > "C:\bigdata\spark\core"): CreateProcess error=2{code} > It seems that the plugin: maven-antrun-plugin cannot run because of windows > no bash. > The following code comes from pom.xml in spark-core module. > {code:java} > > org.apache.maven.plugins > maven-antrun-plugin > > > generate-resources > > > > > > > > > > > > run > > > > > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37030) Maven build failed in windows!
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17438017#comment-17438017 ] Shockang commented on SPARK-37030: -- [~hyukjin.kwon] Thank you for your suggestion. This problem has been solved. > Maven build failed in windows! > -- > > Key: SPARK-37030 > URL: https://issues.apache.org/jira/browse/SPARK-37030 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.0 > Environment: OS: Windows 10 Professional > OS Version: 21H1 > Maven Version: 3.6.3 > >Reporter: Shockang >Priority: Minor > Fix For: 3.2.0 > > Attachments: image-2021-10-17-22-18-16-616.png > > > I pulled the latest Spark master code on my local windows 10 computer and > executed the following command: > {code:java} > mvn -DskipTests clean install{code} > Build failed! > !image-2021-10-17-22-18-16-616.png! > {code:java} > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run > (default) on project spark-core_2.12: An Ant BuildException has occured: > Execute failed: java.io.IOException: Cannot run program "bash" (in directory > "C:\bigdata\spark\core"): CreateProcess error=2{code} > It seems that the plugin: maven-antrun-plugin cannot run because of windows > no bash. > The following code comes from pom.xml in spark-core module. > {code:java} > > org.apache.maven.plugins > maven-antrun-plugin > > > generate-resources > > > > > > > > > > > > run > > > > > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-37030) Maven build failed in windows!
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17437168#comment-17437168 ] Shockang commented on SPARK-37030: -- [~hyukjin.kwon] Even if the community does not support Spark in windows, why no one cares about programmers who use windows ... > Maven build failed in windows! > -- > > Key: SPARK-37030 > URL: https://issues.apache.org/jira/browse/SPARK-37030 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.0 > Environment: OS: Windows 10 Professional > OS Version: 21H1 > Maven Version: 3.6.3 > >Reporter: Shockang >Priority: Minor > Fix For: 3.2.0 > > Attachments: image-2021-10-17-22-18-16-616.png > > > I pulled the latest Spark master code on my local windows 10 computer and > executed the following command: > {code:java} > mvn -DskipTests clean install{code} > Build failed! > !image-2021-10-17-22-18-16-616.png! > {code:java} > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run > (default) on project spark-core_2.12: An Ant BuildException has occured: > Execute failed: java.io.IOException: Cannot run program "bash" (in directory > "C:\bigdata\spark\core"): CreateProcess error=2{code} > It seems that the plugin: maven-antrun-plugin cannot run because of windows > no bash. > The following code comes from pom.xml in spark-core module. > {code:java} > > org.apache.maven.plugins > maven-antrun-plugin > > > generate-resources > > > > > > > > > > > > run > > > > > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-36853: - Attachment: image-2021-10-18-01-57-00-714.png > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > Attachments: image-2021-10-18-01-57-00-714.png, > spark_mvn_clean_install_skip_tests_in_windows.log > > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[88,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[100,25] > (naming) MethodName: Method name 'Once' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[110,25] > (naming) MethodName: Method name 'AvailableNow' must match pattern
[jira] [Commented] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429743#comment-17429743 ] Shockang commented on SPARK-36853: -- Due to the existence of the following issue: [SPARK-37030|https://issues.apache.org/jira/browse/SPARK-37030], maven build failed in windows! I annotated the doubtful code about bash and re executed the command: {code:java} mvn -DskipTests clean install {code} !image-2021-10-18-01-57-00-714.png! For your reference, I have attached the build log. [~hyukjin.kwon] Can this issue be split into multiple subtasks? Because there are 131 errors. > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > Attachments: image-2021-10-18-01-57-00-714.png, > spark_mvn_clean_install_skip_tests_in_windows.log > > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\strea
[jira] [Updated] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-36853: - Attachment: spark_mvn_clean_install_skip_tests_in_windows.log > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > Attachments: spark_mvn_clean_install_skip_tests_in_windows.log > > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[88,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[100,25] > (naming) MethodName: Method name 'Once' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[110,25] > (naming) MethodName: Method name 'AvailableNow' must match pattern > '^[a-z][a-z0-9][a-zA
[jira] [Updated] (SPARK-37030) Maven build failed in windows!
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-37030: - Description: I pulled the latest Spark master code on my local windows 10 computer and executed the following command: {code:java} mvn -DskipTests clean install{code} Build failed! !image-2021-10-17-22-18-16-616.png! {code:java} Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.12: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\bigdata\spark\core"): CreateProcess error=2{code} It seems that the plugin: maven-antrun-plugin cannot run because of windows no bash. The following code comes from pom.xml in spark-core module. {code:java} org.apache.maven.plugins maven-antrun-plugin generate-resources run {code} was: I pulled the latest Spark master code on my local windows 10 computer and executed the following command: {code:java} mvn -DskipTests clean install{code} Build failed! !image-2021-10-17-21-55-33-844.png! {code:java} Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.12: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\bigdata\spark\core"): CreateProcess error=2{code} It seems that the plugin: maven-antrun-plugin cannot run because of windows no bash. The following code comes from pom.xml in spark-core module. {code:java} org.apache.maven.plugins maven-antrun-plugin generate-resources run {code} > Maven build failed in windows! > -- > > Key: SPARK-37030 > URL: https://issues.apache.org/jira/browse/SPARK-37030 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.0 > Environment: OS: Windows 10 Professional > OS Version: 21H1 > Maven Version: 3.6.3 > >Reporter: Shockang >Priority: Minor > Fix For: 3.2.0 > > Attachments: image-2021-10-17-22-18-16-616.png > > > I pulled the latest Spark master code on my local windows 10 computer and > executed the following command: > {code:java} > mvn -DskipTests clean install{code} > Build failed! > !image-2021-10-17-22-18-16-616.png! > {code:java} > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run > (default) on project spark-core_2.12: An Ant BuildException has occured: > Execute failed: java.io.IOException: Cannot run program "bash" (in directory > "C:\bigdata\spark\core"): CreateProcess error=2{code} > It seems that the plugin: maven-antrun-plugin cannot run because of windows > no bash. > The following code comes from pom.xml in spark-core module. > {code:java} > > org.apache.maven.plugins > maven-antrun-plugin > > > generate-resources > > > > > > > > > > > > run > > > > > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-37030) Maven build failed in windows!
[ https://issues.apache.org/jira/browse/SPARK-37030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-37030: - Attachment: image-2021-10-17-22-18-16-616.png > Maven build failed in windows! > -- > > Key: SPARK-37030 > URL: https://issues.apache.org/jira/browse/SPARK-37030 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.2.0 > Environment: OS: Windows 10 Professional > OS Version: 21H1 > Maven Version: 3.6.3 > >Reporter: Shockang >Priority: Minor > Fix For: 3.2.0 > > Attachments: image-2021-10-17-22-18-16-616.png > > > I pulled the latest Spark master code on my local windows 10 computer and > executed the following command: > {code:java} > mvn -DskipTests clean install{code} > Build failed! > !image-2021-10-17-21-55-33-844.png! > {code:java} > Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run > (default) on project spark-core_2.12: An Ant BuildException has occured: > Execute failed: java.io.IOException: Cannot run program "bash" (in directory > "C:\bigdata\spark\core"): CreateProcess error=2{code} > It seems that the plugin: maven-antrun-plugin cannot run because of windows > no bash. > The following code comes from pom.xml in spark-core module. > {code:java} > > org.apache.maven.plugins > maven-antrun-plugin > > > generate-resources > > > > > > > > > > > > run > > > > > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-37030) Maven build failed in windows!
Shockang created SPARK-37030: Summary: Maven build failed in windows! Key: SPARK-37030 URL: https://issues.apache.org/jira/browse/SPARK-37030 Project: Spark Issue Type: Bug Components: Build Affects Versions: 3.2.0 Environment: OS: Windows 10 Professional OS Version: 21H1 Maven Version: 3.6.3 Reporter: Shockang Fix For: 3.2.0 I pulled the latest Spark master code on my local windows 10 computer and executed the following command: {code:java} mvn -DskipTests clean install{code} Build failed! !image-2021-10-17-21-55-33-844.png! {code:java} Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (default) on project spark-core_2.12: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "bash" (in directory "C:\bigdata\spark\core"): CreateProcess error=2{code} It seems that the plugin: maven-antrun-plugin cannot run because of windows no bash. The following code comes from pom.xml in spark-core module. {code:java} org.apache.maven.plugins maven-antrun-plugin generate-resources run {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420438#comment-17420438 ] Shockang commented on SPARK-36853: -- OK > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[88,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[100,25] > (naming) MethodName: Method name 'Once' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[110,25] > (naming) MethodName: Method name 'AvailableNow' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:
[jira] [Commented] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420207#comment-17420207 ] Shockang commented on SPARK-36853: -- [~abhinavofficial] Can you tell me how you caused this error? I think that if it is such an error, the general CI will report an error, and all the full tests will not pass. > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > Fix For: 3.3.0 > > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[88,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[100,25] > (naming) MethodName: Method name 'Once' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\s
[jira] [Comment Edited] (SPARK-36843) Add an iterator method to Dataset
[ https://issues.apache.org/jira/browse/SPARK-36843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420199#comment-17420199 ] Shockang edited comment on SPARK-36843 at 9/26/21, 3:53 AM: [~lxian2] You mean that one job collects all data and returns an iterator of byte array? was (Author: shockang): [~lxian2] You mean that a job collects all data and returns an iterator of byte array. > Add an iterator method to Dataset > - > > Key: SPARK-36843 > URL: https://issues.apache.org/jira/browse/SPARK-36843 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Li Xian >Priority: Minor > > The current org.apache.spark.sql.Dataset#toLocalIterator will submit multiple > jobs for multiple partitions. > In my case, I would like to collect all partition at once to save the job > scheduling cost and also has an iterator to save the memory on > deserialization (instead of deserialize all rows at once, I want only one row > is deserialized during the iteration) > . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36843) Add an iterator method to Dataset
[ https://issues.apache.org/jira/browse/SPARK-36843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420199#comment-17420199 ] Shockang commented on SPARK-36843: -- [~lxian2] You mean that a job collects all data and returns an iterator of byte array. > Add an iterator method to Dataset > - > > Key: SPARK-36843 > URL: https://issues.apache.org/jira/browse/SPARK-36843 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.3.0 >Reporter: Li Xian >Priority: Minor > > The current org.apache.spark.sql.Dataset#toLocalIterator will submit multiple > jobs for multiple partitions. > In my case, I would like to collect all partition at once to save the job > scheduling cost and also has an iterator to save the memory on > deserialization (instead of deserialize all rows at once, I want only one row > is deserialized during the iteration) > . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36853) Code failing on checkstyle
[ https://issues.apache.org/jira/browse/SPARK-36853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420194#comment-17420194 ] Shockang commented on SPARK-36853: -- working on this > Code failing on checkstyle > -- > > Key: SPARK-36853 > URL: https://issues.apache.org/jira/browse/SPARK-36853 > Project: Spark > Issue Type: Bug > Components: Build >Affects Versions: 3.3.0 >Reporter: Abhinav Kumar >Priority: Trivial > Fix For: 3.3.0 > > > There are more - just pasting sample > > [INFO] There are 32 errors reported by Checkstyle 8.43 with > dev/checkstyle.xml ruleset. > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF11.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 107). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF12.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 116). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 104). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF13.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 125). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 109). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF14.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 114). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF15.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 143). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 119). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF16.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 152). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 124). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF17.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 161). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 129). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF18.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 170). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 134). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF19.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 179). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 139). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF20.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 188). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 144). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF21.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 197). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[28] (sizes) > LineLength: Line is longer than 100 characters (found 149). > [ERROR] src\main\java\org\apache\spark\sql\api\java\UDF22.java:[29] (sizes) > LineLength: Line is longer than 100 characters (found 206). > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[44,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[60,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[75,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[88,25] > (naming) MethodName: Method name 'ProcessingTime' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[100,25] > (naming) MethodName: Method name 'Once' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\org\apache\spark\sql\streaming\Trigger.java:[110,25] > (naming) MethodName: Method name 'AvailableNow' must match pattern > '^[a-z][a-z0-9][a-zA-Z0-9_]*$'. > [ERROR] src\main\java\
[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415588#comment-17415588 ] Shockang commented on SPARK-36767: -- [~angerszhuuu] OK > ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT > - > > Key: SPARK-36767 > URL: https://issues.apache.org/jira/browse/SPARK-36767 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.3, 3.1.2, 3.2.1 >Reporter: angerszhu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36767) ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT
[ https://issues.apache.org/jira/browse/SPARK-36767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415583#comment-17415583 ] Shockang commented on SPARK-36767: -- working on this. > ArrayMin/ArrayMax/SortArray/ArraySort add comment and UT > - > > Key: SPARK-36767 > URL: https://issues.apache.org/jira/browse/SPARK-36767 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.0.3, 3.1.2, 3.2.1 >Reporter: angerszhu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Comment: was deleted (was: I'm working on this.) > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > {code:java} > Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > def createPlan(batchId: Long): WriteToDataSourceV2 = { > Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) > {code} > -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala- > {code:java} > Warning:Warning:line (703)a pure expression does nothing in statement > position; multiline expressions might require enclosing parentheses > q1 > {code} > -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala- > {code:java} > Warning:Warning:line (285)object typed in package scalalang is deprecated > (since 3.0.0): please use untyped builtin aggregate functions. > val aggregated = > inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala {code:java} Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead def createPlan(batchId: Long): WriteToDataSourceV2 = { Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) {code} sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala {code:java} Warning:Warning:line (703)a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses q1 {code} sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala {code:java} Warning:Warning:line (285)object typed in package scalalang is deprecated (since 3.0.0): please use untyped builtin aggregate functions. val aggregated = inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) {code} was: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala {code:java} Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead def createPlan(batchId: Long): WriteToDataSourceV2 = { Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) {code} -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala- {code:java} Warning:Warning:line (703)a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses q1 {code} -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala- {code:java} Warning:Warning:line (285)object typed in package scalalang is deprecated (since 3.0.0): please use untyped builtin aggregate functions. val aggregated = inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) {code} > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > {code:java} > Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > def createPlan(batchId: Long): WriteToDataSourceV2 = { > Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) > {code} > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala > {code:java} > Warning:Warning:line (703)a pure expression does nothing in statement > position; multiline expressions might require enclosing parentheses > q1 > {code} > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala > {code:java} > Warning:Warning:line (285)object typed in package scalalang is deprecated > (since 3.0.0): please use untyped builtin aggregate functions. > val aggregated = > inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415564#comment-17415564 ] Shockang commented on SPARK-30177: -- After careful inspection, I found that except for the last problem, everything else had been fixed. But the last one is originally a test for expired APIs. > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > -/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala- > -Warning:Warning:line (108)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) < 0.1)- > -Warning:Warning:line (135)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) == summary.trainingCost)- > -Warning:Warning:line (195)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -model.computeCost(dataset)- > -/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala- > -Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information.- > -jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS)- > -/sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java- > -Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java- > -Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations.- > -Information:Information:java: Recompile with -Xlint:unchecked for details.- > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Comment: was deleted (was: After careful inspection, I found that except for the last one, everything else had been fixed.) > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > -/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala- > -Warning:Warning:line (108)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) < 0.1)- > -Warning:Warning:line (135)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) == summary.trainingCost)- > -Warning:Warning:line (195)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -model.computeCost(dataset)- > -/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala- > -Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information.- > -jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS)- > -/sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java- > -Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java- > -Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations.- > -Information:Information:java: Recompile with -Xlint:unchecked for details.- > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Comment: was deleted (was: I'm working on this.) > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > -/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala- > -Warning:Warning:line (108)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) < 0.1)- > -Warning:Warning:line (135)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -assert(model.computeCost(dataset) == summary.trainingCost)- > -Warning:Warning:line (195)method computeCost in class BisectingKMeansModel > is deprecated (since 3.0.0): This method is deprecated and will be removed in > future versions. Use ClusteringEvaluator instead. You can also get the cost > on the training dataset in the summary.- > -model.computeCost(dataset)- > -/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala- > -Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information.- > -jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS)- > -/sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java- > -Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated- > -/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java- > -Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations.- > -Information:Information:java: Recompile with -Xlint:unchecked for details.- > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30175: - Description: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala {code:java} Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead def createPlan(batchId: Long): WriteToDataSourceV2 = { Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) {code} -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala- {code:java} Warning:Warning:line (703)a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses q1 {code} -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala- {code:java} Warning:Warning:line (285)object typed in package scalalang is deprecated (since 3.0.0): please use untyped builtin aggregate functions. val aggregated = inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) {code} was: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala {code:java} Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead def createPlan(batchId: Long): WriteToDataSourceV2 = { Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is deprecated (since 2.4.0): Use specific logical plans like AppendData instead WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) {code} sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala {code:java} Warning:Warning:line (703)a pure expression does nothing in statement position; multiline expressions might require enclosing parentheses q1 {code} sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala {code:java} Warning:Warning:line (285)object typed in package scalalang is deprecated (since 3.0.0): please use untyped builtin aggregate functions. val aggregated = inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) {code} > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > {code:java} > Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > def createPlan(batchId: Long): WriteToDataSourceV2 = { > Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) > {code} > -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala- > {code:java} > Warning:Warning:line (703)a pure expression does nothing in statement > position; multiline expressions might require enclosing parentheses > q1 > {code} > -sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala- > {code:java} > Warning:Warning:line (285)object typed in package scalalang is deprecated > (since 3.0.0): please use untyped builtin aggregate functions. > val aggregated = > inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shockang updated SPARK-30177: - Description: -/mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala- -Warning:Warning:line (108)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.- -assert(model.computeCost(dataset) < 0.1)- -Warning:Warning:line (135)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.- -assert(model.computeCost(dataset) == summary.trainingCost)- -Warning:Warning:line (195)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary.- -model.computeCost(dataset)- -/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala- -Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java enum Feature is deprecated: see corresponding Javadoc for more information.- -jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS)- -/sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java- -Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated- -Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated- -Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated- -Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated- -Warning:Warning:line (64)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated- -/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java- -Information:Information:java: /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java uses unchecked or unsafe operations.- -Information:Information:java: Recompile with -Xlint:unchecked for details.- /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java Warning:Warning:line (478)java: json(org.apache.spark.api.java.JavaRDD) in org.apache.spark.sql.DataFrameReader has been deprecated was: /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala Warning:Warning:line (108)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) < 0.1) Warning:Warning:line (135)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. assert(model.computeCost(dataset) == summary.trainingCost) Warning:Warning:line (195)method computeCost in class BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated and will be removed in future versions. Use ClusteringEvaluator instead. You can also get the cost on the training dataset in the summary. model.computeCost(dataset) /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java enum Feature is deprecated: see corresponding Javadoc for more information. jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java Warning:Warning:line (28)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (37)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (46)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been deprecated Warning:Warning:line (55)java: org.apache.spark.sql.expressions.javalang.typed in org.apache.spark.sql.expressions.javalang has been dep
[jira] [Commented] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415551#comment-17415551 ] Shockang commented on SPARK-30177: -- I'm working on this. > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala > Warning:Warning:line (108)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) < 0.1) > Warning:Warning:line (135)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) == summary.trainingCost) > Warning:Warning:line (195)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > model.computeCost(dataset) > > /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala > Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information. > jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) > /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java > Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations. > Information:Information:java: Recompile with -Xlint:unchecked for details. > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30177) Eliminate warnings: part 7
[ https://issues.apache.org/jira/browse/SPARK-30177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415550#comment-17415550 ] Shockang commented on SPARK-30177: -- After careful inspection, I found that except for the last one, everything else had been fixed. > Eliminate warnings: part 7 > -- > > Key: SPARK-30177 > URL: https://issues.apache.org/jira/browse/SPARK-30177 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: ABHISHEK KUMAR GUPTA >Priority: Minor > > /mllib/src/test/scala/org/apache/spark/ml/clustering/BisectingKMeansSuite.scala > Warning:Warning:line (108)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) < 0.1) > Warning:Warning:line (135)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > assert(model.computeCost(dataset) == summary.trainingCost) > Warning:Warning:line (195)method computeCost in class > BisectingKMeansModel is deprecated (since 3.0.0): This method is deprecated > and will be removed in future versions. Use ClusteringEvaluator instead. You > can also get the cost on the training dataset in the summary. > model.computeCost(dataset) > > /sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala > Warning:Warning:line (105)Java enum ALLOW_UNQUOTED_CONTROL_CHARS in Java > enum Feature is deprecated: see corresponding Javadoc for more information. > jsonFactory.enable(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS) > /sql/core/src/test/java/test/org/apache/spark/sql/Java8DatasetAggregatorSuite.java > Warning:Warning:line (28)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (37)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (46)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (55)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > Warning:Warning:line (64)java: > org.apache.spark.sql.expressions.javalang.typed in > org.apache.spark.sql.expressions.javalang has been deprecated > /sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > Information:Information:java: > /Users/maxim/proj/eliminate-warning/sql/core/src/test/java/test/org/apache/spark/sql/JavaTestUtils.java > uses unchecked or unsafe operations. > Information:Information:java: Recompile with -Xlint:unchecked for details. > /sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java > Warning:Warning:line (478)java: > json(org.apache.spark.api.java.JavaRDD) in > org.apache.spark.sql.DataFrameReader has been deprecated -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30175) Eliminate warnings: part 5
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415542#comment-17415542 ] Shockang commented on SPARK-30175: -- I'm working on this. > Eliminate warnings: part 5 > -- > > Key: SPARK-30175 > URL: https://issues.apache.org/jira/browse/SPARK-30175 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.1.0 >Reporter: jobit mathew >Priority: Minor > > sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/WriteToMicroBatchDataSource.scala > {code:java} > Warning:Warning:line (36)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > def createPlan(batchId: Long): WriteToDataSourceV2 = { > Warning:Warning:line (37)class WriteToDataSourceV2 in package v2 is > deprecated (since 2.4.0): Use specific logical plans like AppendData instead > WriteToDataSourceV2(new MicroBatchWrite(batchId, write), query) > {code} > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala > {code:java} > Warning:Warning:line (703)a pure expression does nothing in statement > position; multiline expressions might require enclosing parentheses > q1 > {code} > sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala > {code:java} > Warning:Warning:line (285)object typed in package scalalang is deprecated > (since 3.0.0): please use untyped builtin aggregate functions. > val aggregated = > inputData.toDS().groupByKey(_._1).agg(typed.sumLong(_._2)) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36328) HadoopRDD#getPartitions fetches FileSystem Delegation Token for every partition
[ https://issues.apache.org/jira/browse/SPARK-36328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17389960#comment-17389960 ] Shockang commented on SPARK-36328: -- I'm working on it. > HadoopRDD#getPartitions fetches FileSystem Delegation Token for every > partition > --- > > Key: SPARK-36328 > URL: https://issues.apache.org/jira/browse/SPARK-36328 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Priority: Major > > Spark Job creates a separate JobConf for every RDD (every hive table > partition) in HadoopRDD#getPartitions. > {code} > override def getPartitions: Array[Partition] = { > val jobConf = getJobConf() > // add the credentials here as this can be called before SparkContext > initialized > SparkHadoopUtil.get.addCredentials(jobConf) > {code} > Hadoop FileSystem fetches FileSystem Delegation Token and sets into the > Credentials which is part of JobConf. On further requests, will reuse the > token from the Credentials if already exists. > {code} >if (serviceName != null) { // fs has token, grab it > final Text service = new Text(serviceName); > Token token = credentials.getToken(service); > if (token == null) { > token = getDelegationToken(renewer); > if (token != null) { > tokens.add(token); > credentials.addToken(service, token); > } > } > } > {code} > But since Spark Job creates a new JobConf (which will have a new > Credentials) for every hive table partition, the token is not reused and gets > fetched for every partition. This is slowing down the query as each > delegation token has to go through KDC and SSL handshake on Secure Clusters. > *Improvement:* > Spark can add the credentials from previous JobConf into the new JobConf to > reuse the FileSystem Delegation Token similar to how the User Credentials are > added into JobConf after construction. > {code} > val jobConf = getJobConf() > // add the credentials here as this can be called before SparkContext > initialized > SparkHadoopUtil.get.addCredentials(jobConf) > {code} > *Repro* > {code} > beeline> > create table parttable (key char(1), value int) partitioned by (p int); > insert into table parttable partition(p=100) values ('d', 1), ('e', 2), ('f', > 3); > insert into table parttable partition(p=200) values ('d', 1), ('e', 2), ('f', > 3); > insert into table parttable partition(p=300) values ('d', 1), ('e', 2), ('f', > 3); > spark-sql> > select value, count(*) from parttable group by value > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36099) Group exception messages in core/util
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388107#comment-17388107 ] Shockang commented on SPARK-36099: -- It's good for you to wait a day. I was going to submit the PR tonight. Why is it so coincidental![~dc-heros] > Group exception messages in core/util > - > > Key: SPARK-36099 > URL: https://issues.apache.org/jira/browse/SPARK-36099 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: Allison Wang >Priority: Major > > 'core/src/main/scala/org/apache/spark/util' > || Filename || Count || > | AccumulatorV2.scala | 4 | > | ClosureCleaner.scala | 1 | > | DependencyUtils.scala| 1 | > | KeyLock.scala| 1 | > | ListenerBus.scala| 1 | > | NextIterator.scala | 1 | > | SerializableBuffer.scala | 2 | > | ThreadUtils.scala| 4 | > | Utils.scala | 16 | > 'core/src/main/scala/org/apache/spark/util/collection' > || Filename || Count || > | AppendOnlyMap.scala | 1 | > | CompactBuffer.scala | 1 | > | ImmutableBitSet.scala | 6 | > | MedianHeap.scala | 1 | > | OpenHashSet.scala | 2 | > 'core/src/main/scala/org/apache/spark/util/io' > || Filename|| Count || > | ChunkedByteBuffer.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/logging' > || Filename || Count || > | DriverLogger.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/random' > || Filename|| Count || > | RandomSampler.scala | 1 | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36099) Group exception messages in core/util
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388101#comment-17388101 ] Shockang commented on SPARK-36099: -- I’m sorry for my gaffe.[~dc-heros] > Group exception messages in core/util > - > > Key: SPARK-36099 > URL: https://issues.apache.org/jira/browse/SPARK-36099 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: Allison Wang >Priority: Major > > 'core/src/main/scala/org/apache/spark/util' > || Filename || Count || > | AccumulatorV2.scala | 4 | > | ClosureCleaner.scala | 1 | > | DependencyUtils.scala| 1 | > | KeyLock.scala| 1 | > | ListenerBus.scala| 1 | > | NextIterator.scala | 1 | > | SerializableBuffer.scala | 2 | > | ThreadUtils.scala| 4 | > | Utils.scala | 16 | > 'core/src/main/scala/org/apache/spark/util/collection' > || Filename || Count || > | AppendOnlyMap.scala | 1 | > | CompactBuffer.scala | 1 | > | ImmutableBitSet.scala | 6 | > | MedianHeap.scala | 1 | > | OpenHashSet.scala | 2 | > 'core/src/main/scala/org/apache/spark/util/io' > || Filename|| Count || > | ChunkedByteBuffer.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/logging' > || Filename || Count || > | DriverLogger.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/random' > || Filename|| Count || > | RandomSampler.scala | 1 | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36099) Group exception messages in core/util
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388095#comment-17388095 ] Shockang commented on SPARK-36099: -- I think you should ask my permission, or my time will be wasted.[~dc-heros] > Group exception messages in core/util > - > > Key: SPARK-36099 > URL: https://issues.apache.org/jira/browse/SPARK-36099 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: Allison Wang >Priority: Major > > 'core/src/main/scala/org/apache/spark/util' > || Filename || Count || > | AccumulatorV2.scala | 4 | > | ClosureCleaner.scala | 1 | > | DependencyUtils.scala| 1 | > | KeyLock.scala| 1 | > | ListenerBus.scala| 1 | > | NextIterator.scala | 1 | > | SerializableBuffer.scala | 2 | > | ThreadUtils.scala| 4 | > | Utils.scala | 16 | > 'core/src/main/scala/org/apache/spark/util/collection' > || Filename || Count || > | AppendOnlyMap.scala | 1 | > | CompactBuffer.scala | 1 | > | ImmutableBitSet.scala | 6 | > | MedianHeap.scala | 1 | > | OpenHashSet.scala | 2 | > 'core/src/main/scala/org/apache/spark/util/io' > || Filename|| Count || > | ChunkedByteBuffer.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/logging' > || Filename || Count || > | DriverLogger.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/random' > || Filename|| Count || > | RandomSampler.scala | 1 | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36099) Group exception messages in core/util
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388090#comment-17388090 ] Shockang commented on SPARK-36099: -- You should tell me in advance.My code has written more than 200 lines and is preparing to submit pr….[~dc-heros] > Group exception messages in core/util > - > > Key: SPARK-36099 > URL: https://issues.apache.org/jira/browse/SPARK-36099 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: Allison Wang >Priority: Major > > 'core/src/main/scala/org/apache/spark/util' > || Filename || Count || > | AccumulatorV2.scala | 4 | > | ClosureCleaner.scala | 1 | > | DependencyUtils.scala| 1 | > | KeyLock.scala| 1 | > | ListenerBus.scala| 1 | > | NextIterator.scala | 1 | > | SerializableBuffer.scala | 2 | > | ThreadUtils.scala| 4 | > | Utils.scala | 16 | > 'core/src/main/scala/org/apache/spark/util/collection' > || Filename || Count || > | AppendOnlyMap.scala | 1 | > | CompactBuffer.scala | 1 | > | ImmutableBitSet.scala | 6 | > | MedianHeap.scala | 1 | > | OpenHashSet.scala | 2 | > 'core/src/main/scala/org/apache/spark/util/io' > || Filename|| Count || > | ChunkedByteBuffer.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/logging' > || Filename || Count || > | DriverLogger.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/random' > || Filename|| Count || > | RandomSampler.scala | 1 | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36099) Group exception messages in core/util
[ https://issues.apache.org/jira/browse/SPARK-36099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381874#comment-17381874 ] Shockang commented on SPARK-36099: -- [~allisonwang-db] Could I working on this issue? > Group exception messages in core/util > - > > Key: SPARK-36099 > URL: https://issues.apache.org/jira/browse/SPARK-36099 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.3.0 >Reporter: Allison Wang >Priority: Major > > 'core/src/main/scala/org/apache/spark/util' > || Filename || Count || > | AccumulatorV2.scala | 4 | > | ClosureCleaner.scala | 1 | > | DependencyUtils.scala| 1 | > | KeyLock.scala| 1 | > | ListenerBus.scala| 1 | > | NextIterator.scala | 1 | > | SerializableBuffer.scala | 2 | > | ThreadUtils.scala| 4 | > | Utils.scala | 16 | > 'core/src/main/scala/org/apache/spark/util/collection' > || Filename || Count || > | AppendOnlyMap.scala | 1 | > | CompactBuffer.scala | 1 | > | ImmutableBitSet.scala | 6 | > | MedianHeap.scala | 1 | > | OpenHashSet.scala | 2 | > 'core/src/main/scala/org/apache/spark/util/io' > || Filename|| Count || > | ChunkedByteBuffer.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/logging' > || Filename || Count || > | DriverLogger.scala | 1 | > 'core/src/main/scala/org/apache/spark/util/random' > || Filename|| Count || > | RandomSampler.scala | 1 | -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35508) job group and description do not apply on broadcasts
[ https://issues.apache.org/jira/browse/SPARK-35508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381790#comment-17381790 ] Shockang commented on SPARK-35508: -- Put this issue on hold and wait for the complete plans of multiple job groups to fix it. > job group and description do not apply on broadcasts > > > Key: SPARK-35508 > URL: https://issues.apache.org/jira/browse/SPARK-35508 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0 >Reporter: Lior Chaga >Priority: Minor > Attachments: spark2-image.png, spark3-image.png > > > Given the following code: > {code:java} > SparkContext context = new SparkContext("local", "test"); > SparkSession session = new SparkSession(context); > List strings = Lists.newArrayList("a", "b", "c"); > List otherString = Lists.newArrayList( "b", "c", "d"); > Dataset broadcastedDf = session.createDataset(strings, > Encoders.STRING()).toDF(); > Dataset dataframe = session.createDataset(otherString, > Encoders.STRING()).toDF(); > context.setJobGroup("my group", "my job", false); > dataframe.join(broadcast(broadcastedDf), "value").count(); > {code} > Job group and description do not apply on broadcasted dataframe. > With spark 2.x, broadcast creation is given the same job description as the > query itself. > This seems to be broken with spark 3.x > See attached images > !spark3-image.png! !spark2-image.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35508) job group and description do not apply on broadcasts
[ https://issues.apache.org/jira/browse/SPARK-35508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17378483#comment-17378483 ] Shockang commented on SPARK-35508: -- It seems that this bug comes from this PR: [https://github.com/apache/spark/pull/24595] , which will override the settings of job group and job description in the user code. Let me fix this issue. > job group and description do not apply on broadcasts > > > Key: SPARK-35508 > URL: https://issues.apache.org/jira/browse/SPARK-35508 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.0, 3.1.0 >Reporter: Lior Chaga >Priority: Minor > Attachments: spark2-image.png, spark3-image.png > > > Given the following code: > {code:java} > SparkContext context = new SparkContext("local", "test"); > SparkSession session = new SparkSession(context); > List strings = Lists.newArrayList("a", "b", "c"); > List otherString = Lists.newArrayList( "b", "c", "d"); > Dataset broadcastedDf = session.createDataset(strings, > Encoders.STRING()).toDF(); > Dataset dataframe = session.createDataset(otherString, > Encoders.STRING()).toDF(); > context.setJobGroup("my group", "my job", false); > dataframe.join(broadcast(broadcastedDf), "value").count(); > {code} > Job group and description do not apply on broadcasted dataframe. > With spark 2.x, broadcast creation is given the same job description as the > query itself. > This seems to be broken with spark 3.x > See attached images > !spark3-image.png! !spark2-image.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org