[ 
https://issues.apache.org/jira/browse/SPARK-45029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yang Jie updated SPARK-45029:
-----------------------------
    Description: 
The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module 
when testing the connect module. This allows `mvn clean install` and `mvn 
package test` to successfully pass tests. But if `mvn clean test` is executed 
directly, an error in `package org.sparkproject.spark_protobuf.protobuf does 
not exist` will occur. This is because `mvn clean test` directly uses the 
classes file of the `spark-protobuf` module for testing, without the 'package', 
hence it does not `shade` and `relocate` protobuf. 

On the other hand, as reported by @MaxGekk, SPARK-43646 breaks the usability of 
importing Spark as a Maven project into IDEA. 

Therefore, we revert the changes of SPARK-43646 in 
[https://github.com/apache/spark/pull/42746]. 

In summary, when using Maven to test the connect module:
 * sometimes the shaded spark-protobuf jar will be used, such as 'mvn package 
test',
 * sometimes the unshaded classes directory will be used, such as 'mvn clean 
test'.

Therefore, there is no perfect solution to pass both tests, so 'from_protobuf 
messageClassName' and 'from_protobuf messageClassName' in 
'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when 
we find a better solution.

  was:
The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` module 
when testing the connect module. This allows `mvn clean install` and `mvn 
package test` to successfully pass tests. But if `mvn clean test` is executed 
directly, an error in `package org.sparkproject.spark_protobuf.protobuf does 
not exist` will occur. This is because `mvn clean test` directly uses the 
classes file of the `spark-protobuf` module for testing, without the 'package', 
hence it does not `shade` and `relocate` protobuf. 

On the other hand, as reported by @MaxGekk, SPARK-43646 breaks the usability of 
importing Spark as a Maven project into IDEA. 

Therefore, we revert the changes of SPARK-43646 in 
https://github.com/apache/spark/pull/42746. 

In summary, when using Maven to test the connect module:
 * sometimes the shaded spark-protobuf jar will be used, such as 'mvn package 
test',
 * sometimes the unshaded classes directory will be used, such as 'mvn clean 
test'.

Therefore, there is no perfect solution to pass both tests, so 'from_protobuf 
messageClassName' and 'from_protobuf messageClassName' in 
'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled when 
we find a solution.


> Ignore `from_protobuf messageClassName` and `from_protobuf messageClassName 
> options` in `PlanGenerationTestSuite`
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-45029
>                 URL: https://issues.apache.org/jira/browse/SPARK-45029
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect, Tests
>    Affects Versions: 3.5.0, 4.0.0
>            Reporter: Yang Jie
>            Priority: Major
>
> The SPARK-43646 makes both Maven and SBT use the shaded `spark-protobuf` 
> module when testing the connect module. This allows `mvn clean install` and 
> `mvn package test` to successfully pass tests. But if `mvn clean test` is 
> executed directly, an error in `package 
> org.sparkproject.spark_protobuf.protobuf does not exist` will occur. This is 
> because `mvn clean test` directly uses the classes file of the 
> `spark-protobuf` module for testing, without the 'package', hence it does not 
> `shade` and `relocate` protobuf. 
> On the other hand, as reported by @MaxGekk, SPARK-43646 breaks the usability 
> of importing Spark as a Maven project into IDEA. 
> Therefore, we revert the changes of SPARK-43646 in 
> [https://github.com/apache/spark/pull/42746]. 
> In summary, when using Maven to test the connect module:
>  * sometimes the shaded spark-protobuf jar will be used, such as 'mvn package 
> test',
>  * sometimes the unshaded classes directory will be used, such as 'mvn clean 
> test'.
> Therefore, there is no perfect solution to pass both tests, so 'from_protobuf 
> messageClassName' and 'from_protobuf messageClassName' in 
> 'PlanGenerationTestSuite' are ignored for now, and they will be re-enabled 
> when we find a better solution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to