LuciferYang commented on code in PR #40654:
URL: https://github.com/apache/spark/pull/40654#discussion_r1160529800
##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -863,6 +866,46 @@ class ClientE2ETestSuite extends RemoteSparkSession with
SQLHelper {
}.getMessage
assert(message.contains("PARSE_SYNTAX_ERROR"))
}
+
+ test("protobuf functions") {
+ assert(IntegrationTestUtils.isSparkProtobufJarAvailable)
+ // scalastyle:off line.size.limit
+ // If `common.desc` needs to be updated, execute the following command to
regenerate it:
+ // 1. cd connector/connect/common/src/main/protobuf/spark/connect
+ // 2. protoc --include_imports
--descriptor_set_out=../../../../test/resources/protobuf-tests/common.desc
common.proto
+ // scalastyle:on line.size.limit
+ val descPath = {
Review Comment:
At first, I wrote the following case in `PlanGenerationTestSuite ` to test
`from_protobuf` with `descFilePath`:
```
test("from_protobuf messageClassName options descFilePath") {
binary.select(
pbFn.from_protobuf(
data = fn.col("bytes"),
messageName = "StorageLevel",
descFilePath = "file://fake.desc"))
}
```
but when I run `ProtoToParsedPlanTestSuite `, there is following errors:
```
[info] - from_protobuf_messageClassName_options_descFilePath *** FAILED ***
(11 milliseconds)
[info] org.apache.spark.sql.AnalysisException:
[PROTOBUF_DESCRIPTOR_FILE_NOT_FOUND] Error reading Protobuf descriptor file at
path: file://fake.desc.
[info] at
org.apache.spark.sql.errors.QueryCompilationErrors$.cannotFindDescriptorFileError(QueryCompilationErrors.scala:3375)
[info] at
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.parseFileDescriptorSet(ProtobufUtils.scala:234)
[info] at
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:212)
[info] at
org.apache.spark.sql.protobuf.utils.ProtobufUtils$.buildDescriptor(ProtobufUtils.scala:149)
[info] at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor$lzycompute(ProtobufDataToCatalyst.scala:57)
[info] at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.messageDescriptor(ProtobufDataToCatalyst.scala:56)
[info] at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType$lzycompute(ProtobufDataToCatalyst.scala:42)
[info] at
org.apache.spark.sql.protobuf.ProtobufDataToCatalyst.dataType(ProtobufDataToCatalyst.scala:41)
...
[info] Cause: java.io.FileNotFoundException: file:/fake.desc (No such file
or directory)
...
```
`ProtobufDataToCatalyst.dataType` need the `descFilePath` file to resolve
`dataType`, so I change the case as follows:
```
private lazy val descPath = {
getWorkspaceFilePath(
"connector",
"connect",
"common",
"src",
"test",
"resources",
"protobuf-tests",
"common.desc")
test("from_protobuf messageClassName options descFilePath") {
binary.select(
pbFn.from_protobuf(
data = fn.col("bytes"),
messageName = "StorageLevel",
descFilePath = descPath.toFile.getPath))
}
```
the test will success, but the contens of `.explain` as follows:
```
Project [from_protobuf(bytes#0, StorageLevel,
Some(/Users/yangjie01/SourceCode/git/spark-mine-sbt/connector/connect/common/src/test/resources/protobuf-tests/common.desc))
AS from_protobuf(bytes)#0]
+- LocalRelation <empty>, [id#0L, bytes#0]
```
the `descFilePath` is a absolute path, any better suggestions? @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
