grundprinzip commented on code in PR #38347:
URL: https://github.com/apache/spark/pull/38347#discussion_r1003252350
##########
connector/connect/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -207,3 +208,23 @@ message Sample {
int64 seed = 1;
}
}
+
+// Relation of type [[Range]] that generates a sequence of integers.
+message Range {
+ // Optional. Default value = 0
+ int32 start = 1;
+ int32 end = 2;
+ // Optional. Default value = 1
+ Step step = 3;
+ // Optional. Default value is assigned by 1) SQL conf
"spark.sql.leafNodeDefaultParallelism" if
+ // it is set, or 2) spark default parallelism.
+ NumPartitions num_partitions = 4;
Review Comment:
Is this really the best way to express the optionality?
##########
connector/connect/src/main/scala/org/apache/spark/sql/connect/dsl/package.scala:
##########
@@ -175,6 +178,28 @@ package object dsl {
}
object plans { // scalastyle:ignore
+ implicit class DslMockRemoteSession(val session: MockRemoteSession) {
+ def range(
+ start: Option[Int],
+ end: Int,
+ step: Option[Int],
+ numPartitions: Option[Int]): Relation = {
+ val range = proto.Range.newBuilder()
Review Comment:
I've been explicitly requesting this a couple of times already, as a coding
style to always prefix the proto generated classes with their `proto.` prefix.
I know it uses a little bit more horizontal space, but at the same time it
makes always clear where this particular element comes from which is
tremendously useful because we're consistently using the different types
between the catalyst API and Spark Connect in the same code paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
