twosom commented on code in PR #34253:
URL: https://github.com/apache/beam/pull/34253#discussion_r1991760239
##########
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkStreamingPortablePipelineTranslator.java:
##########
@@ -322,13 +342,21 @@ private static <T> void translateFlatten(
dStreams.add(unboundedDataset.getDStream());
} else {
// create a single RDD stream.
- final SingleEmitInputDStream<WindowedValue<T>>
singleEmitInputDStream =
- new SingleEmitInputDStream<WindowedValue<T>>(
- context.getStreamingContext().ssc(), ((BoundedDataset)
dataset).getRDD().rdd());
- final JavaDStream<WindowedValue<T>> dStream =
- JavaDStream.fromDStream(
- singleEmitInputDStream,
JavaSparkContext$.MODULE$.fakeClassTag());
-
+ JavaDStream<WindowedValue<T>> dStream;
+ if (context.getSparkContext().version().startsWith("3")) {
+ Queue<JavaRDD<WindowedValue<T>>> q = new LinkedBlockingQueue<>();
+ q.offer(((BoundedDataset) dataset).getRDD());
+ // TODO (https://github.com/apache/beam/issues/20426): this is not
recoverable from
Review Comment:
I should have been more thorough when working on the PR, and I sincerely
apologize for not doing so. I would also like to express my gratitude to
@Amar3tto for their support and assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
