Should I use Dstream or Structured Stream to transfer data from source to sink and then back from sink to source?

[kant kodali]

Hi All,

I am trying to read data from kafka, insert into Mongo and read from mongo
and insert back into Kafka. I went with structured stream approach first
however I believe I am making some naiver error because my map operations
are not getting invoked.

The pseudo code looks like this

DataSet<String> resultDataSet = jsonDataset.mapPartitions(
insertIntoMongo).mapPartitions(readFromMongo);

StreamingQuery query =
resultDataSet.trigger(ProcesingTime(1000)).format("kafka").start();

query.awaitTermination();

The mapPartitions in this code is not getting executed. Is this because I
am not calling any action on my streaming dataset? In the Dstream case, I
used to call forEachRDD and it worked well. so how do I do this using
structured streaming?

Thanks!