In the case of wayang you have this Reader in the api https://github.com/apache/incubator-wayang/blob/develop/wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/PlanBuilder.scala#L115-L121
Writer in the API https://github.com/apache/incubator-wayang/blob/develop/wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/DataQuanta.scala#L799-L825 In both cases, the writer will be implemented by the platform in the case of flink is as follow writer part https://github.com/apache/incubator-wayang/blob/develop/wayang-platforms/wayang-flink/src/main/java/org/apache/wayang/flink/operators/FlinkObjectFileSink.java#L80-L82 reader part https://github.com/apache/incubator-wayang/blob/develop/wayang-platforms/wayang-flink/src/main/java/org/apache/wayang/flink/operators/FlinkObjectFileSource.java#L89-L103 Also have the implementation on Apache Spark and Stream Java. If you think the code that you show at https://github.com/emmalanguage/emma/blob/master/emma-flink/src/main/scala/org/emmalanguage/api/flink/FlinkOps.scala#L76-L92 is similar to the one that is implemented on the Apache Flink platform. But if you need to read/write in binary it is also possible to create an operator that performs that. Also we can add more options to the operator to perform some behavior in the case of working with binaries. Let us know what do you think or if you need more explain :D Best regards, Bertty On Tue, Sep 28, 2021 at 2:00 PM Gábor E. Gévay <[email protected]> wrote: > Hello, > > I’m working on the Emma integration, and I would need to write a > generic DataQuanta to a temporary file, and then read it back later. > What would be the best way to do this? It’s not trivial because I > don’t know the concrete type of the DataQuanta, i.e., I’m just working > with DataQuanta[A]. (I have a ClassTag for A.) > > For example, the same functionality is achieved when Emma compiles to > Flink by writing and reading the DataSet[A] in a binary format with > the serializer that Flink has for A: > > https://github.com/emmalanguage/emma/blob/master/emma-flink/src/main/scala/org/emmalanguage/api/flink/FlinkOps.scala#L76-L92 > > At first, I thought that the ObjectFileSinks would help, but now I’m > not sure. The ObjectFileSinks seem to be available only at the backend > level. > > Best, > Gábor >
