Hi, It is already fixed. Thank you all for the advices. The main problem was the package containing the mapping and therefore Wayang doesn't understand the output package for spark.
Best regards, Matthew On Fri, Aug 26, 2022, 13:02 MatthewJ Sanyoto <[email protected]> wrote: > Hi, > > The new operator is literally MapOperator under a different name. To test > creating a new operator. I literally copied the code of a mapoperator but I > only changed the name. All the functionality is the same and I only copied > it. > > However, It is possible and I' ll check the input and output channel. > > Best regards, > Matthew > > On Fri, Aug 26, 2022, 12:22 Zoi Kaoudi <[email protected]> wrote: > >> It could also be a problem with the input or output channels of your >> operator and then the enumerator cannot concatenate the partial plans. >> Are you using the RddChannel or something else? >> Best >> -- >> Zoi >> >> Στις Παρασκευή 26 Αυγούστου 2022, 10:17:39 π.μ. CEST, ο χρήστης >> MatthewJ Sanyoto <[email protected]> έγραψε: >> >> Hi, >> >> I have already register the new mapping in the mappings Spark Platform and >> the mapping will map the new operator with the new Spark Operator. >> >> It could also be that Spark doesn't recognize the implementation or the >> mapping isn't done correctly from my side, however if it is so then I >> should got an error no plan to enumerate. However I believe Spark >> recognize >> the operator but the planenumerator doesn't know what to do with it? This >> is my guess. >> >> Best regards, >> Matthew >> >> On Fri, Aug 26, 2022, 10:02 Zoi Kaoudi <[email protected]> wrote: >> >> > Hi Matthew, >> > did you register the new mapping? There should be a class or structure >> > that lists all the mappings available. >> > Best >> > -- >> > Zoi >> > >> > Στις Παρασκευή 26 Αυγούστου 2022, 09:20:24 π.μ. CEST, ο χρήστης jorge >> > Arnulfo Quiané Ruiz <[email protected]> έγραψε: >> > >> > I see. It seems the mapping is not being done correctly as wayang is >> not >> > being able to find a Spark implementation for your wayang operator. >> > >> > I could walk the code with you early next week. In the meantime, perhaps >> > Bertty or Rodrigo had a similar issue before, @Bertty and @Rodrigo did >> you >> > face a similar problem before? >> > >> > — >> > Jorge >> > >> > On Thu, 25 Aug 2022 at 9:00 PM MatthewJ Sanyoto < >> [email protected] >> > > >> > wrote: >> > >> > > Hi, >> > > >> > > So I created a new operator MapHackItOperator in the wayang basic >> modules >> > > operator. This functions essentially the same as MapOperator in the >> same >> > > module. >> > > >> > > Since wayang need a platform to execute this operator and to test, in >> the >> > > wayang platform java modules I updated the Mapping and add >> > MapHackitMapping >> > > and JavaMapHackItOperator , this is the translation part as well as >> the >> > > execution, it works with Java Plugin. >> > > >> > > The problem is for only SparkPlugin I updated the wayang platform >> spark >> > > modules for the SparkPlugin so I created Mapping, MapHackItMapping and >> > > SparkMapHackItOperator that extends to MapHackItOperator with the same >> > code >> > > as the SparkMap to translate it into SparkOperator, I think the >> > translation >> > > part is working but Wayang don't know how to enumerate the plan >> according >> > > to the exception or don't know where to output the results? correct >> me if >> > > I'm wrong. >> > > >> > > I hope it doesn't make it more confusing. >> > > >> > > Best regards, >> > > Matthew >> > > >> > > On Thu, Aug 25, 2022, 20:45 jorge Arnulfo Quiané Ruiz < >> > [email protected] >> > > > >> > > wrote: >> > > >> > > > Hello Matthew, >> > > > >> > > > To better understand the problem, are you trying to copy the >> > > > JavaMapOperator into a Spark operator? Or you are trying to copy a >> > > > SparkOperator into a new spark operator such as you did for Java? >> > > > >> > > > Best, >> > > > Jorge >> > > > >> > > > On Thu, 25 Aug 2022 at 5:56 PM MatthewJ Sanyoto < >> > > [email protected] >> > > > > >> > > > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > I have a problem regarding implementing a new custom operator >> using >> > > > Spark. >> > > > > So I was trying to map a new operator from Wayang Basic operator >> to >> > the >> > > > > Spark Platform to execute in Spark. >> > > > > I tried using the Java Platform (only Java Plugin) and just copied >> > the >> > > > > code(MapOperator, Mapping, JavaMapOperator and MapMapping) but >> with >> > > > > different operators name to test and it worked but not for Spark >> > > > Platform. >> > > > > (only Spark Plugin) >> > > > > >> > > > > Maybe someone here could please explain to me why I got this >> error. >> > > > > Caused by: org.apache.wayang.core.api.exception.WayangException: >> No >> > > > > implementations that concatenate out@Alternative[2x >> > > > > ~Alternative[[MapHackIt[1->1, id=3c947bc5]]], 1e683a3e] with >> > > > > [in@Alternative[2x >> > > > > ~Alternative[[LocalCallbackSink[1->0, id=69fb6037]]], 6babf3bf]]. >> > > > > >> > > > > Maybe I am missing out on something for Spark? >> > > > > >> > > > > Best Regards, >> > > > > Matthew >> > > > > >> > > > >> > > >> > >> > >
