Great! Have a great coding weekend! On Sat, 27 Aug 2022 at 10:55 AM MatthewJ Sanyoto <[email protected]> wrote:
> Hi, > > It is already fixed. Thank you all for the advices. The main problem was > the package containing the mapping and therefore Wayang doesn't understand > the output package for spark. > > Best regards, > Matthew > > On Fri, Aug 26, 2022, 13:02 MatthewJ Sanyoto <[email protected]> > wrote: > > > Hi, > > > > The new operator is literally MapOperator under a different name. To test > > creating a new operator. I literally copied the code of a mapoperator > but I > > only changed the name. All the functionality is the same and I only > copied > > it. > > > > However, It is possible and I' ll check the input and output channel. > > > > Best regards, > > Matthew > > > > On Fri, Aug 26, 2022, 12:22 Zoi Kaoudi <[email protected]> wrote: > > > >> It could also be a problem with the input or output channels of your > >> operator and then the enumerator cannot concatenate the partial plans. > >> Are you using the RddChannel or something else? > >> Best > >> -- > >> Zoi > >> > >> Στις Παρασκευή 26 Αυγούστου 2022, 10:17:39 π.μ. CEST, ο χρήστης > >> MatthewJ Sanyoto <[email protected]> έγραψε: > >> > >> Hi, > >> > >> I have already register the new mapping in the mappings Spark Platform > and > >> the mapping will map the new operator with the new Spark Operator. > >> > >> It could also be that Spark doesn't recognize the implementation or the > >> mapping isn't done correctly from my side, however if it is so then I > >> should got an error no plan to enumerate. However I believe Spark > >> recognize > >> the operator but the planenumerator doesn't know what to do with it? > This > >> is my guess. > >> > >> Best regards, > >> Matthew > >> > >> On Fri, Aug 26, 2022, 10:02 Zoi Kaoudi <[email protected]> > wrote: > >> > >> > Hi Matthew, > >> > did you register the new mapping? There should be a class or structure > >> > that lists all the mappings available. > >> > Best > >> > -- > >> > Zoi > >> > > >> > Στις Παρασκευή 26 Αυγούστου 2022, 09:20:24 π.μ. CEST, ο χρήστης > jorge > >> > Arnulfo Quiané Ruiz <[email protected]> έγραψε: > >> > > >> > I see. It seems the mapping is not being done correctly as wayang is > >> not > >> > being able to find a Spark implementation for your wayang operator. > >> > > >> > I could walk the code with you early next week. In the meantime, > perhaps > >> > Bertty or Rodrigo had a similar issue before, @Bertty and @Rodrigo did > >> you > >> > face a similar problem before? > >> > > >> > — > >> > Jorge > >> > > >> > On Thu, 25 Aug 2022 at 9:00 PM MatthewJ Sanyoto < > >> [email protected] > >> > > > >> > wrote: > >> > > >> > > Hi, > >> > > > >> > > So I created a new operator MapHackItOperator in the wayang basic > >> modules > >> > > operator. This functions essentially the same as MapOperator in the > >> same > >> > > module. > >> > > > >> > > Since wayang need a platform to execute this operator and to test, > in > >> the > >> > > wayang platform java modules I updated the Mapping and add > >> > MapHackitMapping > >> > > and JavaMapHackItOperator , this is the translation part as well as > >> the > >> > > execution, it works with Java Plugin. > >> > > > >> > > The problem is for only SparkPlugin I updated the wayang platform > >> spark > >> > > modules for the SparkPlugin so I created Mapping, MapHackItMapping > and > >> > > SparkMapHackItOperator that extends to MapHackItOperator with the > same > >> > code > >> > > as the SparkMap to translate it into SparkOperator, I think the > >> > translation > >> > > part is working but Wayang don't know how to enumerate the plan > >> according > >> > > to the exception or don't know where to output the results? correct > >> me if > >> > > I'm wrong. > >> > > > >> > > I hope it doesn't make it more confusing. > >> > > > >> > > Best regards, > >> > > Matthew > >> > > > >> > > On Thu, Aug 25, 2022, 20:45 jorge Arnulfo Quiané Ruiz < > >> > [email protected] > >> > > > > >> > > wrote: > >> > > > >> > > > Hello Matthew, > >> > > > > >> > > > To better understand the problem, are you trying to copy the > >> > > > JavaMapOperator into a Spark operator? Or you are trying to copy a > >> > > > SparkOperator into a new spark operator such as you did for Java? > >> > > > > >> > > > Best, > >> > > > Jorge > >> > > > > >> > > > On Thu, 25 Aug 2022 at 5:56 PM MatthewJ Sanyoto < > >> > > [email protected] > >> > > > > > >> > > > wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I have a problem regarding implementing a new custom operator > >> using > >> > > > Spark. > >> > > > > So I was trying to map a new operator from Wayang Basic operator > >> to > >> > the > >> > > > > Spark Platform to execute in Spark. > >> > > > > I tried using the Java Platform (only Java Plugin) and just > copied > >> > the > >> > > > > code(MapOperator, Mapping, JavaMapOperator and MapMapping) but > >> with > >> > > > > different operators name to test and it worked but not for Spark > >> > > > Platform. > >> > > > > (only Spark Plugin) > >> > > > > > >> > > > > Maybe someone here could please explain to me why I got this > >> error. > >> > > > > Caused by: org.apache.wayang.core.api.exception.WayangException: > >> No > >> > > > > implementations that concatenate out@Alternative[2x > >> > > > > ~Alternative[[MapHackIt[1->1, id=3c947bc5]]], 1e683a3e] with > >> > > > > [in@Alternative[2x > >> > > > > ~Alternative[[LocalCallbackSink[1->0, id=69fb6037]]], > 6babf3bf]]. > >> > > > > > >> > > > > Maybe I am missing out on something for Spark? > >> > > > > > >> > > > > Best Regards, > >> > > > > Matthew > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
