Hi Owen, it's omitted from what I pasted but I'm using spark 2.4.4 on both.
On Mon, Aug 17, 2020 at 4:37 PM Sean Owen <sro...@gmail.com> wrote: > Looks like you are building vs Spark 3 and running on Spark 2, or > something along those lines. > > On Mon, Aug 17, 2020 at 4:02 AM Aviad Klein > <aviad.kl...@fundbox.com.invalid> wrote: > >> Hi, I've referenced the same problem on stack overflow and can't seem to >> find answers. >> >> I have custom spark pipelinestages written in scala that are specific to >> my organization. They work well on scala-spark. >> >> However, when I try to wrap them as shown here, so I can use them in >> pyspark, I get weird stuff that's happening. mostly around constructors of >> the java objects >> >> please refer to the stack overflow question >> <https://stackoverflow.com/questions/63439162/referencing-a-scala-java-pipelinestage-from-pyspark-constructor-issues-with-ha>, >> it's the most documented. >> >> Thanks, any help is appreciated >> >> -- >> *Aviad Klein* >> Director of Data Science >> >> >> -- *Aviad Klein* Director of Data Science
