sorry meant to say: we know when we upgrade that we might run into minor inconveniences that are completely our own doing/fault.
also, with yarn it has become really easy to run against an exact spark version of our choosing, since there is no longer such a thing as a centrally managed spark distro on the cluster. On Thu, Mar 30, 2017 at 1:34 PM, Koert Kuipers <ko...@tresata.com> wrote: > i agree with that. > > we work within that assumption. we compile and run against a single exact > spark version. we know when we upgrade that we might run into minor > inconveniences that our completely our own doing/fault. the trade off has > been totally worth it to me. > > > On Thu, Mar 30, 2017 at 1:20 PM, Michael Armbrust <mich...@databricks.com> > wrote: > >> I think really the right way to think about things that are marked >> private is, "this may disappear or change in a future minor release". If >> you are okay with that, working about the visibility restrictions is >> reasonable. >> >> On Thu, Mar 30, 2017 at 5:52 AM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> I stopped asking long time ago why things are private in spark... I >>> mean... The conversion between ml and mllib vectors is private... the >>> conversion between spark vector and breeze used to be (or still is?) >>> private. it just goes on. Lots of useful stuff is private[SQL]. >>> >>> Luckily there are simple ways to get around these visibility restrictions >>> >>> On Mar 29, 2017 22:57, "Ryan" <ryan.hd....@gmail.com> wrote: >>> >>>> I'm writing a transformer and the input column is vector type(which is >>>> the output column from other transformer). But as the VectorUDT is private, >>>> how could I check/transform schema for the vector column? >>>> >>> >> >
