Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Reynold Xin
I used to, before each release during the RC phase, go through every single doc page to make sure we don’t unintentionally leave things public. I no longer have time to do that unfortunately. I find that very useful because I always catch some mistakes through organic development. > On Nov 13,

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Wenchen Fan
> Could you clarify what you mean here? Mima has some known limitations such as not handling "private[blah]" very well Yes that's what I mean. What I want to know here is, which classes/methods we expect them to be private. I think things marked as "private[blabla]" are expected to be private

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Marcelo Vanzin
On Tue, Nov 13, 2018 at 6:26 PM Wenchen Fan wrote: > Recently I updated the MiMa exclusion rules, and found MiMa tracks some > private classes/methods unexpectedly. Could you clarify what you mean here? Mima has some known limitations such as not handling "private[blah]" very well (because that

Re: which classes/methods are considered as private in Spark?

2018-11-13 Thread Sean Owen
You should find that 'surprisingly public' classes are there because of language technicalities. For example DummySerializerInstance is public because it's a Java class, and can't be used outside its package otherwise. LIkewise I think MiMa just looks at bytecode, and private[spark] classes are

which classes/methods are considered as private in Spark?

2018-11-13 Thread Wenchen Fan
Hi all, Recently I updated the MiMa exclusion rules, and found MiMa tracks some private classes/methods unexpectedly. Note that, "private" here means that, we have no guarantee about compatibility. We don't provide documents and users need to take the risk when using them. In the API document,