Re: Why Expression.deterministic method and Nondeterministic trait?

[Jacek Laskowski]

Hi Reynold,

That's my point indeed. If non-determinism is a trait/property of a
class (objects really at runtime) it's by definition part of a trait
and only Nondeterministic expressions are...well...non-deterministic.

I'm yet to review the code regarding the trait, but I suspect that
there are places where you check whether an expression is
deterministic by the method not the trait. If so, there's this
ambiguity I'm talking about.

Anyway, I'm glad to learn from you guys! Thanks.

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sat, Sep 24, 2016 at 12:35 AM, Reynold Xin  wrote:
> deterministic method describes whether this instance of the expression tree
> is deterministic, whereas Nondeterministic trait is about a class.
>
>
> On Fri, Sep 23, 2016 at 10:46 AM, Jacek Laskowski  wrote:
>>
>> Hi Herman,
>>
>> That helps to know that someone can explain why we've got the two
>> nondeterministic states.
>>
>> It's not possible to say...a non-Nondeterministic expression can be
>> non-deterministic (the former is the trait while the latter is the
>> method) #strange
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Fri, Sep 23, 2016 at 6:44 PM, Herman van Hövell tot Westerflier
>>  wrote:
>> > Jacek,
>> >
>> > A non-deterministic expression usually holds some state. The
>> > Nondeterministic trait makes sure a user can initialize this state
>> > properly.
>> > Take a look at InterpretedProjection for instance.
>> >
>> > HTH
>> >
>> > -Herman
>> >
>> > On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> Just came across the Expression trait [1] that can be check for
>> >> determinism by the method deterministic [2] and trait Nondeterministic
>> >> [3]. Why both?
>> >>
>> >> [1]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L53
>> >> [2]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L80
>> >> [3]
>> >>
>> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L271
>> >>
>> >> Pozdrawiam,
>> >> Jacek Laskowski
>> >> 
>> >> https://medium.com/@jaceklaskowski/
>> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> >> Follow me at https://twitter.com/jaceklaskowski
>> >>
>> >> -
>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>> >>
>> >
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Why Expression.deterministic method and Nondeterministic trait?

[Reynold Xin]

deterministic method describes whether this instance of the expression tree
is deterministic, whereas Nondeterministic trait is about a class.


On Fri, Sep 23, 2016 at 10:46 AM, Jacek Laskowski  wrote:

> Hi Herman,
>
> That helps to know that someone can explain why we've got the two
> nondeterministic states.
>
> It's not possible to say...a non-Nondeterministic expression can be
> non-deterministic (the former is the trait while the latter is the
> method) #strange
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Fri, Sep 23, 2016 at 6:44 PM, Herman van Hövell tot Westerflier
>  wrote:
> > Jacek,
> >
> > A non-deterministic expression usually holds some state. The
> > Nondeterministic trait makes sure a user can initialize this state
> properly.
> > Take a look at InterpretedProjection for instance.
> >
> > HTH
> >
> > -Herman
> >
> > On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski 
> wrote:
> >>
> >> Hi,
> >>
> >> Just came across the Expression trait [1] that can be check for
> >> determinism by the method deterministic [2] and trait Nondeterministic
> >> [3]. Why both?
> >>
> >> [1]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L53
> >> [2]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L80
> >> [3]
> >> https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L271
> >>
> >> Pozdrawiam,
> >> Jacek Laskowski
> >> 
> >> https://medium.com/@jaceklaskowski/
> >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> >> Follow me at https://twitter.com/jaceklaskowski
> >>
> >> -
> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
> >>
> >
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Why Expression.deterministic method and Nondeterministic trait?

[Herman van Hövell tot Westerflier]

Jacek,

A non-deterministic expression usually holds some state. The
Nondeterministic trait makes sure a user can initialize this state
properly. Take a look at InterpretedProjection

for instance.

HTH

-Herman

On Fri, Sep 23, 2016 at 8:28 AM, Jacek Laskowski  wrote:

> Hi,
>
> Just came across the Expression trait [1] that can be check for
> determinism by the method deterministic [2] and trait Nondeterministic
> [3]. Why both?
>
> [1] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L53
> [2] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L80
> [3] https://github.com/apache/spark/blob/master/sql/
> catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> expressions/Expression.scala#L271
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Why Expression.deterministic method and Nondeterministic trait?

[Jacek Laskowski]

Hi,

Just came across the Expression trait [1] that can be check for
determinism by the method deterministic [2] and trait Nondeterministic
[3]. Why both?

[1] 
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L53
[2] 
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L80
[3] 
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala#L271

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org