Re: How to use scala.math.Ordering in java

2016-02-13 Thread shcher 
JavaPaidRDD class creates instances of scala.math.Ordering.
I've been able to view the code with Fernflower decompiler in IntelliJ IDEA.
For K=Integer, one can do:
"Ordering ordering =
Ordering$.MODULE$.comparatorToOrdering(Comparator.naturalOrder())"
or implement Comparator directly.
But beware... "Ordering$.MODULE$" is the static reference, and thus is not
Serializable.
So, the whole class performing calculation should implement Serializable.

I do have a working example of RDD.countByValue(Ordering) in Java.

I'm guessing the creators of scala.math.* wanted those classes to be used in
Scala, not in Java (and Scala doesn't have static references).



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-scala-math-Ordering-in-java-tp26019p26218.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: How to use scala.math.Ordering in java

2016-01-21 Thread Dave 

Thanks Ted.


On 20/01/16 18:24, Ted Yu wrote:

Please take a look at the following files for some examples:

sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java
sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java

Cheers

On Wed, Jan 20, 2016 at 1:03 AM, ddav > wrote:


Hi,

I am writing my Spark application in java and I need to use a
RangePartitioner.

JavaPairRDD progRef1 =
sc.textFile(programReferenceDataFile, 12).filter(
(String s) ->
!s.startsWith("#")).mapToPair(
(String s) -> {
ProgramDataRef ref = new ProgramDataRef(s);
return new
Tuple2(ref.genre, ref);
}
);

RangePartitioner rangePart = new
RangePartitioner(12, progRef1.rdd(), true, ?,
progRef1.kClassTag());

I can't determine how to create the correct object for parameter 4
which is
"scala.math.Ordering evidence$1" from the documentation. From the
scala.math.Ordering code I see there are many implicit objects and one
handles Strings. How can I access them from Java.

Thanks,
Dave.



--
View this message in context:

http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-scala-math-Ordering-in-java-tp26019.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org

For additional commands, e-mail: user-h...@spark.apache.org

Re: How to use scala.math.Ordering in java

2016-01-20 Thread Ted Yu 
Please take a look at the following files for some examples:

sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java
sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java

Cheers

On Wed, Jan 20, 2016 at 1:03 AM, ddav  wrote:

> Hi,
>
> I am writing my Spark application in java and I need to use a
> RangePartitioner.
>
> JavaPairRDD progRef1 =
> sc.textFile(programReferenceDataFile, 12).filter(
> (String s) ->
> !s.startsWith("#")).mapToPair(
> (String s) -> {
> ProgramDataRef ref = new
> ProgramDataRef(s);
> return new Tuple2 ProgramDataRef>(ref.genre, ref);
> }
> );
>
> RangePartitioner rangePart = new
> RangePartitioner(12, progRef1.rdd(), true, ?,
> progRef1.kClassTag());
>
> I can't determine how to create the correct object for parameter 4 which is
> "scala.math.Ordering evidence$1" from the documentation. From the
> scala.math.Ordering code I see there are many implicit objects and one
> handles Strings. How can I access them from Java.
>
> Thanks,
> Dave.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-scala-math-Ordering-in-java-tp26019.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

How to use scala.math.Ordering in java

2016-01-20 Thread ddav 
Hi,

I am writing my Spark application in java and I need to use a
RangePartitioner. 

JavaPairRDD progRef1 =
sc.textFile(programReferenceDataFile, 12).filter(
(String s) -> !s.startsWith("#")).mapToPair(
(String s) -> {
ProgramDataRef ref = new 
ProgramDataRef(s);
return new Tuple2(ref.genre, ref);
}
);

RangePartitioner rangePart = new
RangePartitioner(12, progRef1.rdd(), true, ?,
progRef1.kClassTag());

I can't determine how to create the correct object for parameter 4 which is
"scala.math.Ordering evidence$1" from the documentation. From the
scala.math.Ordering code I see there are many implicit objects and one
handles Strings. How can I access them from Java.

Thanks,
Dave.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-scala-math-Ordering-in-java-tp26019.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org