I forgot… it does the same thing with the reducer…
int dartsInCircle = dotsDs.reduce((x, y) -> x + y);
jg
> On Dec 28, 2019, at 12:38 PM, Jean-Georges Perrin <[email protected]> wrote:
>
> Hey guys,
>
> This code:
>
> Dataset<Row> incrementalDf = spark
> .createDataset(l, Encoders.INT())
> .toDF();
> Dataset<Integer> dotsDs = incrementalDf
> .map(status -> {
> double x = Math.random() * 2 - 1;
> double y = Math.random() * 2 - 1;
> counter++;
> if (counter % 100000 == 0) {
> System.out.println("" + counter + " darts thrown so far");
> }
> return (x * x + y * y <= 1) ? 1 : 0;
> }, Encoders.INT());
>
> used to work with Spark 2.x, in the two previous, it says:
>
> The method map(Function1<Row,Integer>, Encoder<Integer>) is ambiguous for the
> type Dataset<Row>
>
> IfI define my mapping function as a class it works fine. Here is the class:
>
> private final class DartMapper
> implements MapFunction<Row, Integer> {
> private static final long serialVersionUID = 38446L;
>
> @Override
> public Integer call(Row r) throws Exception {
> double x = Math.random() * 2 - 1;
> double y = Math.random() * 2 - 1;
> counter++;
> if (counter % 1000 == 0) {
> System.out.println("" + counter + " operations done so far");
> }
> return (x * x + y * y <= 1) ? 1 : 0;
> }
> }
>
> Any hint on what/if I did wrong?
>
> jg
>
>
>
