William, I didn't use SparkR. I use R primarily for plotting.
Spark's basic API is quite simple, it does the distributed computing of
map, filter, group, reduce etc, which are all covered by perl's map, sort,
grep functions IMO.
for instance, this common statistics on Spark:
>>> fruit.take(5)
[('peach', 1), ('apricot', 2), ('apple', 3), ('haw', 1), ('persimmon', 9)]
>>>
>>>
>>> fruit.filter(lambda x:x[0] == 'apple').reduceByKey(lambda
x,y:x+y).collect()
[('apple', 86)]
Which is easily implemented by perl's grep and map functions.
But we need a distributed computing framework of perl6.
Yes there is already the perl-spark project:
https://github.com/perl-spark/Spark
Which didn't get updated for many years. I don't think it's still in active
development.
So I asked the original question.
Thank you.
Piper
On Mon, Nov 29, 2021 at 1:44 PM William Michels <w...@caa.columbia.edu>
wrote:
> Hi Piper!
>
> Have you used SparkR (R on Spark)?
>
> https://spark.apache.org/docs/latest/sparkr.html
>
> I'm encouraged by the data-type mapping between R and Spark. It
> suggests to me that with a reasonable Spark API, mapping data types
> between Raku and Spark should be straightforward:
>
>
> https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark
>
> Best Regards,
>
> Bill.
>
>
> On Sat, Nov 27, 2021 at 12:16 AM Piper H <pott...@gmail.com> wrote:
> >
> > I use perl5 everyday for data statistics.
> > The scripts are running on a single server for the computing tasks.
> > I also use R, which has the similar usage.
> > When we face very large data, we change to Apache Spark for distributed
> computing.
> > Spark's interface languages (python, scala, even ruby) are not flexible,
> but their computing capability is amazing, due to the whole cluster
> contributing the computing powers.
> > Yes I know perl5 is somewhat old, but in perl6 why won't we make that a
> distributed computing framework like Spark? Then it will help a lot to the
> data programmer who already knows perl.
> > I expect a lot from this project.
> >
> > Thanks.
> > Piper
>
