I've been watching this for a bit and was thinking that a combination of a
message bus (Rabbit MQ?) and Cro should provide most of what you'd need for
a backbone.
The fact that Raku has Supplies and Channels built in means it feels like a
problem that's easy enough to fix.
This is probably me coming from a position of not knowing rarely enough
about the problem space though.
On Mon, 29 Nov 2021 at 06:35, Piper H <pott...@gmail.com> wrote:
> William, I didn't use SparkR. I use R primarily for plotting.
>
> Spark's basic API is quite simple, it does the distributed computing of
> map, filter, group, reduce etc, which are all covered by perl's map, sort,
> grep functions IMO.
>
> for instance, this common statistics on Spark:
>
> >>> fruit.take(5)
> [('peach', 1), ('apricot', 2), ('apple', 3), ('haw', 1), ('persimmon', 9)]
> >>>
> >>>
> >>> fruit.filter(lambda x:x[0] == 'apple').reduceByKey(lambda
> x,y:x+y).collect()
> [('apple', 86)]
>
> Which is easily implemented by perl's grep and map functions.
> But we need a distributed computing framework of perl6.
>
> Yes there is already the perl-spark project:
> https://github.com/perl-spark/Spark
> Which didn't get updated for many years. I don't think it's still in
> active development.
>
> So I asked the original question.
>
> Thank you.
> Piper
>
>
> On Mon, Nov 29, 2021 at 1:44 PM William Michels <w...@caa.columbia.edu>
> wrote:
>
>> Hi Piper!
>>
>> Have you used SparkR (R on Spark)?
>>
>> https://spark.apache.org/docs/latest/sparkr.html
>>
>> I'm encouraged by the data-type mapping between R and Spark. It
>> suggests to me that with a reasonable Spark API, mapping data types
>> between Raku and Spark should be straightforward:
>>
>>
>> https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark
>>
>> Best Regards,
>>
>> Bill.
>>
>>
>> On Sat, Nov 27, 2021 at 12:16 AM Piper H <pott...@gmail.com> wrote:
>> >
>> > I use perl5 everyday for data statistics.
>> > The scripts are running on a single server for the computing tasks.
>> > I also use R, which has the similar usage.
>> > When we face very large data, we change to Apache Spark for distributed
>> computing.
>> > Spark's interface languages (python, scala, even ruby) are not
>> flexible, but their computing capability is amazing, due to the whole
>> cluster contributing the computing powers.
>> > Yes I know perl5 is somewhat old, but in perl6 why won't we make that a
>> distributed computing framework like Spark? Then it will help a lot to the
>> data programmer who already knows perl.
>> > I expect a lot from this project.
>> >
>> > Thanks.
>> > Piper
>>
>
--
Simon Proctor
Cognoscite aliquid novum cotidie
http://www.khanate.co.uk/
