Hello Simon, yes I know RMQ quite well. But I don't think perl with RMQ
will provide the functions of distribution.
Maybe Apache Pulsar can do that. Pulsar has a good Websocket API, based on
that a perl client will be implemented easily.
But this project is still on startup. I have tested it, it's quite slow and
unstable right now.
Thanks.
On Mon, Nov 29, 2021 at 3:57 PM Simon Proctor <simon.proc...@gmail.com>
wrote:
> I've been watching this for a bit and was thinking that a combination of a
> message bus (Rabbit MQ?) and Cro should provide most of what you'd need for
> a backbone.
>
> The fact that Raku has Supplies and Channels built in means it feels like
> a problem that's easy enough to fix.
>
> This is probably me coming from a position of not knowing rarely enough
> about the problem space though.
>
> On Mon, 29 Nov 2021 at 06:35, Piper H <pott...@gmail.com> wrote:
>
>> William, I didn't use SparkR. I use R primarily for plotting.
>>
>> Spark's basic API is quite simple, it does the distributed computing of
>> map, filter, group, reduce etc, which are all covered by perl's map, sort,
>> grep functions IMO.
>>
>> for instance, this common statistics on Spark:
>>
>> >>> fruit.take(5)
>> [('peach', 1), ('apricot', 2), ('apple', 3), ('haw', 1), ('persimmon', 9)]
>> >>>
>> >>>
>> >>> fruit.filter(lambda x:x[0] == 'apple').reduceByKey(lambda
>> x,y:x+y).collect()
>> [('apple', 86)]
>>
>> Which is easily implemented by perl's grep and map functions.
>> But we need a distributed computing framework of perl6.
>>
>> Yes there is already the perl-spark project:
>> https://github.com/perl-spark/Spark
>> Which didn't get updated for many years. I don't think it's still in
>> active development.
>>
>> So I asked the original question.
>>
>> Thank you.
>> Piper
>>
>>
>> On Mon, Nov 29, 2021 at 1:44 PM William Michels <w...@caa.columbia.edu>
>> wrote:
>>
>>> Hi Piper!
>>>
>>> Have you used SparkR (R on Spark)?
>>>
>>> https://spark.apache.org/docs/latest/sparkr.html
>>>
>>> I'm encouraged by the data-type mapping between R and Spark. It
>>> suggests to me that with a reasonable Spark API, mapping data types
>>> between Raku and Spark should be straightforward:
>>>
>>>
>>> https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark
>>>
>>> Best Regards,
>>>
>>> Bill.
>>>
>>>
>>> On Sat, Nov 27, 2021 at 12:16 AM Piper H <pott...@gmail.com> wrote:
>>> >
>>> > I use perl5 everyday for data statistics.
>>> > The scripts are running on a single server for the computing tasks.
>>> > I also use R, which has the similar usage.
>>> > When we face very large data, we change to Apache Spark for
>>> distributed computing.
>>> > Spark's interface languages (python, scala, even ruby) are not
>>> flexible, but their computing capability is amazing, due to the whole
>>> cluster contributing the computing powers.
>>> > Yes I know perl5 is somewhat old, but in perl6 why won't we make that
>>> a distributed computing framework like Spark? Then it will help a lot to
>>> the data programmer who already knows perl.
>>> > I expect a lot from this project.
>>> >
>>> > Thanks.
>>> > Piper
>>>
>>
>
> --
> Simon Proctor
> Cognoscite aliquid novum cotidie
>
> http://www.khanate.co.uk/
>
