Re: about binary protocol porting

[Piper H]

Glad to hear these suggestions, @Geoffery.
I also have a question, this product has a clear binary protocol, do you
know how to port it to perl or perl6?
https://ignite.apache.org/docs/latest/binary-client-protocol/binary-client-protocol
I was using their python/ruby clients, but there is not a perl version.

Thanks.
Piper

On Tue, Jan 4, 2022 at 11:15 AM Geoffrey Broadwell  wrote:

> I love doing binary codecs for Raku[1]!  How you approach this really
> depends on what formats and protocols you want to create Raku modules for.
>
> The first thing you need to be able to do is test if your codec is
> correct.  It is notoriously easy to make a tiny mistake in a protocol
> implementation and (especially for binary protocols) miss it entirely
> because it only happens in certain edge cases.
>
> If the format or protocol in question is open and has one or more public
> test suites, you're in good shape.  Raku gives a lot of power for
> refactoring tests to be very clean, and I've had good success doing this
> with several formats.
>
> If there is no public test suite, but you can find RFCs or other detailed
> specs, you can often bootstrap a bespoke test suite from the examples in
> the spec documents.  Failing that, sometimes you can find sites (even
> Wikipedia, for the most common formats) that have known-correct examples to
> start with, or have published reverse engineering of files or captured data.
>
> If the format is truly proprietary, you'll be getting lots of reverse
> engineering practice of your own. 
>
> Now that you have some way of testing correctness, you'll want to be able
> to diagnose the incorrect bits.  Make sure you have some way of presenting
> easily-readable text expansions of the binary format, because just
> comparing raw buffer contents can be rather tedious (though I admit to
> having found bugs in a public test suite by spending so much time staring
> at the buffers I could tell they'd messed up a translation in a way that
> made the test always pass).  If the format or protocol has an official text
> translation/diagnostic/debug format -- CBOR, BSON, Protobuf, etc. all have
> these -- so much the better, you should support that format as soon as
> practical.
>
> Once you get down to the nitty-gritty of writing the codec, I find it is
> very important to make it work before making it fast.  There is a lot of
> room for tuning Raku code, but it is WAY easier to get things going in the
> right direction by starting off with idiomatic Raku -- given/when,
> treating the data buffer as if it was a normal Array (Positional really),
> and so on.
>
> Make sure that with every protocol feature that you add, that you make
> tests newly pass, and (I find at least) that you write the coding and
> decoding bits at the same time, so you can check that you can round-trip
> data successfully.  For the love of all that is good, don't implement any
> obtuse features before the core features are rock solid and pass the test
> suite with nary a hiccup.
>
> After that, when you think you're ready to optimize, write performance
> *tests* first.  Make sure you test with data that will both use your
> codec in a typical manner, and also test out all the odd corners.  You're
> looking for things that seem weirdly slow; this usually indicates a thinko
> like copying the entire buffer each time you read a byte from it, or
> somesuch.
>
> Once you've got the obvious performance kinks worked out, come by and ask
> again, and we can give you further advice from there.  Or heck, just come
> visit us on IRC (#raku at Libera.chat), and we'll be happy to help.  (Do
> stick around for a while though, because traffic varies strongly by time of
> day and day of week.)
>
> Best Regards,
>
>
> Geoff (japhb)
>
>
> [1]  I'm a bit of a nut for it, really.  In the distant past, I wrapped C
> libraries to get the job done, but more recently I've done them as plain
> Raku code (and sometimes NQP, the language that Rakudo is written in).
>
> I've written some of the binary format codecs for Raku:
>
>- https://github.com/japhb/CBOR-Simple
>- https://github.com/japhb/BSON-Simple
>- https://github.com/japhb/Terminal-ANSIParser
>- https://github.com/japhb/TinyFloats
>
> Modified or tuned others:
>
>- https://github.com/samuraisam/p6-pb/commits?author=japhb
>- https://github.com/japhb/serializer-perf
>- (Lots of stuff spread across various Cro
> repositories)
>
> Added a spec extension for an existing standardized format (CBOR):
>
>- https://github.com/japhb/cbor-specs/blob/main/capture.md
>
> And I think I forgot a few things.  
>
>
>

Re: hope we have the distributed computing perl6

[Piper H]

On Tue, Nov 30, 2021 at 1:38 AM William Michels 
wrote:

> Hi Piper,
>
> RE:
>
>
> I have copied some active members of the Perl community on this email,
> in the hopes that they can help transfer the "perl-spark" Github
> project.
>
> Best Regards, Bill.
>
>


Thank you Bill.
If there is a perl based distributed system it would be great anyway.
Today is cloud native days, I hope we can get more benefit from the perl
world.
If we speak only the programming languages, there are too many already, Go,
Erlang, Julia, Rust, they are good enough.
But a distributed system language will help more on today's actual
problems. Just my thoughts...

Regards
Piper

Re: hope we have the distributed computing perl6

[Piper H]

Hello Simon, yes I know RMQ quite well. But I don't think perl with RMQ
will provide the functions of distribution.
Maybe Apache Pulsar can do that. Pulsar has a good Websocket API, based on
that a perl client will be implemented easily.
But this project is still on startup. I have tested it, it's quite slow and
unstable right now.

Thanks.

On Mon, Nov 29, 2021 at 3:57 PM Simon Proctor 
wrote:

> I've been watching this for a bit and was thinking that a combination of a
> message bus (Rabbit MQ?) and Cro should provide most of what you'd need for
> a backbone.
>
> The fact that Raku has Supplies and Channels built in means it feels like
> a problem that's easy enough to fix.
>
> This is probably me coming from a position of not knowing rarely enough
> about the problem space though.
>
> On Mon, 29 Nov 2021 at 06:35, Piper H  wrote:
>
>> William, I didn't use SparkR. I use R primarily for plotting.
>>
>> Spark's basic API is quite simple, it does the distributed computing of
>> map, filter, group, reduce etc, which are all covered by perl's map, sort,
>> grep functions IMO.
>>
>> for instance, this common statistics on Spark:
>>
>> >>> fruit.take(5)
>> [('peach', 1), ('apricot', 2), ('apple', 3), ('haw', 1), ('persimmon', 9)]
>> >>>
>> >>>
>> >>> fruit.filter(lambda x:x[0] == 'apple').reduceByKey(lambda
>> x,y:x+y).collect()
>> [('apple', 86)]
>>
>> Which is easily implemented by perl's grep and map functions.
>> But we need a distributed computing framework of perl6.
>>
>> Yes there is already the perl-spark project:
>> https://github.com/perl-spark/Spark
>> Which didn't get updated for many years. I don't think it's still in
>> active development.
>>
>> So I asked the original question.
>>
>> Thank you.
>> Piper
>>
>>
>> On Mon, Nov 29, 2021 at 1:44 PM William Michels 
>> wrote:
>>
>>> Hi Piper!
>>>
>>> Have you used SparkR (R on Spark)?
>>>
>>> https://spark.apache.org/docs/latest/sparkr.html
>>>
>>> I'm encouraged by the data-type mapping between R and Spark. It
>>> suggests to me that with a reasonable Spark API, mapping data types
>>> between Raku and Spark should be straightforward:
>>>
>>>
>>> https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark
>>>
>>> Best Regards,
>>>
>>> Bill.
>>>
>>>
>>> On Sat, Nov 27, 2021 at 12:16 AM Piper H  wrote:
>>> >
>>> > I use perl5 everyday for data statistics.
>>> > The scripts are running on a single server for the computing tasks.
>>> > I also use R, which has the similar usage.
>>> > When we face very large data, we change to Apache Spark for
>>> distributed computing.
>>> > Spark's interface languages (python, scala, even ruby) are not
>>> flexible, but their computing capability is amazing, due to the whole
>>> cluster contributing the computing powers.
>>> > Yes I know perl5 is somewhat old, but in perl6 why won't we make that
>>> a distributed computing framework like Spark? Then it will help a lot to
>>> the data programmer who already knows perl.
>>> > I expect a lot from this project.
>>> >
>>> > Thanks.
>>> > Piper
>>>
>>
>
> --
> Simon Proctor
> Cognoscite aliquid novum cotidie
>
> http://www.khanate.co.uk/
>

Re: hope we have the distributed computing perl6

[Piper H]

William, I didn't use SparkR. I use R primarily for plotting.

Spark's basic API is quite simple, it does the distributed computing of
map, filter, group, reduce etc, which are all covered by perl's map, sort,
grep functions IMO.

for instance, this common statistics on Spark:

>>> fruit.take(5)
[('peach', 1), ('apricot', 2), ('apple', 3), ('haw', 1), ('persimmon', 9)]
>>>
>>>
>>> fruit.filter(lambda x:x[0] == 'apple').reduceByKey(lambda
x,y:x+y).collect()
[('apple', 86)]

Which is easily implemented by perl's grep and map functions.
But we need a distributed computing framework of perl6.

Yes there is already the perl-spark project:
https://github.com/perl-spark/Spark
Which didn't get updated for many years. I don't think it's still in active
development.

So I asked the original question.

Thank you.
Piper


On Mon, Nov 29, 2021 at 1:44 PM William Michels 
wrote:

> Hi Piper!
>
> Have you used SparkR (R on Spark)?
>
> https://spark.apache.org/docs/latest/sparkr.html
>
> I'm encouraged by the data-type mapping between R and Spark. It
> suggests to me that with a reasonable Spark API, mapping data types
> between Raku and Spark should be straightforward:
>
>
> https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-r-and-spark
>
> Best Regards,
>
> Bill.
>
>
> On Sat, Nov 27, 2021 at 12:16 AM Piper H  wrote:
> >
> > I use perl5 everyday for data statistics.
> > The scripts are running on a single server for the computing tasks.
> > I also use R, which has the similar usage.
> > When we face very large data, we change to Apache Spark for distributed
> computing.
> > Spark's interface languages (python, scala, even ruby) are not flexible,
> but their computing capability is amazing, due to the whole cluster
> contributing the computing powers.
> > Yes I know perl5 is somewhat old, but in perl6 why won't we make that a
> distributed computing framework like Spark? Then it will help a lot to the
> data programmer who already knows perl.
> > I expect a lot from this project.
> >
> > Thanks.
> > Piper
>

Re: hope we have the distributed computing [Raku]

[Piper H]

I would like to be. But I know nothing about the development of Raku
itself, so I most probably would be the product and test engineer for this
project.

Thanks.

On Sun, Nov 28, 2021 at 8:53 AM Darren Duncan 
wrote:

> On 2021-11-27 12:16 a.m., Piper H wrote:
> > but in [Raku] why won't we make that a
> > distributed computing framework like Spark? Then it will help a lot to
> the data
> > programmer who already knows perl.
> > I expect a lot from this project.
>
> Piper, are you offering to lead your proposed project and be the primary
> developer working on it? -- Darren Duncan
>

hope we have the distributed computing perl6

[Piper H]

I use perl5 everyday for data statistics.
The scripts are running on a single server for the computing tasks.
I also use R, which has the similar usage.
When we face very large data, we change to Apache Spark for distributed
computing.
Spark's interface languages (python, scala, even ruby) are not flexible,
but their computing capability is amazing, due to the whole cluster
contributing the computing powers.
Yes I know perl5 is somewhat old, but in perl6 why won't we make that a
distributed computing framework like Spark? Then it will help a lot to the
data programmer who already knows perl.
I expect a lot from this project.

Thanks.
Piper

Re: why not raku ?

[Piper H]

Has Larry Wall joined the development team of raku?
I can't think of the case  if Matz was not in the new Ruby development team.

On Mon, Nov 22, 2021 at 7:19 PM Aureliano Guedes 
wrote:

> Some time ago I found an example of how to integrate C++ through
> NativeCall, but it is not straightforward.
> If we have the advantage of using C and *C++* libs, we will just need one
> more step: "convert legions of dev to our sect".
>
> As more devs became a rakoon, more the language became popular and
> populated of tools and libs.
>
> Imo, the first thing we need to think about is writing really good
> documentation.
> Easy to navigate. Easy to understand. Easy to find practical examples.
>
> Few months ago I was trying to adapt the C++ lib xframe<
> https://github.com/xtensor-stack/xframe>.
> But I missed everything because I didn't put it on github and my HDD died,
> sorry about that.
> Anyway, I'm quite limited in knowledge about Raku.
>
>
> On Mon, Nov 22, 2021 at 12:33 AM Clifton Wood 
> wrote:
>
>> Ralph:
>>
>> It has *two* compelling use cases, and it's the ones that brought me to
>> Raku in the first place:
>>
>> - First class C integration without the need for C (NativeCall)
>>
>> - Grammars
>>
>> I have gone through over 20 C libraries with at least a modicum of
>> success. All in Raku. All easily done to get the basic support (C stubs).
>> I've written utilities that make much of the work of porting an .h file as
>> simple as running a single script.
>>
>> Hopefully I will be able to get these projects out the door, but I have
>> to overcome the CURI output limitations first. I'd really like to have a
>> custom installer built to handle output to the user while pre-compiling,
>> but I've decided that's something that can wait for the future.
>>
>> I guess I'll be releasing the first lib sometime soon.
>>
>> - Xliff
>>
>> On Sun, Nov 21, 2021 at 10:28 PM Clifton Wood 
>> wrote:
>>
>>> Piper: That would be zef
>>>
>>> https://github.com/ugexe/zef
>>>
>>> On Sun, Nov 21, 2021 at 8:51 PM Piper H  wrote:
>>>
>>>> What's the package management tool for raku?
>>>> The stuff like gem/bundle for ruby and cpanm for perl5.
>>>>
>>>> Thanks.
>>>>
>>>> On Mon, Nov 22, 2021 at 9:43 AM Ralph Mellor 
>>>> wrote:
>>>>
>>>>> My 2c:
>>>>>
>>>>> On Fri, Nov 19, 2021 at 9:45 AM Marc Chantreux  wrote:
>>>>> >
>>>>> > > I like ruby and perl
>>>>> >
>>>>> > so do I but raku is by far my prefered interpreted langage now.
>>>>>
>>>>> Nit. It's compiled, in the same way Java or C# is compiled.
>>>>>
>>>>> Consider:
>>>>> ```
>>>>> say 42;
>>>>> constant foo = die;
>>>>> ```
>>>>> If it were interpreted, the `42` would appear, and then the
>>>>> program would die. But instead it dies at compile-time
>>>>> (`constant`s are initialized at compile-time).
>>>>>
>>>>> That said, the usual way of using it is to run a program,
>>>>> which compiles it, and then, if it successfully compiles,
>>>>> immediately runs the compiled program.
>>>>>
>>>>> > I don't raku that much and most of the time, i read the
>>>>> > doc more than i actually write code but when it's writen,
>>>>> > it's always elegant and concise the way i never seen before.
>>>>>
>>>>> Many folk who like Ruby or Python or Lang X say much the
>>>>> same thing about those PLs.
>>>>>
>>>>> > > Maybe perl6 is still not production-ready?
>>>>>
>>>>> Imo it's as production ready as Python.
>>>>>
>>>>> > > but why so few open source projects which were developed by perl6?
>>>>>
>>>>> It's all relative. Compared to most of the thousands of PLs
>>>>> with less projects, there are lots of projects developed in Raku.
>>>>>
>>>>> But you presumably mean in comparison to the likes of Python and Ruby.
>>>>>
>>>>> There are many factors. Some I'd focus on are:
>>>>>
>>>>> * It's unusual. Few folk like that.
>>>>>
>>>>> * It has a large language surface area. Few folk like that.
>>>>

Re: why not raku ?

[Piper H]

What's the package management tool for raku?
The stuff like gem/bundle for ruby and cpanm for perl5.

Thanks.

On Mon, Nov 22, 2021 at 9:43 AM Ralph Mellor 
wrote:

> My 2c:
>
> On Fri, Nov 19, 2021 at 9:45 AM Marc Chantreux  wrote:
> >
> > > I like ruby and perl
> >
> > so do I but raku is by far my prefered interpreted langage now.
>
> Nit. It's compiled, in the same way Java or C# is compiled.
>
> Consider:
> ```
> say 42;
> constant foo = die;
> ```
> If it were interpreted, the `42` would appear, and then the
> program would die. But instead it dies at compile-time
> (`constant`s are initialized at compile-time).
>
> That said, the usual way of using it is to run a program,
> which compiles it, and then, if it successfully compiles,
> immediately runs the compiled program.
>
> > I don't raku that much and most of the time, i read the
> > doc more than i actually write code but when it's writen,
> > it's always elegant and concise the way i never seen before.
>
> Many folk who like Ruby or Python or Lang X say much the
> same thing about those PLs.
>
> > > Maybe perl6 is still not production-ready?
>
> Imo it's as production ready as Python.
>
> > > but why so few open source projects which were developed by perl6?
>
> It's all relative. Compared to most of the thousands of PLs
> with less projects, there are lots of projects developed in Raku.
>
> But you presumably mean in comparison to the likes of Python and Ruby.
>
> There are many factors. Some I'd focus on are:
>
> * It's unusual. Few folk like that.
>
> * It has a large language surface area. Few folk like that.
>
> * It's very slow. Very few folk like that.
>
> * It has no widely recognized distinctive compelling use case.
>
> As a consequence of these and other factors there is minimal
> interest in it so far, let alone adoption.
>
> So now, one can add another factor:
>
> * It isn't interesting to most, and has had minimal adoption so far.
> Almost NO folk are OK with that.
>
> So now, one can add another factor:
>
> * Almost NO folk want to help develop it. And you can't attract
> them either. Unless they get it. Because then they fall in love
> with it. And so it rolls. For now.
>
> So, for now, it needs more work, as it has always done.
>
> > * raku shines on interpreted langages when people are
> > moving to compiled langages
>
> It's a compiled language, so that's not quite right. Perhaps
> you meant it's dynamically typed rather than statically typed,
> but that's not quite right either.
>
> If one squints, it's an open source alternative to Oracle's
> Truffle/Graal/JVM, but it's wy slower.
>
> > * raku is that rich it's hard to get it in a first view
>
> I'd say it's hard to *ever* get most of it. It's as ambitious
> as Truffle/Graal/JVM, perhaps even more so.
>
> But it should and *will* be easy to get it a little at a time.
>
> But we're not there yet.
>
> There's a fairly obvious way to make it vastly easier.
>
> Which is to create mini languages that aren't Raku
> but showcase selected parts of its talents.
>
> But that will have to wait until RakuAST lands.
>
> And perhaps a language version *after* that.
>
> So perhaps 3-4 years if we're lucky.
>
> > * raku is still way too slow to be taken seriously
> > by a large audience
>
> Yes. For now.
>
> > * js or python developpers are legions on the market
> > now so everyone choose this as an argument
>
> Yes. And ts devs too.
>
> > * we need more packages on raku.land
>
> I don't think that's important. We need better Inlines.
>
> We need to deflate the packages/modules/libs argument.
>
> > * i really think technologies are massively adopted when they are
> >   packaged in main linux distros because lot of people don't want to
> >   bother compiling an interpreter or adding extra repos to do it.
>
> I can see there being an opportunity to create a popular
> package before this decade is out in the form of (a fresh
> repackaging of) NQP as a "Raku Rules" engine / latter
> day PCRE / new alternative to Truffle/Graal/JVM.
>
> --
> love, raiph
>

Re: why not raku ?

[Piper H]

I know Numpy well, i can help provide some suggestions.
I even use Rumale for ML.


On Friday, November 19, 2021, Tom Browder  wrote:

> On Fri, Nov 19, 2021 at 07:48 Aureliano Guedes 
> wrote:
>
>> I am still defending that we need a package for data
>> analysis/science/engineer (like the Perl5 PDL, Python Pandas or R
>> data.table) and an IDE for streaming programming like jupyter or rstudio.
>>
>
> Speaking for myself, I agree, and I think there are more than a few of us
> who would help if some expert took the stick and started such a project.
>
> -Tom
>

Re: why not raku ?

[Piper H]

Thanks for the explanation Marc.
I hope there is a chance to cheer up perl/perl6 again.
Such as ML makes python become active, and Rails made ruby popular once.

Regards.

On Fri, Nov 19, 2021 at 5:44 PM Marc Chantreux  wrote:

> hello,
>
> > I like ruby and perl
>
> so do I but raku is by far my prefered interpreted langage now.
> I don't raku that much and most of the time, i read the doc more than i
> actually write code but when it's writen, it's always elegant and
> concise the way i never seen before.
>
> > Maybe perl6 is still not production-ready?
>
> Perl6 is now raku.
>
> it depends: what do you mean by "production" and "ready"? start with
> some few non-critical usecases and you'll see raku is production ready
> enough for lot of things.
>
> > but why so few open source projects which were developed by perl6?
>
> wow ... interesting question. my cents on it:
>
> * raku shines on interpreted langages when people are moving to compiled
> langages
> * raku is that rich it's hard to get it in a first view
> * raku is still way too slow to be taken seriously by a large audience
> * js or python developpers are legions on the market now so everyone
>   choose this as an argument
> * we need more packages on raku.land
> * i really think technologies are massively adopted when they are
>   packaged in main linux distros because lot of people don't want to
>   bother compiling an interpreter or adding extra repos to do it.
>
> regards,
> marc
>

Re: fixing the documentation

[Piper H]

I like ruby and perl, but why so few open source projects which were
developed by perl6?
Maybe perl6 is still not production-ready?

Thanks.

On Fri, Nov 19, 2021 at 2:12 PM JJ Merelo  wrote:

> Thanks a lot.
>
> Cheers
>
> El vie, 19 nov 2021 a las 0:08, Marc Chantreux ()
> escribió:
>
>> Hello,
>>
>> > The best would be if you propose a PR or open an issue at
>> > https://github.com/Raku/doc. Any help with the documentation would
>> > most certainly be appreciated as people working on the docs project
>> > are overloaded.
>>
>> Sorry I was late on this because I wasn't sure how to revamp the whole
>> thing. so i just simplified the 5to6 page this way:
>>
>>
>> https://github.com/Raku/doc/commit/cd32380ab4e1c5ad2017d60a31def9189b54c80f
>>
>> Simple changes are better anyway.
>>
>> thanks.
>> marc
>>
>
>
> --
> JJ
>