Re: about binary protocol porting

[Geoffrey Broadwell]

Just from skimming some of the relevant docs (not having written a 
driver for Apache Ignite before), some thoughts:


 * It does indeed look like there is enough info, both as documentation
   and example code, to write codecs and drivers for Ignite
 * The formats and protocols look rather baroque, with significant
   historical baggage -- it's going to take quite a bit of work to get
   a fully compliant driver, though it does look like a smaller subset
   could be built to match just a particular need
 * There is a strong Java flavor to everything; there is some impedance
   mismatch with Raku (such as the Char array
   

   type, which is an array of UTF-16 code units that doesn't
   necessarily contain valid decodeable text)
 * There seems to be a contention in the design between desire to
   support a schema-less/plain-data mode and a schema/object mode; Raku
   easily has the metaobject protocol chops to make the latter possible
   without invoking truly deep magic, but it does require somewhat more
   advanced knowledge to write

So in short: It looks doable, but quite a fair chunk of work depending 
on how complete you need it to be, and some decisions need to be made 
about how pedantically to support their Java-flavored APIs.



On 1/3/22 7:39 PM, Piper H wrote:

Glad to hear these suggestions, @Geoffery.
I also have a question, this product has a clear binary protocol, do 
you know how to port it to perl or perl6?

https://ignite.apache.org/docs/latest/binary-client-protocol/binary-client-protocol
I was using their python/ruby clients, but there is not a perl version.

Thanks.
Piper

On Tue, Jan 4, 2022 at 11:15 AM Geoffrey Broadwell > wrote:


I love doing binary codecs for Raku[1]!  How you approach this
really depends on what formats and protocols you want to create
Raku modules for.

The first thing you need to be able to do is test if your codec is
correct.  It is notoriously easy to make a tiny mistake in a
protocol implementation and (especially for binary protocols) miss
it entirely because it only happens in certain edge cases.

If the format or protocol in question is open and has one or more
public test suites, you're in good shape.  Raku gives a lot of
power for refactoring tests to be very clean, and I've had good
success doing this with several formats.

If there is no public test suite, but you can find RFCs or other
detailed specs, you can often bootstrap a bespoke test suite from
the examples in the spec documents. Failing that, sometimes you
can find sites (even Wikipedia, for the most common formats) that
have known-correct examples to start with, or have published
reverse engineering of files or captured data.

If the format is truly proprietary, you'll be getting lots of
reverse engineering practice of your own. 😉

Now that you have some way of testing correctness, you'll want to
be able to diagnose the incorrect bits.  Make sure you have some
way of presenting easily-readable text expansions of the binary
format, because just comparing raw buffer contents can be rather
tedious (though I admit to having found bugs in a public test
suite by spending so much time staring at the buffers I could tell
they'd messed up a translation in a way that made the test always
pass).  If the format or protocol has an official text
translation/diagnostic/debug format -- CBOR, BSON, Protobuf, etc.
all have these -- so much the better, you should support that
format as soon as practical.

Once you get down to the nitty-gritty of writing the codec, I find
it is very important to make it work before making it fast.  There
is a lot of room for tuning Raku code, but it is WAY easier to get
things going in the right direction by starting off with idiomatic
Raku -- given/when, treating the data buffer as if it was a normal
Array (Positional really), and so on.

Make sure that with every protocol feature that you add, that you
make tests newly pass, and (I find at least) that you write the
coding and decoding bits at the same time, so you can check that
you can round-trip data successfully.  For the love of all that is
good, don't implement any obtuse features before the core features
are rock solid and pass the test suite with nary a hiccup.

After that, when you think you're ready to optimize, write
performance /tests/ first.  Make sure you test with data that will
both use your codec in a typical manner, and also test out all the
odd corners.  You're looking for things that seem weirdly slow;
this usually indicates a thinko like copying the entire buffer
each time you read a byte from it, or somesuch.

Once you've got the obvious performance kinks worked out, come by
and ask again, and we can g

Re: about binary protocol porting

[Piper H]

Glad to hear these suggestions, @Geoffery.
I also have a question, this product has a clear binary protocol, do you
know how to port it to perl or perl6?
https://ignite.apache.org/docs/latest/binary-client-protocol/binary-client-protocol
I was using their python/ruby clients, but there is not a perl version.

Thanks.
Piper

On Tue, Jan 4, 2022 at 11:15 AM Geoffrey Broadwell  wrote:

> I love doing binary codecs for Raku[1]!  How you approach this really
> depends on what formats and protocols you want to create Raku modules for.
>
> The first thing you need to be able to do is test if your codec is
> correct.  It is notoriously easy to make a tiny mistake in a protocol
> implementation and (especially for binary protocols) miss it entirely
> because it only happens in certain edge cases.
>
> If the format or protocol in question is open and has one or more public
> test suites, you're in good shape.  Raku gives a lot of power for
> refactoring tests to be very clean, and I've had good success doing this
> with several formats.
>
> If there is no public test suite, but you can find RFCs or other detailed
> specs, you can often bootstrap a bespoke test suite from the examples in
> the spec documents.  Failing that, sometimes you can find sites (even
> Wikipedia, for the most common formats) that have known-correct examples to
> start with, or have published reverse engineering of files or captured data.
>
> If the format is truly proprietary, you'll be getting lots of reverse
> engineering practice of your own. 😉
>
> Now that you have some way of testing correctness, you'll want to be able
> to diagnose the incorrect bits.  Make sure you have some way of presenting
> easily-readable text expansions of the binary format, because just
> comparing raw buffer contents can be rather tedious (though I admit to
> having found bugs in a public test suite by spending so much time staring
> at the buffers I could tell they'd messed up a translation in a way that
> made the test always pass).  If the format or protocol has an official text
> translation/diagnostic/debug format -- CBOR, BSON, Protobuf, etc. all have
> these -- so much the better, you should support that format as soon as
> practical.
>
> Once you get down to the nitty-gritty of writing the codec, I find it is
> very important to make it work before making it fast.  There is a lot of
> room for tuning Raku code, but it is WAY easier to get things going in the
> right direction by starting off with idiomatic Raku -- given/when,
> treating the data buffer as if it was a normal Array (Positional really),
> and so on.
>
> Make sure that with every protocol feature that you add, that you make
> tests newly pass, and (I find at least) that you write the coding and
> decoding bits at the same time, so you can check that you can round-trip
> data successfully.  For the love of all that is good, don't implement any
> obtuse features before the core features are rock solid and pass the test
> suite with nary a hiccup.
>
> After that, when you think you're ready to optimize, write performance
> *tests* first.  Make sure you test with data that will both use your
> codec in a typical manner, and also test out all the odd corners.  You're
> looking for things that seem weirdly slow; this usually indicates a thinko
> like copying the entire buffer each time you read a byte from it, or
> somesuch.
>
> Once you've got the obvious performance kinks worked out, come by and ask
> again, and we can give you further advice from there.  Or heck, just come
> visit us on IRC (#raku at Libera.chat), and we'll be happy to help.  (Do
> stick around for a while though, because traffic varies strongly by time of
> day and day of week.)
>
> Best Regards,
>
>
> Geoff (japhb)
>
>
> [1]  I'm a bit of a nut for it, really.  In the distant past, I wrapped C
> libraries to get the job done, but more recently I've done them as plain
> Raku code (and sometimes NQP, the language that Rakudo is written in).
>
> I've written some of the binary format codecs for Raku:
>
>- https://github.com/japhb/CBOR-Simple
>- https://github.com/japhb/BSON-Simple
>- https://github.com/japhb/Terminal-ANSIParser
>- https://github.com/japhb/TinyFloats
>
> Modified or tuned others:
>
>- https://github.com/samuraisam/p6-pb/commits?author=japhb
>- https://github.com/japhb/serializer-perf
>- (Lots of stuff spread across various Cro
> repositories)
>
> Added a spec extension for an existing standardized format (CBOR):
>
>- https://github.com/japhb/cbor-specs/blob/main/capture.md
>
> And I think I forgot a few things.  😁
>
>
>

Re: about binary protocol porting

[Geoffrey Broadwell]

I love doing binary codecs for Raku[1]!  How you approach this really 
depends on what formats and protocols you want to create Raku modules for.


The first thing you need to be able to do is test if your codec is 
correct.  It is notoriously easy to make a tiny mistake in a protocol 
implementation and (especially for binary protocols) miss it entirely 
because it only happens in certain edge cases.


If the format or protocol in question is open and has one or more public 
test suites, you're in good shape.  Raku gives a lot of power for 
refactoring tests to be very clean, and I've had good success doing this 
with several formats.


If there is no public test suite, but you can find RFCs or other 
detailed specs, you can often bootstrap a bespoke test suite from the 
examples in the spec documents.  Failing that, sometimes you can find 
sites (even Wikipedia, for the most common formats) that have 
known-correct examples to start with, or have published reverse 
engineering of files or captured data.


If the format is truly proprietary, you'll be getting lots of reverse 
engineering practice of your own. 😉


Now that you have some way of testing correctness, you'll want to be 
able to diagnose the incorrect bits.  Make sure you have some way of 
presenting easily-readable text expansions of the binary format, because 
just comparing raw buffer contents can be rather tedious (though I admit 
to having found bugs in a public test suite by spending so much time 
staring at the buffers I could tell they'd messed up a translation in a 
way that made the test always pass).  If the format or protocol has an 
official text translation/diagnostic/debug format -- CBOR, BSON, 
Protobuf, etc. all have these -- so much the better, you should support 
that format as soon as practical.


Once you get down to the nitty-gritty of writing the codec, I find it is 
very important to make it work before making it fast. There is a lot of 
room for tuning Raku code, but it is WAY easier to get things going in 
the right direction by starting off with idiomatic Raku -- given/when, 
treating the data buffer as if it was a normal Array (Positional 
really), and so on.


Make sure that with every protocol feature that you add, that you make 
tests newly pass, and (I find at least) that you write the coding and 
decoding bits at the same time, so you can check that you can round-trip 
data successfully.  For the love of all that is good, don't implement 
any obtuse features before the core features are rock solid and pass the 
test suite with nary a hiccup.


After that, when you think you're ready to optimize, write performance 
/tests/ first.  Make sure you test with data that will both use your 
codec in a typical manner, and also test out all the odd corners.  
You're looking for things that seem weirdly slow; this usually indicates 
a thinko like copying the entire buffer each time you read a byte from 
it, or somesuch.


Once you've got the obvious performance kinks worked out, come by and 
ask again, and we can give you further advice from there.  Or heck, just 
come visit us on IRC (#raku at Libera.chat), and we'll be happy to 
help.  (Do stick around for a while though, because traffic varies 
strongly by time of day and day of week.)


Best Regards,


Geoff (japhb)


[1]  I'm a bit of a nut for it, really.  In the distant past, I wrapped 
C libraries to get the job done, but more recently I've done them as 
plain Raku code (and sometimes NQP, the language that Rakudo is written in).


I've written some of the binary format codecs for Raku:

 * https://github.com/japhb/CBOR-Simple
   
 * https://github.com/japhb/BSON-Simple
   
 * https://github.com/japhb/Terminal-ANSIParser
   
 * https://github.com/japhb/TinyFloats
   

Modified or tuned others:

 * https://github.com/samuraisam/p6-pb/commits?author=japhb
   
 * https://github.com/japhb/serializer-perf
   
 * (Lots of stuff spread across various Cro
    repositories)

Added a spec extension for an existing standardized format (CBOR):

 * https://github.com/japhb/cbor-specs/blob/main/capture.md
   

And I think I forgot a few things.  😁

Re: continuous testing

[Vadim Belman]

As long as I'm trying to follow topics, related to continuous testing with 
github actions, I always see ubuntu-based scenarios. What about macOS and 
Windows? Do we have rakudo images for these?

Best regards,
Vadim Belman

> On Jan 1, 2022, at 5:14 AM, JJ Merelo  wrote:
> 
> Try the new GitHub actions with the recently renovated Docker image. See it 
> in action, for instance, here: 
> 
> https://github.com/JJ/p6-pod-load/blob/master/.github/workflows/test.yaml 
> 
> El vie, 31 dic 2021 a las 19:57, Richard Hainsworth ( >) escribió:
> Thanks
> 
> On 31/12/2021 18:27, Fernando Santagata wrote:
>> Hi Richard,
>> this is a link to the GitHub official documentation:
>> 
>> https://docs.github.com/en/actions 
>> 
>> You can also copy a configuration from another project and adapt it to your 
>> needs. For example this one installs some C libraries, some module 
>> dependencies and runs the tests:
>> 
>> https://github.com/frithnanth/raku-Math-Libgsl-Interpolation/blob/master/.github/workflows/test.yml
>>  
>> 
>> On Fri, Dec 31, 2021 at 5:56 PM Richard Hainsworth > > wrote:
>> Fernando,
>> 
>> Thanks.
>> 
>> Any link / blog / article about how to set up GitHub action up for Raku?
>> 
>> Regards,
>> 
>> Richard
>> 
>> On 31/12/2021 16:52, Fernando Santagata wrote:
>>> Hi Richard,
>>> apparently Travis CI has discontinued its free open source plan.
>>> I switched to GitHub actions; don't know what's available on other 
>>> platforms.
>>> 
>>> On Fri, Dec 31, 2021 at 5:43 PM Richard Hainsworth >> > wrote:
>>> I noticed that the .travis files have been removed from some distributions.
>>> 
>>> Also a .circleci file exists in the Raku Docs repo.
>>> 
>>> Is there a preferred / recommended / list of continuous testing 
>>> environments?
>>> 
>>> Is there a preferred / rapid way to handle Raku modules?
>>> 
>>> Regards,
>>> 
>>> Richard
>>> 
>>> 
>>> 
>>> -- 
>>> Fernando Santagata
>> 
>> 
>> -- 
>> Fernando Santagata
> 
> 
> -- 
> JJ

Re: update: meta data in Pod Block

[Richard Hainsworth]

Rakudo v2031.03 

Have I been asleep for a decade?

My Rakudo is v2021.12.

I'm not sure what context you are using, so why you are getting errors. 
POD6 commands do not work with REPL.


Here is the contents of a test file, which can be run with 'prove6 -v 
t/config.t' assuming the following are in 't/config.t'


The first three tests are in the Roast test suite and have been passing 
for years. I'm not aware they fail with any version of Rakudo.


The =defn test is related to my earlier email and shows that Definition 
lists cannot contain embedded POD6.



use Test;
plan 8;
my $p = 0;
my $r;

=begin pod =for DESCRIPTION :title = :authorBrown> :pubdate(2011) =end pod $r = $=pod[$p++].contents[0];

is $r.config, 'presentation template';
is $r.config, 'John Brown';
is $r.config, 2011;

=begin pod =for head1 :attr('key') :battr This is a working header 
=for head1 = :attr('key') :battr
This is what is being tested =defn Bad boy When you Bto include 
formating. Outside a definition list Bis treated differently. 
=end pod $r = $=pod[$p++];


is $r.contents[0].config, 'key', "'attr' is found in first header config";
is $r.contents[1].config, 'key', "'attr' is found in second header 
config";
isa-ok $r.contents[2], Pod::Defn, 'third item is a definition list';
isa-ok $r.contents[3].contents[1], Pod::FormattingCode, 'Ordinary paragraph 
contents has a Formatting Code';
todo 1;
isa-ok $r.contents[2].contents[1], Pod::FormattingCode, 'Definition contents 
has a Formatting Code';

done-testing;

Regards

Richard

On 02/01/2022 23:00, Ralph Mellor wrote:

On Sun, Jan 2, 2022 at 8:00 PM Richard Hainsworth
 wrote:

What does not work (in that 'newkey' is not found in the 'config' part
of the Pod::Block returned by $=pod

=head1 :key

= :newkey

I just tried that with Rakudo v2031.03 and got a compile
time error for this line:

= :newkey

Preceding context expects a term, but found infix = instead.
Did you make a mistake in Pod syntax?


But this does work, and it is a passing test in the Roast suite

=head1 :key

=   :newkey

For me I got the same error as above. This was so for anywhere from
zero to 10 spaces between the `=` and `:newkey`.


However, the intent of S26, as stated in the raku documentation, is for
the first variant above to work, and also for the following case

(this being what is intended as a virtual margin):

   =head1 :newkey

   = :newkey

For me I got the same error (for the `  = :newkey` line).


But this does works (and it seems to be two virtual margins)

   =head1 :key

   =   :newkey

Same error.

My results vs yours suggest someone has changed this behavior
between v2021.3 and your version.

Having looked at commits to Grammar.nqp since 2021.03 I don't
understand how that can be if you're using a later version.


I have looked at the grammar for this situation, which I think is at

https://github.com/finanalyst/rakudo/blob/ae2bf80dae74986c75f9a610f7a25cc1f6cdbb36/src/Perl6/Grammar.nqp#L5093

Blame suggests that code was.last affected (merely moved) 3 years ago:

https://github.com/finanalyst/rakudo/blame/ae2bf80dae74986c75f9a610f7a25cc1f6cdbb36/src/Perl6/Grammar.nqp#L5093


but I cannot see why the actual behaviour occurs, but the documented
(apparently intended) behaviour does not.

The behaviour change and blame suggests it's not due to that line.



Alternatively I am utterly confused.

Either way, happy new year. :)

--
love, ralph

Re: shell to raku

[Andy Bach]

> raku is now my tool of choice when
* manipulexity is important

I had to look it up
Larry Wall: Manipulexity and Whipuptitude - Fortune
If you were a Unix programmer you either programmed in C or shell. And there 
really wasn't much in between. There were these little languages that we used 
on top of shell, but that was the big divide. The big revelation that hatched 
Perl, as it were, was that this opened up into a two-dimensional space. And C 
was good at something I like to call manipulexity, that is the manipulation of 
complex things. While shell was good at something else which I call 
whipuptitude, the aptitude for whipping things up.

So Perl was hatched. As a small egg. That was Perl 1. And it was designed from 
the very beginning to evolve. The fact that we put sigils in front of the 
variables meant that the namespaces were protected from new keywords. And that 
was intentional, so we could evolve the language fairly rapidly without 
impacting.

And it evolved… And it evolved… And finally we got to Perl 5. And… So… Perhaps 
the Perl 6 slogan should be "All Your Paradigms Are Belong To Us". We'll get to 
that.
https://www.shlomifish.org/humour/fortunes/show.cgi?id=larry-wall-big-divide

From: Marc Chantreux 
Sent: Friday, December 31, 2021 7:43 AM
To: Wesley Peng 
Cc: perl6-users 
Subject: shell to raku

CAUTION - EXTERNAL:


Le Fri, Dec 31, 2021 at 01:20:45PM +, Wesley Peng a écrit :
> Replacing Bash scripts with Raku? That’s an interesting thing

Well ... replacing bash is always a good thing but raku is not
always the rhs of the substitution (could be dash/mksh/rc,
make/mk, C, ...).

raku is now my tool of choice when

* manipulexity is important
* performance is not

The way i see the talk is:

* show how to do simple things ($PATH vs $RAKULIB, ...)
* show similarities (if you're confortable with feature X,
  then raku's Y will makes you happy)
* show some pro and cons (Shell vs Raku)

> Can Linux be shipped with Raku by default?

apt install rakudo

Is possible from the stable debian but i don't think it's a good idea
to have it as part of the default install (in a sense that there should
be nothing bigger than dash/busybox/toybox in the default install).
CAUTION - EXTERNAL EMAIL: This email originated outside the Judiciary. Exercise 
caution when opening attachments or clicking on links.