Re: gRPC in D good idea for GSOC 2018?

[Timothee Cour via Digitalmars-d]

for grpc, we should add to dproto (which is pretty good protobuf
library for D but lacks grpc) instead of starting from scratch, see
https://github.com/msoucy/dproto/issues/113 [your advice/opinions on
integrating with grpc?]


On Mon, Jan 22, 2018 at 12:24 PM, Adrian Matoga via Digitalmars-d
 wrote:
> On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
>>
>> I know a project where D could benefit from gRPC in D, which is not among
>> the supported languages:
>>
>>   https://grpc.io/docs/
>>
>> Do you think gRPC support is worth adding to GSOC 2018 ideas?
>>
>>   https://wiki.dlang.org/GSOC_2018_Ideas
>>
>> Ali
>
>
> I can share a fresh experience from mentoring a student in a similar (also
> RPC) project last summer. We built native D-Bus bindings in D based on
> libasync. The student had had no previous experience with D or RPC, and
> within ~2.5 months of focused work she implemented the following:
>
> 1. (de)serialization of all D-Bus data types, including the use of
> compile-time reflection to recursively marshall structs, arrays, and
> variants. Except Variant, for which we decided to make our own
> D-Bus-specific tagged union type, all other D-Bus types are mapped to
> built-in D types.
> 2. A class to connect to the bus via libasync sockets, read the incoming
> messages and dispatch them to the registered handlers, and send messages to
> the bus.
> 3. Proxy (client) and server class templates that generate all the code
> necessary to make the remote calls look almost like local ones (the return
> value/out arguments are passed to a delegate that handles the output instead
> of being returned synchronously).
>
> So, more or less an equivalent of vibe.d's REST interface generator, only
> with fewer customization points.
>
> There were still some opportunities for refactorings and optimizations, so I
> wouldn't consider it production ready. Also, some planned features weren't
> implemented, such as a more convenient way for handling signals or allowing
> transports other than unix sockets on libasync. On the other hand, what we
> have is almost 100% covered with unit tests. This not only made adding
> successive layers quite pleasant, as little (if any) debugging of previously
> written stuff was ever necessary, but also helps to keep the stuff working
> as we modify it.
>
> Based on my experience, I'd say that implementing gRPC may be of a right
> size for a GSoC project, as long as you split it into smaller
> components/layers, prioritize them, and focus on having at least the basic
> stuff usable and tested, instead of expecting it to cover all usage cases
> and be heavily optimized.
>

Re: gRPC in D good idea for GSOC 2018?

[Adrian Matoga via Digitalmars-d]

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which is 
not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


I can share a fresh experience from mentoring a student in a 
similar (also RPC) project last summer. We built native D-Bus 
bindings in D based on libasync. The student had had no previous 
experience with D or RPC, and within ~2.5 months of focused work 
she implemented the following:


1. (de)serialization of all D-Bus data types, including the use 
of compile-time reflection to recursively marshall structs, 
arrays, and variants. Except Variant, for which we decided to 
make our own D-Bus-specific tagged union type, all other D-Bus 
types are mapped to built-in D types.
2. A class to connect to the bus via libasync sockets, read the 
incoming messages and dispatch them to the registered handlers, 
and send messages to the bus.
3. Proxy (client) and server class templates that generate all 
the code necessary to make the remote calls look almost like 
local ones (the return value/out arguments are passed to a 
delegate that handles the output instead of being returned 
synchronously).


So, more or less an equivalent of vibe.d's REST interface 
generator, only with fewer customization points.


There were still some opportunities for refactorings and 
optimizations, so I wouldn't consider it production ready. Also, 
some planned features weren't implemented, such as a more 
convenient way for handling signals or allowing transports other 
than unix sockets on libasync. On the other hand, what we have is 
almost 100% covered with unit tests. This not only made adding 
successive layers quite pleasant, as little (if any) debugging of 
previously written stuff was ever necessary, but also helps to 
keep the stuff working as we modify it.


Based on my experience, I'd say that implementing gRPC may be of 
a right size for a GSoC project, as long as you split it into 
smaller components/layers, prioritize them, and focus on having 
at least the basic stuff usable and tested, instead of expecting 
it to cover all usage cases and be heavily optimized.

Re: gRPC in D good idea for GSOC 2018?

[Andrew Benton via Digitalmars-d]

On Monday, 22 January 2018 at 05:54:31 UTC, Dmitry Olshansky 
wrote:

On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which 
is not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


An http/2 and gRPC solutions is probably necessary with tools 
like linkerd, envoy, and istio if D wants to be competitive in 
service mesh and distributed applications.


http/2 and/or gRPC are both excellent ideas for GSoC 2018.


I would consider them awful in a sense that there is no 
foundation to build them on. At best it will be a self-serving 
artifact poorly fitting with anything else.


I think it would likely be built on top of vibe or collie instead 
of being standalone, but I see your point.



There is not even a standard way on handling IO as of yet.
Basically do we want fiber-aware IO or blocking IO or explicit 
async with future/promise?


You're right.  It's definitely a problem.  Given ease of use and 
the learning curve, I'd be in favor of fiber-aware IO or 
async/await types.



Do we even have protobuf package?


We have at least two.  dproto is a package that supports proto2.  
I've used it before and it's really nice.  There's also the 
eponymous protobuf package, which integrates with protoc and can 
also do proto3.  I haven't used it yet.

Re: gRPC in D good idea for GSOC 2018?

[Timothee Cour via Digitalmars-d]

> Do we even have protobuf package?

https://github.com/msoucy/dproto
it could receive some attention, there are pending issues

for RPC I've been using msgpackrpc since no gRPC was available. But
would be nice to have gRPC.

NOTE: capnproto is a very interesting newer alternative to protobuf;
https://github.com/capnproto/capnproto-dlang shows:
Missing RPC part of Cap'n Proto.
helping out capnproto project (esp around RPC) could be another idea.

we definitely need a good way to do RPC in D, otherwise hard to
integrate D with other services.

> I would consider them awful in a sense that there is no foundation to build 
> them on. At best it will be a self-serving artifact poorly fitting with 
> anything else.

But it would enable using D in places that were not previously
possible (integrating with services); we could imaging providing a
(semi) stable interface for grpc in D code and change implementation
to use better foundations later




On Sun, Jan 21, 2018 at 9:54 PM, Dmitry Olshansky via Digitalmars-d
 wrote:
> On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:
>>
>> On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
>>>
>>> I know a project where D could benefit from gRPC in D, which is not among
>>> the supported languages:
>>>
>>>   https://grpc.io/docs/
>>>
>>> Do you think gRPC support is worth adding to GSOC 2018 ideas?
>>>
>>>   https://wiki.dlang.org/GSOC_2018_Ideas
>>>
>>> Ali
>>
>>
>> An http/2 and gRPC solutions is probably necessary with tools like
>> linkerd, envoy, and istio if D wants to be competitive in service mesh and
>> distributed applications.
>>
>> http/2 and/or gRPC are both excellent ideas for GSoC 2018.
>
>
> I would consider them awful in a sense that there is no foundation to build
> them on. At best it will be a self-serving artifact poorly fitting with
> anything else.
>
> There is not even a standard way on handling IO as of yet.
> Basically do we want fiber-aware IO or blocking IO or explicit async with
> future/promise?
>
> Do we even have protobuf package?
>
>

Re: gRPC in D good idea for GSOC 2018?

[Dmitry Olshansky via Digitalmars-d]

On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which 
is not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


An http/2 and gRPC solutions is probably necessary with tools 
like linkerd, envoy, and istio if D wants to be competitive in 
service mesh and distributed applications.


http/2 and/or gRPC are both excellent ideas for GSoC 2018.


I would consider them awful in a sense that there is no 
foundation to build them on. At best it will be a self-serving 
artifact poorly fitting with anything else.


There is not even a standard way on handling IO as of yet.
Basically do we want fiber-aware IO or blocking IO or explicit 
async with future/promise?


Do we even have protobuf package?

Re: gRPC in D good idea for GSOC 2018?

[Andrew Benton via Digitalmars-d]

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which is 
not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


An http/2 and gRPC solutions is probably necessary with tools 
like linkerd, envoy, and istio if D wants to be competitive in 
service mesh and distributed applications.


http/2 and/or gRPC are both excellent ideas for GSoC 2018.

Re: gRPC in D good idea for GSOC 2018?

[Joakim via Digitalmars-d]

On Friday, 19 January 2018 at 10:14:15 UTC, yawniek wrote:

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which 
is not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


Problem here is that D lacks a proper http/2 stack.
unfortunately due to time constraints my libh2o binding project 
got stuck.


Have you seen Etienne's libhttp2?

http://code.dlang.org/packages/libhttp2

Re: gRPC in D good idea for GSOC 2018?

[yawniek via Digitalmars-d]

On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:

On 01/19/2018 02:14 AM, yawniek wrote:
Could you please summarize what needs to be done and in what 
order. I would be happy to take active part in this effort.


well,  my personal opinion this should be designed/decided on a 
green field and not again make the mistake to try to include too 
much that is just laying around. i think vibe.d's eventcore may 
(or may not) be too opinionated and providing too much:


1a. define an overal strategy how/where protocols should be 
implemented and whether to focus on speed or "elegant" 
implementations.


1b. define how high performance servers should ideally be 
implemented and how that works together with D's concurrency 
models. plus provide the necessary things such as streams/ranges. 
ideally endorse/include/create one primary eventloop 
implementation or interface. possibly rusts tokyo.rs could be an 
inspiration?

https://tokio.rs/docs/getting-started/tls/

2a. create reference implementations for a http parser or e.g. 
port something like picohttpparser 
https://github.com/h2o/picohttpparser but this should adhere to 
the approach defined in 1.



3. implement crypto functions. e.g. port picotls ( 
https://github.com/h2o/picotls ) but also provide/maintain proper 
openssl bindings.


4. implement http/2  grpc

also, think about what happens when the world starts switching to 
quic, the ietf wg makes good progress from what i saw.



a totally different approach/decision would be to say, that D is 
a glue language and one should use C libraries (libuv, openssl, 
picohttpparser). but then i think tooling for integrating those 
should be improved/standardized (e.g. package manager, binding 
automation).

Re: gRPC in D good idea for GSOC 2018?

[aberba via Digitalmars-d]

On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:

On 01/19/2018 02:14 AM, yawniek wrote:

> Just look how
> beautiful Golangs protocol implementations are and the whole
ecosystem
> that focused effort spawned.


I have said this here before.

I agree and do have first-hand experience with that but I'm not 
experienced enough in this field to come up with a modern 
solution. If it were left to me, I would start looking at Go 
and Python solutions but I fear I wouldn't see design decisions 
that could have been better.


Could you please summarize what needs to be done and in what 
order. I would be happy to take active part in this effort.


Thank you,
Ali

Re: gRPC in D good idea for GSOC 2018?

[Ali Çehreli via Digitalmars-d]

On 01/19/2018 02:14 AM, yawniek wrote:

> Just look how
> beautiful Golangs protocol implementations are and the whole ecosystem
> that focused effort spawned.

I agree and do have first-hand experience with that but I'm not 
experienced enough in this field to come up with a modern solution. If 
it were left to me, I would start looking at Go and Python solutions but 
I fear I wouldn't see design decisions that could have been better.


Could you please summarize what needs to be done and in what order. I 
would be happy to take active part in this effort.


Thank you,
Ali

Re: gRPC in D good idea for GSOC 2018?

[yawniek via Digitalmars-d]

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which is 
not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


Problem here is that D lacks a proper http/2 stack.
unfortunately due to time constraints my libh2o binding project 
got stuck.


Imo the real problem with D is still that phobos does not provide 
the important building blocks for today modern infrastrucutre:


- fast and standardized eventloop compatible with Ds concurrency 
primitives

- network protocol libs/ bindings ( http, http/2, tls, dns )

until there is effort to do this D will continue to be an 
ecosystem of hacks.
I really like vibe.d but it comes with a lot of problems and it 
has the focus of providing a usable web framework and not 
implementing protocols in a fast and reusable way.


so as long as "management" does not decide that internet 
protocols is something important in D nothing useful will come 
out. Just look how beautiful Golangs protocol implementations are 
and the whole ecosystem that focused effort spawned.

Re: gRPC in D good idea for GSOC 2018?

[Seb via Digitalmars-d]

On Tuesday, 16 January 2018 at 17:58:40 UTC, jmh530 wrote:

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which 
is not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


Also, the deadline is in 1 week.


Yes it's a good idea for GSoC18 and we still accept ideas. Reply 
here if you have more ideas ;-)

Re: gRPC in D good idea for GSOC 2018?

[jmh530 via Digitalmars-d]

On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
I know a project where D could benefit from gRPC in D, which is 
not among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali


Also, the deadline is in 1 week.

gRPC in D good idea for GSOC 2018?

[Ali Çehreli via Digitalmars-d]

I know a project where D could benefit from gRPC in D, which is not 
among the supported languages:


  https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

  https://wiki.dlang.org/GSOC_2018_Ideas

Ali