[grpc-io] Re: grpc streaming exceeds message limit of 4MB

[Joe Lin]

Thanks. It was a user error. I do have large message > 4M. Setting 
MaxCallSendMsgSize does the trick for me.

On Wednesday, March 14, 2018 at 1:48:43 PM UTC-7, Yuxuan Li wrote:
>
> Hi Joe,
>
> Buffering happens in lower level (transport layer) and shouldn't be able 
> to modify the message size.
>
> There are two places where the error you mentioned could happen, and both 
> in rpc_util.go. The first place (in func recvMsg()) is before we apply 
> decoding, where we compare the encoded message length to the limit. The 
> second place (in func recv()) is after we decode the message, where we 
> compare the decoded (decompressed if compression was enabled) message 
> length to the limit. It could be possible that your encoded message is 
> within 4MB (you said no more than 4k in length), but the decoded message is 
> much larger and beyond the limit.
>
> What you can probably do is:
> on client side, verify where does the error get generated, before the 
> decoding or after the decoding.
> on server side, print out the message size before and after encoding, and 
> see if it's larger than 4MB.
>
> Please let me know if it doesn't help and we can investigate more. And in 
> that case, it would be great if you can provide us with a reproducible 
> example.Thanks!
>
> Yuxuan
>
> On Sunday, March 11, 2018 at 9:44:00 PM UTC-7, Joe Lin wrote:
>>
>> I'm using gRPC streaming between a golang client and a golang server. I'm 
>> seeing quite a few streaming error such as the following on the client side:
>>
>> stream error:rpc error: code = ResourceExhausted desc = grpc: received 
>> message larger than max (14993559 vs. 4194304)
>>
>> I understand that gRPC does have a message limit size of 4MB on the 
>> receiving end. My usage is that client and the server establishes a stream 
>> and the server will feed the client with stream of objects during 
>> throughout the lifetime of client. So the server side streaming is defined 
>> as:
>>
>> func (s *server) FilterResult(req *filter.ResultStreamRequest, stream 
>> filter.FilterService_FilterResultServer) error {
>> // stream the data forever
>> for {
>> // wait for more data
>> result := waitData()
>> // stream the result, 
>> stream.Send(result)
>> }
>> return nil
>> }
>>
>> The size of each result object is no more than 4k in length. But quite 
>> often I would get the above error on the client side. It seems to me that 
>> the server buffered too much results before it sends to the client. 
>>
>> Thanks for any help.
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/ac532921-ff76-477d-a630-b70541395ca2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: grpc streaming exceeds message limit of 4MB

['Yuxuan Li' via grpc.io]

Hi Joe,

Buffering happens in lower level (transport layer) and shouldn't be able to 
modify the message size.

There are two places where the error you mentioned could happen, and both 
in rpc_util.go. The first place (in func recvMsg()) is before we apply 
decoding, where we compare the encoded message length to the limit. The 
second place (in func recv()) is after we decode the message, where we 
compare the decoded (decompressed if compression was enabled) message 
length to the limit. It could be possible that your encoded message is 
within 4MB (you said no more than 4k in length), but the decoded message is 
much larger and beyond the limit.

What you can probably do is:
on client side, verify where does the error get generated, before the 
decoding or after the decoding.
on server side, print out the message size before and after encoding, and 
see if it's larger than 4MB.

Please let me know if it doesn't help and we can investigate more. And in 
that case, it would be great if you can provide us with a reproducible 
example.Thanks!

Yuxuan

On Sunday, March 11, 2018 at 9:44:00 PM UTC-7, Joe Lin wrote:
>
> I'm using gRPC streaming between a golang client and a golang server. I'm 
> seeing quite a few streaming error such as the following on the client side:
>
> stream error:rpc error: code = ResourceExhausted desc = grpc: received 
> message larger than max (14993559 vs. 4194304)
>
> I understand that gRPC does have a message limit size of 4MB on the 
> receiving end. My usage is that client and the server establishes a stream 
> and the server will feed the client with stream of objects during 
> throughout the lifetime of client. So the server side streaming is defined 
> as:
>
> func (s *server) FilterResult(req *filter.ResultStreamRequest, stream 
> filter.FilterService_FilterResultServer) error {
> // stream the data forever
> for {
> // wait for more data
> result := waitData()
> // stream the result, 
> stream.Send(result)
> }
> return nil
> }
>
> The size of each result object is no more than 4k in length. But quite 
> often I would get the above error on the client side. It seems to me that 
> the server buffered too much results before it sends to the client. 
>
> Thanks for any help.
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/16d373fd-9a26-4e1e-9166-3a6f0b7c1cf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: Compiling with Emscripten to Web Assembly

['Wenbo Zhu' via grpc.io]

On Wed, Mar 14, 2018 at 12:56 PM, Matthew Avery 
wrote:

> I was able to compile grpc to asm.js using emscripten and run without
> errors, but I think you are correct, the client was not able to make a
> connection from the browser to the grpc server.  I'm not sure what the path
> forward is yet.  I am going to experiment with a c++ socket and see if
> webassembly will use websockets.  I saw a post on stackoverflow
> 
> that suggested this is the case
>
I would guess websocket calls are bridged over JS calls ..(for all the
security reasons)


>
>
> On Wednesday, March 14, 2018 at 3:45:47 PM UTC-4, Wenbo Zhu wrote:
>>
>> I worked on grpc/grpc-web, and I don't believe you are able to do any
>> native networking I/O from the browser, but if you have any pointer to
>> suggest otherwise, happy to have a look and assess the feasibility.
>>
>>
>> On Tuesday, March 13, 2018 at 6:25:53 AM UTC-7, Matthew Avery wrote:
>>>
>>> I did get grpc and protobuf  compiles to WASM but it is not working yet,
>>> even after enabling SharedArrayBuffer on my browser.  I'm getting a type
>>> exception on the first call to Atomics.store() that is trying to allocate
>>> memory for pthread support.
>>>
>>> On Friday, March 9, 2018 at 11:16:28 AM UTC-5, Matthew Avery wrote:

 I just started looking at this today.  Any progress on your end?

 On Friday, January 19, 2018 at 6:03:04 AM UTC-5, andro...@gmail.com
 wrote:
>
> Hi,
> I'm trying to compile grpc with Emscripten
>  to WebAssembly
>  - trying to
> get a C++-based grpc client working in the browser.
>
> Does anyone know if getting this working is even feasible and has
> anyone else tried it? I'm having issues compiling (mostly Protobuf) which 
> I
> think I'll be able to solve, but I'd like to know if the effort is worth 
> it.
>
 --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/b2a50b3a-fd71-4d75-b77c-8e9a75858fe1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAD3-0rPN%2B4pR%2BVJQCaiP%3DPn-cif2%2BHjJ%2BkLmYrj%2BhCp7xLBvXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: Compiling with Emscripten to Web Assembly

[Matthew Avery]

I was able to compile grpc to asm.js using emscripten and run without 
errors, but I think you are correct, the client was not able to make a 
connection from the browser to the grpc server.  I'm not sure what the path 
forward is yet.  I am going to experiment with a c++ socket and see if 
webassembly will use websockets.  I saw a post on stackoverflow 

 
that suggested this is the case

On Wednesday, March 14, 2018 at 3:45:47 PM UTC-4, Wenbo Zhu wrote:
>
> I worked on grpc/grpc-web, and I don't believe you are able to do any 
> native networking I/O from the browser, but if you have any pointer to 
> suggest otherwise, happy to have a look and assess the feasibility.
>
>
> On Tuesday, March 13, 2018 at 6:25:53 AM UTC-7, Matthew Avery wrote:
>>
>> I did get grpc and protobuf  compiles to WASM but it is not working yet, 
>> even after enabling SharedArrayBuffer on my browser.  I'm getting a type 
>> exception on the first call to Atomics.store() that is trying to allocate 
>> memory for pthread support.
>>
>> On Friday, March 9, 2018 at 11:16:28 AM UTC-5, Matthew Avery wrote:
>>>
>>> I just started looking at this today.  Any progress on your end?
>>>
>>> On Friday, January 19, 2018 at 6:03:04 AM UTC-5, andro...@gmail.com 
>>> wrote:

 Hi,
 I'm trying to compile grpc with Emscripten 
  to WebAssembly 
  - trying to get 
 a C++-based grpc client working in the browser. 

 Does anyone know if getting this working is even feasible and has 
 anyone else tried it? I'm having issues compiling (mostly Protobuf) which 
 I 
 think I'll be able to solve, but I'd like to know if the effort is worth 
 it.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/b2a50b3a-fd71-4d75-b77c-8e9a75858fe1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: Compiling with Emscripten to Web Assembly

['Wenbo Zhu' via grpc.io]

I worked on grpc/grpc-web, and I don't believe you are able to do any 
native networking I/O from the browser, but if you have any pointer to 
suggest otherwise, happy to have a look and assess the feasibility.


On Tuesday, March 13, 2018 at 6:25:53 AM UTC-7, Matthew Avery wrote:
>
> I did get grpc and protobuf  compiles to WASM but it is not working yet, 
> even after enabling SharedArrayBuffer on my browser.  I'm getting a type 
> exception on the first call to Atomics.store() that is trying to allocate 
> memory for pthread support.
>
> On Friday, March 9, 2018 at 11:16:28 AM UTC-5, Matthew Avery wrote:
>>
>> I just started looking at this today.  Any progress on your end?
>>
>> On Friday, January 19, 2018 at 6:03:04 AM UTC-5, andro...@gmail.com 
>> wrote:
>>>
>>> Hi,
>>> I'm trying to compile grpc with Emscripten 
>>>  to WebAssembly 
>>>  - trying to get 
>>> a C++-based grpc client working in the browser. 
>>>
>>> Does anyone know if getting this working is even feasible and has anyone 
>>> else tried it? I'm having issues compiling (mostly Protobuf) which I think 
>>> I'll be able to solve, but I'd like to know if the effort is worth it.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/f0f70531-392a-43be-9f2d-848e394d6bed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: GRPC using UDP

[Harish Patil]

Thanks Srini.
Looks like everyone seems to need grpc over udp but no work has been done
yet.

Harish

On Wed, Mar 14, 2018 at 10:26 AM, 'Srini Polavarapu' via grpc.io <
grpc-io@googlegroups.com> wrote:

> GRPC doesn't support UDP. Some discussion here
> .
>
> On Tuesday, March 13, 2018 at 11:50:55 AM UTC-7, Harish Patil wrote:
>>
>> Hi,
>> Do we have an existing implementations of GRPC using UDP as the
>> underlying transport protocol?
>> If so, please provide the pointers.
>>
>> Thanks.
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "grpc.io" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/grpc-io/MJxQ0z7AQV8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/d138c535-f9f1-4262-a38d-04d0fac50cb1%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CALABnay-mp0AsLE9sPmrmR8Pgw9U4YjKkzMfh9e%2BiZJL6iFrgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: gRPC Performance

[ndun]

Hi Sree, 

any updates? we also found that some data is 
missing: 
https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528

Nan

On Thursday, March 8, 2018 at 8:36:04 PM UTC-8, Sree Kuchibhotla wrote:
>
> Thanks for bringing this to our attention. The numbers look very low for 
> C++ (especially for unary, its way too low). We are investigating..
>
> thanks,
> Sree
>
>
> On Wed, Mar 7, 2018 at 4:38 PM 'Matt Kwong' via grpc.io <
> grp...@googlegroups.com > wrote:
>
>> +grpc-io
>>
>> Unfortunately, I'm not the best person to answer this question. Adding 
>> grpc-io, so that someone working on C++ or Java performance can answer 
>> this. 
>>
>> On Wed, Mar 7, 2018 at 4:10 PM, Nan Dun > > wrote:
>>
>>> Hi Matt,
>>>
>>>  
>>>
>>> This is Nan from Quantcast. Our team at Quantcast is looking for gPRC 
>>> performance numbers and we are glad to find this very insightful chart: 
>>> https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5636470266134528
>>>
>>>  
>>>
>>> However, we are a bit confused, in throughput QPS charts (both unary and 
>>> streaming), why Java has much better performance (almost 2x) than C++? We 
>>> do appreciate if you can give us some hints about this. Thank you!
>>>
>>>  
>>>
>>> Sincerely,
>>>
>>> Nan
>>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com .
>> To post to this group, send email to grp...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/CALc3v%2BOoOwcvFdLuPG7%3DAvaO%2BmKQhT29Kxvb4WthQHLi5Dv9eg%40mail.gmail.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/3e139cb3-6e6b-4ed5-8c3c-aee80d48fa99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: Handshake in TLS

['Srini Polavarapu' via grpc.io]

Please specify the language you are using and whether it is flex or 
standard AppEngine.

On Monday, March 12, 2018 at 10:21:03 AM UTC-7, Baojun Xu wrote:
>
> Hi,
>
> We are calling our grpc endpoint with TLS from app engine. I am wondering 
> how often does a TLS handshake happen in case we are calling from app 
> engine, given our client stub is provided as a Singleton? Especially we 
> don't know how app engine will deal with the client stub.
>
> Best,
> Baojun
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/ed8d2019-e541-4a4b-a65f-ccf14b054902%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: GRPC using UDP

['Srini Polavarapu' via grpc.io]

GRPC doesn't support UDP. Some discussion here 
.

On Tuesday, March 13, 2018 at 11:50:55 AM UTC-7, Harish Patil wrote:
>
> Hi,
> Do we have an existing implementations of GRPC using UDP as the underlying 
> transport protocol? 
> If so, please provide the pointers.
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/d138c535-f9f1-4262-a38d-04d0fac50cb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] [GSoC] Assistance with gRPC Python Ideas

[Sword Chen]

Thanks for your reply.

> Cool! How long? With what programming languages?

 I have used gRPC for several weeks, and I have been building the stream 
algorithm test environment these weeks with gRPC in Python.  

> This sounds like it's going in the right direction - something that I 
> think we sort of expect is that there will be a certain incrementality to 
> static typing of gRPC Python, and having to add stubs describing the types 
> of lower-level systems so that the upper-level systems can have type 
> annotations added and checked is not a surprise. mypy and pytype are both 
> young enough, and gRPC Python is weird enough, that it is certainly not our 
> expectation that we can just throw either tool at our codebase and it will 
> figure types out without any further assistance. :-)

 You mean the gradual typing, which allows one to annotate only part of a 
program, thus leverage desirable aspects of both dynamic and static typing. 
PEP483 .  
If we don't use the other tools, I have tried some basic type check by 
using annotations and decorator PEP 3107 
, here is the decorator 
function which can do the type check:
from inspect import signature
from functools import wraps


def typeassert(*ty_args, **ty_kwargs):
def decorate(func):


# Map function argument names to supplied types
sig = signature(func)
bound_types = sig.bind_partial(*ty_args, **ty_kwargs).arguments


@wraps(func)
def wrapper(*args, **kwargs):
bound_values = sig.bind(*args, **kwargs)
# Enforce type assertions across supplied arguments
for name, value in bound_values.arguments.items():
if name in bound_types:
if not isinstance(value, bound_types[name]):
raise TypeError(
'Argument {} must be {}'.format(name, 
bound_types[name])
)
return func(*args, **kwargs)
return wrapper
return decorate

Here is the example:
@typeassert(int,int z=int)
def spam(x, y, z=42):
print(x, y, z)

So as you said, if we don't throw other tools into the codebase, I am 
considering that we could use this method to help do the type check. But I 
still have some confusions:
1.  the idea is  "Support static type-checking of both gRPC Python itself 
and of code that uses gRPC Python."  when we use the gPRC python, we can 
add the decorator to our own function, but for gPRC Python itself, do we 
add type-checker to every defined function? this will 
2.  Does this need to support PEP 484 
 or just detect the 
errors using the gRPC Python? As I know, mypy verifies standard PEP 484 
 type hints. 
3. About the grpc structure, actually, I am not quite Clear, is there some 
document to help me understand it? 
These are my thoughts. Hope to get further guidance.
- Sword

在 2018年3月13日星期二 UTC+8上午3:59:15，Nathaniel Manista写道：
>
> On Sat, Mar 10, 2018 at 11:07 AM, Sword Chen  > wrote:
>
>> I am Sword Chen, a postgraduate student at Mid Sweden University.
>>
>
> Pleased to meet you!
>
> I have used the gPRC for some time.
>>
>
> Cool! How long? With what programming languages?
>
> So I would love to contribute to the project as part of GSoC. I have seen 
>> the given ideas, and I am quite interested in the "Support static 
>> type-checking of both gRPC Python" , but I can't find the relative open 
>> issue for this idea.  so I did some prework and I put them here.
>>
>> I have tried the Type Checking Tool Mypy, when I tested some file using 
>> the grpc module and I got this error " Cannot find module named 'grpc' " 
>> Then I went through with the Mypy document, Mypy can't do static 
>> type-checking of both gRPC Python itself and of the code that uses gRPC 
>> because Mypy can't find a stub for a grpc library module. and here is the 
>> information about Creating Stubs For Python Modules 
>> https://github.com/python/mypy/wiki/Creating-Stubs-For-Python-Modules. 
>> So what I need to do is to create the stubs for the grpc to help Mypy to 
>> detect the code. Am I working in the right direction now? It would be 
>> really helpful if someone can give me some guidance, so I can have some 
>> confidence to go on.
>>
>
> This sounds like it's going in the right direction - something that I 
> think we sort of expect is that there will be a certain incrementality to 
> static typing of gRPC Python, and having to add stubs describing the types 
> of lower-level systems so that the upper-level systems can have type 
> annotations added and checked is not a surprise. mypy and pytype are both 
> young enough, and gRPC Python is weird enough, that it is certainly not our 
> expectation that we can just throw either tool at our codebase and it will 
>