Re: [grpc-io] Slow gRPC communication with large file in Python

[Josh Parks]

Indeed it does seem to have to do with the M1 mac I'm developing on... I 
just ran it on my windows box and got the following:

(ve-pas) C:\Users\Josh\Code\grpc_min>python grpc_client.py 
 
0.3099939823150635s 18506294 photons in 1008640 bins   
 
0.2660081386566162s 18506294 photons in 1008640 bins   

Is there a way to fix this? This slow of speed is definitely going to 
inhibit development on my laptop...
On Tuesday, February 8, 2022 at 11:26:22 AM UTC-8 Josh Parks wrote:

> Hi all, 
>
> I'm on a Macbook M1Max (perhaps it's the architecture causing the issue?), 
> have 64 GB of RAM as well...
> Sorry for the error in the requirements,  I used `pyreqs` and that seems 
> to mess up...
>
> Running using pyenv virtual environment
>
> Also, added cprofile outputs for server and client to repo
>
>
> On Tuesday, February 8, 2022 at 10:46:03 AM UTC-8 rbel...@google.com 
> wrote:
>
>> Josh,
>>
>> I don't think I'm able to reproduce with your repo. I'm getting something 
>> like 0.2s on my desktop:
>>
>> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
>> 0.28313207626342773s18506294 photons in 1008640 bins
>> 0.14323067665100098s18506294 photons in 1008640 bins
>> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
>> 0.23985695838928223s18506294 photons in 1008640 bins
>> 0.13980460166931152s18506294 photons in 1008640 bins
>>
>> Also, your requirements.txt includes "grpc=1.0.0". I'm assuming this is 
>> just an typo. I used "grpcio".
>>
>> Maybe try running cProfile to generate a profile of the repro on your 
>> machine and sharing that here?
>>
>> On Tue, Feb 8, 2022 at 10:34 AM Josh Parks  wrote:
>>
>>> I'm trying to do a large array transfer (10-50MB) over gRPC in python 
>>> and it's quite slow (5-10 seconds, both client and server on localhost). 
>>> I've tried both streaming and unary requests, and they both seem to run 
>>> slowly.
>>>
>>> For more details/conversation, here's the stackoveflow question: 
>>> https://stackoverflow.com/questions/70993553/grpc-slow-serialization-on-large-dataset
>>>
>>> And for the minimum reproducible example: 
>>> https://github.com/parksj10/grpc_min
>>>
>>> Any help/guidance much appreciated!!!
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to grpc-io+u...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/grpc-io/b713e084-a3d9-46e3-aae2-8501164ca449n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/b3ec83ec-5dae-4bc4-9770-eb99b960021cn%40googlegroups.com.

Re: [grpc-io] Slow gRPC communication with large file in Python

[Josh Parks]

Hi all, 

I'm on a Macbook M1Max (perhaps it's the architecture causing the issue?), 
have 64 GB of RAM as well...
Sorry for the error in the requirements,  I used `pyreqs` and that seems to 
mess up...

Running using pyenv virtual environment

Also, added cprofile outputs for server and client to repo


On Tuesday, February 8, 2022 at 10:46:03 AM UTC-8 rbel...@google.com wrote:

> Josh,
>
> I don't think I'm able to reproduce with your repo. I'm getting something 
> like 0.2s on my desktop:
>
> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
> 0.28313207626342773s18506294 photons in 1008640 bins
> 0.14323067665100098s18506294 photons in 1008640 bins
> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
> 0.23985695838928223s18506294 photons in 1008640 bins
> 0.13980460166931152s18506294 photons in 1008640 bins
>
> Also, your requirements.txt includes "grpc=1.0.0". I'm assuming this is 
> just an typo. I used "grpcio".
>
> Maybe try running cProfile to generate a profile of the repro on your 
> machine and sharing that here?
>
> On Tue, Feb 8, 2022 at 10:34 AM Josh Parks  wrote:
>
>> I'm trying to do a large array transfer (10-50MB) over gRPC in python and 
>> it's quite slow (5-10 seconds, both client and server on localhost). I've 
>> tried both streaming and unary requests, and they both seem to run slowly.
>>
>> For more details/conversation, here's the stackoveflow question: 
>> https://stackoverflow.com/questions/70993553/grpc-slow-serialization-on-large-dataset
>>
>> And for the minimum reproducible example: 
>> https://github.com/parksj10/grpc_min
>>
>> Any help/guidance much appreciated!!!
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/b713e084-a3d9-46e3-aae2-8501164ca449n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/90a78eb7-5840-4647-84c0-2123ebb0cf34n%40googlegroups.com.

Re: [grpc-io] Slow gRPC communication with large file in Python

['Lidi Zheng' via grpc.io]

Windows tends to have slower "localhost" transport comparing to Linux. 
Josh, can you share your deployment setup?

The answer in SO is also helpful, streaming calls perform better in terms 
of throughput.

On Tuesday, February 8, 2022 at 10:46:03 AM UTC-8 Richard Belleville wrote:

> Josh,
>
> I don't think I'm able to reproduce with your repo. I'm getting something 
> like 0.2s on my desktop:
>
> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
> 0.28313207626342773s18506294 photons in 1008640 bins
> 0.14323067665100098s18506294 photons in 1008640 bins
> (venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py 
> 0.23985695838928223s18506294 photons in 1008640 bins
> 0.13980460166931152s18506294 photons in 1008640 bins
>
> Also, your requirements.txt includes "grpc=1.0.0". I'm assuming this is 
> just an typo. I used "grpcio".
>
> Maybe try running cProfile to generate a profile of the repro on your 
> machine and sharing that here?
>
> On Tue, Feb 8, 2022 at 10:34 AM Josh Parks  wrote:
>
>> I'm trying to do a large array transfer (10-50MB) over gRPC in python and 
>> it's quite slow (5-10 seconds, both client and server on localhost). I've 
>> tried both streaming and unary requests, and they both seem to run slowly.
>>
>> For more details/conversation, here's the stackoveflow question: 
>> https://stackoverflow.com/questions/70993553/grpc-slow-serialization-on-large-dataset
>>
>> And for the minimum reproducible example: 
>> https://github.com/parksj10/grpc_min
>>
>> Any help/guidance much appreciated!!!
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/b713e084-a3d9-46e3-aae2-8501164ca449n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/38e794a0-bd37-4640-8fe5-e24e1eeca0e7n%40googlegroups.com.

Re: [grpc-io] Slow gRPC communication with large file in Python

['Richard Belleville' via grpc.io]

Josh,

I don't think I'm able to reproduce with your repo. I'm getting something
like 0.2s on my desktop:

(venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py
0.28313207626342773s18506294 photons in 1008640 bins
0.14323067665100098s18506294 photons in 1008640 bins
(venv) rbellevi@rbell:~/Dev/tmp/grpc_min$ python3 grpc_client.py
0.23985695838928223s18506294 photons in 1008640 bins
0.13980460166931152s18506294 photons in 1008640 bins

Also, your requirements.txt includes "grpc=1.0.0". I'm assuming this is
just an typo. I used "grpcio".

Maybe try running cProfile to generate a profile of the repro on your
machine and sharing that here?

On Tue, Feb 8, 2022 at 10:34 AM Josh Parks  wrote:

> I'm trying to do a large array transfer (10-50MB) over gRPC in python and
> it's quite slow (5-10 seconds, both client and server on localhost). I've
> tried both streaming and unary requests, and they both seem to run slowly.
>
> For more details/conversation, here's the stackoveflow question:
> https://stackoverflow.com/questions/70993553/grpc-slow-serialization-on-large-dataset
>
> And for the minimum reproducible example:
> https://github.com/parksj10/grpc_min
>
> Any help/guidance much appreciated!!!
>
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/b713e084-a3d9-46e3-aae2-8501164ca449n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAOew_sFSgyu9AohYy%2BzbH9N%3DhmXXW_u0MP%3DNXW1wyhZL9UM7UQ%40mail.gmail.com.

[grpc-io] Slow gRPC communication with large file in Python

[Josh Parks]

I'm trying to do a large array transfer (10-50MB) over gRPC in python and 
it's quite slow (5-10 seconds, both client and server on localhost). I've 
tried both streaming and unary requests, and they both seem to run slowly.

For more details/conversation, here's the stackoveflow 
question: 
https://stackoverflow.com/questions/70993553/grpc-slow-serialization-on-large-dataset

And for the minimum reproducible 
example: https://github.com/parksj10/grpc_min

Any help/guidance much appreciated!!!


-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/b713e084-a3d9-46e3-aae2-8501164ca449n%40googlegroups.com.