[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026526#comment-16026526
]
Anand Mazumdar commented on MESOS-4210:
---
This shouldn't be a concern anymore after the upgrade to proto3 as part of
MESOS-7228 that supports message sizes up to 2GB. Marking it as resolved.
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
> Fix For: 1.4.0
>
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864632#comment-15864632
]
Santhosh Kumar Shanmugham commented on MESOS-4210:
--
I traced the message creation to this point in code
https://github.com/apache/mesos/blob/master/src/master/master.cpp#L7025
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864600#comment-15864600
]
Santhosh Kumar Shanmugham commented on MESOS-4210:
--
Looks like this issue more spread out across the codebase (comparing the 2
failure instances in the ticket). Mesos master has to consciously limit the
size of the messages.
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864596#comment-15864596
]
Santhosh Kumar Shanmugham commented on MESOS-4210:
--
Failure message in our case -
{code}
[libprotobuf ERROR google/protobuf/io/coded_stream.cc:180] A protocol message
was rejected because it was too big (more than 67108864 bytes). To increase
the limit (or to disable these warnings), see
CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stre
am.h.
F0213 21:33:57.658892 60996 sched.cpp:895] Check failed: offers.size() ==
pids.size() (32664 vs. 0)
*** Check failure stack trace: ***
@ 0x7f8d1b4d69bd (unknown)
@ 0x7f8d1b4d8750 (unknown)
@ 0x7f8d1b4d6582 (unknown)
@ 0x7f8d1b4d90e9 (unknown)
@ 0x7f8d1aaa646c (unknown)
@ 0x7f8d1aaa7df7 (unknown)
@ 0x7f8d1aa8ee4a (unknown)
@ 0x7f8d1aa9d109 (unknown)
@ 0x7f8d1b46e4e4 (unknown)
@ 0x7f8d1b46e827 (unknown)
@ 0x7f8e319b0220 (unknown)
@ 0x7f8e3355ddc5 start_thread
@ 0x7f8e32c62ced __clone
@ (nil) (unknown)
{code}
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269826#comment-15269826
]
Rob Russo commented on MESOS-4210:
--
This does affect our actual production use case. In the latest versions of
spark when using the parquet relation they attempt to speed up file discovery
by parallelizing it into multiple tasks. Unfortunately if using fine grained
mode they tend to do this using a few number of tasks (always seemed to be 8)
and the smaller the number of tasks, the larger the result that needs to be
sent back to the driver from each task to list the discovered files. With our
file setup this exceeds the max protobuf length once the matched files grows
into the tens of thousands (which is not that much for us).
For now we are getting around this using a custom built protobuf jar with a
higher limit set and building mesos with that jar supplied but it would be
great if we could at least have a option to adjust this limit as a mesos config
since the limit can be set at runtime.
I realize that protobufs are not designed to handle large messages but given
this very standard use case of spark on mesos it seems that there should be
some easy workaround here.
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4210) Investigate increasing protobuf protocol message size limit.
[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073156#comment-15073156
]
John Garcia commented on MESOS-4210:
This came up during a test, where the end-user was attempting to flood the
master with large messages. This test does not represent that user's actual
production or development usage of the product.
[The Google Developer's entro on Protobuf advises the
following:|https://developers.google.com/protocol-buffers/docs/techniques#streaming]
{quote}Protocol Buffers are not designed to handle large messages. As a general
rule of thumb, if you are dealing in messages larger than a megabyte each, it
may be time to consider an alternate strategy.{quote}
> Investigate increasing protobuf protocol message size limit.
>
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
