[ https://issues.apache.org/jira/browse/MESOS-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970063#comment-14970063 ]
Joseph Wu commented on MESOS-3771: ---------------------------------- Sync'd with BenH and [~bmahler] about this (offline). The proposed solution is the following: # None of the state endpoints should be dumping out the binary {{data}} fields in the first place. This includes {{ExecutorInfo}} ([dumped by master|https://github.com/apache/mesos/blob/master/src/common/http.cpp#L317]) and {{TaskInfo}} ([dumped by agent|https://github.com/apache/mesos/blob/master/src/slave/http.cpp#L103]). #* These fields should be removed from the output entirely. Existing frameworks should not be relying on this information. [~stevenschlansker], can you confirm this with Spark? #* We can easily back-port this patch, if absolutely necessary. # Master should not be storing the {{data}} fields from {{ExecutorInfo}}. We currently [store the entire object|https://github.com/apache/mesos/blob/master/src/master/master.hpp#L262-L271], which means master would be as high risk of OOM-ing if a bunch of executors were started with big {{data}} blobs. #* Master should scrub out unneeded bloat from {{ExecutorInfo}} before storing it. #* We can use an alternate internal object, like we do for {{TaskInfo}} vs {{Task}}; see [this|https://github.com/apache/mesos/blob/master/src/messages/messages.proto#L39-L41]. > Mesos JSON API creates invalid JSON due to lack of binary data / non-ASCII > handling > ----------------------------------------------------------------------------------- > > Key: MESOS-3771 > URL: https://issues.apache.org/jira/browse/MESOS-3771 > Project: Mesos > Issue Type: Bug > Components: HTTP API > Affects Versions: 0.24.1, 0.26.0 > Reporter: Steven Schlansker > Assignee: Joseph Wu > Priority: Critical > Labels: mesosphere > > Spark encodes some binary data into the ExecutorInfo.data field. This field > is sent as a "bytes" Protobuf value, which can have arbitrary non-UTF8 data. > If you have such a field, it seems that it is splatted out into JSON without > any regards to proper character encoding: > {code} > 0006b0b0 2e 73 70 61 72 6b 2e 65 78 65 63 75 74 6f 72 2e |.spark.executor.| > 0006b0c0 4d 65 73 6f 73 45 78 65 63 75 74 6f 72 42 61 63 |MesosExecutorBac| > 0006b0d0 6b 65 6e 64 22 7d 2c 22 64 61 74 61 22 3a 22 ac |kend"},"data":".| > 0006b0e0 ed 5c 75 30 30 30 30 5c 75 30 30 30 35 75 72 5c |.\u0000\u0005ur\| > 0006b0f0 75 30 30 30 30 5c 75 30 30 30 66 5b 4c 73 63 61 |u0000\u000f[Lsca| > 0006b100 6c 61 2e 54 75 70 6c 65 32 3b 2e cc 5c 75 30 30 |la.Tuple2;..\u00| > {code} > I suspect this is because the HTTP api emits the executorInfo.data directly: > {code} > JSON::Object model(const ExecutorInfo& executorInfo) > { > JSON::Object object; > object.values["executor_id"] = executorInfo.executor_id().value(); > object.values["name"] = executorInfo.name(); > object.values["data"] = executorInfo.data(); > object.values["framework_id"] = executorInfo.framework_id().value(); > object.values["command"] = model(executorInfo.command()); > object.values["resources"] = model(executorInfo.resources()); > return object; > } > {code} > I think this may be because the custom JSON processing library in stout seems > to not have any idea of what a byte array is. I'm guessing that some > implicit conversion makes it get written as a String instead, but: > {code} > inline std::ostream& operator<<(std::ostream& out, const String& string) > { > // TODO(benh): This escaping DOES NOT handle unicode, it encodes as ASCII. > // See RFC4627 for the JSON string specificiation. > return out << picojson::value(string.value).serialize(); > } > {code} > Thank you for any assistance here. Our cluster is currently entirely down -- > the frameworks cannot handle parsing the invalid JSON produced (it is not > even valid utf-8) -- This message was sent by Atlassian JIRA (v6.3.4#6332)