[ https://issues.apache.org/jira/browse/MESOS-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308222#comment-15308222 ]
Chris Pennello commented on MESOS-4642: --------------------------------------- At the very least, if we aren't going to modify how the code works, then at least document that {{files/read.json}} is _not_ guaranteed to return valid JSON. For example, a consequential, and not _too_ unreasonable workaround, is to ensure that your files are themselves UTF-8 encoded, and that would be a helpful thing to mention in the endpoint documentation. > Mesos Agent Json API can dump binary data from log files out as invalid JSON > ---------------------------------------------------------------------------- > > Key: MESOS-4642 > URL: https://issues.apache.org/jira/browse/MESOS-4642 > Project: Mesos > Issue Type: Bug > Components: json api, slave > Affects Versions: 0.27.0 > Reporter: Steven Schlansker > Priority: Critical > > One of our tasks accidentally started logging binary data to stderr. This > was not intentional and generally should not happen -- however, it causes > severe problems with the Mesos Agent "files/read.json" API, since it gladly > dumps this binary data out as invalid JSON. > {code} > # hexdump -C /path/to/task/stderr | tail > 0003d1f0 6f 6e 6e 65 63 74 69 6f 6e 0a 4e 45 54 3a 20 31 |onnection.NET: 1| > 0003d200 20 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 | onread ENOENT 2| > 0003d210 39 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 |95456 251 295707| > 0003d220 0a 01 00 00 00 00 00 00 ac 57 65 64 2c 20 31 30 |.........Wed, 10| > 0003d230 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 20 69 6e | Unrecognized in| > 0003d240 70 75 74 20 68 65 61 64 65 72 0a |put header.| > {code} > {code} > # curl > 'http://agent-host:5051/files/read.json?path=/path/to/task/stderr&offset=220443&length=90000&grep=' > | hexdump -C > 00007970 6e 65 63 74 69 6f 6e 5c 6e 4e 45 54 3a 20 31 20 |nection\nNET: 1 | > 00007980 6f 6e 72 65 61 64 20 45 4e 4f 45 4e 54 20 32 39 |onread ENOENT 29| > 00007990 35 34 35 36 20 32 35 31 20 32 39 35 37 30 37 5c |5456 251 295707\| > 000079a0 6e 5c 75 30 30 30 31 5c 75 30 30 30 30 5c 75 30 |n\u0001\u0000\u0| > 000079b0 30 30 30 5c 75 30 30 30 30 5c 75 30 30 30 30 5c |000\u0000\u0000\| > 000079c0 75 30 30 30 30 5c 75 30 30 30 30 ac 57 65 64 2c |u0000\u0000.Wed,| > 000079d0 20 31 30 20 55 6e 72 65 63 6f 67 6e 69 7a 65 64 | 10 Unrecognized| > 000079e0 20 69 6e 70 75 74 20 68 65 61 64 65 72 5c 6e 22 | input header\n"| > 000079f0 2c 22 6f 66 66 73 65 74 22 3a 32 32 30 34 34 33 |,"offset":220443| > 00007a00 7d |}| > {code} > This causes downstream sadness: > {code} > ERROR [2016-02-10 18:55:12,303] > io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: > 0ee749630f8b26f1 > ! com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0xac > ! at [Source: org.jboss.netty.buffer.ChannelBufferInputStream@6d69ee8; line: > 1, column: 31181] > ! at > com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1487) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3339) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3333) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2360) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:29) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:12) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:523) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:381) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1073) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserializeFromObject(SuperSonicBeanDeserializer.java:196) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:142) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.module.afterburner.deser.SuperSonicBeanDeserializer.deserialize(SuperSonicBeanDeserializer.java:117) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3562) > ~[singularity-0.4.9.jar:0.4.9] > ! at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2648) > ~[singularity-0.4.9.jar:0.4.9] > ! at com.hubspot.singularity.data.SandboxManager.read(SandboxManager.java:97) > ~[singularity-0.4.9.jar:0.4.9] > {code} > This is extremely similar to https://issues.apache.org/jira/browse/MESOS-3771 > Since this is now the second major issue caused by this, is there any > possibility of using a JSON processing library that actually guarantees > spec-compliant output? I know we can fix the point problem again here, but > it is frustrating that this keeps happening, and I'm sure it will happen > again in the future. > Failing that, maybe we should audit all JSON objects produced to ensure they > cannot contain binary data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)