[VOTE] Release Apache Mesos 0.24.0 (rc1)
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform to the
HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is defined as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the data as
a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
{offers:{offers:[{agent_id:{value:20150820-114552-16777343-5050-43704-S0},framework_id:{value:20150820-114552-16777343-5050-43704-},hostname:localhost,id:{value:20150820-114552-16777343-5050-43704-O0},resources:[{name:cpus,role:*,scalar:{value:8},type:SCALAR},{name:mem,role:*,scalar:{value:15360},type:SCALAR},{name:disk,role:*,scalar:{value:2965448},type:SCALAR},{name:ports,ranges:{range:[{begin:31000,end:32000}]},role:*,type:RANGES}],url:{address:{hostname:localhost,ip:127.0.0.1,port:5051},path:\/slave(1),scheme:http}}]},type:OFFERS”}20
… waiting …
{type:HEARTBEAT”}20
… waiting …
It will receive a couple of messages after successful registration with the
master and the last thing printed is a number (in this case 666). Then after
some time it will print the first offers message followed by the number 20. The
explanation for this behavior is, that curl can’t interpret the data it gets
from Mesos as a complete chunk and waits for the missing data. So it prints
what it thinks is a chunk (a message followed by the size of the next messsage)
and keeps the rest of the message until another message arrives and so on. The
fix for this is to terminate both lines, the message size and the message data,
with CRLF.
Cheers,
Dario
Use docker start rather than docker run?
Hi All, I first posted this to the Marathon list, but someone suggested I try it here. I'm still not sure what component (mesos-master, mesos-slave, marathon) generates the docker run command that launches containers on a slave node. I suppose that it's the framework executor (Marathon) on the slave that actually executes the docker run, but I'm not sure. What I'm really after is whether or not we can cause the use of docker start rather than docker run. At issue here is some persistent data inside /var/lib/docker/aufs/mnt/CTR_ID. docker run will by design (re)launch my application with a different CTR_ID effectively rendering that data inaccessible. But docker start will restart the container and its old data will still be there. Thanks. -Paul
Marathon split brain situation
Hello all, I am running test cluster with Mesos and Marathon in a cluster of 20 compute nodes and 2 head nodes running vm's that host all masters, frameworks etc. Till the 0.23 update there were not many issues but today i seen an issue that i must share and hope you guys know more about. We run an updated Mesos version 0.23 and Marathon 0.10.0. I started a hdfs namenode on docker through marathon and a couple of data nodes on the agents, im slowly building this config further with secondary namenodes, datanodes, journal nodes all in containers. For now its a very basic setup to see how stable everything is and what we should consider when running in containers. Today we found out that the marathon leader suddenly was registered 2 times as framework with different id's and to make it worse: It spawned task again that was already running. Suddenly we had 2 namenodes with the name management. Our consul cluster auto registered both containers and started to forward all traffic to these 2 namenodes. I always thought that zookeeper was taking care of election for marathon and this should prevent scenario's like this. However both frameworks had a different ID, which should explain why zookeeper didn't handle the election. The marathon web interface was no longer responding and everything timed out, i found out that there was only a single marathon process was running. To get hdfs back running again i killed the containers and killed the marathon process. From logs i couldn't gather why this happens, the 10 minutes around the registration of the framework there is nothing but offers, http calls and task syncs. The strange thing i just noticed is that marathon incidentally re-registers itself while its process is not restarted or elected. Does anyone have an idea where to look? -- Rogier Dikkes Systeem Programmeur Hadoop HPC Cloud SURFsara | Science Park 140 | 1098 XG Amsterdam
Re: Marathon split brain situation
Sounds like a marathon issue. You should ask in marathon mailing list. @vinodkone On Aug 28, 2015, at 7:05 AM, Rogier Dikkes rogier.dik...@surfsara.nl wrote: Hello all, I am running test cluster with Mesos and Marathon in a cluster of 20 compute nodes and 2 head nodes running vm's that host all masters, frameworks etc. Till the 0.23 update there were not many issues but today i seen an issue that i must share and hope you guys know more about. We run an updated Mesos version 0.23 and Marathon 0.10.0. I started a hdfs namenode on docker through marathon and a couple of data nodes on the agents, im slowly building this config further with secondary namenodes, datanodes, journal nodes all in containers. For now its a very basic setup to see how stable everything is and what we should consider when running in containers. Today we found out that the marathon leader suddenly was registered 2 times as framework with different id's and to make it worse: It spawned task again that was already running. Suddenly we had 2 namenodes with the name management. Our consul cluster auto registered both containers and started to forward all traffic to these 2 namenodes. I always thought that zookeeper was taking care of election for marathon and this should prevent scenario's like this. However both frameworks had a different ID, which should explain why zookeeper didn't handle the election. The marathon web interface was no longer responding and everything timed out, i found out that there was only a single marathon process was running. To get hdfs back running again i killed the containers and killed the marathon process. From logs i couldn't gather why this happens, the 10 minutes around the registration of the framework there is nothing but offers, http calls and task syncs. The strange thing i just noticed is that marathon incidentally re-registers itself while its process is not restarted or elected. Does anyone have an idea where to look? -- Rogier Dikkes Systeem Programmeur Hadoop HPC Cloud SURFsara | Science Park 140 | 1098 XG Amsterdam
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Dario,
Thanks for the detailed explanation and for trying out the new API. However,
this is not a bug. The output from CURL is the encoding used by Mesos for the
events stream. From the user doc
https://github.com/apache/mesos/blob/master/docs/scheduler_http_api.md:
Master encodes each Event in RecordIO format, i.e., string representation of
length of the event in bytes followed by JSON or binary Protobuf (possibly
compressed) encoded event. Note that the value of length will never be ‘0’ and
the size of the length will be the size of unsigned integer (i.e., 64 bits).
Also, note that the RecordIO encoding should be decoded by the scheduler
whereas the underlying HTTP chunked encoding is typically invisible at the
application (scheduler) layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be something
similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a) as
per the HTTP RFC. As mentioned earlier, the output that you observe on stdout
with CURL is of the Record-IO encoding used for the events stream ( and is not
related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform to
the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is defined
as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the data
as a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
{offers:{offers:[{agent_id:{value:20150820-114552-16777343-5050-43704-S0},framework_id:{value:20150820-114552-16777343-5050-43704-},hostname:localhost,id:{value:20150820-114552-16777343-5050-43704-O0},resources:[{name:cpus,role:*,scalar:{value:8},type:SCALAR},{name:mem,role:*,scalar:{value:15360},type:SCALAR},{name:disk,role:*,scalar:{value:2965448},type:SCALAR},{name:ports,ranges:{range:[{begin:31000,end:32000}]},role:*,type:RANGES}],url:{address:{hostname:localhost,ip:127.0.0.1,port:5051},path:\/slave(1),scheme:http}}]},type:OFFERS”}20
… waiting …
{type:HEARTBEAT”}20
… waiting …
It will receive a couple of messages after successful registration with the
master and the last thing printed is a number (in this case 666). Then after
some time it will print the first offers message followed by the number 20.
The explanation for this behavior is, that curl can’t interpret the data it
gets from Mesos as a complete chunk and waits for the missing data. So it
prints what it thinks is a chunk (a message followed by the size of the next
messsage) and keeps the rest of the message until another message arrives and
so on. The fix for this is to terminate both lines, the message size and the
message data, with CRLF.
Cheers,
Dario
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
I'm happy to add the \n after the event (note it's different from chunk)
if that makes CURL play nicer. I'm not sure about the \r part though? Is
that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released
yet. So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API.
However, this is not a bug. The output from CURL is the encoding used by
Mesos for the events stream. From the user doc
https://github.com/apache/mesos/blob/master/docs/scheduler_http_api.md:
*Master encodes each Event in RecordIO format, i.e., string
representation of length of the event in bytes followed by JSON or binary
Protobuf (possibly compressed) encoded event. Note that the value of
length will never be ‘0’ and the size of the length will be the size of
unsigned integer (i.e., 64 bits). Also, note that the RecordIO encoding
should be decoded by the scheduler whereas the underlying HTTP chunked
encoding is typically invisible at the application (scheduler) layer.“*
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a)
as per the HTTP RFC. As mentioned earlier, the output that you observe on
stdout with CURL is of the Record-IO encoding used for the events stream (
and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform to
the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3 (
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is defined
as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the
data as a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
{offers:{offers:[{agent_id:{value:20150820-114552-16777343-5050-43704-S0},framework_id:{value:20150820-114552-16777343-5050-43704-},hostname:localhost,id:{value:20150820-114552-16777343-5050-43704-O0},resources:[{name:cpus,role:*,scalar:{value:8},type:SCALAR},{name:mem,role:*,scalar:{value:15360},type:SCALAR},{name:disk,role:*,scalar:{value:2965448},type:SCALAR},{name:ports,ranges:{range:[{begin:31000,end:32000}]},role:*,type:RANGES}],url:{address:{hostname:localhost,ip:127.0.0.1,port:5051},path:\/slave(1),scheme:http}}]},type:OFFERS”}20
… waiting …
{type:HEARTBEAT”}20
… waiting …
It will receive a couple of messages after successful registration with
the master and the last thing printed is a number (in this case 666). Then
after some time it will print the first offers message followed by the
number 20. The explanation for this behavior is, that curl can’t interpret
the data it gets from Mesos as a complete chunk and waits for the missing
data. So it prints what it thinks is a chunk (a message followed by the
size of the next messsage) and keeps the rest of the message until another
message arrives and so on. The fix for this is to terminate both lines, the
message size and the message data, with CRLF.
Cheers,
Dario
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves so
strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and receive
'Content-Type: application/json', I actually expect to receive only json in the
message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org wrote:
I'm happy to add the \n after the event (note it's different from chunk) if
that makes CURL play nicer. I'm not sure about the \r part though? Is that
a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released yet.
So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API. However,
this is not a bug. The output from CURL is the encoding used by Mesos for
the events stream. From the user doc:
Master encodes each Event in RecordIO format, i.e., string representation
of length of the event in bytes followed by JSON or binary Protobuf
(possibly compressed) encoded event. Note that the value of length will
never be ‘0’ and the size of the length will be the size of unsigned integer
(i.e., 64 bits). Also, note that the RecordIO encoding should be decoded by
the scheduler whereas the underlying HTTP chunked encoding is typically
invisible at the application (scheduler) layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a) as
per the HTTP RFC. As mentioned earlier, the output that you observe on
stdout with CURL is of the Record-IO encoding used for the events stream (
and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform to
the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is defined
as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the data
as a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
{offers:{offers:[{agent_id:{value:20150820-114552-16777343-5050-43704-S0},framework_id:{value:20150820-114552-16777343-5050-43704-},hostname:localhost,id:{value:20150820-114552-16777343-5050-43704-O0},resources:[{name:cpus,role:*,scalar:{value:8},type:SCALAR},{name:mem,role:*,scalar:{value:15360},type:SCALAR},{name:disk,role:*,scalar:{value:2965448},type:SCALAR},{name:ports,ranges:{range:[{begin:31000,end:32000}]},role:*,type:RANGES}],url:{address:{hostname:localhost,ip:127.0.0.1,port:5051},path:\/slave(1),scheme:http}}]},type:OFFERS”}20
… waiting …
{type:HEARTBEAT”}20
… waiting …
It will receive a couple of messages after successful registration with the
master and the last thing printed is a number (in this case 666). Then
after some time it will print the first offers message followed by the
number 20. The explanation for this behavior is, that curl can’t interpret
the data it gets from Mesos as a complete chunk and waits for the missing
data. So it prints what it thinks is a chunk (a message followed by the
size of the next messsage) and keeps the rest of the message until another
message arrives and so on. The fix for this is to terminate both lines, the
message size and the message data, with CRLF.
Cheers,
Dario
Re: Use docker start rather than docker run?
Alex Tim, Thank you both; most helpful. Alex, can you dispel my confusion on this point: I keep reading that a framework in Mesos (e.g., Marathon) consists of a scheduler and an executor. This reference to executor made me think that Marathon must have *some* kind of presence on the slave node. But the more familiar I become with Mesos the less likely this seems to me. So, what does it mean to talk about the Marathon framework executor? Tim, I did come up with a simple work-around that involves re-copying the needed file into the container each time the application is started. For reasons unknown, this file is not kept in a location that would readily lend itself to my use of persistent storage (Docker -v). That said, I am keenly interested in learning how to write both custom executors schedulers. Any sense for what release of Mesos will see persistent volumes? Thanks again, gents. -Paul On Fri, Aug 28, 2015 at 2:26 PM, Tim Chen t...@mesosphere.io wrote: Hi Paul, We don't [re]start a container since we assume once the task terminated the container is no longer reused. In Mesos to allow tasks to reuse the same executor and handle task logic accordingly people will opt to choose the custom executor route. We're working on a way to keep your sandbox data beyond a container lifecycle, which is called persistent volumes. We haven't integrated that with Docker containerizer yet, so you'll have to wait to use that feature. You could also choose to implement a custom executor for now if you like. Tim On Fri, Aug 28, 2015 at 10:43 AM, Alex Rukletsov a...@mesosphere.com wrote: Paul, that component is called DockerContainerizer and it's part of Mesos Agent (check /Users/alex/Projects/mesos/src/slave/containerizer/docker.hpp). @Tim, could you answer the docker start vs. docker run question? On Fri, Aug 28, 2015 at 1:26 PM, Paul Bell arach...@gmail.com wrote: Hi All, I first posted this to the Marathon list, but someone suggested I try it here. I'm still not sure what component (mesos-master, mesos-slave, marathon) generates the docker run command that launches containers on a slave node. I suppose that it's the framework executor (Marathon) on the slave that actually executes the docker run, but I'm not sure. What I'm really after is whether or not we can cause the use of docker start rather than docker run. At issue here is some persistent data inside /var/lib/docker/aufs/mnt/CTR_ID. docker run will by design (re)launch my application with a different CTR_ID effectively rendering that data inaccessible. But docker start will restart the container and its old data will still be there. Thanks. -Paul
Re: Use docker start rather than docker run?
We have primitives for persistent volumes in next release (0.25.0) but DockerContainerizer integration will happen most likely the version after. Tim On Fri, Aug 28, 2015 at 11:50 AM, Tim Chen t...@mesosphere.io wrote: Hi Paul, Alternatively you can try to launch your task on the same host by specifying a constraint with marathon and mount a directory on the host in your container everytime to work-around as well. Tim On Fri, Aug 28, 2015 at 11:44 AM, Paul Bell arach...@gmail.com wrote: Alex Tim, Thank you both; most helpful. Alex, can you dispel my confusion on this point: I keep reading that a framework in Mesos (e.g., Marathon) consists of a scheduler and an executor. This reference to executor made me think that Marathon must have *some* kind of presence on the slave node. But the more familiar I become with Mesos the less likely this seems to me. So, what does it mean to talk about the Marathon framework executor? Tim, I did come up with a simple work-around that involves re-copying the needed file into the container each time the application is started. For reasons unknown, this file is not kept in a location that would readily lend itself to my use of persistent storage (Docker -v). That said, I am keenly interested in learning how to write both custom executors schedulers. Any sense for what release of Mesos will see persistent volumes? Thanks again, gents. -Paul On Fri, Aug 28, 2015 at 2:26 PM, Tim Chen t...@mesosphere.io wrote: Hi Paul, We don't [re]start a container since we assume once the task terminated the container is no longer reused. In Mesos to allow tasks to reuse the same executor and handle task logic accordingly people will opt to choose the custom executor route. We're working on a way to keep your sandbox data beyond a container lifecycle, which is called persistent volumes. We haven't integrated that with Docker containerizer yet, so you'll have to wait to use that feature. You could also choose to implement a custom executor for now if you like. Tim On Fri, Aug 28, 2015 at 10:43 AM, Alex Rukletsov a...@mesosphere.com wrote: Paul, that component is called DockerContainerizer and it's part of Mesos Agent (check /Users/alex/Projects/mesos/src/slave/containerizer/docker.hpp). @Tim, could you answer the docker start vs. docker run question? On Fri, Aug 28, 2015 at 1:26 PM, Paul Bell arach...@gmail.com wrote: Hi All, I first posted this to the Marathon list, but someone suggested I try it here. I'm still not sure what component (mesos-master, mesos-slave, marathon) generates the docker run command that launches containers on a slave node. I suppose that it's the framework executor (Marathon) on the slave that actually executes the docker run, but I'm not sure. What I'm really after is whether or not we can cause the use of docker start rather than docker run. At issue here is some persistent data inside /var/lib/docker/aufs/mnt/CTR_ID. docker run will by design (re)launch my application with a different CTR_ID effectively rendering that data inaccessible. But docker start will restart the container and its old data will still be there. Thanks. -Paul
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Anand,
in the example from my first mail you can see that curl prints the size of a
message and then waits for the next message and only when it receives that
message it will print the prior message plus the size of the next message, but
not the actual message.
What's the benefit of encoding multiple messages in a single chunk? You could
simply create a single chunk per event.
Cheers,
Dario
On 28.08.2015, at 19:43, Anand Mazumdar an...@mesosphere.io wrote:
Dario,
Can you shed a bit more light on what you still find puzzling about the CURL
behavior after my explanation ?
PS: A single HTTP chunk can have 0 or more Mesos (Scheduler API) Events. So
in your example, the first chunk had complete information about the first
“event”, followed by partial information about the subsequent event from
another chunk.
As for the benefit of using RecordIO format here, how else do you think we
could have de-marcated two events in the response ?
-anand
On Aug 28, 2015, at 10:01 AM, dario.re...@me.com wrote:
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves so
strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and
receive 'Content-Type: application/json', I actually expect to receive only
json in the message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org wrote:
I'm happy to add the \n after the event (note it's different from chunk)
if that makes CURL play nicer. I'm not sure about the \r part though? Is
that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released
yet. So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io
wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API.
However, this is not a bug. The output from CURL is the encoding used by
Mesos for the events stream. From the user doc:
Master encodes each Event in RecordIO format, i.e., string representation
of length of the event in bytes followed by JSON or binary Protobuf
(possibly compressed) encoded event. Note that the value of length will
never be ‘0’ and the size of the length will be the size of unsigned
integer (i.e., 64 bits). Also, note that the RecordIO encoding should be
decoded by the scheduler whereas the underlying HTTP chunked encoding is
typically invisible at the application (scheduler) layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a)
as per the HTTP RFC. As mentioned earlier, the output that you observe on
stdout with CURL is of the Record-IO encoding used for the events stream (
and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform
to the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is
defined as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the
data as a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
Re: Use docker start rather than docker run?
Hi Paul, Alternatively you can try to launch your task on the same host by specifying a constraint with marathon and mount a directory on the host in your container everytime to work-around as well. Tim On Fri, Aug 28, 2015 at 11:44 AM, Paul Bell arach...@gmail.com wrote: Alex Tim, Thank you both; most helpful. Alex, can you dispel my confusion on this point: I keep reading that a framework in Mesos (e.g., Marathon) consists of a scheduler and an executor. This reference to executor made me think that Marathon must have *some* kind of presence on the slave node. But the more familiar I become with Mesos the less likely this seems to me. So, what does it mean to talk about the Marathon framework executor? Tim, I did come up with a simple work-around that involves re-copying the needed file into the container each time the application is started. For reasons unknown, this file is not kept in a location that would readily lend itself to my use of persistent storage (Docker -v). That said, I am keenly interested in learning how to write both custom executors schedulers. Any sense for what release of Mesos will see persistent volumes? Thanks again, gents. -Paul On Fri, Aug 28, 2015 at 2:26 PM, Tim Chen t...@mesosphere.io wrote: Hi Paul, We don't [re]start a container since we assume once the task terminated the container is no longer reused. In Mesos to allow tasks to reuse the same executor and handle task logic accordingly people will opt to choose the custom executor route. We're working on a way to keep your sandbox data beyond a container lifecycle, which is called persistent volumes. We haven't integrated that with Docker containerizer yet, so you'll have to wait to use that feature. You could also choose to implement a custom executor for now if you like. Tim On Fri, Aug 28, 2015 at 10:43 AM, Alex Rukletsov a...@mesosphere.com wrote: Paul, that component is called DockerContainerizer and it's part of Mesos Agent (check /Users/alex/Projects/mesos/src/slave/containerizer/docker.hpp). @Tim, could you answer the docker start vs. docker run question? On Fri, Aug 28, 2015 at 1:26 PM, Paul Bell arach...@gmail.com wrote: Hi All, I first posted this to the Marathon list, but someone suggested I try it here. I'm still not sure what component (mesos-master, mesos-slave, marathon) generates the docker run command that launches containers on a slave node. I suppose that it's the framework executor (Marathon) on the slave that actually executes the docker run, but I'm not sure. What I'm really after is whether or not we can cause the use of docker start rather than docker run. At issue here is some persistent data inside /var/lib/docker/aufs/mnt/CTR_ID. docker run will by design (re)launch my application with a different CTR_ID effectively rendering that data inaccessible. But docker start will restart the container and its old data will still be there. Thanks. -Paul
Re: Use docker start rather than docker run?
Heh, that's a tricky one : ). A framework indeed consists of a scheduler and an executor, both are mandatory. But Mesos provides a default general-purpose executor, which can be used by frameworks. This executor has many names, two most common are MesosExecutor and CommandExecutor. Marathon doesn't have its own executor (in contrast to, say, Aurora) yet, it uses CommandExecutor for all of its tasks. CommandExecutor is implicitly created by Mesos if a task specification do not include executor. This executor can have just a single task and is garbage collected after the task finishes. A task is any command, which will be executed via '/bin/sh -c command'. On Fri, Aug 28, 2015 at 8:50 PM, Tim Chen t...@mesosphere.io wrote: We have primitives for persistent volumes in next release (0.25.0) but DockerContainerizer integration will happen most likely the version after. Tim On Fri, Aug 28, 2015 at 11:50 AM, Tim Chen t...@mesosphere.io wrote: Hi Paul, Alternatively you can try to launch your task on the same host by specifying a constraint with marathon and mount a directory on the host in your container everytime to work-around as well. Tim On Fri, Aug 28, 2015 at 11:44 AM, Paul Bell arach...@gmail.com wrote: Alex Tim, Thank you both; most helpful. Alex, can you dispel my confusion on this point: I keep reading that a framework in Mesos (e.g., Marathon) consists of a scheduler and an executor. This reference to executor made me think that Marathon must have *some* kind of presence on the slave node. But the more familiar I become with Mesos the less likely this seems to me. So, what does it mean to talk about the Marathon framework executor? Tim, I did come up with a simple work-around that involves re-copying the needed file into the container each time the application is started. For reasons unknown, this file is not kept in a location that would readily lend itself to my use of persistent storage (Docker -v). That said, I am keenly interested in learning how to write both custom executors schedulers. Any sense for what release of Mesos will see persistent volumes? Thanks again, gents. -Paul On Fri, Aug 28, 2015 at 2:26 PM, Tim Chen t...@mesosphere.io wrote: Hi Paul, We don't [re]start a container since we assume once the task terminated the container is no longer reused. In Mesos to allow tasks to reuse the same executor and handle task logic accordingly people will opt to choose the custom executor route. We're working on a way to keep your sandbox data beyond a container lifecycle, which is called persistent volumes. We haven't integrated that with Docker containerizer yet, so you'll have to wait to use that feature. You could also choose to implement a custom executor for now if you like. Tim On Fri, Aug 28, 2015 at 10:43 AM, Alex Rukletsov a...@mesosphere.com wrote: Paul, that component is called DockerContainerizer and it's part of Mesos Agent (check /Users/alex/Projects/mesos/src/slave/containerizer/docker.hpp). @Tim, could you answer the docker start vs. docker run question? On Fri, Aug 28, 2015 at 1:26 PM, Paul Bell arach...@gmail.com wrote: Hi All, I first posted this to the Marathon list, but someone suggested I try it here. I'm still not sure what component (mesos-master, mesos-slave, marathon) generates the docker run command that launches containers on a slave node. I suppose that it's the framework executor (Marathon) on the slave that actually executes the docker run, but I'm not sure. What I'm really after is whether or not we can cause the use of docker start rather than docker run. At issue here is some persistent data inside /var/lib/docker/aufs/mnt/CTR_ID. docker run will by design (re)launch my application with a different CTR_ID effectively rendering that data inaccessible. But docker start will restart the container and its old data will still be there. Thanks. -Paul
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Anand,
thanks for the explanation. I didn't think about the case when you have to
split a message, now it makes sense.
But the case I observed with curl is still weird. Even when splitting a
message, it should still receive both parts almost at the same time. Do you
have any idea why it could behave like this?
On 28.08.2015, at 21:31, Anand Mazumdar an...@mesosphere.io wrote:
Dario,
Most HTTP libraries/parsers ( including one that Mesos uses internally )
provide a way to specify a default size of each chunk. If a Mesos Event is
too big , it would get split into smaller chunks and vice-versa.
-anand
On Aug 28, 2015, at 11:51 AM, dario.re...@me.com wrote:
Anand,
in the example from my first mail you can see that curl prints the size of a
message and then waits for the next message and only when it receives that
message it will print the prior message plus the size of the next message,
but not the actual message.
What's the benefit of encoding multiple messages in a single chunk? You
could simply create a single chunk per event.
Cheers,
Dario
On 28.08.2015, at 19:43, Anand Mazumdar an...@mesosphere.io wrote:
Dario,
Can you shed a bit more light on what you still find puzzling about the
CURL behavior after my explanation ?
PS: A single HTTP chunk can have 0 or more Mesos (Scheduler API) Events. So
in your example, the first chunk had complete information about the first
“event”, followed by partial information about the subsequent event from
another chunk.
As for the benefit of using RecordIO format here, how else do you think we
could have de-marcated two events in the response ?
-anand
On Aug 28, 2015, at 10:01 AM, dario.re...@me.com wrote:
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves so
strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and
receive 'Content-Type: application/json', I actually expect to receive
only json in the message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org wrote:
I'm happy to add the \n after the event (note it's different from
chunk) if that makes CURL play nicer. I'm not sure about the \r part
though? Is that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released
yet. So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io
wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API.
However, this is not a bug. The output from CURL is the encoding used by
Mesos for the events stream. From the user doc:
Master encodes each Event in RecordIO format, i.e., string
representation of length of the event in bytes followed by JSON or
binary Protobuf (possibly compressed) encoded event. Note that the
value of length will never be ‘0’ and the size of the length will be the
size of unsigned integer (i.e., 64 bits). Also, note that the RecordIO
encoding should be decoded by the scheduler whereas the underlying HTTP
chunked encoding is typically invisible at the application (scheduler)
layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d
0a) as per the HTTP RFC. As mentioned earlier, the output that you
observe on stdout with CURL is of the Record-IO encoding used for the
events stream ( and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform
to the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is
defined as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Dario,
Can you shed a bit more light on what you still find puzzling about the CURL
behavior after my explanation ?
PS: A single HTTP chunk can have 0 or more Mesos (Scheduler API) Events. So in
your example, the first chunk had complete information about the first “event”,
followed by partial information about the subsequent event from another chunk.
As for the benefit of using RecordIO format here, how else do you think we
could have de-marcated two events in the response ?
-anand
On Aug 28, 2015, at 10:01 AM, dario.re...@me.com wrote:
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves so
strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and receive
'Content-Type: application/json', I actually expect to receive only json in
the message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org
mailto:vinodk...@apache.org wrote:
I'm happy to add the \n after the event (note it's different from chunk)
if that makes CURL play nicer. I'm not sure about the \r part though? Is
that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released yet.
So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io
mailto:an...@mesosphere.io wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API. However,
this is not a bug. The output from CURL is the encoding used by Mesos for
the events stream. From the user doc
https://github.com/apache/mesos/blob/master/docs/scheduler_http_api.md:
Master encodes each Event in RecordIO format, i.e., string representation
of length of the event in bytes followed by JSON or binary Protobuf
(possibly compressed) encoded event. Note that the value of length will
never be ‘0’ and the size of the length will be the size of unsigned integer
(i.e., 64 bits). Also, note that the RecordIO encoding should be decoded by
the scheduler whereas the underlying HTTP chunked encoding is typically
invisible at the application (scheduler) layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a) as
per the HTTP RFC. As mentioned earlier, the output that you observe on
stdout with CURL is of the Record-IO encoding used for the events stream (
and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com
mailto:dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform to
the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is defined
as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the data
as a complete chunk. In curl it currently looks like this:
104
{subscribed:{framework_id:{value:20150820-114552-16777343-5050-43704-}},type:SUBSCRIBED}20
{type:HEARTBEAT”}666
…. waiting …
{offers:{offers:[{agent_id:{value:20150820-114552-16777343-5050-43704-S0},framework_id:{value:20150820-114552-16777343-5050-43704-},hostname:localhost,id:{value:20150820-114552-16777343-5050-43704-O0},resources:[{name:cpus,role:*,scalar:{value:8},type:SCALAR},{name:mem,role:*,scalar:{value:15360},type:SCALAR},{name:disk,role:*,scalar:{value:2965448},type:SCALAR},{name:ports,ranges:{range:[{begin:31000,end:32000}]},role:*,type:RANGES}],url:{address:{hostname:localhost,ip:127.0.0.1,port:5051},path:\/slave(1),scheme:http}}]},type:OFFERS”}20
… waiting …
{type:HEARTBEAT”}20
… waiting …
It will receive a couple of messages after successful
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Dario,
Most HTTP libraries/parsers ( including one that Mesos uses internally )
provide a way to specify a default size of each chunk. If a Mesos Event is too
big , it would get split into smaller chunks and vice-versa.
-anand
On Aug 28, 2015, at 11:51 AM, dario.re...@me.com wrote:
Anand,
in the example from my first mail you can see that curl prints the size of a
message and then waits for the next message and only when it receives that
message it will print the prior message plus the size of the next message,
but not the actual message.
What's the benefit of encoding multiple messages in a single chunk? You could
simply create a single chunk per event.
Cheers,
Dario
On 28.08.2015, at 19:43, Anand Mazumdar an...@mesosphere.io
mailto:an...@mesosphere.io wrote:
Dario,
Can you shed a bit more light on what you still find puzzling about the CURL
behavior after my explanation ?
PS: A single HTTP chunk can have 0 or more Mesos (Scheduler API) Events. So
in your example, the first chunk had complete information about the first
“event”, followed by partial information about the subsequent event from
another chunk.
As for the benefit of using RecordIO format here, how else do you think we
could have de-marcated two events in the response ?
-anand
On Aug 28, 2015, at 10:01 AM, dario.re...@me.com
mailto:dario.re...@me.com wrote:
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves so
strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and
receive 'Content-Type: application/json', I actually expect to receive only
json in the message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org
mailto:vinodk...@apache.org wrote:
I'm happy to add the \n after the event (note it's different from chunk)
if that makes CURL play nicer. I'm not sure about the \r part though? Is
that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released
yet. So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io
mailto:an...@mesosphere.io wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API.
However, this is not a bug. The output from CURL is the encoding used by
Mesos for the events stream. From the user doc
https://github.com/apache/mesos/blob/master/docs/scheduler_http_api.md:
Master encodes each Event in RecordIO format, i.e., string representation
of length of the event in bytes followed by JSON or binary Protobuf
(possibly compressed) encoded event. Note that the value of length will
never be ‘0’ and the size of the length will be the size of unsigned
integer (i.e., 64 bits). Also, note that the RecordIO encoding should be
decoded by the scheduler whereas the underlying HTTP chunked encoding is
typically invisible at the application (scheduler) layer.“
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d 0a)
as per the HTTP RFC. As mentioned earlier, the output that you observe on
stdout with CURL is of the Record-IO encoding used for the events stream (
and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com
mailto:dario.re...@me.com wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform
to the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is
defined as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the
data as a
Re: [VOTE] Release Apache Mesos 0.24.0 (rc1)
Also, if you want to Accept application/json, perhaps you could do
something like:
[recordLength, recordData],
[record2Length, record2Data]
That gives you the same semantics, and the record data can be anything, but
also falls inline with the server returning Content-Type: application/json
For the custom content type, you could then return the custom format.
Possible best of both worlds?
On Friday, August 28, 2015, James DeFelice james.defel...@gmail.com wrote:
Perhaps headers like these would make more sense?
Content-Encoding: recordio
Content-Type: application/x-json-stream
Multiple encoding values could be present to indicate compression, etc.
On Aug 28, 2015 10:43 AM, Anand Mazumdar an...@mesosphere.io
javascript:_e(%7B%7D,'cvml','an...@mesosphere.io'); wrote:
Dario,
Can you shed a bit more light on what you still find puzzling about the
CURL behavior after my explanation ?
PS: A single HTTP chunk can have 0 or more Mesos (Scheduler API) Events.
So in your example, the first chunk had complete information about the
first “event”, followed by partial information about the subsequent event
from another chunk.
As for the benefit of using RecordIO format here, how else do you think
we could have de-marcated two events in the response ?
-anand
On Aug 28, 2015, at 10:01 AM, dario.re...@me.com
javascript:_e(%7B%7D,'cvml','dario.re...@me.com'); wrote:
Anand,
thanks for the explanation. I'm still a little puzzled why curl behaves
so strange. I will check how other client behave as soon as I have a chance.
Vinod,
what exactly is the benefit of using recordio here? Doesn't it make the
content-type somewhat wrong? If I send 'Accept: application/json' and
receive 'Content-Type: application/json', I actually expect to receive only
json in the message.
Thanks,
Dario
On 28.08.2015, at 18:13, Vinod Kone vinodk...@apache.org
javascript:_e(%7B%7D,'cvml','vinodk...@apache.org'); wrote:
I'm happy to add the \n after the event (note it's different from
chunk) if that makes CURL play nicer. I'm not sure about the \r part
though? Is that a nice to have or does it have some other benefit?
The design doc is not set in the stone since this has not been released
yet. So definitely want to do the right/easy thing.
On Fri, Aug 28, 2015 at 7:53 AM, Anand Mazumdar an...@mesosphere.io
javascript:_e(%7B%7D,'cvml','an...@mesosphere.io'); wrote:
Dario,
Thanks for the detailed explanation and for trying out the new API.
However, this is not a bug. The output from CURL is the encoding used by
Mesos for the events stream. From the user doc
https://github.com/apache/mesos/blob/master/docs/scheduler_http_api.md
:
*Master encodes each Event in RecordIO format, i.e., string
representation of length of the event in bytes followed by JSON or binary
Protobuf (possibly compressed) encoded event. Note that the value of
length will never be ‘0’ and the size of the length will be the size of
unsigned integer (i.e., 64 bits). Also, note that the RecordIO encoding
should be decoded by the scheduler whereas the underlying HTTP chunked
encoding is typically invisible at the application (scheduler) layer.“*
If you run CURL with tracing enabled i.e. —trace, the output would be
something similar to this:
= Recv header, 2 bytes (0x2)
: 0d 0a ..
= Recv data, 115 bytes (0x73)
: 36 64 0d 0a 31 30 35 0a 7b 22 73 75 62 73 63 72 6d..105.{subscr
0010: 69 62 65 64 22 3a 7b 22 66 72 61 6d 65 77 6f 72 ibed:{framewor
0020: 6b 5f 69 64 22 3a 7b 22 76 61 6c 75 65 22 3a 22 k_id:{value:
0030: 32 30 31 35 30 38 32 35 2d 31 30 33 30 31 38 2d 20150825-103018-
0040: 33 38 36 33 38 37 31 34 39 38 2d 35 30 35 30 2d 3863871498-5050-
0050: 31 31 38 35 2d 30 30 31 30 22 7d 7d 2c 22 74 79 1185-0010}},ty
0060: 70 65 22 3a 22 53 55 42 53 43 52 49 42 45 44 22 pe:SUBSCRIBED
0070: 7d 0d 0a}..
others
In the output above, the chunks are correctly delimited by ‘CRLF' (0d
0a) as per the HTTP RFC. As mentioned earlier, the output that you observe
on stdout with CURL is of the Record-IO encoding used for the events stream
( and is not related to the RFC ):
event = event-size LF
event-data
Looking forward to more bug reports as you try out the new API !
-anand
On Aug 28, 2015, at 12:56 AM, Dario Rexin dario.re...@me.com
javascript:_e(%7B%7D,'cvml','dario.re...@me.com'); wrote:
-1 (non-binding)
I found a breaking bug in the new HTTP API. The messages do not conform
to the HTTP standard for chunked transfer encoding. in RFC 2616 Sec. 3 (
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html) a chunk is
defined as:
chunk = chunk-size [ chunk-extension ] CRLF
chunk-data CRLF
The HTTP API currently sends a chunk as:
chunk = chunk-size LF
chunk-data
A standard conform HTTP client like curl can’t correctly interpret the
data as a complete chunk. In curl it
