WebUI: CORS in Mesos Design Doc + Discussion

[Jacob Janco]

Hi all, 

There has been a bit of discussion surrounding cross-origin resource sharing in 
Mesos and we would like to float a design doc to delineate/collect thoughts 
from the community and propose an implementation. 

For a bit of context: From the master WebUI we construct agent resource 
requests with a jsonp parameter to wrap returned resources in a callback to 
work around browser same-origin access controls. There are security concerns 
with this approach, one such example is cited in the doc, and there is a 
consensus that jsonp should be removed from the Mesos codebase. Implementing 
CORS support is a step in that direction. 

Here’s a link to the doc: 
https://docs.google.com/document/d/1a6xaljM2yjtteyQ3zz1hmjLcCUOZ_YUHDNzrRm6gOh0/edit?usp=sharing
 

Thanks and comments welcome!

- Jacob Janco

Re: [VOTE] Release Apache Mesos 1.3.0 (rc1)

[Yan Xu]

We work around autotools and protobuf bugs and glibc is only harder for
users and developers to upgrade. :)

I agree that we can establish the minimum glibc version/linux distro
releases etc we support but currently we don't and there are folks who use
Mesos that depend on this version.

We should follow http://mesos.apache.org/documentation/latest/versioning/
and when answers are not in there, we should seek consensus to improve the
process and update the doc and give folks adequate time to adjust to the
new or better defined) process. In the meantime, I think we shouldn't break
the current users.

---
Jiang Yan Xu  | @xujyan 

On Mon, May 8, 2017 at 3:59 PM, Neil Conway  wrote:

> Personally, I'm not convinced that we need to fix MESOS-7378. The
> problem is essentially a bug in glibc that was fixed 6 years ago. (As
> a point of reference, the oldest version of g++ we support was
> released 2 years ago... :) )
>
> Neil
>
> On Mon, May 8, 2017 at 3:45 PM, Yan Xu  wrote:
> > I am still hoping that we get
> > https://issues.apache.org/jira/browse/MESOS-7378 fixed before shipping
> > 0.13.0. :)
> >
> > ---
> > Jiang Yan Xu  | @xujyan
> >
> > On Fri, May 5, 2017 at 6:31 PM, Michael Park  wrote:
> >>
> >> Hi all,
> >>
> >> Please vote on releasing the following candidate as Apache Mesos 1.3.0.
> >>
> >>
> >> 1.3.0 includes the following:
> >>
> >> 
> 
> >>   - Multi-role framework support
> >>   - Executor authentication support
> >>   - Allow frameworks to modify their roles.
> >>   - Hierarchical roles (*EXPERIMENTAL*)
> >>
> >> The CHANGELOG for the release is available at:
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.3.0-rc1
> >>
> >> 
> 
> >>
> >> The candidate for Mesos 1.3.0 release is available at:
> >> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/
> mesos-1.3.0.tar.gz
> >>
> >> The tag to be voted on is 1.3.0-rc1:
> >> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=
> commit;h=1.3.0-rc1
> >>
> >> The MD5 checksum of the tarball can be found at:
> >>
> >> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/
> mesos-1.3.0.tar.gz.md5
> >>
> >> The signature of the tarball can be found at:
> >>
> >> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/
> mesos-1.3.0.tar.gz.asc
> >>
> >> The PGP key used to sign the release is here:
> >> https://dist.apache.org/repos/dist/release/mesos/KEYS
> >>
> >> The JAR is up in Maven in a staging repository here:
> >> https://repository.apache.org/content/repositories/orgapachemesos-1190
> >>
> >> Please vote on releasing this package as Apache Mesos 1.3.0!
> >>
> >> The vote is open until Wed May 10 11:59:59 PDT 2017 and passes if a
> >> majority of at least 3 +1 PMC votes are cast.
> >>
> >> [ ] +1 Release this package as Apache Mesos 1.3.0
> >> [ ] -1 Do not release this package because ...
> >>
> >> Thanks,
> >>
> >> MPark & Neil
> >
> >
>

Re: [VOTE] Release Apache Mesos 1.1.2 (rc1)

[Vinod Kone]

I saw this on ASF CI
.
Expected flaky test?

[ RUN  ] HTTPCommandExecutorTest.TerminateWithACK
I0504 15:43:05.341382 32064 cluster.cpp:158] Creating default 'local' authorizer
I0504 15:43:05.345090 32064 leveldb.cpp:174] Opened db in 3.444533ms
I0504 15:43:05.345728 32064 leveldb.cpp:181] Compacted db in 603462ns
I0504 15:43:05.345772 32064 leveldb.cpp:196] Created db iterator in 16838ns
I0504 15:43:05.345788 32064 leveldb.cpp:202] Seeked to beginning of db in 1987ns
I0504 15:43:05.345799 32064 leveldb.cpp:271] Iterated through 0 keys
in the db in 269ns
I0504 15:43:05.345834 32064 replica.cpp:776] Replica recovered with
log positions 0 -> 0 with 1 holes and 0 unlearned
I0504 15:43:05.346590 32091 recover.cpp:451] Starting replica recovery
I0504 15:43:05.346793 32091 recover.cpp:477] Replica is in EMPTY status
I0504 15:43:05.347823 32098 replica.cpp:673] Replica in EMPTY status
received a broadcasted recover request from
__req_res__(168)@172.17.0.3:41866
I0504 15:43:05.348352 32090 recover.cpp:197] Received a recover
response from a replica in EMPTY status
I0504 15:43:05.348784 32098 recover.cpp:568] Updating replica status to STARTING
I0504 15:43:05.349874 32095 leveldb.cpp:304] Persisting metadata (8
bytes) to leveldb took 840720ns
I0504 15:43:05.349900 32095 replica.cpp:320] Persisted replica status
to STARTING
I0504 15:43:05.350070 32088 recover.cpp:477] Replica is in STARTING status
I0504 15:43:05.350971 32102 master.cpp:380] Master
2075640b-b7dc-44f0-89b5-b0f9af99be7e (41c61dc99119) started on
172.17.0.3:41866
I0504 15:43:05.351112 32088 replica.cpp:673] Replica in STARTING
status received a broadcasted recover request from
__req_res__(169)@172.17.0.3:41866
I0504 15:43:05.350991 32102 master.cpp:382] Flags at startup:
--acls="" --agent_ping_timeout="15secs"
--agent_reregister_timeout="10mins" --allocation_interval="1secs"
--allocator="HierarchicalDRF" --authenticate_agents="true"
--authenticate_frameworks="true" --authenticate_http_frameworks="true"
--authenticate_http_readonly="true"
--authenticate_http_readwrite="true" --authenticators="crammd5"
--authorizers="local" --credentials="/tmp/t7Ea9P/credentials"
--framework_sorter="drf" --help="false" --hostname_lookup="true"
--http_authenticators="basic" --http_framework_authenticators="basic"
--initialize_driver_logging="true" --log_auto_initialize="true"
--logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5"
--max_completed_frameworks="50"
--max_completed_tasks_per_framework="1000" --quiet="false"
--recovery_agent_removal_limit="100%" --registry="replicated_log"
--registry_fetch_timeout="1mins" --registry_gc_interval="15mins"
--registry_max_agent_age="2weeks" --registry_max_agent_count="102400"
--registry_store_timeout="100secs" --registry_strict="false"
--root_submissions="true" --user_sorter="drf" --version="false"
--webui_dir="/mesos/mesos-1.1.2/_inst/share/mesos/webui"
--work_dir="/tmp/t7Ea9P/master" --zk_session_timeout="10secs"
I0504 15:43:05.351322 32102 master.cpp:432] Master only allowing
authenticated frameworks to register
I0504 15:43:05.351335 32102 master.cpp:446] Master only allowing
authenticated agents to register
I0504 15:43:05.351341 32102 master.cpp:459] Master only allowing
authenticated HTTP frameworks to register
I0504 15:43:05.351348 32102 credentials.hpp:37] Loading credentials
for authentication from '/tmp/t7Ea9P/credentials'
I0504 15:43:05.351394 32094 recover.cpp:197] Received a recover
response from a replica in STARTING status
I0504 15:43:05.351594 32102 master.cpp:504] Using default 'crammd5'
authenticator
I0504 15:43:05.351850 32102 http.cpp:887] Using default 'basic' HTTP
authenticator for realm 'mesos-master-readonly'
I0504 15:43:05.352252 32102 http.cpp:887] Using default 'basic' HTTP
authenticator for realm 'mesos-master-readwrite'
I0504 15:43:05.352270 32090 recover.cpp:568] Updating replica status to VOTING
I0504 15:43:05.352500 32102 http.cpp:887] Using default 'basic' HTTP
authenticator for realm 'mesos-master-scheduler'
I0504 15:43:05.352635 32092 leveldb.cpp:304] Persisting metadata (8
bytes) to leveldb took 225076ns
I0504 15:43:05.352660 32092 replica.cpp:320] Persisted replica status to VOTING
I0504 15:43:05.352707 32102 master.cpp:584] Authorization enabled
I0504 15:43:05.352778 32087 recover.cpp:582] Successfully joined the Paxos group
I0504 15:43:05.352880 32091 hierarchical.cpp:149] Initialized
hierarchical allocator process
I0504 15:43:05.352883 32089 whitelist_watcher.cpp:77] No whitelist given
I0504 15:43:05.353144 32087 recover.cpp:466] Recover process terminated
I0504 15:43:05.355403 32085 master.cpp:2017] Elected as the leading master!
I0504 15:43:05.355437 32085 master.cpp:1560] Recovering from registrar

Re: [VOTE] Release Apache Mesos 1.3.0 (rc1)

[Neil Conway]

Personally, I'm not convinced that we need to fix MESOS-7378. The
problem is essentially a bug in glibc that was fixed 6 years ago. (As
a point of reference, the oldest version of g++ we support was
released 2 years ago... :) )

Neil

On Mon, May 8, 2017 at 3:45 PM, Yan Xu  wrote:
> I am still hoping that we get
> https://issues.apache.org/jira/browse/MESOS-7378 fixed before shipping
> 0.13.0. :)
>
> ---
> Jiang Yan Xu  | @xujyan
>
> On Fri, May 5, 2017 at 6:31 PM, Michael Park  wrote:
>>
>> Hi all,
>>
>> Please vote on releasing the following candidate as Apache Mesos 1.3.0.
>>
>>
>> 1.3.0 includes the following:
>>
>> 
>>   - Multi-role framework support
>>   - Executor authentication support
>>   - Allow frameworks to modify their roles.
>>   - Hierarchical roles (*EXPERIMENTAL*)
>>
>> The CHANGELOG for the release is available at:
>>
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.3.0-rc1
>>
>> 
>>
>> The candidate for Mesos 1.3.0 release is available at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos-1.3.0.tar.gz
>>
>> The tag to be voted on is 1.3.0-rc1:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.3.0-rc1
>>
>> The MD5 checksum of the tarball can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos-1.3.0.tar.gz.md5
>>
>> The signature of the tarball can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos-1.3.0.tar.gz.asc
>>
>> The PGP key used to sign the release is here:
>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>
>> The JAR is up in Maven in a staging repository here:
>> https://repository.apache.org/content/repositories/orgapachemesos-1190
>>
>> Please vote on releasing this package as Apache Mesos 1.3.0!
>>
>> The vote is open until Wed May 10 11:59:59 PDT 2017 and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Mesos 1.3.0
>> [ ] -1 Do not release this package because ...
>>
>> Thanks,
>>
>> MPark & Neil
>
>

Re: [VOTE] Release Apache Mesos 1.3.0 (rc1)

[Yan Xu]

s/0.13.0/1.3.0/ :)

---
Jiang Yan Xu  | @xujyan 

On Mon, May 8, 2017 at 3:45 PM, Yan Xu  wrote:

> I am still hoping that we get https://issues.apache.org/
> jira/browse/MESOS-7378 fixed before shipping 0.13.0. :)
>
> ---
> Jiang Yan Xu  | @xujyan 
>
> On Fri, May 5, 2017 at 6:31 PM, Michael Park  wrote:
>
>> Hi all,
>>
>> Please vote on releasing the following candidate as Apache Mesos 1.3.0.
>>
>>
>> 1.3.0 includes the following:
>> 
>> 
>>   - Multi-role framework support
>>   - Executor authentication support
>>   - Allow frameworks to modify their roles.
>>   - Hierarchical roles (*EXPERIMENTAL*)
>>
>> The CHANGELOG for the release is available at:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_p
>> lain;f=CHANGELOG;hb=1.3.0-rc1
>> 
>> 
>>
>> The candidate for Mesos 1.3.0 release is available at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos-1.3.0.tar.gz
>>
>> The tag to be voted on is 1.3.0-rc1:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.3.0-rc1
>>
>> The MD5 checksum of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos
>> -1.3.0.tar.gz.md5
>>
>> The signature of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos
>> -1.3.0.tar.gz.asc
>>
>> The PGP key used to sign the release is here:
>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>
>> The JAR is up in Maven in a staging repository here:
>> https://repository.apache.org/content/repositories/orgapachemesos-1190
>>
>> Please vote on releasing this package as Apache Mesos 1.3.0!
>>
>> The vote is open until Wed May 10 11:59:59 PDT 2017 and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Mesos 1.3.0
>> [ ] -1 Do not release this package because ...
>>
>> Thanks,
>>
>> MPark & Neil
>>
>
>

Re: [VOTE] Release Apache Mesos 1.3.0 (rc1)

[Yan Xu]

I am still hoping that we get
https://issues.apache.org/jira/browse/MESOS-7378 fixed before shipping
0.13.0. :)

---
Jiang Yan Xu  | @xujyan 

On Fri, May 5, 2017 at 6:31 PM, Michael Park  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.3.0.
>
>
> 1.3.0 includes the following:
> 
> 
>   - Multi-role framework support
>   - Executor authentication support
>   - Allow frameworks to modify their roles.
>   - Hierarchical roles (*EXPERIMENTAL*)
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.3.0-rc1
> 
> 
>
> The candidate for Mesos 1.3.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/mesos-1.3.0.tar.gz
>
> The tag to be voted on is 1.3.0-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.3.0-rc1
>
> The MD5 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/
> mesos-1.3.0.tar.gz.md5
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.0-rc1/
> mesos-1.3.0.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is up in Maven in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1190
>
> Please vote on releasing this package as Apache Mesos 1.3.0!
>
> The vote is open until Wed May 10 11:59:59 PDT 2017 and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.3.0
> [ ] -1 Do not release this package because ...
>
> Thanks,
>
> MPark & Neil
>

Re: Choice between LOG(FATAL) and EXIT(EXIT_FAILURE)

[James Peach]

> On May 8, 2017, at 2:02 PM, Zhitao Li  wrote:
> 
> Hi Vinod,
> 
> I'm reviving this old conversation from last year.
> 
> We are feeling some operational pain again, mostly due to journald
> truncates the stderr of Mesos agent so we cannot see a full exit message
> even in journald, and the error is not in GLOG output at all.
> 
> I filed https://issues.apache.org/jira/browse/MESOS-7472 to track this. We
> can submit the patch as long as some committer can shepherd this.


https://reviews.apache.org/r/56681/


> 
> Thanks!
> 
> On Fri, Sep 16, 2016 at 2:17 PM, Vinod Kone  wrote:
> 
>> We typically used LOG(FATAL) when we were interested in the stack trace. If
>> not, we preferred to use EXIT(EXIT_FAILURE). While that was the original
>> intention, not sure if we have been following that distinction diligently.
>> 
>> Separately, we should fix EXIT to log at ERROR level instead of just
>> printing to to stderr.
>> 
>> On Tue, Aug 30, 2016 at 10:42 AM, Zhitao Li  wrote:
>> 
>>> Hi,
>>> 
>>> Can someone explain better about when we should use LOG(FATAL) and when
>>> EXIT(EXIT_FAILURE) in Mesos codebase?
>>> 
>>> One thing is that EXIT(EXIT_FAILURE) does not seem to leave anything in
>> the
>>> level separated GLOG files so it could be mysterious to people who relies
>>> on that for debugging issues.
>>> 
>>> I see we have about 100 call sites to LOG(FATAL) and 200 call sites
>>> to EXIT(EXIT_FAILURE) at the moment.
>>> 
>>> Many thanks!
>>> 
>>> --
>>> Cheers,
>>> 
>>> Zhitao Li
>>> 
>> 
> 
> 
> 
> -- 
> Cheers,
> 
> Zhitao Li

Re: Version numbers during development

[Vinod Kone]

We need to make sure that our release scripts in "support/" and CI jobs
work with this change. Quite a few of them depend on the version number.

Also, I know of at least one organization that does version parsing in
their build scripts. I want to make sure we don't break such orgs.

So we should give more time for people to give feedback.

On Mon, May 8, 2017 at 1:58 PM, Neil Conway  wrote:

> Per offline discussion with mpark, he suggested a preference for
> "-dev" rather than "-devel", which sounds fine to me.
>
> Based on the lack of objections, I'll plan on making this change
> shortly: (1) changing the version number in master to "1.4.0-dev", and
> (2) updating the release process documentation accordingly.
>
> If you have any concerns, please let me know.
>
> Neil
>
>
> On Fri, May 5, 2017 at 4:29 PM, Neil Conway  wrote:
> > In my experience, this is reasonably common. For example, Postgres
> > uses version numbers like "9.5devel" to identify the software in the
> > development period before the next release process starts.
> >
> > Neil
> >
> > On Fri, May 5, 2017 at 4:23 PM, Vinod Kone  wrote:
> >> Is this a common practice that autotools based projects follow?
> >>
> >> For publish snapshot JARs to maven for example, we just add "-SNAPSHOT"
> to
> >> the version tag (and hence to the JAR name) before publishing it to
> maven;
> >> without a need to change the version in source control.
> >>
> >> On Fri, May 5, 2017 at 1:27 PM, Zhitao Li 
> wrote:
> >>
> >>> +1
> >>>
> >>> Sent from my iPhone
> >>>
> >>> > On May 5, 2017, at 12:56 PM, Neil Conway 
> wrote:
> >>> >
> >>> > Our current practice is that when we create a branch for version X,
> we
> >>> > bump the version number in the "master" branch to X+1. For example,
> we
> >>> > just created the 1.3.x branch, and bumped the version number in
> master
> >>> > to "1.4.0".
> >>> >
> >>> > Proposal: we should instead use a version number like "1.4.0-devel"
> in
> >>> > the master branch. When the 1.4.x release branch is created, the
> first
> >>> > commit in that branch would switch to use the "1.4.0" version number.
> >>> > Meanwhile, master would be bumped to use "1.5.0-devel".
> >>> >
> >>> > The main benefit is to make it easier to distinguish official Mesos
> >>> > releases from snapshots that are taken from the master branch at some
> >>> > point during development. Note that according to SemVer,
> "1.4.0-devel"
> >>> > is considered to be "older" than "1.4.0", which is the behavior we'd
> >>> > want.
> >>> >
> >>> > Neil
> >>>
>

Re: Choice between LOG(FATAL) and EXIT(EXIT_FAILURE)

[Zhitao Li]

Hi Vinod,

I'm reviving this old conversation from last year.

We are feeling some operational pain again, mostly due to journald
truncates the stderr of Mesos agent so we cannot see a full exit message
even in journald, and the error is not in GLOG output at all.

I filed https://issues.apache.org/jira/browse/MESOS-7472 to track this. We
can submit the patch as long as some committer can shepherd this.

Thanks!

On Fri, Sep 16, 2016 at 2:17 PM, Vinod Kone  wrote:

> We typically used LOG(FATAL) when we were interested in the stack trace. If
> not, we preferred to use EXIT(EXIT_FAILURE). While that was the original
> intention, not sure if we have been following that distinction diligently.
>
> Separately, we should fix EXIT to log at ERROR level instead of just
> printing to to stderr.
>
> On Tue, Aug 30, 2016 at 10:42 AM, Zhitao Li  wrote:
>
> > Hi,
> >
> > Can someone explain better about when we should use LOG(FATAL) and when
> > EXIT(EXIT_FAILURE) in Mesos codebase?
> >
> > One thing is that EXIT(EXIT_FAILURE) does not seem to leave anything in
> the
> > level separated GLOG files so it could be mysterious to people who relies
> > on that for debugging issues.
> >
> > I see we have about 100 call sites to LOG(FATAL) and 200 call sites
> > to EXIT(EXIT_FAILURE) at the moment.
> >
> > Many thanks!
> >
> > --
> > Cheers,
> >
> > Zhitao Li
> >
>



-- 
Cheers,

Zhitao Li

Re: Version numbers during development

[Neil Conway]

Per offline discussion with mpark, he suggested a preference for
"-dev" rather than "-devel", which sounds fine to me.

Based on the lack of objections, I'll plan on making this change
shortly: (1) changing the version number in master to "1.4.0-dev", and
(2) updating the release process documentation accordingly.

If you have any concerns, please let me know.

Neil


On Fri, May 5, 2017 at 4:29 PM, Neil Conway  wrote:
> In my experience, this is reasonably common. For example, Postgres
> uses version numbers like "9.5devel" to identify the software in the
> development period before the next release process starts.
>
> Neil
>
> On Fri, May 5, 2017 at 4:23 PM, Vinod Kone  wrote:
>> Is this a common practice that autotools based projects follow?
>>
>> For publish snapshot JARs to maven for example, we just add "-SNAPSHOT" to
>> the version tag (and hence to the JAR name) before publishing it to maven;
>> without a need to change the version in source control.
>>
>> On Fri, May 5, 2017 at 1:27 PM, Zhitao Li  wrote:
>>
>>> +1
>>>
>>> Sent from my iPhone
>>>
>>> > On May 5, 2017, at 12:56 PM, Neil Conway  wrote:
>>> >
>>> > Our current practice is that when we create a branch for version X, we
>>> > bump the version number in the "master" branch to X+1. For example, we
>>> > just created the 1.3.x branch, and bumped the version number in master
>>> > to "1.4.0".
>>> >
>>> > Proposal: we should instead use a version number like "1.4.0-devel" in
>>> > the master branch. When the 1.4.x release branch is created, the first
>>> > commit in that branch would switch to use the "1.4.0" version number.
>>> > Meanwhile, master would be bumped to use "1.5.0-devel".
>>> >
>>> > The main benefit is to make it easier to distinguish official Mesos
>>> > releases from snapshots that are taken from the master branch at some
>>> > point during development. Note that according to SemVer, "1.4.0-devel"
>>> > is considered to be "older" than "1.4.0", which is the behavior we'd
>>> > want.
>>> >
>>> > Neil
>>>

Re: [4/6] mesos git commit: Checked validity of master and agent version numbers on startup.

[Neil Conway]

There was an earlier thread on whether to ignore registration attempts
from pre-1.0 Mesos agents:

https://lists.apache.org/thread.html/f5e15f6d7a3f3b08d29e27455e2e1c801775418a148dded953c568e7@%3Cdev.mesos.apache.org%3E

We didn't explicitly discuss what to do when the compiled-in version
of Mesos is not parseable as SemVer. I'll start a separate thread to
let users know about that change.

Neil


On Mon, May 8, 2017 at 10:25 AM, Vinod Kone  wrote:
> @Neil: Have we sent an email about this change to dev list? This might
> break people who were directly building off source and using a custom
> version number.
>
> On Fri, May 5, 2017 at 4:54 PM,  wrote:
>
>> Checked validity of master and agent version numbers on startup.
>>
>> During startup, we now check that the compiled-in version number of the
>> master and agent can be parsed by stout's `Version::parse` (i.e., that
>> `MESOS_VERSION` is valid according to stout's extended variant of the
>> Semver 2.0.0 format).
>>
>> Review: https://reviews.apache.org/r/58708
>>
>>
>> Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
>> Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5a5dd8a4
>> Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5a5dd8a4
>> Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5a5dd8a4
>>
>> Branch: refs/heads/1.2.x
>> Commit: 5a5dd8a4edcb52e2227e2a4607b95a7dcc6aa321
>> Parents: c56851e
>> Author: Neil Conway 
>> Authored: Mon Mar 6 10:55:07 2017 -0800
>> Committer: Neil Conway 
>> Committed: Fri May 5 15:16:38 2017 -0700
>>
>> --
>>  src/master/main.cpp | 11 +++
>>  src/slave/main.cpp  | 11 +++
>>  2 files changed, 22 insertions(+)
>> --
>>
>>
>> http://git-wip-us.apache.org/repos/asf/mesos/blob/5a5dd8a4/
>> src/master/main.cpp
>> --
>> diff --git a/src/master/main.cpp b/src/master/main.cpp
>> index da75fe9..d485a06 100644
>> --- a/src/master/main.cpp
>> +++ b/src/master/main.cpp
>> @@ -57,6 +57,7 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>>
>>  #include "common/build.hpp"
>>  #include "common/http.hpp"
>> @@ -175,6 +176,16 @@ int main(int argc, char** argv)
>>  return EXIT_FAILURE;
>>}
>>
>> +  // Check that master's version has the expected format (SemVer).
>> +  {
>> +Try version = Version::parse(MESOS_VERSION);
>> +if (version.isError()) {
>> +  EXIT(EXIT_FAILURE)
>> +<< "Failed to parse Mesos version '" << MESOS_VERSION << "': "
>> +<< version.error();
>> +}
>> +  }
>> +
>>if (flags.ip_discovery_command.isSome() && flags.ip.isSome()) {
>>  EXIT(EXIT_FAILURE) << flags.usage(
>>  "Only one of `--ip` or `--ip_discovery_command` should be
>> specified");
>>
>> http://git-wip-us.apache.org/repos/asf/mesos/blob/5a5dd8a4/
>> src/slave/main.cpp
>> --
>> diff --git a/src/slave/main.cpp b/src/slave/main.cpp
>> index 31f2b4f..f90aa2f 100644
>> --- a/src/slave/main.cpp
>> +++ b/src/slave/main.cpp
>> @@ -39,6 +39,7 @@
>>  #include 
>>  #include 
>>  #include 
>> +#include 
>>
>>  #include "common/build.hpp"
>>  #include "common/http.hpp"
>> @@ -142,6 +143,16 @@ int main(int argc, char** argv)
>>  return EXIT_FAILURE;
>>}
>>
>> +  // Check that agent's version has the expected format (SemVer).
>> +  {
>> +Try version = Version::parse(MESOS_VERSION);
>> +if (version.isError()) {
>> +  EXIT(EXIT_FAILURE)
>> +<< "Failed to parse Mesos version '" << MESOS_VERSION << "': "
>> +<< version.error();
>> +}
>> +  }
>> +
>>if (flags.master.isNone() && flags.master_detector.isNone()) {
>>  cerr << flags.usage("Missing required option `--master` or "
>>  "`--master_detector`.") << endl;
>>
>>

Re: [4/6] mesos git commit: Checked validity of master and agent version numbers on startup.

[Vinod Kone]

@Neil: Have we sent an email about this change to dev list? This might
break people who were directly building off source and using a custom
version number.

On Fri, May 5, 2017 at 4:54 PM,  wrote:

> Checked validity of master and agent version numbers on startup.
>
> During startup, we now check that the compiled-in version number of the
> master and agent can be parsed by stout's `Version::parse` (i.e., that
> `MESOS_VERSION` is valid according to stout's extended variant of the
> Semver 2.0.0 format).
>
> Review: https://reviews.apache.org/r/58708
>
>
> Project: http://git-wip-us.apache.org/repos/asf/mesos/repo
> Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5a5dd8a4
> Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/5a5dd8a4
> Diff: http://git-wip-us.apache.org/repos/asf/mesos/diff/5a5dd8a4
>
> Branch: refs/heads/1.2.x
> Commit: 5a5dd8a4edcb52e2227e2a4607b95a7dcc6aa321
> Parents: c56851e
> Author: Neil Conway 
> Authored: Mon Mar 6 10:55:07 2017 -0800
> Committer: Neil Conway 
> Committed: Fri May 5 15:16:38 2017 -0700
>
> --
>  src/master/main.cpp | 11 +++
>  src/slave/main.cpp  | 11 +++
>  2 files changed, 22 insertions(+)
> --
>
>
> http://git-wip-us.apache.org/repos/asf/mesos/blob/5a5dd8a4/
> src/master/main.cpp
> --
> diff --git a/src/master/main.cpp b/src/master/main.cpp
> index da75fe9..d485a06 100644
> --- a/src/master/main.cpp
> +++ b/src/master/main.cpp
> @@ -57,6 +57,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>
>  #include "common/build.hpp"
>  #include "common/http.hpp"
> @@ -175,6 +176,16 @@ int main(int argc, char** argv)
>  return EXIT_FAILURE;
>}
>
> +  // Check that master's version has the expected format (SemVer).
> +  {
> +Try version = Version::parse(MESOS_VERSION);
> +if (version.isError()) {
> +  EXIT(EXIT_FAILURE)
> +<< "Failed to parse Mesos version '" << MESOS_VERSION << "': "
> +<< version.error();
> +}
> +  }
> +
>if (flags.ip_discovery_command.isSome() && flags.ip.isSome()) {
>  EXIT(EXIT_FAILURE) << flags.usage(
>  "Only one of `--ip` or `--ip_discovery_command` should be
> specified");
>
> http://git-wip-us.apache.org/repos/asf/mesos/blob/5a5dd8a4/
> src/slave/main.cpp
> --
> diff --git a/src/slave/main.cpp b/src/slave/main.cpp
> index 31f2b4f..f90aa2f 100644
> --- a/src/slave/main.cpp
> +++ b/src/slave/main.cpp
> @@ -39,6 +39,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>
>  #include "common/build.hpp"
>  #include "common/http.hpp"
> @@ -142,6 +143,16 @@ int main(int argc, char** argv)
>  return EXIT_FAILURE;
>}
>
> +  // Check that agent's version has the expected format (SemVer).
> +  {
> +Try version = Version::parse(MESOS_VERSION);
> +if (version.isError()) {
> +  EXIT(EXIT_FAILURE)
> +<< "Failed to parse Mesos version '" << MESOS_VERSION << "': "
> +<< version.error();
> +}
> +  }
> +
>if (flags.master.isNone() && flags.master_detector.isNone()) {
>  cerr << flags.usage("Missing required option `--master` or "
>  "`--master_detector`.") << endl;
>
>

Re: documenting test expactations

[Neil Conway]

My two cents:

(1) I almost always need to read the test to understand/debug a test failure.

(2) As long as the intent of the test is clear, I'm not picky about
whether clarifying comments take the form of C++ comments or
explanatory EXPECT messages.

One thing I would be opposed to is adding explanatory EXPECT messages
in an indiscriminate way; IMO that just adds redundancy and violates
DRY (most test expectations are pretty self-explanatory).

Neil

On Mon, May 8, 2017 at 8:24 AM, James Peach  wrote:
>
>> On May 1, 2017, at 4:28 PM, Benjamin Mahler  wrote:
>>
>> Do you have some examples?
>
> I think that this:
>
> EXPECT_EQ(Bytes(512u), BasicBlocks(Bytes(128)).bytes())
> << "a partial block should round up";
>
> is a strict superset of this:
>
> // A partial block should round up.
> EXPECT_EQ(Bytes(512u), BasicBlocks(Bytes(128)).bytes())
>
> The former is preferable since the person triaging test failures gets the 
> immediate context of what the expectation is doing. This is valuable even if 
> you might also find you need to check the source.
>
>>
>> Thinking through my own experience debugging tests, I tend to only get
>> value out of EXPECT messages when they are providing information that I
>> can't get access to from the line number / actual vs expected printing.
>> (e.g. the value of a variable). If the EXPECT message is simply explaining
>> what the test is doing, then I tend to ignore it and read the test instead,
>> so it would be helpful to discuss some examples to get a better sense. :)
>>
>> On Sat, Apr 29, 2017 at 10:02 AM, James Peach  wrote:
>>
>>> Hi all,
>>>
>>> In a couple of reviews, I've been asked to avoid emitting explanatory
>>> messages from the EXPECT() macro. The rationale for this is that tests
>>> usually use comments. However, I think that emitting the reason for a
>>> failed expectation into the test log is pretty helpful and we should do it
>>> more often.
>>>
>>> What do people think about explicitly allowing (or even encouraging) this?
>>> ie. EXPECT(...) << "some explanation goes here"
>>>
>>> J
>

Re: documenting test expactations

[James Peach]

> On May 1, 2017, at 4:28 PM, Benjamin Mahler  wrote:
> 
> Do you have some examples?

I think that this:

EXPECT_EQ(Bytes(512u), BasicBlocks(Bytes(128)).bytes())
<< "a partial block should round up";

is a strict superset of this:

// A partial block should round up.
EXPECT_EQ(Bytes(512u), BasicBlocks(Bytes(128)).bytes())

The former is preferable since the person triaging test failures gets the 
immediate context of what the expectation is doing. This is valuable even if 
you might also find you need to check the source.

> 
> Thinking through my own experience debugging tests, I tend to only get
> value out of EXPECT messages when they are providing information that I
> can't get access to from the line number / actual vs expected printing.
> (e.g. the value of a variable). If the EXPECT message is simply explaining
> what the test is doing, then I tend to ignore it and read the test instead,
> so it would be helpful to discuss some examples to get a better sense. :)
> 
> On Sat, Apr 29, 2017 at 10:02 AM, James Peach  wrote:
> 
>> Hi all,
>> 
>> In a couple of reviews, I've been asked to avoid emitting explanatory
>> messages from the EXPECT() macro. The rationale for this is that tests
>> usually use comments. However, I think that emitting the reason for a
>> failed expectation into the test log is pretty helpful and we should do it
>> more often.
>> 
>> What do people think about explicitly allowing (or even encouraging) this?
>> ie. EXPECT(...) << "some explanation goes here"
>> 
>> J