compactor
compactor is a pure python implementation of libprocess. I announced it a while back while it was still in the very alpha stage. I've spent the last couple days fixing a bunch of compactor bugs and generally getting it into better shape as we consider running pesos (a pure python implementation of the mesos framework api) in production for the Aurora executor. I've added documentation here: compactor.readthedocs.org The code is available here: https://github.com/wickman/compactor The latest version is published to pypi, so it can be pip installed or pexed to your heart's content. I'd love to hear feedback. It seems like it would be generally useful to be able to write python Processes e.g. lightweight test Processes like allocators, authenticators or health checkers using this. thanks, brian
Re: Review Request 31337: Created a TestLauncher that mocks Launcher.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31337/#review73745 --- boom 31337 - Brian Wickman On Feb. 24, 2015, 1:52 a.m., Vinod Kone wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31337/ --- (Updated Feb. 24, 2015, 1:52 a.m.) Review request for mesos and Jie Yu. Bugs: MESOS-1690 https://issues.apache.org/jira/browse/MESOS-1690 Repository: mesos Description --- Needed for subsequent review. Diffs - src/Makefile.am 65cba64834d8d2224761105516c64c6ba203f9f3 src/tests/launcher.hpp PRE-CREATION Diff: https://reviews.apache.org/r/31337/diff/ Testing --- Tested in subsequent review. Thanks, Vinod Kone
Re: Crashed task is not reaped
A crashed thread does not terminate the Python interpreter, so the executor here will stay alive. If you want an abnormal thread exit to result in an executor termination, you will have to implement that behavior explicitly. We use a thread liveness detector that looks something like: https://gist.github.com/wickman/dc11896d782f9a2160b8 When you create a thread, you do registry.register(thread). That thread should call registry.unregister(self) prior to terminating normally. If it terminates abnormally, the registry.dead event will be set. Our MainThread in practice just does something like: while registry.dead.wait(timeout=10): pass We also have a library (twitter.common.exceptions on pypi) that provides a class called ExceptionalThread which guarantees that sys.excepthook() is called. You could implement similar behavior by making all threads ExceptionalThreads and wrapping sys.excepthook with something that provides an event to MainThread to signal termination as described above. ~brian On Wed, Aug 27, 2014 at 2:55 PM, Alex Rukletsov a...@mesosphere.io wrote: While playing with Rendler https://github.com/mesosphere/RENDLER I noticed that if the task (read: python executor) crashes, the underlying executor stays alive and therefore is not reaped, which renders the task running indefinitely. Here https://gist.github.com/rukletsov/4a74743c5b67f304e661 is a part of the slave log (exception itself doesn't matter, it's there to test the behaviour). Not sure, whether it's a bug or a feature, for me it looks like a bug. Regards, Alex
Re: Review Request 23224: Refactored the python bindings into multiple modules.
On July 24, 2014, 3:53 p.m., Benjamin Hindman wrote: Looks good Thomas, just a few cleanups and we'll get this committed! Also, I still have a few questions regarding the naming/structure of the modules. ;-) In particular, if we didn't have a native component, would we still create sub modules and an empty metapackage? Or at that point would we just put everything inside of the 'mesos' module (including the protobufs). Given that we'll be killing the native component in due time I don't want us to do something that is specific to that now. Thomas Rampelberg wrote: I still like having the sub-modules. It might be overly-verbose, but it is nice to have the protobufs and general interface stuff moved away out of the main package (freeing the main package up for the interface that external users would use). Using meta-packages, we can fold everything back to a single mesos module in the future when it makes sense as most people will be doing `pip install mesos` to begin with. I also like the submodules, since it means things like pesos can just pull in mesos.api (or whatever it's eventually called) which is just a dependency-free interface. We can always choose to consolidate later. - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/#review48480 --- On July 8, 2014, 5:58 p.m., Thomas Rampelberg wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/ --- (Updated July 8, 2014, 5:58 p.m.) Review request for mesos. Bugs: MESOS-857 https://issues.apache.org/jira/browse/MESOS-857 Repository: mesos-git Description --- The existing module has been split into three separate ones: - mesos.api - This contains the stub implementations for the Executor/Scheduler - mesos.native - The old _mesos module. - mesos.protocol - All protobufs. There is also a base metapackage `mesos` that allows a potential `pip install mesos` to correctly install everything required. While mesos.api and mesos.protocol can now be uploaded to the cheeseshop, mesos.native has not changed and will need some more work first. Diffs - Makefile.am b91d8cf011832e6e91b16f03a2d80fbb601eba8f configure.ac e7472081339fc9c773eb2cf2d5f15dc459ac378d mpi/mpiexec-mesos.in da0733fc29f97e67385cab55d60d4e2afd76aba9 mpi/mpiexec-mesos.py 0ab50167eaa43f9d69f37b7c10e26fa7a7d9f250 src/Makefile.am 45afcd145f3b502043424a6dac2197979aefbca2 src/examples/python/test-containerizer.in 569519b3b9755959f9bf931d3c81be9a00b64bc9 src/examples/python/test-executor.in 7e8875f0fd74dc9f9207986864edbce588ec3fb8 src/examples/python/test-framework.in c4683b97b87ba8753e842b0c75cc3d65140a5cf7 src/examples/python/test_containerizer.py c65d891539bcee775741626596997afe8471c930 src/examples/python/test_executor.py 065b50a6146cb39a82024d82c20cf89f940a9e57 src/examples/python/test_framework.py fce090fe542e3863770d7daea3d8764da1d8d5df src/python/api/setup.py.in PRE-CREATION src/python/api/src/mesos/__init__.py PRE-CREATION src/python/native/ext_def.py.in PRE-CREATION src/python/native/mesos_executor_driver_impl.hpp src/python/native/mesos_executor_driver_impl.cpp src/python/native/mesos_scheduler_driver_impl.hpp src/python/native/mesos_scheduler_driver_impl.cpp src/python/native/module.hpp src/python/native/module.cpp b94712681e6f0e9bf5dfdafa10621d1df82dc367 src/python/native/proxy_executor.hpp src/python/native/proxy_executor.cpp src/python/native/proxy_scheduler.hpp src/python/native/proxy_scheduler.cpp src/python/native/setup.py.in PRE-CREATION src/python/native/src/mesos/__init__.py PRE-CREATION src/python/native/src/mesos/native/__init__.py PRE-CREATION src/python/protocol/setup.py.in PRE-CREATION src/python/protocol/src/mesos/__init__.py PRE-CREATION src/python/protocol/src/mesos/protocol/__init__.py PRE-CREATION src/python/setup.py.in b996dfef5c7a6d330991522bf0047ed3cac6760d src/python/src/mesos.py 0152ab456f072f8d4a1c4ab19fe74e181eadbd05 src/python/src/mesos/__init__.py PRE-CREATION Diff: https://reviews.apache.org/r/23224/diff/ Testing --- Thanks, Thomas Rampelberg
Re: Mesos/Libprocess API
They can be installed via setuptools -- the repositories have setup.py files included at the root so that source distributions can be made. Nothing's been published to pypi yet though, since I don't feel they're even ready for an 0.1 release. I pantsified the repositories to make it easier to run tests, but am happy to toxify them instead -- whatever people are most comfortable with. Re: BUILD vs build, you should be able to specify an alternate build path for setuptools so that it doesn't conflict when building distributions. On Wed, Jul 23, 2014 at 12:01 AM, Tom Arnfeld t...@duedil.com wrote: R.E Python 3.3; It doesn't actually require it, but it's importing asyncio directly instead of using the standard try/catch around the import and falling back to trollius (the Python2 port). We can probably import via tornado, which does this try/catch for us. Right now both compactor and pesos can't be installed with setuptools due to the BUILD files (setuptools tries to create build folders and the name conflicts). I'll be fixing this up today by removing pants, at least just to get it working... not sure how Brian feels about that yet ;-) I can ping the mailing list once both compactor and pesos are up and running in an easy-to-use way (next few hours). I've been using https://github.com/tarnfeld/mesos-python-framework as a test framework skeleton, but that's also not easy to set up just yet. Note: Regardless of the above, you need to have installed our fork of tornado for anything to work beyond the first round of messages. Cheers, Tom. On 23 Jul 2014, at 07:42, Vetoshkin Nikita nikita.vetosh...@gmail.com wrote: Hi, Tom! I would gladly help you to debug if you could provide some information about your setup. Is it localhost only communication? Any code snippet to reproduce the problem? P.S. I'm trying to setup pesos and it seems like python3.3 is a requirement but it isn't mentioned anywhere. On Tue, Jul 22, 2014 at 1:45 AM, Tom Arnfeld t...@duedil.com wrote: Hey, I've started to try and finish off the work @wickman started around pesos[1] and compactor[2] - pure language bindings for mesos and libprocess in Python. It's currently far from finished, but have fun into a brick wall around libprocess. If anyone could shed any light that'd be great. To start with, I saw the framework register but disconnect immediately. From a quick chat on IRC someone mentioned this could be related to not keeping the inbound message connection open, the one that sends mesos.internal.FrameworkRegisteredMessage. In doing this, the framework does register with the master and shows up in the UI – however no further messages are received at all. I'm keeping the outbound connection open already. Mesos seems to think it's sending offers (they show up in the logs and are showing in the Offers page) but the master never gets past there, and the framework never receives the HTTP connections. No doubt this is a bug in the socket logic on my end. Struggling to find my way through the libprocess source to figure it out. Any pointers would be much appreciated. Thanks! Tom. [1] https://github.com/wickman/pesos [2] https://github.com/wickman/compactor
Re: Review Request 23224: Refactored the python bindings into multiple modules.
On July 3, 2014, 12:10 a.m., Brian Wickman wrote: src/Makefile.am, line 137 https://reviews.apache.org/r/23224/diff/1/?file=622309#file622309line137 i would love to see all protos contained here, including messages protos. this will allow for the development of a pure python api. mind adding those? or possibly create a mesos.api_internal? Thomas Rampelberg wrote: Would you open a new JIRA for that? I'd like to have a separate patch for that. Filed https://issues.apache.org/jira/browse/MESOS-1561 On July 3, 2014, 12:10 a.m., Brian Wickman wrote: src/python/native/setup.py.in, line 29 https://reviews.apache.org/r/23224/diff/4/?file=623204#file623204line29 what's your philosophy on versioning here? should we always require deps==version or just deps=major,major+1 or similar? can always punt for later. Thomas Rampelberg wrote: At the moment, the separate modules all have versions identical to mesos' (PACKAGE_VERSION). While I'm not sure that is the correct decision in the long term, it works for now. Given that, it seems to make sense to me that the inter-dependencies are `deps==version` (as they're all fixed to the mesos version itself). sgtm - Brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/#review47259 --- On July 3, 2014, 12:46 a.m., Thomas Rampelberg wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/ --- (Updated July 3, 2014, 12:46 a.m.) Review request for mesos. Bugs: MESOS-857 https://issues.apache.org/jira/browse/MESOS-857 Repository: mesos-git Description --- The existing module has been split into three separate ones: - mesos.api - This contains the stub implementations for the Executor/Scheduler - mesos.native - The old _mesos module. - mesos.protocol - All protobufs. There is also a base metapackage `mesos` that allows a potential `pip install mesos` to correctly install everything required. While mesos.api and mesos.protocol can now be uploaded to the cheeseshop, mesos.native has not changed and will need some more work first. Diffs - Makefile.am b91d8cf011832e6e91b16f03a2d80fbb601eba8f configure.ac e7472081339fc9c773eb2cf2d5f15dc459ac378d mpi/mpiexec-mesos.in da0733fc29f97e67385cab55d60d4e2afd76aba9 mpi/mpiexec-mesos.py 0ab50167eaa43f9d69f37b7c10e26fa7a7d9f250 src/Makefile.am e3ff6d71d9324ea8376c14fae056568452f22bdc src/examples/python/test-containerizer.in 569519b3b9755959f9bf931d3c81be9a00b64bc9 src/examples/python/test-executor.in 7e8875f0fd74dc9f9207986864edbce588ec3fb8 src/examples/python/test-framework.in c4683b97b87ba8753e842b0c75cc3d65140a5cf7 src/examples/python/test_containerizer.py c65d891539bcee775741626596997afe8471c930 src/examples/python/test_executor.py 065b50a6146cb39a82024d82c20cf89f940a9e57 src/examples/python/test_framework.py fce090fe542e3863770d7daea3d8764da1d8d5df src/python/api/setup.py.in PRE-CREATION src/python/api/src/mesos/__init__.py PRE-CREATION src/python/native/ext_def.py.in PRE-CREATION src/python/native/mesos_executor_driver_impl.hpp src/python/native/mesos_executor_driver_impl.cpp src/python/native/mesos_scheduler_driver_impl.hpp src/python/native/mesos_scheduler_driver_impl.cpp src/python/native/module.hpp src/python/native/module.cpp b94712681e6f0e9bf5dfdafa10621d1df82dc367 src/python/native/proxy_executor.hpp src/python/native/proxy_executor.cpp src/python/native/proxy_scheduler.hpp src/python/native/proxy_scheduler.cpp src/python/native/setup.py.in PRE-CREATION src/python/native/src/mesos/__init__.py PRE-CREATION src/python/native/src/mesos/native/__init__.py PRE-CREATION src/python/protocol/setup.py.in PRE-CREATION src/python/protocol/src/mesos/__init__.py PRE-CREATION src/python/setup.py.in b996dfef5c7a6d330991522bf0047ed3cac6760d src/python/src/mesos.py 0152ab456f072f8d4a1c4ab19fe74e181eadbd05 src/python/src/mesos/__init__.py PRE-CREATION Diff: https://reviews.apache.org/r/23224/diff/ Testing --- Thanks, Thomas Rampelberg
Re: Review Request 23224: Refactored the python bindings into multiple modules.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/#review47213 --- This seems to be missing the setup.py(.in)s from mesos.api, mesos.native, mesos.protocol. Forget to git add? Otherwise this is looking great! - Brian Wickman On July 2, 2014, 1:32 a.m., Thomas Rampelberg wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/ --- (Updated July 2, 2014, 1:32 a.m.) Review request for mesos. Bugs: MESOS-857 https://issues.apache.org/jira/browse/MESOS-857 Repository: mesos-git Description --- The existing module has been split into three separate ones: - mesos.api - This contains the stub implementations for the Executor/Scheduler - mesos.native - The old _mesos module. - mesos.protocol - All protobufs. There is also a base metapackage `mesos` that allows a potential `pip install mesos` to correctly install everything required. While mesos.api and mesos.protocol can now be uploaded to the cheeseshop, mesos.native has not changed and will need some more work first. Diffs - Makefile.am b91d8cf011832e6e91b16f03a2d80fbb601eba8f configure.ac e7472081339fc9c773eb2cf2d5f15dc459ac378d mpi/mpiexec-mesos.in da0733fc29f97e67385cab55d60d4e2afd76aba9 mpi/mpiexec-mesos.py 0ab50167eaa43f9d69f37b7c10e26fa7a7d9f250 src/Makefile.am 918b0d04a5de69a9213e3d31c8f9424756e4ade5 src/examples/python/test-containerizer.in 569519b3b9755959f9bf931d3c81be9a00b64bc9 src/examples/python/test-executor.in 7e8875f0fd74dc9f9207986864edbce588ec3fb8 src/examples/python/test-framework.in c4683b97b87ba8753e842b0c75cc3d65140a5cf7 src/examples/python/test_containerizer.py c65d891539bcee775741626596997afe8471c930 src/examples/python/test_executor.py 065b50a6146cb39a82024d82c20cf89f940a9e57 src/examples/python/test_framework.py fce090fe542e3863770d7daea3d8764da1d8d5df src/python/api/src/mesos/__init__.py PRE-CREATION src/python/native/mesos_executor_driver_impl.hpp src/python/native/mesos_executor_driver_impl.cpp src/python/native/mesos_scheduler_driver_impl.hpp src/python/native/mesos_scheduler_driver_impl.cpp src/python/native/module.hpp src/python/native/module.cpp b94712681e6f0e9bf5dfdafa10621d1df82dc367 src/python/native/proxy_executor.hpp src/python/native/proxy_executor.cpp src/python/native/proxy_scheduler.hpp src/python/native/proxy_scheduler.cpp src/python/native/src/mesos/__init__.py PRE-CREATION src/python/native/src/mesos/native/__init__.py PRE-CREATION src/python/protocol/src/mesos/__init__.py PRE-CREATION src/python/setup.py.in b996dfef5c7a6d330991522bf0047ed3cac6760d src/python/src/mesos.py 0152ab456f072f8d4a1c4ab19fe74e181eadbd05 src/python/src/mesos/__init__.py PRE-CREATION Diff: https://reviews.apache.org/r/23224/diff/ Testing --- Thanks, Thomas Rampelberg
Re: Review Request 23224: Refactored the python bindings into multiple modules.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/#review47259 --- thanks for doing this work. this will be a huge improvement going forward and opens the possibility of pure python implementations of frameworks. exciting! src/Makefile.am https://reviews.apache.org/r/23224/#comment82924 i would love to see all protos contained here, including messages protos. this will allow for the development of a pure python api. mind adding those? or possibly create a mesos.api_internal? src/python/native/ext_def.py.in https://reviews.apache.org/r/23224/#comment82917 not sure about process but does this need the ASF license headers? src/python/native/setup.py.in https://reviews.apache.org/r/23224/#comment82919 what's your philosophy on versioning here? should we always require deps==version or just deps=major,major+1 or similar? can always punt for later. src/python/protocol/setup.py.in https://reviews.apache.org/r/23224/#comment82920 same here, to be conservative, might want protobuf=2.5.0,3 - Brian Wickman On July 2, 2014, 11:53 p.m., Thomas Rampelberg wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23224/ --- (Updated July 2, 2014, 11:53 p.m.) Review request for mesos. Bugs: MESOS-857 https://issues.apache.org/jira/browse/MESOS-857 Repository: mesos-git Description --- The existing module has been split into three separate ones: - mesos.api - This contains the stub implementations for the Executor/Scheduler - mesos.native - The old _mesos module. - mesos.protocol - All protobufs. There is also a base metapackage `mesos` that allows a potential `pip install mesos` to correctly install everything required. While mesos.api and mesos.protocol can now be uploaded to the cheeseshop, mesos.native has not changed and will need some more work first. Diffs - Makefile.am b91d8cf011832e6e91b16f03a2d80fbb601eba8f configure.ac e7472081339fc9c773eb2cf2d5f15dc459ac378d mpi/mpiexec-mesos.in da0733fc29f97e67385cab55d60d4e2afd76aba9 mpi/mpiexec-mesos.py 0ab50167eaa43f9d69f37b7c10e26fa7a7d9f250 src/Makefile.am e3ff6d71d9324ea8376c14fae056568452f22bdc src/examples/python/test-containerizer.in 569519b3b9755959f9bf931d3c81be9a00b64bc9 src/examples/python/test-executor.in 7e8875f0fd74dc9f9207986864edbce588ec3fb8 src/examples/python/test-framework.in c4683b97b87ba8753e842b0c75cc3d65140a5cf7 src/examples/python/test_containerizer.py c65d891539bcee775741626596997afe8471c930 src/examples/python/test_executor.py 065b50a6146cb39a82024d82c20cf89f940a9e57 src/examples/python/test_framework.py fce090fe542e3863770d7daea3d8764da1d8d5df src/python/api/setup.py.in PRE-CREATION src/python/api/src/mesos/__init__.py PRE-CREATION src/python/native/ext_def.py.in PRE-CREATION src/python/native/mesos_executor_driver_impl.hpp src/python/native/mesos_executor_driver_impl.cpp src/python/native/mesos_scheduler_driver_impl.hpp src/python/native/mesos_scheduler_driver_impl.cpp src/python/native/module.hpp src/python/native/module.cpp b94712681e6f0e9bf5dfdafa10621d1df82dc367 src/python/native/proxy_executor.hpp src/python/native/proxy_executor.cpp src/python/native/proxy_scheduler.hpp src/python/native/proxy_scheduler.cpp src/python/native/setup.py.in PRE-CREATION src/python/native/src/mesos/__init__.py PRE-CREATION src/python/native/src/mesos/native/__init__.py PRE-CREATION src/python/protocol/setup.py.in PRE-CREATION src/python/protocol/src/mesos/__init__.py PRE-CREATION src/python/setup.py.in b996dfef5c7a6d330991522bf0047ed3cac6760d src/python/src/mesos.py 0152ab456f072f8d4a1c4ab19fe74e181eadbd05 src/python/src/mesos/__init__.py PRE-CREATION Diff: https://reviews.apache.org/r/23224/diff/ Testing --- Thanks, Thomas Rampelberg
[jira] [Created] (MESOS-1360) if POST Content-Type is application/json, translate JSON-Protobuf
brian wickman created MESOS-1360: Summary: if POST Content-Type is application/json, translate JSON-Protobuf Key: MESOS-1360 URL: https://issues.apache.org/jira/browse/MESOS-1360 Project: Mesos Issue Type: Improvement Components: libprocess Reporter: brian wickman This seems like a quick and easy way to get easier interoperability between ProtobufProcesses and external implementations. If you see a POST with Content-Type of application/json instead of application/x-protobuf (or whatever), attempt to do json-protobuf translation instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MESOS-857) restructure mesos python namespace
[ https://issues.apache.org/jira/browse/MESOS-857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] brian wickman updated MESOS-857: Priority: Major (was: Minor) Issue Type: Improvement (was: Task) Summary: restructure mesos python namespace (was: split mesos and mesos_pb2 distributions) restructure mesos python namespace -- Key: MESOS-857 URL: https://issues.apache.org/jira/browse/MESOS-857 Project: Mesos Issue Type: Improvement Components: python api Reporter: brian wickman Right now the mesos_pb2 and mesos dependencies are bundled together into the mesos egg. We have some tooling that uses just the compiled protobufs, but because they're lumped together with the mesos egg, we get all the dependency/platform nightmare that comes along with it, not to mention the bloat of including 20MB of .so files. This proposes splitting the mesos protobufs into a separate mesos_pb distribution that the mesos distribution should depend upon via install_requires (e.g. mesos_pb==0.15.0-rc4) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MESOS-857) restructure mesos python namespace
[
https://issues.apache.org/jira/browse/MESOS-857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985801#comment-13985801
]
brian wickman commented on MESOS-857:
-
I propose that we restructure the mesos python project. Right now it's
fractured haphazardly, yet there are idioms made available by the python
packaging ecosystem to do this correctly.
For example, there is src/cli which is a mishmash of C++ and python, which
contains a redeclaration of 'mesos' in unpackaged form which would conflict
with the existing code in src/python. Now src/python bundles mesos_pb2.py,
mesos.py and _mesos.so in a top-level namespace. Ordinarily if you 'pip
install baz', you expect one top level package name and everything residing
underneath, e.g. 'import baz' with baz.foo, baz.bar, baz.bak subpackages.
We should structure the mesos namespace such that bits and pieces of mesos can
be installed a la carte. Right now you have to go all-in, bringing in C
extensions (which are challenging to build and have no pure source distribution
available yet) which is a hindrance for adoption.
It seems reasonable that I might just want API stubs or the code-generated
protobuf classes or just the CLI. We can do this in a few ways, but it means
splitting everything into different packages with dependencies between each
(codified by install_requires in setup.py.) The following proposal uses a
top-level 'mesos' namespace package, but it could be done with separate
top-level packages, e.g. mesos_api, mesos_driver, instead of mesos.api or
mesos.driver.
I propose the following packages (which would also mirror the import namespace):
{noformat}
mesos [nspkg]
mesos.api [pkg]
mesos.cli [pkg]
mesos.driver [pkg]
mesos.native [pkg]
mesos.protocol [pkg]
{noformat}
mesos should be a namespace package: it contains no symbols. But by default it
would have install_requires on everything provided within the mesos project, so
that 'pip install mesos' does approximately the correct thing. But in and of
itself, it would contain no sources.
mesos.api should contain just the Scheduler, SchedulerDriver, Executor,
ExecutorDriver (and in the future, possibly Log, LogDriver, Containerizer,
ContainerizerDriver) stubs. it has no dependencies on anything else.
mesos.cli should contain all the CLI commands. it also shouldn't need to
depend on any other packages except maybe mesos.protocol. we can use the
console_scripts entry point in mesos.cli to handle script installation (see
http://www.scotttorborg.com/python-packaging/command-line-scripts.html#the-console-scripts-entry-point
). this means 'pip install mesos.cli' would create wrapper scripts for
mesos-cat, mesos-ps, etc, that correctly invoke the underlying python modules
with all the dependencies set up correctly, and put onto the $PATH in the same
place as your python interpreter.
mesos.driver should be a package that is a small wrapper around pkg_resources
find_packages + get_entry_map and used to detect any python packages in the
environment exporting concrete driver implementations (e.g.
_mesos.MesosSchedulerDriver or _mesos.MesosExecutorDriver.) this would be done
via EntryPoints (see
https://pythonhosted.org/setuptools/pkg_resources.html#entry-points )
mesos.native should be the package that contains _mesos.so and entry_point
metadata expected by mesos.driver in the setup.py. we could even go so far as
to publish mesos.native.el5 or mesos.native.el6 binary wheels to PyPI in order
to differentiate linux ABIs, but have them correctly detected and picked up by
mesos.driver at runtime. this strategy is also compatible with the pesos
project (https://github.com/wickman/pesos ), which would just publish
PesosSchedulerDriver and PesosExecutorDriver entry points for mesos.driver,
allowing a pure python scheduler or executor to be implemented.
finally, mesos.protocol would be the package containing all of the
code-generated protobuf stubs. we could even split mesos.protocol out as a
namespace package with separate subpackages for mesos.protocol.pb,
mesos.protocol.json. currently protobuf only supports python 2.x (there are
some branches out there with support for 3.x but afaik there is no plan for
those to reach master.) mesos.protocol.pb would have an install_requires on
protobuf, and mesos.protocol.json would be dependency-free, and hence friendly
with python 3.x. ideally there would be helper messages for constructing the
body of libprocess messages (the wire protocol.) in the future that could be
ported over to the Event/Call interface that Ben has described.
in order to support legacy applications, we could have the mesos.legacy
package, which would map all the above names into their _mesos, mesos_pb2 and
mesos.* counterparts.
restructure mesos python namespace
--
Key: MESOS-857
[jira] [Commented] (MESOS-946) setup.py should be able to create publishable sdist
[ https://issues.apache.org/jira/browse/MESOS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13985054#comment-13985054 ] brian wickman commented on MESOS-946: - I went ahead and created a placeholder at https://pypi.python.org/pypi/mesos to prevent a non-project-related organization from swooping in and taking the top level project name (e.g. something that happened for Pants.) Once we have a publishable sdist, we should establish a mesos organization on PyPI and add all committers to it. setup.py should be able to create publishable sdist --- Key: MESOS-946 URL: https://issues.apache.org/jira/browse/MESOS-946 Project: Mesos Issue Type: Bug Components: python api Reporter: Kevin Sweeney Since setup.py uses the build root for compiler flags the sdist produced by python src/python/setup.py sdist can't be used outside the machine that built mesos, uploaded to PyPI, or used with a local PyPI mirror. Aurora's Python executor should be installable by `pip` and depend only on the presence of mesos headers and shared libraries, but since the produced sdist is unusable this currently isn't possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Mesos Wire Protocol Documentation
I've implemented a pure python version of the basic Process/ProtobufProcess mechanics and wire protocol. I haven't gone so far as Kevin and tried to register a framework or maybe talk to a replicated log, but it shouldn't be much more work: https://github.com/wickman/compactor Zero documentation but you can read the tests for a general idea of what's going on. cheers, brian On Sun, Apr 13, 2014 at 10:28 PM, Kevin Sweeney kevin.t.swee...@gmail.comwrote: I've got the start of a JVM verson on github - it can currently register a framework and parse the response. Client-side I still need to figure out how to properly configure Keep-Alive. Servlet can dispatch messages to message handlers in a type-safe way and returns a 202 for messages it's going to handle. Code's a mess currently. https://github.com/kevints/mesos-framework-api On Fri, Apr 11, 2014 at 7:12 PM, Benjamin Hindman b...@eecs.berkeley.eduwrote: First, my apologies for getting to this party so late. It's great to see people interested in helping create native-language Mesos libraries. Vladimir: my presentation was definitely referring to the the low-level protocol between master, framework (scheduler), slave, and executors. I'll do my best here to clarify how the current protocol works and what we need to do to get it to the point where we can write native-language libraries. (Eventually it would be great to move some of this into documentation as necessary.) As Nikita pointed out, the protocol is currently HTTP-like. As my presentation describes, think actors and one-way message passing when considering how the protocol works. To send a message an actor POSTs an HTTP request where the actor that is supposed to receive the message is the first component of the request path and the name of the message is the remaining part of the path. To distinguish one of these messages from a normal HTTP request we look to see if the 'User-Agent' is 'libprocess/...'. For example: POST /master/mesos.internal.RegisterFrameworkMessage HTTP/1.1 User-Agent: libprocess/scheduler(1)@10.0.1.7:53523 ... represents a message with the name 'mesos.internal.RegisterFrameworkMessage' destined for the actor 'master' coming from the actor 'scheduler(1)' at 10.0.1.7:53523. If the 'master' actor were to send a message back it would look something like this: POST /scheduler(1)/mesos.internal.FrameworkRegisteredMessage HTTP/1.1 User-Agent: libprocess/master@10.0.1.7:5050 So, one-way message passing via HTTP POST. The message data is captured as the body of the HTTP request (which can be specified using _either_ Content-Length or a Transfer-Encoding, and as Nikita points out we use chunked transfer encoding internally). The data is arbitrary and the actor ultimately decides how it wants to parse it. In Mesos, 99% of our messages use serialized protobufs, but we also send a few messages with just arbitrary data. All this really means is that knowing the actor and message name is not enough, you also need to know what the body type is supposed to be for that message. In the future we'll probably enable messages with either JSON or serialized protobuf[1] ... for now, just serialized protobuf. Okay, so where does this break down when trying to do this language-natively? I've had some of this in the works and this conversation has motivated me to publish some reviews addressing the issues: (1) We'll need to return a response if one plans to use a native HTTP library since it'll expect request/response. https://reviews.apache.org/r/20276 introduces responding with a '202 Accepted' for these messages (from the HTTP specification, a '202 Accepted': The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.). (2) Most HTTP libraries will set their 'User-Agent' themselves, so https://reviews.apache.org/r/20277 introduces a 'libprocess-from' header that works similar to User-Agent. There is still some cleanup I'd love to do around stringification of PIDs (the underlying type Mesos uses for remote actors, inspired by Erlang). Until then, the 'libprocess-from' string is unfortunately esoteric (see the test). The combination of these two patches should make sending and receiving messages straightforward. However, we still plan to expose the low-level Event and Call protobuf messages and that will be the preferred approach for building a native-language library. Follow along at https://issues.apache.org/jira/browse/MESOS-1127 for more details. (To be clear, you'd still be able to implement native-language libraries with the patches above but we'll be deprecating the protobufs you'd be using in favor of Event and Call protobufs instead. If you're eager to get that
Re: Mesos Wire Protocol Documentation
And I've started a skeleton implementation of the Mesos framework API at https://github.com/wickman/pesos While I vendored the translated protobuf, it would be great to tackle https://issues.apache.org/jira/browse/MESOS-857 at some point. On Sun, Apr 27, 2014 at 12:50 PM, Brian Wickman wick...@gmail.com wrote: I've implemented a pure python version of the basic Process/ProtobufProcess mechanics and wire protocol. I haven't gone so far as Kevin and tried to register a framework or maybe talk to a replicated log, but it shouldn't be much more work: https://github.com/wickman/compactor Zero documentation but you can read the tests for a general idea of what's going on. cheers, brian On Sun, Apr 13, 2014 at 10:28 PM, Kevin Sweeney kevin.t.swee...@gmail.com wrote: I've got the start of a JVM verson on github - it can currently register a framework and parse the response. Client-side I still need to figure out how to properly configure Keep-Alive. Servlet can dispatch messages to message handlers in a type-safe way and returns a 202 for messages it's going to handle. Code's a mess currently. https://github.com/kevints/mesos-framework-api On Fri, Apr 11, 2014 at 7:12 PM, Benjamin Hindman b...@eecs.berkeley.edu wrote: First, my apologies for getting to this party so late. It's great to see people interested in helping create native-language Mesos libraries. Vladimir: my presentation was definitely referring to the the low-level protocol between master, framework (scheduler), slave, and executors. I'll do my best here to clarify how the current protocol works and what we need to do to get it to the point where we can write native-language libraries. (Eventually it would be great to move some of this into documentation as necessary.) As Nikita pointed out, the protocol is currently HTTP-like. As my presentation describes, think actors and one-way message passing when considering how the protocol works. To send a message an actor POSTs an HTTP request where the actor that is supposed to receive the message is the first component of the request path and the name of the message is the remaining part of the path. To distinguish one of these messages from a normal HTTP request we look to see if the 'User-Agent' is 'libprocess/...'. For example: POST /master/mesos.internal.RegisterFrameworkMessage HTTP/1.1 User-Agent: libprocess/scheduler(1)@10.0.1.7:53523 ... represents a message with the name 'mesos.internal.RegisterFrameworkMessage' destined for the actor 'master' coming from the actor 'scheduler(1)' at 10.0.1.7:53523. If the 'master' actor were to send a message back it would look something like this: POST /scheduler(1)/mesos.internal.FrameworkRegisteredMessage HTTP/1.1 User-Agent: libprocess/master@10.0.1.7:5050 So, one-way message passing via HTTP POST. The message data is captured as the body of the HTTP request (which can be specified using _either_ Content-Length or a Transfer-Encoding, and as Nikita points out we use chunked transfer encoding internally). The data is arbitrary and the actor ultimately decides how it wants to parse it. In Mesos, 99% of our messages use serialized protobufs, but we also send a few messages with just arbitrary data. All this really means is that knowing the actor and message name is not enough, you also need to know what the body type is supposed to be for that message. In the future we'll probably enable messages with either JSON or serialized protobuf[1] ... for now, just serialized protobuf. Okay, so where does this break down when trying to do this language-natively? I've had some of this in the works and this conversation has motivated me to publish some reviews addressing the issues: (1) We'll need to return a response if one plans to use a native HTTP library since it'll expect request/response. https://reviews.apache.org/r/20276 introduces responding with a '202 Accepted' for these messages (from the HTTP specification, a '202 Accepted': The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.). (2) Most HTTP libraries will set their 'User-Agent' themselves, so https://reviews.apache.org/r/20277 introduces a 'libprocess-from' header that works similar to User-Agent. There is still some cleanup I'd love to do around stringification of PIDs (the underlying type Mesos uses for remote actors, inspired by Erlang). Until then, the 'libprocess-from' string is unfortunately esoteric (see the test). The combination of these two patches should make sending and receiving messages straightforward. However, we still plan to expose the low-level Event and Call protobuf messages and that will be the preferred approach for building a native
Re: Mesos Wire Protocol Documentation
To my knowledge there is no Zookeeper Group implementation for Go yet. There do however appear to be Go zookeeper bindingshttps://github.com/samuel/go-zookeeper. From my day of spelunking through code, it seems that probably the most challenging thing will be getting a ZookeeperMasterDetector implementation working correctly to enable Scheduler high availability mode for these other languages. Java/Python/C++ have the benefit of preexisting Group implementations (javahttps://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/Group.java , pythonhttps://github.com/twitter/commons/blob/master/src/python/twitter/common/zookeeper/group/kazoo_group.py , c++https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob;f=src/zookeeper/group.cpp;h=70972980dd25618c76eb36e0a31d060c096e36dc;hb=HEAD) which are the foundation for ZK master detection. An intermediate option could be to pull the ZookeeperMasterDetectorProcess out into a standalone binary and have the native implementation talk to it as a sort of proxy to figure out who the leading master is. But that's not nearly as satisfying as a pure language implementation. ~brian On Sun, Apr 27, 2014 at 3:22 PM, Vladimir Vivien vladimir.viv...@gmail.comwrote: It must be the right time for this. I started the same effort for Go. I am following Kevin's example and Ben's well-documented email (in this thread). Very early but should have some proof-of-concept working soon. https://github.com/vladimirvivien/ionos On Sun, Apr 27, 2014 at 6:07 PM, Tom Arnfeld t...@duedil.com wrote: (Sorry to jump into this thread) I have to say, this is very exciting! It only became apparent to me how painful it is having to compile the mesos egg into a Python framework the past week while writing one, especially when running the framework on a different architecture to the one running the executor! Looking forward to being able to use this. :-) It’d be awesome if the API was near identical, so one could switch between `mesos` and `pesos` easily… Tom. On 27 Apr 2014, at 22:42, Brian Wickman wick...@gmail.com wrote: And I've started a skeleton implementation of the Mesos framework API at https://github.com/wickman/pesos While I vendored the translated protobuf, it would be great to tackle https://issues.apache.org/jira/browse/MESOS-857 at some point. On Sun, Apr 27, 2014 at 12:50 PM, Brian Wickman wick...@gmail.com wrote: I've implemented a pure python version of the basic Process/ProtobufProcess mechanics and wire protocol. I haven't gone so far as Kevin and tried to register a framework or maybe talk to a replicated log, but it shouldn't be much more work: https://github.com/wickman/compactor Zero documentation but you can read the tests for a general idea of what's going on. cheers, brian On Sun, Apr 13, 2014 at 10:28 PM, Kevin Sweeney kevin.t.swee...@gmail.com wrote: I've got the start of a JVM verson on github - it can currently register a framework and parse the response. Client-side I still need to figure out how to properly configure Keep-Alive. Servlet can dispatch messages to message handlers in a type-safe way and returns a 202 for messages it's going to handle. Code's a mess currently. https://github.com/kevints/mesos-framework-api On Fri, Apr 11, 2014 at 7:12 PM, Benjamin Hindman b...@eecs.berkeley.edu wrote: First, my apologies for getting to this party so late. It's great to see people interested in helping create native-language Mesos libraries. Vladimir: my presentation was definitely referring to the the low-level protocol between master, framework (scheduler), slave, and executors. I'll do my best here to clarify how the current protocol works and what we need to do to get it to the point where we can write native-language libraries. (Eventually it would be great to move some of this into documentation as necessary.) As Nikita pointed out, the protocol is currently HTTP-like. As my presentation describes, think actors and one-way message passing when considering how the protocol works. To send a message an actor POSTs an HTTP request where the actor that is supposed to receive the message is the first component of the request path and the name of the message is the remaining part of the path. To distinguish one of these messages from a normal HTTP request we look to see if the 'User-Agent' is 'libprocess/...'. For example: POST /master/mesos.internal.RegisterFrameworkMessage HTTP/1.1 User-Agent: libprocess/scheduler(1)@10.0.1.7:53523 ... represents a message with the name 'mesos.internal.RegisterFrameworkMessage' destined for the actor 'master' coming from the actor 'scheduler(1)' at 10.0.1.7:53523. If the 'master' actor were to send a message back it would look something like this: POST /scheduler(1)/mesos.internal.FrameworkRegisteredMessage HTTP/1.1
[jira] [Reopened] (MESOS-692) reservations are reported incorrectly in master's state.json
[
https://issues.apache.org/jira/browse/MESOS-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
brian wickman reopened MESOS-692:
-
I don't see the fix in this CL. The problem is that roles are not reflected in
the json-ification of the Resource, which is what I was looking for.
The slave has a repeated Resource field possibly with multiple Resource roles.
Afaict this change just shows the sum each of them? In practice it's important
to know the allocation of resources to each role individually.
reservations are reported incorrectly in master's state.json
Key: MESOS-692
URL: https://issues.apache.org/jira/browse/MESOS-692
Project: Mesos
Issue Type: Bug
Components: master
Reporter: brian wickman
Assignee: Dominic Hamon
Labels: newbie
Fix For: 0.19.0
When you dump state.json from the master, it lists out a single resources
dict, e.g.
{noformat}
state['slaves'][0]
{..., u'registered_time': 1378851991.87182, u'reregistered_time':
1378917907.3190701, u'id': u'201309042122-193162-5050-55755-120',
u'resources': {u'mem': 21913, u'disk': 40, u'cpus': 14, u'ports':
u'[31000-32000]'}}
{noformat}
Looking at the code, it looks like last value wins:
{noformat}
// Returns a JSON object modeled on a Resources.
JSON::Object model(const Resources resources)
{
JSON::Object object;
foreach (const Resource resource, resources) {
switch (resource.type()) {
case Value::SCALAR:
object.values[resource.name()] = resource.scalar().value();
break;
case Value::RANGES:
object.values[resource.name()] = stringify(resource.ranges());
break;
case Value::SET:
object.values[resource.name()] = stringify(resource.set());
break;
default:
LOG(FATAL) Unexpected Value type: resource.type();
break;
}
}
return object;
}
{noformat}
So for example if you had role * with 15 cores and role hdfs with 1 cores,
the resource dict might just report 1 core. Instead it should probably
aggregate resources by role, and have resources = {'*': {ram, cpu, disk},
'hdfs': {ram, cpu, disk}} etc.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (MESOS-857) split mesos and mesos_pb2 distributions
brian wickman created MESOS-857: --- Summary: split mesos and mesos_pb2 distributions Key: MESOS-857 URL: https://issues.apache.org/jira/browse/MESOS-857 Project: Mesos Issue Type: Task Components: python api Reporter: brian wickman Priority: Minor Right now the mesos_pb2 and mesos dependencies are bundled together into the mesos egg. We have some tooling that uses just the compiled protobufs, but because they're lumped together with the mesos egg, we get all the dependency/platform nightmare that comes along with it, not to mention the bloat of including 20MB of .so files. This proposes splitting the mesos protobufs into a separate mesos_pb distribution that the mesos distribution should depend upon via install_requires (e.g. mesos_pb==0.15.0-rc4) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MESOS-687) slave should export perf hardware counters when available
brian wickman created MESOS-687: --- Summary: slave should export perf hardware counters when available Key: MESOS-687 URL: https://issues.apache.org/jira/browse/MESOS-687 Project: Mesos Issue Type: Story Components: slave Reporter: brian wickman Priority: Minor It would be very valuable for the slave resource monitor to export hardware counters. There are a few ways this could be done -- either by invoking perf directly or by accessing the counters via the syscall/read interface described by http://lxr.free-electrons.com/source/tools/perf/design.txt I'd imagine this would just get plumbed through Isolator::usage in the end and get exported via the resource monitor endpoint. Correct me if I'm wrong but there's currently no information exposed (programmatically) at the granularity level of executor, so I don't really know how this information could ever make its way into the hands of a scheduler for the purpose of making better scheduling decisions vis-a-vis things like task-level interference. Given Google's CPI paper (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/40737.pdf) it sounds like something worth investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
