GitHub user skunert opened a pull request:
https://github.com/apache/incubator-flink/pull/251
Mesos integration of Apache Flink
Hi,
this PR is a mesos client for Flink. It still needs to be tested which i
will do as soon as I have access to a cluster. On a single machine with mesos
running it works fine. Even though it is not ready to merge, I want to gather
feedback and suggestions on it.
I tried to make everything as similar to the yarn client as possible. For
an idea on how this works there is a mesos_setup.md.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/skunert/incubator-flink mesos2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #251
----
commit bf6bedc0c831175aca1bfb6710ab9af1dec18cdb
Author: sebastian kunert <[email protected]>
Date: 2014-10-07T08:33:28Z
initial mesos playaround commit
commit 6f40326d2408f2928d210c6c0a434068ecb63d8b
Author: sebastian kunert <[email protected]>
Date: 2014-10-14T14:26:39Z
initial mesos playaround commit
commit ccd9938a68d74299eb56d148f706d20dc840b873
Author: sebastian kunert <[email protected]>
Date: 2014-10-26T16:19:40Z
trying to start jobmanager
commit abd6decd70b836ac1b66762147031ef657a8c44c
Author: Sebastian Kunert <[email protected]>
Date: 2014-10-29T14:51:26Z
taskmanager and jobmanager starting
commit 2be207f0d080ebaefd9a6b3846682f694f1d6563
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-03T13:42:45Z
moved mesos to a flink-addons module and created own profile
commit 091f21c45435448661f52318cc72defba48f4353
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-03T14:04:36Z
fixed wrong string path
commit 03ff92609041b07d8f180d48867035ab10b9d666
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-05T16:25:51Z
created mesos module and start scripts
commit e0a733382131094369924195e656a9e082e72bb2
Author: sebastian kunert <[email protected]>
Date: 2014-11-04T07:50:43Z
added dynamic flink conf dir
commit 781468d3f4205d68141d06d377773e643150158e
Author: sebastian kunert <[email protected]>
Date: 2014-11-06T00:00:33Z
exported flink into its own executor
commit 6c9425fcc835fe1e9a0e028f0ece988ed8708236
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-06T18:21:17Z
introduced sepparate executors for the task and jobmanager, some code
cleanup. Further cleanup and proper management of the managers required.
commit 9804cd4ae10aa16fd40cfdd952a630e66c662249
Author: sebastian kunert <[email protected]>
Date: 2014-11-07T17:07:54Z
Added logging, advanced MesosUtils, code cleanup.
commit bfe9491b138ff4834431917fe264e05896febc21
Author: sebastian kunert <[email protected]>
Date: 2014-11-08T16:59:33Z
added command line interface
commit 8b87f1c9876cbc56d5fc6c561b931eaa59b59178
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-12T16:46:31Z
added more command line option. It is now possible to limit memory and cpu
usage. Also comments have been added to the code.
commit 626ab44c4408d69b9bcf2f7b3a1e1f5e46972094
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-13T16:58:38Z
added documentation page (currently only clone of yarn documentation page)
and adjusted the slots of taskmanagers according to the available CPUs
commit aeac39482c20fc6cce6c1f83bf28b0b1fc30e46e
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-14T11:03:01Z
added better startup script
commit b1e508df2aead4227b06865dd82daa25e6ceeaa7
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-19T15:00:27Z
implemented handling of task statuses, introduced protobuf serialization of
config parameters, added help option, code cleanup, comments
commit 7b03bb55795587b2691b04944b065dfa393f8222
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-21T16:30:24Z
implemented serialization of configuration with protobuf
commit 0dbd40f8e64979f3156fe3795de17fa8b629fea9
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-26T16:16:57Z
better package structure, web frontend integration, dynamic property
handling, JobManager Port offset, code cleanup
commit c82a5b5872a62882660090237c26fce329a029c2
Author: Sebastian Kunert <[email protected]>
Date: 2014-11-27T14:29:51Z
unified jobmanager executor and web frontend. better cpu handling
commit 20d4939c45970909bd87d83bf11fb4a72aecfe1f
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-03T12:35:11Z
fixed some resource specifications in the scheduler
commit 139fd19ae38f95d7f417b8f912b555a610b374ba
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-03T12:47:11Z
added better comments
commit 068b5809b4717f744bcc8a831f3f0e60c559ef85
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-03T13:12:25Z
better documentation
commit 154910a2e0c12e5f0d73e512b0de3fd267191810
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-03T14:41:34Z
Merge branch 'master' of github.com:apache/incubator-flink into mesos_module
commit 21776c5d5c53543fea0fa8a215ff5b46b5961874
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-03T14:46:26Z
removed unchanged files
commit 6b595b2aae8704077f42052539ed20eb8de261e1
Author: Sebastian Kunert <[email protected]>
Date: 2014-12-04T12:26:40Z
improved webinterface output and logging parameters in the start-mesos.sh
script
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
