Repository: flink Updated Branches: refs/heads/master 9eef3c86c -> 6b9cee33c
[Flink-contrib] Flink dev cluster on Docker with Docker Compose deployment This closes #533 Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/6b9cee33 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/6b9cee33 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/6b9cee33 Branch: refs/heads/master Commit: 6b9cee33c8b046938489f2a635e91fe17a44ec68 Parents: 9eef3c8 Author: Emmanuel <emman...@mediumone.com> Authored: Thu Mar 26 21:32:31 2015 +0100 Committer: Robert Metzger <rmetz...@apache.org> Committed: Fri Mar 27 08:47:40 2015 +0100 ---------------------------------------------------------------------- flink-contrib/docker-flink/LICENSE | 202 +++++++++++++++++++ flink-contrib/docker-flink/README.md | 80 ++++++++ flink-contrib/docker-flink/base/Dockerfile | 49 +++++ flink-contrib/docker-flink/base/supervisor.conf | 21 ++ flink-contrib/docker-flink/build.sh | 31 +++ flink-contrib/docker-flink/docker-compose.yml | 37 ++++ flink-contrib/docker-flink/flink/Dockerfile | 43 ++++ .../docker-flink/flink/conf/flink-conf.yaml | 37 ++++ .../docker-flink/flink/conf/log4j.properties | 26 +++ .../docker-flink/flink/conf/logback-yarn.xml | 39 ++++ .../docker-flink/flink/conf/logback.xml | 31 +++ flink-contrib/docker-flink/flink/conf/slaves | 0 .../docker-flink/flink/config-flink.sh | 55 +++++ pom.xml | 1 + 14 files changed, 652 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/LICENSE ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/LICENSE b/flink-contrib/docker-flink/LICENSE new file mode 100644 index 0000000..e06d208 --- /dev/null +++ b/flink-contrib/docker-flink/LICENSE @@ -0,0 +1,202 @@ +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/README.md ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/README.md b/flink-contrib/docker-flink/README.md new file mode 100644 index 0000000..9dff930 --- /dev/null +++ b/flink-contrib/docker-flink/README.md @@ -0,0 +1,80 @@ +#Apache Flink cluster deployment on Docker using Docker-Compose + +##Installation +###Install Docker + +https://docs.docker.com/installation/ + +if you have issues with Docker-Compose versions incompatible with your version of Docker try + +`curl -sSL https://get.docker.com/ubuntu/ | sudo sh` + +###Install Docker-Compose + +``` +curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose + +chmod +x /usr/local/bin/docker-compose +``` + +###Get the repo + +###Build the images + +Images are based on Ubuntu Trusty 14.04 and run Supervor to stay alive when running containers. + +The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config. + +- Run `./build.sh` + +###Deploy + +- Deploy cluster and see config/setup log output (best run in a screen session) + +`docker-compose up` + +- Deploy as a daemon (and return) + +`docker-compose up -d` + +- Scale the cluster up or down to *N* TaskManagers + +`docker-compose scale taskmanager=<N>` + +- Access the JobManager node with SSH (exposed on Port 220) + +`ssh root@localhost -p 220` + +or on Mac OS X with boot2docker + +`ssh root@$(boot2docker ip) -p 220` + +The password is 'secret' + +- Kill the cluster + +`docker-compose kill` + +- Upload a jar to the cluster + +`scp -P 220 <your_jar> root@localhost:/<your_path>` + +- Run a topology + +`ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>` + +or + +ssh to the job manager and run the topology from there. + +###Ports + +- The Web Dashboard is on port `48080` +- The Web Client is on port `48081` +- JobManager RPC port `6123` (default, not exposed to host) +- TaskManagers RPC port `6121` (default, not exposed to host) +- TaskManagers Data port `6122` (default, not exposed to host) +- JobManager SSH `220` +- TaskManagers SSH: randomly assigned port, check wih `docker ps` + +Edit the `docker-compose.yml` file to edit port settings. http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/base/Dockerfile ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/base/Dockerfile b/flink-contrib/docker-flink/base/Dockerfile new file mode 100644 index 0000000..67a2953 --- /dev/null +++ b/flink-contrib/docker-flink/base/Dockerfile @@ -0,0 +1,49 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM ubuntu:trusty + +#requirements +RUN apt-get update; apt-get install -y curl wget supervisor openssh-server openssh-client nano + +#priviledge separation directory +RUN mkdir /var/run/sshd + +#install Java 7 Oracle JDK +RUN mkdir -p /usr/java/default && \ + curl -Ls 'http://download.oracle.com/otn-pub/java/jdk/7u51-b13/jdk-7u51-linux-x64.tar.gz' -H 'Cookie: oraclelicense=accept-securebackup-cookie' | \ + tar --strip-components=1 -xz -C /usr/java/default/ +ENV JAVA_HOME /usr/java/default/ + +#Install Java Open JDK +#RUN apt-get install -y unzip openjdk-7-jre-headless +#ENV JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/ + + +RUN echo 'root:secret' | chpasswd + +#SSH as root... probably needs to be revised for security! +RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config + +EXPOSE 22 + +# supervisor base configuration +ADD supervisor.conf /etc/supervisor/ + +# default command / not overridable CMD needed for supervisord +#CMD ["supervisord", "-c", "/etc/supervisor.conf"] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/base/supervisor.conf ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/base/supervisor.conf b/flink-contrib/docker-flink/base/supervisor.conf new file mode 100644 index 0000000..ab15222 --- /dev/null +++ b/flink-contrib/docker-flink/base/supervisor.conf @@ -0,0 +1,21 @@ +; Licensed to the Apache Software Foundation (ASF) under one +; or more contributor license agreements. See the NOTICE file +; distributed with this work for additional information +; regarding copyright ownership. The ASF licenses this file +; to you under the Apache License, Version 2.0 (the +; "License"); you may not use this file except in compliance +; with the License. You may obtain a copy of the License at +; +; http://www.apache.org/licenses/LICENSE-2.0 +; +; Unless required by applicable law or agreed to in writing, software +; distributed under the License is distributed on an "AS IS" BASIS, +; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +; See the License for the specific language governing permissions and +; limitations under the License. + +[supervisord] +nodaemon=true + +[include] +files = /etc/supervisor/conf.d/*.conf http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/build.sh ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/build.sh b/flink-contrib/docker-flink/build.sh new file mode 100755 index 0000000..d732a93 --- /dev/null +++ b/flink-contrib/docker-flink/build.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +#Kill cluster and remove all containers +docker-compose kill +#docker rm $(docker ps -a -q) + +#make sure the config file script is executable +chmod +x flink/config-flink.sh + +#rebuild images +docker build -t="base" base +docker build -t="flink" flink http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/docker-compose.yml ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/docker-compose.yml b/flink-contrib/docker-flink/docker-compose.yml new file mode 100644 index 0000000..8fee23b --- /dev/null +++ b/flink-contrib/docker-flink/docker-compose.yml @@ -0,0 +1,37 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +jobmanager: + image: flink + ports: + - "48080:8080" + - "48081:8081" + - "220:22" + expose: + - "6123" + - "22" + command: /usr/local/flink/bin/config-flink.sh jobmanager +taskmanager: + image: flink + ports: + - "22" + expose: + - "6121" + - "6122" + links: + - jobmanager:jobmanager \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/Dockerfile ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/Dockerfile b/flink-contrib/docker-flink/flink/Dockerfile new file mode 100644 index 0000000..8ed7a10 --- /dev/null +++ b/flink-contrib/docker-flink/flink/Dockerfile @@ -0,0 +1,43 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +FROM base + +#add passless key to ssh +RUN ssh-keygen -f ~/.ssh/id_rsa -t rsa -N '' +RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/* + +##Flink 0.8.1 Installation +###Download: +RUN mkdir ~/downloads && cd ~/downloads && \ + wget -q -O - http://mirrors.advancedhosters.com/apache/flink/flink-0.8.1/flink-0.8.1-bin-hadoop2.tgz | tar -zxvf - -C /usr/local/ +RUN cd /usr/local && ln -s ./flink-0.8.1 flink + +ENV FLINK_HOME /usr/local/flink +ENV PATH $PATH:$FLINK_HOME/bin + +#config files (template) +ADD conf/flink-conf.yaml /usr/local/flink/conf/ + +ADD config-flink.sh /usr/local/flink/bin/ +RUN chmod +x /usr/local/flink/bin/config-flink.sh + +EXPOSE 6123 +EXPOSE 22 + +CMD ["/usr/local/flink/bin/config-flink.sh", "taskmanager"] http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/conf/flink-conf.yaml ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/conf/flink-conf.yaml b/flink-contrib/docker-flink/flink/conf/flink-conf.yaml new file mode 100644 index 0000000..e7136e5 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/flink-conf.yaml @@ -0,0 +1,37 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +jobmanager.rpc.address: %jobmanager% +jobmanager.rpc.port: 6123 +jobmanager.heap.mb: 128 + +taskmanager.rpc.port: 6121 +taskmanager.data.port: 6122 +taskmanager.heap.mb: 256 +taskmanager.numberOfTaskSlots: %nb_slots% + +parallelization.degree.default: %parallelism% + +env.java.home: /usr/java/default + +#============================================================================== +# Web Frontend +#============================================================================== + +jobmanager.web.port: 8080 +webclient.port: 8081 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/conf/log4j.properties ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/conf/log4j.properties b/flink-contrib/docker-flink/flink/conf/log4j.properties new file mode 100644 index 0000000..172f566 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/log4j.properties @@ -0,0 +1,26 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +log4j.rootLogger=INFO, file + +# Log all infos in the given file +log4j.appender.file=org.apache.log4j.FileAppender +log4j.appender.file.file=${log.file} +log4j.appender.file.append=false +log4j.appender.file.layout=org.apache.log4j.PatternLayout +log4j.appender.file.layout.ConversionPattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/conf/logback-yarn.xml ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/conf/logback-yarn.xml b/flink-contrib/docker-flink/flink/conf/logback-yarn.xml new file mode 100644 index 0000000..c5511b3 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/logback-yarn.xml @@ -0,0 +1,39 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<configuration> + <appender name="file" class="ch.qos.logback.core.FileAppender"> + <file>${log.file}</file> + <append>false</append> + <encoder> + <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern> + </encoder> + </appender> + + <appender name="console" class="ch.qos.logback.core.ConsoleAppender"> + <encoder> + <pattern>%d{HH:mm:ss} %-5level %logger{60} %X{sourceThread} - %msg%n</pattern> + </encoder> + </appender> + + <logger name="ch.qos.logback" level="WARN" /> + <root level="INFO"> + <appender-ref ref="file"/> + <appender-ref ref="console"/> + </root> +</configuration> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/conf/logback.xml ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/conf/logback.xml b/flink-contrib/docker-flink/flink/conf/logback.xml new file mode 100644 index 0000000..9305b11 --- /dev/null +++ b/flink-contrib/docker-flink/flink/conf/logback.xml @@ -0,0 +1,31 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<configuration> + <appender name="file" class="ch.qos.logback.core.FileAppender"> + <file>${log.file}</file> + <append>false</append> + <encoder> + <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{60} %X{sourceThread} - %msg%n</pattern> + </encoder> + </appender> + + <root level="INFO"> + <appender-ref ref="file"/> + </root> +</configuration> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/conf/slaves ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/conf/slaves b/flink-contrib/docker-flink/flink/conf/slaves new file mode 100644 index 0000000..e69de29 http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/flink-contrib/docker-flink/flink/config-flink.sh ---------------------------------------------------------------------- diff --git a/flink-contrib/docker-flink/flink/config-flink.sh b/flink-contrib/docker-flink/flink/config-flink.sh new file mode 100755 index 0000000..72f2828 --- /dev/null +++ b/flink-contrib/docker-flink/flink/config-flink.sh @@ -0,0 +1,55 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + + +#!/bin/bash + +CONF=/usr/local/flink/conf +EXEC=/usr/local/flink/bin + +#set nb_slots = nb CPUs +#let "nbslots=$2 * `nproc`" +sed -i -e "s/%nb_slots%/`nproc`/g" $CONF/flink-conf.yaml + +#set parallelism +sed -i -e "s/%parallelism%/1/g" $CONF/flink-conf.yaml + +if [ "$1" = "jobmanager" ]; then + echo "Configuring Job Manager on this node" + sed -i -e "s/%jobmanager%/`hostname -i`/g" $CONF/flink-conf.yaml + $EXEC/jobmanager.sh start cluster + $EXEC/start-webclient.sh + +elif [ "$1" = "taskmanager" ]; then + echo "Configuring Task Manager on this node" + sed -i -e "s/%jobmanager%/$JOBMANAGER_PORT_6123_TCP_ADDR/g" $CONF/flink-conf.yaml + $EXEC/taskmanager.sh start +fi + +#print out config - debug +echo "config file: " && cat $CONF/flink-conf.yaml + +#add ENV variable to shell for ssh login +echo "export JAVA_HOME=/usr/java/default;" >> ~/.profile +echo "export PATH=$PATH:$JAVA_HOME/bin;" >> ~/.profile +echo "export F=/usr/local/flink/;" >> ~/.profile +#Uncomment for SSH connection between nodes without prompts +#echo 'export FLINK_SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"' >> ~/.profile + +#run ssh server and supervisor to keep container running. +/usr/sbin/sshd && supervisord -c /etc/supervisor/supervisor.conf \ No newline at end of file http://git-wip-us.apache.org/repos/asf/flink/blob/6b9cee33/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index d534eee..ec063cf 100644 --- a/pom.xml +++ b/pom.xml @@ -710,6 +710,7 @@ under the License. <exclude>out/test/flink-avro/avro/user.avsc</exclude> <!-- Configuration Files. --> <exclude>**/flink-bin/conf/slaves</exclude> + <exclude>flink-contrib/docker-flink/flink/conf/slaves</exclude> <!-- Administrative files in the main trunk. --> <exclude>**/README.md</exclude> <exclude>CHANGELOG</exclude>
