Repository: incubator-reef Updated Branches: refs/heads/master c1ac79639 -> b2e85d12a
[REEF-94]: Replace the guides on the website with links to cwiki This removes living documents on our [website](http://reef.incubator.apache.org/) and instead provides links to cwiki. JIRA: [REEF-94](https://issues.apache.org/jira/browse/REEF-94) Pull Request: Closes #162 Author: Jason (Joo Seong) Jeong <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-reef/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-reef/commit/b2e85d12 Tree: http://git-wip-us.apache.org/repos/asf/incubator-reef/tree/b2e85d12 Diff: http://git-wip-us.apache.org/repos/asf/incubator-reef/diff/b2e85d12 Branch: refs/heads/master Commit: b2e85d12a9caa50d6e1987815e7d21107d56e25e Parents: c1ac796 Author: Jason (Joo Seong) Jeong <[email protected]> Authored: Fri Apr 24 16:06:28 2015 +0900 Committer: Yunseong Lee <[email protected]> Committed: Sun Apr 26 16:59:42 2015 +0900 ---------------------------------------------------------------------- website/src/site/markdown/coding-guideline.md | 51 ------- website/src/site/markdown/committer-guide.md | 159 --------------------- website/src/site/markdown/contributing.md | 159 --------------------- website/src/site/markdown/issue-tracker.md | 21 --- website/src/site/markdown/powered-by.md | 21 --- website/src/site/markdown/reef-examples.md | 135 ----------------- website/src/site/markdown/tutorial.md | 137 ------------------ website/src/site/site.xml | 13 +- 8 files changed, 6 insertions(+), 690 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/coding-guideline.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/coding-guideline.md b/website/src/site/markdown/coding-guideline.md deleted file mode 100644 index 55a6d02..0000000 --- a/website/src/site/markdown/coding-guideline.md +++ /dev/null @@ -1,51 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Coding Guidelines - -###The Basics - -We largely follow [Hadoop coding guidelines](http://wiki.apache.org/hadoop/CodeReviewChecklist): - -- Use 2 (TWO) spaces to indent code. -- Use LF (Unix style) line endings. -- Do **not** use the `@author` Javadoc tag. (Be modest ! :-)) -- For the rest, follow the original [Sun Java coding conventions](http://www.oracle.com/technetwork/java/codeconvtoc-136057.html). -- If some file has different formatting, try to match the existing style. -- Use [CheckStyle](http://checkstyle.sourceforge.net/) to verify your coding style. -- Avoid mixing both style and functionality changes in one commit. -- Make sure all files have the needed [license header](http://www.apache.org/legal/src-headers.html). - -###Comments - -We require committers to add comments to generate proper javadoc pages. In addition to this requirement, we encourage committers to add comments (if needed) to make code easier to understand. - -- Class -- Interface -- Public constructor -- Public method -- Important fields -- Code part that captures complex logic - -###On immutability of why you should make everything final - -REEF favors immutable objects over mutable ones. When you browse the code base, you see that there is an enormous number of final and readonly modifiers used. The reason for this stems from the distributed and parallel nature of REEF: What cannot be changed doesn't have to be guarded with locks. Hence, immutability is an excellent tool to achieve both thread safety and performance. Following this line of thought, we arrive at the following guidelines: - -- Make all instance variables possible `final`. -- Use the [Java Concurrency in Practice annotations](http://jcip.net.s3-website-us-east-1.amazonaws.com/) for the instance variables that aren't `final`. -- When a class has more than 3 non-`final` instance variables, consider breaking the class in two to limit the lock complexity. http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/committer-guide.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/committer-guide.md b/website/src/site/markdown/committer-guide.md deleted file mode 100644 index ce094d9..0000000 --- a/website/src/site/markdown/committer-guide.md +++ /dev/null @@ -1,159 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Committer Guide - -This captures committer-specific information beyond what's covered in the [contributions guide](contributing.html). - -###Accepting Pull Requests on GitHub - -The ASF INFRA team maintains [a mirror of our git repository over on GitHub](https://github.com/apache/incubator-reef).That mirror is strictly one-way: changes from the Apache git get copied over, but not the other way around. Further, the mirror on GitHub is read-only for everyone. Which means we can neither accept nor close pull requests filled there via the GitHub web UI. However, we want to allow for contributions via GitHub Pull Requests. Here's how: - -####Add the ASF git repository - -If you have not done so already, add the ASF git repository. For example, to add as the "apache" remote repository: - - $ git remote add apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git - -As a result, you can refer to the ASF git as "apache". An example setup, with a the GitHub mirror at upstream and a forked GitHub origin at `{username}`: - - $ git remote -v - apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git (fetch) - apache https://git-wip-us.apache.org/repos/asf/incubator-reef.git (push) - origin https://github.com/{username}/incubator-reef.git (fetch) - origin https://github.com/{username}/incubator-reef.git (push) - upstream https://github.com/apache/incubator-reef.git (fetch) - upstream https://github.com/apache/incubator-reef.git (push) - -####Merge changes - -The next step is to merge all changes on the Pull Request as a single commit. There are two methods here: (A) pull the branch from the GitHub Pull Request and squash commits, or (B) get the `.diff` from GitHub and manually create a commit from this information. Each option has its pros and cons. With method A, git does some of the tedious work of preserving commit messages; but in some cases the squash may not apply cleanly, and the merger will have to carefully resolve conflicts. With method B, the `.diff` will apply cleanly (given that the branch is up-to-date, i.e. the GitHub GUI states that the "This pull request can be automatically merged by project collaborators"); but the merger will have to carefully copy over commit messages and edit author information. - -####Commit Message - -Whichever method you choose, the following should be included in the final commit message: - -- Pull request description and commit comments -- A link to the JIRA this is addressing. -- The text "closes #PRNUMBER", where PRNUMBER is the number of the pull request, e.g. "10" - -Following the last statement will close the GitHub pull request. It is important to close via the commit message, because we cannot close pull requests via the GitHub Web UI. - -#####Example Commit message (80 columns) - - [REEF-33] Allow Tasks to initiate an Evaluator Heartbeat - This adds the class HeartBeatTriggerManager which can be used by a Task to - initiate an Evaluator Heartbeat. It is used by injecting it into the Task. - - JIRA: - [REEF-33] https://issues.apache.org/jira/browse/REEF-33 - - Pull Request: - Closes #24 - - Author: - AuthorName AuthorEmail - -####Method A - -#####A-1. Create a branch on your local git repository - -You want to make sure that that branch is current with the the master branch on Apache git: - - $ git checkout -b BRANCH_NAME - $ git pull apache master - -This assumes that you called the remote git repository at the ASF "apache". You can name the branch however you want, but it is customary to name them either after the pull request number or the JIRA id, e.g. REEF-24. - -#####A-2. Pull the contribution into that branch - -This is as simple as - - $ git pull GIT_REPOSITORY_OF_THE_CONTRIBUTOR BRANCH_NAME_OF_THE_CONTRIBUTION - -However, the GitHub web ui makes it somewhat hard to get these two strings. The email from GitHub that informs us of Pull Requests makes this really obvious, though. Consider this quote from pull [request #10](https://github.com/apache/incubator-reef/pull/10): - ->**You can merge this Pull Request by running** ->>`git pull https://github.com/jwang98052/incubator-reef juwang-logfactory` - -This copies the changes from the given remote branch into the one we just created locally. - -#####A-3. Check the pull request - -1. Make sure the code compiles and all tests pass. -2. If the code touches code that you suspect might break on YARN or Mesos, please test on those environments. If you don't have access to a test environment, ask on the mailing list for help. -3. Make sure the code adheres to our [coding guidelines](coding-guideline.html). -4. Make sure that the additions don't create errors in a [Apache RAT](http://creadur.apache.org/rat/) check via `mvn apache-rat:check` - -#####A-4. Rebase the branch onto current apache master - - # Make sure we have the latest bits - $ git fetch apache - # Rebase - $ git rebase -i apache/master - -In the rebase process, make sure that the contribution is squashed to a single commit. From the list of commits, "pick" the commit in the first line (the oldest), and "squash" the commits in the remaining lines: - - pick 7387a49 Comment for first commit - squash 3371411 Comment for second commit - squash 9bf956d Comment for third commit - -[Chapter 3.6](http://www.git-scm.com/book/en/v2/Git-Branching-Rebasing) and [Chapter 7.6](http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) of the [Git Book](http://www.git-scm.com/book/en/v2) contains lots of information on what this means. - -Please make sure that the commit message contains the information given in "Commit Message" above. The latest commit can be changed at any time with the command: - - $ git commit --amend - -#####A-5. Push the code into apache's git - -This is a good time to reflect back on this change and whether it is likely to break the build. If you are certain that it won't, go ahead and do: - - $ git checkout master - $ git merge BRANCH_NAME - $ git push apache master - -This pushes the current branch into the master branch hosted on Apache's git repository. From there, it will be mirrored onto GitHub. And by virtue of the comment added above to the commit message, GitHub will now close the Pull Request. - -####Method B - -#####B-1. Update local master - -In this method, you will work directly on your local master branch. Make sure you have the latest changes on your local master branch, by running: - - $ git pull apache master - -#####B-2. Download the .diff and apply it - -You can download the `.diff` file by appending `.diff` to the GitHub Pull Request url. This file will contain the exact changes that are shown in "Files changed" in the GitHub GUI. For example, for `https://github.com/apache/incubator-reef/pull/24` the `.diff` file is `https://github.com/apache/incubator-reef/pull/24.diff`. Using this url, run apply: - - $ wget https://github.com/apache/incubator-reef/pull/24.diff - $ git apply 24.diff - -#####B-3. Commit and edit author information - -Commit all files, making sure to include all modified and **new** files. In the commit message, make sure to include all information given in "Commit Message" above. After committing you must also change the author information to reflect the original author (and not yourself): - - $ git commit --amend --author "Original Author Name <[email protected]>" - -#####B-4. Push the code into apache's git - -Now push your single commit to apache git: - - $ git push apache master - -This pushes the current branch into the master branch hosted on Apache's git repository. From there, it will be mirrored onto GitHub. And by virtue of the comment added above to the commit message, GitHub will now close the Pull Request. http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/contributing.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/contributing.md b/website/src/site/markdown/contributing.md deleted file mode 100644 index 8e26456..0000000 --- a/website/src/site/markdown/contributing.md +++ /dev/null @@ -1,159 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Contributing to REEF - -####First things first: Welcome! - -Arriving on this page means that you are interested in helping us out. For that: Welcome and thank you! REEF is a community driven project and we always welcome new people joining us. We are inviting contributions in many forms to all parts of the project, including: - -- Bug reports regarding the software, the documentation, the website, guides like this, etc. -- Graphics (for instance, [we don't have a logo yet](https://issues.apache.org/jira/browse/REEF-14)). -- Documentation updates, tutorials, examples. -- Code: Patches, new features, etc - -####Getting started: Join the community. - -The first step is to **join the community** by joining our [mailing list](mailing-list.html). This is where all discussions regarding the project are happening, where we answer each other's questions and where you will find a friendly bunch to welcome you. - -If you want to work on the REEF code base, it is a good idea to [learn how to compile and test REEF](tutorial.html). - -####Finding things to work on - -At any given time, there is any number of [open, unassigned issues on REEF](https://issues.apache.org/jira/issues/?jql=project%20%3D%20REEF%20AND%20status%20%3D%20Open%20AND%20resolution%20%3D%20Unresolved%20AND%20assignee%20in%20\(EMPTY\)%20ORDER%20BY%20priority%20DESC). Note that that list doesn't only contain coding tasks, but also documentation, graphics work, the website and so on. We use JIRA to manage all open todos for the project, software or otherwise. However, some of the items on that list might since have become obsolete or such. Hence, it is always a good idea to get in touch with the rest of the community on the mailing list before getting started. - -####Code contribution process - -While contributing to REEF, you will encounter ASF infrastructure, GitHub infrastructure, and follow ASF and REEF-specific customs. The most important of those customs is to communicate throughout the whole process. That way, the REEF community has a chance to help you along the way. - -####Before you start the work - -An important part of the process comes before you start the contribution process. Most issues should first be brought up in the [[email protected]](mailto:[email protected]) mailing list. If you haven't done so yet, [subscribe to the list](mailing-list.html). After discussion, you or one of the other developers will create an Issue on [ASF JIRA](https://issues.apache.org/jira/browse/REEF/). Again, create an account if you don't have one. Write a succinct description of the Issue, making sure to incorporate any discussion from the mailing list. - -And once you are ready to make changes, have a look at the [coding guidelines](coding-guideline.html) to make sure your code agrees with the rest of REEF. - -####Creating the contribution as a GitHub Pull Request - -REEF uses [GitHub's Pull Requests](https://help.github.com/articles/using-pull-requests/) as the main method to accept changes: you fork REEF into your own repository, make changes, and then send a pull request to the GitHub repository. In more detail: - -1. Fork repository -2. Create branch -3. Make changes in your local machine -4. Merge the master branch into your branch -5. Push commits into the your remote repo (forked) -6. Send a pull request. -7. Participate in the code review. - -#####1. Fork repository - -First, you need to fork the REEF repository. Go to the [Github repository](https://github.com/apache/incubator-reef) mirrored from ASF, and click "Fork" button. Then you will have your own repository. Clone this repository to your local machine: - - $ git clone https://github.com/{your_alias}/incubator-reef.git - -Then, add the Apache GitHub repository as upstream: - - $ git remote add upstream https://github.com/apache/incubator-reef - -A correct git configuration should look similar to the following, with an origin and upstream: - - $ git remote -v - origin https://github.com/{your_alias}/incubator-reef.git (fetch) - origin https://github.com/{your_alias}/incubator-reef.git (push) - upstream https://github.com/apache/incubator-reef.git (fetch) - upstream https://github.com/apache/incubator-reef.git (push) - -If you have an `apache.org` email address, now is the time to [configure git](https://git-wip-us.apache.org/) to use it: - - $ git config user.name "My Name Here" - $ git config user.email [email protected] - -#####2. Create a branch to work on -Before making changes, you have to make sure the issue to resolve (e.g. fix a bug, implement a new feature, etc) is registered in the [REEF JIRA](https://issues.apache.org/jira/browse/REEF). Create a branch to address the issue on your own. The name of the branch should reference the issue, e.g., `REEF-{issue_number}`. You can take a look how others name their branches. - -#####3. Make changes in your local machine -Write the code and make commits as usual. Make sure all new files contain the [ASF licensing header](https://github.com/apache/incubator-reef/blob/master/LICENSE_HEADER.txt). - -#####4. Merge the master branch into your branch -Before sending a pull request, you should make sure that your branch includes the latest changes in the master branch. Please run the following: - - $ git fetch upstream - $ git checkout {your_branch} # Skip this step if you are already on your branch. - $ git merge upstream/master - -Resolve the conflicts if they exist. Test with the merged code so that it does not break the system. Then, check that Apache headers are in place where needed, by running RAT: - - $ mvn apache-rat:check - -Finally, as a courtesy to the merger, you can rebase to master and squash all the commits from your PR into one: - - # Rebase - $ git rebase -i upstream/master - -In the rebase process, make sure that the contribution is squashed to a single commit. From the list of commits, "pick" the commit in the first line (the oldest), and "squash" the commits in the remaining lines: - - pick  7387a49 Comment for first commit - squash 3371411 Comment for second commit - squash 9bf956d Comment for third commit - -[Chapter 3.6](http://www.git-scm.com/book/en/v2/Git-Branching-Rebasing) and [Chapter 7.6](http://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) of the [Git Book](http://www.git-scm.com/book/en/v2) contains lots of information on what this means. - -In this process, git allows you to edit the commit message for the final, squashed commit. This commit message will serve as the description of the pull request and will in all likelihood appear in verbatim in the REEF history. In other words: Spend some time making it good. A common template for this commit message is: - - [REEF-JIRA_ISSUE_NUMBER]: THE_TITLE_OF_THE_JIRA - - This addressed the issue by - * INSERT_DESCRIPTION_OF_SOMETHING - * INSERT_DESCRIPTION_OF_SOMETHING_ELSE - * ... - - JIRA: [REEF-JIRA_ISSUE_NUMBER](https://issues.apache.org/jira/browse/REEF-JIRA_ISSUE_NUMBER) - -You can get a good idea how other people write their commit messages by inspecting the output of `git log`. - -#####5. Push commits into the your remote repo (forked) - -You're almost done! Push your commits into your own repo. - - $ git push origin HEAD - -#####6. Send a pull request - -It is time to send a pull request. If you do not know much about pull requests, you may want to read this [article](https://help.github.com/articles/using-pull-requests/). - -If you go to the repository at the Github website, you can notice that "Compare & Pull request" button appears. Click the button or move into "pull request" tab if you cannot find the button. - -When you create a pull request, you choose the branches to compare and merge. Choose the base as `apache:master` and the head `{your_alias}:{branch_name}`. The description will be the message of your one commit. Feel free to edit it if you aren't satisfied with your commit message. - -Please **update the JIRA issue with a link to the Pull Request**. This triggers an email to the mailing list, which will hopefully result in a committer stepping up for a code review. - -#####7. The code review - -REEF follows a review-then-commit (RTC) model. We perform all our code reviews on GitHub: Community members will leave comments on your pull request and suggest changes. You can have [a look at prior pull requests](https://github.com/apache/incubator-reef/pulls?q=is%3Apr+is%3Aclosed) to get an idea of what to expect. During this process, you may want to change your code. You can do that by pushing additional commits to your branch. Don't worry about squashing the commits into one at this stage. This will be done by the committer once your code is merged into REEF. - -When the code review concludes, one of Committers will merge your work into the REEF codebase. Good job! - -#####8. Merge the pull request (Committers only) - -If you are a committer, you can follow the steps in the [Committer Guide](committer-guide.html) to merge the pull request. Of course, you won't be merging your own pull requests. Nudge committers as needed to make sure everything gets merged correctly. - -####Other guides - -There are other pages in this area you might be interested in: - -- [Coding Guidelines](coding-guideline.html) -- [REEF Tutorial](tutorial.html) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/issue-tracker.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/issue-tracker.md b/website/src/site/markdown/issue-tracker.md deleted file mode 100644 index e5ce3ca..0000000 --- a/website/src/site/markdown/issue-tracker.md +++ /dev/null @@ -1,21 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Issue Tracker - -Please visit the [ASF JIRA](https://issues.apache.org/jira/browse/REEF) website for the most updated issues. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/powered-by.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/powered-by.md b/website/src/site/markdown/powered-by.md deleted file mode 100644 index d8d8bdf..0000000 --- a/website/src/site/markdown/powered-by.md +++ /dev/null @@ -1,21 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Powered By - -- [Azure Stream Analytics](http://azure.microsoft.com/en-us/services/stream-analytics/) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/reef-examples.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/reef-examples.md b/website/src/site/markdown/reef-examples.md deleted file mode 100644 index 74f32e0..0000000 --- a/website/src/site/markdown/reef-examples.md +++ /dev/null @@ -1,135 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#Further REEF Examples - -- [Running HelloREEF on YARN](#yarn) - - [Prerequisites](#yarn-prerequisites) - - [How to configure REEF on YARN](#yarn-configurations) - - [How to launch HelloReefYarn](#yarn-launch) -- [Running a REEF Webserver: HelloREEFHttp](#http) - - [Prerequisites](#http-prerequisites) - - [HttpServerShellCmdtHandler](#http-server-shell) -- [Task Scheduler: Retaining Evaluators](#task-scheduler) - - [Prerequisites](#task-scheduler-prerequisites) - - [REST API](#task-scheduler-rest-api) - - [Reusing the Evaluators](#task-scheduler-reusing-evaluators) - - -###<a name="yarn"></a>Running HelloREEF on YARN - -REEF applications can be run on multiple runtime environments. Using `HelloReefYarn`, we will see how to configure and launch REEF applications on YARN. - -####<a name="yarn-prerequisites"></a>Prerequisites - -* [You have compiled REEF locally](tutorial.html#install) -* [YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) - -####<a name="yarn-configurations"></a>How to configure REEF on YARN - -The only difference between running a REEF application on YARN vs locally is the runtime configuration: - -``` - final LauncherStatus status = DriverLauncher - .getLauncher(YarnClientConfiguration.CONF.build()) - .run(getDriverConfiguration(), JOB_TIMEOUT); -``` - -####<a name="yarn-launch"></a>How to launch HelloReefYarn - -Running `HelloReefYarn` is very similar to running `HelloREEF`: - - yarn jar reef-examples/target/reef-examples-{$REEF_VERSION}-shaded.jar org.apache.reef.examples.hello.HelloREEFYarn - -**Note**: *The path divider may be different for different OS (e.g. Windows uses \\ while Linux uses / for dividers) so change the code as needed.* - -You can see how REEF applications work on YARN environments in [Introduction to REEF](introduction.html). - -###<a name="http"></a>Running a REEF Webserver: HelloREEFHttp - -REEF also has a webserver interface to handle HTTP requests. This webserver can be utilized in many different manners such as in Interprocess Communcation or in conjuction with the REST API. - -To demonstrate a possible use for this interface, `HelloREEFHttp` serves as a simple webserver to execute shell commands requested from user input. The first thing we should do is register a handler to receive the HTTP requests. - -####<a name="http-prerequisites"></a>Prerequisites - -* [You have compiled REEF locally](tutorial.html#install) - -####<a name="http-server-shell"></a>HttpServerShellCmdtHandler - -`HttpServerShellCmdtHandler` implements `HttpHandler` but three methods must be overridden first: `getUriSpecification`, `setUriSpecification`, and `onHttpRequest`. - -- <a name="http-urispecification"></a> -`UriSpecification` defines the URI specification for the handler. More than one handler can exist per application and thus each handler is distinguished using this specification. Since `HelloREEFHttp` defines `UriSpecification` as `Command`, an HTTP request looks like `http://{host_address}:{host_port}/Command/{request}`. - -- <a name="http-onhttprequest"></a> -`onHttpRequest` defines a hook for when an HTTP request for this handler is invoked. - -###<a name="task-scheduler"></a>Retaining Evaluators: Task Scheduler - -Another example is Task scheduler. Getting commands from users using the REST API, it allocates multiple evaluators and submits the tasks. - -It is a basic Task Scheduler example using Reef-webserver. The application receives the task (shell command) list from user and execute the tasks in a FIFO order. - -####<a name="task-scheduler-prerequisites"></a>Prerequisites - -* [You have compiled REEF locally](tutorial.html#install) -* [Running REEF Webserver : HelloREEFHttp](#http) - -####<a name="task-scheduler-rest-api"></a>REST API - -Users can send the HTTP request to the server via URL : - - http://{address}:{port}/reef-example-scheduler/v1 - -And the possible requests are as follows: - -* `/list`: lists all the tasks' statuses. -* `/clear`: clears all the tasks waiting in the queue and returns how many tasks have been removed. -* `/submit?cmd=COMMAND`: submits a task to execute COMMAND and returns the task id. -* `/status?id=ID`: returns the status of the task with the id, "ID". -* `/cancel?id=ID`: cancels the task with the id, "ID". -* `/max-eval?num={num}`: sets the maximum number of evaluators. - -The result of each task is written in the log files - both in the driver's and the evaluators'. - -####<a name="task-scheduler-reusing-evaluators"></a>Reusing the Evaluators - -You can find the method `retainEvaluator()` in SchedulerDriver: - -``` - /** - * Retain the complete evaluators submitting another task - * until there is no need to reuse them. - */ - private synchronized void retainEvaluator(final ActiveContext context) { - if (scheduler.hasPendingTasks()) { - scheduler.submitTask(context); - } else if (nActiveEval > 1) { - nActiveEval--; - context.close(); - } else { - state = State.READY; - waitForCommands(context); - } - } -``` - -When `Task` completes, `EventHandler` for `CompletedTask` event is invoked. An instance of `CompletedTask` is then passed using the parameter to get the `ActiveContext` object from the `CompletedTask`. We can reuse this `Evaluator` by submitting another `Task` to it if there is a task to launch. - -Using the `-retain false` argument disables this functionality and allocates a new evalutor for every task. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/markdown/tutorial.md ---------------------------------------------------------------------- diff --git a/website/src/site/markdown/tutorial.md b/website/src/site/markdown/tutorial.md deleted file mode 100644 index a81729e..0000000 --- a/website/src/site/markdown/tutorial.md +++ /dev/null @@ -1,137 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - -http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -#REEF Tutorial - -- [Installing and Compiling REEF](#install) -- [Running HelloREEF](#running-reef) - - [Local](#local) - - [HelloREEFNoClient](#helloREEFNoClient) - - [YARN](reef-examples.html#yarn) -- [Further Examples](#further-examples) - - -###<a name="install"></a>Installing and Compiling REEF - - -####Requirements - -- [Java](http://www.oracle.com/technetwork/java/index.html) 7 Development Kit -- [Maven 3](http://maven.apache.org/) or newer. Make sure that `mvn` is in your `PATH`. -- [Protocol Buffers Compiler (protoc) 2.5](http://code.google.com/p/protobuf/) Make sure that protoc is in your PATH. **Note**: You need to install version 2.5. Newer versions won't work. - -With these requirements met, the instructions below should work regardless of OS choice and command line interpreter. On Windows, you might find [this](http://cs.markusweimer.com/2013/08/02/how-to-setup-powershell-for-github-maven-and-java-development/) tutorial helpful in setting up PowerShell with Maven, GitHub and Java. You will still have to install the [Protocol Buffers Compiler](https://code.google.com/p/protobuf/), though. - -####Cloning the repository -#####Comitters - $ git clone https://git-wip-us.apache.org/repos/asf/incubator-reef.git - -#####Users - $ git clone git://git.apache.org/incubator-reef.git - -####Compiling the code -REEF is built using Maven. Hence, a simple - - $ mvn clean install - -should suffice. Note that we have quite a few integration tests in the default build. Hence, you might be better off using - - $ mvn -TC1 -DskipTests clean install - -This runs one thread per core (`-TC1`) and skips the tests (`-DskipTests`) - -**Note**: You will see many exception printouts during the compilation of REEF with tests. Those are not, in fact, problems with the build: REEF guarantees that exceptions thrown on remote machines get serialized and shipped back to the Driver. We have extensive unit tests for that feature that produce the confusing printouts. - -### <a name="running-reef"></a>Running HelloREEF - -####Prerequisites - -[You have compiled REEF locally](#install). - -####Running your first REEF program: Hello, REEF! - -The module REEF Examples in the folder `reef-examples` contains several simple programs built on REEF to help you get started with development. As always, the simplest of those is our "Hello World": Hello REEF. Upon launch, it grabs a single Evaluator and submits a single Task to it. That Actvity, fittingly, prints 'Hello REEF!' to stdout. To launch it, navigate to `REEF_HOME` and use the following command: - - java -cp reef-examples/target/reef-examples-{$REEF_VERSION}-incubating-shaded.jar org.apache.reef.examples.hello.HelloREEF - -**Note**: *The path divider may be different for different OS (e.g. Windows uses \\ while Linux uses / for dividers) and the version number of your version of REEF must be placed instead of the \* so change the code as needed.* - -This invokes the shaded jar within the target directory and launches HelloREEF on the local runtime of REEF. During the run, you will see something similar to this output: - - Powered by - ___________ ______ ______ _______ - / ______ / / ___/ / ___/ / ____/ - / _____/ / /__ / /__ / /___ - / /\ \ / ___/ / ___/ / ____/ - / / \ \ / /__ / /__ / / - /__/ \__\ /_____/ /_____/ /__/ - - ... - INFO: REEF Version: 0.10.0-incubating - ... - INFO: The Job HelloREEF is running. - ... - INFO: The Job HelloREEF is done. - ... - INFO: REEF job completed: COMPLETED - -####Where's the output? - -The local runtime simulates a cluster of machines: It executes each Evaluator in a separate process on your machine. Hence, the Evaluator that printed "Hello, REEF" is not executed in the same process as the program you launched above. So, how do you get to the output of the Evaluator? The local runtime generates one folder per job it executes in `REEF_LOCAL_RUNTIME`: - - > cd REEF_LOCAL_RUNTIME - > ls HelloREEF* - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - d---- 1/26/2015 11:21 AM HelloREEF-1422238904731 - -The job folder names are comprised of the job's name (here, `HelloREEF`) and the time stamp of its submission (here, `1422238904731`). If you submit the same job multiple times, you will get multiple folders here. Let's move on: - - > cd HelloREEF-1422238904731 - > ls - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - d---- 1/26/2015 11:21 AM driver - d---- 1/26/2015 11:22 AM Node-1-1422238907421 - -Inside of the job's folder, you will find one folder for the job's Driver (named `driver`) and one per Evaluator. Their name comprises of the virtual node simulated by the local runtime (here, `Node-1`) followed by the time stamp of when this Evaluator was allocated on that node, here `1422238907421`. As the HelloREEF example program only allocated one Evaluator, we only see one of these folders here. Let's peek inside: - - > cd Node-1-1422238907421 - > ls evaluator* - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - -a--- 1/26/2015 11:21 AM 1303 evaluator.stderr - -a--- 1/26/2015 11:21 AM 14 evaluator.stdout - -`evaluator.stderr` contains the output on stderr of this Evaluator, which mostly consists of logs helpful in debugging. `evaluator.stdout` contains the output on stdout. And, sure enough, this is where you find the "Hello, REEF!" message. - -####<a name="helloREEFNoClient"></a>The difference between HelloREEF and HelloREEFNoClient - -The HelloREEF application has multiple versions that all service different needs and one of these applications, `HelloREEFNoClient`, allows the creation of the Driver and Evaluators without the creation of a Client. In many scenarios involving a cluster of machines, one Client will access multiple Drivers so not every Driver needs to create a Client and that is where the `HelloREEFNoClient` application shines. - -Running `HelloREEFNoClient` is nearly identical to running `HelloREEF`: - - java -cp reef-examples/target/reef-examples-{$REEF_VERSION}-incubating-shaded.jar org.apache.reef.examples.hello.HelloREEFNoClient - -**Note**: *The path divider may be different for different OS (e.g. Windows uses \\ while Linux uses / for dividers) and the version number of your version of REEF must be placed instead of the \* so change the code as needed.* - -and the output should be the same with `evaluator.stdout` containing the "Hello, REEF!" message. - -###<a name="further-examples"></a>Further Examples - -Further examples of using REEF can be found [here](reef-examples.html). http://git-wip-us.apache.org/repos/asf/incubator-reef/blob/b2e85d12/website/src/site/site.xml ---------------------------------------------------------------------- diff --git a/website/src/site/site.xml b/website/src/site/site.xml index 3574c6d..d42c697 100644 --- a/website/src/site/site.xml +++ b/website/src/site/site.xml @@ -79,22 +79,21 @@ under the License. <menu name="Documentation"> <item name="Introduction to REEF" href="introduction.html"/> - <item name="REEF Tutorial" href="tutorial.html"/> - <item name="Further REEF Examples" href="reef-examples.html"/> + <item name="REEF Tutorial" href="https://cwiki.apache.org/confluence/display/REEF/Tutorials"/> <item name="Glossary" href="glossary.html"/> <item name="Tang" href="tang.html"/> <item name="Wake" href="wake.html"/> </menu> <menu name="Contribution"> - <item name="Contributing" href="contributing.html"/> - <item name="Committer Guide" href="committer-guide.html"/> - <item name="Coding Guidelines" href="coding-guideline.html"/> + <item name="Contributing" href="https://cwiki.apache.org/confluence/display/REEF/Contributing"/> + <item name="Committer Guide" href="https://cwiki.apache.org/confluence/display/REEF/Committer+Guide"/> + <item name="Coding Guidelines" href="https://cwiki.apache.org/confluence/display/REEF/Coding+Guidelines"/> </menu> <menu name="Community"> <item name="Team" href="team.html"/> <item name="Mailing List" href="mailing-list.html"/> - <item name="Issue Tracker" href="issue-tracker.html"/> - <item name="Powered By" href="powered-by.html"/> + <item name="Issue Tracker" href="https://issues.apache.org/jira/browse/REEF"/> + <item name="Powered By" href="https://cwiki.apache.org/confluence/display/REEF/Powered+by+REEF"/> </menu> <menu name="ASF"> <item name="Apache Software Foundation" href="http://www.apache.org/foundation/"/>
