Repository: incubator-zeppelin Updated Branches: refs/heads/master dff32b2a3 -> f7deadbaf
Vagrant Virtual Machine and Readme.md This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful Python Libraries and one obscure configuration to help with matplotlib plotting inside Zeppelin. The virtual machine consists of: - Ubuntu Server 14.04 LTS - Node.js 0.12.7 - npm 2.11.3 - ruby 1.9.3 + bundler, rake, make (only required for building jekyll documentation) - Maven 3.3.3 - Git - Unzip - libfontconfig to avoid phatomJs missing dependency issues - openjdk-7-jdk - Python addons: pip, matplotlib, scipy, numpy, pandas - Changes are made to the python backend configuration so that pyspark can render matplotlib plots without an interactive window Author: Jeff Steinmetz <[email protected]> Closes #460 from jeffsteinmetz/master and squashes the following commits: cfb1034 [Jeff Steinmetz] add ruby 1.9.3 install + bundler, make and rake for jekyll doc support. Add vagrant port 4000 forwarding for jekyll documentation build watcher 9a78355 [Jeff Steinmetz] remove apache python script from commit 97a5819 [Jeff Steinmetz] remove apache python script. remove nodesource and use official node.js distribution download 0b169a6 [Jeff Steinmetz] remove license from apache-mirror-selector.py. Simplify node install a86c55f [Jeff Steinmetz] typo fixed. licenses in supporting readme and apache mirror python script 3ec6a6b [Jeff Steinmetz] add virtual machine to docs 4b70e63 [Jeff Steinmetz] remove reference to python backend config in readme. 5c31125 [Jeff Steinmetz] add apache-mirror-selector to ansible playbook 11fdc0d [Jeff Steinmetz] removed matplotlib global Agg config. Updated vagrant virtual box settings with hostname and vm name. upated python example to set Agg afc26a4 [Jeff Steinmetz] roll back to original top level README.md de07f00 [Jeff Steinmetz] update readme based on comments. Add license 853ddca [Jeff Steinmetz] vagrant virtual machine and readme ff8078a [Jeff Steinmetz] vagrant virtual machine and readme fb31785 [Jeff Steinmetz] initial commit of vagrant virtual machine script 5888360 [Jeff Steinmetz] merge upstream 1eb27eb [Jeff Steinmetz] expanded build instructions to include pyspark, and clarified maven and node.js requirements Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/f7deadba Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/f7deadba Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/f7deadba Branch: refs/heads/master Commit: f7deadbaf3ed7b2e148bc8666c1e5d0331ae554b Parents: dff32b2 Author: Jeff Steinmetz <[email protected]> Authored: Thu Nov 26 13:24:51 2015 -0800 Committer: Lee moon soo <[email protected]> Committed: Mon Nov 30 18:28:52 2015 +0900 ---------------------------------------------------------------------- docs/docs.md | 1 + docs/install/virtual_machine.md | 182 +++++++++++++++++++ scripts/vagrant/zeppelin-dev/.gitignore | 4 + scripts/vagrant/zeppelin-dev/README.md | 167 +++++++++++++++++ scripts/vagrant/zeppelin-dev/Vagrantfile | 74 ++++++++ scripts/vagrant/zeppelin-dev/ansible-roles.yml | 25 +++ .../zeppelin-dev/roles/common/tasks/main.yml | 31 ++++ .../zeppelin-dev/roles/java7jdk/tasks/main.yml | 18 ++ .../zeppelin-dev/roles/maven/tasks/main.yml | 46 +++++ .../zeppelin-dev/roles/nodejs/tasks/main.yml | 29 +++ .../roles/python-addons/tasks/main.yml | 28 +++ .../zeppelin-dev/roles/ruby/tasks/main.yml | 33 ++++ .../vagrant/zeppelin-dev/show-instructions.sh | 39 ++++ 13 files changed, 677 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/docs/docs.md ---------------------------------------------------------------------- diff --git a/docs/docs.md b/docs/docs.md index 8e338f7..0eebfcf 100644 --- a/docs/docs.md +++ b/docs/docs.md @@ -23,6 +23,7 @@ limitations under the License. * [Install](./install/install.html) * [YARN Install](./install/yarn_install.html) +* [Virtual Machine Install](./install/virtual_machine.html) ### Tutorial http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/docs/install/virtual_machine.md ---------------------------------------------------------------------- diff --git a/docs/install/virtual_machine.md b/docs/install/virtual_machine.md new file mode 100644 index 0000000..f210740 --- /dev/null +++ b/docs/install/virtual_machine.md @@ -0,0 +1,182 @@ +--- +layout: page +title: "Install A Zeppelin ready Virtual Machine" +description: "" +group: install +--- +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +{% include JB/setup %} + + +## Vagrant Virtual Machine for Apache Zeppelin + +The Apache Zeppelin distribution includes a scripts directory + + `scripts/vagrant/zeppelin-dev` + +This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful [Python Libraries](#pythonextras) + +####Installing the required components to launch a virtual machine. + +This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com/downloads "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems. + +### Create a Zeppelin Ready VM in 4 Steps (5 on Windows) + +*If you are running Windows and don't yet have python installed, install Python 2.7.x* [Python Windows Installer](https://www.python.org/downloads/release/python-2710/) + +1. Download and Install Vagrant: [Vagrant Downloads](http://www.vagrantup.com/downloads) +2. Install Ansible: [Ansible Python pip install](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip) +`sudo easy_install pip` then +`sudo pip install ansible` +`ansible --version` should now report version 1.9.2 or higher +3. Install Virtual Box: [Virtual Box Downloads](https://www.virtualbox.org/ "Virtual Box") +4. Type `vagrant up` from within the `/scripts/vagrant/zeppelin-dev` directory + +Thats it! + +You can now run `vagrant ssh` and this will place you into the guest machines terminal prompt. + +If you don't wish to build Zeppelin from scratch, run the z-manager installer script while running in the guest VM: + +``` +curl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh | bash +``` + + + +### Building Zeppelin + +You can now `git clone https://github.com/apache/incubator-zeppelin.git` into a directory on your host machine, or directly in your virtual machine. + +Cloning zeppelin into the `/scripts/vagrant/zeppelin-dev` directory from the host, will allow the directory to be shared between your host and the guest machine. + +Cloning the project again may seem counter intuitive, since this script likley originated from the project repository. Consider copying just the vagrant/zeppelin-dev script from the zeppelin project as a stand alone directory, then once again clone the specific branch you wish to build. + +Synced folders enable Vagrant to sync a folder on the host machine to the guest machine, allowing you to continue working on your project's files on your host machine, but use the resources in the guest machine to compile or run your project. _[(1) Synced Folder Description from Vagrant Up](https://docs.vagrantup.com/v2/synced-folders/index.html)_ + +By default, Vagrant will share your project directory (the directory with the Vagrantfile) to `/vagrant`. Which means you should be able to build within the guest machine after you +`cd /vagrant/incubator-zeppelin` + + +### What's in this VM? + +Running the following commands in the guest machine should display these expected versions: + +`node --version` should report *v0.12.7* +`mvn --version` should report *Apache Maven 3.3.3* and *Java version: 1.7.0_85* + + +The virtual machine consists of: + + - Ubuntu Server 14.04 LTS + - Node.js 0.12.7 + - npm 2.11.3 + - ruby 1.9.3 + rake, make and bundler (only required if building jekyll documentation) + - Maven 3.3.3 + - Git + - Unzip + - libfontconfig to avoid phatomJs missing dependency issues + - openjdk-7-jdk + - Python addons: pip, matplotlib, scipy, numpy, pandas + +### How to build & run Zeppelin + +This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. + +``` +cd /incubator-zeppelin +mvn clean package -Pspark-1.5 -Ppyspark -Dhadoop.version=2.2.0 -Phadoop-2.2 -DskipTests +./bin/zeppelin-daemon.sh start +``` + +On your host machine browse to `http://localhost:8080/` + +If you [turned off port forwarding](#tweakvm) in the `Vagrantfile` browse to `http://192.168.51.52:8080` + + +### [Tweaking the Virtual Machine](id:tweakvm) + +If you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host. + +Comment out the `forward_port` line, and uncomment the `private_network` line in Vagrantfile. The subnet that works best for your local network will vary so adjust `192.168.*.*` accordingly. + +``` +#config.vm.network "forwarded_port", guest: 8080, host: 8080 +config.vm.network "private_network", ip: "192.168.51.52" +``` + +`vagrant halt` followed by `vagrant up` will restart the guest machine bound to the IP address of `192.168.51.52`. +This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources. You may wish to launch nodes in virtual machines with IP Addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc.. + + +### [Python Extras](id:pythonextras) + +With zeppelin running, Numpy, SciPy, Pandas and Matplotlib will be available. Create a pyspark notebook, and try + +``` +%pyspark + +import numpy +import scipy +import pandas +import matplotlib + +print "numpy " + numpy.__version__ +print "scipy " + scipy.__version__ +print "pandas " + pandas.__version__ +print "matplotlib " + matplotlib.__version__ +``` + +To Test plotting using matplotlib into a rendered %html SVG image, try + +``` +%pyspark + +import matplotlib +matplotlib.use('Agg') # turn off interactive charting so this works for server side SVG rendering +import matplotlib.pyplot as plt +import numpy as np +import StringIO + +# clear out any previous plots on this notebook +plt.clf() + +def show(p): + img = StringIO.StringIO() + p.savefig(img, format='svg') + img.seek(0) + print "%html <div style='width:600px'>" + img.buf + "</div>" + +# Example data +people = ('Tom', 'Dick', 'Harry', 'Slim', 'Jim') +y_pos = np.arange(len(people)) +performance = 3 + 10 * np.random.rand(len(people)) +error = np.random.rand(len(people)) + +plt.barh(y_pos, performance, xerr=error, align='center', alpha=0.4) +plt.yticks(y_pos, people) +plt.xlabel('Performance') +plt.title('How fast do you want to go today?') + +show(plt) +``` + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/.gitignore ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/.gitignore b/scripts/vagrant/zeppelin-dev/.gitignore new file mode 100644 index 0000000..c5ab41f --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/.gitignore @@ -0,0 +1,4 @@ +.vagrant +.DS_Store +incubator-zeppelin + http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/README.md ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/README.md b/scripts/vagrant/zeppelin-dev/README.md new file mode 100644 index 0000000..ffb7f23 --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/README.md @@ -0,0 +1,167 @@ +<!-- +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +## Vagrant Virtual Machine for Apache Zeppelin + +This script creates a virtual machine that launches a repeatable, known set of core dependencies required for developing Zeppelin. It can also be used to run an existing Zeppelin build if you don't plan to build from source. For pyspark users, this script also includes several helpful [Python Libraries and one obscure configuration to help with matplotlib plotting inside Zeppelin](#pythonextras) + +####Installing the required components to launch a virtual machine. + +This script requires three applications, [Ansible](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip "Ansible"), [Vagrant](http://www.vagrantup.com/downloads "Vagrant") and [Virtual Box](https://www.virtualbox.org/ "Virtual Box"). All of these applications are freely available as Open Source projects and extremely easy to set up on most operating systems. + +### Create a Zeppelin Ready VM in 4 Steps (5 on Windows) + +*If you are running Windows and don't yet have python installed, install Python 2.7.x* [Python Windows Installer](https://www.python.org/downloads/release/python-2710/) + +1. Download and Install Vagrant: [Vagrant Downloads](http://www.vagrantup.com/downloads) +2. Install Ansible: [Ansible Python pip install](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip) +`sudo easy_install pip` then +`sudo pip install ansible` +`ansible --version` should now report version 1.9.2 or higher +3. Install Virtual Box: [Virtual Box Downloads](https://www.virtualbox.org/ "Virtual Box") +4. Type `vagrant up` from within the `/scripts/vagrant/zeppelin-dev` directory + +Thats it! + +You can now run `vagrant ssh` and this will place you into the guest machines terminal prompt. + +If you don't wish to build Zeppelin from scratch, run the z-manager installer script while running in the guest VM: + +``` +curl -fsSL https://raw.githubusercontent.com/NFLabs/z-manager/master/zeppelin-installer.sh | bash +``` + + + +### Building Zeppelin + +You can now `git clone https://github.com/apache/incubator-zeppelin.git` into a directory on your host machine, or directly in your virtual machine. + +Cloning zeppelin into the `/scripts/vagrant/zeppelin-dev` directory from the host, will allow the directory to be shared between your host and the guest machine. + +Cloning the project again may seem counter intuitive, since this script likley originated from the project repository. Consider copying just the vagrant/zeppelin-dev script from the zeppelin project as a stand alone directory, then once again clone the specific branch you wish to build. + +Synced folders enable Vagrant to sync a folder on the host machine to the guest machine, allowing you to continue working on your project's files on your host machine, but use the resources in the guest machine to compile or run your project. _[(1) Synced Folder Description from Vagrant Up](https://docs.vagrantup.com/v2/synced-folders/index.html)_ + +By default, Vagrant will share your project directory (the directory with the Vagrantfile) to `/vagrant`. Which means you should be able to build within the guest machine after you +`cd /vagrant/incubator-zeppelin` + + +### What's in this VM? + +Running the following commands in the guest machine should display these expected versions: + +`node --version` should report *v0.12.7* +`mvn --version` should report *Apache Maven 3.3.3* and *Java version: 1.7.0_85* + + +The virtual machine consists of: + + - Ubuntu Server 14.04 LTS + - Node.js 0.12.7 + - npm 2.11.3 + - ruby 1.9.3 + rake, make and bundler (only required if building jekyll documentation) + - Maven 3.3.3 + - Git + - Unzip + - libfontconfig to avoid phatomJs missing dependency issues + - openjdk-7-jdk + - Python addons: pip, matplotlib, scipy, numpy, pandas + +### How to build & run Zeppelin + +This assumes you've already cloned the project either on the host machine in the zeppelin-dev directory (to be shared with the guest machine) or cloned directly into a directory while running inside the guest machine. + +``` +cd /incubator-zeppelin +mvn clean package -Pspark-1.5 -Ppyspark -Dhadoop.version=2.2.0 -Phadoop-2.2 -DskipTests +./bin/zeppelin-daemon.sh start +``` + +On your host machine browse to `http://localhost:8080/` + +If you [turned off port forwarding](#tweakvm) in the `Vagrantfile` browse to `http://192.168.51.52:8080` + + +### [Tweaking the Virtual Machine](id:tweakvm) + +If you plan to run this virtual machine along side other Vagrant images, you may wish to bind the virtual machine to a specific IP address, and not use port fowarding from your local host. + +Comment out the `forward_port` line, and uncomment the `private_network` line in Vagrantfile. The subnet that works best for your local network will vary so adjust `192.168.*.*` accordingly. + +``` +#config.vm.network "forwarded_port", guest: 8080, host: 8080 +config.vm.network "private_network", ip: "192.168.51.52" +``` + +`vagrant halt` followed by `vagrant up` will restart the guest machine bound to the IP address of `192.168.51.52`. +This approach usually is typically required if running other virtual machines that discover each other directly by IP address, such as Spark Masters and Slaves as well as Cassandra Nodes, Elasticsearch Nodes, and other Spark data sources. You may wish to launch nodes in virtual machines with IP Addresses in a subnet that works for your local network, such as: 192.168.51.53, 192.168.51.54, 192.168.51.53, etc.. + + +### [Python Extras](id:pythonextras) + +With zeppelin running, Numpy, SciPy, Pandas and Matplotlib will be available. Create a pyspark notebook, and try + +``` +%pyspark + +import numpy +import scipy +import pandas +import matplotlib + +print "numpy " + numpy.__version__ +print "scipy " + scipy.__version__ +print "pandas " + pandas.__version__ +print "matplotlib " + matplotlib.__version__ +``` + +To Test plotting using matplotlib into a rendered %html SVG image, try + +``` +%pyspark + +import matplotlib +matplotlib.use('Agg') # turn off interactive charting so this works for server side SVG rendering +import matplotlib.pyplot as plt +import numpy as np +import StringIO + +# clear out any previous plots on this notebook +plt.clf() + +def show(p): + img = StringIO.StringIO() + p.savefig(img, format='svg') + img.seek(0) + print "%html <div style='width:600px'>" + img.buf + "</div>" + +# Example data +people = ('Tom', 'Dick', 'Harry', 'Slim', 'Jim') +y_pos = np.arange(len(people)) +performance = 3 + 10 * np.random.rand(len(people)) +error = np.random.rand(len(people)) + +plt.barh(y_pos, performance, xerr=error, align='center', alpha=0.4) +plt.yticks(y_pos, people) +plt.xlabel('Performance') +plt.title('How fast do you want to go today?') + +show(plt) +``` + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/Vagrantfile ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/Vagrantfile b/scripts/vagrant/zeppelin-dev/Vagrantfile new file mode 100644 index 0000000..f709023 --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/Vagrantfile @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- mode: ruby -*- +# vi: set ft=ruby : + +# Vagrantfile API/syntax version. Don't touch unless you know what you're doing! +VAGRANTFILE_API_VERSION = "2" + +Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| + # All Vagrant configuration is done here. The most common configuration + # options are documented and commented below. For a complete reference, + # please see the online documentation at vagrantup.com. + + # Every Vagrant virtual environment requires a box to build off of. + config.vm.box = "hashicorp/precise64" + + # Disable automatic box update checking. If you disable this, then + # boxes will only be checked for updates when the user runs + # `vagrant box outdated`. This is not recommended. + #config.vm.box_check_update = false + + # Create a forwarded port mapping which allows access to a specific port + # within the machine from a port on the host machine. + + # Forward local host port 8080 to zeppelin port 8080 + # Forward local host port 4040 to spark UI port 4040 + # Forward local host port 4000 to jekyll documentation port 4000 + # comment out the following lines to use a fixed IP + config.vm.network "forwarded_port", guest: 8080, host: 8080 + config.vm.network "forwarded_port", guest: 4040, host: 4040 + config.vm.network "forwarded_port", guest: 4000, host: 4000 + + + # Uncomment to create a private network, which allows access to the machine + # using a specific IP. Port forwarding not required in this case. + # Select an address range based on a free block available + # in your network environment. 192.168.x.x is an example + #config.vm.network "private_network", ip: "192.168.51.52" + + # Set the hostname. Comment this out if you don't want the hostname managed by the VM + config.vm.hostname = "zeppelinvm" + + # Modify the number of CPUs and memory used by the VM + # by adjusting + # --memory + # and uncommenting and/or updating + # --cpus + + config.vm.provider "virtualbox" do |vb| + vb.customize ["modifyvm", :id, "--memory", "2048"] + #vb.customize ["modifyvm", :id, "--cpus", "2"] + vb.name = "zeppelinvm" + end + + config.vm.provision "ansible" do |ansible| + ansible.playbook = "ansible-roles.yml" + end + + config.vm.provision "shell", path: "show-instructions.sh", run: "always" + +end http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/ansible-roles.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/ansible-roles.yml b/scripts/vagrant/zeppelin-dev/ansible-roles.yml new file mode 100644 index 0000000..ba2a8c8 --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/ansible-roles.yml @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--- +- name: Run Ansible + hosts: all + sudo: yes + roles: + - common + - ruby + - java7jdk + - nodejs + - maven + - python-addons http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/common/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/common/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/common/tasks/main.yml new file mode 100644 index 0000000..0e5fcce --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/common/tasks/main.yml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: call apt-get update + # this must be first so that apt-get installs to point to correct resources + # Only run "update_cache=yes" if the last one is more than 3600 seconds (1 hour) ago + apt: update_cache=yes cache_valid_time=3600 + +- name: Install unzip + apt: pkg=unzip state=present + +- name: Install curl + apt: pkg=curl state=present + +- name: Install git + apt: pkg=git state=present + +- name: Install libfontconfig to avoid phatomJs missing dependency issues + apt: pkg=libfontconfig state=present http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/java7jdk/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/java7jdk/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/java7jdk/tasks/main.yml new file mode 100644 index 0000000..7e78aef --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/java7jdk/tasks/main.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +- name: Install openjdk-7 + apt: pkg=openjdk-7-jdk state=present + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml new file mode 100644 index 0000000..e435aac --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/maven/tasks/main.yml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# setup maven +# Assumes Ubuntu Server 14.04 LTS +# Assumes Java 7+ jdk installed +--- + +- name: Install pythons httplib2 library so that ansible uri module works + apt: pkg=python-httplib2 state=present + + +- name: Call apache web service to find preferred maven download mirror + uri: url=http://www.apache.org/dyn/closer.cgi?path=maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz&asjson=1 return_content=yes + register: webResponse + +#- debug: var=webResponse + +- name: download maven + get_url: url="{{webResponse.json.preferred}}{{webResponse.json.path_info}}" + dest=/tmp/apache-maven-3.3.3-bin.tar.gz + mode=0440 + validate_certs=False + +- name: extract maven tgz + unarchive: src=/tmp/apache-maven-3.3.3-bin.tar.gz + dest=/usr/local/ + copy=no + creates=/usr/local/apache-maven-3.3.3 + +- name: create symlink to this maven version + file: src=/usr/local/apache-maven-3.3.3/bin/mvn + path=/usr/bin/mvn + state=link http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/nodejs/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/nodejs/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/nodejs/tasks/main.yml new file mode 100755 index 0000000..9cf009a --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/nodejs/tasks/main.yml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Install Node.js from the official Node.js +--- + +- name: Download official Linux x64 distribution from Node.js + get_url: url=http://nodejs.org/dist/v0.12.8/node-v0.12.8-linux-x64.tar.gz dest=/tmp/node-v0.12.8-linux-x64.tar.gz + +- name: Unpack node + unarchive: src=/tmp/node-v0.12.8-linux-x64.tar.gz dest=/tmp/ copy=no + +- name: Install Node.js + command: tar -C /usr/local --strip-components 1 -xzf /tmp/node-v0.12.8-linux-x64.tar.gz + args: + creates: /usr/local/bin/node + http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/python-addons/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/python-addons/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/python-addons/tasks/main.yml new file mode 100644 index 0000000..27c4f53 --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/python-addons/tasks/main.yml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# setup zeppelin python dependencies +# Assumes Ubuntu Server 14.04 LTS (HVM) - ami-f64f77b3 64 bit +# Assumes Java 7 installed as part of the top level playbook +--- + +- name: Install python dependencies. This will take a while. + apt: pkg={{ item }} state=latest + with_items: + - python-pip + - python-matplotlib + - python-scipy + - python-numpy + - python-pandas http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/roles/ruby/tasks/main.yml ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/roles/ruby/tasks/main.yml b/scripts/vagrant/zeppelin-dev/roles/ruby/tasks/main.yml new file mode 100644 index 0000000..01cdc5f --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/roles/ruby/tasks/main.yml @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# setup ruby +# Assumes Ubuntu Server 14.04 LTS +--- + +- name: Install ruby 1.9.3 + apt: pkg=ruby1.9.3 state=present + +- name: set global ruby version + alternatives: name=ruby path=/usr/bin/ruby1.9.1 link=/usr/bin/ruby + +- name: Install make + apt: pkg=make state=latest + +- name: Install latest available version of rake. + gem: name=rake state=latest user_install=no + +- name: Install latest available version of bundler. + gem: name=bundler state=latest user_install=no http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/f7deadba/scripts/vagrant/zeppelin-dev/show-instructions.sh ---------------------------------------------------------------------- diff --git a/scripts/vagrant/zeppelin-dev/show-instructions.sh b/scripts/vagrant/zeppelin-dev/show-instructions.sh new file mode 100644 index 0000000..5ce0ce0 --- /dev/null +++ b/scripts/vagrant/zeppelin-dev/show-instructions.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +echo '# Post vagrant up instructions.' +echo '# From your host machine,' +echo '# git clone the incubator-zeppelin branch into this directory' +echo +echo 'git clone https://github.com/apache/incubator-zeppelin.git' +echo +echo '# Cloning the project again may seem counter intuitive, since this script' +echo '# originated from the project repository. Consider copying just the vagrant/zeppelin-dev' +echo '# script from the zeppelin project as a stand alone directory, then once again clone' +echo '# the specific branch you wish to build.' +echo +echo 'vagrant ssh' +echo +echo '# then when running inside the VM' +echo +echo 'cd /vagrant/incubator-zeppelin' +echo 'mvn clean package -DskipTests' +echo +echo '# or for a specific build' +echo +echo 'mvn clean package -Pspark-1.5 -Ppyspark -Dhadoop.version=2.2.0 -Phadoop-2.2 -DskipTests' +echo './bin/zeppelin-daemon.sh start' +
