Author: lidong
Date: Tue May 24 09:12:13 2016
New Revision: 1745335
URL: http://svn.apache.org/viewvc?rev=1745335&view=rev
Log:
refine blog aws emr
Modified:
kylin/site/blog/2016/05/24/aws-emr/index.html
kylin/site/feed.xml
Modified: kylin/site/blog/2016/05/24/aws-emr/index.html
URL:
http://svn.apache.org/viewvc/kylin/site/blog/2016/05/24/aws-emr/index.html?rev=1745335&r1=1745334&r2=1745335&view=diff
==============================================================================
--- kylin/site/blog/2016/05/24/aws-emr/index.html (original)
+++ kylin/site/blog/2016/05/24/aws-emr/index.html Tue May 24 09:12:13 2016
@@ -41,7 +41,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Apache Kylin | Apache Kylin Guide for AWS EMR User</title>
- <meta name="description" content="Install Apache Kylin on EMR Create an EMR
cluster from AWS console, and remember to pick the applications configuration
which contains HBase and Hive as sho...">
+ <meta name="description" content="Install Apache Kylin on EMR Create an EMR
cluster from AWS console, and remember to pick the applications configuration
which contains HBase and Hive. ">
<meta name="author" content="Apache Kylin">
<link rel="shortcut icon" href="fav.png" type="image/png">
@@ -189,14 +189,14 @@
<h1 id="install-apache-kylin-on-emr">Install Apache Kylin on EMR</h1>
<ol>
<li>
- <p>Create an EMR cluster from AWS console, and remember to pick the
applications configuration which contains HBase and Hive as shown here. <br />
+ <p>Create an EMR cluster from AWS console, and remember to pick the
applications configuration which contains HBase and Hive. <br />
<img src="/images/blog/aws_emr_console.png" alt="install EMR cluster"
title="Applications configuration when creating EMR cluster" /></p>
</li>
<li>
<p>Login to master node of the cluster with ssh. (<a
href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-connect-master-node-ssh.html";>instruction</a>)</p>
</li>
<li>
- <p>Download and install Kylin binary, we take Kylin version 1.5.1 as
example here (Please refer to <a href="http://kylin.apache.org/download/";>Kylin
download page</a> for latest binary package). Run the commands below.</p>
+ <p>Download and install Kylin binary, we take Kylin version 1.5.1 as
example here (please refer to <a href="http://kylin.apache.org/download/";>Kylin
download page</a> for latest binary package). Run the commands below.</p>
</li>
</ol>
@@ -209,9 +209,9 @@ export KYLIN_HOME=`pwd`
<h1 id="configure-apache-kylin">Configure Apache Kylin</h1>
-<p>To make Kylin run well on AWS EMR we need to change some scripts. The fix
can be found in this <a
href="https://github.com/apache/kylin/commit/dc08186d570e16b37d9ddaab80aba28801cdb3d0";>commit</a>
and itâll be available in Kylinâs future version.</p>
+<p>To make Kylin run well on AWS EMR we need to modify some scripts. The fix
can be found in <a
href="https://github.com/apache/kylin/commit/dc08186d570e16b37d9ddaab80aba28801cdb3d0";>this
commit</a> and itâll be available in Kylinâs future version.</p>
-<p>*First file is $KYLIN_HOME/bin/find-hive-dependency.sh, starting from line
68.</p>
+<p>First file is <code
class="highlighter-rouge">$KYLIN_HOME/bin/find-hive-dependency.sh</code>,
starting from line 68.</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 68 if [ -z
"$HCAT_HOME" ]
69 then
@@ -225,7 +225,7 @@ export KYLIN_HOME=`pwd`
+ elif [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
+ hcatalog_home=/usr/lib/oozie/lib
76 else
- 77 echo "Couldn't locate hcatalog installation, please make sure it is
installed and set HCAT_HOME to the path."
+ 77 echo "Couldn't locate hcatalog installation, please make sure it is
installed and set HCAT_HOME..."
78 exit 1
79 fi
80 else
@@ -235,9 +235,9 @@ export KYLIN_HOME=`pwd`
</code></pre>
</div>
-<p>When Kylin starts running it tries to detect environment such as Hive home
dictionary. It goes well most time but not for AWS EMR. In AWS EMR nodes,
Hcatalog libs is under â/usr/lib/oozie/libâ, which is not expected. The
modification makes Kylin handle it specially once detecting itâs running on
AWS EMR.</p>
+<p>Kylin tries to detect environment such as Hive home dictionary on starting.
It goes well in most time but not in AWS EMR case. In AWS EMR nodes Hcatalog
libs is under <code class="highlighter-rouge">/usr/lib/oozie/lib</code>, which
is not expected. The modification makes Kylin handle it specially once
detecting itâs on AWS EMR.</p>
-<p>*Second file is $KYLIN_HOME/bin/find-hbase-dependency.sh, starting from
line 20.</p>
+<p>Second file is <code
class="highlighter-rouge">$KYLIN_HOME/bin/find-hbase-dependency.sh</code>,
starting from line 20.</p>
<div class="highlighter-rouge"><pre class="highlight"><code> 20
hbase_classpath=`hbase classpath`
+
@@ -250,7 +250,7 @@ export KYLIN_HOME=`pwd`
</code></pre>
</div>
-<p>In AWS EMR, Hbase scripts reset environment variable HBASE_CLASSPATH in its
first run, which is determined by environment variable HBASE_ENV_INIT. Kylin
builds its own class path according to this variable. To avoid missing libs
caused by HBASE_CLASSPATH reseting, Kylin set HBASE_ENV_INIT to âtrueâ
after running âhbase classpathâ so that HBASE_CLASSPATH wonât be reset
next time run âhbase classpathâ.</p>
+<p>In AWS EMR Hbase scripts reset environment variable <code
class="highlighter-rouge">HBASE_CLASSPATH</code> in its first run, which is
determined by environment variable <code
class="highlighter-rouge">HBASE_ENV_INIT</code>. Kylin builds its own class
path according to this variable. To avoid missing libs caused by <code
class="highlighter-rouge">HBASE_CLASSPATH</code> reseting, Kylin set <code
class="highlighter-rouge">HBASE_ENV_INIT</code> to <code
class="highlighter-rouge">"true"</code> so that <code
class="highlighter-rouge">HBASE_CLASSPATH</code> wonât be reset next time run
<code class="highlighter-rouge">hbase classpath</code>.</p>
<h1 id="load-sample-data-and-start-apache-kylin">Load sample data and start
Apache Kylin</h1>
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1745335&r1=1745334&r2=1745335&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Tue May 24 09:12:13 2016
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Tue, 24 May 2016 09:44:24 -0700</pubDate>
- <lastBuildDate>Tue, 24 May 2016 09:44:24 -0700</lastBuildDate>
+ <pubDate>Tue, 24 May 2016 10:10:54 -0700</pubDate>
+ <lastBuildDate>Tue, 24 May 2016 10:10:54 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
@@ -28,14 +28,14 @@
<description><h1
id="install-apache-kylin-on-emr">Install Apache Kylin on
EMR</h1>
<ol>
<li>
- <p>Create an EMR cluster from AWS console, and remember to pick the
applications configuration which contains HBase and Hive as shown here. <br
/>
+ <p>Create an EMR cluster from AWS console, and remember to pick the
applications configuration which contains HBase and Hive. <br />
<img src="/images/blog/aws_emr_console.png" alt="install EMR
cluster" title="Applications configuration when creating EMR
cluster" /></p>
</li>
<li>
<p>Login to master node of the cluster with ssh. (<a
href="http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-connect-master-node-ssh.html">instruction</a>)</p>
</li>
<li>
- <p>Download and install Kylin binary, we take Kylin version 1.5.1 as
example here (Please refer to <a
href="http://kylin.apache.org/download/">Kylin download
page</a> for latest binary package). Run the commands below.</p>
+ <p>Download and install Kylin binary, we take Kylin version 1.5.1 as
example here (please refer to <a
href="http://kylin.apache.org/download/">Kylin download
page</a> for latest binary package). Run the commands below.</p>
</li>
</ol>
@@ -48,9 +48,9 @@ export KYLIN_HOME=`pwd`
<h1 id="configure-apache-kylin">Configure Apache
Kylin</h1>
-<p>To make Kylin run well on AWS EMR we need to change some scripts. The
fix can be found in this <a
href="https://github.com/apache/kylin/commit/dc08186d570e16b37d9ddaab80aba28801cdb3d0">commit</a>
and itâll be available in Kylinâs future version.</p>
+<p>To make Kylin run well on AWS EMR we need to modify some scripts. The
fix can be found in <a
href="https://github.com/apache/kylin/commit/dc08186d570e16b37d9ddaab80aba28801cdb3d0">this
commit</a> and itâll be available in Kylinâs future
version.</p>
-<p>*First file is $KYLIN_HOME/bin/find-hive-dependency.sh, starting from
line 68.</p>
+<p>First file is <code
class="highlighter-rouge">$KYLIN_HOME/bin/find-hive-dependency.sh</code>,
starting from line 68.</p>
<div class="highlighter-rouge"><pre
class="highlight"><code> 68 if [ -z "$HCAT_HOME" ]
69 then
@@ -64,7 +64,7 @@ export KYLIN_HOME=`pwd`
+ elif [ -n is_aws ] &amp;&amp; [ -d
"/usr/lib/oozie/lib" ]; then
+ hcatalog_home=/usr/lib/oozie/lib
76 else
- 77 echo "Couldn't locate hcatalog installation, please make
sure it is installed and set HCAT_HOME to the path."
+ 77 echo "Couldn't locate hcatalog installation, please make
sure it is installed and set HCAT_HOME..."
78 exit 1
79 fi
80 else
@@ -74,9 +74,9 @@ export KYLIN_HOME=`pwd`
</code></pre>
</div>
-<p>When Kylin starts running it tries to detect environment such as Hive
home dictionary. It goes well most time but not for AWS EMR. In AWS EMR nodes,
Hcatalog libs is under â/usr/lib/oozie/libâ, which is not expected. The
modification makes Kylin handle it specially once detecting itâs running on
AWS EMR.</p>
+<p>Kylin tries to detect environment such as Hive home dictionary on
starting. It goes well in most time but not in AWS EMR case. In AWS EMR nodes
Hcatalog libs is under <code
class="highlighter-rouge">/usr/lib/oozie/lib</code>, which
is not expected. The modification makes Kylin handle it specially once
detecting itâs on AWS EMR.</p>
-<p>*Second file is $KYLIN_HOME/bin/find-hbase-dependency.sh, starting
from line 20.</p>
+<p>Second file is <code
class="highlighter-rouge">$KYLIN_HOME/bin/find-hbase-dependency.sh</code>,
starting from line 20.</p>
<div class="highlighter-rouge"><pre
class="highlight"><code> 20 hbase_classpath=`hbase classpath`
+
@@ -89,7 +89,7 @@ export KYLIN_HOME=`pwd`
</code></pre>
</div>
-<p>In AWS EMR, Hbase scripts reset environment variable HBASE_CLASSPATH
in its first run, which is determined by environment variable HBASE_ENV_INIT.
Kylin builds its own class path according to this variable. To avoid missing
libs caused by HBASE_CLASSPATH reseting, Kylin set HBASE_ENV_INIT to âtrueâ
after running âhbase classpathâ so that HBASE_CLASSPATH wonât be reset
next time run âhbase classpathâ.</p>
+<p>In AWS EMR Hbase scripts reset environment variable <code
class="highlighter-rouge">HBASE_CLASSPATH</code> in its
first run, which is determined by environment variable <code
class="highlighter-rouge">HBASE_ENV_INIT</code>. Kylin
builds its own class path according to this variable. To avoid missing libs
caused by <code
class="highlighter-rouge">HBASE_CLASSPATH</code> reseting,
Kylin set <code
class="highlighter-rouge">HBASE_ENV_INIT</code> to <code
class="highlighter-rouge">"true"</code> so that
<code class="highlighter-rouge">HBASE_CLASSPATH</code>
wonât be reset next time run <code
class="highlighter-rouge">hbase classpath</code>.</p>
<h1 id="load-sample-data-and-start-apache-kylin">Load sample
data and start Apache Kylin</h1>
