Repository: kylin Updated Branches: refs/heads/document 37d31e31a -> 8e639ce01
add more content to advanced setting Project: http://git-wip-us.apache.org/repos/asf/kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/8e639ce0 Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/8e639ce0 Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/8e639ce0 Branch: refs/heads/document Commit: 8e639ce0129e4089491829967f98909c718253b4 Parents: 37d31e3 Author: shaofengshi <shaofeng...@apache.org> Authored: Thu Oct 13 21:37:42 2016 +0800 Committer: shaofengshi <shaofeng...@apache.org> Committed: Thu Oct 13 21:37:42 2016 +0800 ---------------------------------------------------------------------- website/_docs15/install/advance_settings.md | 59 ++++++++++++++---------- website/_docs15/install/kylin_cluster.md | 2 +- 2 files changed, 36 insertions(+), 25 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kylin/blob/8e639ce0/website/_docs15/install/advance_settings.md ---------------------------------------------------------------------- diff --git a/website/_docs15/install/advance_settings.md b/website/_docs15/install/advance_settings.md index 284d4fb..2abd6a4 100644 --- a/website/_docs15/install/advance_settings.md +++ b/website/_docs15/install/advance_settings.md @@ -1,18 +1,35 @@ --- layout: docs15 -title: "Advance Settings of Kylin Environment" +title: "Advance Settings" categories: install permalink: /docs15/install/advance_settings.html --- -## Settings for compression +## Overwrite default kylin.properties at Cube level +In `conf/kylin.properties` there are many prameters, which control/impact on Kylin's behaviors; Most parameters are global configs like security or job related; while some are Cube related; These Cube related parameters can be customized at each Cube level, so you can control the behaviors more flexibly. The GUI to do this is in the "Configuration Overwrites" step of the Cube wizard, as the screenshot below. -By default, Kylin does not enable compression, this is not the recommend settings for production environment, but a tradeoff for new Kylin users. A suitable compression algorithm will reduce the storage overhead. But unsupported algorithm will break the Kylin job build also. There are three kinds of compression used in Kylin, HBase table compression, Hive output compression and MR jobs output compression. Compression settings only take effect after restarting Kylin server instance (by `./kylin.sh start` and `./kylin.sh stop`). + -### HBase table compression +Here take two example: + + * `kylin.cube.algorithm`: it defines the Cubing algorithm that the job engine will select; Its default value is "auto", means the engine will dynamically pick an algorithm ("layer" or "inmem") by sampling the data. If you knows Kylin and your data/cluster well, you can set your preferred algorithm directly (usually "inmem" has better performance but will request more memory). + + * `kylin.hbase.region.cut`: it defines how big a region is when creating the HBase table. The default value is "5" (GB) per region. It might be too big for a small or medium cube, so you can give it a smaller value to get more regions created, then can gain better query performance. + +## Overwrite default Hadoop job conf at Cube level +The `conf/kylin_job_conf.xml` and `conf/kylin_job_conf_inmem.xml` manage the default configurations for Hadoop jobs. If you have the need to customize the configs by cube, you can achieve that with the similar way as above, but need adding a prefix `kylin.job.mr.config.override.`; These configs will be parsed out and then applied when submitting jobs. See two examples below: + + * If want a cube's job getting more memory from Yarn, you can define: `kylin.job.mr.config.override.mapred.map.child.java.opts=-Xmx7g` and `kylin.job.mr.config.override.mapred.map.memory.mb=8192` + * If want a cube's job going to a different Yarn resource queue, you can define: `kylin.job.mr.config.override.mapred.job.queuename=myQueue` (note: "myQueue" is just a sample) + +## Enable compression + +By default, Kylin does not enable compression, this is not the recommend settings for production environment, but a tradeoff for new Kylin users. A suitable compression algorithm will reduce the storage overhead. But unsupported algorithm will break the Kylin job build also. There are three kinds of compression used in Kylin, HBase table compression, Hive output compression and MR jobs output compression. + +* HBase table compression The compression settings define in `kyiln.properties` by `kylin.hbase.default.compression.codec`, default value is *none*. The valid value includes *none*, *snappy*, *lzo*, *gzip* and *lz4*. Before changing the compression algorithm, please make sure the selected algorithm is supported on your HBase cluster. Especially for snappy, lzo and lz4, not all Hadoop distributions include these. -### Hive output compression +* Hive output compression The compression settings define in `kylin_hive_conf.xml`. The default setting is empty which leverages the Hive default configuration. If you want to override the settings, please add (or replace) the following properties into `kylin_hive_conf.xml`. Take the snappy compression for example: {% highlight Groff markup %} <property> @@ -27,8 +44,8 @@ The compression settings define in `kylin_hive_conf.xml`. The default setting is </property> {% endhighlight %} -### MR jobs output compression -The compression settings define in `kylin_job_conf.xml`. The default setting is empty which leverages the MR default configuration. If you want to override the settings, please add (or replace) the following properties into `kylin_job_conf.xml`. Take the snappy compression for example: +* MR jobs output compression +The compression settings define in `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. The default setting is empty which leverages the MR default configuration. If you want to override the settings, please add (or replace) the following properties into `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. Take the snappy compression for example: {% highlight Groff markup %} <property> <name>mapreduce.map.output.compress.codec</name> @@ -42,28 +59,22 @@ The compression settings define in `kylin_job_conf.xml`. The default setting is </property> {% endhighlight %} -### LZO compression +Compression settings only take effect after restarting Kylin server instance. -#### Make sure LZO is working in your environment +## Enable LDAP or SSO authentication -There is a simple tool to test whether LZO is well installed on EVERY SERVER in HBase cluster ( http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/bk_installing_manually_book/content/ch_install_hdfs_yarn_chapter.html#install-snappy-man-install ), and restart the cluster. -To test it on the hadoop CLI that you deployed Kylin, Just run +Check [How to Enable Security with LDAP and SSO](../howto/howto_ldap_and_sso.html) -{% highlight Groff markup %} -hbase org.apache.hadoop.hbase.util.CompressionTest file:///PATH-TO-A-LOCAL-TMP-FILE lzo -{% endhighlight %} -If no exception is printed, you're good to go. Otherwise you'll need to first install LZO properly on this server. -To test if the HBase cluster is ready to create LZO compressed tables, test following HBase command: +## Enable email notification +Kylin can send email notification on job complete/fail; To enable this, edit `conf/kylin.properties`, set the following parameters: {% highlight Groff markup %} -create 'lzoTable', {NAME => 'colFam',COMPRESSION => 'LZO'} +mail.enabled=true +mail.host=your-smtp-server +mail.username=your-smtp-account +mail.password=your-smtp-pwd +mail.sender=your-sender-address {% endhighlight %} -#### Use LZO compression - -Compression settings only take effect after restarting Kylin server instance (by `./kylin.sh start` and `./kylin.sh stop`). To use LZO for compressing MR jobs you need to modify $KYLIN_HOME/conf/kylin_job_conf.xml by replacing compression configuration entries to com.hadoop.compression.lzo. You can refer to other documents for details: http://xiaming.me/posts/2014/05/03/enable-lzo-compression-on-hadoop-pig-and-spark. To use LZO for compressing HBase tables you need to open $KYLIN_HOME/conf/kylin.properties, change kylin.hbase.default.compression.codec=lzo. - -## Enable LDAP or SSO authentication - -Check [How to Enable Security with LDAP and SSO](../howto/howto_ldap_and_sso.html) +Restart Kylin server to take effective. To disable, set `mail.enabled` back to `false`. http://git-wip-us.apache.org/repos/asf/kylin/blob/8e639ce0/website/_docs15/install/kylin_cluster.md ---------------------------------------------------------------------- diff --git a/website/_docs15/install/kylin_cluster.md b/website/_docs15/install/kylin_cluster.md index 97947be..87f693a 100644 --- a/website/_docs15/install/kylin_cluster.md +++ b/website/_docs15/install/kylin_cluster.md @@ -1,6 +1,6 @@ --- layout: docs15 -title: "Deploy in cluster mode" +title: "Deploy in Cluster Mode" categories: install permalink: /docs15/install/kylin_cluster.html ---
