Repository: kylin Updated Branches: refs/heads/document 17a3d91be -> ba80d86f0
revise advanced settings Project: http://git-wip-us.apache.org/repos/asf/kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/ba80d86f Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/ba80d86f Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/ba80d86f Branch: refs/heads/document Commit: ba80d86f01ffa03cceb64976adbb7f41bcabdc42 Parents: 17a3d91 Author: Hongbin Ma <mahong...@apache.org> Authored: Fri Apr 8 14:52:22 2016 +0800 Committer: Hongbin Ma <mahong...@apache.org> Committed: Fri Apr 8 14:59:01 2016 +0800 ---------------------------------------------------------------------- website/_docs15/install/advance_settings.md | 25 ++++++++++++------------ 1 file changed, 12 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kylin/blob/ba80d86f/website/_docs15/install/advance_settings.md ---------------------------------------------------------------------- diff --git a/website/_docs15/install/advance_settings.md b/website/_docs15/install/advance_settings.md index a2da72c..b2f4f04 100644 --- a/website/_docs15/install/advance_settings.md +++ b/website/_docs15/install/advance_settings.md @@ -5,9 +5,17 @@ categories: install permalink: /docs15/install/advance_settings.html --- -## Enable LZO compression +## Settings for compression -By default Kylin leverages snappy compression to compress the output of MR jobs, as well as hbase table storage, reducing the storage overhead. We do not choose LZO compression in Kylin because hadoop venders tend to not include LZO in their distributions due to license(GPL) issues. To enable LZO in Kylin, follow these steps: +### Disable compression (Easiest way to address compression related issues) + +By default Kylin leverages snappy compression to compress the output of MR jobs, as well as hbase table storage, to reduce the storage overhead. We do not choose LZO compression in Kylin because hadoop venders tend to not include LZO in their distributions due to license(GPL) issues. If you compression related issues happened in your cubing job, you have two options: 1. Disable compression 2. Choose other compression algorithms like LZO. + +#### Disable HBase compression + +Compression settings only take effect after restarting Kylin server instance (by `./kylin.sh start` and `./kylin.sh stop`). To disable compressing MR jobs you need to modify $KYLIN_HOME/conf/kylin_job_conf.xml by removing all configuration entries related to compression(Just grep the keyword "compress"). To disable compressing hbase tables you need to open $KYLIN_HOME/conf/kylin.properties and remove the line starting with kylin.hbase.default.compression.codec. + +### LZO compression #### Make sure LZO is working in your environment @@ -25,18 +33,9 @@ To test if the hbase cluster is ready to create LZO compressed tables, test foll create 'lzoTable', {NAME => 'colFam',COMPRESSION => 'LZO'} {% endhighlight %} -#### Use LZO for HBase compression - -You'll need to stop Kylin first by running `./kylin.sh stop`, and then modify $KYLIN_HOME/conf/kylin_job_conf.xml by uncommenting some configuration entries related to LZO compression. -After this, you need to run `./kylin.sh start` to start Kylin again. Now Kylin will use LZO to compress MR outputs and hbase tables. - -Goto $KYLIN_HOME/conf/kylin.properties, change kylin.hbase.default.compression.codec=snappy to kylin.hbase.default.compression.codec=lzo - -#### Use LZO for MR jobs - -Modify $KYLIN_HOME/conf/kylin_job_conf.xml by changing all org.apache.hadoop.io.compress.SnappyCodec to com.hadoop.compression.lzo.LzoCodec. +#### Use LZO compression -Start Kylin again. Now Kylin will use LZO to compress MR outputs and HBase tables. +Compression settings only take effect after restarting Kylin server instance (by `./kylin.sh start` and `./kylin.sh stop`). To use LZO for compressing MR jobs you need to modify $KYLIN_HOME/conf/kylin_job_conf.xml by replacing configuration entries related to compression from org.apache.hadoop.io.compress.SnappyCodec to com.hadoop.compression.lzo. You can refer to other documents for details: http://xiaming.me/posts/2014/05/03/enable-lzo-compression-on-hadoop-pig-and-spark. To use LZO for compressing hbase tables you need to open $KYLIN_HOME/conf/kylin.properties, change kylin.hbase.default.compression.codec=snappy to kylin.hbase.default.compression.codec=lzo. ## Enable LDAP or SSO authentication
