Author: lidong
Date: Thu Aug 8 14:10:46 2019
New Revision: 1864708
URL: http://svn.apache.org/viewvc?rev=1864708&view=rev
Log:
Add detailed analysis for Hive Global Dictionary
Added:
kylin/site/images/Hive-Global-Dictionary/
kylin/site/images/Hive-Global-Dictionary/cube-level-config.png (with
props)
kylin/site/images/Hive-Global-Dictionary/hive-global-dict-table.png (with
props)
kylin/site/images/Hive-Global-Dictionary/set-hive-dict-column.png (with
props)
kylin/site/images/Hive-Global-Dictionary/three-added-steps.png (with
props)
Modified:
kylin/site/docs30/howto/howto_use_hive_mr_dict.html
kylin/site/feed.xml
Modified: kylin/site/docs30/howto/howto_use_hive_mr_dict.html
URL:
http://svn.apache.org/viewvc/kylin/site/docs30/howto/howto_use_hive_mr_dict.html?rev=1864708&r1=1864707&r2=1864708&view=diff
==============================================================================
--- kylin/site/docs30/howto/howto_use_hive_mr_dict.html (original)
+++ kylin/site/docs30/howto/howto_use_hive_mr_dict.html Thu Aug 8 14:10:46 2019
@@ -6357,31 +6357,219 @@ var _hmt = _hmt || [];
<article
class="post-content" >
- <h3
id="global-dictionary">Global Dictionary</h3>
-<p>Count distinct measure is very important for many scenario, such as
PageView statistics, Kylin support count distinct since 1.5.3
(http://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/). <br />
-Apache Kylin implements precisely count distinct based on bitmap, and use
global dictionary to encode string value into a Dict. <br />
-Currently we have to build global dictionary in single process/JVM, which may
take a lot of time and memory for UHC. By in this feature(KYLIN-3841), we use
Hive, a distributed SQL engine to build global dictionary.</p>
+ <h2
id="global-dictionary-in-hive">Global Dictionary in Hive</h2>
+<p>Count distinct(bitmap) measure is very important for many scenario, such as
PageView statistics, and Kylin support count distinct since 1.5.3 .<br />
+Apache Kylin implements precisely count distinct measure based on bitmap, and
use global dictionary to encode string value into integer. <br />
+Currently we have to build global dictionary in single process/JVM, which may
take a lot of time and memory for UHC. By this feature(KYLIN-3841), we use
Hive, a distributed SQL engine to build global dictionary.</p>
<p>This will help to:<br />
-1. Reduce memory pressure of Kylin process, MapReduce will be used to build
dict for Kylin<br />
-2. Make global dictionary reusable<br />
-3. Make global dictionary readable, you may use global dictionary outside
Kylin, maybe useful in many scenario.</p>
-
-<p>And this feature will add three steps if enabled.<br />
-1. Global Dict Mr/Hive extract dict_val from Data<br />
-2. Global Dict Mr/Hive build dict_val<br />
-3. Global Dict Mr/Hive replace dict_val to Data</p>
-
-<h3 id="how-to-use">How to use</h3>
-<p>If you have a count distinct(bitmap) measure for a UHC. Says columns name
are PV_ID and USER_ID, and table name is USER_ACTION, you may add cube-level
configuration <code
class="highlighter-rouge">kylin.dictionary.mr-hive.columns=USER_ACTION_PV_ID,USER_ACTION_USER_ID</code>
to enable this feature.<br />
-You have to know that the value will be replaced into encoded integer in flat
hive table, and this may cause failure of some query.</p>
+1. Reduce memory pressure of Kylin process, MapReduce(or other engine which
hive used) will be used to build dict instead of Kylin process itself.<br />
+2. Make build base cuboid quicker, because string value has been encoded in
previous step.<br />
+3. Make global dictionary reusable.<br />
+4. Make global dictionary readable and bijective, you may use global
dictionary outside Kylin, this maybe useful in many scenario.</p>
+
+<h3 id="step-by-step-analysis">Step by step Analysis</h3>
+<p>This feature will add three additional steps in cube building if enabled,
let us try to understand what Kylin do in these steps.</p>
+
+<ol>
+ <li>
+ <p>Global Dict Mr/Hive extract dict_val from Data</p>
+
+ <ul>
+ <li>Create a Hive table for store global dictionary if it is not exists,
table name should be <em>CubeName_Suffix</em>. This table has two normal column
and one partition column, two normal columns are <code
class="highlighter-rouge">dict_key</code> and <code
class="highlighter-rouge">dict_value</code>, which for origin value and encoded
integer respectively.</li>
+ <li>Create a temporary table with â__group_byâ as its suffix, which
used to store distinct value for specific column. This table has one normal
column and one partition column, normal column is <code
class="highlighter-rouge">dict_key</code> which used to store origin value.</li>
+ <li>Insert distinct value into temporary table created above for each
column by using a hive query âselect cloA from flatTable group by
cloAâ.</li>
+ </ul>
+
+ <p>When this step finished, you should get a temporary table contains
distinct values, each partition for specific Count_Distinct column.</p>
+ </li>
+ <li>
+ <p>Global Dict Mr/Hive build dict_val</p>
+
+ <ul>
+ <li>Find all fresh distinct value which never exists in any older
segments by <em>LEFT JOIN</em> between global dictionary table and temporary
table.</li>
+ <li>Append all fresh distinct value to the tail of global dictionary
table by <em>UNION</em>. By the power of <code
class="highlighter-rouge">row_number</code> function in Hive, added value will
be encoded with integer in incremental way.</li>
+ </ul>
+
+ <p>When this step finished, all distinct value for all Count_Distinct
column will be encoded correctly in global dictionary table.</p>
+ </li>
+ <li>
+ <p>Global Dict Mr/Hive replace dict_val to Data</p>
+
+ <ul>
+ <li>Using <em>LEFT JOIN</em> to replace original string value with
encoded integer on flat table which used to build cuboid later.</li>
+ </ul>
+
+ <p>When this step finished, all string value which belong to
Count_Distinct column will be updated with encoded integer in flat hive
table.</p>
+ </li>
+</ol>
+
+<hr />
+
+<h2 id="how-to-use">How to use</h2>
+
+<p>If you have some count distinct(bitmap) measure, and data type of that
column is String, you may need Hive Global Dictionary. Says columns name are
PV_ID and USER_ID, and table name is USER_ACTION, you may add cube-level
configuration <code
class="highlighter-rouge">kylin.dictionary.mr-hive.columns=USER_ACTION_PV_ID,USER_ACTION_USER_ID</code>
to enable this feature.</p>
+
+<p>Please donât use hive global dictionary on integer type column, you have
to know that the value will be replaced with encoded integer in flat hive
table. If you have sum/max/min measure on the same column, you will get wrong
result in these measures.</p>
+
+<p>And you should know this feature is conflicted with shrunken global
dictionary(KYLIN-3491) because they fix the same thing in different way.</p>
<h3 id="configuration">Configuration</h3>
+
+<ul>
+ <li><code class="highlighter-rouge">kylin.dictionary.mr-hive.columns</code>
is used to specific which columns need to use Hive-MR dict, should be
<em>TABLE1_COLUMN1,TABLE2_COLUMN2</em>. Better configured in cube level,
default value is empty.</li>
+ <li><code class="highlighter-rouge">kylin.dictionary.mr-hive.database</code>
is used to specific which database Hive-MR dict table located, default value is
<em>default</em>.</li>
+ <li><code class="highlighter-rouge">kylin.hive.union.style</code> Sometime
sql which used to build global dict table may have problem in union syntax, you
may refer to Hive Doc for more detail. The default value is <em>UNION</em>,
using lower version of Hive should change to <em>UNION ALL</em>.</li>
+ <li><code
class="highlighter-rouge">kylin.dictionary.mr-hive.table.suffix</code> is used
to specific suffix of global dict table, default value is
<em>_global_dict</em>.</li>
+</ul>
+
+<hr />
+
+<h2 id="screenshot">Screenshot</h2>
+
+<h4 id="sql-in-new-added-step-add-countdistinctbitmap-measure">SQL in new
added step Add count_distinct(bitmap) measure</h4>
+
+<p><img src="/images/Hive-Global-Dictionary/cube-level-config.png"
alt="add_count_distinct_bitmap" /></p>
+
+<h4 id="sql-in-new-added-step-set-hive-dict-column-in-cube-level-config">SQL
in new added step Set hive-dict-column in cube level config</h4>
+
+<p><img src="/images/Hive-Global-Dictionary/set-hive-dict-column.png"
alt="set-hive-dict-column" /></p>
+
+<h4 id="sql-in-new-added-step-three-added-steps-of-cubing-job">SQL in new
added step Three added steps of cubing job</h4>
+
+<p><img src="/images/Hive-Global-Dictionary/three-added-steps.png"
alt="three-added-steps" /></p>
+
+<h4 id="sql-in-new-added-step-hive-global-dictionary-table">SQL in new added
step Hive Global Dictionary Table</h4>
+
+<p><img src="/images/Hive-Global-Dictionary/hive-global-dict-table.png"
alt="hive-global-dict-table" /></p>
+
+<h4 id="sql-in-new-added-step">SQL in new added step</h4>
+
+<ul>
+ <li>Global Dict Mr/Hive extract dict_val from Data</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-groff"
data-lang="groff">CREATE TABLE IF NOT EXISTS
lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
+ ( dict_key STRING COMMENT '',
+ dict_val INT COMMENT ''
+ )
+ COMMENT ''
+ PARTITIONED BY (dict_column string)
+ STORED AS TEXTFILE;
+ DROP TABLE IF EXISTS
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by;
+ CREATE TABLE IF NOT EXISTS
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by
+ (
+ dict_key STRING COMMENT ''
+ )
+ COMMENT ''
+ PARTITIONED BY (dict_column string)
+ STORED AS SEQUENCEFILE
+ ;
+ INSERT OVERWRITE TABLE
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by
+ PARTITION (dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME')
+ SELECT
+ KYLIN_SALES_LSTG_FORMAT_NAME
+ FROM
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
+ GROUP BY KYLIN_SALES_LSTG_FORMAT_NAME
+ ;
+ INSERT OVERWRITE TABLE
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by
+ PARTITION (dict_column = 'KYLIN_SALES_OPS_REGION')
+ SELECT
+ KYLIN_SALES_OPS_REGION
+ FROM
kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
+ GROUP BY KYLIN_SALES_OPS_REGION ;</code></pre></div>
+
+<ul>
+ <li>Global Dict Mr/Hive build dict_val</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-groff"
data-lang="groff">INSERT OVERWRITE TABLE lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
+ PARTITION (dict_column = 'KYLIN_SALES_OPS_REGION')
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
+ WHERE dict_column = 'KYLIN_SALES_OPS_REGION'
+ UNION ALL
+ SELECT a.dict_key as dict_key, (row_number() over(order by a.dict_key
asc)) + (0) as dict_val
+ FROM
+ (
+ SELECT dict_key FROM
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by
WHERE dict_column = 'KYLIN_SALES_OPS_REGION' AND dict_key is not null
+ ) a
+ LEFT JOIN
+ (
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
WHERE dict_column = 'KYLIN_SALES_OPS_REGION'
+ ) b
+ ON a.dict_key = b.dict_key
+ WHERE b.dict_val is null;
+
+ INSERT OVERWRITE TABLE lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
+ PARTITION (dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME')
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
+ WHERE dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME'
+ UNION ALL
+ SELECT a.dict_key as dict_key, (row_number() over(order by a.dict_key
asc)) + (0) as dict_val
+ FROM
+ (
+ SELECT dict_key FROM
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195__group_by
WHERE dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME' AND dict_key is not null
+ ) a
+ LEFT JOIN
+ (
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
WHERE dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME'
+ ) b
+ ON a.dict_key = b.dict_key
+ WHERE b.dict_val is null;</code></pre></div>
+
+<ul>
+ <li>Global Dict Mr/Hive replace dict_val to Data</li>
+</ul>
+
+<div class="highlight"><pre><code class="language-groff"
data-lang="groff">INSERT OVERWRITE TABLE
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
+ SELECT
+ a.KYLIN_SALES_TRANS_ID
+ ,a.KYLIN_SALES_PART_DT
+ ,a.KYLIN_SALES_LEAF_CATEG_ID
+ ,a.KYLIN_SALES_LSTG_SITE_ID
+ ,a.KYLIN_SALES_SELLER_ID
+ ,a.KYLIN_SALES_BUYER_ID
+ ,a.BUYER_ACCOUNT_ACCOUNT_COUNTRY
+ ,a.SELLER_ACCOUNT_ACCOUNT_COUNTRY
+ ,a.KYLIN_SALES_PRICE
+ ,a.KYLIN_SALES_ITEM_COUNT
+ ,a.KYLIN_SALES_LSTG_FORMAT_NAME
+ ,b. dict_val
+ FROM
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
a
+ LEFT OUTER JOIN
+ (
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
WHERE dict_column = 'KYLIN_SALES_OPS_REGION'
+ ) b
+ ON a.KYLIN_SALES_OPS_REGION = b.dict_key;
+ INSERT OVERWRITE TABLE
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
+ SELECT
+ a.KYLIN_SALES_TRANS_ID
+ ,a.KYLIN_SALES_PART_DT
+ ,a.KYLIN_SALES_LEAF_CATEG_ID
+ ,a.KYLIN_SALES_LSTG_SITE_ID
+ ,a.KYLIN_SALES_SELLER_ID
+ ,a.KYLIN_SALES_BUYER_ID
+ ,a.BUYER_ACCOUNT_ACCOUNT_COUNTRY
+ ,a.SELLER_ACCOUNT_ACCOUNT_COUNTRY
+ ,a.KYLIN_SALES_PRICE
+ ,a.KYLIN_SALES_ITEM_COUNT
+ ,b. dict_val
+ ,a.KYLIN_SALES_OPS_REGION
+ FROM
default.kylin_intermediate_kylin_sale_hive_dict_921b0a15_d7cd_a2e6_6852_4ce44158f195
a
+ LEFT OUTER JOIN
+ (
+ SELECT dict_key, dict_val FROM lacus.KYLIN_SALE_HIVE_DICT_HIVE_GLOBAL
WHERE dict_column = 'KYLIN_SALES_LSTG_FORMAT_NAME'
+ ) b
+ ON a.KYLIN_SALES_LSTG_FORMAT_NAME = b.dict_key;</code></pre></div>
+
+<h3 id="reference-link">Reference Link</h3>
+
<ul>
- <li><code class="highlighter-rouge">kylin.dictionary.mr-hive.columns</code>
is used to specific which columns need to be Hive-MR dict.</li>
- <li><code class="highlighter-rouge">kylin.dictionary.mr-hive.database</code>
is used to specific which database Hive-MR dict located.</li>
- <li><code class="highlighter-rouge">kylin.hive.union.style</code> Sometime
sql which used to build global dict table may have syntax problem. This should
be fixed by specific this entry with <em>UNION ALL</em>.</li>
- <li><code
class="highlighter-rouge">kylin.dictionary.mr-hive.table.suffix</code> is used
to specific suffix of global dict table.</li>
+ <li>https://issues.apache.org/jira/browse/KYLIN-3491</li>
+ <li>https://issues.apache.org/jira/browse/KYLIN-3841</li>
+ <li>https://issues.apache.org/jira/browse/KYLIN-3905</li>
+
<li>https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Union</li>
+ <li>http://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/</li>
</ul>
</article>
Modified: kylin/site/feed.xml
URL:
http://svn.apache.org/viewvc/kylin/site/feed.xml?rev=1864708&r1=1864707&r2=1864708&view=diff
==============================================================================
--- kylin/site/feed.xml (original)
+++ kylin/site/feed.xml Thu Aug 8 14:10:46 2019
@@ -19,8 +19,8 @@
<description>Apache Kylin Home</description>
<link>http://kylin.apache.org/</link>
<atom:link href="http://kylin.apache.org/feed.xml"; rel="self"
type="application/rss+xml"/>
- <pubDate>Thu, 01 Aug 2019 06:52:56 -0700</pubDate>
- <lastBuildDate>Thu, 01 Aug 2019 06:52:56 -0700</lastBuildDate>
+ <pubDate>Thu, 08 Aug 2019 06:59:20 -0700</pubDate>
+ <lastBuildDate>Thu, 08 Aug 2019 06:59:20 -0700</lastBuildDate>
<generator>Jekyll v2.5.3</generator>
<item>
@@ -1191,114 +1191,114 @@ Security: (depend on your security setti
</item>
<item>
- <title>Apache Kylin v3.0.0-alpha åå¸</title>
- <description><p>è¿æ¥ Apache Kylin 社åºå¾é«å
´å°å®£å¸ï¼Apache Kylin v3.0.0-alpha æ£å¼åå¸ã</p>
+ <title>Apache Kylin v3.0.0-alpha Release Announcement</title>
+ <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v3.0.0-alpha.</p>
-<p>Apache Kylin
æ¯ä¸ä¸ªå¼æºçåå¸å¼åæå¼æï¼æ¨å¨ä¸ºæ大æ°æ®éæä¾ SQL
æ¥å£åå¤ç»´åæï¼OLAPï¼çè½åã</p>
+<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Big Data
supporting extremely large datasets.</p>
-<p>è¿æ¯ Kylin ä¸ä¸ä»£ v3.x
ç第ä¸ä¸ªåå¸çæ¬ï¼ç¨äºæ©æé¢è§ï¼ä¸»è¦çåè½æ¯å®æ¶
ï¼Real-timeï¼ OLAPãå®æ´çæ¹å¨å表请åè§<a
href="/docs/release_notes.html">release
notes</a>ï¼è¿éæä¸äºä¸»è¦æ¹è¿å说æã</p>
+<p>This is the first release of the new generation v3.x, the main
feature introduced is the Real-time OLAP. All of the changes can be found in
the <a href="/docs/release_notes.html">release
notes</a>. Here we just highlight the main features.</p>
-<h1 id="section">éè¦æ°åè½</h1>
+<h1 id="important-features">Important features</h1>
-<h3 id="kylin-3654----olap">KYLIN-3654 - å®æ¶ OLAP</h3>
-<p>éçå¼å
¥æ°ç real-time receiver å coordinator ç»ä»¶ï¼Kylin
è½å¤å®ç°æ¯«ç§çº§å«çæ°æ®åå¤å»¶è¿ï¼æ°æ®æºæ¥èªæµå¼æ°æ®å¦
Apache Kafkaãè¿æå³çï¼ä» v3.0 å¼å§ï¼Kylin
æ¢è½å¤æ¯æåå²æ¹éæ°æ®ç
OLAPï¼ä¹æ¯æ对æµå¼æ°æ®çåå®æ¶ï¼Near real-timeï¼ä»¥åå®å
¨å®æ¶(real-time)åæãç¨æ·å¯ä»¥ä½¿ç¨ä¸ä¸ª OLAP
å¹³å°æ¥æå¡ä¸åç使ç¨åºæ¯ãæ¤æ¹æ¡å·²ç»å¨æ©æç¨æ·å¦ eBay
å¾å°é¨ç½²åéªè¯ãå
³äºå¦ä½ä½¿ç¨æ¤åè½ï¼è¯·åè<a
href="/docs30/tutorial/realtime_olap.html">æ¤æ
ç¨</a>ã</p>
+<h3 id="kylin-3654---real-time-olap">KYLIN-3654 - Real-time
OLAP</h3>
+<p>With the newly introduced Kylin real-time receiver and coordinator
components, Kylin can implement a millisecond-level data preparation delay for
streaming data from sources like Apache Kafka. This means since v3.0 on, Kylin
can support sub-second level OLAP over historical batch data, near real-time
streaming as well as real-time streaming. The user can use one OLAP platform to
serve different scenarios. This solution has been deployed and verified in
early adopters like eBay since 2018. For how to enable it, please refer to
<a href="/docs30/tutorial/realtime_olap.html">this
tutorial</a>.</p>
-<h3 id="kylin-3795----apache-livy--spark-">KYLIN-3795 - éè¿
Apache Livy é交 Spark ä»»å¡</h3>
-<p>è¿ä¸ªåè½å
许管çå为 Kylin é
ç½®ä½¿ç¨ Apache Livy
(incubating) æ¥å®æä»»å¡çé交ãSpark ä½ä¸çæ交éè¿ Livy ç
REST API æ¥æ交ï¼èæ éå¨æ¬å°å¯å¨ Spark Driver
è¿ç¨ï¼ä»èæ¹ä¾¿å¯¹ Spark èµæºç管ççæ§ï¼åæ¶ä¹éä½å¯¹ Kylin
ä»»å¡è¿ç¨æå¨èç¹çååã</p>
+<h3
id="kylin-3795---submit-spark-jobs-via-apache-livy">KYLIN-3795 -
Submit Spark jobs via Apache Livy</h3>
+<p>This feature allows the administrator to configure Kylin to integrate
with Apache Livy (incubating) for Spark job submissions. The Spark job is
submitted to the Livy Server through Livyâs REST API, instead of starting the
Spark Driver process in local, which facilitates the management and monitoring
of the Spark resources, and also releases the pressure of the nodes where the
Kylin job server is running.</p>
-<h3 id="kylin-3820----curator-">KYLIN-3820 - åºäº Curator
çä»»å¡èç¹åé
åæå¡åç°</h3>
-<p>æ°å¢ä¸ç§åºäºApache Zookeeper å
Curatorä½ä¸è°åº¦å¨ï¼å¯ä»¥èªå¨åç° Kylin èç¹ï¼å¹¶èªå¨åé
ä¸ä¸ªèç¹æ¥è¿è¡ä»»å¡ç管ç以åæ
éæ¢å¤ãæäºè¿ä¸ªåè½åï¼ç®¡çåå¯ä»¥æ´å
容æå°é¨ç½²åæ©å± Kylin èç¹ï¼èä¸åéè¦å¨ <code
class="highlighter-rouge">kylin.properties</code> ä¸é
ç½®æ¯ä¸ª Kylin èç¹çå°å并éå¯ Kylin 以使ä¹çæã</p>
+<h3 id="kylin-3820---a-curator-based-job-scheduler">KYLIN-3820
- A curator-based job scheduler</h3>
+<p>A new job scheduler is added to automatically discover the Kylin
nodes and do an automatic leader selection among them (only the leader will
submit jobs). With this feature, you can easily deploy and scale out Kylin
nodes without manually update the node address in <code
class="highlighter-rouge">kylin.properties</code> and
restart Kylin to take effective.</p>
-<h1 id="section-1">å
¶å®æ¹è¿</h1>
+<h1 id="other-enhancements">Other enhancements</h1>
-<h3 id="kylin-3716---fastthreadlocal--threadlocal">KYLIN-3716
- FastThreadLocal æ¿æ¢ ThreadLocal</h3>
-<p>ä½¿ç¨ Netty ä¸ç FastThreadLocal æ¿ä»£ JDK åçç
ThreadLocalï¼å¯ä»¥ä¸å®ç¨åº¦ä¸æå Kylin
å¨é«å¹¶åä¸çæ§è½ã</p>
+<h3
id="kylin-3716---fastthreadlocal-replaces-threadlocal">KYLIN-3716
- FastThreadLocal replaces ThreadLocal</h3>
+<p>Using FastThreadLocal instead of ThreadLocal can improve Kylinâs
overall performance to some extent.</p>
<h3
id="kylin-3867---enable-jdbc-to-use-key-store--trust-store-for-https-connection">KYLIN-3867
- Enable JDBC to use key store &amp; trust store for https
connection</h3>
-<p>éè¿ä½¿ç¨HTTPSï¼ä¿æ¤äºJDBC使ç¨ç身份éªè¯ä¿¡æ¯ï¼ä½¿å¾Kylinæ´å
å®å
¨</p>
+<p>By using HTTPS, the authentication information used by JDBC is
protected, making Kylin more secure.</p>
<h3
id="kylin-3905---enable-shrunken-dictionary-default">KYLIN-3905 -
Enable shrunken dictionary default</h3>
-<p>é»è®¤å¼å¯ shrunken
dictionaryï¼é对é«åºç»´è¿è¡ç²¾ç¡®å»éçåºæ¯ï¼å¯ä»¥æ¾èåå°æ建ç¨æ¶ã</p>
+<p>By default, the shrunken dictionary is enabled, and the precise
counting scene for high cardinal dimensions can significantly reduce the build
time.</p>
<h3
id="kylin-3839---storage-clean-up-after-the-refreshing-and-deleting-a-segment">KYLIN-3839
- Storage clean up after the refreshing and deleting a segment</h3>
-<p>æ´å åæ¶å°æ¸
é¤ä¸å¿
è¦çæ°æ®æ件</p>
+<p>Clear unnecessary data files in a timely manner</p>
-<p><strong>ä¸è½½</strong></p>
+<p><strong>Download</strong></p>
-<p>è¦ä¸è½½Apache Kylin æºä»£ç æäºè¿å¶å
ï¼è¯·è®¿é®<a
href="/download">ä¸è½½é¡µé¢</a> page.</p>
+<p>To download Apache Kylin v3.0.0-alpha source code or binary package,
visit the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
-<p><strong>å级</strong></p>
+<p><strong>Upgrade</strong></p>
-<p>åè<a
href="/docs/howto/howto_upgrade.html">å级æå</a>.</p>
+<p>Follow the <a
href="/docs/howto/howto_upgrade.html">upgrade
guide</a>.</p>
-<p><strong>åé¦</strong></p>
+<p><strong>Feedback</strong></p>
-<p>å¦ææ¨éå°é®é¢æçé®ï¼è¯·åéé®ä»¶è³ Apache Kylin dev
æ user é®ä»¶å表ï¼d...@kylin.apache.orgï¼u...@kylin.apache.org;
å¨åéä¹åï¼è¯·ç¡®ä¿æ¨å·²éè¿åéçµåé®ä»¶è³
dev-subscr...@kylin.apache.org æ user-subscr...@kylin.apache.org 订é
äºé®ä»¶å表ã</p>
+<p>If you face issue or question, please send mail to Apache Kylin dev
or user mailing list:Â d...@kylin.apache.org , u...@kylin.apache.org; Before
sending, please make sure you have subscribed the mailing list by dropping an
email to dev-subscr...@kylin.apache.org or
user-subscr...@kylin.apache.org.</p>
-<p><em>é常æè°¢ææè´¡ç®Apache
Kylinçæå!</em></p>
+<p><em>Great thanks to everyone who
contributed!</em></p>
</description>
<pubDate>Fri, 19 Apr 2019 13:00:00 -0700</pubDate>
-
<link>http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</link>
- <guid
isPermaLink="true">http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</guid>
+
<link>http://kylin.apache.org/blog/2019/04/19/release-v3.0.0-alpha/</link>
+ <guid
isPermaLink="true">http://kylin.apache.org/blog/2019/04/19/release-v3.0.0-alpha/</guid>
- <category>cn_blog</category>
+ <category>blog</category>
</item>
<item>
- <title>Apache Kylin v3.0.0-alpha Release Announcement</title>
- <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v3.0.0-alpha.</p>
+ <title>Apache Kylin v3.0.0-alpha åå¸</title>
+ <description><p>è¿æ¥ Apache Kylin 社åºå¾é«å
´å°å®£å¸ï¼Apache Kylin v3.0.0-alpha æ£å¼åå¸ã</p>
-<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Big Data
supporting extremely large datasets.</p>
+<p>Apache Kylin
æ¯ä¸ä¸ªå¼æºçåå¸å¼åæå¼æï¼æ¨å¨ä¸ºæ大æ°æ®éæä¾ SQL
æ¥å£åå¤ç»´åæï¼OLAPï¼çè½åã</p>
-<p>This is the first release of the new generation v3.x, the main
feature introduced is the Real-time OLAP. All of the changes can be found in
the <a href="/docs/release_notes.html">release
notes</a>. Here we just highlight the main features.</p>
+<p>è¿æ¯ Kylin ä¸ä¸ä»£ v3.x
ç第ä¸ä¸ªåå¸çæ¬ï¼ç¨äºæ©æé¢è§ï¼ä¸»è¦çåè½æ¯å®æ¶
ï¼Real-timeï¼ OLAPãå®æ´çæ¹å¨å表请åè§<a
href="/docs/release_notes.html">release
notes</a>ï¼è¿éæä¸äºä¸»è¦æ¹è¿å说æã</p>
-<h1 id="important-features">Important features</h1>
+<h1 id="section">éè¦æ°åè½</h1>
-<h3 id="kylin-3654---real-time-olap">KYLIN-3654 - Real-time
OLAP</h3>
-<p>With the newly introduced Kylin real-time receiver and coordinator
components, Kylin can implement a millisecond-level data preparation delay for
streaming data from sources like Apache Kafka. This means since v3.0 on, Kylin
can support sub-second level OLAP over historical batch data, near real-time
streaming as well as real-time streaming. The user can use one OLAP platform to
serve different scenarios. This solution has been deployed and verified in
early adopters like eBay since 2018. For how to enable it, please refer to
<a href="/docs30/tutorial/realtime_olap.html">this
tutorial</a>.</p>
+<h3 id="kylin-3654----olap">KYLIN-3654 - å®æ¶ OLAP</h3>
+<p>éçå¼å
¥æ°ç real-time receiver å coordinator ç»ä»¶ï¼Kylin
è½å¤å®ç°æ¯«ç§çº§å«çæ°æ®åå¤å»¶è¿ï¼æ°æ®æºæ¥èªæµå¼æ°æ®å¦
Apache Kafkaãè¿æå³çï¼ä» v3.0 å¼å§ï¼Kylin
æ¢è½å¤æ¯æåå²æ¹éæ°æ®ç
OLAPï¼ä¹æ¯æ对æµå¼æ°æ®çåå®æ¶ï¼Near real-timeï¼ä»¥åå®å
¨å®æ¶(real-time)åæãç¨æ·å¯ä»¥ä½¿ç¨ä¸ä¸ª OLAP
å¹³å°æ¥æå¡ä¸åç使ç¨åºæ¯ãæ¤æ¹æ¡å·²ç»å¨æ©æç¨æ·å¦ eBay
å¾å°é¨ç½²åéªè¯ãå
³äºå¦ä½ä½¿ç¨æ¤åè½ï¼è¯·åè<a
href="/docs30/tutorial/realtime_olap.html">æ¤æ
ç¨</a>ã</p>
-<h3
id="kylin-3795---submit-spark-jobs-via-apache-livy">KYLIN-3795 -
Submit Spark jobs via Apache Livy</h3>
-<p>This feature allows the administrator to configure Kylin to integrate
with Apache Livy (incubating) for Spark job submissions. The Spark job is
submitted to the Livy Server through Livyâs REST API, instead of starting the
Spark Driver process in local, which facilitates the management and monitoring
of the Spark resources, and also releases the pressure of the nodes where the
Kylin job server is running.</p>
+<h3 id="kylin-3795----apache-livy--spark-">KYLIN-3795 - éè¿
Apache Livy é交 Spark ä»»å¡</h3>
+<p>è¿ä¸ªåè½å
许管çå为 Kylin é
ç½®ä½¿ç¨ Apache Livy
(incubating) æ¥å®æä»»å¡çé交ãSpark ä½ä¸çæ交éè¿ Livy ç
REST API æ¥æ交ï¼èæ éå¨æ¬å°å¯å¨ Spark Driver
è¿ç¨ï¼ä»èæ¹ä¾¿å¯¹ Spark èµæºç管ççæ§ï¼åæ¶ä¹éä½å¯¹ Kylin
ä»»å¡è¿ç¨æå¨èç¹çååã</p>
-<h3 id="kylin-3820---a-curator-based-job-scheduler">KYLIN-3820
- A curator-based job scheduler</h3>
-<p>A new job scheduler is added to automatically discover the Kylin
nodes and do an automatic leader selection among them (only the leader will
submit jobs). With this feature, you can easily deploy and scale out Kylin
nodes without manually update the node address in <code
class="highlighter-rouge">kylin.properties</code> and
restart Kylin to take effective.</p>
+<h3 id="kylin-3820----curator-">KYLIN-3820 - åºäº Curator
çä»»å¡èç¹åé
åæå¡åç°</h3>
+<p>æ°å¢ä¸ç§åºäºApache Zookeeper å
Curatorä½ä¸è°åº¦å¨ï¼å¯ä»¥èªå¨åç° Kylin èç¹ï¼å¹¶èªå¨åé
ä¸ä¸ªèç¹æ¥è¿è¡ä»»å¡ç管ç以åæ
éæ¢å¤ãæäºè¿ä¸ªåè½åï¼ç®¡çåå¯ä»¥æ´å
容æå°é¨ç½²åæ©å± Kylin èç¹ï¼èä¸åéè¦å¨ <code
class="highlighter-rouge">kylin.properties</code> ä¸é
ç½®æ¯ä¸ª Kylin èç¹çå°å并éå¯ Kylin 以使ä¹çæã</p>
-<h1 id="other-enhancements">Other enhancements</h1>
+<h1 id="section-1">å
¶å®æ¹è¿</h1>
-<h3
id="kylin-3716---fastthreadlocal-replaces-threadlocal">KYLIN-3716
- FastThreadLocal replaces ThreadLocal</h3>
-<p>Using FastThreadLocal instead of ThreadLocal can improve Kylinâs
overall performance to some extent.</p>
+<h3 id="kylin-3716---fastthreadlocal--threadlocal">KYLIN-3716
- FastThreadLocal æ¿æ¢ ThreadLocal</h3>
+<p>ä½¿ç¨ Netty ä¸ç FastThreadLocal æ¿ä»£ JDK åçç
ThreadLocalï¼å¯ä»¥ä¸å®ç¨åº¦ä¸æå Kylin
å¨é«å¹¶åä¸çæ§è½ã</p>
<h3
id="kylin-3867---enable-jdbc-to-use-key-store--trust-store-for-https-connection">KYLIN-3867
- Enable JDBC to use key store &amp; trust store for https
connection</h3>
-<p>By using HTTPS, the authentication information used by JDBC is
protected, making Kylin more secure.</p>
+<p>éè¿ä½¿ç¨HTTPSï¼ä¿æ¤äºJDBC使ç¨ç身份éªè¯ä¿¡æ¯ï¼ä½¿å¾Kylinæ´å
å®å
¨</p>
<h3
id="kylin-3905---enable-shrunken-dictionary-default">KYLIN-3905 -
Enable shrunken dictionary default</h3>
-<p>By default, the shrunken dictionary is enabled, and the precise
counting scene for high cardinal dimensions can significantly reduce the build
time.</p>
+<p>é»è®¤å¼å¯ shrunken
dictionaryï¼é对é«åºç»´è¿è¡ç²¾ç¡®å»éçåºæ¯ï¼å¯ä»¥æ¾èåå°æ建ç¨æ¶ã</p>
<h3
id="kylin-3839---storage-clean-up-after-the-refreshing-and-deleting-a-segment">KYLIN-3839
- Storage clean up after the refreshing and deleting a segment</h3>
-<p>Clear unnecessary data files in a timely manner</p>
+<p>æ´å åæ¶å°æ¸
é¤ä¸å¿
è¦çæ°æ®æ件</p>
-<p><strong>Download</strong></p>
+<p><strong>ä¸è½½</strong></p>
-<p>To download Apache Kylin v3.0.0-alpha source code or binary package,
visit the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
+<p>è¦ä¸è½½Apache Kylin æºä»£ç æäºè¿å¶å
ï¼è¯·è®¿é®<a
href="/download">ä¸è½½é¡µé¢</a> page.</p>
-<p><strong>Upgrade</strong></p>
+<p><strong>å级</strong></p>
-<p>Follow the <a
href="/docs/howto/howto_upgrade.html">upgrade
guide</a>.</p>
+<p>åè<a
href="/docs/howto/howto_upgrade.html">å级æå</a>.</p>
-<p><strong>Feedback</strong></p>
+<p><strong>åé¦</strong></p>
-<p>If you face issue or question, please send mail to Apache Kylin dev
or user mailing list:Â d...@kylin.apache.org , u...@kylin.apache.org; Before
sending, please make sure you have subscribed the mailing list by dropping an
email to dev-subscr...@kylin.apache.org or
user-subscr...@kylin.apache.org.</p>
+<p>å¦ææ¨éå°é®é¢æçé®ï¼è¯·åéé®ä»¶è³ Apache Kylin dev
æ user é®ä»¶å表ï¼d...@kylin.apache.orgï¼u...@kylin.apache.org;
å¨åéä¹åï¼è¯·ç¡®ä¿æ¨å·²éè¿åéçµåé®ä»¶è³
dev-subscr...@kylin.apache.org æ user-subscr...@kylin.apache.org 订é
äºé®ä»¶å表ã</p>
-<p><em>Great thanks to everyone who
contributed!</em></p>
+<p><em>é常æè°¢ææè´¡ç®Apache
Kylinçæå!</em></p>
</description>
<pubDate>Fri, 19 Apr 2019 13:00:00 -0700</pubDate>
-
<link>http://kylin.apache.org/blog/2019/04/19/release-v3.0.0-alpha/</link>
- <guid
isPermaLink="true">http://kylin.apache.org/blog/2019/04/19/release-v3.0.0-alpha/</guid>
+
<link>http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</link>
+ <guid
isPermaLink="true">http://kylin.apache.org/cn_blog/2019/04/19/release-v3.0.0-alpha/</guid>
- <category>blog</category>
+ <category>cn_blog</category>
</item>
@@ -1455,84 +1455,6 @@ The checkpoint info is the smallest part
</item>
<item>
- <title>Apache Kylin v2.6.0 Release Announcement</title>
- <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v2.6.0.</p>
-
-<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Big Data
supporting extremely large datasets.</p>
-
-<p>This is a major release after 2.5.0, including many enhancements. All
of the changes can be found in the <a
href="https://kylin.apache.org/docs/release_notes.html">release
notes</a>. Here just highlight the major ones:</p>
-
-<h3 id="sdk-for-jdbc-sources">SDK for JDBC sources</h3>
-<p>Apache Kylin has already supported several data sources like Amazon
Redshift, SQL Server through JDBC. <br />
-To help developers handle SQL dialect differences and easily implement a new
data source through JDBC, Kylin provides a new data source SDK with APIs
for:<br />
-* Synchronize metadata and data from JDBC source<br />
-* Build cube from JDBC source<br />
-* Query pushdown to JDBC source engine when cube is unmatched</p>
-
-<p>Check KYLIN-3552 for more.</p>
-
-<h3 id="memcached-as-distributed-cache">Memcached as
distributed cache</h3>
-<p>In the past, query caches are not efficiently used in Kylin due to
two aspects: aggressive cache expiration strategy and local cache. <br />
-Because of the aggressive cache expiration strategy, useful caches are often
cleaned up unnecessarily. <br />
-Because query caches are stored in local servers, they cannot be shared
between servers. <br />
-And because of the size limitation of local cache, not all useful query
results can be cached.</p>
-
-<p>To deal with these shortcomings, we change the query cache expiration
strategy by signature checking and introduce the memcached as Kylinâs
distributed cache so that Kylin servers are able to share cache between
servers. <br />
-And itâs easy to add memcached servers to scale out distributed cache. With
enough memcached servers, we can cached things as much as possible. <br />
-Then we also introduce segment level query cache which can not only speed up
query but also reduce the rpcs to HBase. <br />
-The related tasks are KYLIN-2895, KYLIN-2894, KYLIN-2896, KYLIN-2897,
KYLIN-2898, KYLIN-2899.</p>
-
-<h3 id="forkjoinpool-for-fast-cubing">ForkJoinPool for fast
cubing</h3>
-<p>In the past, fast cubing uses split threads, task threads and main
thread to do the cube building, there is complex join and error handling
logic.</p>
-
-<p>The new implement leverages the ForkJoinPool from JDK, the event
split logic is handled in<br />
-main thread. Cuboid task and sub-tasks are handled in fork join pool, cube
results are collected<br />
-async and can be write to output earlier. Check KYLIN-2932 for more.</p>
-
-<h3 id="improve-hllcounter-performance">Improve HLLCounter
performance</h3>
-<p>In the past, the way to create HLLCounter and to compute harmonic
mean are not efficient.</p>
-
-<p>The new implement improve the HLLCounter creation by copy register
from another HLLCounter instead of merge. To compute harmonic mean in the
HLLCSnapshot, it does the enhancement by <br />
-* using table to cache all 1/2^r without computing on the fly<br />
-* remove floating addition by using integer addition in the bigger loop<br
/>
-* remove branch, e.g. neednât checking whether registers[i] is zero or not,
although this is minor improvement.</p>
-
-<p>Check KYLIN-3656 for more.</p>
-
-<h3 id="improve-cuboid-recommendation-algorithm">Improve
Cuboid Recommendation Algorithm</h3>
-<p>In the past, to add cuboids which are not prebuilt, the cube planner
turns to mandatory cuboids which are selected if its rollup row count is above
some threshold. <br />
-There are two shortcomings:<br />
-* The way to estimate the rollup row count is not good<br />
-* Itâs hard to determine the threshold of rollup row count for recommending
mandatory cuboids</p>
-
-<p>The new implement improves the way to estimate the row count of
un-prebuilt cuboids by rollup ratio rather than exact rollup row count. <br
/>
-With better estimated row counts for un-prebuilt cuboids, the cost-based cube
planner algorithm will decide which cuboid to be built or not and the threshold
for previous mandatory cuboids is not needed. <br />
-By this improvement, we donât need the threshold for mandatory cuboids
recommendation, and mandatory cuboids can only be manually set and will not be
recommended. Check KYLIN-3540 for more.</p>
-
-<p><strong>Download</strong></p>
-
-<p>To download Apache Kylin v2.6.0 source code or binary package, visit
the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
-
-<p><strong>Upgrade</strong></p>
-
-<p>Follow the <a
href="/docs/howto/howto_upgrade.html">upgrade
guide</a>.</p>
-
-<p><strong>Feedback</strong></p>
-
-<p>If you face issue or question, please send mail to Apache Kylin dev
or user mailing list:Â d...@kylin.apache.org , u...@kylin.apache.org; Before
sending, please make sure you have subscribed the mailing list by dropping an
email to dev-subscr...@kylin.apache.org or
user-subscr...@kylin.apache.org.</p>
-
-<p><em>Great thanks to everyone who
contributed!</em></p>
-</description>
- <pubDate>Fri, 18 Jan 2019 12:00:00 -0800</pubDate>
- <link>http://kylin.apache.org/blog/2019/01/18/release-v2.6.0/</link>
- <guid
isPermaLink="true">http://kylin.apache.org/blog/2019/01/18/release-v2.6.0/</guid>
-
-
- <category>blog</category>
-
- </item>
-
- <item>
<title>Apache Kylin v2.6.0 æ£å¼åå¸</title>
<description><p>è¿æ¥Apache Kylin 社åºå¾é«å
´å°å®£å¸ï¼Apache Kylin 2.6.0 æ£å¼åå¸ã</p>
@@ -1611,5 +1533,83 @@ By this improvement, we donât need
</item>
+ <item>
+ <title>Apache Kylin v2.6.0 Release Announcement</title>
+ <description><p>The Apache Kylin community is pleased to
announce the release of Apache Kylin v2.6.0.</p>
+
+<p>Apache Kylin is an open source Distributed Analytics Engine designed
to provide SQL interface and multi-dimensional analysis (OLAP) on Big Data
supporting extremely large datasets.</p>
+
+<p>This is a major release after 2.5.0, including many enhancements. All
of the changes can be found in the <a
href="https://kylin.apache.org/docs/release_notes.html">release
notes</a>. Here just highlight the major ones:</p>
+
+<h3 id="sdk-for-jdbc-sources">SDK for JDBC sources</h3>
+<p>Apache Kylin has already supported several data sources like Amazon
Redshift, SQL Server through JDBC. <br />
+To help developers handle SQL dialect differences and easily implement a new
data source through JDBC, Kylin provides a new data source SDK with APIs
for:<br />
+* Synchronize metadata and data from JDBC source<br />
+* Build cube from JDBC source<br />
+* Query pushdown to JDBC source engine when cube is unmatched</p>
+
+<p>Check KYLIN-3552 for more.</p>
+
+<h3 id="memcached-as-distributed-cache">Memcached as
distributed cache</h3>
+<p>In the past, query caches are not efficiently used in Kylin due to
two aspects: aggressive cache expiration strategy and local cache. <br />
+Because of the aggressive cache expiration strategy, useful caches are often
cleaned up unnecessarily. <br />
+Because query caches are stored in local servers, they cannot be shared
between servers. <br />
+And because of the size limitation of local cache, not all useful query
results can be cached.</p>
+
+<p>To deal with these shortcomings, we change the query cache expiration
strategy by signature checking and introduce the memcached as Kylinâs
distributed cache so that Kylin servers are able to share cache between
servers. <br />
+And itâs easy to add memcached servers to scale out distributed cache. With
enough memcached servers, we can cached things as much as possible. <br />
+Then we also introduce segment level query cache which can not only speed up
query but also reduce the rpcs to HBase. <br />
+The related tasks are KYLIN-2895, KYLIN-2894, KYLIN-2896, KYLIN-2897,
KYLIN-2898, KYLIN-2899.</p>
+
+<h3 id="forkjoinpool-for-fast-cubing">ForkJoinPool for fast
cubing</h3>
+<p>In the past, fast cubing uses split threads, task threads and main
thread to do the cube building, there is complex join and error handling
logic.</p>
+
+<p>The new implement leverages the ForkJoinPool from JDK, the event
split logic is handled in<br />
+main thread. Cuboid task and sub-tasks are handled in fork join pool, cube
results are collected<br />
+async and can be write to output earlier. Check KYLIN-2932 for more.</p>
+
+<h3 id="improve-hllcounter-performance">Improve HLLCounter
performance</h3>
+<p>In the past, the way to create HLLCounter and to compute harmonic
mean are not efficient.</p>
+
+<p>The new implement improve the HLLCounter creation by copy register
from another HLLCounter instead of merge. To compute harmonic mean in the
HLLCSnapshot, it does the enhancement by <br />
+* using table to cache all 1/2^r without computing on the fly<br />
+* remove floating addition by using integer addition in the bigger loop<br
/>
+* remove branch, e.g. neednât checking whether registers[i] is zero or not,
although this is minor improvement.</p>
+
+<p>Check KYLIN-3656 for more.</p>
+
+<h3 id="improve-cuboid-recommendation-algorithm">Improve
Cuboid Recommendation Algorithm</h3>
+<p>In the past, to add cuboids which are not prebuilt, the cube planner
turns to mandatory cuboids which are selected if its rollup row count is above
some threshold. <br />
+There are two shortcomings:<br />
+* The way to estimate the rollup row count is not good<br />
+* Itâs hard to determine the threshold of rollup row count for recommending
mandatory cuboids</p>
+
+<p>The new implement improves the way to estimate the row count of
un-prebuilt cuboids by rollup ratio rather than exact rollup row count. <br
/>
+With better estimated row counts for un-prebuilt cuboids, the cost-based cube
planner algorithm will decide which cuboid to be built or not and the threshold
for previous mandatory cuboids is not needed. <br />
+By this improvement, we donât need the threshold for mandatory cuboids
recommendation, and mandatory cuboids can only be manually set and will not be
recommended. Check KYLIN-3540 for more.</p>
+
+<p><strong>Download</strong></p>
+
+<p>To download Apache Kylin v2.6.0 source code or binary package, visit
the <a
href="http://kylin.apache.org/download">download</a>
page.</p>
+
+<p><strong>Upgrade</strong></p>
+
+<p>Follow the <a
href="/docs/howto/howto_upgrade.html">upgrade
guide</a>.</p>
+
+<p><strong>Feedback</strong></p>
+
+<p>If you face issue or question, please send mail to Apache Kylin dev
or user mailing list:Â d...@kylin.apache.org , u...@kylin.apache.org; Before
sending, please make sure you have subscribed the mailing list by dropping an
email to dev-subscr...@kylin.apache.org or
user-subscr...@kylin.apache.org.</p>
+
+<p><em>Great thanks to everyone who
contributed!</em></p>
+</description>
+ <pubDate>Fri, 18 Jan 2019 12:00:00 -0800</pubDate>
+ <link>http://kylin.apache.org/blog/2019/01/18/release-v2.6.0/</link>
+ <guid
isPermaLink="true">http://kylin.apache.org/blog/2019/01/18/release-v2.6.0/</guid>
+
+
+ <category>blog</category>
+
+ </item>
+
</channel>
</rss>
Added: kylin/site/images/Hive-Global-Dictionary/cube-level-config.png
URL:
http://svn.apache.org/viewvc/kylin/site/images/Hive-Global-Dictionary/cube-level-config.png?rev=1864708&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/Hive-Global-Dictionary/cube-level-config.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: kylin/site/images/Hive-Global-Dictionary/hive-global-dict-table.png
URL:
http://svn.apache.org/viewvc/kylin/site/images/Hive-Global-Dictionary/hive-global-dict-table.png?rev=1864708&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/Hive-Global-Dictionary/hive-global-dict-table.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: kylin/site/images/Hive-Global-Dictionary/set-hive-dict-column.png
URL:
http://svn.apache.org/viewvc/kylin/site/images/Hive-Global-Dictionary/set-hive-dict-column.png?rev=1864708&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/Hive-Global-Dictionary/set-hive-dict-column.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: kylin/site/images/Hive-Global-Dictionary/three-added-steps.png
URL:
http://svn.apache.org/viewvc/kylin/site/images/Hive-Global-Dictionary/three-added-steps.png?rev=1864708&view=auto
==============================================================================
Binary file - no diff available.
Propchange: kylin/site/images/Hive-Global-Dictionary/three-added-steps.png
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
