Added: kylin/site/cn_blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/index.html URL: http://svn.apache.org/viewvc/kylin/site/cn_blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/index.html?rev=1891980&view=auto ============================================================================== --- kylin/site/cn_blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/index.html (added) +++ kylin/site/cn_blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/index.html Tue Aug 3 10:55:02 2021 @@ -0,0 +1,9130 @@ +<!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> +<!doctype html> +<html> + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<head> + <meta charset="utf-8"> + <meta http-equiv="X-UA-Compatible" content="IE=edge"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + + <title>Apache Kylin | Kylin å¨ç¾å¢å°åºé¤é¥®çå®è·µåä¼å</title> + <meta name="description" content="ä»2016å¹´å¼å§ï¼ç¾å¢å°åºé¤é¥®ææ¯å¢éå°±å¼å§ä½¿ç¨Apache Kylinä½ä¸ºOLAPå¼æï¼ä½æ¯éçä¸å¡çé«éåå±ï¼å¨æ建åæ¥è¯¢å±é¢é½åºç°äºæçé®é¢ãäºæ¯ï¼ææ¯å¢éä»åç解读å¼å§ï¼ç¶å对è¿ç¨è¿è¡å±å±æ解ï¼å¹¶å¶å®äºç±ç¹åé¢çå®æ½è·¯çº¿ãæ¬ææ»ç»äºä¸äºç»éªåå¿å¾ï¼å¸æè½å¤å¸®å©ä¸çæ´å¤çææ¯å¢éæé«æ°æ®ç产åºæçã"> + <meta name="author" content="Apache Kylin"> + <link rel="shortcut icon" href="fav.png" type="image/png"> + + + +<link rel="stylesheet" href="/assets/css/animate.css"> +<!-- Bootstrap --> +<link rel="stylesheet" href="/assets/css/bootstrap.min.css"> + +<!-- Fonts --> +<!-- <link rel="stylesheet" href="http://fonts.googleapis.com/css?family=Alice|Open+Sans:400,300,700"> --> + +<!-- Icons --> +<link rel="stylesheet" href="/assets/css/font-awesome.min.css"> + + <!-- Custom styles --> + <link rel="stylesheet" href="/assets/css/styles.css"> + <link rel="stylesheet" href="/assets/css/docs.css"> + <link rel="stylesheet" href="/assets/css/pygments.css"> + + <link rel="canonical" href="http://kylin.apache.org/cn_blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/"> + <link rel="alternate" type="application/rss+xml" title="Apache Kylin" href="http://kylin.apache.org/feed.xml" /> + +<!--[if lt IE 9]> <script src="assets/js/html5shiv.js"></script> <![endif]--> +<!-- Global site tag (gtag.js) - Google Analytics --> +<script async src="https://www.googletagmanager.com/gtag/js?id=UA-120788561-1"></script> +<script> + window.dataLayer = window.dataLayer || []; + function gtag(){dataLayer.push(arguments);} + gtag('js', new Date()); + + gtag('config', 'UA-120788561-1'); +</script> +<script type="text/javascript" src="/assets/js/jquery-1.9.1.min.js"></script> +<script type="text/javascript" src="/assets/js/nside.js"></script> </script> +<script type="text/javascript" src="/assets/js/nnav.js"></script> </script> +<script> +var _hmt = _hmt || []; +(function() { + var hm = document.createElement("script"); + hm.src = "https://hm.baidu.com/hm.js?bdc5e03add430c0b72cc0eb91eabfa99"; + var s = document.getElementsByTagName("script")[0]; + s.parentNode.insertBefore(hm, s); +})(); +</script> + +</head> + + <body> + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<header id="header" > + + <!-- Main Menu --> + <nav class="navbar navbar-default" role="navigation" id="nav-wrapper"> + <div class="container-fluid" id="nav"> + <!-- + <img class="img-circle" width="40px" height="40px" id="circlelogo" src="/assets/images/kylin_logo.jpg"> + --> + <!-- Brand and toggle get grouped for better mobile display --> + <div class="navbar-header"> + <img class="navbar-logo" width="46" src="/assets/images/kylin_logo.png" ></img> + <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1"> + <span class="sr-only">Toggle navigation</span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + <span class="icon-bar"></span> + </button> + <ul class="nav icon-navbar"> + <li><a href="https://twitter.com/apachekylin" target="_blank" class="fa fa-twitter fa-lg" title="Twitter: @ApacheKylin" ></a></li> + <li><a href="https://github.com/apache/kylin" target="_blank" class="fa fa-github-alt fa-lg" title="Github: apache/kylin" ></a></li> + <li><a href="https://www.facebook.com/kylinio" target="_blank" class="fa fa-facebook fa-lg" title="Facebook: kylin.io" ></a></li> + </ul> + </div> + + <!-- Collect the nav links, forms, and other content for toggling --> + <div class="navbar-collapse collapse" id="bs-example-navbar-collapse-1"> + + <ul class="nav navbar-nav"> + + <li><a href="/">Home</a></li> + <li> + <a href="/docs">Docs</a> +<!-- + <a href="#" class="dropdown-toggle" data-toggle="dropdown">Docs</a> + <ul class="dropdown-menu"> + + <li class="dropdown-submenu"><a href="#titleGetting Started" data-toggle="collapse" class="navtitle">Getting Started</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/index.html" class="list-group-item-lay pjaxlink">Overview</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/release_notes.html" class="list-group-item-lay pjaxlink">Release Notes</a> + </li> + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/concepts.html" class="list-group-item-lay pjaxlink">Technical Concepts</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/terminology.html" class="list-group-item-lay pjaxlink">Terminology</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/faq.html" class="list-group-item-lay pjaxlink">FAQ</a> + </li> + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/events.html" class="list-group-item-lay pjaxlink">Events and Conferences</a> + </li> + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/best_practices.html" class="list-group-item-lay pjaxlink">Community Best Practices</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/gettingstarted/kylin-quickstart.html" class="list-group-item-lay pjaxlink">Quick Start</a> + </li> + + + + + + + + + <ul> + </li> + + <li class="dropdown-submenu"><a href="#titleInstallation" data-toggle="collapse" class="navtitle">Installation</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/index.html" class="list-group-item-lay pjaxlink">Installation Guide</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/kylin_cluster.html" class="list-group-item-lay pjaxlink">Deploy in Cluster Mode</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/configuration.html" class="list-group-item-lay pjaxlink">Kylin Configuration</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/kylin_aws_emr.html" class="list-group-item-lay pjaxlink">Install Kylin on AWS EMR</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/kylin_docker.html" class="list-group-item-lay pjaxlink">Run Kylin with Docker</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/install/kylin_on_kubernetes.html" class="list-group-item-lay pjaxlink">Deploy Kylin on Kubernetes</a> + </li> + + + + + + + + + <ul> + </li> + + <li class="dropdown-submenu"><a href="#titleTutorial" data-toggle="collapse" class="navtitle">Tutorial</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/kylin_sample.html" class="list-group-item-lay pjaxlink">Quick Start with Sample Cube</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/web.html" class="list-group-item-lay pjaxlink">Web Interface</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/create_cube.html" class="list-group-item-lay pjaxlink">Cube Wizard</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_build_job.html" class="list-group-item-lay pjaxlink">Cube Build and Job Monitoring</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_migration.html" class="list-group-item-lay pjaxlink">Cube Migration</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/sql_reference.html" class="list-group-item-lay pjaxlink">SQL Reference</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/project_level_acl.html" class="list-group-item-lay pjaxlink">Project And Table Level ACL</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_spark.html" class="list-group-item-lay pjaxlink">Build Cube with Spark</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_flink.html" class="list-group-item-lay pjaxlink">Build Cube with Flink</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_streaming.html" class="list-group-item-lay pjaxlink">Cubing from Kafka(Near RT Streaming)</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/realtime_olap.html" class="list-group-item-lay pjaxlink">Real-time OLAP</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/cube_build_performance.html" class="list-group-item-lay pjaxlink">Cube Build Tuning</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/kylin_client_tool.html" class="list-group-item-lay pjaxlink">Kylin Python Client</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/setup_systemcube.html" class="list-group-item-lay pjaxlink">Set Up System Cube</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/use_cube_planner.html" class="list-group-item-lay pjaxlink">Use Cube Planner</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/use_dashboard.html" class="list-group-item-lay pjaxlink">Use Dashboard</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/setup_jdbc_datasource.html" class="list-group-item-lay pjaxlink">Setup JDBC Data Source</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/hybrid.html" class="list-group-item-lay pjaxlink">Hybrid Model</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/mysql_metastore.html" class="list-group-item-lay pjaxlink">Use MySQL as Metastore</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/lambda_mode_and_timezone_realtime_olap.html" class="list-group-item-lay pjaxlink">Lambda mode and Timezone in Real-time OLAP</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/query_pushdown.html" class="list-group-item-lay pjaxlink">Enable Query Pushdown</a> + </li> + + + + + + + + + <ul> + </li> + + <li class="dropdown-submenu"><a href="#titleIntegration" data-toggle="collapse" class="navtitle">Integration</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/odbc.html" class="list-group-item-lay pjaxlink">Kylin ODBC Driver</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/jdbc.html" class="list-group-item-lay pjaxlink">Kylin JDBC Driver</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/tableau.html" class="list-group-item-lay pjaxlink">Tableau 8</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/tableau_91.html" class="list-group-item-lay pjaxlink">Tableau 9</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/powerbi.html" class="list-group-item-lay pjaxlink">MS Excel and Power BI</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/microstrategy.html" class="list-group-item-lay pjaxlink">MicroStrategy</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/squirrel.html" class="list-group-item-lay pjaxlink">SQuirreL</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/flink.html" class="list-group-item-lay pjaxlink">Apache Flink</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/spark.html" class="list-group-item-lay pjaxlink">Apache Spark</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/hue.html" class="list-group-item-lay pjaxlink">Hue</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/Qlik.html" class="list-group-item-lay pjaxlink">Qlik Sense</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/tutorial/superset.html" class="list-group-item-lay pjaxlink">Superset</a> + </li> + + + + + + + + + <ul> + </li> + + <li class="dropdown-submenu"><a href="#titleHow To" data-toggle="collapse" class="navtitle">How To</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_use_restapi.html" class="list-group-item-lay pjaxlink">Use RESTful API</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_build_cube_with_restapi.html" class="list-group-item-lay pjaxlink">Build Cube with API</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_optimize_cubes.html" class="list-group-item-lay pjaxlink">Optimize Cube Design</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_backup_metadata.html" class="list-group-item-lay pjaxlink">Backup Metadata</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_cleanup_storage.html" class="list-group-item-lay pjaxlink">Cleanup Storage</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_upgrade.html" class="list-group-item-lay pjaxlink">Upgrade From Old Versions</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_use_cli.html" class="list-group-item-lay pjaxlink">Use Utility CLIs</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_ldap_and_sso.html" class="list-group-item-lay pjaxlink">Secure with LDAP and SSO</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_use_beeline.html" class="list-group-item-lay pjaxlink">Use Beeline for Hive</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_update_coprocessor.html" class="list-group-item-lay pjaxlink">Update Coprocessor</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_install_ranger_kylin_plugin.html" class="list-group-item-lay pjaxlink">Install Ranger Plugin</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_enable_zookeeper_acl.html" class="list-group-item-lay pjaxlink">Enable Zookeeper ACL</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/howto_use_hive_mr_dict.html" class="list-group-item-lay pjaxlink">Use Hive to build global dictionary</a> + </li> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/howto/sample_dataset.html" class="list-group-item-lay pjaxlink">Sample Dataset</a> + </li> + + + + + + + + + <ul> + </li> + + <li class="dropdown-submenu"><a href="#titleSecurity" data-toggle="collapse" class="navtitle">Security</a> + <ul class="dropdown-menu"> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <li class="navlist"> + <a href="/docs/security.html" class="list-group-item-lay pjaxlink">Security Issues</a> + </li> + + + + + + + + + <ul> + </li> + + </ul> --> + </li> + <li><a href="/download">Download</a></li> + <li><a href="/community" >Community</a></li> + <li><a href="/development" >Development</a></li> + <li><a href="/blog">Blog</a></li> + <li><a href="/cn" >ä¸æç</a></li> + </ul> + </div><!-- /.navbar-collapse --> + </div><!-- /.container-fluid --> + </nav> + + <div id="head" class="parallax normal-header" > + <div class="text-center header-apache"> + <a href="http://apache.org/foundation/contributing.html" title="Support Apache" style="margin-left: 150px;"> + <div> + <img src="https://www.apache.org/images/SupportApache-small.png" > + </div> + </a> + </div> + </div> + + </header> + + <div class="page-content main"> + <header style=" padding:2em 0 0 "> + <div class="container" > + <div style=" padding:0 4em"> + <div class="blog-icon"> + <img width="30" src="/assets/images/icon_blog_w.png"> + </div> + <h4 class="index-title" style=" float:left;"><span>Apache Kylin⢠Technical Blog</span></h4> + </div> + </div> + </div> + + <div class="container blog"> + <div> + <article class="post-content" > + <!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +<div class="post" style=" padding:2em 4em 4em 4em"> + + <header class="post-header"> + <h1 class="post-title">Kylin å¨ç¾å¢å°åºé¤é¥®çå®è·µåä¼å</h1> + <p class="post-meta" >Aug 3, 2021 ⢠岳åº</p> + </header> + + <article class="post-content" > + <p>ä»2016å¹´å¼å§ï¼ç¾å¢å°åºé¤é¥®ææ¯å¢éå°±å¼å§ä½¿ç¨Apache Kylinä½ä¸ºOLAPå¼æï¼ä½æ¯éçä¸å¡çé«éåå±ï¼å¨æ建åæ¥è¯¢å±é¢é½åºç°äºæçé®é¢ãäºæ¯ï¼ææ¯å¢éä»åç解读å¼å§ï¼ç¶å对è¿ç¨è¿è¡å±å±æ解ï¼å¹¶å¶å®äºç±ç¹åé¢çå®æ½è·¯çº¿ãæ¬ææ»ç»äºä¸äºç»éªåå¿å¾ï¼å¸æè½å¤å¸®å©ä¸çæ´å¤çææ¯å¢éæé«æ°æ®ç产åºæçã</p> + +<h2 id="section">èæ¯</h2> + +<p>éå®ä¸å¡çç¹ç¹æ¯è§æ¨¡å¤§ãé¢åå¤ãéæ±å¯ãç¾å¢å°åºé¤é¥®æ天éå®ç³»ç»ï¼<strong>以ä¸ç®ç§°âæ天â</strong>ï¼ä½ä¸ºéå®æ°æ®æ¯æç主è¦è½½ä½ï¼ä¸ä» æ¶åçèå´è¾å¹¿ï¼èä¸é¢ä¸´çææ¯åºæ¯ä¹é常å¤æï¼<strong>å¤ç»ç»å±çº§æ°æ®å±ç¤ºåé´æãè¶ è¿1/3çææ éè¦ç²¾åå»éï¼å³°å¼æ¥è¯¢å·²ç»è¾¾å°æ°ä¸çº§å«</strong>ï¼ãå¨è¿æ ·çä¸å¡èæ¯ä¸ï¼å»ºè®¾ç¨³å®é«æçOLAPå¼æï¼åå©åæ人åå¿«éå³çï¼å·²ç»æ为å°é¤æå¤� �çæ ¸å¿ç®æ ã</p> + +<p>Apache Kylinæ¯ä¸ä¸ªåºäºHadoop大æ°æ®å¹³å°æé çå¼æºOLAPå¼æï¼å®éç¨äºå¤ç»´ç«æ¹ä½é¢è®¡ç®ææ¯ï¼å©ç¨ç©ºé´æ¢æ¶é´çæ¹æ³ï¼å°æ¥è¯¢é度æåè³äºç§çº§å«ï¼æ大å°æé«äºæ°æ®åæçæçï¼å¹¶å¸¦æ¥äºä¾¿æ·ãçµæ´»çæ¥è¯¢åè½ãåºäºææ¯ä¸ä¸å¡å¹é 度ï¼æ天äº2016å¹´éç¨Kylinä½ä¸ºOLAPå¼æï¼æ¥ä¸æ¥çå å¹´éï¼è¿å¥ç³»ç»é«æå°æ¯æäºæ们çæ°æ®åæä½ç³»ã</p> + +<p>2020å¹´ï¼ç¾å¢å°é¤ä¸å¡åå±è¾å¿«ï¼æ°æ®ææ ä¹è¿ éå¢å ãåºäºKylinçè¿å¥ç³»ç»ï¼å¨æ建åæ¥è¯¢ä¸ååºç°äºä¸¥éçæçé®é¢ï¼ä»èå½±åå°æ°æ®çåæå³çï¼å¹¶ç»ç¨æ·ä½éªä¼å带æ¥äºå¾å¤§çé»ç¢ãææ¯å¢éç»è¿åå¹´å·¦å³çæ¶é´ï¼å¯¹Kylinè¿è¡ä¸ç³»åçä¼åè¿ä»£ï¼å æ¬ç»´åº¦è£åªã模å设计以åèµæºéé çççï¼å¸®å©éå®ä¸ç»©æ°æ®SLAä»90%æåè³99.99%ãåºäºè¿æ¬¡å®æï¼æ们æ²æ·äºä¸å¥æ¶µçäºâåçè§£è ¯»âãâè¿ç¨æ解âãâå®æ½è·¯çº¿âçææ¯æ¹æ¡ãå¸æè¿äºç»éªä¸æ»ç»ï¼è½å¤å¸®å©ä¸çæ´å¤çææ¯å¢éæé«æ°æ®äº§åºä¸ä¸å¡å³ççæçã</p> + +<h2 id="section-1">é®é¢ä¸ç®æ </h2> + +<p>éå®ä½ä¸ºè¡æ¥å¹³å°åå家çæ¡¥æ¢ï¼å å«éå®å°åºåçµè¯æ访两ç§ä¸å¡æ¨¡å¼ï¼ä»¥æåºã人åç»ç»æ¶æé级管çï¼ææåæåéè¦æ2å¥ç»ç»å±çº§æ¥çãå¨ææ å£å¾ä¸è´ãæ°æ®äº§åºåæ¶çè¦æ±ä¸ï¼æ们ç»åKylinçé¢è®¡ç®ææ³ï¼è¿è¡äºæ°æ®çæ¶æ设计ãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-01.png" alt="" /></p> + +<p>èKylin计ç®ç»´åº¦ç»åçå ¬å¼æ¯2^Nï¼<strong>N为维度个æ°</strong>ï¼ï¼å®æ¹æä¾ç»´åº¦åªæçæ¹å¼ï¼åå°ç»´åº¦ç»å个æ°ãä½ç±äºå°é¤ä¸å¡çç¹æ®æ§ï¼åä»»å¡ä¸å¯è£åªçç»å个æ°ä»é«è¾¾1000+ãå¨éæ±è¿ä»£ä»¥å人åãæåºç»ç»åå¨çåºæ¯ä¸ï¼éè¦åæº¯å ¨é¨åå²æ°æ®ï¼ä¼è费大éçèµæºä»¥åè¶ é«çæ建æ¶é¿ãèåºäºä¸å¡ååçæ¶æ设计ï¼è½è½å¤æ大å°ä¿è¯æ°æ®äº§åºç解è¦ï¼ä¿è¯ææ å£å¾çä¸è´æ§ï¼ä½æ¯å� �¹Kylinæ建产çäºå¾å¤§çååï¼è¿è导è´èµæºå ç¨å¤§ãèæ¶é¿ãåºäºä»¥ä¸ä¸å¡ç°ç¶ï¼æ们å½çº³äºKylinçMOLAP模å¼ä¸åå¨çé®é¢ï¼å ·ä½å¦ä¸ï¼</p> + +<ul> + <li><strong>æçé®é¢å½ä¸é¾ï¼å®ç°åçï¼</strong>ï¼æ建è¿ç¨æ¥éª¤å¤ï¼åæ¥éª¤ä¹é´å¼ºå ³èï¼ä» ä»é®é¢ç表象å¾é¾åç°é®é¢çæ ¹æ¬åå ï¼æ æ³è¡ä¹ææå°è§£å³é®é¢ã</li> + <li><strong>æ建å¼ææªè¿ä»£ï¼æ建è¿ç¨ï¼</strong>ï¼åå²ä»»å¡ä»éç¨MapReduceä½ä¸ºæ建å¼æï¼æ²¡æåæ¢å°æ建æçæ´é«çSparkã</li> + <li><strong>èµæºå©ç¨ä¸åçï¼æ建è¿ç¨ï¼</strong>ï¼èµæºæµªè´¹ãèµæºçå¾ ï¼é»è®¤å¹³å°å¨æèµæºéé æ¹å¼ï¼å¯¼è´å°ä»»å¡ç³è¯·äºå¤§éèµæºï¼æ°æ®ååä¸åçï¼äº§çäºå¤§éçå°æ件ï¼ä»èé æèµæºæµªè´¹ã大éä»»å¡çå¾ ã</li> + <li><strong>æ ¸å¿ä»»å¡èæ¶é¿ï¼å®æ½è·¯çº¿ï¼</strong>ï¼æ天éå®äº¤æä¸ç»©æ°æ®ææ çæºè¡¨æ°æ®é大ã维度ç»åå¤ãè¨èçé«ï¼å¯¼è´æ¯å¤©æ建çæ¶é¿è¶ è¿2个å°æ¶ã</li> + <li><strong>SLAè´¨éä¸è¾¾æ ï¼å®æ½è·¯çº¿ï¼</strong>ï¼SLAçæ´ä½è¾¾æçæªè½è¾¾å°é¢æç®æ ã</li> +</ul> + +<p>å¨è®¤çåæå®é®é¢ï¼å¹¶ç¡®å®ææç大ç®æ åï¼æ们对Kylinçæ建è¿ç¨è¿è¡äºåç±»ï¼æ解åºå¨æ建è¿ç¨ä¸è½æåæççæ ¸å¿ç¯èï¼éè¿âåç解读âãâå±å±æ解âãâç±ç¹åé¢âçæ段ï¼è¾¾æååéä½çç®æ ãå ·ä½éåç®æ å¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-02.png" alt="" /></p> + +<h2 id="section-2">ä¼ååæ-åç解读</h2> + +<p>为äºè§£å³æçæåå®ä½é¾ãå½å é¾çé®é¢ï¼æ们解读äºKylinæ建åçï¼å å«äºé¢è®¡ç®ææ³ä»¥åBy-layeréå±ç®æ³ã</p> + +<h3 id="section-3">é¢è®¡ç®</h3> + +<p>æ ¹æ®ç»´åº¦ç»ååºææå¯è½ç维度ï¼å¯¹å¤ç»´åæå¯è½ç¨å°çææ è¿è¡é¢è®¡ç®ï¼å°è®¡ç®å¥½çç»æä¿åæCubeãå设æ们æ4个维度ï¼è¿ä¸ªCubeä¸æ¯ä¸ªèç¹ï¼<strong>称ä½Cuboid</strong>ï¼é½æ¯è¿4个维度çä¸åç»åï¼æ¯ä¸ªç»åå®ä¹äºä¸ç»åæç维度ï¼<strong>å¦group by</strong>ï¼ï¼ææ çèåç»æå°±ä¿åå¨æ¯ä¸ªCuboidä¸ãæ¥è¯¢æ¶ï¼æä»¬æ ¹æ®SQLæ¾å°å¯¹åºçCuboidï¼è¯»åææ çå¼ï¼å³å¯è¿åãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-03.png" alt="" /></p> + +<h3 id="by-layer">By-layeréå±ç®æ³</h3> + +<p>ä¸ä¸ªNç»´çCubeï¼æ¯ç±1个Nç»´åç«æ¹ä½ãN个ï¼N-1ï¼ç»´åç«æ¹ä½ãN*(N-1)/2个(N-2)ç»´åç«æ¹ä½ãâ¦â¦N个1ç»´åç«æ¹ä½å1个0ç»´åç«æ¹ä½ææï¼æ»å ±æ 2^N个åç«æ¹ä½ãå¨éå±ç®æ³ä¸ï¼æç §ç»´åº¦æ°éå±åå°æ¥è®¡ç®ï¼æ¯ä¸ªå±çº§ç计ç®ï¼é¤äºç¬¬ä¸å±ï¼ç±åå§æ°æ®èåèæ¥ï¼ï¼æ¯åºäºä¸ä¸å±çº§ç计ç®ç»ææ¥è®¡ç®çã</p> + +<p>ä¾å¦ï¼group by [A,B]çç»æï¼å¯ä»¥åºäºgroup by [A,B,C]çç»æï¼éè¿å»æCåèåå¾æ¥çï¼è¿æ ·å¯ä»¥åå°éå¤è®¡ç®ï¼å½0ç»´Cuboid计ç®åºæ¥çæ¶åï¼æ´ä¸ªCubeç计ç®ä¹å°±å®æäºãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-04.png" alt="" /></p> + +<h2 id="section-4">è¿ç¨åæ-å±å±æ解</h2> + +<p>å¨äºè§£å®Kylinçåºå±åçåï¼æ们å°ä¼åçæ¹åéå®å¨âå¼æéæ©âãâæ°æ®è¯»åâãâæ建åå ¸âãâåå±æ建âãâæ件转æ¢âäºä¸ªç¯èï¼åç»ååé¶æ®µçé®é¢ãæè·¯åç®æ åï¼æ们ç»äºåå°äºå¨éä½è®¡ç®èµæºçåæ¶éä½äºèæ¶ã详æ å¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-05.png" alt="" /></p> + +<h3 id="section-5">æ建å¼æéæ©</h3> + +<p>ç®åï¼æ们已ç»å°æ建å¼æå·²éæ¥åæ¢ä¸ºSparkãæ天æ©å¨2016年就使ç¨Kylinä½ä¸ºOLAPå¼æï¼åå²ä»»å¡æ²¡æåæ¢ï¼ä» ä» é对MapReduceåäºåæ°ä¼åãå ¶å®å¨2017å¹´ï¼Kylinå®ç½å·²å¯ç¨Sparkä½ä¸ºæ建å¼æï¼å®ç½å¯ç¨Sparkæ建å¼æï¼ï¼æ建æçç¸è¾MapReduceæå1è³3åï¼è¿å¯éè¿Cube设计éæ©åæ¢ï¼å¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-06.png" alt="" /></p> + +<h3 id="section-6">读åæºæ°æ®</h3> + +<p>Kylin以å¤é¨è¡¨çæ¹å¼è¯»åHiveä¸çæºæ°æ®ï¼è¡¨ä¸çæ°æ®æ件ï¼<strong>åå¨å¨HDFS</strong>ï¼ä½ä¸ºä¸ä¸ä¸ªåä»»å¡çè¾å ¥ï¼æ¤è¿ç¨å¯è½åå¨å°æ件é®é¢ãå½åï¼Kylinä¸æ¸¸æ°æ®å®½è¡¨æ件æ°åå¸æ¯è¾åçï¼æ éå¨ä¸æ¸¸è®¾ç½®å并ï¼å¦æ强è¡å并åèä¼å¢å ä¸æ¸¸æºè¡¨æ°æ®å å·¥æ¶é´ã</p> + +<p>对äºé¡¹ç®éæ±ï¼è¦åå·åå²æ°æ®æå¢å 维度ç»åï¼éè¦éæ°æå»ºå ¨é¨çæ°æ®ï¼é常éç¨æææ建çæ¹å¼åå·åå²ï¼å è½½çååºè¿å¤åºç°å°æ件é®é¢ï¼å¯¼è´æ¤è¿ç¨æ§è¡ç¼æ ¢ãå¨Kylin级å«éåé ç½®æ件ï¼å¯¹å°æ件è¿è¡å并ï¼åå°Mapæ°éï¼å¯ææå°æå读åæçã</p> + +<p><strong>å并æºè¡¨å°æ件</strong>ï¼å并Hiveæºè¡¨ä¸å°æ件个æ°ï¼æ§å¶æ¯ä¸ªJob并è¡çTask个æ°ãè°æ´åæ°å¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-07.png" alt="" /></p> + +<p><strong>Kylin级å«åæ°éå</strong>ï¼è®¾ç½®Map读åè¿ç¨çæ件大å°ãè°æ´åæ°å¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-08.png" alt="" /></p> + +<h3 id="section-7">æ建åå ¸</h3> + +<p>Kylinéè¿è®¡ç®Hive表åºç°ç维度å¼ï¼å建维度åå ¸ï¼å°ç»´åº¦å¼æ å°æç¼ç ï¼å¹¶ä¿åä¿åç»è®¡ä¿¡æ¯ï¼è约HBaseåå¨èµæºãæ¯ä¸ç§ç»´åº¦ç»åï¼ç§°ä¸ºä¸ä¸ªCuboidãç论ä¸æ¥è¯´ï¼ä¸ä¸ªNç»´çCubeï¼ä¾¿æ2^Nç§ç»´åº¦ç»åã</p> + +<h4 id="section-8">ç»åæ°éæ¥ç</h4> + +<p>å¨å¯¹ç»´åº¦ç»ååªæåï¼å®é 计ç®ç»´åº¦ç»åé¾ä»¥è®¡ç®ï¼å¯éè¿æ§è¡æ¥å¿ï¼<strong>æªå¾ä¸ºæåäºå®è¡¨å¯ä¸åçæ¥éª¤ä¸ï¼æåä¸ä¸ªReduceçæ¥å¿</strong>ï¼ï¼æ¥çå ·ä½ç维度ç»åæ°éãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-09.png" alt="" /></p> + +<h4 id="section-9">å ¨å±åå ¸ä¾èµ</h4> + +<p>æ天æå¾å¤ä¸å¡åºæ¯éè¦ç²¾ç¡®å»éï¼å½åå¨å¤ä¸ªå ¨å±åå ¸åæ¶ï¼å¯è®¾ç½®åä¾èµï¼ä¾å¦ï¼å½åæ¶åå¨âé¨åºæ°éâãâå¨çº¿é¨åºæ°éâæ°æ®ææ ï¼å¯è®¾ç½®åä¾èµï¼åå°å¯¹è¶ é«åºç»´åº¦ç计ç®ãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-10.png" alt="" /></p> + +<h4 id="section-10">计ç®èµæºé ç½®</h4> + +<p>å½ææ ä¸åå¨å¤ä¸ªç²¾åå»éææ æ¶ï¼å¯éå½å¢å 计ç®èµæºï¼æå对é«åºç»´åº¦æ建çæçãåæ°è®¾ç½®å¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-11.png" alt="" /></p> + +<h3 id="section-11">åå±æ建</h3> + +<p>æ¤è¿ç¨ä¸ºKylinæ建çæ ¸å¿ï¼åæ¢Sparkå¼æåï¼é»è®¤åªéç¨By-layeréå±ç®æ³ï¼ä¸åèªå¨éæ©ï¼By-layeréå±ç®æ³ãå¿«éç®æ³ï¼ãSparkå¨å®ç°By-layeréå±ç®æ³çè¿ç¨ä¸ï¼ä»æåºå±çCuboidä¸å±ä¸å±å°åä¸è®¡ç®ï¼ç´å°è®¡ç®åºæ顶å±çCuboidï¼ç¸å½äºæ§è¡äºä¸ä¸ªä¸å¸¦group byçæ¥è¯¢ï¼ï¼å°åå±çç»ææ°æ®ç¼åå°å åä¸ï¼è·³è¿æ¯æ¬¡æ°æ®ç读åè¿ç¨ï¼ç´æ¥ä¾èµä¸å±çç¼åæ°æ®ï¼å¤§å¤§æé«äºæ§è¡æçãSparkæ§è¡è¿ç¨å ·ä½å 容å¦ä¸ ã</p> + +<h4 id="job">Jobé¶æ®µ</h4> + +<p>Job个æ°ä¸ºBy-layerç®æ³æ çå±æ°ï¼Sparkå°æ¯å±ç»ææ°æ®çè¾åºï¼ä½ä¸ºä¸ä¸ªJobãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-12.png" alt="" /></p> + +<h4 id="stage">Stageé¶æ®µ</h4> + +<p>æ¯ä¸ªJob对åºä¸¤ä¸ªStageé¶æ®µï¼å为读åä¸å±ç¼åæ°æ®åç¼å该å±è®¡ç®åçç»ææ°æ®ãå¦ä¸å¾æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-13.png" alt="" /></p> + +<h4 id="task">Task并è¡åº¦è®¾ç½®</h4> + +<p>Kylinæ ¹æ®é¢ä¼°æ¯å±æ建Cuboidç»åæ°æ®ç大å°ï¼<strong>å¯éè¿ç»´åº¦åªæçæ¹å¼ï¼åå°ç»´åº¦ç»åçæ°éï¼éä½Cuboidç»åæ°æ®ç大å°ï¼æåæ建æçï¼æ¬ææä¸è¯¦ç»ä»ç»</strong>ï¼ååå²æ°æ®çåæ°å¼è®¡ç®åºä»»å¡å¹¶è¡åº¦ã计ç®å ¬å¼å¦ä¸ï¼</p> + +<ul> + <li> + <p><strong>Task个æ°è®¡ç®å ¬å¼</strong>ï¼Min(MapSize/cut-mb ï¼MaxPartition) ï¼Max(MapSize/cut-mb ï¼MinPartition)</p> + </li> + <li> + <ul> + <li><strong>MapSize</strong>ï¼æ¯å±æ建çCuboidç»å大å°ï¼å³ï¼Kylin对åå±çº§ç»´åº¦ç»å大å°çé¢ä¼°å¼ã</li> + <li><strong>cut-mb</strong>ï¼åå²æ°æ®å¤§å°ï¼æ§å¶Taskä»»å¡å¹¶è¡ä¸ªæ°ï¼å¯éè¿kylin.engine.spark.rdd-partition-cut-mbåæ°è®¾ç½®ã</li> + <li><strong>MaxPartition</strong>ï¼æ大ååºï¼å¯éè¿kylin.engine.spark.max-partitionåæ°è®¾ç½®ã</li> + <li><strong>MinPartition</strong>ï¼æå°ååºï¼å¯éè¿kylin.engine.spark.min-partitionåæ°è®¾ç½®ã</li> + </ul> + </li> + <li><strong>è¾åºæ件个æ°è®¡ç®</strong>ï¼æ¯ä¸ªTaskä»»å¡å°æ§è¡å®æåçç»ææ°æ®å缩ï¼åå ¥HDFSï¼ä½ä¸ºæ件转æ¢è¿ç¨çè¾å ¥ãæ件个æ°å³ä¸ºï¼Taskä»»å¡è¾åºæ件个æ°çæ±æ»ã</li> +</ul> + +<h4 id="section-12">èµæºç³è¯·è®¡ç®</h4> + +<p>å¹³å°é»è®¤éç¨å¨ææ¹å¼ç³è¯·è®¡ç®èµæºï¼å个Executorç计ç®è½åå å«ï¼1个é»è¾CPUï¼ä»¥ä¸ç®ç§°CPUï¼ã6GBå å å åã1GBçå å¤å åã计ç®å ¬å¼å¦ä¸ï¼</p> + +<ul> + <li><strong>CPU</strong> = kylin.engine.spark-conf.spark.executor.cores * å®é ç³è¯·çExecutors个æ°ã</li> + <li><strong>å å</strong> =ï¼kylin.engine.spark-conf.spark.executor.memory + spark.yarn.executor.memoryOverheadï¼* å®é ç³è¯·çExecutors个æ°ã</li> + <li><strong>å个Executorçæ§è¡è½å</strong> = kylin.engine.spark-conf.spark.executor.memory / kylin.engine.spark-conf.spark.executor.coresï¼å³ï¼1个CPUæ§è¡è¿ç¨ä¸ç³è¯·çå å大å°ã</li> + <li><strong>æ大Executors个æ°</strong> = kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutorsï¼å¹³å°é»è®¤å¨æç³è¯·ï¼è¯¥åæ°éå¶æ大ç³è¯·ä¸ªæ°ã</li> +</ul> + +<p>å¨èµæºå 足çæ åµä¸ï¼è¥å个Stageé¶æ®µç³è¯·1000个并è¡ä»»å¡ï¼åéè¦ç³è¯·èµæºè¾¾å°7000GBå åå1000个CPUï¼å³ï¼<code class="highlighter-rouge">CPUï¼1*1000=1000ï¼å åï¼ï¼6+1ï¼*1000=7000GB</code>ã</p> + +<h4 id="section-13">èµæºåçåéé </h4> + +<p>ç±äºBy-layeréå±ç®æ³çç¹æ§ï¼ä»¥åSparkå¨å®é æ§è¡è¿ç¨ä¸çå缩æºå¶ï¼å®é æ§è¡çTaskä»»å¡å è½½çååºæ°æ®è¿è¿å°äºåæ°è®¾ç½®å¼ï¼ä»è导è´ä»»å¡è¶ é«å¹¶è¡ï¼å ç¨å¤§éèµæºï¼åæ¶äº§ç大éçå°æ件ï¼å½±åä¸æ¸¸æ件转æ¢è¿ç¨ãå æ¤ï¼åççååæ°æ®æ为ä¼åçå ³é®ç¹ãéè¿Kylinæ建æ¥å¿ï¼å¯æ¥çåå±çº§çCuboidç»åæ°æ®çé¢ä¼°å¤§å°ï¼ä»¥åååçååºä¸ªæ°ï¼çäºStageé¶æ®µå®é çæçTask个æ°ï¼ãå¦ä¸å¾æ� �示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-14.png" alt="" /></p> + +<p>ç»åSpark UIå¯æ¥çå®æ§è¡æ åµï¼è°æ´å åçç³è¯·ï¼æ»¡è¶³æ§è¡æéè¦çèµæºå³å¯ï¼åå°èµæºæµªè´¹ã</p> + +<ol> + <li>æ´ä½èµæºç³è¯·æå°å¼å¤§äºStageé¶æ®µTop1ãTop2å±çº§çç¼åæ°æ®ä¹åï¼ä¿è¯ç¼åæ°æ®å ¨é¨å¨å åãå¦ä¸å¾æ示ï¼</li> +</ol> + +<p><img src="/images/blog/meituan_cn/chart-15.png" alt="" /></p> + +<p><strong>计ç®å ¬å¼</strong>ï¼Stageé¶æ®µTop1ãTop2å±çº§çç¼åæ°æ®ä¹å < kylin.engine.spark-conf.spark.executor.memory * kylin.engine.spark-conf.spark.memory.fraction * spark.memory.storageFraction *æ大Executors个æ°</p> + +<ol> + <li>å个Taskå®é æéè¦çå ååCPUï¼<strong>1个Taskæ§è¡ä½¿ç¨1个CPU</strong>ï¼å°äºå个Executorçæ§è¡è½åãå¦ä¸å¾æ示ï¼</li> +</ol> + +<p><img src="/images/blog/meituan_cn/chart-16.png" alt="" /></p> + +<p><strong>计ç®å ¬å¼</strong>ï¼å个Taskå®é æéè¦çå å < kylin.engine.spark-conf.spark.executor.memory * kylin.engine.spark-conf.spark.memory.fraction * spark.memory.st·orageFraction / kylin.engine.spark-conf.spark.executor.coresãåæ°è¯´æå¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-17.png" alt="" /></p> + +<h3 id="section-14">æ件转æ¢</h3> + +<p>Kylinå°æ建ä¹åçCuboidæ件转æ¢æHTableæ ¼å¼çHfileæ件ï¼éè¿BulkLoadçæ¹å¼å°æ件åHTableè¿è¡å ³èï¼å¤§å¤§éä½äºHBaseçè´è½½ãæ¤è¿ç¨éè¿ä¸ä¸ªMapReduceä»»å¡å®æï¼Map个æ°ä¸ºåå±æ建é¶æ®µè¾åºæ件个æ°ãæ¥å¿å¦ä¸ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-18.png" alt="" /></p> + +<p>æ¤é¶æ®µå¯æ ¹æ®å®é è¾å ¥çæ°æ®æ件大å°ï¼<strong>å¯éè¿MapReduceæ¥å¿æ¥ç</strong>ï¼ï¼åçç³è¯·è®¡ç®èµæºï¼é¿å èµæºæµªè´¹ã</p> + +<p>计ç®å ¬å¼ï¼Mapé¶æ®µèµæºç³è¯· = kylin.job.mr.config.override.mapreduce.map.memory.mb * åå±æ建é¶æ®µè¾åºæ件个æ°ãå ·ä½åæ°å¦ä¸è¡¨æ示ï¼</p> + +<p><img src="/images/blog/meituan_cn/chart-19.png" alt="" /></p> + +<h2 id="section-15">å®æ½è·¯çº¿-ç±ç¹åé¢</h2> + +<h3 id="section-16">交æè¯ç¹å®è·µ</h3> +
[... 99 lines stripped ...]