A JIRA ticket has been opened, waiting for INFRA : https://issues.apache.org/jira/browse/INFRA-25238 . ------------------------ With warm regard Xiaoxiang Yu
On Tue, Dec 5, 2023 at 10:30 AM Nam Đỗ Duy <na...@vnpay.vn.invalid> wrote: > Thank you Xiaoxiang, please update me when you have changed your default > branch. In case people are impressed by the numbers then I hope to turn > this situation to reverse direction. > > On Tue, Dec 5, 2023 at 9:02 AM Xiaoxiang Yu <x...@apache.org> wrote: > >> The default branch is for 4.X which is a maintained branch, the active >> branch is kylin5. >> I will change the default branch to kylin5 later. >> >> ------------------------ >> With warm regard >> Xiaoxiang Yu >> >> >> >> On Tue, Dec 5, 2023 at 9:12 AM Nam Đỗ Duy <na...@vnpay.vn.invalid> wrote: >> >>> Hi Xiaoxiang, Sirs / Madams >>> >>> Can you see the atttached photo >>> >>> My boss asked that why druid commit code regularly but kylin had not >>> been committed since July >>> >>> >>> On Mon, 4 Dec 2023 at 15:33 Xiaoxiang Yu <x...@apache.org> wrote: >>> >>>> I think so. >>>> >>>> Response time is not the only factor to make a decision. Kylin could be >>>> cheaper >>>> when the query pattern is suitable for the Kylin model, and Kylin can >>>> guarantee >>>> reasonable query latency. Clickhouse will be quicker in an ad hoc query >>>> scenario. >>>> >>>> By the way, Youzan and Kyligence combine them together to provide >>>> unified data analytics services for their customers. >>>> >>>> ------------------------ >>>> With warm regard >>>> Xiaoxiang Yu >>>> >>>> >>>> >>>> On Mon, Dec 4, 2023 at 4:01 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>> wrote: >>>> >>>>> Hi Xiaoxiang, thank you >>>>> >>>>> In case my client uses cloud computing service like gcp or aws, which >>>>> will cost more: precalculation feature of kylin or clickhouse (incase >>>>> of >>>>> kylin, I have a thought that the query execution has been done once and >>>>> stored in cube to be used many times so kylin uses less cloud >>>>> computation, >>>>> is that true)? >>>>> >>>>> On Mon, Dec 4, 2023 at 2:46 PM Xiaoxiang Yu <x...@apache.org> wrote: >>>>> >>>>> > Following text is part of an article( >>>>> > https://zhuanlan.zhihu.com/p/343394287) . >>>>> > >>>>> > >>>>> > >>>>> =============================================================================== >>>>> > >>>>> > Kylin is suitable for aggregation queries with fixed modes because >>>>> of its >>>>> > pre-calculated technology, for example, join, group by, and where >>>>> condition >>>>> > modes in SQL are relatively fixed, etc. The larger the data volume >>>>> is, the >>>>> > more obvious the advantages of using Kylin are; in particular, Kylin >>>>> is >>>>> > particularly advantageous in the scenarios of de-emphasis (count >>>>> distinct), >>>>> > Top N, and Percentile. In particular, Kylin's advantages in >>>>> de-weighting >>>>> > (count distinct), Top N, Percentile and other scenarios are >>>>> especially >>>>> > huge, and it is used in a large number of scenarios, such as >>>>> Dashboard, all >>>>> > kinds of reports, large-screen display, traffic statistics, and user >>>>> > behavior analysis. Meituan, Aurora, Shell Housing, etc. use Kylin to >>>>> build >>>>> > their data service platforms, providing millions to tens of millions >>>>> of >>>>> > queries per day, and most of the queries can be completed within 2 - >>>>> 3 >>>>> > seconds. There is no better alternative for such a high concurrency >>>>> > scenario. >>>>> > >>>>> > ClickHouse, because of its MPP architecture, has high computing >>>>> power and >>>>> > is more suitable when the query request is more flexible, or when >>>>> there is >>>>> > a need for detailed queries with low concurrency. Scenarios include: >>>>> very >>>>> > many columns and where conditions are arbitrarily combined with the >>>>> user >>>>> > label filtering, not a large amount of concurrency of complex >>>>> on-the-spot >>>>> > query and so on. If the amount of data and access is large, you need >>>>> to >>>>> > deploy a distributed ClickHouse cluster, which is a higher challenge >>>>> for >>>>> > operation and maintenance. >>>>> > >>>>> > If some queries are very flexible but infrequent, it is more >>>>> > resource-efficient to use now-computing. Since the number of queries >>>>> is >>>>> > small, even if each query consumes a lot of computational resources, >>>>> it is >>>>> > still cost-effective overall. If some queries have a fixed pattern >>>>> and the >>>>> > query volume is large, it is more suitable for Kylin, because the >>>>> query >>>>> > volume is large, and by using large computational resources to save >>>>> the >>>>> > results, the upfront computational cost can be amortized over each >>>>> query, >>>>> > so it is the most economical. >>>>> > >>>>> > --- Translated with DeepL.com (free version) >>>>> > >>>>> > >>>>> > ------------------------ >>>>> > With warm regard >>>>> > Xiaoxiang Yu >>>>> > >>>>> > >>>>> > >>>>> > On Mon, Dec 4, 2023 at 3:16 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>>> wrote: >>>>> > >>>>> >> Thank you Xiaoxiang for the near real time streaming feature. That's >>>>> >> great. >>>>> >> >>>>> >> This morning there has been a new challenge to my team: clickhouse >>>>> offered >>>>> >> us the speed of calculating 8 billion rows in millisecond which is >>>>> faster >>>>> >> than my demonstration (I used Kylin to do calculating 1 billion >>>>> rows in >>>>> >> 2.9 >>>>> >> seconds) >>>>> >> >>>>> >> Can you briefly suggest the advantages of kylin over clickhouse so >>>>> that I >>>>> >> can defend my demonstration. >>>>> >> >>>>> >> On Mon, Dec 4, 2023 at 1:55 PM Xiaoxiang Yu <x...@apache.org> >>>>> wrote: >>>>> >> >>>>> >> > 1. "In this important scenario of realtime analytics, the reason >>>>> here is >>>>> >> > that >>>>> >> > kylin has lag time due to model update of new segment build, is >>>>> that >>>>> >> > correct?" >>>>> >> > >>>>> >> > You are correct. >>>>> >> > >>>>> >> > 2. "If that is true, then can you suggest a work-around of >>>>> combination >>>>> >> of >>>>> >> > ... " >>>>> >> > >>>>> >> > Kylin is planning to introduce NRT streaming(coding is completed >>>>> but not >>>>> >> > released), >>>>> >> > which can make the time-lag to about 3 minutes(that is my >>>>> estimation >>>>> >> but I >>>>> >> > am >>>>> >> > quite certain about it). >>>>> >> > NRT stands for 'near real-time', it will run a job and do >>>>> micro-batch >>>>> >> > aggregation and persistence periodically. The price is that you >>>>> need to >>>>> >> run >>>>> >> > and monitor a long-running >>>>> >> > job. This feature is based on Spark Streaming, so you need >>>>> knowledge of >>>>> >> > it. >>>>> >> > >>>>> >> > I am curious about what is the maximum time-lag your customers >>>>> >> > can tolerate? >>>>> >> > Personally, I guess minute level time-lag is ok for most cases. >>>>> >> > >>>>> >> > ------------------------ >>>>> >> > With warm regard >>>>> >> > Xiaoxiang Yu >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > On Mon, Dec 4, 2023 at 12:28 PM Nam Đỗ Duy <na...@vnpay.vn.invalid >>>>> > >>>>> >> wrote: >>>>> >> > >>>>> >> > > Druid is better in >>>>> >> > > - Have a real-time datasource like Kafka etc. >>>>> >> > > >>>>> >> > > ========================== >>>>> >> > > >>>>> >> > > Hi Xiaoxiang, thank you for your response. >>>>> >> > > >>>>> >> > > In this important scenario of realtime alalytics, the reason >>>>> here is >>>>> >> that >>>>> >> > > kylin has lag time due to model update of new segment build, is >>>>> that >>>>> >> > > correct? >>>>> >> > > >>>>> >> > > If that is true, then can you suggest a work-around of >>>>> combination of >>>>> >> : >>>>> >> > > >>>>> >> > > (time - lag kylin cube) + (realtime DB update) to provide >>>>> >> > > realtime capability ? >>>>> >> > > >>>>> >> > > IMO, the point here is to find that (realtime DB update) and >>>>> >> integrate it >>>>> >> > > with (time - lag kylin cube). >>>>> >> > > >>>>> >> > > On Fri, Dec 1, 2023 at 1:53 PM Xiaoxiang Yu <x...@apache.org> >>>>> wrote: >>>>> >> > > >>>>> >> > > > I researched and tested Druid two years ago(I don't know too >>>>> much >>>>> >> about >>>>> >> > > > the change of Druid in these two years. New features that I >>>>> know >>>>> >> are : >>>>> >> > > > new UI, fully on K8s etc). >>>>> >> > > > >>>>> >> > > > Here are some cases you should consider using Druid other >>>>> than Kylin >>>>> >> > > > at the moment (using Kylin 5.0-beta to compare the Druid >>>>> which I >>>>> >> used >>>>> >> > two >>>>> >> > > > years ago): >>>>> >> > > > >>>>> >> > > > - Have a real-time datasource like Kafka etc. >>>>> >> > > > - Most queries are small(Based on my test result, I think >>>>> Druid had >>>>> >> > > better >>>>> >> > > > response time for small queries two years ago.) >>>>> >> > > > - Don't know how to optimize Spark/Hadoop, want to use the >>>>> >> K8S/public >>>>> >> > > > cloud platform as your deployment platform. >>>>> >> > > > >>>>> >> > > > But I do think there are many scenarios in which Kylin could >>>>> be >>>>> >> better, >>>>> >> > > > like: >>>>> >> > > > >>>>> >> > > > - Better performance for complex/big queries. Kylin can have >>>>> a more >>>>> >> > > > exact-match/fine-grained >>>>> >> > > > Index for queries containing different `Group By >>>>> dimensions`. >>>>> >> > > > - User-friendly UI for modeling. >>>>> >> > > > - Support 'Join' better? (Not sure at the moment) >>>>> >> > > > - ODBC driver for different BI.(its website did not show it >>>>> supports >>>>> >> > ODBC >>>>> >> > > > well) >>>>> >> > > > - Looks like Kylin supports ANSI SQL better than Druid. >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > I don't know Pinot, so I have nothing to say about it. >>>>> >> > > > Hope to help you, or you are free to share your opinion. >>>>> >> > > > >>>>> >> > > > ------------------------ >>>>> >> > > > With warm regard >>>>> >> > > > Xiaoxiang Yu >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > On Fri, Dec 1, 2023 at 11:11 AM Nam Đỗ Duy >>>>> <na...@vnpay.vn.invalid> >>>>> >> > > wrote: >>>>> >> > > > >>>>> >> > > >> Dear Xiaoxiang, >>>>> >> > > >> Sirs/Madams, >>>>> >> > > >> >>>>> >> > > >> May I post my boss's question: >>>>> >> > > >> >>>>> >> > > >> What are the pros and cons of the OLAP platform Kylin >>>>> compared to >>>>> >> > Pinot >>>>> >> > > >> and >>>>> >> > > >> Druid? >>>>> >> > > >> >>>>> >> > > >> Please kindly let me know >>>>> >> > > >> >>>>> >> > > >> Thank you very much and best regards >>>>> >> > > >> >>>>> >> > > > >>>>> >> > > >>>>> >> > >>>>> >> >>>>> > >>>>> >>>>
