Done. Github branch changed to kylin5. ------------------------ With warm regard Xiaoxiang Yu
On Tue, Dec 5, 2023 at 11:13 AM Xiaoxiang Yu <x...@apache.org> wrote: > A JIRA ticket has been opened, waiting for INFRA : > https://issues.apache.org/jira/browse/INFRA-25238 . > ------------------------ > With warm regard > Xiaoxiang Yu > > > > On Tue, Dec 5, 2023 at 10:30 AM Nam Đỗ Duy <na...@vnpay.vn.invalid> wrote: > >> Thank you Xiaoxiang, please update me when you have changed your default >> branch. In case people are impressed by the numbers then I hope to turn >> this situation to reverse direction. >> >> On Tue, Dec 5, 2023 at 9:02 AM Xiaoxiang Yu <x...@apache.org> wrote: >> >>> The default branch is for 4.X which is a maintained branch, the active >>> branch is kylin5. >>> I will change the default branch to kylin5 later. >>> >>> ------------------------ >>> With warm regard >>> Xiaoxiang Yu >>> >>> >>> >>> On Tue, Dec 5, 2023 at 9:12 AM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>> wrote: >>> >>>> Hi Xiaoxiang, Sirs / Madams >>>> >>>> Can you see the atttached photo >>>> >>>> My boss asked that why druid commit code regularly but kylin had not >>>> been committed since July >>>> >>>> >>>> On Mon, 4 Dec 2023 at 15:33 Xiaoxiang Yu <x...@apache.org> wrote: >>>> >>>>> I think so. >>>>> >>>>> Response time is not the only factor to make a decision. Kylin could >>>>> be cheaper >>>>> when the query pattern is suitable for the Kylin model, and Kylin can >>>>> guarantee >>>>> reasonable query latency. Clickhouse will be quicker in an ad hoc >>>>> query scenario. >>>>> >>>>> By the way, Youzan and Kyligence combine them together to provide >>>>> unified data analytics services for their customers. >>>>> >>>>> ------------------------ >>>>> With warm regard >>>>> Xiaoxiang Yu >>>>> >>>>> >>>>> >>>>> On Mon, Dec 4, 2023 at 4:01 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>>> wrote: >>>>> >>>>>> Hi Xiaoxiang, thank you >>>>>> >>>>>> In case my client uses cloud computing service like gcp or aws, which >>>>>> will cost more: precalculation feature of kylin or clickhouse (incase >>>>>> of >>>>>> kylin, I have a thought that the query execution has been done once >>>>>> and >>>>>> stored in cube to be used many times so kylin uses less cloud >>>>>> computation, >>>>>> is that true)? >>>>>> >>>>>> On Mon, Dec 4, 2023 at 2:46 PM Xiaoxiang Yu <x...@apache.org> wrote: >>>>>> >>>>>> > Following text is part of an article( >>>>>> > https://zhuanlan.zhihu.com/p/343394287) . >>>>>> > >>>>>> > >>>>>> > >>>>>> =============================================================================== >>>>>> > >>>>>> > Kylin is suitable for aggregation queries with fixed modes because >>>>>> of its >>>>>> > pre-calculated technology, for example, join, group by, and where >>>>>> condition >>>>>> > modes in SQL are relatively fixed, etc. The larger the data volume >>>>>> is, the >>>>>> > more obvious the advantages of using Kylin are; in particular, >>>>>> Kylin is >>>>>> > particularly advantageous in the scenarios of de-emphasis (count >>>>>> distinct), >>>>>> > Top N, and Percentile. In particular, Kylin's advantages in >>>>>> de-weighting >>>>>> > (count distinct), Top N, Percentile and other scenarios are >>>>>> especially >>>>>> > huge, and it is used in a large number of scenarios, such as >>>>>> Dashboard, all >>>>>> > kinds of reports, large-screen display, traffic statistics, and user >>>>>> > behavior analysis. Meituan, Aurora, Shell Housing, etc. use Kylin >>>>>> to build >>>>>> > their data service platforms, providing millions to tens of >>>>>> millions of >>>>>> > queries per day, and most of the queries can be completed within 2 >>>>>> - 3 >>>>>> > seconds. There is no better alternative for such a high concurrency >>>>>> > scenario. >>>>>> > >>>>>> > ClickHouse, because of its MPP architecture, has high computing >>>>>> power and >>>>>> > is more suitable when the query request is more flexible, or when >>>>>> there is >>>>>> > a need for detailed queries with low concurrency. Scenarios >>>>>> include: very >>>>>> > many columns and where conditions are arbitrarily combined with the >>>>>> user >>>>>> > label filtering, not a large amount of concurrency of complex >>>>>> on-the-spot >>>>>> > query and so on. If the amount of data and access is large, you >>>>>> need to >>>>>> > deploy a distributed ClickHouse cluster, which is a higher >>>>>> challenge for >>>>>> > operation and maintenance. >>>>>> > >>>>>> > If some queries are very flexible but infrequent, it is more >>>>>> > resource-efficient to use now-computing. Since the number of >>>>>> queries is >>>>>> > small, even if each query consumes a lot of computational >>>>>> resources, it is >>>>>> > still cost-effective overall. If some queries have a fixed pattern >>>>>> and the >>>>>> > query volume is large, it is more suitable for Kylin, because the >>>>>> query >>>>>> > volume is large, and by using large computational resources to save >>>>>> the >>>>>> > results, the upfront computational cost can be amortized over each >>>>>> query, >>>>>> > so it is the most economical. >>>>>> > >>>>>> > --- Translated with DeepL.com (free version) >>>>>> > >>>>>> > >>>>>> > ------------------------ >>>>>> > With warm regard >>>>>> > Xiaoxiang Yu >>>>>> > >>>>>> > >>>>>> > >>>>>> > On Mon, Dec 4, 2023 at 3:16 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>>>> wrote: >>>>>> > >>>>>> >> Thank you Xiaoxiang for the near real time streaming feature. >>>>>> That's >>>>>> >> great. >>>>>> >> >>>>>> >> This morning there has been a new challenge to my team: clickhouse >>>>>> offered >>>>>> >> us the speed of calculating 8 billion rows in millisecond which is >>>>>> faster >>>>>> >> than my demonstration (I used Kylin to do calculating 1 billion >>>>>> rows in >>>>>> >> 2.9 >>>>>> >> seconds) >>>>>> >> >>>>>> >> Can you briefly suggest the advantages of kylin over clickhouse so >>>>>> that I >>>>>> >> can defend my demonstration. >>>>>> >> >>>>>> >> On Mon, Dec 4, 2023 at 1:55 PM Xiaoxiang Yu <x...@apache.org> >>>>>> wrote: >>>>>> >> >>>>>> >> > 1. "In this important scenario of realtime analytics, the reason >>>>>> here is >>>>>> >> > that >>>>>> >> > kylin has lag time due to model update of new segment build, is >>>>>> that >>>>>> >> > correct?" >>>>>> >> > >>>>>> >> > You are correct. >>>>>> >> > >>>>>> >> > 2. "If that is true, then can you suggest a work-around of >>>>>> combination >>>>>> >> of >>>>>> >> > ... " >>>>>> >> > >>>>>> >> > Kylin is planning to introduce NRT streaming(coding is completed >>>>>> but not >>>>>> >> > released), >>>>>> >> > which can make the time-lag to about 3 minutes(that is my >>>>>> estimation >>>>>> >> but I >>>>>> >> > am >>>>>> >> > quite certain about it). >>>>>> >> > NRT stands for 'near real-time', it will run a job and do >>>>>> micro-batch >>>>>> >> > aggregation and persistence periodically. The price is that you >>>>>> need to >>>>>> >> run >>>>>> >> > and monitor a long-running >>>>>> >> > job. This feature is based on Spark Streaming, so you need >>>>>> knowledge of >>>>>> >> > it. >>>>>> >> > >>>>>> >> > I am curious about what is the maximum time-lag your customers >>>>>> >> > can tolerate? >>>>>> >> > Personally, I guess minute level time-lag is ok for most cases. >>>>>> >> > >>>>>> >> > ------------------------ >>>>>> >> > With warm regard >>>>>> >> > Xiaoxiang Yu >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > On Mon, Dec 4, 2023 at 12:28 PM Nam Đỗ Duy >>>>>> <na...@vnpay.vn.invalid> >>>>>> >> wrote: >>>>>> >> > >>>>>> >> > > Druid is better in >>>>>> >> > > - Have a real-time datasource like Kafka etc. >>>>>> >> > > >>>>>> >> > > ========================== >>>>>> >> > > >>>>>> >> > > Hi Xiaoxiang, thank you for your response. >>>>>> >> > > >>>>>> >> > > In this important scenario of realtime alalytics, the reason >>>>>> here is >>>>>> >> that >>>>>> >> > > kylin has lag time due to model update of new segment build, >>>>>> is that >>>>>> >> > > correct? >>>>>> >> > > >>>>>> >> > > If that is true, then can you suggest a work-around of >>>>>> combination of >>>>>> >> : >>>>>> >> > > >>>>>> >> > > (time - lag kylin cube) + (realtime DB update) to provide >>>>>> >> > > realtime capability ? >>>>>> >> > > >>>>>> >> > > IMO, the point here is to find that (realtime DB update) and >>>>>> >> integrate it >>>>>> >> > > with (time - lag kylin cube). >>>>>> >> > > >>>>>> >> > > On Fri, Dec 1, 2023 at 1:53 PM Xiaoxiang Yu <x...@apache.org> >>>>>> wrote: >>>>>> >> > > >>>>>> >> > > > I researched and tested Druid two years ago(I don't know too >>>>>> much >>>>>> >> about >>>>>> >> > > > the change of Druid in these two years. New features that I >>>>>> know >>>>>> >> are : >>>>>> >> > > > new UI, fully on K8s etc). >>>>>> >> > > > >>>>>> >> > > > Here are some cases you should consider using Druid other >>>>>> than Kylin >>>>>> >> > > > at the moment (using Kylin 5.0-beta to compare the Druid >>>>>> which I >>>>>> >> used >>>>>> >> > two >>>>>> >> > > > years ago): >>>>>> >> > > > >>>>>> >> > > > - Have a real-time datasource like Kafka etc. >>>>>> >> > > > - Most queries are small(Based on my test result, I think >>>>>> Druid had >>>>>> >> > > better >>>>>> >> > > > response time for small queries two years ago.) >>>>>> >> > > > - Don't know how to optimize Spark/Hadoop, want to use the >>>>>> >> K8S/public >>>>>> >> > > > cloud platform as your deployment platform. >>>>>> >> > > > >>>>>> >> > > > But I do think there are many scenarios in which Kylin could >>>>>> be >>>>>> >> better, >>>>>> >> > > > like: >>>>>> >> > > > >>>>>> >> > > > - Better performance for complex/big queries. Kylin can have >>>>>> a more >>>>>> >> > > > exact-match/fine-grained >>>>>> >> > > > Index for queries containing different `Group By >>>>>> dimensions`. >>>>>> >> > > > - User-friendly UI for modeling. >>>>>> >> > > > - Support 'Join' better? (Not sure at the moment) >>>>>> >> > > > - ODBC driver for different BI.(its website did not show it >>>>>> supports >>>>>> >> > ODBC >>>>>> >> > > > well) >>>>>> >> > > > - Looks like Kylin supports ANSI SQL better than Druid. >>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > > I don't know Pinot, so I have nothing to say about it. >>>>>> >> > > > Hope to help you, or you are free to share your opinion. >>>>>> >> > > > >>>>>> >> > > > ------------------------ >>>>>> >> > > > With warm regard >>>>>> >> > > > Xiaoxiang Yu >>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > > >>>>>> >> > > > On Fri, Dec 1, 2023 at 11:11 AM Nam Đỗ Duy >>>>>> <na...@vnpay.vn.invalid> >>>>>> >> > > wrote: >>>>>> >> > > > >>>>>> >> > > >> Dear Xiaoxiang, >>>>>> >> > > >> Sirs/Madams, >>>>>> >> > > >> >>>>>> >> > > >> May I post my boss's question: >>>>>> >> > > >> >>>>>> >> > > >> What are the pros and cons of the OLAP platform Kylin >>>>>> compared to >>>>>> >> > Pinot >>>>>> >> > > >> and >>>>>> >> > > >> Druid? >>>>>> >> > > >> >>>>>> >> > > >> Please kindly let me know >>>>>> >> > > >> >>>>>> >> > > >> Thank you very much and best regards >>>>>> >> > > >> >>>>>> >> > > > >>>>>> >> > > >>>>>> >> > >>>>>> >> >>>>>> > >>>>>> >>>>>
