1. How can I automate the build index daily for newest data?
I guess your team/teammate will manage a ETL pipeline(Jenkins,
DolphinScheduler etc),
you may call Kylin by a rest api in your pipeline, here is the link:
https://kylin.apache.org/5.0/docs/restapi/segment_management_api
2. Can I apply the above automate process for near realtime or realtime
data?
There will be a latency of about 10 min to 2 hours in most cases, it
depends on how fast
the build index job is completed.
------------------------
With warm regard
Xiaoxiang Yu
On Thu, Nov 2, 2023 at 2:10 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> wrote:
> Hi Xiaoxiang
>
> My case is not in date range and I need to do daily.
>
> 1. How can I automate the build index daily for newest data?
>
> 2. Can I apply the above automate process for near realtime or realtime
> data (load realtime data from Hive into new index/segment)
>
> Thank you very much for your help
>
>
> On Thu, 2 Nov 2023 at 12:58 Xiaoxiang Yu <x...@apache.org> wrote:
>
> > If the new data 's date range is covered by a segment in your model, you
> > should refresh your existing segment, refer to :
> >
> >
> >
> https://kylin.apache.org/5.0/docs/modeling/load_data/segment_operation_settings/intro#segment-operation
> > .
> >
> > If not, create a new segment and build index, refer to :
> > https://kylin.apache.org/5.0/docs/modeling/load_data/by_date
> >
> > ------------------------
> > With warm regard
> > Xiaoxiang Yu
> >
> >
> >
> > On Thu, Nov 2, 2023 at 11:57 AM Nam Đỗ Duy <na...@vnpay.vn.invalid>
> wrote:
> >
> > > Thank you XiangXiao, I still have 1 question as follows:
> > >
> > > When the Hive Datasource to be added with new data, how to reflect
> those
> > in
> > > Cube (index) and query result?
> > >
> > >
> > > On Thu, Nov 2, 2023 at 10:00 AM Xiaoxiang Yu <x...@apache.org> wrote:
> > >
> > > > Congratulations, hope you will make good use of the ability of Kylin
> 5
> > > for
> > > > your use cases.
> > > >
> > > >
> > > > ------------------------
> > > > With warm regard
> > > > Xiaoxiang Yu
> > > >
> > > >
> > > >
> > > > On Thu, Nov 2, 2023 at 10:50 AM Nam Đỗ Duy <na...@vnpay.vn.invalid>
> > > wrote:
> > > >
> > > >> The query is too fast, less than a second, can you make it a little
> > bit
> > > >> slower so that I can see it clearly 😀😀
> > > >> [image: image.png]
> > > >>
> > > >> On Thu, Nov 2, 2023 at 9:32 AM Nam Đỗ Duy <na...@vnpay.vn> wrote:
> > > >>
> > > >>> Thank you Xiaoxiang for the guideline. Will definitely read
> > > >>> it carefully. Kindly help the following questions:
> > > >>>
> > > >>> 1. Computed column
> > > >>>
> > > >>> I created a “computed column” and add it to dimensions (among other
> > > >>> dimensions)
> > > >>>
> > > >>> When I use query to select the computed column it returned error
> > > >>>
> > > >>> 2. Datatype optimization: will you think that the int be better
> than
> > > >>> string for key join columns?
> > > >>>
> > > >>> Please advise
> > > >>>
> > > >>>
> > > >>> On Wed, 1 Nov 2023 at 17:32 Xiaoxiang Yu <x...@apache.org> wrote:
> > > >>>
> > > >>>> Yes, that is almost correct.
> > > >>>>
> > > >>>> If you have a lot of complex queries, and you want to using Kylin
> 5
> > to
> > > >>>> accelerate them, the recommended steps of mine are as follows:
> > > >>>>
> > > >>>> 1. You analyse all queries and collect all join relation/pattern.
> > > >>>> 2. You create Models for each specific join relation/pattern, with
> > the
> > > >>>> join
> > > >>>> relation you find in above step.
> > > >>>> 3. You analyse and collect dimensions and measures from all
> queries,
> > > and
> > > >>>> add them to the corresponding Model.
> > > >>>> 4. You build segments of all Models with proper data range.
> > > >>>> 5. You turned off the pushdown switch, and sent all queries to
> > Kylin.
> > > If
> > > >>>> there are some queries which failed, fix them.
> > > >>>> Here are some common situations.
> > > >>>> 5.1 Join relation/pattern is not matched
> > > >>>> 5.2 If the join relation is matched, the Model might not
> contain
> > > >>>> every
> > > >>>> column that your query needs, please check kylin.query.log with
> > > keyword
> > > >>>> '
> > > >>>> unmatched'.
> > > >>>> 6. (Optional) If you find some of your queries do not exactly
> match
> > > with
> > > >>>> your Index(your query on [colA, colB], but your index contains
> more
> > > >>>> columns
> > > >>>> than colA and colB), you can add some aggregate groups(or smaller
> > > Table
> > > >>>> Index) to optimize the query performance.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> ------------------------
> > > >>>> With warm regard
> > > >>>> Xiaoxiang Yu
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Wed, Nov 1, 2023 at 5:57 PM Nam Đỗ Duy <na...@vnpay.vn.invalid
> >
> > > >>>> wrote:
> > > >>>>
> > > >>>> > Thank you Xiaoxiang, I nearly got to the point.
> > > >>>> >
> > > >>>> > So can I interpret that: 1 model equal (~) to a set of Joins of
> > > >>>> (Dim/Fact)
> > > >>>> > table, that is to say we need to create several models according
> > to
> > > >>>> > multiple kinds of joins queries?
> > > >>>> >
> > > >>>> > Best regards
> > > >>>> >
> > > >>>> > On Wed, Nov 1, 2023 at 4:50 PM Xiaoxiang Yu <x...@apache.org>
> > > wrote:
> > > >>>> >
> > > >>>> >> Have you ever tried to analyse the reason why your query can
> not
> > > hit
> > > >>>> >> Model 'sample_ssb'?
> > > >>>> >> It is because the join relation of your query is not suitable
> for
> > > the
> > > >>>> >> join relation/pattern of Model 'sample_ssb'.
> > > >>>> >>
> > > >>>> >> Your query used a join relation/pattern like: A inner join B.
> > > >>>> >> But the Model 'sample_ssb' used a join relation/pattern like :
> A
> > > >>>> inner
> > > >>>> >> join B inner join C.
> > > >>>> >>
> > > >>>> >> If you are familiar with the definition of Inner join, you may
> > know
> > > >>>> that
> > > >>>> >> the
> > > >>>> >> relation/pattern 'A inner join B inner join C' will have a
> chance
> > > >>>> >> to lose some rows when compared to pattern 'A inner join B'.
> > > >>>> >> So the Model 'sample_ssb' will be excluded to serve your query.
> > > >>>> >>
> > > >>>> >> That is to say, you need to create a new model that is similar
> to
> > > >>>> Model
> > > >>>> >> 'sample_ssb',
> > > >>>> >> but with additional tables removed.
> > > >>>> >>
> > > >>>> >>
> > > >>>> >>
> > > >>>> >> ------------------------
> > > >>>> >> With warm regard
> > > >>>> >> Xiaoxiang Yu
> > > >>>> >>
> > > >>>> >>
> > > >>>> >>
> > > >>>> >> On Wed, Nov 1, 2023 at 5:21 PM Nam Đỗ Duy
> <na...@vnpay.vn.invalid
> > >
> > > >>>> wrote:
> > > >>>> >>
> > > >>>> >>> Hi Xiaoxiang,
> > > >>>> >>>
> > > >>>> >>> Thank you very much
> > > >>>> >>>
> > > >>>> >>> I have clearer picture of Kylin already thanks to your
> > > explanation.
> > > >>>> >>>
> > > >>>> >>> Now back to the sample project of SSB in attached photo, when
> I
> > > run
> > > >>>> this
> > > >>>> >>> query with push_down option OFF, why the OLAP error appears,
> and
> > > in
> > > >>>> such
> > > >>>> >>> case, how to create a new cube for this query?
> > > >>>> >>>
> > > >>>> >>> [image: image.png]
> > > >>>> >>>
> > > >>>> >>> On Wed, Nov 1, 2023 at 3:49 PM Xiaoxiang Yu <x...@apache.org>
> > > >>>> wrote:
> > > >>>> >>>
> > > >>>> >>>> Here is some of my explanation and it may not be perfect.
> > > >>>> >>>> Segment in Kylin is part of model/cube pre-computed data, in
> > most
> > > >>>> >>>> cases, divided by date column.
> > > >>>> >>>>
> > > >>>> >>>> Here is some difference between Segment and Snapshot.
> > > >>>> >>>> Segment, whose source data comes from one fact table joins
> some
> > > >>>> dimension
> > > >>>> >>>> tables with 'specific date range', is 'precomputed', and will
> > > >>>> accelerate
> > > >>>> >>>> complex query.
> > > >>>> >>>> Snapshot, whose source data comes from one specific dimension
> > > >>>> table without
> > > >>>> >>>> specific date range, is "not precomputed", and can join with
> > > >>>> segments
> > > >>>> >>>> at runtime .
> > > >>>> >>>>
> > > >>>> >>>> -
> > https://kylin.apache.org/5.0/docs/snapshot/snapshot_management
> > > >>>> >>>> -
> > > >>>> >>>>
> > > >>>>
> > >
> >
> https://kylin.apache.org/5.0/docs/modeling/load_data/segment_operation_settings/intro
> > > >>>> >>>>
> > > >>>> >>>> ------------------------
> > > >>>> >>>> With warm regard
> > > >>>> >>>> Xiaoxiang Yu
> > > >>>> >>>>
> > > >>>> >>>>
> > > >>>> >>>>
> > > >>>> >>>> On Wed, Nov 1, 2023 at 3:53 PM Nam Đỗ Duy <na...@vnpay.vn>
> > > wrote:
> > > >>>> >>>>
> > > >>>> >>>>> Thank you again, very smart of you to automatically select
> > cube
> > > >>>> for a
> > > >>>> >>>>> certain query. Sorry If I ask too much: Is the concept of
> > > Segment
> > > >>>> in Kylin
> > > >>>> >>>>> model similar to Slice-and-Dice concept of Cube, what is the
> > > >>>> different
> > > >>>> >>>>> between Kylin Segment and Kylin Snapshot?
> > > >>>> >>>>>
> > > >>>> >>>>> PS. I sent you the log files for your help in investigating
> > why
> > > my
> > > >>>> >>>>> cube has not been used.
> > > >>>> >>>>>
> > > >>>> >>>>> On Wed, Nov 1, 2023 at 2:36 PM Xiaoxiang Yu <
> x...@apache.org>
> > > >>>> wrote:
> > > >>>> >>>>>
> > > >>>> >>>>>> I guess there is a misunderstanding from your sentences.
> > > >>>> >>>>>>
> > > >>>> >>>>>> -- 'I need to select Cube from a combo box below the query
> > > >>>> window'
> > > >>>> >>>>>> It is not right to use 'need', that combo box is for some
> > > >>>> specific
> > > >>>> >>>>>> cases(for example, Kylin did not choose a cube which is the
> > > most
> > > >>>> >>>>>> efficient), not the most cases.
> > > >>>> >>>>>> In most cases(both for Kylin 4 and Kylin 5), you don't need
> > to
> > > >>>> select
> > > >>>> >>>>>> a Cube in the combo box, Kylin will do the choice for you.
> > > >>>> >>>>>>
> > > >>>> >>>>>> ------------------------
> > > >>>> >>>>>> With warm regard
> > > >>>> >>>>>> Xiaoxiang Yu
> > > >>>> >>>>>>
> > > >>>> >>>>>>
> > > >>>> >>>>>>
> > > >>>> >>>>>> On Wed, Nov 1, 2023 at 3:24 PM Nam Đỗ Duy
> > > <na...@vnpay.vn.invalid
> > > >>>> >
> > > >>>> >>>>>> wrote:
> > > >>>> >>>>>>
> > > >>>> >>>>>>> Hi Xiaoxiang, sorry if I made you confused (Anyway, it is
> > > just a
> > > >>>> >>>>>>> question of a beginner)
> > > >>>> >>>>>>>
> > > >>>> >>>>>>> "obviously" means "clearly"
> > > >>>> >>>>>>>
> > > >>>> >>>>>>> because I need to select Cube from a combo box below the
> > query
> > > >>>> window
> > > >>>> >>>>>>>
> > > >>>> >>>>>>> Thank you very much
> > > >>>> >>>>>>>
> > > >>>> >>>>>>> On Wed, Nov 1, 2023 at 2:20 PM Xiaoxiang Yu <
> > x...@apache.org>
> > > >>>> wrote:
> > > >>>> >>>>>>>
> > > >>>> >>>>>>>> From my side, I cannot understand why you say Kylin 4 is
> > > 'very
> > > >>>> >>>>>>>> obviously'. Can you give an example?
> > > >>>> >>>>>>>> From the source code, the basic logic of choosing the
> right
> > > >>>> >>>>>>>> cube/model are similar.
> > > >>>> >>>>>>>> ------------------------
> > > >>>> >>>>>>>> With warm regard
> > > >>>> >>>>>>>> Xiaoxiang Yu
> > > >>>> >>>>>>>>
> > > >>>> >>>>>>>>
> > > >>>> >>>>>>>>
> > > >>>> >>>>>>>> On Wed, Nov 1, 2023 at 3:01 PM Nam Đỗ Duy <
> na...@vnpay.vn>
> > > >>>> wrote:
> > > >>>> >>>>>>>>
> > > >>>> >>>>>>>>> Thank you for your kind reply, please answer 1 more
> > question
> > > >>>> about
> > > >>>> >>>>>>>>> version 5:
> > > >>>> >>>>>>>>>
> > > >>>> >>>>>>>>> In version 4.x we run query against a Cube very
> obviously,
> > > >>>> but in
> > > >>>> >>>>>>>>> version 5, the cube usage is a implication socan you
> > advise:
> > > >>>> for a given
> > > >>>> >>>>>>>>> query, which model will be used, which index (cube) will
> > be
> > > >>>> used for this
> > > >>>> >>>>>>>>> query?
> > > >>>> >>>>>>>>>
> > > >>>> >>>>>>>>> Thank you
> > > >>>> >>>>>>>>>
> > > >>>> >>>>>>>>> On Wed, Nov 1, 2023 at 1:42 PM Xiaoxiang Yu <
> > > x...@apache.org>
> > > >>>> >>>>>>>>> wrote:
> > > >>>> >>>>>>>>>
> > > >>>> >>>>>>>>>> 1. How do I measure the size of the index (cube) in
> > version
> > > >>>> 5?
> > > >>>> >>>>>>>>>> You can check storage of specific Indexes from the
> > Index
> > > >>>> page.
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>>
> > > >>>>
> > >
> >
> https://kylin.apache.org/5.0/docs/modeling/model_design/aggregation_group#view-aggregate-index
> > > >>>> >>>>>>>>>> or
> > > >>>> >>>>>>>>>>
> > > >>>>
> > >
> >
> https://kylin.apache.org/5.0/assets/images/index_1-6ad3f55183d4ed61962359d9408ba192.png
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>> 2. How to create the cardinality for each column?
> > > >>>> >>>>>>>>>> You should check this link :
> > > >>>> >>>>>>>>>>
> > > https://kylin.apache.org/5.0/docs/datasource/data_sampling/
> > > >>>> .
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>> 3. In your default project sample named SSB project,
> you
> > > have
> > > >>>> >>>>>>>>>> only 4 simple aggregate group index and no table index
> as
> > > in
> > > >>>> attached file
> > > >>>> >>>>>>>>>> so what is the best strategy to select index for our
> > OLAP?
> > > >>>> >>>>>>>>>> 1. There does exist a 'Base Table Index' by
> default
> > > >>>> >>>>>>>>>> actually, its id is 20000000001.
> > > >>>> >>>>>>>>>> 2. I think it is a good question and Kylin 5 lacks
> > > such a
> > > >>>> >>>>>>>>>> guide for better modeling. You are free to ask your
> > > question
> > > >>>> to
> > > >>>> >>>>>>>>>> mailing list and I will try to reply.
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>> ------------------------
> > > >>>> >>>>>>>>>> With warm regard
> > > >>>> >>>>>>>>>> Xiaoxiang Yu
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>> On Wed, Nov 1, 2023 at 2:12 PM Xiaoxiang Yu <
> > > x...@apache.org
> > > >>>> >
> > > >>>> >>>>>>>>>> wrote:
> > > >>>> >>>>>>>>>>
> > > >>>> >>>>>>>>>>> OK, I didn't read all the mail history so I
> > misunderstand
> > > >>>> the
> > > >>>> >>>>>>>>>>> situation. Looks like you need to analyse
> > > >>>> >>>>>>>>>>> the cause why the query didn't hit the cube correctly.
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> Please generate query diagnosis package and send it to
> > me
> > > >>>> >>>>>>>>>>> privately. I will analyse the query log.
> > > >>>> >>>>>>>>>>> You can refer to the following steps in screenshots.
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> [image: image.png]
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> If the screenshots are not displaying correctly,
> please
> > > read
> > > >>>> >>>>>>>>>>> this guide :
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>>
> > > >>>>
> > >
> >
> https://kylin.apache.org/5.0/docs/operations/system-operation/diagnosis/#generate-query-diagnosis-package-in-web-ui
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> By the way, you need to analyse the cause by reading
> > > >>>> >>>>>>>>>>> kylin.query.log, not the kylin.log,
> > > >>>> >>>>>>>>>>> refer to
> > > >>>> >>>>>>>>>>>
> > > >>>> https://kylin.apache.org/5.0/docs/operations/logs/system_log
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> ------------------------
> > > >>>> >>>>>>>>>>> With warm regard
> > > >>>> >>>>>>>>>>> Xiaoxiang Yu
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>> On Wed, Nov 1, 2023 at 12:18 PM Nam Đỗ Duy <
> > > na...@vnpay.vn>
> > > >>>> >>>>>>>>>>> wrote:
> > > >>>> >>>>>>>>>>>
> > > >>>> >>>>>>>>>>>> Thank you Xiaoxiang for your advice. As my title
> email
> > > >>>> shown, I
> > > >>>> >>>>>>>>>>>> guessed that the OLAP functionalities has not been
> > > >>>> correctly set up in my
> > > >>>> >>>>>>>>>>>> computer.
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>> The evidence about it is that: when I disable the
> > > Pushdown
> > > >>>> >>>>>>>>>>>> option box to use solely the precomputation cube
> only,
> > it
> > > >>>> showed following
> > > >>>> >>>>>>>>>>>> error: Please kindly advise how to properly build the
> > > OLAP
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>> LIMIT 500": No realization found for OLAPContext,
> > > >>>> MODEL_UNMATCHED_JOIN,
> > > >>>>
> > rel#2240:KapTableScan.OLAP.[](table=[VNEVENT_HIVE_DWH_400MILLION_ROWS,
> > > >>>> FACTUSEREVENT],ctx=0@null,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
> 10,
> > > >>>> 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>> On Wed, Nov 1, 2023 at 10:40 AM Xiaoxiang Yu <
> > > >>>> x...@apache.org>
> > > >>>> >>>>>>>>>>>> wrote:
> > > >>>> >>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> Hi,
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> Yesterday, I tried to see if query pushdown
> > > functions
> > > >>>> work
> > > >>>> >>>>>>>>>>>>> well in the Kylin5 docker, and all of my queries
> > return
> > > >>>> proper responses .
> > > >>>> >>>>>>>>>>>>> After checking your logs from Shaofeng, I found
> > > these
> > > >>>> >>>>>>>>>>>>> error messages repeated many times:
> > > >>>> >>>>>>>>>>>>> 1. 'java.io.IOException: All datanodes
> > > >>>> >>>>>>>>>>>>> DatanodeInfoWithStorage[127.0.0.1:9866
> > > >>>> ,DS-5093899b-06c7-4386-95d5-6fc271d92b52,DISK]
> > > >>>> >>>>>>>>>>>>> are bad. Aborting...'
> > > >>>> >>>>>>>>>>>>> 2. 'curator.ConnectionState : Connection timed
> out
> > > for
> > > >>>> >>>>>>>>>>>>> connection string (localhost:2181) and timeout
> > (15000) /
> > > >>>> elapsed (41794)
> > > >>>> >>>>>>>>>>>>> org.apache.curator.CuratorConnectionLossException:
> > > >>>> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss'
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> I guess the root cause is that the container
> > didn't
> > > >>>> not
> > > >>>> >>>>>>>>>>>>> have enough resources. I found you query on a table
> > > called
> > > >>>> >>>>>>>>>>>>> 'XXX_hive_dwh_400million_rows', looks like you gave
> a
> > > >>>> complex query on a
> > > >>>> >>>>>>>>>>>>> table which contains 400 million rows?
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> Since I am the uploader of kylin5 's docker
> > image, I
> > > >>>> want
> > > >>>> >>>>>>>>>>>>> to give some explainment. Kylin5 docker is not a
> place
> > > >>>> for performance
> > > >>>> >>>>>>>>>>>>> benchmarks, it is only for demonstration. It is only
> > > >>>> allocated with very
> > > >>>> >>>>>>>>>>>>> little resources(8G memory) if you are using the
> > default
> > > >>>> command from
> > > >>>> >>>>>>>>>>>>> docker hub page. Before I uploaded my image, I only
> > > >>>> tested my image using
> > > >>>> >>>>>>>>>>>>> the ssb dataset, which the biggest table only
> contains
> > > >>>> about 60k rows. If
> > > >>>> >>>>>>>>>>>>> you are using a larger dataset and complexer
> queries,
> > > you
> > > >>>> have to scale the
> > > >>>> >>>>>>>>>>>>> resource properly. Try querying tables which contain
> > not
> > > >>>> more than 100k
> > > >>>> >>>>>>>>>>>>> rows by default.
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> Here are some tips which may help you to check
> if
> > > the
> > > >>>> >>>>>>>>>>>>> daemon service is in health status and
> > > >>>> resources(particularly disk space)
> > > >>>> >>>>>>>>>>>>> is configured properly.
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> 1. Checking HDFS 's web ui(
> > > >>>> >>>>>>>>>>>>> http://localhost:9870/dfshealth.html#tab-datanode )
> > to
> > > >>>> >>>>>>>>>>>>> confirm whether HDFS service is in 'In service'
> > status.
> > > >>>> >>>>>>>>>>>>> 2. Checking Datanode 's log in
> > > >>>> >>>>>>>>>>>>>
> > > >>>> `/opt/hadoop-3.2.1/logs/hadoop-root-datanode-Kylin5-Machine.log`,
> > > check if
> > > >>>> >>>>>>>>>>>>> there is any error message. Like: cat
> > > >>>> >>>>>>>>>>>>>
> > > >>>> /opt/hadoop-3.2.1/logs/hadoop-root-datanode-Kylin5-Machine.log |
> > grep
> > > ERROR
> > > >>>> >>>>>>>>>>>>> | wc -l
> > > >>>> >>>>>>>>>>>>> 3. Checking if your docker engine is configured
> > with
> > > >>>> >>>>>>>>>>>>> enough disk space, if you are using Docker Desktop
> > like
> > > >>>> me,please go to
> > > >>>> >>>>>>>>>>>>> "Settings" - "Resources" - "Advanced", make sure you
> > > have
> > > >>>> allocated 40GB+
> > > >>>> >>>>>>>>>>>>> disk space to the docker container.
> > > >>>> >>>>>>>>>>>>> 4. Checking the available disk space of your
> > > >>>> container by
> > > >>>> >>>>>>>>>>>>> `df -h`, make sure the 'Use%' of 'overlay' is less
> > than
> > > >>>> 60% .
> > > >>>> >>>>>>>>>>>>> 5. Checking the load average/ cpu usage/ jvm gc.
> > > Make
> > > >>>> sure
> > > >>>> >>>>>>>>>>>>> these metrics are not really high when you send a
> > query.
> > > >>>> >>>>>>>>>>>>> ------------------------
> > > >>>> >>>>>>>>>>>>> With warm regard
> > > >>>> >>>>>>>>>>>>> Xiaoxiang Yu
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>> On Tue, Oct 31, 2023 at 5:13 PM Nam Đỗ Duy
> > > >>>> >>>>>>>>>>>>> <na...@vnpay.vn.invalid> wrote:
> > > >>>> >>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> Hi ShaoFeng
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> Thank you very much for your valuable feedback
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> I saw the application to be there (if I see it
> right)
> > > as
> > > >>>> in
> > > >>>> >>>>>>>>>>>>>> the attachment photo. Kindly advise so that I can
> run
> > > >>>> this query on OLAP.
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> PS. I sent you the log file in private.
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> [image: image.png]
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 3:11 PM ShaoFeng Shi <
> > > >>>> >>>>>>>>>>>>>> shaofeng...@apache.org> wrote:
> > > >>>> >>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>> Can you provide the messages in logs/kylin.log
> when
> > > >>>> >>>>>>>>>>>>>>> executing the SQL? and you can also check the
> Spark
> > UI
> > > >>>> from yarn resource
> > > >>>> >>>>>>>>>>>>>>> manager (there should be one running application
> > > called
> > > >>>> Spardar, which is
> > > >>>> >>>>>>>>>>>>>>> Kylin's backend spark application). If the
> > application
> > > >>>> is not there, it may
> > > >>>> >>>>>>>>>>>>>>> indicates the yarn doesn't have resource to
> startup
> > > it.
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>> Best regards,
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>> Shaofeng Shi 史少锋
> > > >>>> >>>>>>>>>>>>>>> Apache Kylin PMC,
> > > >>>> >>>>>>>>>>>>>>> Apache Incubator PMC,
> > > >>>> >>>>>>>>>>>>>>> Email: shaofeng...@apache.org
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>> Apache Kylin FAQ:
> > > >>>> >>>>>>>>>>>>>>>
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > > >>>> >>>>>>>>>>>>>>> Join Kylin user mail group:
> > > >>>> user-subscr...@kylin.apache.org
> > > >>>> >>>>>>>>>>>>>>> Join Kylin dev mail group:
> > > >>>> dev-subscr...@kylin.apache.org
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>> Nam Đỗ Duy <na...@vnpay.vn> 于2023年10月31日周二
> 10:35写道：
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> Dear Sir/Madam,
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> I have a fact with 500million rows then I build
> > > model,
> > > >>>> >>>>>>>>>>>>>>>> index according to the website help.
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> I chose full incremental because this is the
> first
> > > >>>> times I
> > > >>>> >>>>>>>>>>>>>>>> load data
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> I create both index types Aggregate group index,
> > > table
> > > >>>> >>>>>>>>>>>>>>>> index as photo attached.
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> But the query always failed after timeout of 300
> > > >>>> seconds (I
> > > >>>> >>>>>>>>>>>>>>>> run in docker), I dont want to increase the value
> > of
> > > >>>> 300 seconds because I
> > > >>>> >>>>>>>>>>>>>>>> wish the OLAP can run within 1 minutes (is that
> > > >>>> possible?)
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> It seems that the OLAP function in indexing not
> > > >>>> working to
> > > >>>> >>>>>>>>>>>>>>>> speedup the query by precomputed cube.
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> Can you advise to check whether the index did
> > really
> > > >>>> work?
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> It is quite urgent task for me so prompt response
> > is
> > > >>>> highly
> > > >>>> >>>>>>>>>>>>>>>> appreciated.
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>> Thank you very much
> > > >>>> >>>>>>>>>>>>>>>>
> > > >>>> >>>>>>>>>>>>>>>
> > > >>>>
> > > >>>
> > >
> >
>
