Thank you XiangXiao, I still have 1 question as follows: When the Hive Datasource to be added with new data, how to reflect those in Cube (index) and query result?
On Thu, Nov 2, 2023 at 10:00 AM Xiaoxiang Yu <x...@apache.org> wrote: > Congratulations, hope you will make good use of the ability of Kylin 5 for > your use cases. > > > ------------------------ > With warm regard > Xiaoxiang Yu > > > > On Thu, Nov 2, 2023 at 10:50 AM Nam Đỗ Duy <na...@vnpay.vn.invalid> wrote: > >> The query is too fast, less than a second, can you make it a little bit >> slower so that I can see it clearly 😀😀 >> [image: image.png] >> >> On Thu, Nov 2, 2023 at 9:32 AM Nam Đỗ Duy <na...@vnpay.vn> wrote: >> >>> Thank you Xiaoxiang for the guideline. Will definitely read >>> it carefully. Kindly help the following questions: >>> >>> 1. Computed column >>> >>> I created a “computed column” and add it to dimensions (among other >>> dimensions) >>> >>> When I use query to select the computed column it returned error >>> >>> 2. Datatype optimization: will you think that the int be better than >>> string for key join columns? >>> >>> Please advise >>> >>> >>> On Wed, 1 Nov 2023 at 17:32 Xiaoxiang Yu <x...@apache.org> wrote: >>> >>>> Yes, that is almost correct. >>>> >>>> If you have a lot of complex queries, and you want to using Kylin 5 to >>>> accelerate them, the recommended steps of mine are as follows: >>>> >>>> 1. You analyse all queries and collect all join relation/pattern. >>>> 2. You create Models for each specific join relation/pattern, with the >>>> join >>>> relation you find in above step. >>>> 3. You analyse and collect dimensions and measures from all queries, and >>>> add them to the corresponding Model. >>>> 4. You build segments of all Models with proper data range. >>>> 5. You turned off the pushdown switch, and sent all queries to Kylin. If >>>> there are some queries which failed, fix them. >>>> Here are some common situations. >>>> 5.1 Join relation/pattern is not matched >>>> 5.2 If the join relation is matched, the Model might not contain >>>> every >>>> column that your query needs, please check kylin.query.log with keyword >>>> ' >>>> unmatched'. >>>> 6. (Optional) If you find some of your queries do not exactly match with >>>> your Index(your query on [colA, colB], but your index contains more >>>> columns >>>> than colA and colB), you can add some aggregate groups(or smaller Table >>>> Index) to optimize the query performance. >>>> >>>> >>>> >>>> ------------------------ >>>> With warm regard >>>> Xiaoxiang Yu >>>> >>>> >>>> >>>> On Wed, Nov 1, 2023 at 5:57 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>> wrote: >>>> >>>> > Thank you Xiaoxiang, I nearly got to the point. >>>> > >>>> > So can I interpret that: 1 model equal (~) to a set of Joins of >>>> (Dim/Fact) >>>> > table, that is to say we need to create several models according to >>>> > multiple kinds of joins queries? >>>> > >>>> > Best regards >>>> > >>>> > On Wed, Nov 1, 2023 at 4:50 PM Xiaoxiang Yu <x...@apache.org> wrote: >>>> > >>>> >> Have you ever tried to analyse the reason why your query can not hit >>>> >> Model 'sample_ssb'? >>>> >> It is because the join relation of your query is not suitable for the >>>> >> join relation/pattern of Model 'sample_ssb'. >>>> >> >>>> >> Your query used a join relation/pattern like: A inner join B. >>>> >> But the Model 'sample_ssb' used a join relation/pattern like : A >>>> inner >>>> >> join B inner join C. >>>> >> >>>> >> If you are familiar with the definition of Inner join, you may know >>>> that >>>> >> the >>>> >> relation/pattern 'A inner join B inner join C' will have a chance >>>> >> to lose some rows when compared to pattern 'A inner join B'. >>>> >> So the Model 'sample_ssb' will be excluded to serve your query. >>>> >> >>>> >> That is to say, you need to create a new model that is similar to >>>> Model >>>> >> 'sample_ssb', >>>> >> but with additional tables removed. >>>> >> >>>> >> >>>> >> >>>> >> ------------------------ >>>> >> With warm regard >>>> >> Xiaoxiang Yu >>>> >> >>>> >> >>>> >> >>>> >> On Wed, Nov 1, 2023 at 5:21 PM Nam Đỗ Duy <na...@vnpay.vn.invalid> >>>> wrote: >>>> >> >>>> >>> Hi Xiaoxiang, >>>> >>> >>>> >>> Thank you very much >>>> >>> >>>> >>> I have clearer picture of Kylin already thanks to your explanation. >>>> >>> >>>> >>> Now back to the sample project of SSB in attached photo, when I run >>>> this >>>> >>> query with push_down option OFF, why the OLAP error appears, and in >>>> such >>>> >>> case, how to create a new cube for this query? >>>> >>> >>>> >>> [image: image.png] >>>> >>> >>>> >>> On Wed, Nov 1, 2023 at 3:49 PM Xiaoxiang Yu <x...@apache.org> >>>> wrote: >>>> >>> >>>> >>>> Here is some of my explanation and it may not be perfect. >>>> >>>> Segment in Kylin is part of model/cube pre-computed data, in most >>>> >>>> cases, divided by date column. >>>> >>>> >>>> >>>> Here is some difference between Segment and Snapshot. >>>> >>>> Segment, whose source data comes from one fact table joins some >>>> dimension >>>> >>>> tables with 'specific date range', is 'precomputed', and will >>>> accelerate >>>> >>>> complex query. >>>> >>>> Snapshot, whose source data comes from one specific dimension >>>> table without >>>> >>>> specific date range, is "not precomputed", and can join with >>>> segments >>>> >>>> at runtime . >>>> >>>> >>>> >>>> - https://kylin.apache.org/5.0/docs/snapshot/snapshot_management >>>> >>>> - >>>> >>>> >>>> https://kylin.apache.org/5.0/docs/modeling/load_data/segment_operation_settings/intro >>>> >>>> >>>> >>>> ------------------------ >>>> >>>> With warm regard >>>> >>>> Xiaoxiang Yu >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Nov 1, 2023 at 3:53 PM Nam Đỗ Duy <na...@vnpay.vn> wrote: >>>> >>>> >>>> >>>>> Thank you again, very smart of you to automatically select cube >>>> for a >>>> >>>>> certain query. Sorry If I ask too much: Is the concept of Segment >>>> in Kylin >>>> >>>>> model similar to Slice-and-Dice concept of Cube, what is the >>>> different >>>> >>>>> between Kylin Segment and Kylin Snapshot? >>>> >>>>> >>>> >>>>> PS. I sent you the log files for your help in investigating why my >>>> >>>>> cube has not been used. >>>> >>>>> >>>> >>>>> On Wed, Nov 1, 2023 at 2:36 PM Xiaoxiang Yu <x...@apache.org> >>>> wrote: >>>> >>>>> >>>> >>>>>> I guess there is a misunderstanding from your sentences. >>>> >>>>>> >>>> >>>>>> -- 'I need to select Cube from a combo box below the query >>>> window' >>>> >>>>>> It is not right to use 'need', that combo box is for some >>>> specific >>>> >>>>>> cases(for example, Kylin did not choose a cube which is the most >>>> >>>>>> efficient), not the most cases. >>>> >>>>>> In most cases(both for Kylin 4 and Kylin 5), you don't need to >>>> select >>>> >>>>>> a Cube in the combo box, Kylin will do the choice for you. >>>> >>>>>> >>>> >>>>>> ------------------------ >>>> >>>>>> With warm regard >>>> >>>>>> Xiaoxiang Yu >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> On Wed, Nov 1, 2023 at 3:24 PM Nam Đỗ Duy <na...@vnpay.vn.invalid >>>> > >>>> >>>>>> wrote: >>>> >>>>>> >>>> >>>>>>> Hi Xiaoxiang, sorry if I made you confused (Anyway, it is just a >>>> >>>>>>> question of a beginner) >>>> >>>>>>> >>>> >>>>>>> "obviously" means "clearly" >>>> >>>>>>> >>>> >>>>>>> because I need to select Cube from a combo box below the query >>>> window >>>> >>>>>>> >>>> >>>>>>> Thank you very much >>>> >>>>>>> >>>> >>>>>>> On Wed, Nov 1, 2023 at 2:20 PM Xiaoxiang Yu <x...@apache.org> >>>> wrote: >>>> >>>>>>> >>>> >>>>>>>> From my side, I cannot understand why you say Kylin 4 is 'very >>>> >>>>>>>> obviously'. Can you give an example? >>>> >>>>>>>> From the source code, the basic logic of choosing the right >>>> >>>>>>>> cube/model are similar. >>>> >>>>>>>> ------------------------ >>>> >>>>>>>> With warm regard >>>> >>>>>>>> Xiaoxiang Yu >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> On Wed, Nov 1, 2023 at 3:01 PM Nam Đỗ Duy <na...@vnpay.vn> >>>> wrote: >>>> >>>>>>>> >>>> >>>>>>>>> Thank you for your kind reply, please answer 1 more question >>>> about >>>> >>>>>>>>> version 5: >>>> >>>>>>>>> >>>> >>>>>>>>> In version 4.x we run query against a Cube very obviously, >>>> but in >>>> >>>>>>>>> version 5, the cube usage is a implication socan you advise: >>>> for a given >>>> >>>>>>>>> query, which model will be used, which index (cube) will be >>>> used for this >>>> >>>>>>>>> query? >>>> >>>>>>>>> >>>> >>>>>>>>> Thank you >>>> >>>>>>>>> >>>> >>>>>>>>> On Wed, Nov 1, 2023 at 1:42 PM Xiaoxiang Yu <x...@apache.org> >>>> >>>>>>>>> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>>> 1. How do I measure the size of the index (cube) in version >>>> 5? >>>> >>>>>>>>>> You can check storage of specific Indexes from the Index >>>> page. >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> https://kylin.apache.org/5.0/docs/modeling/model_design/aggregation_group#view-aggregate-index >>>> >>>>>>>>>> or >>>> >>>>>>>>>> >>>> https://kylin.apache.org/5.0/assets/images/index_1-6ad3f55183d4ed61962359d9408ba192.png >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> 2. How to create the cardinality for each column? >>>> >>>>>>>>>> You should check this link : >>>> >>>>>>>>>> https://kylin.apache.org/5.0/docs/datasource/data_sampling/ >>>> . >>>> >>>>>>>>>> >>>> >>>>>>>>>> 3. In your default project sample named SSB project, you have >>>> >>>>>>>>>> only 4 simple aggregate group index and no table index as in >>>> attached file >>>> >>>>>>>>>> so what is the best strategy to select index for our OLAP? >>>> >>>>>>>>>> 1. There does exist a 'Base Table Index' by default >>>> >>>>>>>>>> actually, its id is 20000000001. >>>> >>>>>>>>>> 2. I think it is a good question and Kylin 5 lacks such a >>>> >>>>>>>>>> guide for better modeling. You are free to ask your question >>>> to >>>> >>>>>>>>>> mailing list and I will try to reply. >>>> >>>>>>>>>> >>>> >>>>>>>>>> ------------------------ >>>> >>>>>>>>>> With warm regard >>>> >>>>>>>>>> Xiaoxiang Yu >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> On Wed, Nov 1, 2023 at 2:12 PM Xiaoxiang Yu <x...@apache.org >>>> > >>>> >>>>>>>>>> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>>> OK, I didn't read all the mail history so I misunderstand >>>> the >>>> >>>>>>>>>>> situation. Looks like you need to analyse >>>> >>>>>>>>>>> the cause why the query didn't hit the cube correctly. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> Please generate query diagnosis package and send it to me >>>> >>>>>>>>>>> privately. I will analyse the query log. >>>> >>>>>>>>>>> You can refer to the following steps in screenshots. >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> [image: image.png] >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> If the screenshots are not displaying correctly, please read >>>> >>>>>>>>>>> this guide : >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> https://kylin.apache.org/5.0/docs/operations/system-operation/diagnosis/#generate-query-diagnosis-package-in-web-ui >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> By the way, you need to analyse the cause by reading >>>> >>>>>>>>>>> kylin.query.log, not the kylin.log, >>>> >>>>>>>>>>> refer to >>>> >>>>>>>>>>> >>>> https://kylin.apache.org/5.0/docs/operations/logs/system_log >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> ------------------------ >>>> >>>>>>>>>>> With warm regard >>>> >>>>>>>>>>> Xiaoxiang Yu >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> On Wed, Nov 1, 2023 at 12:18 PM Nam Đỗ Duy <na...@vnpay.vn> >>>> >>>>>>>>>>> wrote: >>>> >>>>>>>>>>> >>>> >>>>>>>>>>>> Thank you Xiaoxiang for your advice. As my title email >>>> shown, I >>>> >>>>>>>>>>>> guessed that the OLAP functionalities has not been >>>> correctly set up in my >>>> >>>>>>>>>>>> computer. >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> The evidence about it is that: when I disable the Pushdown >>>> >>>>>>>>>>>> option box to use solely the precomputation cube only, it >>>> showed following >>>> >>>>>>>>>>>> error: Please kindly advise how to properly build the OLAP >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> LIMIT 500": No realization found for OLAPContext, >>>> MODEL_UNMATCHED_JOIN, >>>> rel#2240:KapTableScan.OLAP.[](table=[VNEVENT_HIVE_DWH_400MILLION_ROWS, >>>> FACTUSEREVENT],ctx=0@null,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, >>>> 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]) >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>> On Wed, Nov 1, 2023 at 10:40 AM Xiaoxiang Yu < >>>> x...@apache.org> >>>> >>>>>>>>>>>> wrote: >>>> >>>>>>>>>>>> >>>> >>>>>>>>>>>>> Hi, >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Yesterday, I tried to see if query pushdown functions >>>> work >>>> >>>>>>>>>>>>> well in the Kylin5 docker, and all of my queries return >>>> proper responses . >>>> >>>>>>>>>>>>> After checking your logs from Shaofeng, I found these >>>> >>>>>>>>>>>>> error messages repeated many times: >>>> >>>>>>>>>>>>> 1. 'java.io.IOException: All datanodes >>>> >>>>>>>>>>>>> DatanodeInfoWithStorage[127.0.0.1:9866 >>>> ,DS-5093899b-06c7-4386-95d5-6fc271d92b52,DISK] >>>> >>>>>>>>>>>>> are bad. Aborting...' >>>> >>>>>>>>>>>>> 2. 'curator.ConnectionState : Connection timed out for >>>> >>>>>>>>>>>>> connection string (localhost:2181) and timeout (15000) / >>>> elapsed (41794) >>>> >>>>>>>>>>>>> org.apache.curator.CuratorConnectionLossException: >>>> >>>>>>>>>>>>> KeeperErrorCode = ConnectionLoss' >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> I guess the root cause is that the container didn't >>>> not >>>> >>>>>>>>>>>>> have enough resources. I found you query on a table called >>>> >>>>>>>>>>>>> 'XXX_hive_dwh_400million_rows', looks like you gave a >>>> complex query on a >>>> >>>>>>>>>>>>> table which contains 400 million rows? >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Since I am the uploader of kylin5 's docker image, I >>>> want >>>> >>>>>>>>>>>>> to give some explainment. Kylin5 docker is not a place >>>> for performance >>>> >>>>>>>>>>>>> benchmarks, it is only for demonstration. It is only >>>> allocated with very >>>> >>>>>>>>>>>>> little resources(8G memory) if you are using the default >>>> command from >>>> >>>>>>>>>>>>> docker hub page. Before I uploaded my image, I only >>>> tested my image using >>>> >>>>>>>>>>>>> the ssb dataset, which the biggest table only contains >>>> about 60k rows. If >>>> >>>>>>>>>>>>> you are using a larger dataset and complexer queries, you >>>> have to scale the >>>> >>>>>>>>>>>>> resource properly. Try querying tables which contain not >>>> more than 100k >>>> >>>>>>>>>>>>> rows by default. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> Here are some tips which may help you to check if the >>>> >>>>>>>>>>>>> daemon service is in health status and >>>> resources(particularly disk space) >>>> >>>>>>>>>>>>> is configured properly. >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> 1. Checking HDFS 's web ui( >>>> >>>>>>>>>>>>> http://localhost:9870/dfshealth.html#tab-datanode ) to >>>> >>>>>>>>>>>>> confirm whether HDFS service is in 'In service' status. >>>> >>>>>>>>>>>>> 2. Checking Datanode 's log in >>>> >>>>>>>>>>>>> >>>> `/opt/hadoop-3.2.1/logs/hadoop-root-datanode-Kylin5-Machine.log`, check if >>>> >>>>>>>>>>>>> there is any error message. Like: cat >>>> >>>>>>>>>>>>> >>>> /opt/hadoop-3.2.1/logs/hadoop-root-datanode-Kylin5-Machine.log | grep ERROR >>>> >>>>>>>>>>>>> | wc -l >>>> >>>>>>>>>>>>> 3. Checking if your docker engine is configured with >>>> >>>>>>>>>>>>> enough disk space, if you are using Docker Desktop like >>>> me,please go to >>>> >>>>>>>>>>>>> "Settings" - "Resources" - "Advanced", make sure you have >>>> allocated 40GB+ >>>> >>>>>>>>>>>>> disk space to the docker container. >>>> >>>>>>>>>>>>> 4. Checking the available disk space of your >>>> container by >>>> >>>>>>>>>>>>> `df -h`, make sure the 'Use%' of 'overlay' is less than >>>> 60% . >>>> >>>>>>>>>>>>> 5. Checking the load average/ cpu usage/ jvm gc. Make >>>> sure >>>> >>>>>>>>>>>>> these metrics are not really high when you send a query. >>>> >>>>>>>>>>>>> ------------------------ >>>> >>>>>>>>>>>>> With warm regard >>>> >>>>>>>>>>>>> Xiaoxiang Yu >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>> On Tue, Oct 31, 2023 at 5:13 PM Nam Đỗ Duy >>>> >>>>>>>>>>>>> <na...@vnpay.vn.invalid> wrote: >>>> >>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> Hi ShaoFeng >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> Thank you very much for your valuable feedback >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> I saw the application to be there (if I see it right) as >>>> in >>>> >>>>>>>>>>>>>> the attachment photo. Kindly advise so that I can run >>>> this query on OLAP. >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> PS. I sent you the log file in private. >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> [image: image.png] >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 3:11 PM ShaoFeng Shi < >>>> >>>>>>>>>>>>>> shaofeng...@apache.org> wrote: >>>> >>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> Can you provide the messages in logs/kylin.log when >>>> >>>>>>>>>>>>>>> executing the SQL? and you can also check the Spark UI >>>> from yarn resource >>>> >>>>>>>>>>>>>>> manager (there should be one running application called >>>> Spardar, which is >>>> >>>>>>>>>>>>>>> Kylin's backend spark application). If the application >>>> is not there, it may >>>> >>>>>>>>>>>>>>> indicates the yarn doesn't have resource to startup it. >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> Best regards, >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> Shaofeng Shi 史少锋 >>>> >>>>>>>>>>>>>>> Apache Kylin PMC, >>>> >>>>>>>>>>>>>>> Apache Incubator PMC, >>>> >>>>>>>>>>>>>>> Email: shaofeng...@apache.org >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> Apache Kylin FAQ: >>>> >>>>>>>>>>>>>>> https://kylin.apache.org/docs/gettingstarted/faq.html >>>> >>>>>>>>>>>>>>> Join Kylin user mail group: >>>> user-subscr...@kylin.apache.org >>>> >>>>>>>>>>>>>>> Join Kylin dev mail group: >>>> dev-subscr...@kylin.apache.org >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> Nam Đỗ Duy <na...@vnpay.vn> 于2023年10月31日周二 10:35写道： >>>> >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Dear Sir/Madam, >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I have a fact with 500million rows then I build model, >>>> >>>>>>>>>>>>>>>> index according to the website help. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I chose full incremental because this is the first >>>> times I >>>> >>>>>>>>>>>>>>>> load data >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> I create both index types Aggregate group index, table >>>> >>>>>>>>>>>>>>>> index as photo attached. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> But the query always failed after timeout of 300 >>>> seconds (I >>>> >>>>>>>>>>>>>>>> run in docker), I dont want to increase the value of >>>> 300 seconds because I >>>> >>>>>>>>>>>>>>>> wish the OLAP can run within 1 minutes (is that >>>> possible?) >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> It seems that the OLAP function in indexing not >>>> working to >>>> >>>>>>>>>>>>>>>> speedup the query by precomputed cube. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Can you advise to check whether the index did really >>>> work? >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> It is quite urgent task for me so prompt response is >>>> highly >>>> >>>>>>>>>>>>>>>> appreciated. >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>>> Thank you very much >>>> >>>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> >>>
