[jira] [Created] (KYLIN-3633) Dead lock may happen in building global dictionary
nichunen created KYLIN-3633: --- Summary: Dead lock may happen in building global dictionary Key: KYLIN-3633 URL: https://issues.apache.org/jira/browse/KYLIN-3633 Project: Kylin Issue Type: Bug Components: Measure - Count Distinct Affects Versions: v2.4.1 Reporter: nichunen Assignee: nichunen During the building of global dict, if an error happens, the builder's lock maybe un-released. This will block all other building jobs with global dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[ https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651086#comment-16651086 ] Shaofeng SHI commented on KYLIN-3297: - can you change its type to Date and try again? You can also read this blog: https://blog.bcmeng.com/post/kylin-sql-type.html > sql在解析>大值 and <小值的时候，kylin出现内存溢出。 > - > > Key: KYLIN-3297 > URL: https://issues.apache.org/jira/browse/KYLIN-3297 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: oracle linux 6.8 > cup 8C > memory 48G > apache kylin v2.2.0 >Reporter: 陈家宇 >Priority: Major > Fix For: v2.6.0 > > > select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a, > case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from ( > select h.business_area_desc,h.plant_code, > case when p.report_business_area like '%切大片%' then '切大片' > else p.report_business_area end report_business_area, > case when p.report_business_area like '%切大片%' then '00' > else p.orderby end orderby, > sum(p.quantity) qty,sum(a)a from pkn_kpi p > inner join hcm_area h on p.plant_id=h.plant_id > where 1=1 > and h.business_area_desc='上海' > and h.plant_code='S1' > and p.calendar_day>='2018-03-19' > and p.calendar_day<='2018-03-18' > and trim(p.report_business_area) is not null > group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby > ) f > group by f.plant_code,f.report_business_area,f.orderby > order by orderby > 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and > calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651078#comment-16651078
]
huaicui commented on KYLIN-3601:
As we know, create a query instance is very impact performance；We use
PrepareStatement that help us improve a little of performance, but this
pattern also often create query instance that limit throughput rate;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[ https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651063#comment-16651063 ] 陈家宇 commented on KYLIN-3297: type of "calendar_day" is varchar > sql在解析>大值 and <小值的时候，kylin出现内存溢出。 > - > > Key: KYLIN-3297 > URL: https://issues.apache.org/jira/browse/KYLIN-3297 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: oracle linux 6.8 > cup 8C > memory 48G > apache kylin v2.2.0 >Reporter: 陈家宇 >Priority: Major > Fix For: v2.6.0 > > > select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a, > case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from ( > select h.business_area_desc,h.plant_code, > case when p.report_business_area like '%切大片%' then '切大片' > else p.report_business_area end report_business_area, > case when p.report_business_area like '%切大片%' then '00' > else p.orderby end orderby, > sum(p.quantity) qty,sum(a)a from pkn_kpi p > inner join hcm_area h on p.plant_id=h.plant_id > where 1=1 > and h.business_area_desc='上海' > and h.plant_code='S1' > and p.calendar_day>='2018-03-19' > and p.calendar_day<='2018-03-18' > and trim(p.report_business_area) is not null > group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby > ) f > group by f.plant_code,f.report_business_area,f.orderby > order by orderby > 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and > calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649978#comment-16649978
]
Ma Gang commented on KYLIN-3601:
I use apache ab to do some test: ab -n 10 -c 2 -T
'application/json;charset=UTF-8' -p query_request.json -A
[http://localhost/kylin/api/query]
the result is:
Document Path: /kylin/api/query
Document Length: 951 bytes
Concurrency Level: 2
Time taken for tests: 739.392 seconds
Complete requests: 1
Failed requests: 3746
(Connect: 0, Receive: 0, Length: 3746, Exceptions: 0)
Write errors: 0
Non-2xx responses: 30
Total transferred: 13928764 bytes
Total POSTed: 4610461
HTML transferred: 9728194 bytes
Requests per second: 13.52 [#/sec] (mean)
Time per request: 147.878 [ms] (mean)
Time per request: 73.939 [ms] (mean, across all concurrent requests)
the log in kylin server side is:
2018-10-15 09:07:40,448 DEBUG [Query 2f60cebe-3b9f-432a-b971-7d2285e2e25c-615]
service.QueryService:664 :
BorrowedCount:20737,DestroyedCount:62,CreatedCount:70,ReturnedCount:20737,IdleCount:8,ActiveCount:
0
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649787#comment-16649787
]
huaicui edited comment on KYLIN-3601 at 10/15/18 8:20 AM:
--
We don't use 1 concurrent client for performance test. The number of concurrent
is more 50; For example 70,100,150,200;
was (Author: kofiori):
We don't use 1 concurrent client to performance test. The number of concurrent
is more 50; For example 70,100,150,200;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649787#comment-16649787
]
huaicui commented on KYLIN-3601:
We don't use 1 concurrent client to performance test. The number of concurrent
is more 50; For example 70,100,150,200;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[
https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649768#comment-16649768
]
Yichen Zhou commented on KYLIN-3297:
Hi [~xer001], I didn't meet this issue when executing the following query for
sample cube.
{quote}select part_dt, trans_id from KYLIN_SALES where part_dt>='2012-09-09'
and part_dt<='2012-01-08';
{quote}
It returns 'No Result.', which is the correct answer.
Could you provide the kylin log or any other information so that I can
reproduce this issue?
> sql在解析>大值 and <小值的时候，kylin出现内存溢出。
> -
>
> Key: KYLIN-3297
> URL: https://issues.apache.org/jira/browse/KYLIN-3297
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.2.0
> Environment: oracle linux 6.8
> cup 8C
> memory 48G
> apache kylin v2.2.0
>Reporter: 陈家宇
>Priority: Major
> Fix For: v2.6.0
>
>
> select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a,
> case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from (
> select h.business_area_desc,h.plant_code,
> case when p.report_business_area like '%切大片%' then '切大片'
> else p.report_business_area end report_business_area,
> case when p.report_business_area like '%切大片%' then '00'
> else p.orderby end orderby,
> sum(p.quantity) qty,sum(a)a from pkn_kpi p
> inner join hcm_area h on p.plant_id=h.plant_id
> where 1=1
> and h.business_area_desc='上海'
> and h.plant_code='S1'
> and p.calendar_day>='2018-03-19'
> and p.calendar_day<='2018-03-18'
> and trim(p.report_business_area) is not null
> group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby
> ) f
> group by f.plant_code,f.report_business_area,f.orderby
> order by orderby
> 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and
> calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649767#comment-16649767
]
Ma Gang commented on KYLIN-3601:
So you mean that only 1 concurrent client can also reproduce it? I will try to
reproduce it in my side.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
[ https://issues.apache.org/jira/browse/KYLIN-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649760#comment-16649760 ] ASF GitHub Bot commented on KYLIN-3632: --- coveralls commented on issue #290: KYLIN-3632 Add configuration that can switch on/off preparedStatement cache URL: https://github.com/apache/kylin/pull/290#issuecomment-429717932 ## Pull Request Test Coverage Report for [Build 3769](https://coveralls.io/builds/19518586) * **0** of **2** **(0.0%)** changed or added relevant lines in **2** files are covered. * **1** unchanged line in **1** file lost coverage. * Overall coverage decreased (**-0.0003%**) to **23.304%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/19518586/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L1461) | 0 | 1 | 0.0% | [server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java](https://coveralls.io/builds/19518586/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FQueryService.java#L590) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/19518586/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | Totals | [](https://coveralls.io/builds/19518586) | | :-- | --: | | Change from base [Build 3768](https://coveralls.io/builds/19516792): | -0.0003% | | Covered Lines: | 16309 | | Relevant Lines: | 69984 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add configuration that can switch on/off preparedStatement cache in Kylin > server > > > Key: KYLIN-3632 > URL: https://issues.apache.org/jira/browse/KYLIN-3632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.1 > > > From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn > on/off for each request by adding a new field "enableStatementCache" in the > query request, by default it is on. We need to add a switch in server level, > the preparedStatement cache can take effective only when the switch is on, by > default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
