[
https://issues.apache.org/jira/browse/KYLIN-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607490#comment-17607490
]
Liu Zhao commented on KYLIN-5268:
---------------------------------
hi，我刚刚看了下，如果tableau发出的奇怪sql不多固定，似乎修改源码的
org.apache.kylin.rest.util.TableauInterceptor 类更合理，相较于修改 QueryUtil.java
强制为没有limit的sql加limit，前者可以直接拦截不提交spark查询作业，后者还是会发起spark查询作业有资源消耗。
{code:java}
// QueryService.java
conn = QueryConnection.getConnection(sqlRequest.getProject());
String userInfo =
SecurityContextHolder.getContext().getAuthentication().getName();
QueryContext context = QueryContextFacade.current();
context.setUsername(userInfo);
context.setGroups(AclPermissionUtil.getCurrentUserGroups());
context.setProject(sqlRequest.getProject());
final Collection<? extends GrantedAuthority> grantedAuthorities =
SecurityContextHolder.getContext()
.getAuthentication().getAuthorities();
for (GrantedAuthority grantedAuthority : grantedAuthorities) {
userInfo += ",";
userInfo += grantedAuthority.getAuthority();
}
SQLResponse fakeResponse =
TableauInterceptor.tableauIntercept(sqlRequest.getSql());
if (null != fakeResponse) {
logger.debug("Return fake response, is exception? " +
fakeResponse.getIsException());
return fakeResponse;
}
String correctedSql = QueryUtil.massageSql(sqlRequest.getSql(),
sqlRequest.getProject(),
sqlRequest.getLimit(), sqlRequest.getOffset(),
conn.getSchema(), Constant.FakeCatalogName);
if (!correctedSql.equals(sqlRequest.getSql())) {
logger.info("The corrected query: " + correctedSql);
//CAUTION: should not change sqlRequest content!
//sqlRequest.setSql(correctedSql);
}
{code}
> kylin4.0.1服务每天都会内存溢出java.lang.OutOfMemoryError: Java heap space
> ---------------------------------------------------------------
>
> Key: KYLIN-5268
> URL: https://issues.apache.org/jira/browse/KYLIN-5268
> Project: Kylin
> Issue Type: Bug
> Reporter: zhenkuan_zhang
> Priority: Major
> Attachments: kylin_query.log
>
>
> Hi，最近发现我们的kylin服务非常不稳定，三个节点，每天都会有节点出现内存溢出的状况，jvm从16G调到48G也同样会内存溢出。通过分析Dump 出
> java_pid34184.hprof文件，发现内存会被两个查询完全占满导致jvm内存溢出，sql是通过tableau发送来的。sql如下（该事实表行数上亿）：
> 1.
> SELECT "自訂 SQL 查詢"."FAIL" AS "FAIL" FROM (
> SELECT DT TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6)
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
> SUBSTRING(ATL FROM 1 for POSITION('*-' IN ATL from 1)-1 )
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3)
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4)
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) EEEEATL ,
> INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
> ) "自訂 SQL 查詢"
> 2.
> SELECT "自訂 SQL 查詢"."RETEST" AS "RETEST" FROM (
> SELECT DT TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6)
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
> SUBSTRING(ATL FROM 1 for POSITION('*-' IN ATL from 1)-1 )
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3)
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4)
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) EEEEATL ,
> INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
> ) "自訂 SQL 查詢"
> 一方面我们在与tableau方联系，沟通为什么会发出这么奇怪的sql。
> 另一方面希望得到kylin的支持，看看有没有什么解决方法，
> 比如kylin.query.force-limit=1000 可以为 select * from table 加上 limit 1000，有没有办法限制
> select a from table。
> 或者有没有其他的解决方法。
> 期待回复，十分感谢！！！
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
