[jira] [Updated] (KYLIN-2268) Potential NPE in ModelDimensionDesc#init()
[
https://issues.apache.org/jira/browse/KYLIN-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-2268:
--
Description:
{code}
if (columns != null) {
StringUtil.toUpperCaseArray(columns, columns);
}
if (model != null) {
table = model.findTable(table).getAlias();
for (int i = 0; i < columns.length; i++) {
{code}
In the second if block, columns is accessed without null check.
was:
{code}
if (columns != null) {
StringUtil.toUpperCaseArray(columns, columns);
}
if (model != null) {
table = model.findTable(table).getAlias();
for (int i = 0; i < columns.length; i++) {
{code}
In the second if block, columns is accessed without null check.
> Potential NPE in ModelDimensionDesc#init()
> --
>
> Key: KYLIN-2268
> URL: https://issues.apache.org/jira/browse/KYLIN-2268
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> if (columns != null) {
> StringUtil.toUpperCaseArray(columns, columns);
> }
> if (model != null) {
> table = model.findTable(table).getAlias();
> for (int i = 0; i < columns.length; i++) {
> {code}
> In the second if block, columns is accessed without null check.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2424) Optimize the integration test's performance
[ https://issues.apache.org/jira/browse/KYLIN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852812#comment-15852812 ] Billy Liu commented on KYLIN-2424: -- Hi [~Shaofengshi], I found you added new optional property index with default value 'T', is 'T' a kind of index? > Optimize the integration test's performance > --- > > Key: KYLIN-2424 > URL: https://issues.apache.org/jira/browse/KYLIN-2424 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Kylin's integration test is slow, especially the ITCombinationTest. Most of > time are spent on H2 to execute the test queries. In a latest integration > test, this test case take 90 minutes to finish. > By checking H2's document, I think the main problem is the absence of index > on the tables, while index is very important for a relational database's > query performance. So when Kylin create the tables in H2, shoud create index > on the columns that will be used in the queries, like the pk/fk, the > filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2424) Optimize the integration test's performance
[ https://issues.apache.org/jira/browse/KYLIN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852787#comment-15852787 ] Shaofeng SHI commented on KYLIN-2424: - The change is pushed to master at: https://github.com/apache/kylin/commit/5d83c80fbc0d6c3db434368cc8c29d786eebcd94 Now the ITCombinationTest takes less than 3 minutes in my local. > Optimize the integration test's performance > --- > > Key: KYLIN-2424 > URL: https://issues.apache.org/jira/browse/KYLIN-2424 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Kylin's integration test is slow, especially the ITCombinationTest. Most of > time are spent on H2 to execute the test queries. In a latest integration > test, this test case take 90 minutes to finish. > By checking H2's document, I think the main problem is the absence of index > on the tables, while index is very important for a relational database's > query performance. So when Kylin create the tables in H2, shoud create index > on the columns that will be used in the queries, like the pk/fk, the > filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2424) Optimize the integration test's performance
Shaofeng SHI created KYLIN-2424: --- Summary: Optimize the integration test's performance Key: KYLIN-2424 URL: https://issues.apache.org/jira/browse/KYLIN-2424 Project: Kylin Issue Type: Improvement Components: Tools, Build and Test Reporter: Shaofeng SHI Assignee: Shaofeng SHI Fix For: v2.0.0 Kylin's integration test is slow, especially the ITCombinationTest. Most of time are spent on H2 to execute the test queries. In a latest integration test, this test case take 90 minutes to finish. By checking H2's document, I think the main problem is the absence of index on the tables, while index is very important for a relational database's query performance. So when Kylin create the tables in H2, shoud create index on the columns that will be used in the queries, like the pk/fk, the filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2391) Unclosed FileInputStream in KylinConfig#getConfigAsString()
[
https://issues.apache.org/jira/browse/KYLIN-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852754#comment-15852754
]
Shaofeng SHI commented on KYLIN-2391:
-
fix in:
https://github.com/apache/kylin/commit/6f0bc1c3edd75f0b180afeeb68e09cf01298d4d8
> Unclosed FileInputStream in KylinConfig#getConfigAsString()
> ---
>
> Key: KYLIN-2391
> URL: https://issues.apache.org/jira/browse/KYLIN-2391
> Project: Kylin
> Issue Type: Bug
> Components: General
>Reporter: Ted Yu
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.0.0
>
>
> {code}
> OrderedProperties orderedProperties = new OrderedProperties();
> orderedProperties.load(new FileInputStream(propertiesFile));
> {code}
> The stream created by FileInputStream should be closed upon return.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KYLIN-2284) intersect_count function error
[ https://issues.apache.org/jira/browse/KYLIN-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2284. --- Resolution: Not A Bug Fix Version/s: v2.0.0 > intersect_count function error > -- > > Key: KYLIN-2284 > URL: https://issues.apache.org/jira/browse/KYLIN-2284 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.6.0 > Environment: windows >Reporter: Luyuan Zhai >Assignee: liyang >Priority: Minor > Fix For: v2.0.0 > > Attachments: intersect_count.correct result.png, > intersect_count.error result.png > > > I explored on the Kylin Web GUI, using datasource learn_kylin, when I inputed > sql query"select intersect_count(seller_id, LSTG_FORMAT_NAME, > array['Others']) from KYLIN_SALES", it led to an error"Error while executing > SQL "select intersect_count(seller_id, LSTG_FORMAT_NAME, array['Others']) > from KYLIN_SALES LIMIT 5": Index: 1, Size: 1". > Yet when I inputed similar sql query including intersect_count function as > well, then it worked well and the result returned correctly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2284) intersect_count function error
[ https://issues.apache.org/jira/browse/KYLIN-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852708#comment-15852708 ] liyang commented on KYLIN-2284: --- Reproduced the error on master branch, the error message now reads good enough: "Error while executing SQL "select week_beg_dt as week, intersect_count( SELLER_ID, lstg_format_name, array['FP-GTC']) as a, count(*) as cnt from test_kylin_fact left join edw.test_cal_dt on test_kylin_fact.cal_dt = edw.test_cal_dt.CAL_DT where week_beg_dt in (DATE '2013-12-22', DATE '2012-06-23') group by week_beg_dt LIMIT 5": org.apache.kylin.measure.hllc.HLLCounter cannot be cast to org.apache.kylin.measure.bitmap.BitmapCounter" > intersect_count function error > -- > > Key: KYLIN-2284 > URL: https://issues.apache.org/jira/browse/KYLIN-2284 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.6.0 > Environment: windows >Reporter: Luyuan Zhai >Assignee: liyang >Priority: Minor > Fix For: v2.0.0 > > Attachments: intersect_count.correct result.png, > intersect_count.error result.png > > > I explored on the Kylin Web GUI, using datasource learn_kylin, when I inputed > sql query"select intersect_count(seller_id, LSTG_FORMAT_NAME, > array['Others']) from KYLIN_SALES", it led to an error"Error while executing > SQL "select intersect_count(seller_id, LSTG_FORMAT_NAME, array['Others']) > from KYLIN_SALES LIMIT 5": Index: 1, Size: 1". > Yet when I inputed similar sql query including intersect_count function as > well, then it worked well and the result returned correctly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2284) intersect_count function error
[ https://issues.apache.org/jira/browse/KYLIN-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852703#comment-15852703 ] liyang commented on KYLIN-2284: --- This is expected, intersect_count() only works with bitmap count distinct, but in learn_kylin, the seller_id is of type HLL count distinct. > intersect_count function error > -- > > Key: KYLIN-2284 > URL: https://issues.apache.org/jira/browse/KYLIN-2284 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.6.0 > Environment: windows >Reporter: Luyuan Zhai >Assignee: liyang >Priority: Minor > Attachments: intersect_count.correct result.png, > intersect_count.error result.png > > > I explored on the Kylin Web GUI, using datasource learn_kylin, when I inputed > sql query"select intersect_count(seller_id, LSTG_FORMAT_NAME, > array['Others']) from KYLIN_SALES", it led to an error"Error while executing > SQL "select intersect_count(seller_id, LSTG_FORMAT_NAME, array['Others']) > from KYLIN_SALES LIMIT 5": Index: 1, Size: 1". > Yet when I inputed similar sql query including intersect_count function as > well, then it worked well and the result returned correctly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2344) Package spark into Kylin binary package
[ https://issues.apache.org/jira/browse/KYLIN-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2344. - Resolution: Fixed > Package spark into Kylin binary package > --- > > Key: KYLIN-2344 > URL: https://issues.apache.org/jira/browse/KYLIN-2344 > Project: Kylin > Issue Type: Task >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > To support KYLIN-2331, Kylin need a Spark; need build Spark into Kylin's > binary package, just like we did for Tomcat -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2344) Package spark into Kylin binary package
[ https://issues.apache.org/jira/browse/KYLIN-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2344: Request participants: (was: ) Fix Version/s: v2.0.0 > Package spark into Kylin binary package > --- > > Key: KYLIN-2344 > URL: https://issues.apache.org/jira/browse/KYLIN-2344 > Project: Kylin > Issue Type: Task >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > To support KYLIN-2331, Kylin need a Spark; need build Spark into Kylin's > binary package, just like we did for Tomcat -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2404) Add "hive.merge.mapfiles" and "hive.merge.mapredfiles" to kylin_hive_conf.xml
[ https://issues.apache.org/jira/browse/KYLIN-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2404. - Resolution: Fixed > Add "hive.merge.mapfiles" and "hive.merge.mapredfiles" to kylin_hive_conf.xml > - > > Key: KYLIN-2404 > URL: https://issues.apache.org/jira/browse/KYLIN-2404 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Minor > Fix For: v2.0.0 > > > Since 1.5.3, Kylin uses a "redistribute" step to merge the small files to > proper size after creating the intermediate hive table. While in some users' > environment, hive merge small files is enabled by default, that will cause > additional CPU and will impact on the cube building performance (in extreme > case the files will be merged to 256MB, then only very small number of > mappers be started in building). > So Kylin should explicitly tell Hive to disable the merge small files feature > when creating and redistributing the intermediate flat table. Adding > "hive.merge.mapfiles" and "hive.merge.mapredfiles" to > conf/kylin_hive_conf.xml with value "false" will solve this. The meaning of > these two parameters can be found in > https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2411) Kill MR job on pause
[ https://issues.apache.org/jira/browse/KYLIN-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2411. - Resolution: Fixed > Kill MR job on pause > > > Key: KYLIN-2411 > URL: https://issues.apache.org/jira/browse/KYLIN-2411 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Two minor improvements here: > 1) when user pause a job, if the running step is a MR job, kill it > immediately to release cluster resource; > 2) clear job output and information (like job id) when admin rollback a step. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2415) Change back default metadata name to "kylin_metadata"
[ https://issues.apache.org/jira/browse/KYLIN-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2415. - Resolution: Fixed > Change back default metadata name to "kylin_metadata" > - > > Key: KYLIN-2415 > URL: https://issues.apache.org/jira/browse/KYLIN-2415 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > In the dev of v2.0, the default metadata name was changed as > "kylin.metadata.url=kylin_default_instance@hbase", the name is better, but > considering that many user are using the old name, so when they upgrade to > v2.0, if not change this back to the old name, the cubes will be invisible; > To avoid the confusing user, we should keep using the legacy name; -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2361) Upgrade to Tomcat 8.X
[ https://issues.apache.org/jira/browse/KYLIN-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15852697#comment-15852697 ] Billy Liu commented on KYLIN-2361: -- [INFO] Apache Kylin ... SUCCESS [ 3.211 s] [INFO] Apache Kylin - Calcite Overrides ... SUCCESS [ 2.743 s] [INFO] Apache Kylin - Core Common . SUCCESS [ 5.418 s] [INFO] Apache Kylin - Core Metadata ... SUCCESS [ 27.684 s] [INFO] Apache Kylin - Core Dictionary . SUCCESS [01:22 min] [INFO] Apache Kylin - Core Cube ... SUCCESS [01:54 min] [INFO] Apache Kylin - Core Job SUCCESS [01:58 min] [INFO] Apache Kylin - Core Storage SUCCESS [ 39.640 s] [INFO] Apache Kylin - MapReduce Engine SUCCESS [ 22.566 s] [INFO] Apache Kylin - HBase Storage ... SUCCESS [ 13.671 s] [INFO] Apache Kylin - Spark Engine SUCCESS [ 15.224 s] [INFO] Apache Kylin - Hive Source . SUCCESS [ 4.526 s] [INFO] Apache Kylin - Kafka Source SUCCESS [ 3.537 s] [INFO] Apache Kylin - Query ... SUCCESS [ 2.860 s] [INFO] Apache Kylin - Tool SUCCESS [ 4.780 s] [INFO] Apache Kylin - REST Server Base SUCCESS [ 8.026 s] [INFO] Apache Kylin - REST Server . SUCCESS [ 40.812 s] [INFO] Apache Kylin - JDBC Driver . SUCCESS [ 8.145 s] [INFO] Apache Kylin - Assembly SUCCESS [ 6.611 s] [INFO] Apache Kylin - Integration Test SUCCESS [ 03:02 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 1.025 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:10 h [INFO] Finished at: 2017-02-04T16:15:54+00:00 [INFO] Final Memory: 76M/926M [INFO] > Upgrade to Tomcat 8.X > - > > Key: KYLIN-2361 > URL: https://issues.apache.org/jira/browse/KYLIN-2361 > Project: Kylin > Issue Type: Task > Components: Web >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu >Priority: Minor > Fix For: v2.0.0 > > > Apache Tomcat 8.5.x supports the same Servlet, JSP, EL, and WebSocket > Specification versions as Apache Tomcat 8.0.x. In addition to that, it also > implements the JASPIC 1.1 specification. There are significant changes in > many areas under the hood, resulting in improved performance, stability, and > total cost of ownership. Please refer to the Apache Tomcat 8.5 Changelog for > details. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2321) Message: Error while executing SQL "select * from root.id_card LIMIT 50000": AppendTrieDictionary can't retrive value from id
[ https://issues.apache.org/jira/browse/KYLIN-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2321: Fix Version/s: (was: v1.6.0) v2.0.0 > Message: Error while executing SQL "select * from root.id_card LIMIT 5": > AppendTrieDictionary can't retrive value from id > - > > Key: KYLIN-2321 > URL: https://issues.apache.org/jira/browse/KYLIN-2321 > Project: Kylin > Issue Type: Bug > Components: 3rd Party, Job Engine >Affects Versions: v1.6.0 > Environment: Message: Error while executing SQL "select * from > root.id_card LIMIT 5": AppendTrieDictionary can't retrive value from id >Reporter: konglei >Assignee: Dong Li > Fix For: v2.0.0 > > > Message: Error while executing SQL "select * from root.id_card LIMIT 5": > AppendTrieDictionary can't retrive value from id > ==[QUERY]=== > 2016-12-26 16:59:16,553 ERROR [http-bio-7070-exec-10] > controller.BasicController:44 : > org.apache.kylin.rest.exception.InternalErrorException: Error while executing > SQL "select * from root.id_card LIMIT 5": AppendTrieDictionary can't > retrive value from id > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:389) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:69) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualF
[jira] [Updated] (KYLIN-2311) Build Cube
[ https://issues.apache.org/jira/browse/KYLIN-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2311: Fix Version/s: (was: v1.6.0) v2.0.0 > Build Cube > -- > > Key: KYLIN-2311 > URL: https://issues.apache.org/jira/browse/KYLIN-2311 > Project: Kylin > Issue Type: Test > Components: Job Engine >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh5.7-bin.tar.gz > system:centos 6.8 >Reporter: konglei >Assignee: Dong Li >Priority: Critical > Fix For: v2.0.0 > > > Error: java.io.IOException: Failed to build cube in mapper 1 at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148) at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:415) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.io.IOException: java.io.IOException: java.lang.IllegalStateException at > java.util.concurrent.FutureTask.report(FutureTask.java:122) at > java.util.concurrent.FutureTask.get(FutureTask.java:188) at > org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) > ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: > java.io.IOException: java.lang.IllegalStateException at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:84) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) Caused by: java.io.IOException: > java.io.IOException: java.lang.IllegalStateException at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) > at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) > ... 5 more Caused by: java.io.IOException: java.lang.IllegalStateException > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:101) > ... 7 more Caused by: java.lang.IllegalStateException at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) at > org.apache.kylin.common.util.MemoryBudgetController.(MemoryBudgetController.java:73) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.makeMemoryBudget(InMemCubeBuilder.java:324) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:174) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2316: Fix Version/s: (was: v1.6.0) v2.0.0 > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: Web >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: Zhong,Jason > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2325) kylin sql optimization is very slow about 100ms or so, and consumes a lot of cpu
[ https://issues.apache.org/jira/browse/KYLIN-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2325: Fix Version/s: (was: v1.6.0) v2.0.0 > kylin sql optimization is very slow about 100ms or so, and consumes a lot of > cpu > > > Key: KYLIN-2325 > URL: https://issues.apache.org/jira/browse/KYLIN-2325 > Project: Kylin > Issue Type: Improvement >Reporter: WangTaiZe > Labels: performance > Fix For: v2.0.0 > > Original Estimate: 12h > Remaining Estimate: 12h > > kylin sql optimization is too slow, and consumes too much cpu. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2423) Model should always include PK/FK as dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2423. --- Resolution: Fixed Fix Version/s: v2.0.0 > Model should always include PK/FK as dimensions > --- > > Key: KYLIN-2423 > URL: https://issues.apache.org/jira/browse/KYLIN-2423 > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: liyang > Fix For: v2.0.0 > > > Model should always include PK/FK as dimensions, as they participate in model > any way. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2391) Unclosed FileInputStream in KylinConfig#getConfigAsString()
[
https://issues.apache.org/jira/browse/KYLIN-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2391:
Request participants: (was: )
Fix Version/s: v2.0.0
Component/s: General
> Unclosed FileInputStream in KylinConfig#getConfigAsString()
> ---
>
> Key: KYLIN-2391
> URL: https://issues.apache.org/jira/browse/KYLIN-2391
> Project: Kylin
> Issue Type: Bug
> Components: General
>Reporter: Ted Yu
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.0.0
>
>
> {code}
> OrderedProperties orderedProperties = new OrderedProperties();
> orderedProperties.load(new FileInputStream(propertiesFile));
> {code}
> The stream created by FileInputStream should be closed upon return.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KYLIN-2421) Add spark engine to Integration Test
[ https://issues.apache.org/jira/browse/KYLIN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2421: Description: Let Jenkins to test Spark cubing daily. I have changed the engine type for "ci_inner_join_cube" from "2" (MRV2) to "4" (spark). After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test: 1. manually install spark-1.6.3-bin-hadoop2.6 in a local folder like /usr/local/spark cd /usr/local tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz ln -s spark-1.6.3-bin-hadoop2.6 spark 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS): hadoop fs -mkdir /kylin/spark/ hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/ 3. manually create local temp folder for hbase client (if it doesn't exist): mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars 4. Specify SPARK_HOME property when run mvn: mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox was: Let Jenkins to test Spark cubing daily. Will change the engine type for "ci_inner_join_cube" to "4" (spark). After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test: 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like /usr/local/spark cd /usr/local tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz ln -s spark-1.6.3-bin-hadoop2.6 spark 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS): hadoop fs -mkdir /kylin/spark/ hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/ 3. manually create local temp folder for hbase client (if it doesn't exist): mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars 4. Add SPARK_HOME property when run mvn: mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox > Add spark engine to Integration Test > > > Key: KYLIN-2421 > URL: https://issues.apache.org/jira/browse/KYLIN-2421 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Let Jenkins to test Spark cubing daily. I have changed the engine type for > "ci_inner_join_cube" from "2" (MRV2) to "4" (spark). > After this change, you need do a couple changes manually in your HDP 2.4 > sandbox to finish the Integration Test: > 1. manually install spark-1.6.3-bin-hadoop2.6 in a local folder like > /usr/local/spark > cd /usr/local > tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz > ln -s spark-1.6.3-bin-hadoop2.6 spark > 2. manually upload the spark-assembly jar to HDFS as > /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading > the jar to HDFS): > hadoop fs -mkdir /kylin/spark/ > hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar > /kylin/spark/ > 3. manually create local temp folder for hbase client (if it doesn't exist): > mkdir -p /hadoop/hbase/local/jars > chmod 777 /hadoop/hbase/local/jars > 4. Specify SPARK_HOME property when run mvn: > mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P > sandbox -- This message was sent by Atlassian JIRA (v6.3.15#6346)
