[jira] [Created] (KYLIN-2250) Make HiveMRInput support Hive View
Wang Cheng created KYLIN-2250: - Summary: Make HiveMRInput support Hive View Key: KYLIN-2250 URL: https://issues.apache.org/jira/browse/KYLIN-2250 Project: Kylin Issue Type: Bug Reporter: Wang Cheng Priority: Minor When hive table is a view, it can not be accessed by HCatalog. A compromised way is to materialize the view firstly. In this purpose, make a little bit changes in HiveMRInput. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-2218) Kylin Hive Column Cardinality Job table using hdfs /tmp/kylin/cardinality/ as output folder
[
https://issues.apache.org/jira/browse/KYLIN-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wang Cheng closed KYLIN-2218.
-
Resolution: Fixed
> Kylin Hive Column Cardinality Job table using hdfs /tmp/kylin/cardinality/ as
> output folder
> ---
>
> Key: KYLIN-2218
> URL: https://issues.apache.org/jira/browse/KYLIN-2218
> Project: Kylin
> Issue Type: Improvement
>Reporter: Wang Cheng
>Priority: Trivial
>
> It is not safe to save kylin cardinality intermediate data into hdfs
> directionary: /tmp, because it might be modified by other hadoop processes.
> Thus move the dir from /tmp/kylin/cardinality/ to
> /kylin/kylin_metadata/{jobid}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (KYLIN-2219) Tables can not show "extend information" correctly
[ https://issues.apache.org/jira/browse/KYLIN-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng closed KYLIN-2219. - Resolution: Fixed > Tables can not show "extend information" correctly > -- > > Key: KYLIN-2219 > URL: https://issues.apache.org/jira/browse/KYLIN-2219 > Project: Kylin > Issue Type: Improvement >Reporter: Wang Cheng >Assignee: Wang Cheng >Priority: Minor > > When clicking DataSource->Database->Table->Extend Information, it can not > show full information. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-2245) Refine CubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng closed KYLIN-2245. - Resolution: Fixed > Refine CubeSegment > -- > > Key: KYLIN-2245 > URL: https://issues.apache.org/jira/browse/KYLIN-2245 > Project: Kylin > Issue Type: Bug >Reporter: Wang Cheng >Priority: Minor > Labels: patch > > List can not present the relations among CubeSegments, for > example, there are a lot of operations among Segments in CubeInstance. > Will refine List with new class Segments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2249) Build cube error when use "inmem" but ok with "layer"
[ https://issues.apache.org/jira/browse/KYLIN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hoangle updated KYLIN-2249: --- Environment: CentOS > Build cube error when use "inmem" but ok with "layer" > - > > Key: KYLIN-2249 > URL: https://issues.apache.org/jira/browse/KYLIN-2249 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.6.0 > Environment: CentOS >Reporter: hoangle > Original Estimate: 410h > Remaining Estimate: 410h > > My cube building is ok if I set configure: > kylin.cube.algorithm=layer > But It always error (at Step Name: Build Cube) when I change "layer" to > "inmem" > > 2016-12-05 17:17:37,451 ERROR [Thread-13] > org.apache.kylin.dict.TrieDictionary: Not a valid value: 122594010041 > 2016-12-05 17:17:38,452 ERROR [pool-8-thread-1] > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error > java.io.IOException: java.lang.IllegalArgumentException: Value not exists! > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) > at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: Value not exists! > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) > at > org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167) > at > org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98) > at > org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121) > at > org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121) > at > org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110) > at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93) > at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:544) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:525) > at > org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:341) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:168) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-2249) Build cube error when use "inmem" but ok with "layer"
[ https://issues.apache.org/jira/browse/KYLIN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hoangle updated KYLIN-2249: --- Remaining Estimate: 410h Original Estimate: 410h > Build cube error when use "inmem" but ok with "layer" > - > > Key: KYLIN-2249 > URL: https://issues.apache.org/jira/browse/KYLIN-2249 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.6.0 >Reporter: hoangle > Original Estimate: 410h > Remaining Estimate: 410h > > My cube building is ok if I set configure: > kylin.cube.algorithm=layer > But It always error (at Step Name: Build Cube) when I change "layer" to > "inmem" > > 2016-12-05 17:17:37,451 ERROR [Thread-13] > org.apache.kylin.dict.TrieDictionary: Not a valid value: 122594010041 > 2016-12-05 17:17:38,452 ERROR [pool-8-thread-1] > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error > java.io.IOException: java.lang.IllegalArgumentException: Value not exists! > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) > at > org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: Value not exists! > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) > at > org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167) > at > org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98) > at > org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121) > at > org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121) > at > org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110) > at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93) > at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:544) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:525) > at > org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:341) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:168) > at > org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) > at > org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2249) Build cube error when use "inmem" but ok with "layer"
[ https://issues.apache.org/jira/browse/KYLIN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724265#comment-15724265 ] hoangle commented on KYLIN-2249: add more log 2016-12-05 17:18:01,859 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Failed to build cube in mapper 0 at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143) ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) ... 5 more Caused by: java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) ... 7 more Caused by: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167) at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98) at org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121) at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121) at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110) at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93) at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:544) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:525) at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:341) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:168) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) > Build cube error when use "inmem" but ok with "layer" > - > > Key: KYLIN-2249 > URL: https://issues.apache.org/jira/browse/KYLIN-2249 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.6.0 >
[jira] [Updated] (KYLIN-2249) Build cube error when use "inmem" but ok with "layer"
[ https://issues.apache.org/jira/browse/KYLIN-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hoangle updated KYLIN-2249: --- Description: My cube building is ok if I set configure: kylin.cube.algorithm=layer But It always error (at Step Name: Build Cube) when I change "layer" to "inmem" 2016-12-05 17:17:37,451 ERROR [Thread-13] org.apache.kylin.dict.TrieDictionary: Not a valid value: 122594010041 2016-12-05 17:17:38,452 ERROR [pool-8-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167) at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98) at org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121) at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:121) at org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:110) at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:93) at org.apache.kylin.gridtable.GTRecord.setValues(GTRecord.java:81) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilderInputConverter.convert(InMemCubeBuilderInputConverter.java:74) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:544) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$InputConverter$1.next(InMemCubeBuilder.java:525) at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:139) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.createBaseCuboid(InMemCubeBuilder.java:341) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:168) at org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:137) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:284) was: 2016-12-05 17:17:37,451 ERROR [Thread-13] org.apache.kylin.dict.TrieDictionary: Not a valid value: 122594010041 2016-12-05 17:17:38,452 ERROR [pool-8-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error java.io.IOException: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:196) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:169) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116) at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75) at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167) at org.apache.kylin.common.util.Dictionary.getIdFromValue(Dictionary.java:98) at org.apache.kylin.dimension.DictionaryDimEnc$DictionarySerializer.serialize(DictionaryDimEnc.java:121) at
[jira] [Closed] (KYLIN-2228) Creating a Custom Interceptor in Apache DS 2.0.0-M23
[ https://issues.apache.org/jira/browse/KYLIN-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu closed KYLIN-2228. Resolution: Invalid > Creating a Custom Interceptor in Apache DS 2.0.0-M23 > > > Key: KYLIN-2228 > URL: https://issues.apache.org/jira/browse/KYLIN-2228 > Project: Kylin > Issue Type: Improvement >Reporter: Gautham Gurudatta Shet >Priority: Blocker > > The link > "http://directory.apache.org/apacheds/advanced-ug/6-implementing-interceptor.html; > does not have info on how to create a custom interceptor for Apache DS > 2.0.0-M23. The info is present for Apache DS 1.5.5. Can you please provide > this info? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2221) rethink on KYLIN-1684
[ https://issues.apache.org/jira/browse/KYLIN-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721942#comment-15721942 ] qianqiaoneng commented on KYLIN-2221: - +1. Better to resolve the issue from the view that how to avoid these "fake" empty segment instead of change the query behavior for empty segments, as it will impact "fake" empty segment as well as real empty segments. For real empty segment, there is no need to do hbase scan. > rethink on KYLIN-1684 > - > > Key: KYLIN-2221 > URL: https://issues.apache.org/jira/browse/KYLIN-2221 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > Fix For: v1.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2228) Creating a Custom Interceptor in Apache DS 2.0.0-M23
[ https://issues.apache.org/jira/browse/KYLIN-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721873#comment-15721873 ] qianqiaoneng commented on KYLIN-2228: - Is this JIRA is supposed to issued against Apache DS instead of Kylin? General (documentation, site) DIR http://issues.apache.org/jira/browse/DIR > Creating a Custom Interceptor in Apache DS 2.0.0-M23 > > > Key: KYLIN-2228 > URL: https://issues.apache.org/jira/browse/KYLIN-2228 > Project: Kylin > Issue Type: Improvement >Reporter: Gautham Gurudatta Shet >Priority: Blocker > > The link > "http://directory.apache.org/apacheds/advanced-ug/6-implementing-interceptor.html; > does not have info on how to create a custom interceptor for Apache DS > 2.0.0-M23. The info is present for Apache DS 1.5.5. Can you please provide > this info? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2248) TopN merge further optimization after KYLIN-1917
[
https://issues.apache.org/jira/browse/KYLIN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721764#comment-15721764
]
Shaofeng SHI commented on KYLIN-2248:
-
Change made in
https://github.com/apache/kylin/commit/59a30f66d47cc1838e6852405699fd7957bfac29
> TopN merge further optimization after KYLIN-1917
>
>
> Key: KYLIN-2248
> URL: https://issues.apache.org/jira/browse/KYLIN-2248
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
> Fix For: v1.6.1
>
>
> After KYLIN-1917, there still be room for performance optimization when
> building a cube which has very large amount rows but the cardinality of all
> dimension are quite small.
> Then there will be much aggregation happens in building base cuboid. The
> reducer has a big pressure on CPU. With JStack we observed the CPU was spent
> on the TopNCounter.merge(), in the HashMap.get() method.
> {code}
> Thread 28679: (state = IN_JAVA)
> - java.util.HashMap.getEntry(java.lang.Object) @bci=81, line=465 (Compiled
> frame; information may be imprecise)
> - java.util.HashMap.get(java.lang.Object) @bci=11, line=417 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNCounter.merge(org.apache.kylin.measure.topn.TopNCounter)
> @bci=117, line=174 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNAggregator.aggregate(org.apache.kylin.measure.topn.TopNCounter)
> @bci=38, line=44 (Compiled frame)
> - org.apache.kylin.measure.topn.TopNAggregator.aggregate(java.lang.Object)
> @bci=5, line=27 (Compiled frame)
> - org.apache.kylin.measure.MeasureAggregators.aggregate(java.lang.Object[])
> @bci=42, line=76 (Compiled frame)
> -
> org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(org.apache.hadoop.io.Text,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=95,
> line=97 (Compiled frame)
> - org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=7,
> line=42 (Interpreted frame)
> - org.apache.kylin.engine.mr.KylinReducer.reduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=4,
> line=40 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.Reducer.run(org.apache.hadoop.mapreduce.Reducer$Context)
> @bci=22, line=171 (Interpreted frame)
> -
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol,
> org.apache.hadoop.mapred.Task$TaskReporter,
> org.apache.hadoop.mapred.RawKeyValueIterator,
> org.apache.hadoop.io.RawComparator, java.lang.Class, java.lang.Class)
> @bci=119, line=627 (Interpreted frame)
> - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=384, line=389
> (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild$2.run() @bci=36, line=164 (Interpreted
> frame)
> -
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
> java.security.AccessControlContext) @bci=0 (Interpreted frame)
> - javax.security.auth.Subject.doAs(javax.security.auth.Subject,
> java.security.PrivilegedExceptionAction) @bci=42, line=415 (Interpreted frame)
> -
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
> @bci=14, line=1709 (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild.main(java.lang.String[]) @bci=514,
> line=158 (Interpreted frame)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (KYLIN-2248) TopN merge further optimization after KYLIN-1917
[
https://issues.apache.org/jira/browse/KYLIN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2248:
Comment: was deleted
(was: Change made in
https://github.com/apache/kylin/commit/59a30f66d47cc1838e6852405699fd7957bfac29)
> TopN merge further optimization after KYLIN-1917
>
>
> Key: KYLIN-2248
> URL: https://issues.apache.org/jira/browse/KYLIN-2248
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
> Fix For: v1.6.1
>
>
> After KYLIN-1917, there still be room for performance optimization when
> building a cube which has very large amount rows but the cardinality of all
> dimension are quite small.
> Then there will be much aggregation happens in building base cuboid. The
> reducer has a big pressure on CPU. With JStack we observed the CPU was spent
> on the TopNCounter.merge(), in the HashMap.get() method.
> {code}
> Thread 28679: (state = IN_JAVA)
> - java.util.HashMap.getEntry(java.lang.Object) @bci=81, line=465 (Compiled
> frame; information may be imprecise)
> - java.util.HashMap.get(java.lang.Object) @bci=11, line=417 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNCounter.merge(org.apache.kylin.measure.topn.TopNCounter)
> @bci=117, line=174 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNAggregator.aggregate(org.apache.kylin.measure.topn.TopNCounter)
> @bci=38, line=44 (Compiled frame)
> - org.apache.kylin.measure.topn.TopNAggregator.aggregate(java.lang.Object)
> @bci=5, line=27 (Compiled frame)
> - org.apache.kylin.measure.MeasureAggregators.aggregate(java.lang.Object[])
> @bci=42, line=76 (Compiled frame)
> -
> org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(org.apache.hadoop.io.Text,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=95,
> line=97 (Compiled frame)
> - org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=7,
> line=42 (Interpreted frame)
> - org.apache.kylin.engine.mr.KylinReducer.reduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=4,
> line=40 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.Reducer.run(org.apache.hadoop.mapreduce.Reducer$Context)
> @bci=22, line=171 (Interpreted frame)
> -
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol,
> org.apache.hadoop.mapred.Task$TaskReporter,
> org.apache.hadoop.mapred.RawKeyValueIterator,
> org.apache.hadoop.io.RawComparator, java.lang.Class, java.lang.Class)
> @bci=119, line=627 (Interpreted frame)
> - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=384, line=389
> (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild$2.run() @bci=36, line=164 (Interpreted
> frame)
> -
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
> java.security.AccessControlContext) @bci=0 (Interpreted frame)
> - javax.security.auth.Subject.doAs(javax.security.auth.Subject,
> java.security.PrivilegedExceptionAction) @bci=42, line=415 (Interpreted frame)
> -
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
> @bci=14, line=1709 (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild.main(java.lang.String[]) @bci=514,
> line=158 (Interpreted frame)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-2248) TopN merge further optimization after KYLIN-1917
[
https://issues.apache.org/jira/browse/KYLIN-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721765#comment-15721765
]
Shaofeng SHI commented on KYLIN-2248:
-
Change made in
https://github.com/apache/kylin/commit/59a30f66d47cc1838e6852405699fd7957bfac29
> TopN merge further optimization after KYLIN-1917
>
>
> Key: KYLIN-2248
> URL: https://issues.apache.org/jira/browse/KYLIN-2248
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
> Fix For: v1.6.1
>
>
> After KYLIN-1917, there still be room for performance optimization when
> building a cube which has very large amount rows but the cardinality of all
> dimension are quite small.
> Then there will be much aggregation happens in building base cuboid. The
> reducer has a big pressure on CPU. With JStack we observed the CPU was spent
> on the TopNCounter.merge(), in the HashMap.get() method.
> {code}
> Thread 28679: (state = IN_JAVA)
> - java.util.HashMap.getEntry(java.lang.Object) @bci=81, line=465 (Compiled
> frame; information may be imprecise)
> - java.util.HashMap.get(java.lang.Object) @bci=11, line=417 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNCounter.merge(org.apache.kylin.measure.topn.TopNCounter)
> @bci=117, line=174 (Compiled frame)
> -
> org.apache.kylin.measure.topn.TopNAggregator.aggregate(org.apache.kylin.measure.topn.TopNCounter)
> @bci=38, line=44 (Compiled frame)
> - org.apache.kylin.measure.topn.TopNAggregator.aggregate(java.lang.Object)
> @bci=5, line=27 (Compiled frame)
> - org.apache.kylin.measure.MeasureAggregators.aggregate(java.lang.Object[])
> @bci=42, line=76 (Compiled frame)
> -
> org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(org.apache.hadoop.io.Text,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=95,
> line=97 (Compiled frame)
> - org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=7,
> line=42 (Interpreted frame)
> - org.apache.kylin.engine.mr.KylinReducer.reduce(java.lang.Object,
> java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer$Context) @bci=4,
> line=40 (Interpreted frame)
> -
> org.apache.hadoop.mapreduce.Reducer.run(org.apache.hadoop.mapreduce.Reducer$Context)
> @bci=22, line=171 (Interpreted frame)
> -
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol,
> org.apache.hadoop.mapred.Task$TaskReporter,
> org.apache.hadoop.mapred.RawKeyValueIterator,
> org.apache.hadoop.io.RawComparator, java.lang.Class, java.lang.Class)
> @bci=119, line=627 (Interpreted frame)
> - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf,
> org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=384, line=389
> (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild$2.run() @bci=36, line=164 (Interpreted
> frame)
> -
> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
> java.security.AccessControlContext) @bci=0 (Interpreted frame)
> - javax.security.auth.Subject.doAs(javax.security.auth.Subject,
> java.security.PrivilegedExceptionAction) @bci=42, line=415 (Interpreted frame)
> -
> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
> @bci=14, line=1709 (Interpreted frame)
> - org.apache.hadoop.mapred.YarnChild.main(java.lang.String[]) @bci=514,
> line=158 (Interpreted frame)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
