[jira] [Updated] (KYLIN-4964) Receiver consumer thread should be stoped while encounting unrecoverable error
[ https://issues.apache.org/jira/browse/KYLIN-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-4964: --- Description: org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data. Here is our solution which already tested: # Throw IllegalStorageException while we cannot do the persist # Then stop the consumer thread # Add `consumer_thread_alive` in ConsumerStats for monitoring # Also fix an another issue which will cause wrong result after restore from checkpoint was: org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data. Here is our sulotion which already tested: # Throw IllegalStorageException while we cannot do the persist # Then stop the consumer thread # Add `consumer_thread_alive` in ConsumerStats for monitoring # Also fix an another issue which will cause wrong result after restore from checkpoint > Receiver consumer thread should be stoped while encounting unrecoverable error > -- > > Key: KYLIN-4964 > URL: https://issues.apache.org/jira/browse/KYLIN-4964 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.1.1 >Reporter: Xingxing Di >Priority: Major > > org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist > will catch exception and just log it, this behavior will cause critical > problem, like our case: there is no space left on device, `persist` method > failed for many times, we lost several hours of data. > Here is our solution which already tested: > # Throw IllegalStorageException while we cannot do the persist > # Then stop the consumer thread > # Add `consumer_thread_alive` in ConsumerStats for monitoring > # Also fix an another issue which will cause wrong result after restore from > checkpoint > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4964) Receiver consumer thread should be stoped while encounting unrecoverable error
[ https://issues.apache.org/jira/browse/KYLIN-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-4964: --- Summary: Receiver consumer thread should be stoped while encounting unrecoverable error (was: Receiver consumer thread should be stoped while while encounting unrecoverable error) > Receiver consumer thread should be stoped while encounting unrecoverable error > -- > > Key: KYLIN-4964 > URL: https://issues.apache.org/jira/browse/KYLIN-4964 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.1.1 >Reporter: Xingxing Di >Priority: Major > > org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist > will catch exception and just log it, this behavior will cause critical > problem, like our case: there is no space left on device, `persist` method > failed for many times, we lost several hours of data. > Here is our sulotion which already tested: > # Throw IllegalStorageException while we cannot do the persist > # Then stop the consumer thread > # Add `consumer_thread_alive` in ConsumerStats for monitoring > # Also fix an another issue which will cause wrong result after restore from > checkpoint > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4964) Receiver consumer thread should be stoped while while encounting unrecoverable error
[ https://issues.apache.org/jira/browse/KYLIN-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-4964: --- Description: org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data. Here is our sulotion which already tested: # Throw IllegalStorageException while we cannot do the persist # Then stop the consumer thread # Add `consumer_thread_alive` in ConsumerStats for monitoring # Also fix an another issue which will cause wrong result after restore from checkpoint was: org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data. > Receiver consumer thread should be stoped while while encounting > unrecoverable error > > > Key: KYLIN-4964 > URL: https://issues.apache.org/jira/browse/KYLIN-4964 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.1.1 >Reporter: Xingxing Di >Priority: Major > > org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist > will catch exception and just log it, this behavior will cause critical > problem, like our case: there is no space left on device, `persist` method > failed for many times, we lost several hours of data. > Here is our sulotion which already tested: > # Throw IllegalStorageException while we cannot do the persist > # Then stop the consumer thread > # Add `consumer_thread_alive` in ConsumerStats for monitoring > # Also fix an another issue which will cause wrong result after restore from > checkpoint > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4964) Receiver consumer thread should be stoped while while encounting unrecoverable error
Xingxing Di created KYLIN-4964: -- Summary: Receiver consumer thread should be stoped while while encounting unrecoverable error Key: KYLIN-4964 URL: https://issues.apache.org/jira/browse/KYLIN-4964 Project: Kylin Issue Type: Bug Components: Real-time Streaming Affects Versions: v3.1.1 Reporter: Xingxing Di org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4806) Native kubernetes support for kylin streaming receivers
[ https://issues.apache.org/jira/browse/KYLIN-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-4806: --- Description: Currently kylin manually maintains stream receiver replicasets. Under this design, when we have many cubes, we cannot achieve a perfect balance between resource isolation and maintenance costs: 1. Maintain a large number of replicasets to maintain resource isolation, which will cause a lot of maintenance costs 2. Maintain less replicasets to reduce maintenance costs, so one replicaset should be used for multiple cubes, resource isolation will be a problem. eg: One cube’s source data(Kafka topic) has grown abnormally, which may cause the streaming receiver process unstable. In the context of cloud native, kylin should add native support for kubernetes, and then we can easily achieve *cube level streaming receiver cluster* (a set of replicatset ) by define the necessary parameters such as parallelism, cpu, memory, etc. was: Currently kylin manually maintains stream receiver replicasets. Under this design, when we have many cubes, we cannot achieve a perfect balance between resource isolation and maintenance costs: 1. Maintain a large number of replicasets to maintain resource isolation, which will cause a lot of maintenance costs 2. Maintain less replicasets to reduce maintenance costs, so one replicaset should be used for multiple cubes, resource isolation will be a problem. eg: One cube’s source data(Kafka topic) has grown abnormally, which may cause the streaming receiver process unstable. In the context of cloud native, kylin should add native support for kubernetes, and then we can easily achieve *cube level streaming receiver cluster* (a set of replicatset ) by define the necessary parameters such as parallelism, cpu, memory, etc. > Native kubernetes support for kylin streaming receivers > --- > > Key: KYLIN-4806 > URL: https://issues.apache.org/jira/browse/KYLIN-4806 > Project: Kylin > Issue Type: New Feature > Components: Real-time Streaming >Reporter: Xingxing Di >Priority: Major > > Currently kylin manually maintains stream receiver replicasets. Under this > design, when we have many cubes, we cannot achieve a perfect balance between > resource isolation and maintenance costs: > 1. Maintain a large number of replicasets to maintain resource isolation, > which will cause a lot of maintenance costs > 2. Maintain less replicasets to reduce maintenance costs, so one replicaset > should be used for multiple cubes, resource isolation will be a problem. eg: > One cube’s source data(Kafka topic) has grown abnormally, which may cause the > streaming receiver process unstable. > In the context of cloud native, kylin should add native support for > kubernetes, and then we can easily achieve *cube level streaming receiver > cluster* (a set of replicatset ) by define the necessary parameters such as > parallelism, cpu, memory, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4806) Native kubernetes support for kylin streaming receivers
Xingxing Di created KYLIN-4806: -- Summary: Native kubernetes support for kylin streaming receivers Key: KYLIN-4806 URL: https://issues.apache.org/jira/browse/KYLIN-4806 Project: Kylin Issue Type: New Feature Components: Real-time Streaming Reporter: Xingxing Di Currently kylin manually maintains stream receiver replicasets. Under this design, when we have many cubes, we cannot achieve a perfect balance between resource isolation and maintenance costs: 1. Maintain a large number of replicasets to maintain resource isolation, which will cause a lot of maintenance costs 2. Maintain less replicasets to reduce maintenance costs, so one replicaset should be used for multiple cubes, resource isolation will be a problem. eg: One cube’s source data(Kafka topic) has grown abnormally, which may cause the streaming receiver process unstable. In the context of cloud native, kylin should add native support for kubernetes, and then we can easily achieve *cube level streaming receiver cluster* (a set of replicatset ) by define the necessary parameters such as parallelism, cpu, memory, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521957#comment-16521957
]
Xingxing Di edited comment on KYLIN-3282 at 6/25/18 7:58 AM:
-
Hi Shaofeng, now I can't find the error log which cause job disappeared, but i
saw a few "Overwriting conflict " error logs like the origin post by
readme_kylin.
{code:java}
2018-06-10 19:34:51,044 ERROR [Scheduler 372961027 Job
39a3698c-c961-456d-a655-3a9c5f8dc188-2194] common.MapReduceExecutable:195 :
error execute Map
ReduceExecutable{id=39a3698c-c961-456d-a655-3a9c5f8dc188-15, name=Convert
Cuboid Data to HFile, state=RUNNING}
java.lang.IllegalStateException: Overwriting conflict
/execute_output/39a3698c-c961-456d-a655-3a9c5f8dc188-15, expect old TS
1528630470288, but it i
s 1528630481128
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:294)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:280)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:260)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:104)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:218)
at
org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:470)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:160)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
was (Author: dixingx...@yeah.net):
Hi Shaofeng, now I can't find the error log cause job disappeared, but i saw
few "Overwriting conflict " error logs like the origin post by readme_kylin.
{code:java}
2018-06-10 19:34:51,044 ERROR [Scheduler 372961027 Job
39a3698c-c961-456d-a655-3a9c5f8dc188-2194] common.MapReduceExecutable:195 :
error execute Map
ReduceExecutable{id=39a3698c-c961-456d-a655-3a9c5f8dc188-15, name=Convert
Cuboid Data to HFile, state=RUNNING}
java.lang.IllegalStateException: Overwriting conflict
/execute_output/39a3698c-c961-456d-a655-3a9c5f8dc188-15, expect old TS
1528630470288, but it i
s 1528630481128
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:294)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:280)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:260)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:104)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:218)
at
org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:470)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:160)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at
[jira] [Commented] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521957#comment-16521957
]
Xingxing Di commented on KYLIN-3282:
Hi Shaofeng, now I can't find the error log cause job disappeared, but i saw
few "Overwriting conflict " error logs like the origin post by readme_kylin.
{code:java}
2018-06-10 19:34:51,044 ERROR [Scheduler 372961027 Job
39a3698c-c961-456d-a655-3a9c5f8dc188-2194] common.MapReduceExecutable:195 :
error execute Map
ReduceExecutable{id=39a3698c-c961-456d-a655-3a9c5f8dc188-15, name=Convert
Cuboid Data to HFile, state=RUNNING}
java.lang.IllegalStateException: Overwriting conflict
/execute_output/39a3698c-c961-456d-a655-3a9c5f8dc188-15, expect old TS
1528630470288, but it i
s 1528630481128
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:294)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:280)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:260)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:104)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:218)
at
org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:470)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:160)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2018-03-09 12:52:10,191 ERROR [Scheduler 9772827 Job
> 499477a7-4c1a-4c5a-8d4a-0b3218a58dca-516] execution.AbstractExecutable:100 :
> 1th retries for onExecuteFinished fails due to {}
> java.lang.IllegalStateException: Overwriting conflict
> /execute_output/499477a7-4c1a-4c5a-8d4a-0b3218a58dca-13, expect old TS
> 1520571099067, but it is 1520571112216
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
> at
>
[jira] [Comment Edited] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507411#comment-16507411
]
Xingxing Di edited comment on KYLIN-3282 at 6/10/18 3:10 PM:
-
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed ， the error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in this error, i don't see a regular way to resolve
this situation. In this case I can delete the segment and rebuild , but i think
this error may occur in any hbase persistance cases, so there should be a
regular way to deal with hbase persist timeout situation.
was (Author: dixingx...@yeah.net):
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed， this error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in the error, i don't see a regular way to resolve
this situation.
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2018-03-09 12:52:10,191 ERROR [Scheduler 9772827 Job
> 499477a7-4c1a-4c5a-8d4a-0b3218a58dca-516] execution.AbstractExecutable:100 :
> 1th retries for onExecuteFinished fails due to {}
> java.lang.IllegalStateException: Overwriting conflict
> /execute_output/499477a7-4c1a-4c5a-8d4a-0b3218a58dca-13, expect old TS
> 1520571099067, but it is 1520571112216
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
> at
> org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
> at
> org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:415)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
> at
>
[jira] [Commented] (KYLIN-3282) hbase timeout cause the endless status.
[
https://issues.apache.org/jira/browse/KYLIN-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507411#comment-16507411
]
Xingxing Di commented on KYLIN-3282:
Is anyone working on this? we had same issue on kylin2.2.0, the same error
occurred while after "Build Base Cuboid" step succeed， this error cause the job
disappeared in " Monitor view" , and the segment can't refresh for "Segments
overlaps".
Now this cube just stucked in the error, i don't see a regular way to resolve
this situation.
> hbase timeout cause the endless status.
> ---
>
> Key: KYLIN-3282
> URL: https://issues.apache.org/jira/browse/KYLIN-3282
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.0
>Reporter: readme_kylin
>Priority: Major
>
> ri Mar 09 12:52:07 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1520571112216, pause=100, retries=1},
> java.io.IOException: Call to QZ140/10.0.0.140:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=8030361,
> waitTime=15002, operationTimeout=15000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> 2018-03-09 12:52:10,191 ERROR [Scheduler 9772827 Job
> 499477a7-4c1a-4c5a-8d4a-0b3218a58dca-516] execution.AbstractExecutable:100 :
> 1th retries for onExecuteFinished fails due to {}
> java.lang.IllegalStateException: Overwriting conflict
> /execute_output/499477a7-4c1a-4c5a-8d4a-0b3218a58dca-13, expect old TS
> 1520571099067, but it is 1520571112216
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:316)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:305)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:291)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:271)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
> at
> org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
> at
> org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:415)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
[jira] [Closed] (KYLIN-3170) Kylin generate wrong cuboids
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di closed KYLIN-3170. -- Resolution: Not A Bug > Kylin generate wrong cuboids > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.6.0, v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3170) Kylin generate wrong cuboids
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328503#comment-16328503 ] Xingxing Di edited comment on KYLIN-3170 at 1/17/18 8:52 AM: - I debugged "Cuboid.isValid" method, i found there is a config property : "kylin.cube.aggrgroup.isMandatoryOnlyValid" , after i set it to true the generated cuboids just like i expected. I will close this issue. was (Author: dixingx...@yeah.net): I debugged "Cuboid.isValid" method, i found there is a config property : "kylin.cube.aggrgroup.isMandatoryOnlyValid" , after i set it to true the generated cuboids just like i expected. I will close this issue. > Kylin generate wrong cuboids > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.6.0, v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3170) Kylin generate wrong cuboids
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328503#comment-16328503 ] Xingxing Di commented on KYLIN-3170: I debugged "Cuboid.isValid" method, i found there is a config property : "kylin.cube.aggrgroup.isMandatoryOnlyValid" , after i set it to true the generated cuboids just like i expected. I will close this issue. > Kylin generate wrong cuboids > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.6.0, v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3170) Kylin generate wrong cuboids
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-3170: --- Affects Version/s: v1.6.0 > Kylin generate wrong cuboids > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.6.0, v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3170) Kylin generate wrong cuboids
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-3170: --- Summary: Kylin generate wrong cuboids (was: Cuboid wrong) > Kylin generate wrong cuboids > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.6.0, v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3170) Cuboid wrong
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-3170: --- Description: We have a cube which has 5 dimensions and 2 agg groups: 5 dimensions : A, B, C, D, E agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) The cube generate 5 cuboids : Cuboid 1 Cuboid 11011 Cuboid 11010 Cuboid 11101 Cuboid 11100 there this no cuboid 11000 , I'm confused about this, I think this should be a bug. was: We have a cube which has 5 dimensions and 2 agg groups: 5 dimensions : A, B, C, D, E agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) The cube generate 5 cuboids : | Cuboid 1 | Cuboid 11011| | Cuboid 11010 | Cuboid 11101 | Cuboid 11100 there this no cuboid 11000 , I'm confused about this, I think this should be a bug. > Cuboid wrong > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > > Cuboid 1 > Cuboid 11011 > Cuboid 11010 > Cuboid 11101 > Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3170) Cuboid
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-3170: --- Description: We have a cube which has 5 dimensions and 2 agg groups: 5 dimensions : A, B, C, D, E agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) The cube generate 5 cuboids : | Cuboid 1 | Cuboid 11011| | Cuboid 11010 | Cuboid 11101 | Cuboid 11100 there this no cuboid 11000 , I'm confused about this, I think this should be a bug. was: We have a cube which has 5 dimensions and 2 agg groups: 5 dimensions : A, B, C, D, E agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) The cube generate 5 cuboids : | Cuboid 1 | Cuboid 11011 | Cuboid 11010 | Cuboid 11101 | Cuboid 11100 there this no cuboid 11000 , I'm confused about this, I think this should be a bug. > Cuboid > --- > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > | Cuboid 1 > | > Cuboid 11011| > | Cuboid 11010 > | Cuboid 11101 > | Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3170) Cuboid wrong
[ https://issues.apache.org/jira/browse/KYLIN-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-3170: --- Summary: Cuboid wrong (was: Cuboid ) > Cuboid wrong > > > Key: KYLIN-3170 > URL: https://issues.apache.org/jira/browse/KYLIN-3170 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0 > Environment: kylin2.2.0, hbase1.2.4 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Major > > We have a cube which has 5 dimensions and 2 agg groups: > 5 dimensions : A, B, C, D, E > agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) > agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) > > The cube generate 5 cuboids : > | Cuboid 1 > | > Cuboid 11011| > | Cuboid 11010 > | Cuboid 11101 > | Cuboid 11100 > > there this no cuboid 11000 , I'm confused about this, I think this should be > a bug. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3170) Cuboid
Xingxing Di created KYLIN-3170: -- Summary: Cuboid Key: KYLIN-3170 URL: https://issues.apache.org/jira/browse/KYLIN-3170 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.2.0 Environment: kylin2.2.0, hbase1.2.4 Reporter: Xingxing Di Assignee: Shaofeng SHI We have a cube which has 5 dimensions and 2 agg groups: 5 dimensions : A, B, C, D, E agg group1 : Mandatory Dimensions (A, B), Hierarchy Dimensions (D, E) agg group2 : Mandatory Dimensions (A, B), Hierarchy Dimensions (C, E) The cube generate 5 cuboids : | Cuboid 1 | Cuboid 11011 | Cuboid 11010 | Cuboid 11101 | Cuboid 11100 there this no cuboid 11000 , I'm confused about this, I think this should be a bug. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274038#comment-16274038
]
Xingxing Di edited comment on KYLIN-3059 at 12/1/17 7:04 AM:
-
[~liyang.g...@gmail.com] Happy to hear that .
I saw this in kylin2.2 :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc().getIdentity().equals(column.getTable().getIdentity()));
this.table = table;
this.column = column;
}
{code}
I had merge this into our kylin (v1.6) , I'm not sure if that can resolve the
problem.
was (Author: dixingx...@yeah.net):
[~liyang.g...@gmail.com] Happy to hear that .
I saw this in kylin2.2 :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc().getIdentity().equals(column.getTable().getIdentity()));
this.table = table;
this.column = column;
}
{code}
I had merge this into our kylin(v 1.6) , I'm not sure if that can resolve the
problem.
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274038#comment-16274038
]
Xingxing Di commented on KYLIN-3059:
[~liyang.g...@gmail.com] Happy to hear that .
I saw this in kylin2.2 :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc().getIdentity().equals(column.getTable().getIdentity()));
this.table = table;
this.column = column;
}
{code}
I had merge this into our kylin(v 1.6) , I'm not sure if that can resolve the
problem.
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266294#comment-16266294
]
Xingxing Di commented on KYLIN-3059:
I am using v1.6.0, is there any risk to upgrade to v2.2 ? There are a lot of
cubes, is there a way to rollback safely ?
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingxing Di updated KYLIN-3059:
---
Fix Version/s: (was: all)
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingxing Di updated KYLIN-3059:
---
Description:
A query got an exception , the error trace was :
{code:java}
Caused by: java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
at
org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
at
org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
{code}
the code is :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc() == column.getTable()); // this line
cause exception
this.table = table;
this.column = column;
}
{code}
and after we did "Reload metadata", query executed successful.
I dont know what will cause this exception, i think it may related to
ProjectL2Cache or ProjectManager.
was:
A query got an exception , the error trace was :
Caused by: java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
at
org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
at
org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
the code is :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc() == column.getTable()); // this line
cause exception
this.table = table;
this.column = column;
}
{code}
and after we did "Reload metadata", query executed successful.
I dont know what will cause this exception, i think it may related to
ProjectL2Cache or ProjectManager.
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-3059) Query got IllegalArgumentException
[
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingxing Di updated KYLIN-3059:
---
Description:
A query got an exception , the error trace was :
Caused by: java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
at
org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
at
org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
the code is :
{code:java}
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc() == column.getTable()); // this line
cause exception
this.table = table;
this.column = column;
}
{code}
and after we did "Reload metadata", query executed successful.
I dont know what will cause this exception, i think it may related to
ProjectL2Cache or ProjectManager.
was:
A query got an exception , the error trace was :
Caused by: java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
at
org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
at
org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
the code is :
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc() == column.getTable()); // this line
cause exception
this.table = table;
this.column = column;
}
and after we did "Reload metadata", query executed successful.
I dont know what will cause this exception, i think it may related to
ProjectL2Cache or ProjectManager.
> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> Caused by: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> the code is :
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this
> line cause exception
> this.table = table;
> this.column = column;
> }
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception, i think it may related to
> ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-3059) Query got IllegalArgumentException
Xingxing Di created KYLIN-3059:
--
Summary: Query got IllegalArgumentException
Key: KYLIN-3059
URL: https://issues.apache.org/jira/browse/KYLIN-3059
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v1.6.0
Environment: apache-kylin-1.6.0-hbase1.x
apache hadoop 2.7.2
hbase-1.2.4
Reporter: Xingxing Di
Assignee: liyang
Fix For: all
Attachments: query.log
A query got an exception , the error trace was :
Caused by: java.lang.IllegalArgumentException
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
at
org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
at
org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
the code is :
TblColRef(TableRef table, ColumnDesc column) {
checkArgument(table.getTableDesc() == column.getTable()); // this line
cause exception
this.table = table;
this.column = column;
}
and after we did "Reload metadata", query executed successful.
I dont know what will cause this exception, i think it may related to
ProjectL2Cache or ProjectManager.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-1809) kylin1.5.2 query count distinct with condition "in" will cause timeout .
[ https://issues.apache.org/jira/browse/KYLIN-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392964#comment-15392964 ] Xingxing Di commented on KYLIN-1809: Of cause, it has nothing to do with query syntax, but the sql with IN clause will cause hbase error, i dont know the reason, and all cubes have the same problem. > kylin1.5.2 query count distinct with condition "in" will cause timeout . > -- > > Key: KYLIN-1809 > URL: https://issues.apache.org/jira/browse/KYLIN-1809 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2 > Environment: hbase1.1.2 hadoop 2.4.1 >Reporter: Xingxing Di >Assignee: hongbin ma > > Kylin1.5.2 query count distinct with condition "in" will cause exception . > Note that , Kylin1.5 has no such problem. > select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt ='2016-05-29' and t2.city_id in (110100,440100,440300,310100,330100) > group by dt > The error stack: > 2016-06-21 09:28:59,995 ERROR [http-bio-8070-exec-2] > controller.QueryController:209 : Exception when execute sql > java.sql.SQLException: Error while executing SQL "select > t2.city_name,sum(pv),count(distinct device_id) from > olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt='2016-06-18' and t2.city_id in (110100,440100,440300,310100,330100) > group by t2.city_name > LIMIT 5": Timeout visiting cube! > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:361) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:273) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) > at > org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$c462a47d.query() > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) > at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at >
[jira] [Commented] (KYLIN-1809) kylin1.5.2 query count distinct with condition "in" will cause timeout .
[ https://issues.apache.org/jira/browse/KYLIN-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391279#comment-15391279 ] Xingxing Di commented on KYLIN-1809: This sql you post works fine. This sql also works fine : select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 left join olap.olap_dim_city_all t2 on t1.city_id=t2.id where dt ='2016-05-29' and t2.city_id =110100 group by dt; This sql does not work : select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 left join olap.olap_dim_city_all t2 on t1.city_id=t2.id where dt ='2016-05-29' and t2.city_id in (110100,440100) group by dt; This sql also does not work : select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 left join olap.olap_dim_city_all t2 on t1.city_id=t2.id where dt ='2016-05-29' and (t2.city_id = 110100 or t2.city_id = 440100) group by dt; Hope can help you. > kylin1.5.2 query count distinct with condition "in" will cause timeout . > -- > > Key: KYLIN-1809 > URL: https://issues.apache.org/jira/browse/KYLIN-1809 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2 > Environment: hbase1.1.2 hadoop 2.4.1 >Reporter: Xingxing Di >Assignee: hongbin ma > > Kylin1.5.2 query count distinct with condition "in" will cause exception . > Note that , Kylin1.5 has no such problem. > select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt ='2016-05-29' and t2.city_id in (110100,440100,440300,310100,330100) > group by dt > The error stack: > 2016-06-21 09:28:59,995 ERROR [http-bio-8070-exec-2] > controller.QueryController:209 : Exception when execute sql > java.sql.SQLException: Error while executing SQL "select > t2.city_name,sum(pv),count(distinct device_id) from > olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt='2016-06-18' and t2.city_id in (110100,440100,440300,310100,330100) > group by t2.city_name > LIMIT 5": Timeout visiting cube! > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:361) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:273) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) > at > org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$c462a47d.query() > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) > at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) > at >
[jira] [Issue Comment Deleted] (KYLIN-1809) kylin1.5.2 query count distinct with condition "in" will cause timeout .
[ https://issues.apache.org/jira/browse/KYLIN-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-1809: --- Comment: was deleted (was: Sorry, i almost forgot this. The error log from hbase is : 2016-07-22 14:54:39,593 INFO [B.defaultRpcServer.handler=216,queue=6,port=16020] endpoint.CubeVisitService: Scanned 1 rows. 2016-07-22 14:54:39,595 ERROR [B.defaultRpcServer.handler=216,queue=6,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) at org.apache.kylin.storage.hbase.cube.v2.HBaseReadonlyStore$1$1.next(HBaseReadonlyStore.java:118) at org.apache.kylin.storage.hbase.cube.v2.HBaseReadonlyStore$1$1.next(HBaseReadonlyStore.java:93) at org.apache.kylin.gridtable.GTFilterScanner$2.hasNext(GTFilterScanner.java:88) at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:133) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:286) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:4164) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7435) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1875) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1857) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:722) ) > kylin1.5.2 query count distinct with condition "in" will cause timeout . > -- > > Key: KYLIN-1809 > URL: https://issues.apache.org/jira/browse/KYLIN-1809 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2 > Environment: hbase1.1.2 hadoop 2.4.1 >Reporter: Xingxing Di > > Kylin1.5.2 query count distinct with condition "in" will cause exception . > Note that , Kylin1.5 has no such problem. > select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt ='2016-05-29' and t2.city_id in (110100,440100,440300,310100,330100) > group by dt > The error stack: > 2016-06-21 09:28:59,995 ERROR [http-bio-8070-exec-2] > controller.QueryController:209 : Exception when execute sql > java.sql.SQLException: Error while executing SQL "select > t2.city_name,sum(pv),count(distinct device_id) from > olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt='2016-06-18' and t2.city_id in (110100,440100,440300,310100,330100) > group by t2.city_name > LIMIT 5": Timeout visiting cube! > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:361) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:273) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) > at > org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$c462a47d.query() > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) > at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) >
[jira] [Reopened] (KYLIN-1809) kylin1.5.2 query count distinct with condition "in" will cause timeout .
[ https://issues.apache.org/jira/browse/KYLIN-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di reopened KYLIN-1809: Sorry, i almost forgot this. The error log from hbase is : 2016-07-22 14:54:39,593 INFO [B.defaultRpcServer.handler=216,queue=6,port=16020] endpoint.CubeVisitService: Scanned 1 rows. 2016-07-22 14:54:39,595 ERROR [B.defaultRpcServer.handler=216,queue=6,port=16020] ipc.RpcServer: Unexpected throwable object java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191) at org.apache.kylin.storage.hbase.cube.v2.HBaseReadonlyStore$1$1.next(HBaseReadonlyStore.java:118) at org.apache.kylin.storage.hbase.cube.v2.HBaseReadonlyStore$1$1.next(HBaseReadonlyStore.java:93) at org.apache.kylin.gridtable.GTFilterScanner$2.hasNext(GTFilterScanner.java:88) at org.apache.kylin.gridtable.GTAggregateScanner.iterator(GTAggregateScanner.java:133) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:286) at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:4164) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7435) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1875) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1857) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:722) > kylin1.5.2 query count distinct with condition "in" will cause timeout . > -- > > Key: KYLIN-1809 > URL: https://issues.apache.org/jira/browse/KYLIN-1809 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2 > Environment: hbase1.1.2 hadoop 2.4.1 >Reporter: Xingxing Di > > Kylin1.5.2 query count distinct with condition "in" will cause exception . > Note that , Kylin1.5 has no such problem. > select dt,sum(pv),count(distinct device_id) from olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt ='2016-05-29' and t2.city_id in (110100,440100,440300,310100,330100) > group by dt > The error stack: > 2016-06-21 09:28:59,995 ERROR [http-bio-8070-exec-2] > controller.QueryController:209 : Exception when execute sql > java.sql.SQLException: Error while executing SQL "select > t2.city_name,sum(pv),count(distinct device_id) from > olap.olap_app_flw_platform t1 > left join olap.olap_dim_city_all t2 on t1.city_id=t2.id > where dt='2016-06-18' and t2.city_id in (110100,440100,440300,310100,330100) > group by t2.city_name > LIMIT 5": Timeout visiting cube! > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:143) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:186) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:361) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:273) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:121) > at > org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$c462a47d.query() > at > org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94) > at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at >
[jira] [Commented] (KYLIN-1653) Cube metadata lost
[ https://issues.apache.org/jira/browse/KYLIN-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319815#comment-15319815 ] Xingxing Di commented on KYLIN-1653: Thank you , this helps. > Cube metadata lost > -- > > Key: KYLIN-1653 > URL: https://issues.apache.org/jira/browse/KYLIN-1653 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v1.3.0 >Reporter: Xingxing Di >Assignee: Shaofeng SHI > > I have a cube in READY status, today when i build it i got errors > "java.lang.IllegalStateException: No resource found at -- > /cube_desc/flw_topic_supper.json", i checked metadata in hbase ,it doesn't > exist. but i found "/cube_desc/flw_topic_supper.json" still there. > java.lang.IllegalStateException: No resource found at -- > /cube_desc/flw_topic_supper.json > at > org.apache.kylin.job.hadoop.AbstractHadoopJob.dumpResources(AbstractHadoopJob.java:352) > at > org.apache.kylin.job.hadoop.AbstractHadoopJob.attachKylinPropsAndMetadata(AbstractHadoopJob.java:298) > at > org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:81) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > result code:2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1758) createLookupHiveViewMaterializationStep will create intermediate table for fact table
[ https://issues.apache.org/jira/browse/KYLIN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318437#comment-15318437 ] Xingxing Di commented on KYLIN-1758: I am looking forward that ! > createLookupHiveViewMaterializationStep will create intermediate table for > fact table > -- > > Key: KYLIN-1758 > URL: https://issues.apache.org/jira/browse/KYLIN-1758 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2 > Environment: hadoop2.4, hbase1.1.2 >Reporter: Xingxing Di >Assignee: Shaofeng SHI >Priority: Critical > Fix For: v1.5.2.1 > > > In our model, the fact table is a hive view. When i build cube(I selected > one partition for one day's data), the job was trying to create intermediate > table by sql : > DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; > CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU > LOCATION > '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' > AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; > 1. There is no partition string in where string (I selected only one > partition), which cause a very very big MR. > 2. Note that, our lookup table is not a view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1758) createLookupHiveViewMaterializationStep will create intermediate table for fact table
[ https://issues.apache.org/jira/browse/KYLIN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-1758: --- Description: In our model, the fact table is a hive view. When i build cube(I selected one partition for one day's data), the job was trying to create intermediate table by sql : DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU LOCATION '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; 1. There is no partition string in where string (I selected only one partition), which cause a very very big MR. 2. Note that, our lookup table is not a view. was: In our model, the fact table is a hive view. When i build cube(I selected one partition for one day's data), the job was trying to create intermediate table by sql : DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU LOCATION '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; Note that , there is no partition specified (I selected only one partition), and run a very very big MR. > createLookupHiveViewMaterializationStep will create intermediate table for > fact table > -- > > Key: KYLIN-1758 > URL: https://issues.apache.org/jira/browse/KYLIN-1758 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2 > Environment: hadoop2.4, hbase1.1.2 >Reporter: Xingxing Di >Assignee: Dong Li >Priority: Critical > > In our model, the fact table is a hive view. When i build cube(I selected > one partition for one day's data), the job was trying to create intermediate > table by sql : > DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; > CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU > LOCATION > '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' > AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; > 1. There is no partition string in where string (I selected only one > partition), which cause a very very big MR. > 2. Note that, our lookup table is not a view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1758) createLookupHiveViewMaterializationStep will create intermediate table for fact table
[ https://issues.apache.org/jira/browse/KYLIN-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated KYLIN-1758: --- Description: In our model, the fact table is a hive view. When i build cube(I selected one partition for one day's data), the job was trying to create intermediate table by sql : DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU LOCATION '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; Note that , there is no partition specified (I selected only one partition), and run a very very big MR. was: In out model, the fact table is a hive view. When i build cube, the job was trying to create intermediate table by sql : DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU LOCATION '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; Note that , there is no partition specified (I selected only one partition), and run a very very big MR. > createLookupHiveViewMaterializationStep will create intermediate table for > fact table > -- > > Key: KYLIN-1758 > URL: https://issues.apache.org/jira/browse/KYLIN-1758 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2 > Environment: hadoop2.4, hbase1.1.2 >Reporter: Xingxing Di >Assignee: Dong Li >Priority: Critical > > In our model, the fact table is a hive view. When i build cube(I selected > one partition for one day's data), the job was trying to create intermediate > table by sql : > DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; > CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU > LOCATION > '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' > AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; > Note that , there is no partition specified (I selected only one partition), > and run a very very big MR. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1758) createLookupHiveViewMaterializationStep will create intermediate table for fact table
Xingxing Di created KYLIN-1758: -- Summary: createLookupHiveViewMaterializationStep will create intermediate table for fact table Key: KYLIN-1758 URL: https://issues.apache.org/jira/browse/KYLIN-1758 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v1.5.2 Environment: hadoop2.4, hbase1.1.2 Reporter: Xingxing Di Assignee: Dong Li Priority: Critical In out model, the fact table is a hive view. When i build cube, the job was trying to create intermediate table by sql : DROP TABLE IF EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU; CREATE TABLE IF NOT EXISTS kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU LOCATION '/team/db/kylin15/kylin_metadata15/kylin-65c7ee9a-3024-4633-927c-19e992ed155a/kylin_intermediate_OLAP_OLAP_FLW_PCM_MAU' AS SELECT * FROM OLAP.OLAP_FLW_PCM_MAU; Note that , there is no partition specified (I selected only one partition), and run a very very big MR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1658) Clone cube will change the original cube's cube desc name
[
https://issues.apache.org/jira/browse/KYLIN-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277525#comment-15277525
]
Xingxing Di commented on KYLIN-1658:
And yes , you can close this.
> Clone cube will change the original cube's cube desc name
> --
>
> Key: KYLIN-1658
> URL: https://issues.apache.org/jira/browse/KYLIN-1658
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.x-HBase1.1.3
>Reporter: Xingxing Di
>Assignee: Zhong,Jason
>
> I have a cube "coin" , after i cloned it to "coin_clone" i found the
> original cube's ("coin") cubeDesc name changed to coin_clone(in cubes ->
> JSON(cube) view).
> It seems not right , I don't know if it is a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1658) Clone cube will change the original cube's cube desc name
[
https://issues.apache.org/jira/browse/KYLIN-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277522#comment-15277522
]
Xingxing Di commented on KYLIN-1658:
Can you show me the jira page or the patch of this bug ?
> Clone cube will change the original cube's cube desc name
> --
>
> Key: KYLIN-1658
> URL: https://issues.apache.org/jira/browse/KYLIN-1658
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: 1.x-HBase1.1.3
>Reporter: Xingxing Di
>Assignee: Zhong,Jason
>
> I have a cube "coin" , after i cloned it to "coin_clone" i found the
> original cube's ("coin") cubeDesc name changed to coin_clone(in cubes ->
> JSON(cube) view).
> It seems not right , I don't know if it is a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-1658) Clone cube will change the original cube's cube desc name
Xingxing Di created KYLIN-1658:
--
Summary: Clone cube will change the original cube's cube desc name
Key: KYLIN-1658
URL: https://issues.apache.org/jira/browse/KYLIN-1658
Project: Kylin
Issue Type: Bug
Components: Metadata
Affects Versions: 1.x-HBase1.1.3
Reporter: Xingxing Di
Assignee: Shaofeng SHI
I have a cube "coin" , after i cloned it to "coin_clone" i found the original
cube's ("coin") cubeDesc name changed to coin_clone(in cubes -> JSON(cube)
view).
It seems not right , I don't know if it is a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1653) Cube metadata lost
[ https://issues.apache.org/jira/browse/KYLIN-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272055#comment-15272055 ] Xingxing Di commented on KYLIN-1653: [~Shaofengshi] It seems like https://issues.apache.org/jira/browse/KYLIN-1536 , but this time we didnot change any tables or colunm name. Can you give me some hints? Here is an another question , I see the restore metadata steping is : backup -> reset -> restore , when i reset the whole metadata will kylin working all right ? Is there a way to backup project1 -> reset project1 -> restore project1 , this looks less risk for me. And also when i try to backup metadata i got an exception , so i can not backup the whole metadata : Exception in thread "main" java.io.FileNotFoundException: File does not exist: /team/db/kylin/kylin_metadata/resources/dict/APP.A_INDEX_TIMECONTENT_SERIES_CITY_OBJECT_D_FACT/DIM_OBJECT_ID/e6b5639f-24e2-460e-9197-26dbee48a0cb.dict at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1728) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1671) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1651) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1625) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:497) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:322) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1167) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1155) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1145) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:268) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:235) at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:228) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1318) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:289) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:289) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) at org.apache.kylin.common.persistence.HBaseResourceStore.getInputStream(HBaseResourceStore.java:181) at org.apache.kylin.common.persistence.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:201) at org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:149) at org.apache.kylin.common.persistence.ResourceTool.copyR(ResourceTool.java:88) at org.apache.kylin.common.persistence.ResourceTool.copyR(ResourceTool.java:100) at org.apache.kylin.common.persistence.ResourceTool.copyR(ResourceTool.java:100) at org.apache.kylin.common.persistence.ResourceTool.copyR(ResourceTool.java:100) at
[jira] [Commented] (KYLIN-1568) Got java.nio.BufferOverflowException when Build Base Cuboid Data
[
https://issues.apache.org/jira/browse/KYLIN-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229855#comment-15229855
]
Xingxing Di commented on KYLIN-1568:
Already added the description.
> Got java.nio.BufferOverflowException when Build Base Cuboid Data
> -
>
> Key: KYLIN-1568
> URL: https://issues.apache.org/jira/browse/KYLIN-1568
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.0
> Environment: hadoop2.4.1 , hive1.0 , hbase0.98
>Reporter: Xingxing Di
>Assignee: Dong Li
>
> I looked into the code , found this :
> valueBuf = ByteBuffer.allocate(RowConstants.ROWVALUE_BUFFER_SIZE);
> It looks like the ROWVALUE_BUFFER_SIZE is smaller than the value bytes length.
> The exeption is :
> 2016-04-07 10:18:57,311 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.nio.BufferOverflowException at
> java.nio.HeapByteBuffer.put(HeapByteBuffer.java:200) at
> org.apache.kylin.measure.bitmap.BitmapCounter.writeRegisters(BitmapCounter.java:85)
> at
> org.apache.kylin.measure.bitmap.BitmapSerializer.serialize(BitmapSerializer.java:40)
> at
> org.apache.kylin.measure.bitmap.BitmapSerializer.serialize(BitmapSerializer.java:30)
> at
> org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:89) at
> org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:105)
> at
> org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:43)
>at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> The cube info :
> {
> "uuid": "6990333d-ef59-44e3-b46a-7e571d002698",
> "version": "1.5.0",
> "name": "flw_test2_cube_1",
> "description": "",
> "dimensions": [
> {
> "name": "TMP.TMP_FLW_MAIN_2.SITE_CLASS",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "SITE_CLASS",
> "derived": null
> },
> {
> "name": "TMP.TMP_FLW_MAIN_2.DEVICE_ID",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "DEVICE_ID",
> "derived": null
> },
> {
> "name": "TMP.TMP_FLW_MAIN_2.REF_DOMAIN",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "REF_DOMAIN",
> "derived": null
> },
> {
> "name": "TMP.TMP_FLW_MAIN_2.NEW_FLAG",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "NEW_FLAG",
> "derived": null
> },
> {
> "name": "TMP.TMP_FLW_MAIN_2.DEALER_ID",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "DEALER_ID",
> "derived": null
> },
> {
> "name": "TMP.TMP_FLW_MAIN_2.KEY_WORD",
> "table": "TMP.TMP_FLW_MAIN_2",
> "column": "KEY_WORD",
> "derived": null
> },
> {
> "name": "TMP.OLAP_DIM_CITY_ALL_DERIVED",
> "table": "TMP.OLAP_DIM_CITY_ALL",
> "column": null,
> "derived": [
> "ID",
> "CITY_ID",
> "CITY_NAME",
> "PROVINCE_ID",
> "PROVINCE_NAME"
> ]
> },
> {
> "name": "TMP.OLAP_DIM_SITE_ALL_DERIVED",
> "table": "TMP.OLAP_DIM_SITE_ALL",
> "column": null,
> "derived": [
> "ID",
> "SITE_ID",
> "SITE_NAME",
> "CATEGORY_ID",
> "CATEGORY_NAME",
> "SUB_CATEGORY_ID",
> "SUB_CATEGORY_NAME"
> ]
> },
> {
> "name": "OLAP.OLAP_DIM_DAY_DERIVED",
> "table": "OLAP.OLAP_DIM_DAY",
> "column": null,
> "derived": [
> "DIM_DAY_ID",
> "DAY_DATE",
> "WEEKDAY_ID",
> "DIM_MONTH_ID"
> ]
> },
> {
> "name": "TMP.OLAP_DIM_SPEC_ALL_DERIVED",
> "table": "TMP.OLAP_DIM_SPEC_ALL",
> "column": null,
> "derived": [
> "ID",
> "SPEC_ID",
> "SPEC_NAME",
> "SERIES_ID",
> "SERIES_NAME",
> "BRAND_ID",
> "BRAND_NAME",
> "LEVEL_ID",
> "LEVEL_NAME",
> "FACTORY_ID",
> "FACTORY_NAME"
> ]
> }
> ],
> "measures": [
> {
> "name": "_COUNT_",
> "function": {
> "expression": "COUNT",
> "parameter": {
> "type": "constant",
> "value": "1",
> "next_parameter": null
>
[jira] [Updated] (KYLIN-1568) Got java.nio.BufferOverflowException when Build Base Cuboid Data
[
https://issues.apache.org/jira/browse/KYLIN-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingxing Di updated KYLIN-1568:
---
Description:
I looked into the code , found this :
valueBuf = ByteBuffer.allocate(RowConstants.ROWVALUE_BUFFER_SIZE);
It looks like the ROWVALUE_BUFFER_SIZE is smaller than the value bytes length.
The exeption is :
2016-04-07 10:18:57,311 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.nio.BufferOverflowException at
java.nio.HeapByteBuffer.put(HeapByteBuffer.java:200) at
org.apache.kylin.measure.bitmap.BitmapCounter.writeRegisters(BitmapCounter.java:85)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.serialize(BitmapSerializer.java:40)
at
org.apache.kylin.measure.bitmap.BitmapSerializer.serialize(BitmapSerializer.java:30)
at org.apache.kylin.measure.MeasureCodec.encode(MeasureCodec.java:89)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:105)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.reduce(CuboidReducer.java:43)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
The cube info :
{
"uuid": "6990333d-ef59-44e3-b46a-7e571d002698",
"version": "1.5.0",
"name": "flw_test2_cube_1",
"description": "",
"dimensions": [
{
"name": "TMP.TMP_FLW_MAIN_2.SITE_CLASS",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "SITE_CLASS",
"derived": null
},
{
"name": "TMP.TMP_FLW_MAIN_2.DEVICE_ID",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "DEVICE_ID",
"derived": null
},
{
"name": "TMP.TMP_FLW_MAIN_2.REF_DOMAIN",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "REF_DOMAIN",
"derived": null
},
{
"name": "TMP.TMP_FLW_MAIN_2.NEW_FLAG",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "NEW_FLAG",
"derived": null
},
{
"name": "TMP.TMP_FLW_MAIN_2.DEALER_ID",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "DEALER_ID",
"derived": null
},
{
"name": "TMP.TMP_FLW_MAIN_2.KEY_WORD",
"table": "TMP.TMP_FLW_MAIN_2",
"column": "KEY_WORD",
"derived": null
},
{
"name": "TMP.OLAP_DIM_CITY_ALL_DERIVED",
"table": "TMP.OLAP_DIM_CITY_ALL",
"column": null,
"derived": [
"ID",
"CITY_ID",
"CITY_NAME",
"PROVINCE_ID",
"PROVINCE_NAME"
]
},
{
"name": "TMP.OLAP_DIM_SITE_ALL_DERIVED",
"table": "TMP.OLAP_DIM_SITE_ALL",
"column": null,
"derived": [
"ID",
"SITE_ID",
"SITE_NAME",
"CATEGORY_ID",
"CATEGORY_NAME",
"SUB_CATEGORY_ID",
"SUB_CATEGORY_NAME"
]
},
{
"name": "OLAP.OLAP_DIM_DAY_DERIVED",
"table": "OLAP.OLAP_DIM_DAY",
"column": null,
"derived": [
"DIM_DAY_ID",
"DAY_DATE",
"WEEKDAY_ID",
"DIM_MONTH_ID"
]
},
{
"name": "TMP.OLAP_DIM_SPEC_ALL_DERIVED",
"table": "TMP.OLAP_DIM_SPEC_ALL",
"column": null,
"derived": [
"ID",
"SPEC_ID",
"SPEC_NAME",
"SERIES_ID",
"SERIES_NAME",
"BRAND_ID",
"BRAND_NAME",
"LEVEL_ID",
"LEVEL_NAME",
"FACTORY_ID",
"FACTORY_NAME"
]
}
],
"measures": [
{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "PV",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "PV",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "USER_ID",
"function": {
"expression": "COUNT_DISTINCT",
"parameter": {
"type": "column",
"value": "USER_ID",
"next_parameter": null
},
"returntype": "bitmap"
},
"dependent_measure_ref": null
}
],
"rowkey": {
"rowkey_columns": [
{
"column": "DT",
"encoding": "dict"
},
{
"column":
[jira] [Created] (KYLIN-1568) Got java.nio.BufferOverflowException when Build Base Cuboid Data
Xingxing Di created KYLIN-1568: -- Summary: Got java.nio.BufferOverflowException when Build Base Cuboid Data Key: KYLIN-1568 URL: https://issues.apache.org/jira/browse/KYLIN-1568 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v1.5.0 Environment: hadoop2.4.1 , hive1.0 , hbase0.98 Reporter: Xingxing Di Assignee: Dong Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1559) MR job link is wrong
[ https://issues.apache.org/jira/browse/KYLIN-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229585#comment-15229585 ] Xingxing Di commented on KYLIN-1559: Sorry for bothering, i will close this issue. > MR job link is wrong > > > Key: KYLIN-1559 > URL: https://issues.apache.org/jira/browse/KYLIN-1559 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.0 > Environment: v1.5.0, hadoop2.4.1 hive1.0.0 hbase0.98 >Reporter: Xingxing Di >Assignee: Zhong,Jason >Priority: Minor > > In monitor view, the link of "MRJob" start with "http//" which should be > start with "http://;. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1559) MR job link is wrong
Xingxing Di created KYLIN-1559: -- Summary: MR job link is wrong Key: KYLIN-1559 URL: https://issues.apache.org/jira/browse/KYLIN-1559 Project: Kylin Issue Type: Bug Components: REST Service Affects Versions: v1.5.0 Environment: v1.5.0, hadoop2.4.1 hive1.0.0 hbase0.98 Reporter: Xingxing Di Assignee: Zhong,Jason Priority: Minor In monitor view, the link of "MRJob" start with "http//" which should be start with "http://;. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1536) Cube instance failed to load.
[ https://issues.apache.org/jira/browse/KYLIN-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213924#comment-15213924 ] Xingxing Di commented on KYLIN-1536: Yes , one table was changed and reloaded. So this is not a bug ? > Cube instance failed to load. > - > > Key: KYLIN-1536 > URL: https://issues.apache.org/jira/browse/KYLIN-1536 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: 1.x-HBase1.1.3 > Environment: apache hadoop 2.4.1, hbase1.1.2, hive1.1.0 >Reporter: Xingxing Di > > The cube disapeared in "Cubes" view, got error : > [http-bio-7070-exec-3]:[2016-03-23 > 10:03:13,927][ERROR][org.apache.kylin.rest.service.CubeService.listAllCubes(CubeService.java:197)] > - Cube instance flw_pcm is failed to load ... > Tables seem like ok, and query don't work , tips : > From line 1, column 22 to line 1, column 33: Table 'OLAP_FLW_PCM' not found > while executing SQL: "select count(1) from olap_flw_pcm LIMIT 5" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1536) Cube instance failed to load.
Xingxing Di created KYLIN-1536: -- Summary: Cube instance failed to load. Key: KYLIN-1536 URL: https://issues.apache.org/jira/browse/KYLIN-1536 Project: Kylin Issue Type: Bug Components: General Affects Versions: 1.x-HBase1.1.3 Environment: apache hadoop 2.4.1, hbase1.1.2, hive1.1.0 Reporter: Xingxing Di The cube disapeared in "Cubes" view, got error : [http-bio-7070-exec-3]:[2016-03-23 10:03:13,927][ERROR][org.apache.kylin.rest.service.CubeService.listAllCubes(CubeService.java:197)] - Cube instance flw_pcm is failed to load ... Tables seem like ok, and query don't work , tips : >From line 1, column 22 to line 1, column 33: Table 'OLAP_FLW_PCM' not found >while executing SQL: "select count(1) from olap_flw_pcm LIMIT 5" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
