[jira] [Created] (KYLIN-2923) FILTER clause support
Alexander Sterligov created KYLIN-2923: -- Summary: FILTER clause support Key: KYLIN-2923 URL: https://issues.apache.org/jira/browse/KYLIN-2923 Project: Kylin Issue Type: New Feature Reporter: Alexander Sterligov -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2788) HFile is not written to S3
[ https://issues.apache.org/jira/browse/KYLIN-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128412#comment-16128412 ] Alexander Sterligov commented on KYLIN-2788: emr-5.4.0 > HFile is not written to S3 > -- > > Key: KYLIN-2788 > URL: https://issues.apache.org/jira/browse/KYLIN-2788 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: Alexander Sterligov > > I set kylin.hbase.cluster.fs to s3 bucket where hbase lives. > Step "Convert Cuboid Data to HFile" finished without errors. Statistics at > the end of the job said that it has written lot's of data to s3. > But there is no hfiles in kylin_metadata folder (kylin_metadata > /kylin-1e436685-7102-4621-a4cb-6472b866126d//hfile), but only > _temporary folder and _SUCCESS file. > _temporary contains hfiles inside attempt folders. it looks like there were > not copied from _temporary to result dir. But there is no errors neither in > kylin log, nor in reducers' logs. > Then loading empty hfiles produces empty segments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2788) HFile is not written to S3
Alexander Sterligov created KYLIN-2788: -- Summary: HFile is not written to S3 Key: KYLIN-2788 URL: https://issues.apache.org/jira/browse/KYLIN-2788 Project: Kylin Issue Type: Bug Affects Versions: v2.0.0 Reporter: Alexander Sterligov I set kylin.hbase.cluster.fs to s3 bucket where hbase lives. Step "Convert Cuboid Data to HFile" finished without errors. Statistics at the end of the job said that it has written lot's of data to s3. But there is no hfiles in kylin_metadata folder (kylin_metadata /kylin-1e436685-7102-4621-a4cb-6472b866126d//hfile), but only _temporary folder and _SUCCESS file. _temporary contains hfiles inside attempt folders. it looks like there were not copied from _temporary to result dir. But there is no errors neither in kylin log, nor in reducers' logs. Then loading empty hfiles produces empty segments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122999#comment-16122999
]
Alexander Sterligov commented on KYLIN-2749:
No, my core nodes are on-demand.
My configuration was:
- master is on-demand m4.2xlarge, I rised kylin Xmx to 16Gb.
- 5 on-demand core nodes m4.xlarge
- from 0 to 50 task nodes of spot instances m4.xlarge with auto-scaling by
ContainerPendingRatio
Amazon starts regions servers on task nodes and if it's spot instances it work
really bad. I had a lot of issues with lost information. Then I fixed number of
task nodes to 20 and it worked much better.
Finally I moved hbase to separate cluster with on-demand nodes only and I use
another cluster with spot-instances for cube build. It works good and I don's
see such issues anymore. (The only issue is that hfiles are not placed well to
s3).
I've seen an email at user mailing list about best practices running kylin in
clouds. It looks like on-demand instances for hbase regionservers is a must
have.
I think this ticket may be closed as not a bug of kylin. Or you can replace
closeQuietly by some other helper with logging. That's the only place where
exceptions may be swallowed.
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
> Attachments: kylin.log.2017-07-22.tar
>
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
>
[jira] [Comment Edited] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119718#comment-16119718
]
Alexander Sterligov edited comment on KYLIN-2749 at 8/9/17 10:57 AM:
-
It is related to read replica synchronization with rw-replica. I don't use
them. And as you can see from the log - data was successfully written. I've
checked that there is no setAutoFlush(false) in kylin, so every put is
guaranteed to be flushed before return. I think my problem was caused by hbck
and some lost data from WAL.
By the way, I propose to replace closeQuietly by helper which doesn't swallow
exceptions from close() without logging.
was (Author: sterligovak):
It is related to read replica synchronization with rw-replica. I don't use
them. And as you can see from the log - data was successfully written. I've
checked that there is not setAutoFlush(false) in kylin, so every put is
guaranteed to be flushed before return. I think my problem was caused by hbck
and some lost data from WAL.
By the way, I propose to replace closeQuietly by helper which doesn't swallow
exceptions from close() without logging.
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
> Attachments: kylin.log.2017-07-22.tar
>
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
>
[jira] [Comment Edited] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119718#comment-16119718
]
Alexander Sterligov edited comment on KYLIN-2749 at 8/9/17 10:56 AM:
-
It is related to read replica synchronization with rw-replica. I don't use
them. And as you can see from the log - data was successfully written. I've
checked that there is not setAutoFlush(false) in kylin, so every put is
guaranteed to be flushed before return. I think my problem was caused by hbck
and some lost data from WAL.
By the way, I propose to replace closeQuietly by helper which doesn't swallow
exceptions from close() without logging.
was (Author: sterligovak):
It is related to read replicate synchronization with rw-replica. I don't use
them. And as you can see from the log - data was successfully written. I've
checked that there is not setAutoFlush(false) in kylin, so every put is
guaranteed to be flushed before return. I think my problem was caused by hbck
and some lost data from WAL.
By the way, I propose to replace closeQuietly by helper which doesn't swallow
exceptions from close() without logging.
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
> Attachments: kylin.log.2017-07-22.tar
>
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
>
[jira] [Commented] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119718#comment-16119718
]
Alexander Sterligov commented on KYLIN-2749:
It is related to read replicate synchronization with rw-replica. I don't use
them. And as you can see from the log - data was successfully written. I've
checked that there is not setAutoFlush(false) in kylin, so every put is
guaranteed to be flushed before return. I think my problem was caused by hbck
and some lost data from WAL.
By the way, I propose to replace closeQuietly by helper which doesn't swallow
exceptions from close() without logging.
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
> Attachments: kylin.log.2017-07-22.tar
>
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
> at
>
[jira] [Comment Edited] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115922#comment-16115922
]
Alexander Sterligov edited comment on KYLIN-2749 at 8/6/17 8:29 PM:
Sorry for long response.
*The problem seams to be starting from*
2017-07-22 09:22:34,544 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
client.HBaseAdmin:965 : Deleted KYLIN_7MMHCHKVVB
2017-07-22 09:22:34,545 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
steps.MergeGCStep:86 : Dropped htable: KYLIN_7MMHCHKVVB
*Then the job is successfully finished:*
2017-07-22 09:25:41,056 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
execution.ExecutableManager:396 : job id:1655adfa-112e-4ed0-b44e-75a021ec6fcb
from RUNNING to SUCCEED
*You can see no errors in between*
*After 30 minutes Kylin was shut down:*
2017-07-22 09:53:47,666 INFO [Thread-11] threadpool.DefaultScheduler:228 :
Shutingdown Job Engine
2017-07-22 09:55:06,622 WARN [localhost-startStop-1] common.KylinConfig:232 :
KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
*Then merge job is started again:*
2017-07-22 09:55:32,257 INFO [pool-7-thread-1] threadpool.DefaultScheduler:114
: CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=READY} prepare to schedule
2017-07-22 09:55:32,259 INFO [pool-7-thread-1] threadpool.DefaultScheduler:117
: CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=READY} scheduled
*Then it crashes:*
2017-07-22 09:55:32,546 ERROR [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:126 : error running Executable:
MergeDictionaryStep{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb-00, name=Merge
Cuboid Dictionary, state=RUNNING}
2017-07-22 09:55:32,547 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb-00
2017-07-22 09:55:32,551 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb-00
2017-07-22 09:55:32,554 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.ExecutableManager:396 : job
id:1655adfa-112e-4ed0-b44e-75a021ec6fcb-00 from RUNNING to ERROR
2017-07-22 09:55:32,555 ERROR [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:126 : error running Executable:
CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=RUNNING}
2017-07-22 09:55:32,556 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb
2017-07-22 09:55:32,559 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb
2017-07-22 09:55:32,561 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.ExecutableManager:396 : job id:1655adfa-112e-4ed0-b44e-75a021ec6fcb
from RUNNING to ERROR
2017-07-22 09:55:32,562 WARN [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:258 : no need to send email, user list is empty
2017-07-22 09:55:32,562 ERROR [pool-8-thread-1] threadpool.DefaultScheduler:146
: ExecuteException job:1655adfa-112e-4ed0-b44e-75a021ec6fcb
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException: java.lang.NullPointerException
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException:
java.lang.NullPointerException
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
... 4 more
Caused by: java.lang.NullPointerException
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:148)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
at
[jira] [Updated] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Sterligov updated KYLIN-2749:
---
Attachment: kylin.log.2017-07-22.tar
Sorry for long response.
*The problem seams to be starting from*
2017-07-22 09:22:34,544 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
client.HBaseAdmin:965 : Deleted KYLIN_7MMHCHKVVB
2017-07-22 09:22:34,545 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
steps.MergeGCStep:86 : Dropped htable: KYLIN_7MMHCHKVVB
*Then it is successfully finished:*
2017-07-22 09:25:41,056 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-261]
execution.ExecutableManager:396 : job id:1655adfa-112e-4ed0-b44e-75a021ec6fcb
from RUNNING to SUCCEED
*You can see no errors in between*
*After 30 minutes Kylin was shut down:*
2017-07-22 09:53:47,666 INFO [Thread-11] threadpool.DefaultScheduler:228 :
Shutingdown Job Engine
2017-07-22 09:55:06,622 WARN [localhost-startStop-1] common.KylinConfig:232 :
KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
*Then merge job is started again:*
2017-07-22 09:55:32,257 INFO [pool-7-thread-1] threadpool.DefaultScheduler:114
: CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=READY} prepare to schedule
2017-07-22 09:55:32,259 INFO [pool-7-thread-1] threadpool.DefaultScheduler:117
: CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=READY} scheduled
*Then it crashes:*
2017-07-22 09:55:32,546 ERROR [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:126 : error running Executable:
MergeDictionaryStep{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb-00, name=Merge
Cuboid Dictionary, state=RUNNING}
2017-07-22 09:55:32,547 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb-00
2017-07-22 09:55:32,551 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb-00
2017-07-22 09:55:32,554 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.ExecutableManager:396 : job
id:1655adfa-112e-4ed0-b44e-75a021ec6fcb-00 from RUNNING to ERROR
2017-07-22 09:55:32,555 ERROR [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:126 : error running Executable:
CubingJob{id=1655adfa-112e-4ed0-b44e-75a021ec6fcb, name=cohort_main -
2017071500_2017072200 - MERGE - GMT+03:00 2017-07-22 12:11:32,
state=RUNNING}
2017-07-22 09:55:32,556 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb
2017-07-22 09:55:32,559 DEBUG [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
dao.ExecutableDao:217 : updating job output, id:
1655adfa-112e-4ed0-b44e-75a021ec6fcb
2017-07-22 09:55:32,561 INFO [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.ExecutableManager:396 : job id:1655adfa-112e-4ed0-b44e-75a021ec6fcb
from RUNNING to ERROR
2017-07-22 09:55:32,562 WARN [Job 1655adfa-112e-4ed0-b44e-75a021ec6fcb-197]
execution.AbstractExecutable:258 : no need to send email, user list is empty
2017-07-22 09:55:32,562 ERROR [pool-8-thread-1] threadpool.DefaultScheduler:146
: ExecuteException job:1655adfa-112e-4ed0-b44e-75a021ec6fcb
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException: java.lang.NullPointerException
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException:
java.lang.NullPointerException
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
... 4 more
Caused by: java.lang.NullPointerException
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:148)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
at
org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
at
[jira] [Commented] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098088#comment-16098088
]
Alexander Sterligov commented on KYLIN-2749:
It is reproduced again. It is caused by kylin OOM (system, not OutOfMemoryError)
Merge process is in progress, old segment tables are already removed, but cube
json contains references for them.
Merged segment is in NEW state and "to hfile" job is running.
It looks like it started job, which successfully worked almost up to the end
and then started a new one after OOM.
Does kylin run its jobs in separate process?
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
> at
>
[jira] [Comment Edited] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097540#comment-16097540
]
Alexander Sterligov edited comment on KYLIN-2749 at 7/23/17 7:29 AM:
-
The merge job had finished successfully and new merged segment contained new
segment, but cube.json was old and contained references to old removed
segments. I don't see any errors at this moment in log, but hbase had a failure
at that moment.
What if exception was ignored at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl
during Table.close and internal hbase buffers were not flushed during the
failure?
I mean IOUtils.closeQuietly(table);
was (Author: sterligovak):
The merge job had finished successfully and new merged segment contained new
segment, but cube.json was old and contained references to old removed
segments. I don't see any errors at this moment in log, but hbase had a failure
at that moment.
What if exception was ignored at
org.apache.kylin.storage.hbase.HBaseResourceStore during Table.close and
internal hbase buffers were not flushed during the failure?
{quote}
@Override
protected long checkAndPutResourceImpl(String resPath, byte[] content, long
oldTS, long newTS) throws IOException, IllegalStateException {
Table table =
getConnection().getTable(TableName.valueOf(getAllInOneTableName()));
try {
byte[] row = Bytes.toBytes(resPath);
byte[] bOldTS = oldTS == 0 ? null : Bytes.toBytes(oldTS);
Put put = buildPut(resPath, newTS, row, content, table);
boolean ok = table.checkAndPut(row, B_FAMILY, B_COLUMN_TS, bOldTS,
put);
logger.trace("Update row " + resPath + " from oldTs: " + oldTS + ",
to newTs: " + newTS + ", operation result: " + ok);
if (!ok) {
long real = getResourceTimestampImpl(resPath);
throw new IllegalStateException("Overwriting conflict " +
resPath + ", expect old TS " + oldTS + ", but it is " + real);
}
return newTS;
} finally {
IOUtils.closeQuietly(table);
}
}
{quote}
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
[jira] [Commented] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097540#comment-16097540
]
Alexander Sterligov commented on KYLIN-2749:
The merge job had finished successfully and new merged segment contained new
segment, but cube.json was old and contained references to old removed
segments. I don't see any errors at this moment in log, but hbase had a failure
at that moment.
What if exception was ignored at
org.apache.kylin.storage.hbase.HBaseResourceStore during Table.close and
internal hbase buffers were not flushed during the failure?
{quote}
@Override
protected long checkAndPutResourceImpl(String resPath, byte[] content, long
oldTS, long newTS) throws IOException, IllegalStateException {
Table table =
getConnection().getTable(TableName.valueOf(getAllInOneTableName()));
try {
byte[] row = Bytes.toBytes(resPath);
byte[] bOldTS = oldTS == 0 ? null : Bytes.toBytes(oldTS);
Put put = buildPut(resPath, newTS, row, content, table);
boolean ok = table.checkAndPut(row, B_FAMILY, B_COLUMN_TS, bOldTS,
put);
logger.trace("Update row " + resPath + " from oldTs: " + oldTS + ",
to newTs: " + newTS + ", operation result: " + ok);
if (!ok) {
long real = getResourceTimestampImpl(resPath);
throw new IllegalStateException("Overwriting conflict " +
resPath + ", expect old TS " + oldTS + ", but it is " + real);
}
return newTS;
} finally {
IOUtils.closeQuietly(table);
}
}
{quote}
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
>
[jira] [Commented] (KYLIN-2711) NPE if job output is lost
[ https://issues.apache.org/jira/browse/KYLIN-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097537#comment-16097537 ] Alexander Sterligov commented on KYLIN-2711: [^KYLIN-2711-step-output-parsing-failure-fix] The patch is attached. > NPE if job output is lost > - > > Key: KYLIN-2711 > URL: https://issues.apache.org/jira/browse/KYLIN-2711 > Project: Kylin > Issue Type: Bug >Affects Versions: all >Reporter: Alexander Sterligov >Assignee: Alexander Sterligov > Fix For: v2.1.0 > > Attachments: > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost-for-2.0.x.patch, > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost.patch, > KYLIN-2711-step-output-parsing-failure-fix > > > After job crash caused by YARN failure Kylin entered inconsistent state which > causes NullPointerException when trying to get jobs. > 2017-07-05 09:53:04,471 ERROR [http-bio-7070-exec-18] > controller.BasicController:54 : > org.apache.kylin.rest.exception.InternalErrorException: > java.lang.NullPointerException > at > org.apache.kylin.rest.controller.JobController.list(JobController.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at >
[jira] [Updated] (KYLIN-2740) FileNotFoundException on base cuboid build if GlobalDictionary is used
[ https://issues.apache.org/jira/browse/KYLIN-2740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Sterligov updated KYLIN-2740: --- Attachment: KYLIN-2740-patch > FileNotFoundException on base cuboid build if GlobalDictionary is used > -- > > Key: KYLIN-2740 > URL: https://issues.apache.org/jira/browse/KYLIN-2740 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: Alexander Sterligov >Assignee: kangkaisen > Attachments: KYLIN-2740-patch > > > 2017-07-13 15:25:20,515 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.RuntimeException: > java.io.FileNotFoundException: No such file or directory: > 'home/production/bi/kylin/kylin_metadata/resources/GlobalDict/dict/MART.STAR_MAIN_EVENT/DEVICE_ID/version_1499959477799/.index' > at > org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:129) > at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:264) > at org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:329) > at > org.apache.kylin.cube.CubeSegment.buildDictionaryMap(CubeSegment.java:321) > at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.(BaseCuboidBuilder.java:86) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.setup(BaseCuboidMapperBase.java:70) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.setup(HiveToBaseCuboidMapper.java:36) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.io.FileNotFoundException: No such file or directory: > 'home/production/bi/kylin/kylin_metadata/resources/GlobalDict/dict/MART.STAR_MAIN_EVENT/DEVICE_ID/version_1499959477799/.index' > The reason of the exception is that flushIndex in > org.apache.kylin.dict.AppendTrieDictionary flushes and closes file after > CachedTreeMap is committed. .index file is left in working directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2711) NPE if job output is lost
[ https://issues.apache.org/jira/browse/KYLIN-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097280#comment-16097280 ] Alexander Sterligov commented on KYLIN-2711: I've just found that it's not enough. May I create pull request on github or only patch is acceptable? > NPE if job output is lost > - > > Key: KYLIN-2711 > URL: https://issues.apache.org/jira/browse/KYLIN-2711 > Project: Kylin > Issue Type: Bug >Affects Versions: all >Reporter: Alexander Sterligov >Assignee: Alexander Sterligov > Fix For: v2.1.0 > > Attachments: > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost-for-2.0.x.patch, > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost.patch > > > After job crash caused by YARN failure Kylin entered inconsistent state which > causes NullPointerException when trying to get jobs. > 2017-07-05 09:53:04,471 ERROR [http-bio-7070-exec-18] > controller.BasicController:54 : > org.apache.kylin.rest.exception.InternalErrorException: > java.lang.NullPointerException > at > org.apache.kylin.rest.controller.JobController.list(JobController.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at >
[jira] [Updated] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Sterligov updated KYLIN-2749:
---
Description:
Merge started to work on last 7 segments.
During the process hbase had a failure because of spot-instances shutdown in
Amazon. Data was not lost, because it is at S3.
I stopped kylin and did hbase hbck --repair. During the report of repair I
didn't see any information about lost data, just redistribution of regions.
Then after kylin was started I cannot query data from the last 7 segments:
{quote}
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
not found, got: KYLIN_7H3WSPX1UJ.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
at
com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
at
org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
at
org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
at Baz$1$1.moveNext(Unknown Source)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
at
org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
... 77 more
Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:691)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1796)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1751)
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:182)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
{quote}
[jira] [Updated] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Sterligov updated KYLIN-2749:
---
Description:
Merge started to work on last 7 segments.
During the process hbase had a failure because of spot-instances shutdown in
Amazon. Data was not lost, because it is at S3.
I stopped kylin and did hbase hbck --repair. During the report of repair I
didn't see any information about lost data, just redistribution of regions.
Then after kylin was started I cannot query data from the last 7 segments:
{quote}
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
not found, got: KYLIN_7H3WSPX1UJ.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
at
com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
at
org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
at
org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
at Baz$1$1.moveNext(Unknown Source)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
at
org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
... 77 more
Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:691)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1796)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1751)
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:182)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
{quote}
[jira] [Updated] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Sterligov updated KYLIN-2749:
---
Summary: Merge may remove old segments without saving merged segment (was:
Merge may remove merged segments before proper register of merged segment)
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 query
> segments with error:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
> at
> org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
> at
> org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
> at
> org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
> at
>
[jira] [Created] (KYLIN-2749) Merge may remove merged segments before proper register of merged segment
Alexander Sterligov created KYLIN-2749:
--
Summary: Merge may remove merged segments before proper register
of merged segment
Key: KYLIN-2749
URL: https://issues.apache.org/jira/browse/KYLIN-2749
Project: Kylin
Issue Type: Bug
Affects Versions: v2.0.0
Reporter: Alexander Sterligov
Merge started to work on last 7 segments.
During the process hbase had a failure because of spot-instances shutdown in
Amazon. Data was not lost, because it is at S3.
I stopped kylin and did hbase hbck --repair. During the report of repair I
didn't see any information about lost data, just redistribution of regions.
Then after kylin was started I cannot query data from the last 7 query segments
with error:
{quote}
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
not found, got: KYLIN_7H3WSPX1UJ.
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
at
org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
at
com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
at
org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
at
org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
at
org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
at Baz$1$1.moveNext(Unknown Source)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
at
org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
at
org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
at Baz.bind(Unknown Source)
at
org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
at
org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
at
org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
... 77 more
Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
at
org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
at
org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:505)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:721)
at
org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:691)
at
org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1796)
at
org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1751)
at
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:182)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
[jira] [Commented] (KYLIN-2715) StorageCleanupJob removes intermediate hive tables of jobs in progress
[ https://issues.apache.org/jira/browse/KYLIN-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088508#comment-16088508 ] Alexander Sterligov commented on KYLIN-2715: I've just double checked - it really deletes intermediate table, not working directory. > StorageCleanupJob removes intermediate hive tables of jobs in progress > -- > > Key: KYLIN-2715 > URL: https://issues.apache.org/jira/browse/KYLIN-2715 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: Alexander Sterligov > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2715) StorageCleanupJob removes intermediate hive tables of jobs in progress
[ https://issues.apache.org/jira/browse/KYLIN-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088498#comment-16088498 ] Alexander Sterligov edited comment on KYLIN-2715 at 7/15/17 7:13 AM: - Yes, code looks good, but it is reproduced every time. I can investigate it further, but it seems to me, that the problem is that for segments which build is in progress getLastBuildJobID returns null and StorageCleanupJob removes the whole working dir. Is null last build job id for segments being built expected? It looks strange for me. was (Author: sterligovak): Yes, code looks good, but it reproduces every time. I can investigate it further, but it seems to me, that the problem is that for segments which build is in progress getLastBuildJobID returns null and StorageCleanupJob removes the whole working dir. Is null last build job id for segments being built expected? It looks strange for me. > StorageCleanupJob removes intermediate hive tables of jobs in progress > -- > > Key: KYLIN-2715 > URL: https://issues.apache.org/jira/browse/KYLIN-2715 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: Alexander Sterligov > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2715) StorageCleanupJob removes intermediate hive tables of jobs in progress
[ https://issues.apache.org/jira/browse/KYLIN-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088498#comment-16088498 ] Alexander Sterligov commented on KYLIN-2715: Yes, code looks good, but it reproduces every time. I can investigate it further, but it seems to me, that the problem is that for segments which build is in progress getLastBuildJobID returns null and StorageCleanupJob removes the whole working dir. Is null last build job id for segments being built expected? It looks strange for me. > StorageCleanupJob removes intermediate hive tables of jobs in progress > -- > > Key: KYLIN-2715 > URL: https://issues.apache.org/jira/browse/KYLIN-2715 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: Alexander Sterligov > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2740) FileNotFoundException on base cuboid build if GlobalDictionary is used
Alexander Sterligov created KYLIN-2740: -- Summary: FileNotFoundException on base cuboid build if GlobalDictionary is used Key: KYLIN-2740 URL: https://issues.apache.org/jira/browse/KYLIN-2740 Project: Kylin Issue Type: Bug Affects Versions: v2.0.0 Reporter: Alexander Sterligov 2017-07-13 15:25:20,515 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: java.io.FileNotFoundException: No such file or directory: 'home/production/bi/kylin/kylin_metadata/resources/GlobalDict/dict/MART.STAR_MAIN_EVENT/DEVICE_ID/version_1499959477799/.index' at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:129) at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:264) at org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:329) at org.apache.kylin.cube.CubeSegment.buildDictionaryMap(CubeSegment.java:321) at org.apache.kylin.engine.mr.common.BaseCuboidBuilder.(BaseCuboidBuilder.java:86) at org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.setup(BaseCuboidMapperBase.java:70) at org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.setup(HiveToBaseCuboidMapper.java:36) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.io.FileNotFoundException: No such file or directory: 'home/production/bi/kylin/kylin_metadata/resources/GlobalDict/dict/MART.STAR_MAIN_EVENT/DEVICE_ID/version_1499959477799/.index' The reason of the exception is that flushIndex in org.apache.kylin.dict.AppendTrieDictionary flushes and closes file after CachedTreeMap is committed. .index file is left in working directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2715) StorageCleanupJob removes intermediate hive tables of jobs in progress
Alexander Sterligov created KYLIN-2715: -- Summary: StorageCleanupJob removes intermediate hive tables of jobs in progress Key: KYLIN-2715 URL: https://issues.apache.org/jira/browse/KYLIN-2715 Project: Kylin Issue Type: Bug Affects Versions: v2.0.0 Reporter: Alexander Sterligov -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2711) NPE if job output is lost
[ https://issues.apache.org/jira/browse/KYLIN-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Sterligov updated KYLIN-2711: --- Attachment: 0001-KYLIN-2711-avoid-NPE-if-output-is-lost-for-2.0.x.patch > NPE if job output is lost > - > > Key: KYLIN-2711 > URL: https://issues.apache.org/jira/browse/KYLIN-2711 > Project: Kylin > Issue Type: Bug >Affects Versions: all >Reporter: Alexander Sterligov > Attachments: > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost-for-2.0.x.patch, > 0001-KYLIN-2711-avoid-NPE-if-output-is-lost.patch > > > After job crash caused by YARN failure Kylin entered inconsistent state which > causes NullPointerException when trying to get jobs. > 2017-07-05 09:53:04,471 ERROR [http-bio-7070-exec-18] > controller.BasicController:54 : > org.apache.kylin.rest.exception.InternalErrorException: > java.lang.NullPointerException > at > org.apache.kylin.rest.controller.JobController.list(JobController.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) >
[jira] [Updated] (KYLIN-2711) NPE if job output is lost
[ https://issues.apache.org/jira/browse/KYLIN-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Sterligov updated KYLIN-2711: --- Attachment: 0001-KYLIN-2711-avoid-NPE-if-output-is-lost.patch > NPE if job output is lost > - > > Key: KYLIN-2711 > URL: https://issues.apache.org/jira/browse/KYLIN-2711 > Project: Kylin > Issue Type: Bug >Affects Versions: all >Reporter: Alexander Sterligov > Attachments: 0001-KYLIN-2711-avoid-NPE-if-output-is-lost.patch > > > After job crash caused by YARN failure Kylin entered inconsistent state which > causes NullPointerException when trying to get jobs. > 2017-07-05 09:53:04,471 ERROR [http-bio-7070-exec-18] > controller.BasicController:54 : > org.apache.kylin.rest.exception.InternalErrorException: > java.lang.NullPointerException > at > org.apache.kylin.rest.controller.JobController.list(JobController.java:76) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at >
[jira] [Created] (KYLIN-2711) NPE if job output is lost
Alexander Sterligov created KYLIN-2711: -- Summary: NPE if job output is lost Key: KYLIN-2711 URL: https://issues.apache.org/jira/browse/KYLIN-2711 Project: Kylin Issue Type: Bug Affects Versions: all Reporter: Alexander Sterligov After job crash caused by YARN failure Kylin entered inconsistent state which causes NullPointerException when trying to get jobs. 2017-07-05 09:53:04,471 ERROR [http-bio-7070-exec-18] controller.BasicController:54 : org.apache.kylin.rest.exception.InternalErrorException: java.lang.NullPointerException at org.apache.kylin.rest.controller.JobController.list(JobController.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842) at javax.servlet.http.HttpServlet.service(HttpServlet.java:624) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91) at
