[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515978#comment-15515978 ] liyang commented on KYLIN-1828: --- Thanks Cheng, this is merged to master. > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: Wang Cheng > Fix For: v1.5.4 > > Attachments: 0001-KYLIN-1828-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436159#comment-15436159 ] hongbin ma commented on KYLIN-1828: --- The patch might need revising as we recently changed segment name convention in KYLIN-1859. Also we need to figure out why so many intermediate tables survived after the GC step (org.apache.kylin.source.hive.HiveMRInput.BatchCubingInputSide#addStepPhase4_Cleanup) > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba >Assignee: hongbin ma > Fix For: v1.5.4 > > Attachments: 0001-KYLIN-1828-fix-OutOfBounds-StorageCleanupJob.patch > > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391071#comment-15391071 ] hongbin ma commented on KYLIN-1828: --- thank you [~cal...@gmail.com] for raising the issue. It seems to be a bug here, I'll follow > java.lang.StringIndexOutOfBoundsException in > org.apache.kylin.storage.hbase.util.StorageCleanupJob > -- > > Key: KYLIN-1828 > URL: https://issues.apache.org/jira/browse/KYLIN-1828 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2.1 >Reporter: Richard Calaba > > While running storage cleanup job: > ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete > true > I see Hive tables in form > kylin_intermediate__1970010100_20160701031500 > in the defaul schema. > While running the above storage cleaner (v.1.5.2.1 - all previously built > Cubes Disabled & Dropped) I am getting an error: > 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with > job status ERROR > usage: StorageCleanupJob > -deleteDelete the unused storage > Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String > index out of range: -2 > at java.lang.String.substring(String.java:1904) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308) > 2016-06-27 22:28:08,486 INFO [Thread-0 > HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper > sessionid=0x154c97461586119 > 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session: > 0x154c97461586119 closed > 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread > shut down -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[
https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352252#comment-15352252
]
Richard Calaba commented on KYLIN-1828:
---
Additional workaround for those who need to cleanup the hive tables manually:
To remove ALL kylin_intermediate hive tables from default schema I run:
hive -e 'use default;show tables "kylin_intermediate_*";' | xargs -I '{}' hive
-e 'use default;drop table {}'
> java.lang.StringIndexOutOfBoundsException in
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> --
>
> Key: KYLIN-1828
> URL: https://issues.apache.org/jira/browse/KYLIN-1828
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2.1
>Reporter: Richard Calaba
>
> While running storage cleanup job:
> ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete
> true
> I see Hive tables in form
> kylin_intermediate__1970010100_20160701031500
> in the defaul schema.
> While running the above storage cleaner (v.1.5.2.1 - all previously built
> Cubes Disabled & Dropped) I am getting an error:
> 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with
> job status ERROR
> usage: StorageCleanupJob
> -deleteDelete the unused storage
> Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String
> index out of range: -2
> at java.lang.String.substring(String.java:1904)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308)
> 2016-06-27 22:28:08,486 INFO [Thread-0
> HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper
> sessionid=0x154c97461586119
> 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x154c97461586119 closed
> 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread
> shut down
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1828) java.lang.StringIndexOutOfBoundsException in org.apache.kylin.storage.hbase.util.StorageCleanupJob
[
https://issues.apache.org/jira/browse/KYLIN-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352250#comment-15352250
]
Richard Calaba commented on KYLIN-1828:
---
Further Analysis of the problem:
StorageCleanupJob.java - line 266 - 270:
266while ((line = reader.readLine()) != null) {
267if (line.startsWith("kylin_intermediate_")) {
268boolean isNeedDel = false;
269String uuid = line.substring(line.length() - uuidLength,
line.length());
270uuid = uuid.replace("_", "-");
Obviously the " String uuid = line.substring(line.length() - uuidLength,
line.length());" on line 269 is causing the String out of bounds exception ->
not sure why - do not have enough info about the assumed pattern for the table
names -> but in my hive DB (default schema) I see kylin table names containing
cube (or model) name and not uuid (kylin_intermediate_ - maybe
this is the cause of the problem - not sure.
Exception rasied there at line 269 is further causing bail out from the method
cleanUnusedIntermediateHiveTable through the call stack - causing the
additional confusing message in the log:
usage: StorageCleanupJob
-delete Delete the unused storage
which indicates that the class was not called with correct parameters - BUT it
was - according to
https://kylin.apache.org/docs/howto/howto_cleanup_storage.html.
> java.lang.StringIndexOutOfBoundsException in
> org.apache.kylin.storage.hbase.util.StorageCleanupJob
> --
>
> Key: KYLIN-1828
> URL: https://issues.apache.org/jira/browse/KYLIN-1828
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v1.5.2.1
>Reporter: Richard Calaba
>
> While running storage cleanup job:
> ./bin/kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete
> true
> I see Hive tables in form
> kylin_intermediate__1970010100_20160701031500
> in the defaul schema.
> While running the above storage cleaner (v.1.5.2.1 - all previously built
> Cubes Disabled & Dropped) I am getting an error:
> 2016-06-27 22:28:08,480 INFO [main StorageCleanupJob:262]: Remove
> intermediate hive table with job id fc44da88-cffc-4710-8726-ff910cf83451 with
> job status ERROR
> usage: StorageCleanupJob
> -deleteDelete the unused storage
> Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String
> index out of range: -2
> at java.lang.String.substring(String.java:1904)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:269)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.run(StorageCleanupJob.java:91)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:308)
> 2016-06-27 22:28:08,486 INFO [Thread-0
> HConnectionManager$HConnectionImplementation:1907]: Closing zookeeper
> sessionid=0x154c97461586119
> 2016-06-27 22:28:08,491 INFO [Thread-0 ZooKeeper:684]: Session:
> 0x154c97461586119 closed
> 2016-06-27 22:28:08,491 INFO [main-EventThread ClientCnxn:509]: EventThread
> shut down
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
