[ https://issues.apache.org/jira/browse/KYLIN-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nichunen updated KYLIN-4107: ---------------------------- Fix Version/s: v3.0.0-beta > StorageCleanupJob fails to delete Hive tables with "Argument list too long" > error > --------------------------------------------------------------------------------- > > Key: KYLIN-4107 > URL: https://issues.apache.org/jira/browse/KYLIN-4107 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase > Affects Versions: v2.6.2 > Environment: CentOS 7.6, HDP 2.6.5, Kylin 2.6.3 > Reporter: Vsevolod Ostapenko > Priority: Major > Fix For: v3.0.0-beta > > > On a system with multiple Kylin developers that experiment with cube design > and (re)build/drop cube segments often intermediate Hive tables and HBase > left over tables accumulate very quickly. > After a certain point storage cleanup cannot be executed using suggested > method: > {{${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete > true}} > Apparently, storage cleanup job creates a single shell command to drop all > Hive tables, which fails to execute because command line is just too long. > For example: > {quote} > 2019-07-23 17:47:31,611 ERROR [main] job.StorageCleanupJob:377 : Error during > deleting Hive tables > java.io.IOException: Cannot run program "/bin/bash": error=7, Argument list > too long > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) > at > org.apache.kylin.common.util.CliCommandExecutor.runNativeCommand(CliCommandExecutor.java:133) > at > org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:89) > at > org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:83) > at > org.apache.kylin.rest.job.StorageCleanupJob.deleteHiveTables(StorageCleanupJob.java:409) > at > org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTableInternal(StorageCleanupJob.java:375) > at > org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:278) > at > org.apache.kylin.rest.job.StorageCleanupJob.cleanup(StorageCleanupJob.java:151) > at > org.apache.kylin.rest.job.StorageCleanupJob.execute(StorageCleanupJob.java:145) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:27) > Caused by: java.io.IOException: error=7, Argument list too long > at java.lang.UNIXProcess.forkAndExec(Native Method) > at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) > at java.lang.ProcessImpl.start(ProcessImpl.java:134) > at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) > ... 10 moreĀ > {quote} > Instead of composing one long command, storage cleanup need to generate a > script and feed that into beeline or hive CLI. -- This message was sent by Atlassian JIRA (v7.6.14#76016)