I don’t know the details of how the storm application that streams into Hive works, but this sounds like the transaction batches weren’t getting closed. Compaction can’t happen until those batches are closed. Do you know how you had storm configured? Also, you might ask separately on the storm list to see if people have seen this issue before.
Alan. > On Jul 27, 2016, at 03:31, Igor Kuzmenko <f1she...@gmail.com> wrote: > > One more thing. I'm using Apache Storm to stream data in Hive. And when I > turned off Storm topology compactions started to work properly. > > On Tue, Jul 26, 2016 at 6:28 PM, Igor Kuzmenko <f1she...@gmail.com> wrote: > I'm using Hive 1.2.1 transactional table. Inserting data in it via Hive > Streaming API. After some time i expect compaction to start but it didn't > happen: > > Here's part of log, which shows that compactor initiator thread doesn't see > any delta files: > 2016-07-26 18:06:52,459 INFO [Thread-8]: compactor.Initiator > (Initiator.java:run(89)) - Checking to see if we should compact > default.data_aaa.dt=20160726 > 2016-07-26 18:06:52,496 DEBUG [Thread-8]: io.AcidUtils > (AcidUtils.java:getAcidState(432)) - in directory > hdfs://sorm-master01.msk.mts.ru:8020/apps/hive/warehouse/data_aaa/dt=20160726 > base = null deltas = 0 > 2016-07-26 18:06:52,496 DEBUG [Thread-8]: compactor.Initiator > (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 0 > threshold: 0.1 will major compact: false > > But in that directory there's actually 23 files: > > hadoop fs -ls /apps/hive/warehouse/data_aaa/dt=20160726 > Found 23 items > -rw-r--r-- 3 storm hdfs 4 2016-07-26 17:20 > /apps/hive/warehouse/data_aaa/dt=20160726/_orc_acid_version > drwxrwxrwx - storm hdfs 0 2016-07-26 17:22 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71741256_71741355 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:23 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71762456_71762555 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:25 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71787756_71787855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:26 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71795756_71795855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:27 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71804656_71804755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:29 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71828856_71828955 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:30 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71846656_71846755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:32 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71850756_71850855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:33 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71867356_71867455 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:34 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71891556_71891655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:36 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71904856_71904955 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:37 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71907256_71907355 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:39 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71918756_71918855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:40 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71947556_71947655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:41 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71960656_71960755 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:43 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71963156_71963255 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:44 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71964556_71964655 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:46 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71987156_71987255 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:47 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72015756_72015855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:48 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72021356_72021455 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72048756_72048855 > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72070856_72070955 > > Full log here. > > What could go wrong? >
