Thanks for reply, Alan. My guess with Storm was wrong. Today I get same behavior with running Storm topology. Anyway, I'd like to know, how can I check that transaction batch was closed correctly?
On Wed, Jul 27, 2016 at 8:09 PM, Alan Gates <[email protected]> wrote: > I don’t know the details of how the storm application that streams into > Hive works, but this sounds like the transaction batches weren’t getting > closed. Compaction can’t happen until those batches are closed. Do you > know how you had storm configured? Also, you might ask separately on the > storm list to see if people have seen this issue before. > > Alan. > > > On Jul 27, 2016, at 03:31, Igor Kuzmenko <[email protected]> wrote: > > > > One more thing. I'm using Apache Storm to stream data in Hive. And when > I turned off Storm topology compactions started to work properly. > > > > On Tue, Jul 26, 2016 at 6:28 PM, Igor Kuzmenko <[email protected]> > wrote: > > I'm using Hive 1.2.1 transactional table. Inserting data in it via Hive > Streaming API. After some time i expect compaction to start but it didn't > happen: > > > > Here's part of log, which shows that compactor initiator thread doesn't > see any delta files: > > 2016-07-26 18:06:52,459 INFO [Thread-8]: compactor.Initiator > (Initiator.java:run(89)) - Checking to see if we should compact > default.data_aaa.dt=20160726 > > 2016-07-26 18:06:52,496 DEBUG [Thread-8]: io.AcidUtils > (AcidUtils.java:getAcidState(432)) - in directory hdfs:// > sorm-master01.msk.mts.ru:8020/apps/hive/warehouse/data_aaa/dt=20160726 > base = null deltas = 0 > > 2016-07-26 18:06:52,496 DEBUG [Thread-8]: compactor.Initiator > (Initiator.java:determineCompactionType(271)) - delta size: 0 base size: 0 > threshold: 0.1 will major compact: false > > > > But in that directory there's actually 23 files: > > > > hadoop fs -ls /apps/hive/warehouse/data_aaa/dt=20160726 > > Found 23 items > > -rw-r--r-- 3 storm hdfs 4 2016-07-26 17:20 > /apps/hive/warehouse/data_aaa/dt=20160726/_orc_acid_version > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:22 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71741256_71741355 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:23 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71762456_71762555 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:25 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71787756_71787855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:26 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71795756_71795855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:27 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71804656_71804755 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:29 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71828856_71828955 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:30 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71846656_71846755 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:32 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71850756_71850855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:33 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71867356_71867455 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:34 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71891556_71891655 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:36 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71904856_71904955 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:37 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71907256_71907355 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:39 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71918756_71918855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:40 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71947556_71947655 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:41 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71960656_71960755 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:43 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71963156_71963255 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:44 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71964556_71964655 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:46 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_71987156_71987255 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:47 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72015756_72015855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:48 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72021356_72021455 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72048756_72048855 > > drwxrwxrwx - storm hdfs 0 2016-07-26 17:50 > /apps/hive/warehouse/data_aaa/dt=20160726/delta_72070856_72070955 > > > > Full log here. > > > > What could go wrong? > > > >
