unsubscribe On Tue, Apr 14, 2015 at 4:28 PM, Gopal Vijayaraghavan <gop...@apache.org> wrote:
> > >0.14 . Acid tables have been a real pain for us. We don¹t believe they are > >production ready. At least in our use cases, Tez crashes for assorted > >reasons or only assigns 1 mapper to the partition. Having delta files and > >no base files borks mapper assignments. > > Some of the chicken-egg problems for those were solved recently in > HIVE-10114. > > Then TEZ-1993 is coming out in the next version of Tez, into which we¹re > plugging in HIVE-7428 (no fix yet). > > Currently delta-only splits have 0 bytes as the ³file size², so it grouped > together to make a 16Mb chunk (rather a huge single 0 sized split). > > Those patches are the effect of me shaving the yak from the ³1 mapper² > issue. > > After which the writer has to follow up on HIVE-9933 to get the locality > of files fixed. > > >name are left scattered about, borking queries. Latency is higher with > >streaming than writing to an orc file in hdfs, forcing obscene quantities > >of buckets and orc files smaller than any reasonable orc stripe / hdfs > >block size. The compactor hangs seemingly at random for no reason we¹ve > >been able to discern. > > I haven¹t seen these issues yet, but I am not dealing with a large volume > insert rate, so haven¹t produced latency issues there. > > Since I work on Hive performance and I haven¹t seen too many bugs filed, > so I haven¹t paid attention to the performance of ACID. > > Please file bugs when you find them, so that it appears on the radar for > folks like me. > > I¹m poking about because I want a live stream into LLAP to work seamlessly > & return sub-second query results when queried (pre-cache/stage & merge > etc). > > >An orc file without a footer is junk data (or, at least, the last stripe > >is junk data). I suppose my question should have been 'what will the hive > >query do when it encounters this? Skip the stripe / file? Error out the > >query? Something else?¹ > > It should throw an exception, because that¹s a corrupt ORC file. > > The trucking demo uses Storm without ACID - this is likely to get better > once we use Apache Falcon to move the data around. > > Cheers, > Gopal > > > -- Chad J. Dotzenrod (630)669-6095 cdotzen...@gmail.com
