I’m guessing that this is an issue in the metastore database where it is unable to read from the transaction tables due to the ingestion rate. What version of Hive are you using? What database are you storing the metadata in?
Alan. > On Nov 29, 2016, at 00:05, Diego Fustes Villadóniga <dfus...@oesia.com> wrote: > > Hi all, > > We are trying to use Hive streaming to ingest data in real time from Flink. > We send batches of data every 5 seconds to Hive. We are working version > 1.1.0-cdh5.8.2. > > The ingestión works fine. However, compactions are not working, the log shows > this error: > > Unable to select next element for compaction, ERROR: could not serialize > access due to concurrent update > > In addition, when we run simple queries like SELECT COUNT(1) FROM events, we > are getting OutOfMemory errors, even though we have assigned 10GB to each > Mapper/Reducer. Seeing the logs, each map task tries to load > all delta files, until it breaks, which does not make much sense to me. > > > I think that we have followed all the steps described in the documentation, > so we are blocked in this point. > > Could you help us?
