Thanks for the jira, will see if that works for us. On Sep 2, 2015 7:11 PM, "Prasanth Jayachandran" < [email protected]> wrote:
> Memory manager is made thread local > https://issues.apache.org/jira/browse/HIVE-10191 > > Can you try the patch from HIVE-10191 and see if that helps? > > On Sep 2, 2015, at 8:58 PM, David Capwell <[email protected]> wrote: > > I'll try that out and see if it goes away (not seen this in the past 24 > hours, no code change). > > Doing this now means that I can't share the memory, so will prob go with a > thread local and allocate fixed sizes to the pool per thread (50% heap / 50 > threads). Will most likely be awhile before I can report back (unless it > fails fast in testing) > On Sep 2, 2015 2:11 PM, "Owen O'Malley" <[email protected]> wrote: > >> (Dropping dev) >> >> Well, that explains the non-determinism, because the MemoryManager will >> be shared across threads and thus the stripes will get flushed at >> effectively random times. >> >> Can you try giving each writer a unique MemoryManager? You'll need to put >> a class into the org.apache.hadoop.hive.ql.io.orc package to get access to >> the necessary class (MemoryManager) and method >> (OrcFile.WriterOptions.memory). We may be missing a synchronization on the >> MemoryManager somewhere and thus be getting a race condition. >> >> Thanks, >> Owen >> >> On Wed, Sep 2, 2015 at 12:57 PM, David Capwell <[email protected]> >> wrote: >> >>> We have multiple threads writing, but each thread works on one file, so >>> orc writer is only touched by one thread (never cross threads) >>> On Sep 2, 2015 11:18 AM, "Owen O'Malley" <[email protected]> wrote: >>> >>>> I don't see how it would get there. That implies that minimum was null, >>>> but the count was non-zero. >>>> >>>> The ColumnStatisticsImpl$StringStatisticsImpl.serialize looks like: >>>> >>>> @Override >>>> OrcProto.ColumnStatistics.Builder serialize() { >>>> OrcProto.ColumnStatistics.Builder result = super.serialize(); >>>> OrcProto.StringStatistics.Builder str = >>>> OrcProto.StringStatistics.newBuilder(); >>>> if (getNumberOfValues() != 0) { >>>> str.setMinimum(getMinimum()); >>>> str.setMaximum(getMaximum()); >>>> str.setSum(sum); >>>> } >>>> result.setStringStatistics(str); >>>> return result; >>>> } >>>> >>>> and thus shouldn't call down to setMinimum unless it had at least some >>>> non-null values in the column. >>>> >>>> Do you have multiple threads working? There isn't anything that should be >>>> introducing non-determinism so for the same input it would fail at the >>>> same point. >>>> >>>> .. Owen >>>> >>>> >>>> >>>> >>>> On Tue, Sep 1, 2015 at 10:51 PM, David Capwell <[email protected]> >>>> wrote: >>>> >>>>> We are writing ORC files in our application for hive to consume. >>>>> Given enough time, we have noticed that writing causes a NPE when >>>>> working with a string column's stats. Not sure whats causing it on >>>>> our side yet since replaying the same data is just fine, it seems more >>>>> like this just happens over time (different data sources will hit this >>>>> around the same time in the same JVM). >>>>> >>>>> Here is the code in question, and below is the exception: >>>>> >>>>> final Writer writer = OrcFile.createWriter(path, >>>>> OrcFile.writerOptions(conf).inspector(oi)); >>>>> try { >>>>> for (Data row : rows) { >>>>> List<Object> struct = Orc.struct(row, inspector); >>>>> writer.addRow(struct); >>>>> } >>>>> } finally { >>>>> writer.close(); >>>>> } >>>>> >>>>> >>>>> Here is the exception: >>>>> >>>>> java.lang.NullPointerException: null >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics$Builder.setMinimum(OrcProto.java:1803) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$StringStatisticsImpl.serialize(ColumnStatisticsImpl.java:411) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$StringTreeWriter.createRowIndexEntry(WriterImpl.java:1255) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:775) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.createRowIndexEntry(WriterImpl.java:775) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.createRowIndexEntry(WriterImpl.java:1978) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1985) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.checkMemory(WriterImpl.java:322) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.MemoryManager.notifyWriters(MemoryManager.java:168) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.MemoryManager.addedRow(MemoryManager.java:157) >>>>> ~[hive-exec-0.14.0.jar:0.14.0] >>>>> at >>>>> org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:2276) >>>>> ~[hive-exec-0.14.0.jar: >>>>> >>>>> >>>>> Versions: >>>>> >>>>> Hadoop: apache 2.2.0 >>>>> Hive Apache: 0.14.0 >>>>> Java 1.7 >>>>> >>>>> >>>>> Thanks for your time reading this email. >>>>> >>>> >>>> >> >
