Thanks for the update Steve! Mukund: could you please share the command to get the error above? I tried a few approaches but couldn't reproduce it :(
Thanks again! Best, Chao On Mon, Jan 24, 2022 at 7:16 AM Steve Loughran <ste...@cloudera.com> wrote: > > fix is in t disable auditing, which is now the default > https://issues.apache.org/jira/browse/HADOOP-18094 > > everything is OK for apps which retain the same fs instances for the life > of the app, but not for Hive... > > will do a better fix ASAP where in exchange for loss of auditing after a > GC event, only weak refs are held in maps private to the auditor. > > i will put that in hadoop common as i would want to use the same code in > thread-levek IOStatistics tracking. > there we;d demand create an IOStatistics snapshot per thread, short lived > worker threads for stream io would still update the stats of the thread the > stream was created in. this will let lus collect stats on store io through > the orc/paquet readers for each thread doing work for a job, and include > them in job stats. > > and how would that be useful? well. look at this coimparison of job/task > commit performance with the manifest committer > https://gist.github.com/steveloughran/7dc1e68220db67327b781b345b42c0b8 >
