9114 really does nothing but confirm that 8880 is confusing. What you call a log, I view as a metrics sink that writes to a log, but doesn’t actually use any of configuration bits to configure a sink. That makes it a one-off, and those tend to cause significant issues down the road.
At this point, I’m going to -1 9114 since it does nothing but spread 8880’s pain to more of the system. I’m still inclined to revert 8880 until it can be fixed to be more compliant with the rest of Hadoop. On Oct 7, 2015, at 11:10 AM, Arpit Agarwal <aagar...@hortonworks.com> wrote: > HDFS-8880 does not require HDFS-9114. It is configured like any other logger > by adding '-Dnamenode.metrics.logger=DEBUG,NNMETRICSRFA' or equivalent to > HADOOP_NAMENODE_OPTS. Your concerns sound specific to 9114 and not inherent > to 8880. > > > On 10/6/15, 1:28 PM, "Allen Wittenauer" <a...@altiscale.com> wrote: > >> >> As I mentioned in the original message, take a look at HDFS-9114, which is >> basically making HDFS-8880 operable. >> >> Generating a sink that is not a sink (which is what HDFS-8880 basically >> does), isn’t particularly practical. 8880 would be much more appealing if >> it was a sink. If FileSink isn’t fixable, then let's make a Log4jSink which >> would basically be FileSink with those issues fixed. >> >> If not all NN metrics are getting logged, then that’s probably a bug in >> *all* of the metrics systems, no? Having them show up only in this one off >> sounds absolutely terrible. >> >> On Oct 6, 2015, at 1:11 PM, Arpit Agarwal <aagar...@hortonworks.com> wrote: >> >>> There is no property called hadoop.metrics.log.file. Where do you see it? >>> And there is no property that would show up in the command-line of every >>> process. namenode.metrics.logger would show up in the NN command-line and >>> even that is off by default. >>> >>> FileSink looks limited for practical use. >>> - No support for rolling output files. >>> - The format is hard-coded and not practical to grep. Changing it is >>> incompatible. >>> - Won't write to new output files unless I create the files first with >>> 'touch'. >>> >>> - Not all NN metrics are logged. These last two are likely bugs. >>> >>> >>> >>> >>> >>> >>> On 10/6/15, 11:46 AM, "Allen Wittenauer" <a...@altiscale.com> wrote: >>> >>>> Current state is the custom stuff in HDFS-8880 is mostly undocumented >>>> except for some tuning bits in hdfs-default.xml vs. the javadocs for >>>> metrics2. Neither of which is ideal. >>>> >>>> There’s no doubt the documentation for all of the metrics2 sinks are… >>>> sparse. Which is likely what led to the duplicate functionality. :( >>>> >>>> On Oct 6, 2015, at 11:29 AM, Andrew Wang <andrew.w...@cloudera.com> wrote: >>>> >>>>> If it's duplicate we should probably back it out, but taking a step back, >>>>> is the issue that there isn't good documentation about configuring >>>>> Metrics2 >>>>> / FileSync? I see the API docs, but a user-focused guide on how to >>>>> configure Metrics2 would probably be a welcome addition. >>>>> >>>>> HBase has a blog at https://blogs.apache.org/hbase/ this could also be >>>>> good >>>>> content for a blog post. >>>>> >>>>> Best, >>>>> Andrew >>>>> >>>>> On Tue, Oct 6, 2015 at 11:12 AM, Allen Wittenauer <a...@altiscale.com> >>>>> wrote: >>>>> >>>>>> >>>>>> Folks, >>>>>> >>>>>> I’ve been looking over HDFS-8880 and it’s various follow-on >>>>>> JIRAs. The intentions are good, but the implementation is >>>>>> mostly/effectively a duplicate of the FileSink that’s already part of the >>>>>> Hadoop metrics subsystem. (which therefore means it works with all >>>>>> daemons, >>>>>> out of the box already). Reading through HDFS-9114, it’s pretty obvious >>>>>> now that users are going to get *very* confused as just what happens when >>>>>> they set the “hadoop.metrics.log.file” property. It’s opening a >>>>>> pandora’s >>>>>> box of work, since that property only partially works with one >>>>>> sub-project, >>>>>> will show up on the command line of every daemon, and isn’t documented... >>>>>> >>>>>> I’d like to see this series of patches reverted (they haven’t >>>>>> shipped yet, so now is the time!) and effort placed into updating the >>>>>> metrics2 FileSink to have whatever functionality is missing. >>>>>> >>>>>> Thoughts? >>>> >>>> >> >> >
