[ https://issues.apache.org/jira/browse/PIG-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-2924: ------------------------------- Attachment: PIG-2924-3.patch I updated the patch as follows: {quote} Having the word Computer in the interface name and configs could cause confusion, due to how it's an overloaded term. I don't have any great suggestions though. PigStatsOutputSizeReader? {quote} Changed to {{PigStatsOutputSizeReader}}. {quote} Instead of registering a single new computer it would be ideal if we could register a list of computers. {quote} Fixed. {quote} Each computer could have a boolean supports(POStore poStore) method that returns whether this class supports a given POStore. This can often be done by inspecting the output path. A default URI-based abstract class could help with that part. {quote} Each reader implements {{boolean supports(String uri)}} method. For {{FileBasedOutputSizeReader}}, the output of {{UriUtil.isHDFSFileOrLocalOrS3N()}} is returned. {quote} The computers would then be consulted in order, where the first to support the POStore wins. {quote} Fixed. {quote} If a computer can't determine a size for some reason (i.e., it doesn't support it or an exception occurred), it shouldn't return 0. Instead maybe we reserve -1 for this case and document it as such. {quote} Fixed. In addition, I replaced {{POStore}} with {{String}}. Please let me know what you think. Thanks! > PigStats should not be assuming all Storage classes to be file-based storage > ---------------------------------------------------------------------------- > > Key: PIG-2924 > URL: https://issues.apache.org/jira/browse/PIG-2924 > Project: Pig > Issue Type: Bug > Components: tools > Affects Versions: 0.9.2, 0.10.0 > Reporter: Harsh J > Assignee: Cheolsoo Park > Attachments: PIG-2924-2.patch, PIG-2924-3.patch, PIG-2924.patch > > > Using PigStatsUtil (like Oozie does) to collect JobStats for jobs that use a > HBaseStorage blows up when the stats are asked to be accumulated. > This is because JobStats (which adds stuff up) is assuming all storages are > file based and that it can do listStatus/etc. operations on their > filespec-provided filename. For HBaseStorage, this is set to the tablename > and there's no such file, leading to an exception (FileNotFound or Invalid > URI - depending on using 'tablename' or 'hbase://tablename'). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira