[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878674#action_12878674 ] Richard Ding commented on PIG-1428: --- +1 > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[
https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877591#action_12877591
]
Ashutosh Chauhan commented on PIG-1428:
---
So, I read through PIG-889. It seems that there never was a documented way to
use counters, reporters etc from UDFs, Load/Store Funcs. Actually, there is a
hacky way to do it, which exists in DefaultAbstractBag.java
{code}
protected void incSpillCount(Enum counter) {
// Increment the spill count
// warn is a misnomer. The function updates the counter. If the update
// fails, it dumps a warning
PigHadoopLogger.getInstance().warn(this, "Spill counter incremented",
counter);
}
{code}
But in PIG-889 Santhosh has argued against for this (mis)use of PigLogger. I
think we need to provide a formal way to Pig users to access counters,
reporters from our interfaces (UDFs, L/S) as PigHadoopLogger is designed for
error-handling (warning aggregation in particular) and not for this purpose.
And we shall mark this class as Internal only, before some one starts using it.
With the same argument, above method where Pig is internally making use of its
own Counters is flawed and needs to be corrected.
> Add getPigStatusReporter() to PigHadoopLogger
> -
>
> Key: PIG-1428
> URL: https://issues.apache.org/jira/browse/PIG-1428
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Ashutosh Chauhan
>Assignee: Dmitriy V. Ryaboy
> Fix For: 0.8.0
>
> Attachments: PIG-1428.patch, PIG-1428.patch
>
>
> Without this getter method, its not possible to get counters, report progress
> etc. from UDFs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877616#action_12877616 ] Ashutosh Chauhan commented on PIG-1428: --- I propose a slightly different approach here. Instead of adding getPigStatusReporter() to PigLogger() interface and the corresponding implementation in PigHadoopLogger, we can add a static singleton method in PigStatusReporter and also add a setContext( TaskInputOutputContext context) We can then set the context in map() and reduce() functions and users will have full access of the reporter object through the static method. This will allow us to keep error logging different then status reporting. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877324#action_12877324 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446095/PIG-1428.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 379 release audit warnings (more than the trunk's current 378 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/318/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876708#action_12876708 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446095/PIG-1428.patch against trunk revision 952098. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 383 release audit warnings (more than the trunk's current 382 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/332/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874865#action_12874865 ] Dmitriy V. Ryaboy commented on PIG-1428: I notice that the issue has been discussed before in PIG-889, and Santosh argued (convincingly) that adding this method to PigLogger might not make sense. Santosh, would you like to suggest a different place to put this functionality? I am not married to using this method, it's just the path of least resistance. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch, PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874218#action_12874218 ] Richard Ding commented on PIG-1428: --- Remove the keyword synchronized from the getter will fix this findbugs warning. Setters are called only once per map/reduce task (during the setup). > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873908#action_12873908 ] Dmitriy V. Ryaboy commented on PIG-1428: Findbugs is quite right to call me out on the synchronization thing. I am not sure why the setter needs to by synchronized; I am even less sure the getter should be. Seems like this would add one more lock every time we want to increment a counter or write a log line, which is unfortunate (I assume those objects handle their own concurrency issues). Can Richard or Pradeep comment on that? > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873901#action_12873901 ] Hadoop QA commented on PIG-1428: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445985/PIG-1428.patch against trunk revision 949057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. -1 release audit. The applied patch generated 386 release audit warnings (more than the trunk's current 385 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h1.grid.sp2.yahoo.net/17/console This message is automatically generated. > Add getPigStatusReporter() to PigHadoopLogger > - > > Key: PIG-1428 > URL: https://issues.apache.org/jira/browse/PIG-1428 > Project: Pig > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Ashutosh Chauhan >Assignee: Dmitriy V. Ryaboy > Fix For: 0.8.0 > > Attachments: PIG-1428.patch > > > Without this getter method, its not possible to get counters, report progress > etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
