[jira] [Commented] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507886#comment-13507886 ] Chris Nauroth commented on HADOOP-8958: --- The only user-facing issue that comes to mind is if someone attempts to configure viewfs on Windows with mount points that are just drive specs (C:, D:, etc.). This will fail fast at initialization time with an error message stating that the mount point is not an absolute path. That could cause some initial confusion, but I think most Windows engineers will make the realization that C: is not absolute and that they need to adjust the configuration to add a path like C:\ or C:\foo. > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507878#comment-13507878 ] Bikas Saha commented on HADOOP-8958: I actually liked the first base class approach better because that makes it more future proof and dev friendly. > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507879#comment-13507879 ] Bikas Saha commented on HADOOP-8958: Do you think the same issue might be triggered in any normal user scenario which does operations equivalent to the test? > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9110) winutils ls off-by-one error indexing MONTHS array can cause access violation
[ https://issues.apache.org/jira/browse/HADOOP-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507860#comment-13507860 ] Chuan Liu commented on HADOOP-9110: --- +1 Thanks for the quick fix! > winutils ls off-by-one error indexing MONTHS array can cause access violation > - > > Key: HADOOP-9110 > URL: https://issues.apache.org/jira/browse/HADOOP-9110 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-9110-branch-1-win.1.patch, > HADOOP-9110-branch-trunk-win.1.patch > > > In ls.c, the LsPrintLine function uses the wMonth field of a SYSTEMTIME > struct to index into MONTHS, an array of 12 elements containing string > representations of the months. The wMonth field is 1-based (1=January), but > indexing into an array is zero-based. This gives the wrong month, and since > we just crossed into month 12, it also has started causing an access > violation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9110) winutils ls off-by-one error indexing MONTHS array can cause access violation
[ https://issues.apache.org/jira/browse/HADOOP-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9110: -- Attachment: HADOOP-9110-branch-trunk-win.1.patch HADOOP-9110-branch-1-win.1.patch I have attached patches for both branch-1-win and branch-trunk-win. Please credit these patches to Thejas Nair, the original author. Thank you, --Chris > winutils ls off-by-one error indexing MONTHS array can cause access violation > - > > Key: HADOOP-9110 > URL: https://issues.apache.org/jira/browse/HADOOP-9110 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-9110-branch-1-win.1.patch, > HADOOP-9110-branch-trunk-win.1.patch > > > In ls.c, the LsPrintLine function uses the wMonth field of a SYSTEMTIME > struct to index into MONTHS, an array of 12 elements containing string > representations of the months. The wMonth field is 1-based (1=January), but > indexing into an array is zero-based. This gives the wrong month, and since > we just crossed into month 12, it also has started causing an access > violation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9110) winutils ls off-by-one error indexing MONTHS array can cause access violation
Chris Nauroth created HADOOP-9110: - Summary: winutils ls off-by-one error indexing MONTHS array can cause access violation Key: HADOOP-9110 URL: https://issues.apache.org/jira/browse/HADOOP-9110 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 1-win, trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth In ls.c, the LsPrintLine function uses the wMonth field of a SYSTEMTIME struct to index into MONTHS, an array of 12 elements containing string representations of the months. The wMonth field is 1-based (1=January), but indexing into an array is zero-based. This gives the wrong month, and since we just crossed into month 12, it also has started causing an access violation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Support on-demand publish of metrics
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507802#comment-13507802 ] Luke Lu commented on HADOOP-9090: - Thanks Mostafa! +1 on v8 pending jenkins. > Support on-demand publish of metrics > > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.4.patch, > HADOOP-9090.justEnhanceDefaultImpl.5.patch, > HADOOP-9090.justEnhanceDefaultImpl.6.patch, > HADOOP-9090.justEnhanceDefaultImpl.7.patch, > HADOOP-9090.justEnhanceDefaultImpl.8.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > Updated description based on feedback: > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way this JIRA solves this problem is adding a new method > publishMetricsNow() to the MetricsSystemImpl() class, that does a synchronous > out-of-band push of the metrics from the sources to the sink. I also add a > method to MetricsSinkAdapter (putMetricsImmediate) to support that change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[ https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507800#comment-13507800 ] Suresh Srinivas commented on HADOOP-9094: - bq. lean towards subclasses of FNF and ACE to maintain consistent manipulation of path based exceptions Daryn can you explain what you mean by this? > Add interface audience and stability annotation to PathExceptions > - > > Key: HADOOP-9094 > URL: https://issues.apache.org/jira/browse/HADOOP-9094 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks > adding interface audience and stability to notation to those exceptions. It > also tracks the comment from HADOOP-9093: > bq. I propose using FileNotFoundException instead of PathNotFoundException as > it is already extensively used. Similarly use AccessControlException instead > of PathAccessException. If folks agree, I will make that change in the next > patch. Alternatively we could at least make these exceptions subclasses of > the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[ https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507796#comment-13507796 ] Daryn Sharp commented on HADOOP-9094: - bq. Question is why is this necessary? This is an error condition and what can you programmatically do? Mincing strings is always problematic and makes it difficult to enhance error messages w/o breaking compatibility. Allowing an exception to be interrogated for information frees us to change the message string. When performing an operation on multiple paths you have the ability to easily determine which failed and why, so perhaps corrective or cleanup action can be performed. Awhile back I think pig requested this ability to avoid mincing strings. Since they call into FsShell for some operations we'd need to ensure we don't break them if a change is deemed necessary. bq. I feel adding equivalent exceptions already available in java and used in hadoop code to me seems unnecessary, even at the expense of losing some grouping. I understand where you're coming from, and I pondered the same issues when initially creating the exceptions. Since the exceptions were confined to FsShell, I favored consistent and easy manipulation of exceptions. If we change the existing exceptions, I'd reluctantly (in the sense of more catch clauses) lean towards subclasses of FNF and ACE to maintain consistent manipulation of path based exceptions. Those that don't care to interrogate path exceptions can catch the base FNF or ACE directly? > Add interface audience and stability annotation to PathExceptions > - > > Key: HADOOP-9094 > URL: https://issues.apache.org/jira/browse/HADOOP-9094 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks > adding interface audience and stability to notation to those exceptions. It > also tracks the comment from HADOOP-9093: > bq. I propose using FileNotFoundException instead of PathNotFoundException as > it is already extensively used. Similarly use AccessControlException instead > of PathAccessException. If folks agree, I will make that change in the next > patch. Alternatively we could at least make these exceptions subclasses of > the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Support on-demand publish of metrics
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.8.patch Like this? > Support on-demand publish of metrics > > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.4.patch, > HADOOP-9090.justEnhanceDefaultImpl.5.patch, > HADOOP-9090.justEnhanceDefaultImpl.6.patch, > HADOOP-9090.justEnhanceDefaultImpl.7.patch, > HADOOP-9090.justEnhanceDefaultImpl.8.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > Updated description based on feedback: > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way this JIRA solves this problem is adding a new method > publishMetricsNow() to the MetricsSystemImpl() class, that does a synchronous > out-of-band push of the metrics from the sources to the sink. I also add a > method to MetricsSinkAdapter (putMetricsImmediate) to support that change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507772#comment-13507772 ] Ivan Mitic commented on HADOOP-8981: Thanks Xuan for the patch! I don't have anything else to add on top of what Luke already called out. Patch looks good. > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch, > HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[ https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507767#comment-13507767 ] Suresh Srinivas commented on HADOOP-9094: - bq. The downside to #2 is you cannot interrogate the exception for the path that had the problem. You'd be forced to mince the message string which is one of the other reasons I made them subclasses of PathIOException. Question is why is this necessary? This is an error condition and what can you programmatically do? bq. Multiple catch clauses would be required to handle path exceptions, FNF, ACE, etc. Any ideas on how to avoid this? One could then catch IOException, no? I feel adding equivalent exceptions already available in java and used in hadoop code to me seems unnecessary, even at the expense of losing some grouping. > Add interface audience and stability annotation to PathExceptions > - > > Key: HADOOP-9094 > URL: https://issues.apache.org/jira/browse/HADOOP-9094 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks > adding interface audience and stability to notation to those exceptions. It > also tracks the comment from HADOOP-9093: > bq. I propose using FileNotFoundException instead of PathNotFoundException as > it is already extensively used. Similarly use AccessControlException instead > of PathAccessException. If folks agree, I will make that change in the next > patch. Alternatively we could at least make these exceptions subclasses of > the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507756#comment-13507756 ] Luke Lu commented on HADOOP-8981: - Thanks for the patch Xuan! The patch lgtm besides some minor nits: # Please add a comment about atMost(2) in the original test to document the intention. # Please fix the style issues according to https://wiki.apache.org/hadoop/CodeReviewChecklist ** Indentation needs to be 2 spaces, I see 2, 3, 4, 8 being used at the same time :) > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch, > HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9090) Support on-demand publish of metrics
[
https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507744#comment-13507744
]
Luke Lu commented on HADOOP-9090:
-
Why don't you just make WaitableMetricsBuffer extends MetricsBuffer? like this:
{code}
static WaitableMetricsBuffer extends MetricsBuffer {
WaitableMetricsBuffer(MetricsBuffer buffer) {
super(buffer);
}
...
}
{code}
Less change to existing code...
> Support on-demand publish of metrics
>
>
> Key: HADOOP-9090
> URL: https://issues.apache.org/jira/browse/HADOOP-9090
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
>Reporter: Mostafa Elhemali
>Priority: Minor
> Attachments: HADOOP-9090.2.patch,
> HADOOP-9090.justEnhanceDefaultImpl.2.patch,
> HADOOP-9090.justEnhanceDefaultImpl.3.patch,
> HADOOP-9090.justEnhanceDefaultImpl.4.patch,
> HADOOP-9090.justEnhanceDefaultImpl.5.patch,
> HADOOP-9090.justEnhanceDefaultImpl.6.patch,
> HADOOP-9090.justEnhanceDefaultImpl.7.patch,
> HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch
>
>
> Updated description based on feedback:
> We have a need to publish metrics out of some short-living processes, which
> is not really well-suited to the current metrics system implementation which
> periodically publishes metrics asynchronously (a behavior that works great
> for long-living processes). Of course I could write my own metrics system,
> but it seems like such a waste to rewrite all the awesome code currently in
> the MetricsSystemImpl and supporting classes.
> The way this JIRA solves this problem is adding a new method
> publishMetricsNow() to the MetricsSystemImpl() class, that does a synchronous
> out-of-band push of the metrics from the sources to the sink. I also add a
> method to MetricsSinkAdapter (putMetricsImmediate) to support that change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8980) TestRPC fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507735#comment-13507735 ] Xuan Gong commented on HADOOP-8980: --- I agree with you. The test failure, i think, is because the race condition. But I think the race condition happens when the client try to set up the I/O stream with the server. It will send the headers to the server and start the connection thread that waits for response. The code is at Client.java, function setupIOstreams(), if we add the thread.sleep(200L) at the end of code (between start() and return), we can make the test pass. So, I think, the test failure because it tried to write before the I/O stream betweem server and client is set up. > TestRPC fails on Windows > > > Key: HADOOP-8980 > URL: https://issues.apache.org/jira/browse/HADOOP-8980 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > This failure may indicate a difference in socket handling on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Support on-demand publish of metrics
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9090: - Attachment: HADOOP-9090.justEnhanceDefaultImpl.7.patch Luke: is the code in the new patch what you have in mind? > Support on-demand publish of metrics > > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.4.patch, > HADOOP-9090.justEnhanceDefaultImpl.5.patch, > HADOOP-9090.justEnhanceDefaultImpl.6.patch, > HADOOP-9090.justEnhanceDefaultImpl.7.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > Updated description based on feedback: > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way this JIRA solves this problem is adding a new method > publishMetricsNow() to the MetricsSystemImpl() class, that does a synchronous > out-of-band push of the metrics from the sources to the sink. I also add a > method to MetricsSinkAdapter (putMetricsImmediate) to support that change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9082) Select and document a platform-independent scripting language for use in Hadoop environment
[ https://issues.apache.org/jira/browse/HADOOP-9082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507715#comment-13507715 ] Eli Collins commented on HADOOP-9082: - I agree with Allen and Doug as well. IMO we should make the existing bash scripts simpler such that maintaining a parallel set of BAT files (if necessary) isn't a big deal. > Select and document a platform-independent scripting language for use in > Hadoop environment > --- > > Key: HADOOP-9082 > URL: https://issues.apache.org/jira/browse/HADOOP-9082 > Project: Hadoop Common > Issue Type: Bug >Reporter: Matt Foley > > This issue is going to be discussed at length in the common-dev@ mailing > list, under topic "[PROPOSAL] introduce Python as build-time and run-time > dependency for Hadoop and throughout Hadoop stack". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9081) Port TestWinUtils from branch-1-win to branch-trunk-win
[ https://issues.apache.org/jira/browse/HADOOP-9081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507705#comment-13507705 ] Chris Nauroth commented on HADOOP-9081: --- Thanks, Ivan. Sounds like a good plan. The scope of this jira then is a merge of branch-1-win winutils to branch-trunk-win (no merge conflicts expected) and TestWinUtils at 100% test pass rate. > Port TestWinUtils from branch-1-win to branch-trunk-win > --- > > Key: HADOOP-9081 > URL: https://issues.apache.org/jira/browse/HADOOP-9081 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > branch-1-win has a test suite named TestWinUtils to cover winutils.exe > functionality. We need to port this to branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507701#comment-13507701
]
Karthik Kambatla commented on HADOOP-9107:
--
>From HADOOP-6221:
bq. I think a good tactic would be rather than trying to make the old RPC stack
interruptible, focus on making Avro something that you can interrupt, so that
going forward you can interrupt client programs trying to talk to unresponsive
servers.
Steve, is there a reason for not making the old RPC stack interruptible?
I feel we should do both - what Hari is proposing here, and what HADOOP-6221
addresses.
> Hadoop IPC client eats InterruptedException and sets interrupt on the thread
> which is not documented
>
>
> Key: HADOOP-9107
> URL: https://issues.apache.org/jira/browse/HADOOP-9107
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 1.1.0, 2.0.2-alpha
>Reporter: Hari Shreedharan
>
> This code in Client.java looks fishy:
> {code}
> public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
> ConnectionId remoteId) throws InterruptedException, IOException {
> Call call = new Call(rpcKind, rpcRequest);
> Connection connection = getConnection(remoteId, call);
> connection.sendParam(call); // send the parameter
> boolean interrupted = false;
> synchronized (call) {
> while (!call.done) {
> try {
> call.wait(); // wait for the result
> } catch (InterruptedException ie) {
> // save the fact that we were interrupted
> interrupted = true;
> }
> }
> if (interrupted) {
> // set the interrupt flag now that we are done waiting
> Thread.currentThread().interrupt();
> }
> if (call.error != null) {
> if (call.error instanceof RemoteException) {
> call.error.fillInStackTrace();
> throw call.error;
> } else { // local exception
> InetSocketAddress address = connection.getRemoteAddress();
> throw NetUtils.wrapException(address.getHostName(),
> address.getPort(),
> NetUtils.getHostname(),
> 0,
> call.error);
> }
> } else {
> return call.getRpcResult();
> }
> }
> }
> {code}
> Blocking calls are expected to throw InterruptedException if that is
> interrupted. Also it seems like this method waits on the call objects even if
> it is interrupted. Currently, this method does not throw an
> InterruptedException, nor is it documented that this method interrupts the
> thread calling it. If it is interrupted, this method should still throw
> InterruptedException, it should not matter if the call was successful or not.
> This is a major issue for clients which do not call this directly, but call
> HDFS client API methods to write to HDFS, which may be interrupted by the
> client due to timeouts, but does not throw InterruptedException. Any HDFS
> client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9057) TestMetricsSystemImpl.testInitFirst fails intermittently
[ https://issues.apache.org/jira/browse/HADOOP-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9057: --- Resolution: Duplicate Status: Resolved (was: Patch Available) Duplicate of HADOOP-8981. > TestMetricsSystemImpl.testInitFirst fails intermittently > > > Key: HADOOP-9057 > URL: https://issues.apache.org/jira/browse/HADOOP-9057 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HADOOP-9057.branch-1-win.metricsrace.patch > > > Error Message > Wanted but not invoked: metricsSink.putMetrics(); -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > Stacktrace > Wanted but not invoked: > metricsSink.putMetrics(); > -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > at > org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) > at > org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) > Standard Output > 2012-10-04 11:43:55,641 INFO impl.MetricsConfig > (MetricsConfig.java:loadFirst(99)) - loaded properties from > hadoop-metrics2-test.properties -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9057) TestMetricsSystemImpl.testInitFirst fails intermittently
[ https://issues.apache.org/jira/browse/HADOOP-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507697#comment-13507697 ] Ivan Mitic commented on HADOOP-9057: Thanks Xuan for posting the patch. I will take a look on the other Jira. Will resolve this one as a duplicate and we can backport from the other one. > TestMetricsSystemImpl.testInitFirst fails intermittently > > > Key: HADOOP-9057 > URL: https://issues.apache.org/jira/browse/HADOOP-9057 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HADOOP-9057.branch-1-win.metricsrace.patch > > > Error Message > Wanted but not invoked: metricsSink.putMetrics(); -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > Stacktrace > Wanted but not invoked: > metricsSink.putMetrics(); > -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > at > org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) > at > org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) > Standard Output > 2012-10-04 11:43:55,641 INFO impl.MetricsConfig > (MetricsConfig.java:loadFirst(99)) - loaded properties from > hadoop-metrics2-test.properties -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9081) Port TestWinUtils from branch-1-win to branch-trunk-win
[ https://issues.apache.org/jira/browse/HADOOP-9081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507691#comment-13507691 ] Ivan Mitic commented on HADOOP-9081: Thanks Chris for filing this Jira. It might be useful to tag along another snapshot of winutils which includes a few bugfixes. At this point, winutils is in pretty stable state in branch-1-win, and converging the two would be great. > Port TestWinUtils from branch-1-win to branch-trunk-win > --- > > Key: HADOOP-9081 > URL: https://issues.apache.org/jira/browse/HADOOP-9081 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > > branch-1-win has a test suite named TestWinUtils to cover winutils.exe > functionality. We need to port this to branch-trunk-win. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9098) Add missing license headers
[ https://issues.apache.org/jira/browse/HADOOP-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-9098: -- Assignee: Arpit Agarwal > Add missing license headers > --- > > Key: HADOOP-9098 > URL: https://issues.apache.org/jira/browse/HADOOP-9098 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1.1.1 >Reporter: Tom White >Assignee: Arpit Agarwal >Priority: Blocker > Fix For: 1.2.0 > > > There are missing license headers in some source files (e.g. > TestUnderReplicatedBlocks.java is one) according to the RAT report. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507660#comment-13507660
]
Hari Shreedharan commented on HADOOP-9107:
--
I agree that both are pretty similar, but I think we still need to do the
cleanup I am proposing here right?
> Hadoop IPC client eats InterruptedException and sets interrupt on the thread
> which is not documented
>
>
> Key: HADOOP-9107
> URL: https://issues.apache.org/jira/browse/HADOOP-9107
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 1.1.0, 2.0.2-alpha
>Reporter: Hari Shreedharan
>
> This code in Client.java looks fishy:
> {code}
> public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
> ConnectionId remoteId) throws InterruptedException, IOException {
> Call call = new Call(rpcKind, rpcRequest);
> Connection connection = getConnection(remoteId, call);
> connection.sendParam(call); // send the parameter
> boolean interrupted = false;
> synchronized (call) {
> while (!call.done) {
> try {
> call.wait(); // wait for the result
> } catch (InterruptedException ie) {
> // save the fact that we were interrupted
> interrupted = true;
> }
> }
> if (interrupted) {
> // set the interrupt flag now that we are done waiting
> Thread.currentThread().interrupt();
> }
> if (call.error != null) {
> if (call.error instanceof RemoteException) {
> call.error.fillInStackTrace();
> throw call.error;
> } else { // local exception
> InetSocketAddress address = connection.getRemoteAddress();
> throw NetUtils.wrapException(address.getHostName(),
> address.getPort(),
> NetUtils.getHostname(),
> 0,
> call.error);
> }
> } else {
> return call.getRpcResult();
> }
> }
> }
> {code}
> Blocking calls are expected to throw InterruptedException if that is
> interrupted. Also it seems like this method waits on the call objects even if
> it is interrupted. Currently, this method does not throw an
> InterruptedException, nor is it documented that this method interrupts the
> thread calling it. If it is interrupted, this method should still throw
> InterruptedException, it should not matter if the call was successful or not.
> This is a major issue for clients which do not call this directly, but call
> HDFS client API methods to write to HDFS, which may be interrupted by the
> client due to timeouts, but does not throw InterruptedException. Any HDFS
> client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunk-win.3.patch > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch, > HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9057) TestMetricsSystemImpl.testInitFirst fails intermittently
[ https://issues.apache.org/jira/browse/HADOOP-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507645#comment-13507645 ] Xuan Gong commented on HADOOP-9057: --- The timeout in mockito verify is a great idea. It actually verify the callback before we call the stop function. But I think that we may need add another test case to verify the stop(). I think that the purpose of this stop() is when I call this function, all the sink threads should stop no matter they are idle or they are doing something. And actually the test is fail, on the other side, it proved that our stop function is actually worked correctly. When we call the stop, everything is stopped no matter what they are doing right now. Based on that, I runned this test several times, I found that sometimes it will be successful, sometimes half rigth half wrong, sometimes all fails. That is why orignally we try to add Thread.sleep() and try to buy some times for the sink threads to consume all the items in queue. Maybe you can take a look at the HADOOP-8981, this is the duplication issue. I uploaded patches verion 3, including the test case that using the mockito timeout and another test case to verify stop function. > TestMetricsSystemImpl.testInitFirst fails intermittently > > > Key: HADOOP-9057 > URL: https://issues.apache.org/jira/browse/HADOOP-9057 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1-win >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: HADOOP-9057.branch-1-win.metricsrace.patch > > > Error Message > Wanted but not invoked: metricsSink.putMetrics(); -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > Stacktrace > Wanted but not invoked: > metricsSink.putMetrics(); > -> at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > Actually, there were zero interactions with this mock. > at > org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testInitFirst(TestMetricsSystemImpl.java:80) > at > org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37) > at > org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62) > Standard Output > 2012-10-04 11:43:55,641 INFO impl.MetricsConfig > (MetricsConfig.java:loadFirst(99)) - loaded properties from > hadoop-metrics2-test.properties -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: (was: HADOOP-8981-branch-trunck-win.3.patch) > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunk-win.1.patch, > HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated HADOOP-8981: -- Attachment: HADOOP-8981-branch-trunck-win.3.patch Actually, using mockito timeout works. We actually verify the callback before we call stop functions. But I still keep the test case that verify the stop function correctly. Patch 3 is the mockito timeout version > TestMetricsSystemImpl fails on Windows > -- > > Key: HADOOP-8981 > URL: https://issues.apache.org/jira/browse/HADOOP-8981 > Project: Hadoop Common > Issue Type: Bug > Components: metrics >Affects Versions: trunk-win >Reporter: Chris Nauroth >Assignee: Arpit Agarwal > Attachments: HADOOP-8981-branch-trunck-win.3.patch, > HADOOP-8981-branch-trunk-win.1.patch, HADOOP-8981-branch-trunk-win.2.patch > > > The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9054) Add AuthenticationHandler that uses Kerberos but allows for an alternate form of authentication for browsers
[ https://issues.apache.org/jira/browse/HADOOP-9054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507599#comment-13507599 ] Alejandro Abdelnur commented on HADOOP-9054: +1. On Allen's comment "the user is going to get bounced around a lot. As a result, this means they might need to re-authenticate on every host if one isn't careful about the implementation." Hadoop-Auth already has provisions for this by allowing you to define a cookie domain. Once you have a hadoop-auth cookie, the authentication-handler code does not kicks in until the cookie expires. > Add AuthenticationHandler that uses Kerberos but allows for an alternate form > of authentication for browsers > > > Key: HADOOP-9054 > URL: https://issues.apache.org/jira/browse/HADOOP-9054 > Project: Hadoop Common > Issue Type: New Feature > Components: security >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.0.3-alpha > > Attachments: HADOOP-9054.patch, HADOOP-9054.patch > > > It would be useful for some Oozie users if, when using Kerberos, that browser > access to the oozie web UI could be authenticated in a different way (w/o > Kerberos). This may be useful for other projects using Hadoop-Auth, so this > feature is to add a new AuthenticationHandler that uses Kerberos by default, > unless a browser (user-agents are configurable) is used, in which case some > other form of authentication can be used. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9109) Support remote shell comands other than ssh in startup scripts
[ https://issues.apache.org/jira/browse/HADOOP-9109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated HADOOP-9109: --- Attachment: HADOOP-9109-tag-release-1.1.0.patch Patch illustrating change in 1.1.0 tag. > Support remote shell comands other than ssh in startup scripts > -- > > Key: HADOOP-9109 > URL: https://issues.apache.org/jira/browse/HADOOP-9109 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: Albert Chu >Priority: Minor > Attachments: HADOOP-9109-tag-release-1.1.0.patch > > > Some organizations have remote shell launching mechanisms other than ssh. It > would be nice to be able to use these other remote shell commands without > hand modifying the startup scripts. > I've attached a locally developed patch for this against Hadoop 1.1.0. It > modifies conf/hadoop-env.sh, bin/slaves.sh, and bin/hadoop-daemon.sh and > supports a HADOOP_SSH_CMD environment variable to allow users to set whatever > remote shell command they want to use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9109) Support remote shell comands other than ssh in startup scripts
Albert Chu created HADOOP-9109: -- Summary: Support remote shell comands other than ssh in startup scripts Key: HADOOP-9109 URL: https://issues.apache.org/jira/browse/HADOOP-9109 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: Albert Chu Priority: Minor Some organizations have remote shell launching mechanisms other than ssh. It would be nice to be able to use these other remote shell commands without hand modifying the startup scripts. I've attached a locally developed patch for this against Hadoop 1.1.0. It modifies conf/hadoop-env.sh, bin/slaves.sh, and bin/hadoop-daemon.sh and supports a HADOOP_SSH_CMD environment variable to allow users to set whatever remote shell command they want to use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507541#comment-13507541 ] Hudson commented on HADOOP-8958: Integrated in Hadoop-trunk-Commit #3076 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3076/]) HADOOP-8958. ViewFs:Non absolute mount name failures when running multiple tests on Windows. Contributed by Chris Nauroth. (Revision 1415784) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1415784 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsTestSetup.java > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8958: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to trunk. Thank you Chris. > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 3.0.0 > > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8958) ViewFs:Non absolute mount name failures when running multiple tests on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507524#comment-13507524 ] Sanjay Radia commented on HADOOP-8958: -- +1 > ViewFs:Non absolute mount name failures when running multiple tests on Windows > -- > > Key: HADOOP-8958 > URL: https://issues.apache.org/jira/browse/HADOOP-8958 > Project: Hadoop Common > Issue Type: Bug > Components: viewfs >Affects Versions: 3.0.0, trunk-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-8958.2.patch, HADOOP-8958.patch > > > This appears to be an issue with parsing a Windows-specific path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
[ https://issues.apache.org/jira/browse/HADOOP-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507466#comment-13507466 ] Bikas Saha commented on HADOOP-9102: Looks like I had written it this way to differentiate between task being alive/dead vs an error in executing the command. This change means any error in execution that results in a non-zero exist code would mean the task is assumed to be dead. Not sure if thats correct but lets leave that for now since the same thing could happen for Linux and the Java code is written with that expectation too. This also makes the isAlive parsing redundant in Hadoop code. +1 > winutils task isAlive does not return a non-zero exit code if the requested > task is not alive > - > > Key: HADOOP-9102 > URL: https://issues.apache.org/jira/browse/HADOOP-9102 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-9102-branch-1-win.1.patch > > > Work on YARN-233 noted that winutils task isAlive does not return a non-zero > exit code if the requested task is not alive. This is inconsistent with the > equivalent check on Unix, kill -0, which uses a non-zero exit code to > communicate status. By changing winutils task isAlive to return a non-zero > exit code, we can make the error handling code on the Java side consistent, > avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9016) Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
[ https://issues.apache.org/jira/browse/HADOOP-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9016: --- Attachment: HADOOP-9016--e.patch HADOOP-9016-branch-0.23--e.patch The patches of version "e" are updated to merge over fix YARN-204. > Provide unit tests for class > org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream > - > > Key: HADOOP-9016 > URL: https://issues.apache.org/jira/browse/HADOOP-9016 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky >Priority: Minor > Attachments: HADOOP-9016--b.patch, HADOOP-9016-branch-0.23--d.patch, > HADOOP-9016-branch-0.23--e.patch, HADOOP-9016--c.patch, HADOOP-9016--d.patch, > HADOOP-9016--e.patch, HADOOP-9016.patch > > > unit-test coverage of classes > org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream, > org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream.HarFsInputStream is > zero. > Suggested to provide unit-tests covering these classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9102) winutils task isAlive does not return a non-zero exit code if the requested task is not alive
[ https://issues.apache.org/jira/browse/HADOOP-9102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9102: -- Attachment: HADOOP-9102-branch-1-win.1.patch This patch ports a similar change applied to branch-trunk-win for YARN-233 to branch-1-win. > winutils task isAlive does not return a non-zero exit code if the requested > task is not alive > - > > Key: HADOOP-9102 > URL: https://issues.apache.org/jira/browse/HADOOP-9102 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 1-win >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-9102-branch-1-win.1.patch > > > Work on YARN-233 noted that winutils task isAlive does not return a non-zero > exit code if the requested task is not alive. This is inconsistent with the > equivalent check on Unix, kill -0, which uses a non-zero exit code to > communicate status. By changing winutils task isAlive to return a non-zero > exit code, we can make the error handling code on the Java side consistent, > avoiding the need for additional if (WINDOWS) checks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[
https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507437#comment-13507437
]
Daryn Sharp commented on HADOOP-9094:
-
The downside to #2 is you cannot interrogate the exception for the path that
had the problem. You'd be forced to mince the message string which is one of
the other reasons I made them subclasses of {{PathIOException}}.
I think Nicholas also pointed out in the related jira:
bq. I also have thought about changing the base class to interface. However, it
may not be very useful since we cannot catch interface as it no longer extends
Throwable.
So as it stands now, you can simply catch {{PathIOException}} for any path
related exception. #1 suffers from essentially the same ailment Nicholas
points out. Multiple catch clauses would be required to handle path
exceptions, FNF, ACE, etc. Any ideas on how to avoid this?
> Add interface audience and stability annotation to PathExceptions
> -
>
> Key: HADOOP-9094
> URL: https://issues.apache.org/jira/browse/HADOOP-9094
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>
> HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks
> adding interface audience and stability to notation to those exceptions. It
> also tracks the comment from HADOOP-9093:
> bq. I propose using FileNotFoundException instead of PathNotFoundException as
> it is already extensively used. Similarly use AccessControlException instead
> of PathAccessException. If folks agree, I will make that change in the next
> patch. Alternatively we could at least make these exceptions subclasses of
> the exception that I am proposing replacing them with.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9082) Select and document a platform-independent scripting language for use in Hadoop environment
[ https://issues.apache.org/jira/browse/HADOOP-9082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507430#comment-13507430 ] Doug Cutting commented on HADOOP-9082: -- I agree with Allen. We should minimize scripting. I fear that Python would encourage more scripting when what we need is less. Scripts should be limited to things that cannot be done in Java. Parallel versions (bash & bat) of a few simple scripts may be a necessary evil if Cygwin is unacceptable. > Select and document a platform-independent scripting language for use in > Hadoop environment > --- > > Key: HADOOP-9082 > URL: https://issues.apache.org/jira/browse/HADOOP-9082 > Project: Hadoop Common > Issue Type: Bug >Reporter: Matt Foley > > This issue is going to be discussed at length in the common-dev@ mailing > list, under topic "[PROPOSAL] introduce Python as build-time and run-time > dependency for Hadoop and throughout Hadoop stack". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[ https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507383#comment-13507383 ] Suresh Srinivas commented on HADOOP-9094: - bq. Maybe I'm confused about what option #2 Sorry my comment was not very clear. Here is another attempt to make it clear: * Move formatting exception message into a static utility method * Option 1: Subclass the well known java/hadoop exception ** PathNotFoundException subclasses FileNotFoundException. It formats the exception message using the utility. ** PathAccessException - rename to PathAccessControlException. Make it a subclass of AccessControlException. It also formats the exception message using the utility. * Option 2: Just use the well known java/hadoop exception ** Remove PathNotFoundException. Use FileNotFoundException. Retain the format of the exception message. ** Remove PathAccessException. Use AccessControlException. Retain the format of the exception message. I am leaning towards Option 2. > Add interface audience and stability annotation to PathExceptions > - > > Key: HADOOP-9094 > URL: https://issues.apache.org/jira/browse/HADOOP-9094 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks > adding interface audience and stability to notation to those exceptions. It > also tracks the comment from HADOOP-9093: > bq. I propose using FileNotFoundException instead of PathNotFoundException as > it is already extensively used. Similarly use AccessControlException instead > of PathAccessException. If folks agree, I will make that change in the next > patch. Alternatively we could at least make these exceptions subclasses of > the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9094) Add interface audience and stability annotation to PathExceptions
[ https://issues.apache.org/jira/browse/HADOOP-9094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507354#comment-13507354 ] Daryn Sharp commented on HADOOP-9094: - bq. I am leaning towards the second option. Maybe I'm confused about what option #2 refers to. Is it this: bq. Other alternative is to blow away the Path*Exception in above cases and use the super class I have propose. If so, I'm unclear which super class that would be? If you are referring to the standard FNF and ACE exceptions, won't you need a subclass which is option 1? > Add interface audience and stability annotation to PathExceptions > - > > Key: HADOOP-9094 > URL: https://issues.apache.org/jira/browse/HADOOP-9094 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > HADOOP-9093 moved path related exceptions to o.a.h.fs. This jira tracks > adding interface audience and stability to notation to those exceptions. It > also tracks the comment from HADOOP-9093: > bq. I propose using FileNotFoundException instead of PathNotFoundException as > it is already extensively used. Similarly use AccessControlException instead > of PathAccessException. If folks agree, I will make that change in the next > patch. Alternatively we could at least make these exceptions subclasses of > the exception that I am proposing replacing them with. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9083) Port HADOOP-9020 Add a SASL PLAIN server to branch 1
[ https://issues.apache.org/jira/browse/HADOOP-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507348#comment-13507348 ] Daryn Sharp commented on HADOOP-9083: - +1 Makes sense, I just wanted to make sure. Although I think this means we need to pull this patch into 23. It's currently in trunk & 2.x, so if it's incorporated into 1.x, will it cause incompatibilities for hive? > Port HADOOP-9020 Add a SASL PLAIN server to branch 1 > > > Key: HADOOP-9083 > URL: https://issues.apache.org/jira/browse/HADOOP-9083 > Project: Hadoop Common > Issue Type: Task > Components: ipc, security >Affects Versions: 1.0.3 >Reporter: Yu Gao >Assignee: Yu Gao > Attachments: HADOOP-9020-branch-1.patch, test-patch.result, > test-TestSaslRPC.result > > > It would be good if the patch of HADOOP-9020 for adding SASL PLAIN server > implementation could be ported to branch 1 as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507321#comment-13507321
]
Steve Loughran commented on HADOOP-9107:
This is similar to HADOOP-6221, though you are proposing more cleanup.
Could you use that patch and test as a starting point?
> Hadoop IPC client eats InterruptedException and sets interrupt on the thread
> which is not documented
>
>
> Key: HADOOP-9107
> URL: https://issues.apache.org/jira/browse/HADOOP-9107
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 1.1.0, 2.0.2-alpha
>Reporter: Hari Shreedharan
>
> This code in Client.java looks fishy:
> {code}
> public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
> ConnectionId remoteId) throws InterruptedException, IOException {
> Call call = new Call(rpcKind, rpcRequest);
> Connection connection = getConnection(remoteId, call);
> connection.sendParam(call); // send the parameter
> boolean interrupted = false;
> synchronized (call) {
> while (!call.done) {
> try {
> call.wait(); // wait for the result
> } catch (InterruptedException ie) {
> // save the fact that we were interrupted
> interrupted = true;
> }
> }
> if (interrupted) {
> // set the interrupt flag now that we are done waiting
> Thread.currentThread().interrupt();
> }
> if (call.error != null) {
> if (call.error instanceof RemoteException) {
> call.error.fillInStackTrace();
> throw call.error;
> } else { // local exception
> InetSocketAddress address = connection.getRemoteAddress();
> throw NetUtils.wrapException(address.getHostName(),
> address.getPort(),
> NetUtils.getHostname(),
> 0,
> call.error);
> }
> } else {
> return call.getRpcResult();
> }
> }
> }
> {code}
> Blocking calls are expected to throw InterruptedException if that is
> interrupted. Also it seems like this method waits on the call objects even if
> it is interrupted. Currently, this method does not throw an
> InterruptedException, nor is it documented that this method interrupts the
> thread calling it. If it is interrupted, this method should still throw
> InterruptedException, it should not matter if the call was successful or not.
> This is a major issue for clients which do not call this directly, but call
> HDFS client API methods to write to HDFS, which may be interrupted by the
> client due to timeouts, but does not throw InterruptedException. Any HDFS
> client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9108) Add a method to clear terminateCalled to ExitUtil for test cases
[ https://issues.apache.org/jira/browse/HADOOP-9108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507296#comment-13507296 ] Hudson commented on HADOOP-9108: Integrated in Hadoop-Hdfs-0.23-Build #451 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/451/]) HADOOP-9108. Add a method to clear terminateCalled to ExitUtil for test cases (Kihwal Lee via tgraves) (Revision 1415489) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1415489 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java > Add a method to clear terminateCalled to ExitUtil for test cases > > > Key: HADOOP-9108 > URL: https://issues.apache.org/jira/browse/HADOOP-9108 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.23.5 >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: 0.23.6 > > Attachments: hadoop-9108.branch-0.23.patch > > > Currently once terminateCalled is set, it will stay set since it's a class > static variable. This can break test cases where multiple test cases run in > one jvm. In MiniDfsCluster, it should be cleared during shutdown for the next > test case to run properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9090) Support on-demand publish of metrics
[ https://issues.apache.org/jira/browse/HADOOP-9090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated HADOOP-9090: Summary: Support on-demand publish of metrics (was: Refactor MetricsSystemImpl to allow for an on-demand publish system) Mostafa, I'm still a little bothered by the fact that a wrapper object is created for per publish per sink in normal cases. Since MetricsBuffer is trivially decoratable, how about creating a WaitalbeMetricsBuffer for OOB case only? > Support on-demand publish of metrics > > > Key: HADOOP-9090 > URL: https://issues.apache.org/jira/browse/HADOOP-9090 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Reporter: Mostafa Elhemali >Priority: Minor > Attachments: HADOOP-9090.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.2.patch, > HADOOP-9090.justEnhanceDefaultImpl.3.patch, > HADOOP-9090.justEnhanceDefaultImpl.4.patch, > HADOOP-9090.justEnhanceDefaultImpl.5.patch, > HADOOP-9090.justEnhanceDefaultImpl.6.patch, > HADOOP-9090.justEnhanceDefaultImpl.patch, HADOOP-9090.patch > > > Updated description based on feedback: > We have a need to publish metrics out of some short-living processes, which > is not really well-suited to the current metrics system implementation which > periodically publishes metrics asynchronously (a behavior that works great > for long-living processes). Of course I could write my own metrics system, > but it seems like such a waste to rewrite all the awesome code currently in > the MetricsSystemImpl and supporting classes. > The way this JIRA solves this problem is adding a new method > publishMetricsNow() to the MetricsSystemImpl() class, that does a synchronous > out-of-band push of the metrics from the sources to the sink. I also add a > method to MetricsSinkAdapter (putMetricsImmediate) to support that change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
