[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15926858#comment-15926858 ] Hitesh Shah commented on HADOOP-10101: -- IIRC, YarnClient has a guava object being used in one of its public APIs. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha2 >Reporter: Rakesh R >Assignee: Tsuyoshi Ozawa > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.012.patch, HADOOP-10101.013.patch, HADOOP-10101.patch, > HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13037) Refactor Azure Data Lake Store as an independent FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893529#comment-15893529 ] Hitesh Shah commented on HADOOP-13037: -- [~chris.douglas] [~cnauroth] [~vishwajeet.dusane] [~steve_l] Given that HADOOP-13687 and HADOOP-13257 are resolved, can this be backported to branch-2? > Refactor Azure Data Lake Store as an independent FileSystem > --- > > Key: HADOOP-13037 > URL: https://issues.apache.org/jira/browse/HADOOP-13037 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Shrikant Naidu >Assignee: Vishwajeet Dusane > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13037-001.patch, HADOOP-13037-002.patch, > HADOOP-13037-003.patch, HADOOP-13037-004.patch, HADOOP-13037.005.patch, > HADOOP-13037.006.patch, HADOOP-13037 Proposal.pdf > > > The jira proposes an improvement over HADOOP-12666 to remove webhdfs > dependencies from the ADL file system client and build out a standalone > client. At a high level, this approach would extend the Hadoop file system > class to provide an implementation for accessing Azure Data Lake. The scheme > used for accessing the file system will continue to be > adl://.azuredatalake.net/path/to/file. > The Azure Data Lake Cloud Store will continue to provide a webHDFS rest > interface. The client will access the ADLS store using WebHDFS Rest APIs > provided by the ADLS store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546286#comment-15546286 ] Hitesh Shah commented on HADOOP-10101: -- I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use. Some questions: - why version 18 and not 19? Guava 19 was released in 2015 or so - have you confirmed that no deprecated and/or beta apis are being used? - I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment. > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0, 2.6.0 >Reporter: Rakesh R >Assignee: Vinayakumar B > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-10101: - Target Version/s: 3.0.0-alpha2 (was: ) > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0, 2.6.0 >Reporter: Rakesh R >Assignee: Vinayakumar B > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13065) Add a new interface for retrieving FS and FC Statistics
[ https://issues.apache.org/jira/browse/HADOOP-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-13065: - Assignee: Mingliang Liu (was: Aaron Fabbri) > Add a new interface for retrieving FS and FC Statistics > --- > > Key: HADOOP-13065 > URL: https://issues.apache.org/jira/browse/HADOOP-13065 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Ram Venkatesh >Assignee: Mingliang Liu > Fix For: 2.8.0 > > Attachments: HADOOP-13065-007.patch, HADOOP-13065.008.patch, > HADOOP-13065.009.patch, HADOOP-13065.010.patch, HADOOP-13065.011.patch, > HADOOP-13065.012.patch, HADOOP-13065.013.patch, HDFS-10175.000.patch, > HDFS-10175.001.patch, HDFS-10175.002.patch, HDFS-10175.003.patch, > HDFS-10175.004.patch, HDFS-10175.005.patch, HDFS-10175.006.patch, > TestStatisticsOverhead.java > > > Currently FileSystem.Statistics exposes the following statistics: > BytesRead > BytesWritten > ReadOps > LargeReadOps > WriteOps > These are in-turn exposed as job counters by MapReduce and other frameworks. > There is logic within DfsClient to map operations to these counters that can > be confusing, for instance, mkdirs counts as a writeOp. > Proposed enhancement: > Add a statistic for each DfsClient operation including create, append, > createSymlink, delete, exists, mkdirs, rename and expose them as new > properties on the Statistics object. The operation-specific counters can be > used for analyzing the load imposed by a particular job on HDFS. > For example, we can use them to identify jobs that end up creating a large > number of files. > Once this information is available in the Statistics object, the app > frameworks like MapReduce can expose them as additional counters to be > aggregated and recorded as part of job summary. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359782#comment-15359782
]
Hitesh Shah commented on HADOOP-13283:
--
Any reason why FileSystem.clearStatistics() is not being changed to call a
reset on GlobalStorageStats?
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13288) Guard null stats key in FileSystemStorageStatistics
[
https://issues.apache.org/jira/browse/HADOOP-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337514#comment-15337514
]
Hitesh Shah commented on HADOOP-13288:
--
{code}
if (key == null) {
return null;
}
{code}
Shouldnt this be a precondition assert where key passed in should never be
null? i.e. if a bad app asks for a value for a null key, throw an error.
> Guard null stats key in FileSystemStorageStatistics
> ---
>
> Key: HADOOP-13288
> URL: https://issues.apache.org/jira/browse/HADOOP-13288
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13288.000.patch
>
>
> Currently in {{FileSystemStorageStatistics}} we simply returns data from
> {{FileSystem#Statistics}}. However there is no null key check, which leads to
> NPE problems to downstream applications. For example, we got a NPE when
> passing a null key to {{FileSystemStorageStatistics#getLong()}}, exception
> stack as following:
> {quote}
> NullPointerException
> at
> org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80)
> at
> org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108)
> at
> org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60)
> at
> org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118)
> at
> org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {quote}
> This jira is to add null stat key check to {{FileSystemStorageStatistics}}.
> Thanks [~hitesh] for trying in Tez and reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13123) Permit the default hadoop delegation token file format to be configurable
[ https://issues.apache.org/jira/browse/HADOOP-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279085#comment-15279085 ] Hitesh Shah commented on HADOOP-13123: -- +1 on reverting the change if it did go into branch-2. > Permit the default hadoop delegation token file format to be configurable > - > > Key: HADOOP-13123 > URL: https://issues.apache.org/jira/browse/HADOOP-13123 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Matthew Paduano >Assignee: Matthew Paduano > Attachments: HADOOP-13123.01.patch > > > If one environment updates to using the new dtutil code and accompanying > Credentials code, there is a backward compatibility issue with the default > file format being JAVA. Older clients need to be updated to ask for a file > in the legacy format (FORMAT_JAVA). > As an aid to users in this trap, we can add a configuration property to set > the default file format. When set to FORMAT_JAVA, the new server code will > serve up legacy files without being asked. The default value for this > property will remain FORMAT_PB. But affected users can add this config > option to the services using the newer code. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
[ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278775#comment-15278775 ] Hitesh Shah commented on HADOOP-12563: -- [~raviprak] Mind updating the fix versions to denote which version of 2.x this commit went into. > Updated utility to create/modify token files > > > Key: HADOOP-12563 > URL: https://issues.apache.org/jira/browse/HADOOP-12563 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Matthew Paduano > Fix For: 3.0.0 > > Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, > HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch, > HADOOP-12563.06.patch, HADOOP-12563.07.patch, HADOOP-12563.07.patch, > HADOOP-12563.08.patch, HADOOP-12563.09.patch, HADOOP-12563.10.patch, > HADOOP-12563.11.patch, HADOOP-12563.12.patch, HADOOP-12563.13.patch, > HADOOP-12563.14.patch, HADOOP-12563.15.patch, HADOOP-12563.16.patch, > dtutil-test-out, example_dtutil_commands_and_output.txt, > generalized_token_case.pdf > > > hdfs fetchdt is missing some critical features and is geared almost > exclusively towards HDFS operations. Additionally, the token files that are > created use Java serializations which are hard/impossible to deal with in > other languages. It should be replaced with a better utility in common that > can read/write protobuf-based token files, has enough flexibility to be used > with other services, and offers key functionality such as append and rename. > The old version file format should still be supported for backward > compatibility, but will be effectively deprecated. > A follow-on JIRA will deprecrate fetchdt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
[
https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277434#comment-15277434
]
Hitesh Shah commented on HADOOP-12563:
--
[~aw] Did you take a look at the stack trace in question?
{code}
Caused by: java.io.IOException: Unknown version 1 in token storage.
at
org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:*215*)
at
org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:*194*)
{code}
Could you clarify what you mean by "Tez should not be reading the file outside
of the provided hadoop classes" ?
> Updated utility to create/modify token files
>
>
> Key: HADOOP-12563
> URL: https://issues.apache.org/jira/browse/HADOOP-12563
> Project: Hadoop Common
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: Matthew Paduano
> Fix For: 3.0.0
>
> Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch,
> HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch,
> HADOOP-12563.06.patch, HADOOP-12563.07.patch, HADOOP-12563.07.patch,
> HADOOP-12563.08.patch, HADOOP-12563.09.patch, HADOOP-12563.10.patch,
> HADOOP-12563.11.patch, HADOOP-12563.12.patch, HADOOP-12563.13.patch,
> HADOOP-12563.14.patch, HADOOP-12563.15.patch, HADOOP-12563.16.patch,
> dtutil-test-out, example_dtutil_commands_and_output.txt,
> generalized_token_case.pdf
>
>
> hdfs fetchdt is missing some critical features and is geared almost
> exclusively towards HDFS operations. Additionally, the token files that are
> created use Java serializations which are hard/impossible to deal with in
> other languages. It should be replaced with a better utility in common that
> can read/write protobuf-based token files, has enough flexibility to be used
> with other services, and offers key functionality such as append and rename.
> The old version file format should still be supported for backward
> compatibility, but will be effectively deprecated.
> A follow-on JIRA will deprecrate fetchdt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12969) Mark IPC.Client and IPC.Server as @Public, @Evolving
[ https://issues.apache.org/jira/browse/HADOOP-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215071#comment-15215071 ] Hitesh Shah commented on HADOOP-12969: -- Anyone using hadoop rpc ( HBase, Tez, etc. ) for client-server communication are already using this regardless of it being private. > Mark IPC.Client and IPC.Server as @Public, @Evolving > > > Key: HADOOP-12969 > URL: https://issues.apache.org/jira/browse/HADOOP-12969 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > > Per the discussion in > [HADOOP-12909|https://issues.apache.org/jira/browse/HADOOP-12909?focusedCommentId=15211745&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15211745], > this is to propose marking IPC.Client and IPC.Server as @Public, @Evolving > as a result of HADOOP-12909 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632199#comment-14632199 ] Hitesh Shah commented on HADOOP-10101: -- I hit something similar in tez recently and looking at this jira, I am wondering as to why shading ( relocating all classes ) the guava jar is not an option? Consider the following approach: - introduce a new hadoop-guava module which depends on guava version X. - hadoop-guava relocates all classes in guava to org.apache.hadoop.shaded.guava - all code in hadoop depends on hadoop-guava and excludes the google-guava dependency I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem ) > Update guava dependency to the latest version > - > > Key: HADOOP-10101 > URL: https://issues.apache.org/jira/browse/HADOOP-10101 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0, 2.6.0 >Reporter: Rakesh R >Assignee: Vinayakumar B > Labels: BB2015-05-TBR > Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, > HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, > HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101-009.patch, > HADOOP-10101-010.patch, HADOOP-10101-010.patch, HADOOP-10101-011.patch, > HADOOP-10101.patch, HADOOP-10101.patch > > > The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Resolution: Not a Problem Status: Resolved (was: Patch Available) > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch, HADOOP-9726.2.patch, > HADOOP-9726.3.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-8109) Using the default rpms of hadoop, /usr/etc/hadoop symlink gets removed on upgrade
[ https://issues.apache.org/jira/browse/HADOOP-8109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8109: Resolution: Not a Problem Status: Resolved (was: Patch Available) > Using the default rpms of hadoop, /usr/etc/hadoop symlink gets removed on > upgrade > -- > > Key: HADOOP-8109 > URL: https://issues.apache.org/jira/browse/HADOOP-8109 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.0.0, 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-8109.branch-1.patch, HADOOP-8109.trunk.patch > > > Given that for rpms, the pre-uninstall scripts for the older version run > after the post-install scripts of the package being installed, the symlink > created from /usr/etc/hadoop to /etc/hadoop gets deleted. This breaks running > /usr/bin/hadoop without providing --config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11254) Promoting AccessControlList to be public
[ https://issues.apache.org/jira/browse/HADOOP-11254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192225#comment-14192225 ] Hitesh Shah commented on HADOOP-11254: -- [~zjshen] Thanks for filing the issue. The context/use-case for this came about when looking at trying to integrate Tez with Timeline for ACLs. Given that Timeline today does not support incremental updates to ACLs, to update ACLs one needs to parse the original ACL string and update it with additional users/groups. The AccessControlList class being made public would be very useful for this. > Promoting AccessControlList to be public > > > Key: HADOOP-11254 > URL: https://issues.apache.org/jira/browse/HADOOP-11254 > Project: Hadoop Common > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > The motivation of promoting AccessControlList to be a public API is to > facilitate the users to programmatically parse or construct an ACL string. A > typical use case may be the timeline domain, where we have the client lib to > accept strings complied with ACL string format to define the authorized > readers/writers. It will be more convenient and less buggy if users can > compose this string with AccessControlList. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-10951) Make org.apache.hadoop.security.Groups public
[ https://issues.apache.org/jira/browse/HADOOP-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved HADOOP-10951. -- Resolution: Not a Problem > Make org.apache.hadoop.security.Groups public > - > > Key: HADOOP-10951 > URL: https://issues.apache.org/jira/browse/HADOOP-10951 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hitesh Shah > > This class seems like useful functionality for most developers building yarn > applications with respect to application acls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10951) Make org.apache.hadoop.security.Groups public
Hitesh Shah created HADOOP-10951: Summary: Make org.apache.hadoop.security.Groups public Key: HADOOP-10951 URL: https://issues.apache.org/jira/browse/HADOOP-10951 Project: Hadoop Common Issue Type: Improvement Reporter: Hitesh Shah This class seems like useful functionality for most developers building yarn applications with respect to application acls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved HADOOP-9741. - Resolution: Won't Fix > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch, INDEX.LIST > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks for the patience [~benoyantony] > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Nishan Shetty, Huawei >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.4.0 > > Attachments: HADOOP-8753.1.copy.patch, HADOOP-8753.1.patch, > YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Attachment: HADOOP-8753.1.copy.patch Re-attaching same patch as earlier to trigger a Jenkins pre-commit build. [~benoyantony] Sorry about this. This completely slipped below my radar. Triggering a jenkins check - once it goes through, will commit. > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.2-alpha >Reporter: Nishan Shetty, Huawei >Assignee: Benoy Antony >Priority: Minor > Attachments: HADOOP-8753.1.copy.patch, HADOOP-8753.1.patch, > YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HADOOP-10050) Update single node and cluster install instructions to work with latest bits
[
https://issues.apache.org/jira/browse/HADOOP-10050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797411#comment-13797411
]
Hitesh Shah commented on HADOOP-10050:
--
Comments:
The capacity-scheduler configs are removed from SingleCluster.apt.vm - any
reason for this?
Generated SingleCluster.html has "Set $$YARN_CONF_DIR" - mind fixing this too?
{code}
Run ResourceManager and NodeManager as:
$ cd $HADOOP_MAPRED_HOME
$ sbin/yarn-daemon.sh start resourcemanager
$ sbin/yarn-daemon.sh start nodemanager
$ sbin/mr-jobhistory-daemon.sh start historyserver
{code}
yarn-daemon.sh is not meant to be in HADOOP_MAPRED_HOME.
If this doc needs to be fixed, there are probably a lot of confusing references
to HADOOP_PREFIX vs HADOOP_YARN_HOME etc which probably need to be ironed out.
It may be good to either address them in this jira or file a new one for a
follow-up.
> Update single node and cluster install instructions to work with latest bits
>
>
> Key: HADOOP-10050
> URL: https://issues.apache.org/jira/browse/HADOOP-10050
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.2.0
>Reporter: Arpit Gupta
>Assignee: Arpit Gupta
>Priority: Minor
> Attachments: ClusterSetup.html, HADOOP-10050.patch, mapred-site.xml,
> SingleCluster.html, yarn-site.xml
>
>
> A few things i noticed
> 1. changes to yarn.nodemanager.aux-services
> 2. Set the framework to yarn in mapred-site.xml
> 3. Start the history server
> Also noticed no change to the capacity scheduler configs was needed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Attachment: HADOOP-9726.3.patch Removed unnecessary log message from previous patch. > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch, HADOOP-9726.2.patch, > HADOOP-9726.3.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Status: Patch Available (was: Open) > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch, HADOOP-9726.2.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Attachment: HADOOP-9726.2.patch > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch, HADOOP-9726.2.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9741: Status: Open (was: Patch Available) > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch, INDEX.LIST > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9741: Attachment: INDEX.LIST Attaching an index file for hadoop-common-3.0.0-SNAPSHOT.jar. It is relative to the top of the jar so re-location of the jar itself should not affect this. The performance improvement from what I have seen is quite small - around 10-20 ms when cycling through the jars in the hadoop classpath with a test program. Given your concerns with jar-of-jars and the other tomcat issue that I had not come across, I am becoming inclined to not push this in until I can prove more substantial gains on a cluster. > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch, INDEX.LIST > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Status: Open (was: Patch Available) > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711712#comment-13711712 ] Hitesh Shah commented on HADOOP-9726: - [~lmccay] attached a change which is less invasive based on the current usage. I would prefer to leave the re-factor to anyone else who is more familiar with the usage of SecurityUtil and willing to take a crack at it. > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Attachment: HADOOP-9726.1.patch > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah > Attachments: HADOOP-9726.1.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9726: Assignee: Hitesh Shah Status: Patch Available (was: Open) > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Attachments: HADOOP-9726.1.patch > > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711706#comment-13711706 ] Hitesh Shah commented on HADOOP-9741: - [~tucu00] Was mostly looking at it from a performance improvement point of view. I don't believe hadoop has any nested jars or fat-jars that could create the problem raised in the tomcat bug. Do you see a cause for concern with use of indexed jars? > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9741: Status: Patch Available (was: Open) > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9741) Make build created indexed jars
[ https://issues.apache.org/jira/browse/HADOOP-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9741: Attachment: HADOOP-9741.1.patch > Make build created indexed jars > --- > > Key: HADOOP-9741 > URL: https://issues.apache.org/jira/browse/HADOOP-9741 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Minor > Attachments: HADOOP-9741.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9741) Make build created indexed jars
Hitesh Shah created HADOOP-9741: --- Summary: Make build created indexed jars Key: HADOOP-9741 URL: https://issues.apache.org/jira/browse/HADOOP-9741 Project: Hadoop Common Issue Type: Bug Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
[ https://issues.apache.org/jira/browse/HADOOP-9726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707381#comment-13707381 ] Hitesh Shah commented on HADOOP-9726: - [~lmccay] My concern is mostly related to creation of a configuration object that can be avoided by passing in an already initialized one with say HADOOP_SECURITY_TOKEN_SERVICE_USE_IP set to the non-default value. > org.apache.hadoop.security.SecurityUtil has its own static Configuration > which cannot be overridden > --- > > Key: HADOOP-9726 > URL: https://issues.apache.org/jira/browse/HADOOP-9726 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah > > There is a static block which loads a new Configuration object and uses it to > initialize the SSLFactory and HostResolver. > Should this class have a similar static setConfiguration() function similar > to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9726) org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden
Hitesh Shah created HADOOP-9726: --- Summary: org.apache.hadoop.security.SecurityUtil has its own static Configuration which cannot be overridden Key: HADOOP-9726 URL: https://issues.apache.org/jira/browse/HADOOP-9726 Project: Hadoop Common Issue Type: Bug Reporter: Hitesh Shah There is a static block which loads a new Configuration object and uses it to initialize the SSLFactory and HostResolver. Should this class have a similar static setConfiguration() function similar to UGI? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9628) Setup a daily build job for branch-2.1.0-beta
Hitesh Shah created HADOOP-9628: --- Summary: Setup a daily build job for branch-2.1.0-beta Key: HADOOP-9628 URL: https://issues.apache.org/jira/browse/HADOOP-9628 Project: Hadoop Common Issue Type: Bug Reporter: Hitesh Shah -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9618) Add thread which detects JVM pauses
[ https://issues.apache.org/jira/browse/HADOOP-9618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13676236#comment-13676236 ] Hitesh Shah commented on HADOOP-9618: - [~tlipcon] does it make sense to expose this information through a metric too ( i.e. increment counters when warn/info levels are hit? > Add thread which detects JVM pauses > --- > > Key: HADOOP-9618 > URL: https://issues.apache.org/jira/browse/HADOOP-9618 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-9618.txt > > > Often times users struggle to understand what happened when a long JVM pause > (GC or otherwise) causes things to malfunction inside a Hadoop daemon. For > example, a long GC pause while logging an edit to the QJM may cause the edit > to timeout, or a long GC pause may make other IPCs to the NameNode timeout. > We should add a simple thread which loops on 1-second sleeps, and if the > sleep ever takes significantly longer than 1 second, log a WARN. This will > make GC pauses obvious in logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell
[ https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13608625#comment-13608625 ] Hitesh Shah commented on HADOOP-9422: - @Harsh, there is some static code in Shell.java that requires HADOOP_HOME to be set. As per @Bikas and @Chris, it seems it is only required for correct behavior on windows. > HADOOP_HOME should not be required to be set to be able to launch commands > using hadoop.util.Shell > -- > > Key: HADOOP-9422 > URL: https://issues.apache.org/jira/browse/HADOOP-9422 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Hitesh Shah > > Not sure why this is an enforced requirement especially in cases where a > deployment is done using multiple tar-balls ( one each for > common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell
[ https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9422: Affects Version/s: (was: 2.0.3-alpha) 3.0.0 > HADOOP_HOME should not be required to be set to be able to launch commands > using hadoop.util.Shell > -- > > Key: HADOOP-9422 > URL: https://issues.apache.org/jira/browse/HADOOP-9422 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Hitesh Shah > > Not sure why this is an enforced requirement especially in cases where a > deployment is done using multiple tar-balls ( one each for > common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9422) HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell
[ https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9422: Summary: HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell (was: HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to run commands) > HADOOP_HOME should not be required to be set to be able to launch commands > using hadoop.util.Shell > -- > > Key: HADOOP-9422 > URL: https://issues.apache.org/jira/browse/HADOOP-9422 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Hitesh Shah > > Not sure why this is an enforced requirement especially in cases where a > deployment is done using multiple tar-balls ( one each for > common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9422) HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to run commands
[ https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9422: Affects Version/s: 2.0.3-alpha > HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to > run commands > -- > > Key: HADOOP-9422 > URL: https://issues.apache.org/jira/browse/HADOOP-9422 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Hitesh Shah > > Not sure why this is an enforced requirement especially in cases where a > deployment is done using multiple tar-balls ( one each for > common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9422) HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to run commands
[ https://issues.apache.org/jira/browse/HADOOP-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9422: Summary: HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to run commands (was: HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to launch commands) > HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to > run commands > -- > > Key: HADOOP-9422 > URL: https://issues.apache.org/jira/browse/HADOOP-9422 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hitesh Shah > > Not sure why this is an enforced requirement especially in cases where a > deployment is done using multiple tar-balls ( one each for > common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9422) HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to launch commands
Hitesh Shah created HADOOP-9422: --- Summary: HADOOP_HOME needs to be set to be able to use org.apache.hadoop.util.Shell to launch commands Key: HADOOP-9422 URL: https://issues.apache.org/jira/browse/HADOOP-9422 Project: Hadoop Common Issue Type: Bug Reporter: Hitesh Shah Not sure why this is an enforced requirement especially in cases where a deployment is done using multiple tar-balls ( one each for common/hdfs/mapreduce/yarn ). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[ https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9276: Resolution: Fixed Fix Version/s: 2.0.3-alpha Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks Arun. > Allow BoundedByteArrayOutputStream to be resettable > --- > > Key: HADOOP-9276 > URL: https://issues.apache.org/jira/browse/HADOOP-9276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HADOOP-9276.3.patch, HADOOP-9276.patch, HADOOP-9276.patch > > > Allow BoundedByteArrayOutputStream's byte[] to be resettable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[ https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9276: Status: Patch Available (was: Open) > Allow BoundedByteArrayOutputStream to be resettable > --- > > Key: HADOOP-9276 > URL: https://issues.apache.org/jira/browse/HADOOP-9276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Attachments: HADOOP-9276.3.patch, HADOOP-9276.patch, HADOOP-9276.patch > > > Allow BoundedByteArrayOutputStream's byte[] to be resettable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[ https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9276: Attachment: HADOOP-9276.3.patch Attaching another patch addressing my minor comment. > Allow BoundedByteArrayOutputStream to be resettable > --- > > Key: HADOOP-9276 > URL: https://issues.apache.org/jira/browse/HADOOP-9276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Attachments: HADOOP-9276.3.patch, HADOOP-9276.patch, HADOOP-9276.patch > > > Allow BoundedByteArrayOutputStream's byte[] to be resettable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[ https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-9276: Status: Open (was: Patch Available) > Allow BoundedByteArrayOutputStream to be resettable > --- > > Key: HADOOP-9276 > URL: https://issues.apache.org/jira/browse/HADOOP-9276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Attachments: HADOOP-9276.patch, HADOOP-9276.patch > > > Allow BoundedByteArrayOutputStream's byte[] to be resettable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[
https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569189#comment-13569189
]
Hitesh Shah commented on HADOOP-9276:
-
Sorry - found one more issue.
{code}
public void reset() {
this.limit = buffer.length;
-this.count = 0;
+this.currentPointer = startOffset;
{code}
limit here should be set to buffer.length - startOffset to prevent problems
when startOffset is non-zero.
> Allow BoundedByteArrayOutputStream to be resettable
> ---
>
> Key: HADOOP-9276
> URL: https://issues.apache.org/jira/browse/HADOOP-9276
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
>Priority: Minor
> Attachments: HADOOP-9276.patch, HADOOP-9276.patch
>
>
> Allow BoundedByteArrayOutputStream's byte[] to be resettable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9276) Allow BoundedByteArrayOutputStream to be resettable
[ https://issues.apache.org/jira/browse/HADOOP-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569021#comment-13569021 ] Hitesh Shah commented on HADOOP-9276: - Minor comments: - BoundedByteArrayOutputStream(byte[] buf, int offset, int limit) could call the resetBuffer function itself. - the new ctor and resetBuffer functions need a check on buffer.length against limit ( similar to the original ctor's check for capacity ) > Allow BoundedByteArrayOutputStream to be resettable > --- > > Key: HADOOP-9276 > URL: https://issues.apache.org/jira/browse/HADOOP-9276 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Minor > Attachments: HADOOP-9276.patch > > > Allow BoundedByteArrayOutputStream's byte[] to be resettable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450859#comment-13450859 ] Hitesh Shah commented on HADOOP-8753: - @Benoy, I modified your patch slightly to remove the whitespaces and fix the use of tabs. Patch seems trivial and looks fine. However, could you look at adding a unit test for this? > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Assignee: Benoy Antony >Priority: Minor > Attachments: HADOOP-8753.1.patch, YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8774) Incorrect maven configuration causes mrapp-generated-classpath file to be generated in multiple places
Hitesh Shah created HADOOP-8774: --- Summary: Incorrect maven configuration causes mrapp-generated-classpath file to be generated in multiple places Key: HADOOP-8774 URL: https://issues.apache.org/jira/browse/HADOOP-8774 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.1.0-alpha Reporter: Hitesh Shah Priority: Minor Running a simple mvn clean install at the top of tree shows logs such as the ones below in the HDFS tree: [INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-hdfs-httpfs --- [INFO] Wrote classpath file '/Users/Hitesh/dev/hadoop-common/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/classes/mrapp-generated-classpath'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Status: Patch Available (was: Open) > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Assignee: Benoy Antony >Priority: Minor > Attachments: HADOOP-8753.1.patch, YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Attachment: HADOOP-8753.1.patch > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Priority: Minor > Attachments: HADOOP-8753.1.patch, YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned HADOOP-8753: --- Assignee: Benoy Antony > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Assignee: Benoy Antony >Priority: Minor > Attachments: HADOOP-8753.1.patch, YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Status: Open (was: Patch Available) Cancelling patch to re-upload a new one with whitespaces/tabs cleaned up. > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Priority: Minor > Attachments: HADOOP-8753.1.patch, YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8753) LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Summary: LocalDirAllocator throws "ArithmeticException: / by zero" when there is no available space on configured local dir (was: getLocalPathForWrite throws "ArithmeticException: / by zero" when there is no available space on configured local dir) > LocalDirAllocator throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > -- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Priority: Minor > Attachments: YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8753) getLocalPathForWrite throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated HADOOP-8753: Summary: getLocalPathForWrite throws "ArithmeticException: / by zero" when there is no available space on configured local dir (was: NM throws "ArithmeticException: / by zero" when there is no available space on configured local dir) > getLocalPathForWrite throws "ArithmeticException: / by zero" when there is no > available space on configured local dir > - > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Priority: Minor > Attachments: YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HADOOP-8753) NM throws "ArithmeticException: / by zero" when there is no available space on configured local dir
[ https://issues.apache.org/jira/browse/HADOOP-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah moved YARN-16 to HADOOP-8753: - Component/s: (was: nodemanager) Affects Version/s: (was: 2.2.0-alpha) 2.2.0-alpha Key: HADOOP-8753 (was: YARN-16) Project: Hadoop Common (was: Hadoop YARN) > NM throws "ArithmeticException: / by zero" when there is no available space > on configured local dir > --- > > Key: HADOOP-8753 > URL: https://issues.apache.org/jira/browse/HADOOP-8753 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: Nishan Shetty >Priority: Minor > Attachments: YARN-16.patch > > > 12/08/09 13:59:49 INFO mapreduce.Job: Task Id : > attempt_1344492468506_0023_m_00_0, Status : FAILED > java.lang.ArithmeticException: / by zero > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:371) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) > at > org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:257) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) > Instead of throwing exception directly we can log a warning saying no > available space on configured local dir -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8027) Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
[
https://issues.apache.org/jira/browse/HADOOP-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated HADOOP-8027:
Attachment: HADOOP-8027.branch-1.patch
Attaching a patch for branch-1.
> Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
> --
>
> Key: HADOOP-8027
> URL: https://issues.apache.org/jira/browse/HADOOP-8027
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Harsh J
>Assignee: Aaron T. Myers
>Priority: Minor
> Fix For: 0.23.1, 0.24.0
>
> Attachments: HADOOP-8027.branch-1.patch, HDFS-2584.patch
>
>
> Logs that follow a {{/jmx}} servlet visit:
> {code}
> 11/11/22 12:09:52 ERROR jmx.JMXJsonServlet: getting attribute UsageThreshold
> of java.lang:type=MemoryPool,name=Par Eden Space threw an exception
> javax.management.RuntimeMBeanException:
> java.lang.UnsupportedOperationException: Usage threshold is not supported
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
> ...
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
