[jira] [Resolved] (HADOOP-12980) Document RPC scheduler/callqueue configuration keys
[ https://issues.apache.org/jira/browse/HADOOP-12980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin resolved HADOOP-12980. Resolution: Duplicate > Document RPC scheduler/callqueue configuration keys > --- > > Key: HADOOP-12980 > URL: https://issues.apache.org/jira/browse/HADOOP-12980 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation >Reporter: Xiaoyu Yao >Assignee: Ryan Wu >Priority: Major > > This ticket is opened to document RPC scheduler, callqueue and handler > related keys. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15805) Hadoop logo not showed correctly in old site
Yiqun Lin created HADOOP-15805:
--
Summary: Hadoop logo not showed correctly in old site
Key: HADOOP-15805
URL: https://issues.apache.org/jira/browse/HADOOP-15805
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.1.1
Reporter: Yiqun Lin
Hadoop logo not showed correctly in old site. In old site pages, we use the
address {{[http://hadoop.apache.org/images/hadoop-logo.jpg]}} to show the
hadoop logo. Actually, this addressed is outdated. The right address now is
[http://hadoop.apache.org/hadoop-logo.jpg].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15768) Do not use Time#now to calculate the rpc queue time duration
Yiqun Lin created HADOOP-15768:
--
Summary: Do not use Time#now to calculate the rpc queue time
duration
Key: HADOOP-15768
URL: https://issues.apache.org/jira/browse/HADOOP-15768
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Yiqun Lin
Assignee: Ryan Wu
For the rpc queue time calculation, we are using a not-recommended way: Using
{{Time#now}} to calculate for this.
{code}
// Invoke the protocol method
long startTime = Time.now();
int qTime = (int) (startTime-receivedTime);
server.updateMetrics(detailedMetricsName, qTime, processingTime, false);
{code}
We should use {{Time#monotonicNow()}} instead. This JIRA will fix these across
RpcEngine impl classes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15742) Log if the ipc backoff is enabled in CallQueueManager
Yiqun Lin created HADOOP-15742:
--
Summary: Log if the ipc backoff is enabled in CallQueueManager
Key: HADOOP-15742
URL: https://issues.apache.org/jira/browse/HADOOP-15742
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.1.1
Reporter: Yiqun Lin
Assignee: Ryan Wu
Currently we don't log the info of ipc backoff. It will look good to print this
as well so that makes users know if we enable this.
{code:java}
public CallQueueManager(Class> backingClass,
Class schedulerClass,
boolean clientBackOffEnabled, int maxQueueSize, String namespace,
Configuration conf) {
int priorityLevels = parseNumLevels(namespace, conf);
this.scheduler = createScheduler(schedulerClass, priorityLevels,
namespace, conf);
BlockingQueue bq = createCallQueueInstance(backingClass,
priorityLevels, maxQueueSize, namespace, conf);
this.clientBackOffEnabled = clientBackOffEnabled;
this.putRef = new AtomicReference>(bq);
this.takeRef = new AtomicReference>(bq);
LOG.info("Using callQueue: " + backingClass + " queueCapacity: " +
maxQueueSize + " scheduler: " + schedulerClass);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15391) Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules
Yiqun Lin created HADOOP-15391: -- Summary: Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules Key: HADOOP-15391 URL: https://issues.apache.org/jira/browse/HADOOP-15391 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.1 Reporter: Yiqun Lin Assignee: Yiqun Lin The documentation pages for hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake render error (see screen-shot attached or [here|http://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html]). The reason of this is that the css file is missing in these modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14839) DistCp log output should contain copied and deleted files and directories
[
https://issues.apache.org/jira/browse/HADOOP-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin reopened HADOOP-14839:
> DistCp log output should contain copied and deleted files and directories
> -
>
> Key: HADOOP-14839
> URL: https://issues.apache.org/jira/browse/HADOOP-14839
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.7.1
>Reporter: Konstantin Shaposhnikov
>Assignee: Yiqun Lin
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14839.006.patch, HADOOP-14839-branch-2.001.patch,
> HDFS-10234.001.patch, HDFS-10234.002.patch, HDFS-10234.003.patch,
> HDFS-10234.004.patch, HDFS-10234.005.patch
>
>
> DistCp log output (specified via {{-log}} command line option) currently
> contains only skipped and failed (when failures are ignored via {{-i}}) files.
> It will be more useful if it also contains copied and deleted files and
> created directories.
> This should be fixed in
> https://github.com/apache/hadoop/blob/branch-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14568) GenericTestUtils#waitFor missing parameter verification
Yiqun Lin created HADOOP-14568: -- Summary: GenericTestUtils#waitFor missing parameter verification Key: HADOOP-14568 URL: https://issues.apache.org/jira/browse/HADOOP-14568 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0-alpha3 Reporter: Yiqun Lin Assignee: Yiqun Lin GenericTestUtils#waitFor missed the parameters verification. This will lead the method ran error or meanless. This point was suggested from this [comment|https://issues.apache.org/jira/browse/HDFS-11711?focusedCommentId=16039292&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16039292]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13965) Groups should be consistent in using group mapping class
Yiqun Lin created HADOOP-13965:
--
Summary: Groups should be consistent in using group mapping class
Key: HADOOP-13965
URL: https://issues.apache.org/jira/browse/HADOOP-13965
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
{code:title=Groups.java}
public Groups(Configuration conf, final Timer timer) {
impl =
ReflectionUtils.newInstance(
conf.getClass(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING,
ShellBasedUnixGroupsMapping.class,
GroupMappingServiceProvider.class),
conf);
...
}
{code}
The default value of setting {{hadoop.security.group.mapping}} is different in
code and config file. In the core-default.xml, it uses the class
{{JniBasedUnixGroupsMappingWithFallback}} and this should be a true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13940) Document the missing envvars commands
Yiqun Lin created HADOOP-13940:
--
Summary: Document the missing envvars commands
Key: HADOOP-13940
URL: https://issues.apache.org/jira/browse/HADOOP-13940
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
In HADOOP-12366, it introduced the new command to display computed Hadoop
environment variables, for example JAVA_HOME. But the new commands haven't been
documented except {{hdfs envvars}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13883) Add description of -fs option in generic command usage
[
https://issues.apache.org/jira/browse/HADOOP-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin reopened HADOOP-13883:
> Add description of -fs option in generic command usage
> --
>
> Key: HADOOP-13883
> URL: https://issues.apache.org/jira/browse/HADOOP-13883
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13883.001.patch, HADOOP-13883.addendum001.patch
>
>
> Currently the description of option '-fs' is missing in generic command
> usage in documentation {{CommandManual.md}}. And the users won't know to use
> this option, while this option already makes sense to {{hdfs dfsadmin}},
> {{hdfs fsck}}, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13883) Add description of -fs option in generic command usage
Yiqun Lin created HADOOP-13883:
--
Summary: Add description of -fs option in generic command usage
Key: HADOOP-13883
URL: https://issues.apache.org/jira/browse/HADOOP-13883
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
Currently the description of option '-fs' is missing in generic command usage
in documentation {{CommandManual.md}}. And the users won't know to use this
option, while this option already makes sense to {{hdfs dfsadmin}}, {{hdfs
fsck}}, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13880) Fix dead links in relevant APIs of Job setting
Yiqun Lin created HADOOP-13880: -- Summary: Fix dead links in relevant APIs of Job setting Key: HADOOP-13880 URL: https://issues.apache.org/jira/browse/HADOOP-13880 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 3.0.0-alpha2 Reporter: Yiqun Lin Assignee: Yiqun Lin Priority: Minor There are some dead links in Job relevant classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13690) Fix typos in core-default.xml
Yiqun Lin created HADOOP-13690:
--
Summary: Fix typos in core-default.xml
Key: HADOOP-13690
URL: https://issues.apache.org/jira/browse/HADOOP-13690
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor
There are three typos in file {{core-default.xml}}:
* In {{hadoop.user.group.static.mapping.overrides}}: in otherwords-> in other
words
* In {{hadoop.security.group.mapping.ldap.search.group.hierarchy.levels}}:
exectue->execute
* In {{hadoop.security.dns.interface}}: subsitution->substitution
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13538) Deprecate getInstance and initiate methods with Path in TrashPolicy
Yiqun Lin created HADOOP-13538: -- Summary: Deprecate getInstance and initiate methods with Path in TrashPolicy Key: HADOOP-13538 URL: https://issues.apache.org/jira/browse/HADOOP-13538 Project: Hadoop Common Issue Type: Improvement Reporter: Yiqun Lin Priority: Minor As HADOOP-13534 memtioned, the getInstance and initiate APIs with Path is not used anymore. We should deprecate these methods before removing them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
Yiqun Lin created HADOOP-13210:
--
Summary: MetricsSinkAdapter always reset retryDelay and retryCount
Key: HADOOP-13210
URL: https://issues.apache.org/jira/browse/HADOOP-13210
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Yiqun Lin
In {{MetricsSinkAdapter#publishMetricsFromQueue}}, it always reset the
{{retryDelay}} and {{retryCount}} in each loop.
{code}
void publishMetricsFromQueue() {
int retryDelay = firstRetryDelay;
int n = retryCount;
int minDelay = Math.min(500, retryDelay * 1000); // millis
Random rng = new Random(System.nanoTime());
while (!stopping) {
try {
queue.consumeAll(this);
refreshQueueSizeGauge();
retryDelay = firstRetryDelay;
n = retryCount;
inError = false;
} catch (InterruptedException e) {
LOG.info(name +" thread interrupted.");
} catch (Exception e) {
if (n > 0) {
int retryWindow = Math.max(0, 1000 / 2 * retryDelay - minDelay);
int awhile = rng.nextInt(retryWindow) + minDelay;
if (!inError) {
LOG.error("Got sink exception, retry in "+ awhile +"ms", e);
}
retryDelay *= retryBackoff;
try { Thread.sleep(awhile); }
catch (InterruptedException e2) {
LOG.info(name +" thread interrupted while waiting for retry", e2);
}
--n;
} else {
if (!inError) {
LOG.error("Got sink exception and over retry limit, "+
"suppressing further error messages", e);
}
queue.clear();
refreshQueueSizeGauge();
inError = true; // Don't keep complaining ad infinitum
}
}
}
}
{code}
It looks the change of these in catch block code will not comes into effect. I
think we should remove these two line because the values have been initalized
before in {{publishMetricsFromQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
