[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16816818#comment-16816818
]
Yiqun Lin edited comment on HDFS-14403 at 4/13/19 3:35 AM:
-----------------------------------------------------------
[~daryn], your patch looks very good. It simplified the codes a lot and support
more type times. You put lots of logic in IPC layer not in HDFS module, This
can be reused by other module as well.
Looking through two approaches, we can reuse cost provider definition to return
cost value from first approach. That can make this more pluggable to use rather
than directly calculating cost value in scheduler class. For the actual cost
calculation, process details based approach seems a more better way.
Based my proposal, sample classes:
{noformat}
public interface CostProvider {
/**
* Get cost from process details which will be used in scheduler.
* @param details Process details.
* @return The cost of the latest call invoked by user.
*/
long getCost(ProcessingDetails details);
}
/**
* For the default cost provieder, we use the simple count based way
* and process details is no used here.
* @author yiqlin
*
*/
public class DefaultCostProvider implements CostProvider {
@Override
public long getCost(ProcessingDetails details) {
return 1;
}
}
public class LockHoldCostProvider implements CostProvider {
private final long[] processingCostWeights;
public LockHoldCostProvider(Configuration conf) {
// initialize cost weights
}
@Override
public long getCost(ProcessingDetails details) {
long processingCost = details.applyWeights(processingCostWeights);
details.set(Timing.COST, processingCost);
return processingCost;
}
}
{noformat}
The cost provider will be used in
{noformat}
public void addResponseTime(String name, Schedulable obj,
ProcessingDetails details) {
if (!processingCostEnabled) {
return;
}
try {
String user = identityProvider.makeIdentity(obj);
getAndIncrementCost(user, costProvider.getCost(details));
} catch (InterruptedException e) {
// method never actually throws.
}
}
{noformat}
Since process detail contains many type elapsed times value, we can extend the
new cost provider based on other algorithm if we need in future. Any thoughts
from others?
was (Author: linyiqun):
[~daryn], your patch looks very good. It simplified the codes a lot and support
more type times. You put lots of logic in IPC layer not in HDFS module, This
can be reused by other module as well.
Looking through two approaches, we can reuse cost provider definition to return
cost value. That can make this more pluggable to use rather than directly
calculating cost value in class. For the actual cost calculation, process
details based approach seems a more better way.
Based my proposal, sample classes:
{noformat}
public interface CostProvider {
/**
* Get cost from process details which will be used in scheduler.
* @param details Process details.
* @return The cost of the latest call invoked by user.
*/
long getCost(ProcessingDetails details);
}
/**
* For the default cost provieder, we use the simple count based way
* and process details is no used here.
* @author yiqlin
*
*/
public class DefaultCostProvider implements CostProvider {
@Override
public long getCost(ProcessingDetails details) {
return 1;
}
}
public class LockHoldCostProvider implements CostProvider {
private final long[] processingCostWeights;
public LockHoldCostProvider(Configuration conf) {
// initialize cost weights
}
@Override
public long getCost(ProcessingDetails details) {
long processingCost = details.applyWeights(processingCostWeights);
details.set(Timing.COST, processingCost);
return processingCost;
}
}
{noformat}
The cost provider will be used in
{noformat}
public void addResponseTime(String name, Schedulable obj,
ProcessingDetails details) {
if (!processingCostEnabled) {
return;
}
try {
String user = identityProvider.makeIdentity(obj);
getAndIncrementCost(user, costProvider.getCost(details));
} catch (InterruptedException e) {
// method never actually throws.
}
}
{noformat}
Since process detail contains many type elapsed times value, we can extend the
new cost provider based on other algorithm if we need in future. Any thoughts
from others?
> Cost-Based RPC FairCallQueue
> ----------------------------
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
> Reporter: Erik Krogen
> Assignee: Christopher Gregorian
> Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf,
> HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch,
> HDFS-14403.branch-2.8.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
