dlogothetis commented on a change in pull request #110: Counter Mechanism
URL: https://github.com/apache/giraph/pull/110#discussion_r342750159
##########
File path:
giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
##########
@@ -1236,6 +1235,76 @@ public void cleanup(FinishedSuperstepStats
finishedSuperstepStats)
LOG.error("cleanup: Got InterruptedException on notification " +
"to master about cleanup", e);
}
+ }
+
+ /**
+ * Method to send the counter values to the master using the zookeeper
+ * This is called after finishing all the supersteps
+ * @param allSuperstepsDone Whether the job has finished all supersteps
+ * This is needed to ensure the superstep number is the same for master and
+ * worker, when all supersteps are finished
+ *
+ */
+ public void storeCountersInZooKeeper(boolean allSuperstepsDone) {
+ Set<CustomCounter> additionalCounters =
+ CustomCounters.getCustomCounters();
+
+ JSONArray jsonCounters = new JSONArray();
+ Mapper.Context context = getContext();
+ Counter counter;
+ for (CustomCounter customCounter : additionalCounters) {
+ String groupName = customCounter.getGroupName();
+ String counterName = customCounter.getCounterName();
+ counter = context.getCounter(groupName, counterName);
+ customCounter.setValue(counter.getValue());
+ jsonCounters.put(Base64.encodeBytes(
+ WritableUtils.writeToByteArray(customCounter)));
+ }
+ // Add the Netty counters
+ Map<String, Set<String>> nettyCounters =
+ NettyClient.getCounterGroupsAndNames();
+ for (Map.Entry<String, Set<String>> entry : nettyCounters.entrySet()) {
+ String groupName = entry.getKey();
+ for (String counterName: entry.getValue()) {
+ CustomCounter customCounter = new CustomCounter(groupName, counterName,
+ CustomCounter.AGGREGATION.SUM);
+ counter = context.getCounter(groupName, counterName);
+ customCounter.setValue(counter.getValue());
+ jsonCounters.put(Base64.encodeBytes(
+ WritableUtils.writeToByteArray(customCounter)));
+ }
+ }
+ long superStep = getSuperstep() + (allSuperstepsDone ? 1 : 0);
+ String finishedWorkerPath =
+ getWorkerFinishedPath(
+ getApplicationAttempt(), superStep, true) +
+ "/" + workerInfo.getHostnameId();
+ LOG.info("worker finished path: " + finishedWorkerPath);
Review comment:
Related to the commend about having a separate method for getting the path
for the counters, let's change the log line too, to reflect we're writing
counters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
