P.S., you might want to file this on jira and assign it to me or yourself. if you need some help, I can help you. ;)
On Thu, Mar 28, 2013 at 10:27 AM, Edward J. Yoon <[email protected]> wrote: > The cost of a superstep described in Wikipedia[1] is as below: > > The cost of a superstep S = max(Wi) + max(hg) + l = max cost of the > local computation + global communication + cost of the barrier > synchronization. As you already might know, max(Wi) means just the > wait time for the slowest process. > >> III - In relation to the cost of computation, is there any way to implement >> on >> Apache Hama? > > Sure, you can calculate the cost of each superstep using Counters, and > can sum finally. With this, you can also add something like "estimated > time remaining". Please see our implementation of sync() method[2] > (especially PeerCounter.TIME_IN_SYNC_MS). > > 1. http://en.wikipedia.org/wiki/Bulk_synchronous_parallel > 2. > http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java > > On Wed, Mar 27, 2013 at 11:26 PM, Júlio Pires <[email protected]> wrote: >> Hi Suraj, >> >> Thanks for the response. >> I've been by analyzing and I got the following doubts: >> >> I - How can I know when a super step ended? >> II - With the information of finish of a superstep, it is possible to >> capture the costs discussed in the previous email (computing, communication >> and synchronization)? >> III - In relation to the cost of computation, is there any way to implement >> on >> Apache Hama? >> >> Thanks! >> Júlio >> >> >> 2013/3/16 Suraj Menon <[email protected]> >> >>> Hi Julio, >>> >>> - Cost of the longest running local computation; >>> We don't have this yet per superstep. >>> >>> - Cost of global communication between the processors; >>> We provide information on count/size in bytes of messages sent and received >>> by peers.( IO_BYTES_READ, MESSAGE_BYTES_TRANSFERED, MESSAGE_BYTES_RECEIVED, >>> TOTAL_MESSAGES_SENT, TOTAL_MESSAGES_RECEIVED, COMPRESSED_BYTES_SENT, >>> COMPRESSED_BYTES_RECEIVED) >>> - Cost of the barrier synchronisation at the end of the superstep. >>> This is again provided in the final output in TIME_IN_SYNC_MS. >>> >>> The stats above are aggregated throughout the lifecycle and we don't have a >>> breakdown per superstep yet. You may create a JIRA for tracking. >>> >>> Thanks, >>> Suraj >>> >>> >>> On Fri, Mar 15, 2013 at 8:10 PM, Edward J. Yoon <[email protected] >>> >wrote: >>> >>> > Hi, of course. >>> > >>> > On Fri, Mar 15, 2013 at 9:39 PM, Júlio Pires <[email protected]> >>> > wrote: >>> > > Hi, >>> > > >>> > > Thanks for the response. >>> > > >>> > > So, thinking in a more abstract level, there are means to achieve these >>> > > three cost information on Apache Hama (another type of information) >>> (see >>> > > >>> > >>> http://en.wikipedia.org/wiki/Bulk_synchronous_parallelThe_Cost_of_a_BSP_algorithm >>> > )? >>> > > >>> > > - Cost of the longest running local computation; >>> > > - Cost of global communication between the processors; >>> > > - Cost of the barrier synchronisation at the end of the superstep. The >>> > cost >>> > > of one superstep for p processors >>> > > >>> > > Thanks! >>> > > >>> > > >>> > > 2013/3/14 Edward J. Yoon <[email protected]> >>> > > >>> > >> Unfortunately, we currently don't have any monitoring tool for metrics >>> > >> such as CPU, memory. >>> > >> >>> > >> For the cluster and BSP job monitoring, there's a web UI but not >>> mature >>> > >> yet. >>> > >> >>> > >> On Fri, Mar 15, 2013 at 2:17 AM, Júlio Pires <[email protected]> >>> > >> wrote: >>> > >> > Hi, >>> > >> > >>> > >> > It's possible to retrieve information of performance such as CPU >>> > usage or >>> > >> > memory in each task on each peer? If so, how do? >>> > >> > >>> > >> > Also, details of the functioning of job monitoring in Apache Hama. >>> > >> > >>> > >> > Thanks! >>> > >> > Júlio >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> Best Regards, Edward J. Yoon >>> > >> @eddieyoon >>> > >> >>> > >>> > >>> > >>> > -- >>> > Best Regards, Edward J. Yoon >>> > @eddieyoon >>> > >>> > > > > -- > Best Regards, Edward J. Yoon > @eddieyoon -- Best Regards, Edward J. Yoon @eddieyoon
