[ https://issues.apache.org/jira/browse/YARN-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amit Tiwari updated YARN-2556: ------------------------------ Attachment: YARN-2556.patch Hi guys, I've done the following enhancements to the previous patches that were posted: 1) Earlier, the payload was getting set as the entityId. Since the entityId is used as a key, by LevelDB it was crashing under moderate loads, because each key size was ~2MB. Hence I've changed it to send the payload as a part of OtherInfo. This is handled well. 2) Instead of posting a string of repeated 'a's as a payload, I choose from a set of characters. This ensures that the LevelDB does not get away easily with compression ('cos algos can easily compress a string if it comprises a single repeated character) Here are some of the performance numbers that I've got: I run 20 concurrent jobs, with the argument -m 300 -s 10 -t 20 On a 36 node cluster, this results in ~830 concurrent containers (e.g maps), each firing 10KB of payload, 20 times. Level DB seems to hold up fine. Would you have other ways that I could stress/load the system even more? thanks --amit > Tool to measure the performance of the timeline server > ------------------------------------------------------ > > Key: YARN-2556 > URL: https://issues.apache.org/jira/browse/YARN-2556 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver > Reporter: Jonathan Eagles > Assignee: Chang Li > Attachments: YARN-2556-WIP.patch, YARN-2556-WIP.patch, > YARN-2556.patch, yarn2556.patch, yarn2556.patch, yarn2556_wip.patch > > > We need to be able to understand the capacity model for the timeline server > to give users the tools they need to deploy a timeline server with the > correct capacity. > I propose we create a mapreduce job that can measure timeline server write > and read performance. Transactions per second, I/O for both read and write > would be a good start. > This could be done as an example or test job that could be tied into gridmix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)