[
https://issues.apache.org/jira/browse/HAMA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Jungblut updated HAMA-559:
---------------------------------
Attachment: SpillingBufferProfile-2012-10-27.snapshot
{noformat}
Call tree (all threads together, CPU usage estimation)
+------------------------------------------------------------------------------------------------------------+-----------------+-------------+
| Name
| Time (ms) | Samples |
+------------------------------------------------------------------------------------------------------------+-----------------+-------------+
| +---<All threads>
| 19.344 100 % | 34 100 % |
| |
| | |
| +---de.jungblut.benchmark.SpillingBufferProfile.main(String[])
| 18.720 97 % | 19 56 % |
| | |
| | |
| | +---org.apache.hadoop.io.IntWritable.write(DataOutput)
| 16.707 86 % | 17 50 % |
| | | |
| | |
| | | +---java.io.DataOutputStream.writeInt(int)
| 16.707 86 % | 17 50 % |
| | |
| | |
| | +---java.util.Random.nextInt()
| 2.012 10 % | 2 6 % |
| |
| | |
| +---java.lang.Thread.run()
| 624 3 % | 15 44 % |
| |
| | |
| +---java.util.concurrent.ThreadPoolExecutor$Worker.run()
| 624 3 % | 15 44 % |
| |
| | |
|
+---java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
| 624 3 % | 15 44 % |
| |
| | |
| +---java.util.concurrent.FutureTask.run()
| 624 3 % | 15 44 % |
| |
| | |
| +---java.util.concurrent.FutureTask$Sync.innerRun()
| 624 3 % | 15 44 % |
| |
| | |
|
+---de.jungblut.benchmark.SpillingBuffer$SpillingWriteThread.call()
| 624 3 % | 15 44 % |
| |
| | |
|
+---de.jungblut.benchmark.SpillingBuffer$SpillingWriteThread.call()
| 624 3 % | 15 44 % |
| |
| | |
|
+---de.jungblut.benchmark.SpillingBuffer$SpillingWriteThread.keepWritingToFile()
| 624 3 % | 15 44 % |
| |
| | |
|
+---de.jungblut.benchmark.SpillingBuffer$SpillWriteIndexStatus.getNextFileWriteIndex()
| 561 3 % | 14 41 % |
| |
| | |
| +---java.nio.DirectByteBuffer.put(ByteBuffer)
| 62 0 % | 1 3 % |
| |
| | |
| +---sun.misc.Unsafe.copyMemory(long, long, long)
| 62 0 % | 1 3 % |
| |
| | |
| +---sun.misc.Unsafe.copyMemory(Object, long, Object,
long, long) | 62 0 % | 1 3 % |
+------------------------------------------------------------------------------------------------------------+-----------------+-------------+
Generated by YourKit Java Profiler 11.0.7 27.10.2012 21:15:06
{noformat}
> Add a spilling message queue
> ----------------------------
>
> Key: HAMA-559
> URL: https://issues.apache.org/jira/browse/HAMA-559
> Project: Hama
> Issue Type: Sub-task
> Components: bsp core
> Affects Versions: 0.5.0
> Reporter: Thomas Jungblut
> Assignee: Suraj Menon
> Priority: Minor
> Fix For: 0.7.0
>
> Attachments: HAMA-559.patch-v1,
> spilling_buffer_cpu_usage_text_write.png,
> SpillingBufferProfile-2012-10-27.snapshot,
> spilling_buffer_profile_cpu_graph_test_write.png,
> spilling_buffer_profile_cpugraph_writeUTF.png,
> spillingbuffer_profile_cpu_writeUTF.png, spilling_buffer_profile_LOCK.JPG,
> spilling_buffer_profile_timesplit_text_write.png,
> spilling_buffer_profile_writeUTF.png
>
>
> After HAMA-521 is done, we can add a spilling queue which just holds the
> messages in RAM that fit into the heap space. The rest can be flushed to disk.
> We may call this a HybridQueue or something like that.
> The benefits should be that we don't have to flush to disk so often and get
> faster. However we may have more GC so it is always overall faster.
> The requirements for this queue also include:
> - The message object once written to the queue (after returning from the
> write call) could be modified, but the changes should not be reflected in the
> messages stored in the queue.
> - For now let's implement a queue that does not support concurrent reading
> and writing. This feature is needed when we implement asynchronous
> communication.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
