[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[ https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117011#comment-13117011 ] Avery Ching commented on GIRAPH-12: --- If the default stack size is 1 MB, then for instance if you have 1024 workers, you are talking about 1 GB just wasted for thread stack space per node. The aggregate wasted memory would be 1 GB * 1024 = 1 TB, that's a lot of memory =). The issue is that many clusters (including Yahoo!'s) have are running only 32-bit JVMs. So if you are using 1 GB just for stack space, you only get so much left for heap (graph + messages). I think this should help quite a bit until GIRAPH-37 is taken on. Can you run the unittests against a real Hadoop instance as well? Then I'd say +1, unless someone disagrees. > Investigate communication improvements > -- > > Key: GIRAPH-12 > URL: https://issues.apache.org/jira/browse/GIRAPH-12 > Project: Giraph > Issue Type: Improvement > Components: bsp >Reporter: Avery Ching >Assignee: Hyunsik Choi >Priority: Minor > Attachments: GIRAPH-12_1.patch, GIRAPH-12_2.patch > > > Currently every worker will start up a thread to communicate with every other > workers. Hadoop RPC is used for communication. For instance if there are > 400 workers, each worker will create 400 threads. This ends up using a lot > of memory, even with the option > -Dmapred.child.java.opts="-Xss64k". > It would be good to investigate using frameworks like Netty or custom roll > our own to improve this situation. By moving away from Hadoop RPC, we would > also make compatibility of different Hadoop versions easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[
https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116961#comment-13116961
]
Hyunsik Choi commented on GIRAPH-12:
Avery,
Thank you for your review. You are right. Runtime's totalMem() and freeMem()
methods doesn't measure stack sizes. I'm sure of it after testing the below
code.
https://gist.github.com/1249761
I have looked for how to measure the stack size of a java application. I could
not find about that. Still, I'm not sure how to show that thread stack memory
is reduced by the thread pool approach. Now, your way seems a only method to
prove them.
However, I'm curious to know how much thread overhead is in terms of memory
consumption. Before I try your approach. I conducted some simple experiments.
I used the above source code to investigate the memory usage of threads. This
is executed on a machine with intel i3, ubuntu 11.10 (64bit), and 8G memory. I
measure their memory by using 'top'. 'top' shows several columns including VIRT
and RES, and SHR. We only need to focus RES, resident memory. RES includes all
resident memory usages, such as heap and stack. I could know this from this
page (http://goo.gl/JE7fD).
Firstly, I executed the above code with 1000 threads and without a jvm option
'-Xss'. Accoring to this page (http://goo.gl/sz2qM), the default stack size
'Xss' is 1024k on the jvm of 64bit linux. After all threads are created, I
executed 'top' to print the memory usages as follows:
1k threads with default thread stack size.
{noformat}
VIRT RES SHR
9163 hyunsik 20 0 3366m 30m 8296 S 18 0.4 0:01.52 java
{noformat}
2k threads with default thread stack size.
{noformat}
VIRT RES SHR
11223 hyunsik 20 0 4434m 46m 8340 S 40 0.6 0:04.11 java
{noformat}
With 1k and 2k threads, that program consumes only 30 and 46 mega bytes
respectively. The memory usage of threads are smaller than I expected. I wonder
if thread stack size is the main cause of the memory problem that we have faced.
Besides, the default stack size is 1024k. The thread stack size seems to not
affect RES. I had more tests with 'Xss' in order to investigate more the thread
stack size.
1k threads with '-Xss4096k'.
{noformat}
28301 hyunsik 20 0 6380m 30m 8292 S 17 0.4 0:05.25 java
{noformat}
2k threads with '-Xss4096k'
{noformat}
29326 hyunsik 20 0 10.1g 46m 8300 S 38 0.6 0:03.42 java
{noformat}
VIRT surely is affected by '-Xss', but RES is not. 'Xss' seems the maximum
stack size of each thread because it doesn't affect RES.
What do you think about that?
> Investigate communication improvements
> --
>
> Key: GIRAPH-12
> URL: https://issues.apache.org/jira/browse/GIRAPH-12
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
>Reporter: Avery Ching
>Assignee: Hyunsik Choi
>Priority: Minor
> Attachments: GIRAPH-12_1.patch, GIRAPH-12_2.patch
>
>
> Currently every worker will start up a thread to communicate with every other
> workers. Hadoop RPC is used for communication. For instance if there are
> 400 workers, each worker will create 400 threads. This ends up using a lot
> of memory, even with the option
> -Dmapred.child.java.opts="-Xss64k".
> It would be good to investigate using frameworks like Netty or custom roll
> our own to improve this situation. By moving away from Hadoop RPC, we would
> also make compatibility of different Hadoop versions easier.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[ https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116960#comment-13116960 ] Hyunsik Choi commented on GIRAPH-12: Dmitriy, Thank you for your comments. Regardless of the problem caused by thread stack size, those approaches look promising. Especially, spilling messages to disk looks necessary so that Giraph deals with really large graph data. Otherwise, out of memory may occur when the message generating rate are higher than network bandwidth. I'll open a separate issue about this. > Investigate communication improvements > -- > > Key: GIRAPH-12 > URL: https://issues.apache.org/jira/browse/GIRAPH-12 > Project: Giraph > Issue Type: Improvement > Components: bsp >Reporter: Avery Ching >Assignee: Hyunsik Choi >Priority: Minor > Attachments: GIRAPH-12_1.patch, GIRAPH-12_2.patch > > > Currently every worker will start up a thread to communicate with every other > workers. Hadoop RPC is used for communication. For instance if there are > 400 workers, each worker will create 400 threads. This ends up using a lot > of memory, even with the option > -Dmapred.child.java.opts="-Xss64k". > It would be good to investigate using frameworks like Netty or custom roll > our own to improve this situation. By moving away from Hadoop RPC, we would > also make compatibility of different Hadoop versions easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent Messages' doesn't work.
[
https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116928#comment-13116928
]
Severin Corsten commented on GIRAPH-42:
---
In addition I found an issue implementing point 1 and 3 together, thus I am not
quite sure whether its issued by it:
I run the SimpleShortestPathVertex in a standard Giraph on a big dataset and
got as a result: Giraph Stats: Superstep: 15
After adding a counter in Vertex.sendMsg and one in the GiraphMapper just
behind the compute call, the result changed to: Giraph Stats: Superstep: 16
while the number of supersteps show in Giraph Timers is the same (15).
I am not sure whether is caused by an other change or by a bad condition and I
didn't reproduce it yet.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
>Reporter: Hyunsik Choi
>Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished
> vertices=60
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=60
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent Messages' doesn't work.
[
https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116775#comment-13116775
]
Severin Corsten commented on GIRAPH-42:
---
I don't see a reson why not to implement both 1 and 3. That allows the user to
have a better view on his Combiner.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
>Reporter: Hyunsik Choi
>Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished
> vertices=60
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=60
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-42) The MapReduce counter 'Sent Messages' doesn't work.
[
https://issues.apache.org/jira/browse/GIRAPH-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116592#comment-13116592
]
Hyunsik Choi commented on GIRAPH-42:
I also like the third approach. The current implementation seems to aim at the
third approach.
> The MapReduce counter 'Sent Messages' doesn't work.
> ---
>
> Key: GIRAPH-42
> URL: https://issues.apache.org/jira/browse/GIRAPH-42
> Project: Giraph
> Issue Type: Bug
> Components: bsp
>Reporter: Hyunsik Choi
>Priority: Minor
>
> The MapReduce counter 'Sent Messages' doesn't work. It always shows 0.
> {noformat}
> .
> .
> 11/09/28 10:51:22 INFO mapred.JobClient: Current workers=20
> 11/09/28 10:51:22 INFO mapred.JobClient: Current master task partition=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Sent messages=0
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate finished
> vertices=60
> 11/09/28 10:51:22 INFO mapred.JobClient: Aggregate vertices=60
> .
> .
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[ https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116259#comment-13116259 ] Avery Ching commented on GIRAPH-12: --- I wonder if Runtime methods measure the stack sizes of the threads consuming all that memory? My guess is no. Since we're using less threads, we should have less stacks consuming all the minimum memory. I agree that the heap memory won't change much in going to your approach. I'm not sure how to prove that thread stack memory is being reduced if Runtime fails to capture this though. One crude way would simply be to increase the heap space with your test until you find a maximum heap size that can be used in your code and the original code. If your code can reach a higher heap allocation, than that should prove a memory win (more memory can be used for the heap). Here's some arguments to try out that approach if you're interested: -Dmapred.child.java.opts="-Xms2750m -Xmx2750m" As you bump up the -Xms and -Xmx values simultaneously, eventually your job won't start and hopefully your changes enable a higher limit... > Investigate communication improvements > -- > > Key: GIRAPH-12 > URL: https://issues.apache.org/jira/browse/GIRAPH-12 > Project: Giraph > Issue Type: Improvement > Components: bsp >Reporter: Avery Ching >Assignee: Hyunsik Choi >Priority: Minor > Attachments: GIRAPH-12_1.patch, GIRAPH-12_2.patch > > > Currently every worker will start up a thread to communicate with every other > workers. Hadoop RPC is used for communication. For instance if there are > 400 workers, each worker will create 400 threads. This ends up using a lot > of memory, even with the option > -Dmapred.child.java.opts="-Xss64k". > It would be good to investigate using frameworks like Netty or custom roll > our own to improve this situation. By moving away from Hadoop RPC, we would > also make compatibility of different Hadoop versions easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[ https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116253#comment-13116253 ] Dmitriy V. Ryaboy commented on GIRAPH-12: - Julien has a nice post describing how one goes about detecting low memory conditions: https://techblug.wordpress.com/2011/07/21/detecting-low-memory-in-java-part-2/ . The first thing to do when this happens is probably to run through combiners to attempt to free some memory. Assuming we still need to do something with the messages, there are two approaches that come to mind: 1) Spill to disk and keep track of spilled messages. This is going to cost us, but it'll make it possible to make progress when otherwise an OOM error would occur. 2) Send the messages to intended recipients instead of spilling to disk. That will be speedier, but does run the risk of the other side being out of memory and unable to accumulate, too. Either way, this ticket is more about reworking the communication code than about memory improvements -- we haven't measured how much memory individual threads are taking up, but I am betting their impact is dwarfed by buffered messages and the in-memory graph segment we are working on, so it would be surprising if we could substantially reduce the amount of memory by simply switching to thread pools. Let's open a separate JIRA to deal with message accumulation better, and consider this code on merits other than memory footprint. > Investigate communication improvements > -- > > Key: GIRAPH-12 > URL: https://issues.apache.org/jira/browse/GIRAPH-12 > Project: Giraph > Issue Type: Improvement > Components: bsp >Reporter: Avery Ching >Assignee: Hyunsik Choi >Priority: Minor > Attachments: GIRAPH-12_1.patch, GIRAPH-12_2.patch > > > Currently every worker will start up a thread to communicate with every other > workers. Hadoop RPC is used for communication. For instance if there are > 400 workers, each worker will create 400 threads. This ends up using a lot > of memory, even with the option > -Dmapred.child.java.opts="-Xss64k". > It would be good to investigate using frameworks like Netty or custom roll > our own to improve this situation. By moving away from Hadoop RPC, we would > also make compatibility of different Hadoop versions easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
