[graph-tool] Re: memory handling when using graph-tool's MCMCs and multiprocessing on large networks

[Sam G]

thanks for your reply.

here's an example running minimize_blockmodel_dl() 10 times on 10 cores. when i 
run this on a large network (2GB, 2M vertices, 20M edges) i get a MemoryError. 

```
import graph_tool.all as gt
import multiprocessign as mp
import numpy as np

g = gt.load_graph("large_graph", fmt="graphml")

N_iter = 10
N_core = 10

def fit_sbm():
state = gt.minimize_blockmodel_dl(g)
b = state.get_blocks()
return b

def _parallel_sbm(iter = N_iter):
pool = mp.Pool(N_core)
future_res = [pool.apply_async(fit_sbm) for m in range(iter)]
res = [f.get() for f in future_res]
return res

def parallel_fit_sbm(iter = M_iter):
results = parallel_sbm(iter)
return results

results = parallel_fit_sbm()

___
graph-tool mailing list -- graph-tool@skewed.de
To unsubscribe send an email to graph-tool-le...@skewed.de

[graph-tool] Re: memory handling when using graph-tool's MCMCs and multiprocessing on large networks

[Tiago de Paula Peixoto]

Am 21.10.21 um 22:38 schrieb Sam G:

hi,

i was wondering if anyone had any tips on conserving memory when running 
graph-tool MCMCs on large networks in parallel.

i have a large network (2GB), and about 260GB of memory, and was surprised to 
receive a MemoryError when i ran 10 MCMC chains in parallel using 
multiprocessing. my impression is that MCMC was relatively light on memory.


As usual, it is not possible to say anything concrete without a minimal 
complete working example that shows the problem.


--
Tiago de Paula Peixoto 
___
graph-tool mailing list -- graph-tool@skewed.de
To unsubscribe send an email to graph-tool-le...@skewed.de

[graph-tool] memory handling when using graph-tool's MCMCs and multiprocessing on large networks

[Sam G]

hi,

i was wondering if anyone had any tips on conserving memory when running 
graph-tool MCMCs on large networks in parallel.

i have a large network (2GB), and about 260GB of memory, and was surprised to 
receive a MemoryError when i ran 10 MCMC chains in parallel using 
multiprocessing. my impression is that MCMC was relatively light on memory.

cheers,
-sam
___
graph-tool mailing list -- graph-tool@skewed.de
To unsubscribe send an email to graph-tool-le...@skewed.de