Yes, it's probably the best if you do the relabeling externally. Let us know if it still doesn't work after using Read_Edgelist() with a relabeled file. T.
On Mon, Aug 1, 2016 at 2:37 PM, Raphael C <[email protected]> wrote: > Thank you for the quick reply. My system is certainly 64 bit. The > problem is just the amount of RAM > > g = Graph.Read_Ncol('edges.txt') > > uses it seems. > > Here is some code to produce a fake edge list that reproduces my problem. > > import random > > #Number of edges, vertices > m = 62500000 > n = m/2 > > for i in xrange(m): > fromnode = str(random.randint(0, n-1)).zfill(9) > tonode = str(random.randint(0, n-1)).zfill(9) > print fromnode, tonode > > If I produce a file edges.txt using this code and then run > > from igraph import Graph > g = Graph.Read_Ncol('edges.txt') > > it runs out of RAM. > > To get a better picture of the RAM usage I ran the same test with m = > 20000000 (that is about one third of the edges and vertices). > > /usr/bin/time -v python ./test.py > > shows > > Maximum resident set size (kbytes): 3172988 > > With m = 30000000 I see Maximum resident set size (kbytes): 4750440 > > Maybe one solution is to relabel the nodes myself external so I can > avoid the overhead of Ncol? > > Raphael > > > > > On 1 August 2016 at 10:23, Tamas Nepusz <[email protected]> wrote: >> Hello, >> >> Read_Edgelist() won't work because that assumes that the vertex IDs >> are in the range [0; |V|-1] so it would create lots of isolated >> vertices if your vertex ID range has "gaps" in it. Read_Ncol() is the >> way to go, but it has an additional space penalty as it has to >> maintain a mapping from the numeric IDs in the file to the range [0; >> |V|-1]. >> >> igraph requires 32 bytes per edge and 16 bytes per vertex to store the >> graph itself, plus additional data structures to store the vertex/edge >> attributes. Therefore, a graph of your size would require ~2.5 GB of >> memory plus the attributes. 8 GB of RAM should therefore be enough -- >> however, note that Python might not be able to utilize all that >> memory. In particular, 32-bit Python on Windows is limited to 2 or 3 >> GBs of memory (see >> https://msdn.microsoft.com/en-us/library/aa366778(v=vs.85).aspx#memory_limits >> ). If you happen to use a 32-bit Python on a 64-bit machine, you will >> need to install a 64-bit Python with a corresponding igraph package >> that is also built for 64-bit, and then try again. >> >> Best, >> T. >> >> >> On Mon, Aug 1, 2016 at 9:52 AM, Raphael C <[email protected]> wrote: >>> I have 8GB of RAM and I have a simple edge list text file of size >>> 1.2GB. It was 62500000 edges and about half that many vertices. Each >>> line looks like >>> >>> 287111206 357850135 >>> >>> I would like to read in the graph and output a sparse adjacency >>> matrix. I am failing on all counts. I have tried >>> >>> >>> g = Graph.Read_Edgelist('edges.txt') >>> >>> but this fails immediately with >>> >>> MemoryError: Error at vector.pmt:439: cannot reserve space for vector, >>> Out of memory >>> >>> This seems unrelated to the size of the graph is just a function of >>> the node ids being large. >>> >>> So instead I tried >>> >>> g = Graph.Read_Ncol('edges.txt') >>> >>> This eats up all the RAM in my PC forcing me to kill the code. >>> >>> I fact I tested g = Graph.Read_Ncol('edges.txt') with the first 1/5 of >>> the edges and have the same memory problem. >>> >>> Each node id is a 32 bit integer so the graph should fit easily in 8GB of >>> RAM. >>> >>> What can I do? >>> >>> Thanks very much for any help. >>> Raphael >>> >>> _______________________________________________ >>> igraph-help mailing list >>> [email protected] >>> https://lists.nongnu.org/mailman/listinfo/igraph-help >> >> _______________________________________________ >> igraph-help mailing list >> [email protected] >> https://lists.nongnu.org/mailman/listinfo/igraph-help > > _______________________________________________ > igraph-help mailing list > [email protected] > https://lists.nongnu.org/mailman/listinfo/igraph-help _______________________________________________ igraph-help mailing list [email protected] https://lists.nongnu.org/mailman/listinfo/igraph-help
