Hello Community, I'm trying to make a function to determine the betweenness of the Vertices in a Graph. I'm using Gelly for this and a custom shortestpath function This is my input graph: http://prntscr.com/d7y51y <http://prntscr.com/d7y51y>
What i've done is use collect() on the vertice values and loop over the list to determine the shortest path from those nodes to the rest of the nodes, in the loop i use a Union function to put everything in one big DataSet called "collectionDataSet" Here's the code: http://paste.thezomg.com/19919/79295357/ <http://paste.thezomg.com/19919/79295357/> After the 9th iteration i get the error: Too few memory segments provided. Hash Table needs at least 33 memory segments. I had this problem before, and it was fixed by increasing the TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY. Currently that's at 16000 after increasing it a couple of times, but the error keeps popping up after the 9th iteration. When i don't use the Union the error won't pop up The full stack trace can be found here: http://paste.thezomg.com/19921/79296084/ <http://paste.thezomg.com/19921/79296084/> I tried different methods like using a join, or using reduce right after the union, but it didn't change anything on the result. Are there other settings i need to adjust? And why is this exactly happening -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/33-segments-problem-with-configuration-set-tp10144.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
