Robert Nishihara created ARROW-1410: ---------------------------------------
Summary: Plasma object store occasionally pauses for a long time Key: ARROW-1410 URL: https://issues.apache.org/jira/browse/ARROW-1410 Project: Apache Arrow Issue Type: Improvement Environment: Ubuntu 16.04 Reporter: Robert Nishihara The problem can be reproduced as follows. First start a plasma store with {code} plasma_store -s /tmp/s1 -m 500000000000 {code} Then continuously put in objects using a script like the following. {code} import pyarrow.plasma as plasma import numpy as np client = plasma.connect('/tmp/s1', '', 0) for i in range(20000): print(i) object_id = plasma.ObjectID(np.random.bytes(20)) client.create(object_id, np.random.randint(0, 100000000)) client.seal(object_id) {code} As the loop counters are being printed, you will see long pauses. The problem is the fact that we are mmapping pages with the MAP_POPULATE flag. Though this can be used to improve performance of subsequent object creations, it isn't worth the long pauses. We may want to find a way to populate the pages in the background. -- This message was sent by Atlassian JIRA (v6.4.14#64029)