Hi all, I am testing indexing with elasticsearch 1.3.2 on my Ubuntu-PC with 8GB of RAM and an SSD-Disk. export ES_HEAP_SIZE=5g
in elasticsearch.yml I set bootstrap.mlockall: true all other attributes are commented out First i am creating an index with curl -XPUT 'http://localhost:9200/mih' then i start a Java-Program that in a loop indexes quite big JSON-Documents, using Apache HttpClient CURL-Equivalent to the requests Java-Program sends: curl -XPOST 'http://localhost:9200/mih/mih' -d '{ "id":"AaqOeBfCwZGBXTBP", "parentId":"VOWyQAzBqPFpveBV", "contentModelId":"data", "state":"PENDING", "version":1, "children":[ ], "label":"benchtool-FpqSInvLvZlqkYEs", "utcCreated":"2014-12-17T06:24:22.728Z", "utcLastModified":"2014-12-17T06:24:22.728Z", "tags":null, "metadata":{ "DCgOxyg":{ "name":"DCgOxyg", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"6,0f0428220f57", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCgOxyg/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.747Z", "utcLastModified":"2014-12-17T06:24:22.747Z" }, "DCsgYyF":{ "name":"DCsgYyF", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"1,0f054af03d1f", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCsgYyF/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.754Z", "utcLastModified":"2014-12-17T06:24:22.754Z" }, "DCVFQgr":{ "name":"DCVFQgr", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"2,0f0651292ba2", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/metadata/DCVFQgr/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.792Z", "utcLastModified":"2014-12-17T06:24:22.792Z" } }, "binaries":{ "binary-BFpeqrSaGFekwGCX":{ "name":"binary-BFpeqrSaGFekwGCX", "size":81920, "mimetype":"application/octet-stream", "metadata":{ "DCYCtYC":{ "name":"DCYCtYC", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"3,0f0b7df76af5", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/metadata/DCYCtYC/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.809Z", "utcLastModified":"2014-12-17T06:24:22.809Z" }, "DCHCyyf":{ "name":"DCHCyyf", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"1,0f0d1709f367", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/metadata/DCHCyyf/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.820Z", "utcLastModified":"2014-12-17T06:24:22.820Z" } }, "filename":"binary-BFpeqrSaGFekwGCX.bin", "checksum":"677ac6824c100d474255e8717523ea19", "checksumType":"MD5", "path":"5,0f09c92d98d1", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-BFpeqrSaGFekwGCX/content", "internal":true }, "utcCreated":"2014-12-17T06:24:22.801Z", "utcLastModified":"2014-12-17T06:24:22.801Z" }, "binary-kOyemkEPEQkCtgTd":{ "name":"binary-kOyemkEPEQkCtgTd", "size":81920, "mimetype":"application/octet-stream", "metadata":{ "DCMDzAk":{ "name":"DCMDzAk", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"2,0f138a21e637", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/metadata/DCMDzAk/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.900Z", "utcLastModified":"2014-12-17T06:24:22.900Z" }, "DCZEQsj":{ "name":"DCZEQsj", "size":316, "mimetype":"text/xml", "filename":"dc.xml", "checksum":"e35fd65bf9081eb62b1b6bfd95c9cea5", "checksumType":"MD5", "path":"2,0f16ce4bfc03", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/metadata/DCZEQsj/content", "internal":true }, "type":"DC", "indexInline":true, "jsonData":{ "metadata":{ "title":"Test Object", "creator":"fasseg", "subject":"Testing Groven", "description":"Test Object to implement integration Tests" } }, "utcCreated":"2014-12-17T06:24:22.911Z", "utcLastModified":"2014-12-17T06:24:22.911Z" } }, "filename":"binary-kOyemkEPEQkCtgTd.bin", "checksum":"8923305820da6534f63b8bf4d40ae21c", "checksumType":"MD5", "path":"5,0f0fefcdd8f2", "source":{ "type":"url-source", "uri":"/entity/AaqOeBfCwZGBXTBP/binary/binary-kOyemkEPEQkCtgTd/content", "internal":true }, "utcCreated":"2014-12-17T06:24:22.826Z", "utcLastModified":"2014-12-17T06:24:22.826Z" } }, "alternativeIdentifiers":[], "relations":null }' The more Documents i index the slower indexing gets. So for the first 100 Documents i need about 20 ms to index 1 Document. When i have about 1000 Documents in the Index, indexing slows down and takes me about 500 ms to index 1 Document. Restart of Elasticsearch or of my Computer doesnt help, indexing still needs 500 ms per Document if I already have 1000 indexed Documents in the index. Index-Size at that point is about 60MB, containing about 30 Segments. Is there any Hint how i can accelerate indexing? 1000 Documents on one Machine shouldnt be too much. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a18976f5-898a-4d18-8e8f-ddce84841437%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
