Hi Michael,
Thanks for your response. Now I used this config with batchinserter, still
its very slow.
cache_type=none
use_memory_mapped_buffers=true
# 14 bytes per node
neostore.nodestore.db.mapped_memory=30G
# 33 bytes per relationships
neostore.relationshipstore.db.mapped_memory=30G
# 38 bytes per property
neostore.propertystore.db.mapped_memory=50G
# 60 bytes per long-string block
neostore.propertystore.db.strings.mapped_memory=50G
neostore.propertystore.db.index.keys.mapped_memory=50G
neostore.propertystore.db.index.mapped_memory=50G
dbms.pagecache.memory =50G
*****************************************************************************************************8
Exaplaining my code:(using batchinserterindex)
private BatchInserterIndexProvider indexProvider = new
LuceneBatchInserterIndexProvider(getInserter());
1)Index Creation:
BatchInserterIndex index = indexProvider.nodeIndex(indexName, EXACT_CONFIG);
index.setCacheCapacity("ID", 10000000);
2)Loading to index:
index.add(NeonodeID, keyValPair(ID, "asdasdasdasdasd");
3)using the index for lookup:
index.get("ID", "asdasdasdasdasd").getSingle();
Please let me know if I am missing something here. Your help in this is
much appreciated.
Thanks
On Sun, Jul 10, 2016 at 11:53 PM, 'Michael Hunger' via Neo4j <
neo4j@googlegroups.com> wrote:
> Perhaps you can share your code?
> I think it is rather the too low pagecache config
>
>
> Von meinem iPhone gesendet
>
> Am 10.07.2016 um 12:32 schrieb Ganesh Selvaraj <ganeshkumar1...@gmail.com
> >:
>
> Update<> - I tried with batchinsertIndexProvider with both lucene and also
> mapDB(the same code from here
> http://www.programcreek.com/java-api-examples/index.php?api=org.neo4j.unsafe.batchinsert.BatchInserters
> ).
> Still the lookup takes so much time. For around 100+ million relationships
> its taking around 6+ hours.
>
> On Sunday, July 10, 2016 at 8:26:49 PM UTC+12, Ganesh Selvaraj wrote:
>>
>> Hi,
>>
>> I am using batchinserter in my work to import large amounts of data
>> initially.
>> I am aware that there is a neo4j import tool, but I am not using it, as
>> we do some transformations programatically.
>>
>> When using batchinserter, after creating nodes, I use a map to store the
>> domainId-Neoid references. And later while creating relationships I lookup
>> the map for every relationship to find the corresponding Neoid.
>> Now since I am dealing with lot of relationships, the map I use is
>> slow(Tried java map, mapDb maps).
>> Can some one help with some idea to speed up the lookup process ? I am
>> also keen to know how neo4j import tool (how it handles lookups) works as
>> it is super fast.
>> Your help in this is much appreciated.
>>
>> Thanks
>> Ganesh
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Neo4j" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/neo4j/BnQkwjEfP5k/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
*Cheers*
*Ganesh*
--
You received this message because you are subscribed to the Google Groups
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
