Re: [Neo4j] Cats and Dogs, living together
On Mon, Dec 6, 2010 at 16:57, Peter Neubauer peter.neuba...@neotechnology.com wrote: Very very cool Javier! Thank you :) Is this built using the Neo4j Python bindings or pure REST? Also, is there a public website available to refer to? By now we are using only REST, but the performance is not what we expected when we process large nodes returned by a traversal (so much HTTP requests). So we are now evaluating whether using de Python binding or building a Java socket server and a Python socket client could be better. When you make a traversal or use the indices, the REST API returns the URLs of each of the nodes returned, so we need make one HTTP request more per node. It would be great if we could send an optional param to make the server returned all properties. Another question - regarding visualisation, what was your experience of the best performing lib for JavaScript out there regarding large amount of nodes and relationships to render, and adaptability for UI? Currently, it seems there is - TheJIT - Processing.js - Graphdracula TheJIT was our first approach, but with large datasets the behaviour is not very fast. Besides, the interaction ways are a bit limited and hard to expand. Processing.js is, with no doubt, the most promising solution. We were happy using Porcessing.js, but you need build all you need to represent graphs, nodes and edges. It's very low level programation and by now the browsers can't with it, therefore we used a mixed version between Processing.js in the browser and NetworkX in the server side for some calculations of layout and etc. Graphdracula was an inspiration for us. It's very beautiful, but it's also very incomplete yet. I guess in the near future will be very useful, but we need total control of all happens in the UI. So, now we are using Räphael, the core library of Graphdracula, and we are implementing several layout algorithms in Javascript and some ways to interact with nodes to expand the graph by browsing. I hope to setup a Sylva test site soon, then I will e-mail to this list, if that's alright with you. Best regards. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Dec 2, 2010 at 6:03 PM, Javier de la Rosa ver...@gmail.com wrote: Hi, everybody, I work in a lab at University of Western Ontario, with humanists people. The needs of the humanist research make useless the current SQL databases because it's hard to change the schemas or build queries with several JOINs. So, we are developing a system which mixes a relational database and Neo4j. Sylva [1], as it's called (previously Graphgamel), stores all data in the Neo4j database as a graph. On the other hand, de multimedia files (image, video and audio files) are stores using the relational one (over Django). Besides, the relational part allows the definition of lazy and dynamic schemas, very usefull to model the world from the humanistic point of view. The users can create nodes and relationships but according to a certain kind of integrity defined in the schema. Sylva also has a very early version of visualization through Raphäel and Processing.js. Here you can see some screenshots [2, 3, 4] and a video demo [5]. Our goal is to adapt the django-qbe project [6] to our schema tool in order to produce Gremlin queries in a visual way. But by now we are using the Neo4 REST component but it's not very fast and it has some limitations. It's an alpha version, but it goes without saying :-) [1] https://github.com/escalant3/graphgamel [2] http://dl.dropbox.com/u/2630535/sylva.png [3] http://dl.dropbox.com/u/2630535/plexigraph.png [4] http://dl.dropbox.com/u/2630535/grafo.png [5] http://www.youtube.com/watch?v=r04eV7vghfs (sorry, not subtitles or audio yet) [6] http://versae.github.com/qbe/ On Wed, Dec 1, 2010 at 12:52, Andreas Kollegger andreas.kolleg...@neotechnology.com wrote: Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? We're starting to think about best practices for integration: * Hybrid data-modeling: what goes where? * XA transactions * message queues for data distribution * data migration strategies Any problems or feature-requests related to living in a multi-storage-platform world are welcome. Cheers, Andreas ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Javier de la Rosa http://versae.es ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Javier, thanks for the feedback! There is a strong desire for having some graph visualization in the Neo4j Admin console, so - if you think it is interesting, I think there might be a strong case for the projects working together on the visualization component. I don't have the timeframe laid out yet but Neo Technology can dedicate resources to it early next year. Would that make sense to you? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Tue, Dec 7, 2010 at 3:44 PM, Javier de la Rosa ver...@gmail.com wrote: On Mon, Dec 6, 2010 at 16:57, Peter Neubauer peter.neuba...@neotechnology.com wrote: Very very cool Javier! Thank you :) Is this built using the Neo4j Python bindings or pure REST? Also, is there a public website available to refer to? By now we are using only REST, but the performance is not what we expected when we process large nodes returned by a traversal (so much HTTP requests). So we are now evaluating whether using de Python binding or building a Java socket server and a Python socket client could be better. When you make a traversal or use the indices, the REST API returns the URLs of each of the nodes returned, so we need make one HTTP request more per node. It would be great if we could send an optional param to make the server returned all properties. Another question - regarding visualisation, what was your experience of the best performing lib for JavaScript out there regarding large amount of nodes and relationships to render, and adaptability for UI? Currently, it seems there is - TheJIT - Processing.js - Graphdracula TheJIT was our first approach, but with large datasets the behaviour is not very fast. Besides, the interaction ways are a bit limited and hard to expand. Processing.js is, with no doubt, the most promising solution. We were happy using Porcessing.js, but you need build all you need to represent graphs, nodes and edges. It's very low level programation and by now the browsers can't with it, therefore we used a mixed version between Processing.js in the browser and NetworkX in the server side for some calculations of layout and etc. Graphdracula was an inspiration for us. It's very beautiful, but it's also very incomplete yet. I guess in the near future will be very useful, but we need total control of all happens in the UI. So, now we are using Räphael, the core library of Graphdracula, and we are implementing several layout algorithms in Javascript and some ways to interact with nodes to expand the graph by browsing. I hope to setup a Sylva test site soon, then I will e-mail to this list, if that's alright with you. Best regards. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Dec 2, 2010 at 6:03 PM, Javier de la Rosa ver...@gmail.com wrote: Hi, everybody, I work in a lab at University of Western Ontario, with humanists people. The needs of the humanist research make useless the current SQL databases because it's hard to change the schemas or build queries with several JOINs. So, we are developing a system which mixes a relational database and Neo4j. Sylva [1], as it's called (previously Graphgamel), stores all data in the Neo4j database as a graph. On the other hand, de multimedia files (image, video and audio files) are stores using the relational one (over Django). Besides, the relational part allows the definition of lazy and dynamic schemas, very usefull to model the world from the humanistic point of view. The users can create nodes and relationships but according to a certain kind of integrity defined in the schema. Sylva also has a very early version of visualization through Raphäel and Processing.js. Here you can see some screenshots [2, 3, 4] and a video demo [5]. Our goal is to adapt the django-qbe project [6] to our schema tool in order to produce Gremlin queries in a visual way. But by now we are using the Neo4 REST component but it's not very fast and it has some limitations. It's an alpha version, but it goes without saying :-) [1] https://github.com/escalant3/graphgamel [2] http://dl.dropbox.com/u/2630535/sylva.png [3] http://dl.dropbox.com/u/2630535/plexigraph.png [4] http://dl.dropbox.com/u/2630535/grafo.png [5] http://www.youtube.com/watch?v=r04eV7vghfs (sorry, not subtitles or audio yet) [6] http://versae.github.com/qbe/ On Wed, Dec 1, 2010 at
Re: [Neo4j] Cats and Dogs, living together
On Tue, Dec 7, 2010 at 10:39, Peter Neubauer peter.neuba...@neotechnology.com wrote: There is a strong desire for having some graph visualization in the Neo4j Admin console, so - if you think it is interesting, I think there might be a strong case for the projects working together on the visualization component. I don't have the timeframe laid out yet but Neo Technology can dedicate resources to it early next year. Would that make sense to you? Of course yes. We're not very familiar using Java, but I think it's not needed, because the most of the code in the web admin console would be Javascript. The only thing to decide is the format of data to interchange between the server and the browser to paint the graph and the properties. Anyways, we will keep working on Sylva and its visual component, although in a slower way than we would like. For whatever, I'm always available at my e-mail. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Tue, Dec 7, 2010 at 3:44 PM, Javier de la Rosa ver...@gmail.com wrote: On Mon, Dec 6, 2010 at 16:57, Peter Neubauer peter.neuba...@neotechnology.com wrote: Very very cool Javier! Thank you :) Is this built using the Neo4j Python bindings or pure REST? Also, is there a public website available to refer to? By now we are using only REST, but the performance is not what we expected when we process large nodes returned by a traversal (so much HTTP requests). So we are now evaluating whether using de Python binding or building a Java socket server and a Python socket client could be better. When you make a traversal or use the indices, the REST API returns the URLs of each of the nodes returned, so we need make one HTTP request more per node. It would be great if we could send an optional param to make the server returned all properties. Another question - regarding visualisation, what was your experience of the best performing lib for JavaScript out there regarding large amount of nodes and relationships to render, and adaptability for UI? Currently, it seems there is - TheJIT - Processing.js - Graphdracula TheJIT was our first approach, but with large datasets the behaviour is not very fast. Besides, the interaction ways are a bit limited and hard to expand. Processing.js is, with no doubt, the most promising solution. We were happy using Porcessing.js, but you need build all you need to represent graphs, nodes and edges. It's very low level programation and by now the browsers can't with it, therefore we used a mixed version between Processing.js in the browser and NetworkX in the server side for some calculations of layout and etc. Graphdracula was an inspiration for us. It's very beautiful, but it's also very incomplete yet. I guess in the near future will be very useful, but we need total control of all happens in the UI. So, now we are using Räphael, the core library of Graphdracula, and we are implementing several layout algorithms in Javascript and some ways to interact with nodes to expand the graph by browsing. I hope to setup a Sylva test site soon, then I will e-mail to this list, if that's alright with you. Best regards. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Dec 2, 2010 at 6:03 PM, Javier de la Rosa ver...@gmail.com wrote: Hi, everybody, I work in a lab at University of Western Ontario, with humanists people. The needs of the humanist research make useless the current SQL databases because it's hard to change the schemas or build queries with several JOINs. So, we are developing a system which mixes a relational database and Neo4j. Sylva [1], as it's called (previously Graphgamel), stores all data in the Neo4j database as a graph. On the other hand, de multimedia files (image, video and audio files) are stores using the relational one (over Django). Besides, the relational part allows the definition of lazy and dynamic schemas, very usefull to model the world from the humanistic point of view. The users can create nodes and relationships but according to a certain kind of integrity defined in the schema. Sylva also has a very early version of visualization through Raphäel and Processing.js. Here you can see some screenshots [2, 3, 4] and a video demo [5]. Our goal is to adapt the django-qbe project [6] to our schema tool in order to produce Gremlin
Re: [Neo4j] Cats and Dogs, living together
Very very cool Javier! Is this built using the Neo4j Python bindings or pure REST? Also, is there a public website available to refer to? Another question - regarding visualisation, what was your experience of the best performing lib for JavaScript out there regarding large amount of nodes and relationships to render, and adaptability for UI? Currently, it seems there is - TheJIT - Processing.js - Graphdracula Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Dec 2, 2010 at 6:03 PM, Javier de la Rosa ver...@gmail.com wrote: Hi, everybody, I work in a lab at University of Western Ontario, with humanists people. The needs of the humanist research make useless the current SQL databases because it's hard to change the schemas or build queries with several JOINs. So, we are developing a system which mixes a relational database and Neo4j. Sylva [1], as it's called (previously Graphgamel), stores all data in the Neo4j database as a graph. On the other hand, de multimedia files (image, video and audio files) are stores using the relational one (over Django). Besides, the relational part allows the definition of lazy and dynamic schemas, very usefull to model the world from the humanistic point of view. The users can create nodes and relationships but according to a certain kind of integrity defined in the schema. Sylva also has a very early version of visualization through Raphäel and Processing.js. Here you can see some screenshots [2, 3, 4] and a video demo [5]. Our goal is to adapt the django-qbe project [6] to our schema tool in order to produce Gremlin queries in a visual way. But by now we are using the Neo4 REST component but it's not very fast and it has some limitations. It's an alpha version, but it goes without saying :-) [1] https://github.com/escalant3/graphgamel [2] http://dl.dropbox.com/u/2630535/sylva.png [3] http://dl.dropbox.com/u/2630535/plexigraph.png [4] http://dl.dropbox.com/u/2630535/grafo.png [5] http://www.youtube.com/watch?v=r04eV7vghfs (sorry, not subtitles or audio yet) [6] http://versae.github.com/qbe/ On Wed, Dec 1, 2010 at 12:52, Andreas Kollegger andreas.kolleg...@neotechnology.com wrote: Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? We're starting to think about best practices for integration: * Hybrid data-modeling: what goes where? * XA transactions * message queues for data distribution * data migration strategies Any problems or feature-requests related to living in a multi-storage-platform world are welcome. Cheers, Andreas ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Javier de la Rosa http://versae.es ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Kalin, I think the different shell access possibilities are good candidates for fast integration. You have at least Neo4j Shell, distributed with Neo4j downloads, see http://wiki.neo4j.org/content/Getting_Started_with_Neo4j_Server#Enter_the_Shell JRuby scripts, see http://neo4j.rubyforge.org/ and https://github.com/andreasronge/neo4j Gremlin, https://github.com/tinkerpop/gremlin/wiki/Getting-Started Python and others, many of these can be used from the command line to read and pipe data into Neo4j. Would one of these be an option? Especially the Neo4j Shell connects remotely (the others will when the RemoteGraphDB component is fixed up) and should be able to do most of the usual tasks... WDYT? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Thu, Dec 2, 2010 at 8:12 AM, Kalin Wilson Development d...@kalinwilson.com wrote: I'm currently prototyping an application that will use both a RDBMS and Neo4j in parallel. The RDBMS currently exists as the backing store for a web application. While it's feasible to move all of the data into Neo4j, that's not politically palatable right now. So I plan to use Neo4j to store dependency relationships and associated data with references back to the RDBMS. To minimize synchronization issues, I plan to only store entity type and id information in nodes unless specific information is needed to control traversals. Right now most of the data access is via JPA using hibernate through a pretty good DAO-domain abstraction. I plan to extend the domain model to include neo4j nodes and relationships along the lines of your examples and extend the DAOs to include management of the embedded Neo4j instance. My prototype is a Spring app and currently uses aspect oriented transaction management, but in order to manage the transactions on the two databases, I'll probably have to handle transactions programmatically, probably wrapping one in the other. It would be nice to combine the transaction management somehow. In terms of data migration, although I'm trying to minimize migration right now, I have thought about how I might migrate if I were to use Neo4j exclusively. I'm still not sure what the best approaches are to properties vs nodes for entity attributes, indexes (lucene) vs type subnodes, etc. One of the hurdles to migrating to Neo4j is the lack of tool support, particularly for general access. This has been acknowledged by Emil and others, so I'm not complaining. But with our current app, if we need to import data or fix general data problems, we can use a SQL workbench to directly access the database rather than write a special capability in the app. That's not always the best approach but it does allow other developers more familiar with other technologies the ability to access the DB with their tools/language due to the prevalence of SQL. With Neo4j it seems that programmatic access will be required. Granted, I haven't explored the REST server, SPARQL support, or the shell very much. They may offer more generalized access. I'm interested to hear how others are approaching the polyglot-persistence task. As I move forward, I'll share what I learn or have problems with. Thanks, Kalin On Dec 1, 2010, at 10:52 AM, Andreas Kollegger wrote: Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? We're starting to think about best practices for integration: * Hybrid data-modeling: what goes where? * XA transactions * message queues for data distribution * data migration strategies Any problems or feature-requests related to living in a multi-storage-platform world are welcome. Cheers, Andreas ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? Let me contribute my 2 cents here. I have developed a rough demo that shows what my thoughts are for integration of Neo4j instances in a managed environment. Using this JCA 1.6 compliant connector module I have successfully - Injected a ConnectionFactory (akin to a DataSource from JDBC) that returns NeoConnections (similar to javax.sql.Connection). - Performed operations on it (node creations/property settings) in tandem with operations on a mySQL instance. That is, in the same EJB method, with a container managed transaction. Both resources were enlisted in the top level tx and were successfully committed or rolledback (when I violated constraints in either db, for instance), in proper 2PC manner. So far, based on my experience working with JDBC, the feeling is the same, as far as usability goes. The GraphDatabaseService interface is not yet completely available. The code is available at the repository: The transaction manager to use (based on previous work): https://svn.neo4j.org/laboratory/users/cgioran/ContainerProvidedTxManager/ The connector module: https://svn.neo4j.org/laboratory/users/cgioran/NeoJCA/ The modified kernel (nothing major, just exposed retrieval of XAResources and enforced the new txm) https://svn.neo4j.org/laboratory/users/cgioran/neo4j-kernel-jta/ The idea is that you build/install all three above, add the jta kernel and the txm service in your ear's libraries and the connector module as an ear connector module (in application.xml). Create a pool and bind it to the JNDI and retrieve it from there in your EJB/Servlet. DISCLAIMER: The above code is a toy and completely temporary. I have not verified its operation in any significant way, I do not even know if (in fact I doubt that) the pooling mechanism works properly. Do not use it in any way that might be remotely important. The connector will work only in glassfish. Other app servers (JonAS and Geronimo for certain) require additional descriptor files apart from ra.xml If you want to deploy in another container, please consult its manual on the proper procedure. I will now go and write a post explaining in more detail how to build, install and use the above. If, in the meantime, someone feels adventurous and decides to look at the code, feedback is always welcome. The important thing for me is not the correctness of the code - there is not much there anyway. What I am interested in is its dynamics and the possible interactions in an app server. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Peter, Those are certainly viable alternatives (need to investigate further), but most involve learning another language/interface or working at the command line or within a script. The ubiquity of SQL access to a database is a big factor when considering migration. One of my teammates is much better with SQL than I am and his preferred language is Visual Basic. His role in supporting a database driven app has fallen to mostly data maintenance which he accomplishes either through the Mysql workbench using it's browsing capabilities and direct SQL, or quick and dirty utilities he can write in VB to manipulate the data. So as I consider implementing part of the system with Neo4j, I have to consider that I'll be removing access to and manipulation of that data from other teammates who prefer or have expertise with SQL and require that access to the Neo4j data be programmatic or via shell/command line which can be limiting in the view you have. The flexible, schema-less nature of Neo4j is a great feature. But at some point in order to be able to traverse and understand a graph, a user needs to understand how the graph has been built and it's semantics. With tools like Mysql workbench you can browse the DB metadata to discover or remember table/field names and types, relationships, etc. Not having such a browser to discover and visualize a Neo4j graph requires more rigor among a team to document and adhere to the network model. Rigor is good but not always followed up on. I guess you could use the shell to traverse a graph and dump all of the relationship types, etc, but could you dump all of the properties on nodes and relationships to get a view of the model? Even with that would you have enough information to begin writing traversals to extract or manipulate data? Would you understand the semantics? This sort of tool support and general access capability will be very useful to Neo4j when its available, IMO. Thanks for the dialog! Kalin On Dec 2, 2010, at 5:48 AM, Peter Neubauer wrote: Kalin, I think the different shell access possibilities are good candidates for fast integration. You have at least Neo4j Shell, distributed with Neo4j downloads, see http://wiki.neo4j.org/content/Getting_Started_with_Neo4j_Server#Enter_the_Shell JRuby scripts, see http://neo4j.rubyforge.org/ and https://github.com/andreasronge/neo4j Gremlin, https://github.com/tinkerpop/gremlin/wiki/Getting-Started Python and others, many of these can be used from the command line to read and pipe data into Neo4j. Would one of these be an option? Especially the Neo4j Shell connects remotely (the others will when the RemoteGraphDB component is fixed up) and should be able to do most of the usual tasks... WDYT? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Kalin (and list), My lab project for a few weeks now has been an alternative shell to Neo4j. Coming from a DBA background, I felt that we are lacking tools to do DBA-work - import/export data, tune indexes, browse the data in an easy way, and make backups. So that's been the focus for my shell. It hosts Rhino, so you write scripts in Javascript. This gives you a Turing complete scripting environment, which I missed. The API looks like the Java API, but has been simplified, to make REPL more enjoyable (stuff like auto-commits). I've also built in some support to make it easy to import CSV-files. It's available herehttps://trac.neo4j.org/browser/laboratory/users/andres/awesome-shellif anyone want to take a look. It's still just a lab project, so be cautious if you test drive it. I've been doing this pretty much on my own, so I'd love feedback if there's any to be had. Cheers, Andrés On Thu, Dec 2, 2010 at 3:02 PM, Kalin Wilson Development d...@kalinwilson.com wrote: Peter, Those are certainly viable alternatives (need to investigate further), but most involve learning another language/interface or working at the command line or within a script. The ubiquity of SQL access to a database is a big factor when considering migration. One of my teammates is much better with SQL than I am and his preferred language is Visual Basic. His role in supporting a database driven app has fallen to mostly data maintenance which he accomplishes either through the Mysql workbench using it's browsing capabilities and direct SQL, or quick and dirty utilities he can write in VB to manipulate the data. So as I consider implementing part of the system with Neo4j, I have to consider that I'll be removing access to and manipulation of that data from other teammates who prefer or have expertise with SQL and require that access to the Neo4j data be programmatic or via shell/command line which can be limiting in the view you have. The flexible, schema-less nature of Neo4j is a great feature. But at some point in order to be able to traverse and understand a graph, a user needs to understand how the graph has been built and it's semantics. With tools like Mysql workbench you can browse the DB metadata to discover or remember table/field names and types, relationships, etc. Not having such a browser to discover and visualize a Neo4j graph requires more rigor among a team to document and adhere to the network model. Rigor is good but not always followed up on. I guess you could use the shell to traverse a graph and dump all of the relationship types, etc, but could you dump all of the properties on nodes and relationships to get a view of the model? Even with that would you have enough information to begin writing traversals to extract or manipulate data? Would you understand the semantics? This sort of tool support and general access capability will be very useful to Neo4j when its available, IMO. Thanks for the dialog! Kalin On Dec 2, 2010, at 5:48 AM, Peter Neubauer wrote: Kalin, I think the different shell access possibilities are good candidates for fast integration. You have at least Neo4j Shell, distributed with Neo4j downloads, see http://wiki.neo4j.org/content/Getting_Started_with_Neo4j_Server#Enter_the_Shell JRuby scripts, see http://neo4j.rubyforge.org/ and https://github.com/andreasronge/neo4j Gremlin, https://github.com/tinkerpop/gremlin/wiki/Getting-Started Python and others, many of these can be used from the command line to read and pipe data into Neo4j. Would one of these be an option? Especially the Neo4j Shell connects remotely (the others will when the RemoteGraphDB component is fixed up) and should be able to do most of the usual tasks... WDYT? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
I will now go and write a post explaining in more detail how to build, install and use the above. And it is available at http://digitalstain.blogspot.com/2010/12/towards-neo4j-connector-implementation.html be gentle. thank you for your time, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
I'm currently prototyping an application that will use both a RDBMS and Neo4j in parallel. The RDBMS currently exists as the backing store for a web application. While it's feasible to move all of the data into Neo4j, that's not politically palatable right now. So I plan to use Neo4j to store dependency relationships and associated data with references back to the RDBMS. To minimize synchronization issues, I plan to only store entity type and id information in nodes unless specific information is needed to control traversals. Right now most of the data access is via JPA using hibernate through a pretty good DAO-domain abstraction. I plan to extend the domain model to include neo4j nodes and relationships along the lines of your examples and extend the DAOs to include management of the embedded Neo4j instance. My prototype is a Spring app and currently uses aspect oriented transaction management, but in order to manage the transactions on the two databases, I'll probably have to handle transactions programmatically, probably wrapping one in the other. It would be nice to combine the transaction management somehow. In terms of data migration, although I'm trying to minimize migration right now, I have thought about how I might migrate if I were to use Neo4j exclusively. I'm still not sure what the best approaches are to properties vs nodes for entity attributes, indexes (lucene) vs type subnodes, etc. One of the hurdles to migrating to Neo4j is the lack of tool support, particularly for general access. This has been acknowledged by Emil and others, so I'm not complaining. But with our current app, if we need to import data or fix general data problems, we can use a SQL workbench to directly access the database rather than write a special capability in the app. That's not always the best approach but it does allow other developers more familiar with other technologies the ability to access the DB with their tools/language due to the prevalence of SQL. With Neo4j it seems that programmatic access will be required. Granted, I haven't explored the REST server, SPARQL support, or the shell very much. They may offer more generalized access. I'm interested to hear how others are approaching the polyglot-persistence task. As I move forward, I'll share what I learn or have problems with. Thanks, Kalin On Dec 1, 2010, at 10:52 AM, Andreas Kollegger wrote: Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? We're starting to think about best practices for integration: * Hybrid data-modeling: what goes where? * XA transactions * message queues for data distribution * data migration strategies Any problems or feature-requests related to living in a multi-storage-platform world are welcome. Cheers, Andreas ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
