[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092850#comment-16092850 ] Dougal Watt commented on ATLAS-1757: The state of integration for graph DB's is somewhat in flux, with Tinkerpop being more oriented towards analytics use cases than straight out integration. For the latter, GraphQL is proving to be a very nice solution (graphql.org). It's a query language and information-centric API rolled into one, and both the query syntax and outputs are structured in the same way. Nicely, the query return is in JSON format, so everything on the upstream side of the API can be totally standard internet technologies. The architecture of GraphQL also allows for integration of any back end data store so you can have more than one, pluggable databases behind the scenes. I've started two companies building SaaS products on top of graph DB's, and choose Blazegraph due to it's support for RDF and GPU acceleration. An RDF graph would allow Atlas to define the required metamodel in RDF and use it to reason over instance data received and send by Atlas between different data lakes - you'd also get metamodel validation and conformance, and a highly evolvable schema. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092891#comment-16092891 ] David Radley commented on ATLAS-1757: - [~dougal_watt] Interesting - I do not know anything about this project. I am wondering about the license around this facebook open source. Apache has recently said no to react.js https://www.theregister.co.uk/2017/07/17/apache_says_no_to_facebook_code_libraries/ . I like Tinkerpop, as it is an open standard - I am not sure how open GraphQL is; is this something that Facebook would be willing to contribute to the Atlas code base for the Atlas community to assess? The Atlas DSL and the new facetted search do provide structured queries (json in json out) . I agree RDF is very powerful - though RDF and Owl may not be for the faint hearted; so excludes (many) people who may not be willing to embrace the steep learning curve involved. We are also looking at GaianDB as part of the VDC project. This is an open source project that exposes heterogeneous data sources over SQL. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093033#comment-16093033 ] Nigel Jones commented on ATLAS-1757: Interesting info Dougal :-) ... on the license Apache specifically have an issue with the revocation option... I found Simon Phipps blog informative, but clearly it's the apache pmcs, and apache legal that would have to make any call on what's allowed/not-allowed. for now though react is a problem. More at https://meshedinsights.com/2017/07/16/apache-bans-facebooks-license-combo/ . As an aside a quick glance at the graphQL java libraries suggest they are MIT licensed so not an issue. I'm not an expert on graph, or graph libraries... from my uninformed (!) perspective Tinkerpop has significant attraction in being an open apache project, with meritocratic governance - I'm not sure that graphQL is in that same space? > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093035#comment-16093035 ] Dougal Watt commented on ATLAS-1757: Interesting point David. GraphQL isn't in the FB open source projects library, but is published as a specification. There are many reference implementations that use the spec. My company will likely use a Java Client (here's one under MIT license: https://github.com/graphql-java/graphql-java), and build the server we need for Blazegraph (under the covers it will parse the query and convert to SPARQL). So as I read it, the spec is just the spec, it's up to the end user to create the client and server implementations of the spec and license them as they want. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093054#comment-16093054 ] Dougal Watt commented on ATLAS-1757: Hi Nigel, good to see you online again! Tinkerpop is a great system, especially in how it abstracts the db and sits across OLAP and OLTP use cases, and it's strong governance, which GraphQL lacks. There are sometimes wrinkles in how graph db's support it though e.g. Blazegraph doesn't enable it for full HA clustered configurations. The really nice thing with GraphQL is you get back JSON so it's very accessible for developers, and the type system, mutations, introspection and validation are all very easy to use, and directly resemble the query system so it's very easy to understand. If all you need to do is support transactional processing (and my companies are using it for a whole range of businesses) then it's likely easier to use from a development perspective than Tinkerpop. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206328#comment-16206328 ] Andrew Hulbert commented on ATLAS-1757: --- Is there anything on the roadmap for the future to support visibility filtering with Atlas + JanusGraph + HBase? For example, it would be interesting to have the ability to access control specific vertices, attributes, or relationships based on visibilities which systems like bigtable systems like HBase and Accumulo can support. I'd be interested to see if anyone else desires this and if so I'd probably put a vote in for JanusGraph and contributing to help make that a top level feature in both Atlas and JanusGraph. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206420#comment-16206420 ] Sarath Subramanian commented on ATLAS-1757: --- [~grahamwallis] could you rebase your latest patch and upload it to review board, it fails to apply on master. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206458#comment-16206458 ] Jerry He commented on ATLAS-1757: - [~hulbs] Please see this ticket on JanusGraph https://github.com/JanusGraph/janusgraph/issues/493 > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207263#comment-16207263 ] Graham Wallis commented on ATLAS-1757: -- [~sarath.ku...@gmail.com] Apologies - I have just replaced the v2 patch with a rebase on master. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207273#comment-16207273 ] Graham Wallis commented on ATLAS-1757: -- A quick, general update on this JIRA. The attached proposal doc defined 4 stages: Stage 1: The first stage was to restructure the build files, which was in the v1 patch. This changed the graph profiles and introduced the GRAPH-PROVIDER system variable as a way of selecting graph provider. See the graphdb/readme.txt in the v2 patch for details. Stage 2: The second stage is the introduction of JanusGraph to Atlas. The v2 patch introduces the graphdb/janus module and contains some restructuring of graphdb/common and graphdb/titan0, graphdb/titan1; so that graphdb/common applies to any tinkerpop database (of which titan0, titan1 and janus are all examples). Work is ongoing on stage 2. At this time, the v2 patch has been built clean but not tested. I intend to start running UTs/ITs on it and fixing up problems as I find them. There is additional work to do for stage 2 to update any other Atlas components that are not yet at Tinkerpop3. On a recent Atlas dev call it was decided that the catalog component should be removed - this is one of the components that is back-level at Tinkerpop2. There are also other Atlas components that will need to be updated. There may also be further updates to exploit features of Janus and look at introducing the visibility feature described above (thanks [~hulbs] and [~jerryhe]) or to modify the Atlas SOE in line with Janus dependencies. The desired result of stage 2 is to provide a choice between graphdb/titan0 and graphdb/janus, both of which should be fully operational. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210772#comment-16210772 ] David Radley commented on ATLAS-1757: - [~hulbs] A thought on your visibility suggestion. The way we are thinking about filtering or masking is in an architecture that is being worked on in the Virtual Data Connector (documented in the Atlas Wiki); it aspires to bring these sort of decisions up to the business level. It does this by working with Glossary Terms that are classified. For example a term called National Insurance Number could be classified as a personal sensitive information. Apache Ranger can be setup to run policies and rules against the classifications (it can already do this against classifications applied to assets). Governance policies can then be specified more naturally in a technology agnostic way. The Atlas type system helps in the architecture and lives above the Janus persistence layer. It might be that this Janus visibility feature could fit in this Virtual Data connector architecture; though I am not seeing how at the moment. I am sure it could be useful for more IT orientated use cases. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211131#comment-16211131 ] Andrew Hulbert commented on ATLAS-1757: --- [~davidrad] I'll try to take a look over at the Virtual Data Connector pages and discussions. I think that "field-level" visibility is definitely something that's useful for what I'm working on but the "cell-level" visibility is probably (as you mentioned) not as much on the radar at the moment. But I'll try to chat with he VD connector group and see what they are thinking. We have customers that need to segregate data from different divisions and want that strong guarantee on a per "entity" basis. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211258#comment-16211258 ] David Radley commented on ATLAS-1757: - [~hulbs] The equivalent to the RDB grant is all done through Ranger - which gives us great separation of responsibility and flexibility. I have no reason to think that Apache Ranger mechanism is not strong enough for the use cases we have looked at. If the data really needs to be separate - and does not need to be joined, then it might make sense to have them in separate data lakes - i.e. separate Atlas's. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222553#comment-16222553 ] Graham Wallis commented on ATLAS-1757: -- Attaching ATLAS-1757-v4.patch. Please see RBT for details: https://reviews.apache.org/r/63041 It has been decided that we will only go as far as JanusGraph 0.1.1 for the time being, due to needing extra migration work for JanusGraph 0.2.0, which will happen soon. This patch runs almost clean with JanusGraph 0.1.1, the caveat being two UT failures in repository: * GraphBackedMetadataRepositoryTest.testConcurrentCalls * AtlasRelationshipStoreV1Test.testRelationshipAttributeUpdate_NonComposite_OneToMany > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch, ATLAS-1757-v3.patch, > ATLAS-1757-v4.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1757) Proposal to update graph DB
[ https://issues.apache.org/jira/browse/ATLAS-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16287426#comment-16287426 ] Graham Wallis commented on ATLAS-1757: -- Brief update on graph changes to Atlas - most of the work proposed in this JIRA is now complete. * ATLAS-2264 updated JanusGraph to 0.2.0 [https://issues.apache.org/jira/browse/ATLAS-2264] * ATLAS-2308 removed titan1 [https://issues.apache.org/jira/browse/ATLAS-2308] With these JIRAs, Atlas supports JanusGraph 0.2.0 (tinkerpop3) and Titan 0.5.4 (tinkerpop2). The parts (of ATLAS-1757) that were not completed are removal of the titan0 profile (which will happen at some point in the future under a separate JIRA) and the possible option of building with multiple providers and selecting provider via configuration, (which is less likely to happen). [ATLAS-2270] contains a discussion of choice of persistence and indexing backends. > Proposal to update graph DB > --- > > Key: ATLAS-1757 > URL: https://issues.apache.org/jira/browse/ATLAS-1757 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: trunk >Reporter: Graham Wallis >Assignee: Graham Wallis > Fix For: 1.0.0 > > Attachments: ATLAS-1757 Proposal to change graph database.pdf, > ATLAS-1757-v1.patch, ATLAS-1757-v2.patch, ATLAS-1757-v3.patch, > ATLAS-1757-v4.patch > > > Given the formation of the JanusGraph open source project (under the Linux > Foundation) to continue the development and support of the Titan DB, should > we aim to deprecate Titan and move over to JanusGraph? > If we did this, we could keep the graph abstraction layer and use it to > support Titan 0, Titan 1 and JanusGraph. > Are there other graph databases that we should consider? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
