Re: [Neo4j] REST batch support - transaction support for java rest client?
yes, i'll see if i get time to go down that path. it's mostly to keep the programmer happy and productive, not a 100% requirement. On Thu, Jul 7, 2011 at 9:58 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Right those are some of the issues. > > So one way would be to specify a tx timeout upfront which automatically > rolls back the tx (you can just add a kind of timer/TTL to the tx-session > object) and clears it as well. > > Keeping state on the server is always a problem but I don't see a different > solution for that. But it might worth a try especially if it helps you with > your concrete scenario. > > Michael > > > > Am 07.07.2011 um 15:30 schrieb Patrik Sundberg: > > > good idea. i'll ponder it for a bit. > > > > but yes, we clearly need to keep state around, so for REST it'd be > carried > > around in session. but on server side I guess you have issues with never > > ending transactions, how to cull them, etc. since it's a stateless > > req/response comm channel. on a permanent channel it's easy to detect > > disconnect and clean up, over http not as easy. > > > > thanks > > > > > > On Thu, Jul 7, 2011 at 12:42 PM, Michael Hunger < > > michael.hun...@neotechnology.com> wrote: > > > >> But then it would be possible to write a RequestFilter for the > Neo4j-Server > >> that does start and commit/rollbacks transactions. > >> > >> I.e. you create a tx object and put it in the session-context if there > is > >> none and return a tx-token that the filter uses (e.g. as header-field). > >> then later you can pull it out again and attach it to the current thread > >> (that's the tricky part). > >> On commit or rollback you just do that with the tx (after attaching it > to > >> the thread). > >> > >> As the RestfulGraphDb and the Filter share the same execution thread > this > >> could/should work. > >> > >> I wouldn't want to support that in the neo4j server by default as this > >> creates a lot of server-side state that has to be managed. > >> > >> But if it works out one could publish that as server-extension. > >> > >> HTH > >> > >> Michael > >> > >> Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: > >> > >>> Following up on the topic of transactions for client API. > >>> > >>> What is the current plan for some sort of client side API supporting > >>> transactions? > >>> > >>> I'm playing around with some ideas here and the lack of transaction > >> support > >>> in the client API is problematic. I know there's BATCH support in the > >> REST > >>> API which effectively is a transaction, but it doesn't always suit. For > >>> example I have the following steps that I'd like to accomplish: > >>> - create a reference node > >>> - check if a node with a given domain id exist in an index, if it does, > >> fail > >>> - create an entity node for the given domain id > >>> - add entity node to the index > >>> - attach entity node to ref node > >>> - create a node representing a specific version of the entity node > >>> - attach the version node to the entity node, with some properties on > the > >>> relationships signifying valid time > >>> > >>> That should all be considered an atomic operation, all or nothing. > Doing > >> it > >>> step by step is very easy and natural with REST API, but trying to roll > >> back > >>> on error is flaky. > >>> > >>> I think could batch it, but from a programming style it becomes pretty > >>> unnatural. Same thing with a plugin for doing the steps. The natural > flow > >> of > >>> code client side gets distorted by having to collect a lot of data > >> upfront > >>> and then provide all that data to a method call. It's doable, just > >> doesn't > >>> seem ideal. > >>> > >>> Using an embedded db, exposing as some sort of service etc is also > >> doable, > >>> it's just that my domain is graph related and I'm pretty happy with > just > >> the > >>> "primitives" and using a remote server (if I could have transactions). > >>> Number of clients are quite a few and need to share their data + don't > >> all > >>> run all the time so can't make the client API the embedded api. > >>> > >>> I'd think it's not an uncommon situation and many people wishing for a > >>> support for natural client side transaction API (similar to embedded > >> api). > >>> > >>> Patrik > >>> > >>> > >>> On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg > >>> wrote: > >>> > yeah, harder problem than my first hunch. > > sounds like plugins is the way to go for now, hopefully introduction > of > non-rest protocol with same interface as embedded API in 1.5 will > >> simplify > things in the future. > > thanks > > > On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < > michael.hun...@neotechnology.com> wrote: > > > Patrick, > > > > I've already thought long and hard about that. > > > > The problem is you can't implement that transparently as you can > never > > allow code in a second call rely on data derived from a previo
Re: [Neo4j] REST batch support - transaction support for java rest client?
Right those are some of the issues. So one way would be to specify a tx timeout upfront which automatically rolls back the tx (you can just add a kind of timer/TTL to the tx-session object) and clears it as well. Keeping state on the server is always a problem but I don't see a different solution for that. But it might worth a try especially if it helps you with your concrete scenario. Michael Am 07.07.2011 um 15:30 schrieb Patrik Sundberg: > good idea. i'll ponder it for a bit. > > but yes, we clearly need to keep state around, so for REST it'd be carried > around in session. but on server side I guess you have issues with never > ending transactions, how to cull them, etc. since it's a stateless > req/response comm channel. on a permanent channel it's easy to detect > disconnect and clean up, over http not as easy. > > thanks > > > On Thu, Jul 7, 2011 at 12:42 PM, Michael Hunger < > michael.hun...@neotechnology.com> wrote: > >> But then it would be possible to write a RequestFilter for the Neo4j-Server >> that does start and commit/rollbacks transactions. >> >> I.e. you create a tx object and put it in the session-context if there is >> none and return a tx-token that the filter uses (e.g. as header-field). >> then later you can pull it out again and attach it to the current thread >> (that's the tricky part). >> On commit or rollback you just do that with the tx (after attaching it to >> the thread). >> >> As the RestfulGraphDb and the Filter share the same execution thread this >> could/should work. >> >> I wouldn't want to support that in the neo4j server by default as this >> creates a lot of server-side state that has to be managed. >> >> But if it works out one could publish that as server-extension. >> >> HTH >> >> Michael >> >> Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: >> >>> Following up on the topic of transactions for client API. >>> >>> What is the current plan for some sort of client side API supporting >>> transactions? >>> >>> I'm playing around with some ideas here and the lack of transaction >> support >>> in the client API is problematic. I know there's BATCH support in the >> REST >>> API which effectively is a transaction, but it doesn't always suit. For >>> example I have the following steps that I'd like to accomplish: >>> - create a reference node >>> - check if a node with a given domain id exist in an index, if it does, >> fail >>> - create an entity node for the given domain id >>> - add entity node to the index >>> - attach entity node to ref node >>> - create a node representing a specific version of the entity node >>> - attach the version node to the entity node, with some properties on the >>> relationships signifying valid time >>> >>> That should all be considered an atomic operation, all or nothing. Doing >> it >>> step by step is very easy and natural with REST API, but trying to roll >> back >>> on error is flaky. >>> >>> I think could batch it, but from a programming style it becomes pretty >>> unnatural. Same thing with a plugin for doing the steps. The natural flow >> of >>> code client side gets distorted by having to collect a lot of data >> upfront >>> and then provide all that data to a method call. It's doable, just >> doesn't >>> seem ideal. >>> >>> Using an embedded db, exposing as some sort of service etc is also >> doable, >>> it's just that my domain is graph related and I'm pretty happy with just >> the >>> "primitives" and using a remote server (if I could have transactions). >>> Number of clients are quite a few and need to share their data + don't >> all >>> run all the time so can't make the client API the embedded api. >>> >>> I'd think it's not an uncommon situation and many people wishing for a >>> support for natural client side transaction API (similar to embedded >> api). >>> >>> Patrik >>> >>> >>> On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg >>> wrote: >>> yeah, harder problem than my first hunch. sounds like plugins is the way to go for now, hopefully introduction of non-rest protocol with same interface as embedded API in 1.5 will >> simplify things in the future. thanks On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Patrick, > > I've already thought long and hard about that. > > The problem is you can't implement that transparently as you can never > allow code in a second call rely on data derived from a previous one. > > The simplest form that I came up with is a "BatchCommand" that gets an >> API > interface injected that allows requests but doesn't return data. > > The execution of this Batch command would then return a "BatchResult" >> with > all the data acquired during the batch operation. > > Another way would be to inject the normal GraphDatabaseService >> interface, > record the invocations in a first phase and then
Re: [Neo4j] REST batch support - transaction support for java rest client?
good idea. i'll ponder it for a bit. but yes, we clearly need to keep state around, so for REST it'd be carried around in session. but on server side I guess you have issues with never ending transactions, how to cull them, etc. since it's a stateless req/response comm channel. on a permanent channel it's easy to detect disconnect and clean up, over http not as easy. thanks On Thu, Jul 7, 2011 at 12:42 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > But then it would be possible to write a RequestFilter for the Neo4j-Server > that does start and commit/rollbacks transactions. > > I.e. you create a tx object and put it in the session-context if there is > none and return a tx-token that the filter uses (e.g. as header-field). > then later you can pull it out again and attach it to the current thread > (that's the tricky part). > On commit or rollback you just do that with the tx (after attaching it to > the thread). > > As the RestfulGraphDb and the Filter share the same execution thread this > could/should work. > > I wouldn't want to support that in the neo4j server by default as this > creates a lot of server-side state that has to be managed. > > But if it works out one could publish that as server-extension. > > HTH > > Michael > > Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: > > > Following up on the topic of transactions for client API. > > > > What is the current plan for some sort of client side API supporting > > transactions? > > > > I'm playing around with some ideas here and the lack of transaction > support > > in the client API is problematic. I know there's BATCH support in the > REST > > API which effectively is a transaction, but it doesn't always suit. For > > example I have the following steps that I'd like to accomplish: > > - create a reference node > > - check if a node with a given domain id exist in an index, if it does, > fail > > - create an entity node for the given domain id > > - add entity node to the index > > - attach entity node to ref node > > - create a node representing a specific version of the entity node > > - attach the version node to the entity node, with some properties on the > > relationships signifying valid time > > > > That should all be considered an atomic operation, all or nothing. Doing > it > > step by step is very easy and natural with REST API, but trying to roll > back > > on error is flaky. > > > > I think could batch it, but from a programming style it becomes pretty > > unnatural. Same thing with a plugin for doing the steps. The natural flow > of > > code client side gets distorted by having to collect a lot of data > upfront > > and then provide all that data to a method call. It's doable, just > doesn't > > seem ideal. > > > > Using an embedded db, exposing as some sort of service etc is also > doable, > > it's just that my domain is graph related and I'm pretty happy with just > the > > "primitives" and using a remote server (if I could have transactions). > > Number of clients are quite a few and need to share their data + don't > all > > run all the time so can't make the client API the embedded api. > > > > I'd think it's not an uncommon situation and many people wishing for a > > support for natural client side transaction API (similar to embedded > api). > > > > Patrik > > > > > > On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg > > wrote: > > > >> yeah, harder problem than my first hunch. > >> > >> sounds like plugins is the way to go for now, hopefully introduction of > >> non-rest protocol with same interface as embedded API in 1.5 will > simplify > >> things in the future. > >> > >> thanks > >> > >> > >> On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < > >> michael.hun...@neotechnology.com> wrote: > >> > >>> Patrick, > >>> > >>> I've already thought long and hard about that. > >>> > >>> The problem is you can't implement that transparently as you can never > >>> allow code in a second call rely on data derived from a previous one. > >>> > >>> The simplest form that I came up with is a "BatchCommand" that gets an > API > >>> interface injected that allows requests but doesn't return data. > >>> > >>> The execution of this Batch command would then return a "BatchResult" > with > >>> all the data acquired during the batch operation. > >>> > >>> Another way would be to inject the normal GraphDatabaseService > interface, > >>> record the invocations in a first phase and then execute the batch > command > >>> again (this time ignoring the inputs but then returning the results) > but > >>> this is bad from a usability perspective. > >>> > >>> One critical issue is the creation of relationships as they depend on > the > >>> correct node-ids of previously created nodes. Jacob already thought > about > >>> some means of referring to previous output data but I think kept away > from > >>> that as we didn't want to make this batch-interface a turing complete > >>> language. > >>> > >>> So you see, it's not that simple. > >
Re: [Neo4j] REST batch support - transaction support for java rest client?
But then it would be possible to write a RequestFilter for the Neo4j-Server that does start and commit/rollbacks transactions. I.e. you create a tx object and put it in the session-context if there is none and return a tx-token that the filter uses (e.g. as header-field). then later you can pull it out again and attach it to the current thread (that's the tricky part). On commit or rollback you just do that with the tx (after attaching it to the thread). As the RestfulGraphDb and the Filter share the same execution thread this could/should work. I wouldn't want to support that in the neo4j server by default as this creates a lot of server-side state that has to be managed. But if it works out one could publish that as server-extension. HTH Michael Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: > Following up on the topic of transactions for client API. > > What is the current plan for some sort of client side API supporting > transactions? > > I'm playing around with some ideas here and the lack of transaction support > in the client API is problematic. I know there's BATCH support in the REST > API which effectively is a transaction, but it doesn't always suit. For > example I have the following steps that I'd like to accomplish: > - create a reference node > - check if a node with a given domain id exist in an index, if it does, fail > - create an entity node for the given domain id > - add entity node to the index > - attach entity node to ref node > - create a node representing a specific version of the entity node > - attach the version node to the entity node, with some properties on the > relationships signifying valid time > > That should all be considered an atomic operation, all or nothing. Doing it > step by step is very easy and natural with REST API, but trying to roll back > on error is flaky. > > I think could batch it, but from a programming style it becomes pretty > unnatural. Same thing with a plugin for doing the steps. The natural flow of > code client side gets distorted by having to collect a lot of data upfront > and then provide all that data to a method call. It's doable, just doesn't > seem ideal. > > Using an embedded db, exposing as some sort of service etc is also doable, > it's just that my domain is graph related and I'm pretty happy with just the > "primitives" and using a remote server (if I could have transactions). > Number of clients are quite a few and need to share their data + don't all > run all the time so can't make the client API the embedded api. > > I'd think it's not an uncommon situation and many people wishing for a > support for natural client side transaction API (similar to embedded api). > > Patrik > > > On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg > wrote: > >> yeah, harder problem than my first hunch. >> >> sounds like plugins is the way to go for now, hopefully introduction of >> non-rest protocol with same interface as embedded API in 1.5 will simplify >> things in the future. >> >> thanks >> >> >> On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < >> michael.hun...@neotechnology.com> wrote: >> >>> Patrick, >>> >>> I've already thought long and hard about that. >>> >>> The problem is you can't implement that transparently as you can never >>> allow code in a second call rely on data derived from a previous one. >>> >>> The simplest form that I came up with is a "BatchCommand" that gets an API >>> interface injected that allows requests but doesn't return data. >>> >>> The execution of this Batch command would then return a "BatchResult" with >>> all the data acquired during the batch operation. >>> >>> Another way would be to inject the normal GraphDatabaseService interface, >>> record the invocations in a first phase and then execute the batch command >>> again (this time ignoring the inputs but then returning the results) but >>> this is bad from a usability perspective. >>> >>> One critical issue is the creation of relationships as they depend on the >>> correct node-ids of previously created nodes. Jacob already thought about >>> some means of referring to previous output data but I think kept away from >>> that as we didn't want to make this batch-interface a turing complete >>> language. >>> >>> So you see, it's not that simple. >>> >>> Michael >>> >>> Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: >>> Hi, Since there is now possible to send off batches of operations via the >>> REST interface, I was wondering if anyone has started to look at implementing transactions in the java REST client ( https://github.com/jexp/neo4j-java-rest-binding) ? It would seem possible, but I can also see it could involve some major reorganizing of the internals of the client to make everything aware of transactions and submit via batch command. Patrik ___ Neo4j mailing list User@lists.neo4j.org ht
Re: [Neo4j] REST batch support - transaction support for java rest client?
Following up on the topic of transactions for client API. What is the current plan for some sort of client side API supporting transactions? I'm playing around with some ideas here and the lack of transaction support in the client API is problematic. I know there's BATCH support in the REST API which effectively is a transaction, but it doesn't always suit. For example I have the following steps that I'd like to accomplish: - create a reference node - check if a node with a given domain id exist in an index, if it does, fail - create an entity node for the given domain id - add entity node to the index - attach entity node to ref node - create a node representing a specific version of the entity node - attach the version node to the entity node, with some properties on the relationships signifying valid time That should all be considered an atomic operation, all or nothing. Doing it step by step is very easy and natural with REST API, but trying to roll back on error is flaky. I think could batch it, but from a programming style it becomes pretty unnatural. Same thing with a plugin for doing the steps. The natural flow of code client side gets distorted by having to collect a lot of data upfront and then provide all that data to a method call. It's doable, just doesn't seem ideal. Using an embedded db, exposing as some sort of service etc is also doable, it's just that my domain is graph related and I'm pretty happy with just the "primitives" and using a remote server (if I could have transactions). Number of clients are quite a few and need to share their data + don't all run all the time so can't make the client API the embedded api. I'd think it's not an uncommon situation and many people wishing for a support for natural client side transaction API (similar to embedded api). Patrik On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg wrote: > yeah, harder problem than my first hunch. > > sounds like plugins is the way to go for now, hopefully introduction of > non-rest protocol with same interface as embedded API in 1.5 will simplify > things in the future. > > thanks > > > On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < > michael.hun...@neotechnology.com> wrote: > >> Patrick, >> >> I've already thought long and hard about that. >> >> The problem is you can't implement that transparently as you can never >> allow code in a second call rely on data derived from a previous one. >> >> The simplest form that I came up with is a "BatchCommand" that gets an API >> interface injected that allows requests but doesn't return data. >> >> The execution of this Batch command would then return a "BatchResult" with >> all the data acquired during the batch operation. >> >> Another way would be to inject the normal GraphDatabaseService interface, >> record the invocations in a first phase and then execute the batch command >> again (this time ignoring the inputs but then returning the results) but >> this is bad from a usability perspective. >> >> One critical issue is the creation of relationships as they depend on the >> correct node-ids of previously created nodes. Jacob already thought about >> some means of referring to previous output data but I think kept away from >> that as we didn't want to make this batch-interface a turing complete >> language. >> >> So you see, it's not that simple. >> >> Michael >> >> Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: >> >> > Hi, >> > >> > Since there is now possible to send off batches of operations via the >> REST >> > interface, I was wondering if anyone has started to look at implementing >> > transactions in the java REST client ( >> > https://github.com/jexp/neo4j-java-rest-binding) ? >> > >> > It would seem possible, but I can also see it could involve some major >> > reorganizing of the internals of the client to make everything aware of >> > transactions and submit via batch command. >> > >> > Patrik >> > ___ >> > Neo4j mailing list >> > User@lists.neo4j.org >> > https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] REST batch support - transaction support for java rest client?
yeah, harder problem than my first hunch. sounds like plugins is the way to go for now, hopefully introduction of non-rest protocol with same interface as embedded API in 1.5 will simplify things in the future. thanks On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Patrick, > > I've already thought long and hard about that. > > The problem is you can't implement that transparently as you can never > allow code in a second call rely on data derived from a previous one. > > The simplest form that I came up with is a "BatchCommand" that gets an API > interface injected that allows requests but doesn't return data. > > The execution of this Batch command would then return a "BatchResult" with > all the data acquired during the batch operation. > > Another way would be to inject the normal GraphDatabaseService interface, > record the invocations in a first phase and then execute the batch command > again (this time ignoring the inputs but then returning the results) but > this is bad from a usability perspective. > > One critical issue is the creation of relationships as they depend on the > correct node-ids of previously created nodes. Jacob already thought about > some means of referring to previous output data but I think kept away from > that as we didn't want to make this batch-interface a turing complete > language. > > So you see, it's not that simple. > > Michael > > Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: > > > Hi, > > > > Since there is now possible to send off batches of operations via the > REST > > interface, I was wondering if anyone has started to look at implementing > > transactions in the java REST client ( > > https://github.com/jexp/neo4j-java-rest-binding) ? > > > > It would seem possible, but I can also see it could involve some major > > reorganizing of the internals of the client to make everything aware of > > transactions and submit via batch command. > > > > Patrik > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] REST batch support - transaction support for java rest client?
Patrick, I've already thought long and hard about that. The problem is you can't implement that transparently as you can never allow code in a second call rely on data derived from a previous one. The simplest form that I came up with is a "BatchCommand" that gets an API interface injected that allows requests but doesn't return data. The execution of this Batch command would then return a "BatchResult" with all the data acquired during the batch operation. Another way would be to inject the normal GraphDatabaseService interface, record the invocations in a first phase and then execute the batch command again (this time ignoring the inputs but then returning the results) but this is bad from a usability perspective. One critical issue is the creation of relationships as they depend on the correct node-ids of previously created nodes. Jacob already thought about some means of referring to previous output data but I think kept away from that as we didn't want to make this batch-interface a turing complete language. So you see, it's not that simple. Michael Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: > Hi, > > Since there is now possible to send off batches of operations via the REST > interface, I was wondering if anyone has started to look at implementing > transactions in the java REST client ( > https://github.com/jexp/neo4j-java-rest-binding) ? > > It would seem possible, but I can also see it could involve some major > reorganizing of the internals of the client to make everything aware of > transactions and submit via batch command. > > Patrik > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
