[jira] [Created] (AIRFLOW-6601) Define Neo4j database connection type
Timothy Findlay created AIRFLOW-6601: Summary: Define Neo4j database connection type Key: AIRFLOW-6601 URL: https://issues.apache.org/jira/browse/AIRFLOW-6601 Project: Apache Airflow Issue Type: Sub-task Components: configuration Affects Versions: master Reporter: Timothy Findlay Done when: * Airflow UI allows a Neo4j type of connection which ** Supports protocol type `BOLT` and `BOLT+ROUTING` ** Dose not require `schema` ** Assembles an auth tuple for username+password -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6600) Make an operator to use the neo4j hook
Timothy Findlay created AIRFLOW-6600: Summary: Make an operator to use the neo4j hook Key: AIRFLOW-6600 URL: https://issues.apache.org/jira/browse/AIRFLOW-6600 Project: Apache Airflow Issue Type: Sub-task Components: operators Affects Versions: master Reporter: Timothy Findlay Done when: * DAG Task can ** Successfully execute a query with no result ** Successfully execute a query with results and save to CSV file ** Successfully execute a DML query (eg. Create node) ** Fail a task with no results gracefully when configured to do so ** Fail a task when connetion fails ** Fail a task when there is insufficent disk space -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6599) Make a hook for neo4j
Timothy Findlay created AIRFLOW-6599: Summary: Make a hook for neo4j Key: AIRFLOW-6599 URL: https://issues.apache.org/jira/browse/AIRFLOW-6599 Project: Apache Airflow Issue Type: Sub-task Components: hooks Affects Versions: master Reporter: Timothy Findlay Done when: * The hook can use the Python Neo4j driver to ** Read connection parameters from Airflow ** Execute a cypher query ** Save the results to a CSV file -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6598) DAG task queries a neo4j compatiable graph database
Timothy Findlay created AIRFLOW-6598: Summary: DAG task queries a neo4j compatiable graph database Key: AIRFLOW-6598 URL: https://issues.apache.org/jira/browse/AIRFLOW-6598 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Affects Versions: master Reporter: Timothy Findlay h3. Story *As* a developer *When* I create a DAG task *I want* to query a neo4j compatiable graph database *So that* I can use the results h3. Assumptions * DAG Task can specify output type * Processing the results of the query is up to the developer h3. In Scope * Ability to export results to CSV * Support for DML queries. * Resulting row count validation * Target space validation * Support for handling empty query responses * Bolt and Bolt+routing protocol * Correct failure handling (Connection, Query failure) h3. Out of Scope * No other output format (other than CSV) supported. eg. Parquet, Avro, JSON * Ability to use the bulk export API * Ability to use the bulk import API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay closed AIRFLOW-5920. Resolution: Won't Do > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business > UPDATE: > Feedback through the development effort has highlighted some considerations: > * AIP-21 and any other future plans for the Apache Airflow project need to > be taken into consideration > * A complete featrue set should be implemented considering all use cases: > ** Export to common formats > ** Importing data from common formats > ** DML queries to manipulate the graph > * A suitable connection definition should be implmented for Neo4J (not JDBC) > ** Connection should support bolt and bolt+routing protocols > ** Connection pooling should be managed by the hook to ensure correct > transation bounaries within the session > * A detailed failure mode analysis should also be submitted including: > ** Failed connection handling > ** Query execution handling > ** Transaction handling for multiple queries > ** Data integrity checking (import/export validation) > * Considerations to ensure export does not fill the local disk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008881#comment-17008881 ] Timothy Findlay commented on AIRFLOW-5920: -- This feature deserves more thought than passing interest. At a minimum the Neo4j organisation should be engaged to approve use of the trademark. They should also provide resources to support development and mainteance of this functionality within Apache Airflow. The re-work at this stage is not worth pursuing. A fresh fork/PR can be started by interested parties if there is sufficient demand. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business > UPDATE: > Feedback through the development effort has highlighted some considerations: > * AIP-21 and any other future plans for the Apache Airflow project need to > be taken into consideration > * A complete featrue set should be implemented considering all use cases: > ** Export to common formats > ** Importing data from common formats > ** DML queries to manipulate the graph > * A suitable connection definition should be implmented for Neo4J (not JDBC) > ** Connection should support bolt and bolt+routing protocols > ** Connection pooling should be managed by the hook to ensure correct > transation bounaries within the session > * A detailed failure mode analysis should also be submitted including: > ** Failed connection handling > ** Query execution handling > ** Transaction handling for multiple queries > ** Data integrity checking (import/export validation) > * Considerations to ensure export does not fill the local disk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay updated AIRFLOW-5920: - Description: As a DAG developer I want to create DAG tasks to execute OpenCypher queries against a graph database So that the output can be used elsewhere in a DAG / business UPDATE: Feedback through the development effort has highlighted some considerations: * AIP-21 and any other future plans for the Apache Airflow project need to be taken into consideration * A complete featrue set should be implemented considering all use cases: ** Export to common formats ** Importing data from common formats ** DML queries to manipulate the graph * A suitable connection definition should be implmented for Neo4J (not JDBC) ** Connection should support bolt and bolt+routing protocols ** Connection pooling should be managed by the hook to ensure correct transation bounaries within the session * A detailed failure mode analysis should also be submitted including: ** Failed connection handling ** Query execution handling ** Transaction handling for multiple queries ** Data integrity checking (import/export validation) * Considerations to ensure export does not fill the local disk was: As a DAG developer I want to create DAG tasks to execute OpenCypher queries against a graph database So that the output can be used elsewhere in a DAG / business UPDATE: Feedback through the development effort has highlighted some considerations: * AIP-21 and any other future plans for the Apache Airflow project need to be taken into consideration * A complete featrue set should be implemented considering other use cases (export to different formats & importing data into a graph database) * > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business > UPDATE: > Feedback through the development effort has highlighted some considerations: > * AIP-21 and any other future plans for the Apache Airflow project need to > be taken into consideration > * A complete featrue set should be implemented considering all use cases: > ** Export to common formats > ** Importing data from common formats > ** DML queries to manipulate the graph > * A suitable connection definition should be implmented for Neo4J (not JDBC) > ** Connection should support bolt and bolt+routing protocols > ** Connection pooling should be managed by the hook to ensure correct > transation bounaries within the session > * A detailed failure mode analysis should also be submitted including: > ** Failed connection handling > ** Query execution handling > ** Transaction handling for multiple queries > ** Data integrity checking (import/export validation) > * Considerations to ensure export does not fill the local disk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[
https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008851#comment-17008851
]
Timothy Findlay commented on AIRFLOW-5920:
--
Additionally:
Currently this operator supports only CSV but I think it would be better if it
would have param for {{export_format}} configurable by the user so it would be
easier to add additional formats in the future.
Also like any CSV export you need to allow {{field_delimiter}},
{{print_header}} so users can configure how the csv output would look like.
> Add support to execute OpenCypher query against Neo4j
> -
>
> Key: AIRFLOW-5920
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5920
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks, operators
>Affects Versions: 1.10.7
>Reporter: Timothy Findlay
>Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> As a DAG developer
> I want to create DAG tasks to execute OpenCypher queries against a graph
> database
> So that the output can be used elsewhere in a DAG / business
> UPDATE:
> Feedback through the development effort has highlighted some considerations:
> * AIP-21 and any other future plans for the Apache Airflow project need to
> be taken into consideration
> * A complete featrue set should be implemented considering other use cases
> (export to different formats & importing data into a graph database)
> *
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[
https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008850#comment-17008850
]
Timothy Findlay commented on AIRFLOW-5920:
--
Some feedback includes:
The operator doesn't perform action on Neo4J (like running query, creating
table etc..) but move data from Neo4J to CSV. Usually such operators have
naming convention of {{x To y}} Maybe {{Neo4JOperatorDownloadOperator}} ?
> Add support to execute OpenCypher query against Neo4j
> -
>
> Key: AIRFLOW-5920
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5920
> Project: Apache Airflow
> Issue Type: New Feature
> Components: hooks, operators
>Affects Versions: 1.10.7
>Reporter: Timothy Findlay
>Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> As a DAG developer
> I want to create DAG tasks to execute OpenCypher queries against a graph
> database
> So that the output can be used elsewhere in a DAG / business
> UPDATE:
> Feedback through the development effort has highlighted some considerations:
> * AIP-21 and any other future plans for the Apache Airflow project need to
> be taken into consideration
> * A complete featrue set should be implemented considering other use cases
> (export to different formats & importing data into a graph database)
> *
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay updated AIRFLOW-5920: - Description: As a DAG developer I want to create DAG tasks to execute OpenCypher queries against a graph database So that the output can be used elsewhere in a DAG / business UPDATE: Feedback through the development effort has highlighted some considerations: * AIP-21 and any other future plans for the Apache Airflow project need to be taken into consideration * A complete featrue set should be implemented considering other use cases (export to different formats & importing data into a graph database) * was: As a DAG developer I want to create DAG tasks to execute OpenCypher queries against a graph database So that the output can be used elsewhere in a DAG / business > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business > UPDATE: > Feedback through the development effort has highlighted some considerations: > * AIP-21 and any other future plans for the Apache Airflow project need to > be taken into consideration > * A complete featrue set should be implemented considering other use cases > (export to different formats & importing data into a graph database) > * -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989413#comment-16989413 ] Timothy Findlay commented on AIRFLOW-5920: -- Unfortunately I am not going to have the time to keep working on this. The level of effort is simply too high for me. If there is no one else that is interested in this feature I'm happy for the community to drop it. I think most graph database users have found/made alternatives alread anyway. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay updated AIRFLOW-5920: - Priority: Major (was: Minor) > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Major > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay reassigned AIRFLOW-5920: Assignee: (was: Timothy Findlay) > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987540#comment-16987540 ] Timothy Findlay commented on AIRFLOW-5920: -- Thanks [~jackjack10] . I have removed "draft" from the title. The build is green, and I have updated with feedback from others. AIP-21 was mentioned, but its not clear what I can change/implement now to support that. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983132#comment-16983132 ] Timothy Findlay commented on AIRFLOW-5920: -- [~jackjack10], - I have made a few more updates, the draft PR is now green for the build. Can you cast your eye over this, if it looks ok, I'll make this a formal PR an look for it to be merged in. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979000#comment-16979000 ] Timothy Findlay commented on AIRFLOW-5920: -- [~jackjack10] - Please see draft PR #6604 I've included documentation (and example dag), tests and the hook + operator. I welcome any feedback you might have. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973014#comment-16973014 ] Timothy Findlay commented on AIRFLOW-5920: -- I've drafted something that works, but need to think more about what sort of options developers will want for the output. First draft only outputs to CSV, this can link into an email operator or S3 file operation or whatever else people want. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Findlay reassigned AIRFLOW-5920: Assignee: Timothy Findlay > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
[ https://issues.apache.org/jira/browse/AIRFLOW-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-5920 started by Timothy Findlay. > Add support to execute OpenCypher query against Neo4j > - > > Key: AIRFLOW-5920 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks, operators >Affects Versions: 1.10.7 >Reporter: Timothy Findlay >Assignee: Timothy Findlay >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > As a DAG developer > I want to create DAG tasks to execute OpenCypher queries against a graph > database > So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5920) Add support to execute OpenCypher query against Neo4j
Timothy Findlay created AIRFLOW-5920: Summary: Add support to execute OpenCypher query against Neo4j Key: AIRFLOW-5920 URL: https://issues.apache.org/jira/browse/AIRFLOW-5920 Project: Apache Airflow Issue Type: New Feature Components: hooks, operators Affects Versions: 1.10.7 Reporter: Timothy Findlay As a DAG developer I want to create DAG tasks to execute OpenCypher queries against a graph database So that the output can be used elsewhere in a DAG / business -- This message was sent by Atlassian Jira (v8.3.4#803005)
