bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1938503762
Fine to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
T
potiuk commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1938494404
Ok. Thought a bit and I propose, we take it slower (reassigned it to 2.9.0).
I think we have some other - more important issues to solvve for 2.8.*, it was
a bit "bendig" the rules to pu
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1938476594
@potiuk the code assumes >= 2.9.0 now. So no code change is required for
targeting 2.9.0 from my point of view.
--
This is an automated message from the Apache Git Service.
To re
potiuk commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1934427763
> Targeting 2.8.2 will require a code change.
Just chenging >= 2.9.0 right ? That's the only thing to change as far as I
understand @bolkedebruin ?
--
This is an automated messa
potiuk commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1934376102
> @ephraimbuddy @bolkedebruin - I want to promote this PR - just confirming
that it's going out in 2.8.2 for real? I agree with @jedcunningham that this
feels like a new feature to me, b
cmarteepants commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1934245016
I want to promote this PR - just confirming that it's going out in 2.8.2 for
real? I agree with @jedcunningham that this feels like a new feature to me, but
won't object to people
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1931382851
O yes the "expansion bug" ;-). I'm fine either way. Targeting 2.8.2 will
require a code change.
--
This is an automated message from the Apache Git Service.
To respond to the m
jedcunningham commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1930742577
Sorry, that decision was in a collapsed discussion I didn't expand :) It
doesn't "feel" like a bugfix to me, but I'm happy to step back on this.
@ephraimbuddy is the rm ultimately
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1928868964
@jedcunningham it was agreed to have this in 2.8.2 as we consider this a bug
fix more than a feature release. What has changed?
--
This is an automated message from the Apache Gi
jedcunningham commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1928121474
(@bolkedebruin, moving this back to 2.9 as we won't ship this in a core
patch release)
--
This is an automated message from the Apache Git Service.
To respond to the message, pl
bolkedebruin merged PR #37058:
URL: https://github.com/apache/airflow/pull/37058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@air
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1476602650
##
airflow/providers/common/io/xcom/backend.py:
##
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1476602650
##
airflow/providers/common/io/xcom/backend.py:
##
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1476602650
##
airflow/providers/common/io/xcom/backend.py:
##
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1476602650
##
airflow/providers/common/io/xcom/backend.py:
##
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
dirrao commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1473833512
##
airflow/providers/common/io/xcom/backend.py:
##
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472851000
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472726210
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.Bas
uranusjr commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472722713
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.B
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472669140
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472668469
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472667511
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472667511
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
ephraimbuddy commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472594220
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472546549
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
ephraimbuddy commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472535420
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472528714
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.Bas
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472522606
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agree
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472498210
##
airflow/providers/common/io/xcom/__init__.py:
##
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472495236
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472494935
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1472493470
##
docs/apache-airflow/core-concepts/xcoms.rst:
##
@@ -56,6 +56,31 @@ XComs are a relative of :doc:`variables`, with the main
difference being that XC
If t
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1471129414
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
uranusjr commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.B
uranusjr commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.B
uranusjr commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1470830063
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.B
ephraimbuddy commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1470785477
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,32 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
hussein-awala commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1916322884
> @hussein-awala sure I can do that. Is that also what we want? Shouldn't we
bump the requirements of the provider? I understand that this forces people to
upgrade but the functio
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1916308845
@hussein-awala sure I can do that. Is that also what we want? Shouldn't we
bump the requirements of the provider? I understand that this forces people to
upgrade but the functional
hussein-awala commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915398665
> > > If we chose 2 then it makes sense to move it into common.io (with all
the extras that it requires cause it is a 'new' capability for a provider).
> >
> >
> > I thi
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915338997
Ok ill move it to the provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
potiuk commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1915207986
Yeah. My suggestion was bad idea and you pointed out it's not needed indeed.
And yes I thought that the change could actually be split In two : Base XCOM
change for small/big values for
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1914950829
> > If we chose 2 then it makes sense to move it into common.io (with all
the extras that it requires cause it is a 'new' capability for a provider).
>
> I think I''d support
potiuk commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1914597782
> If we chose 2 then it makes sense to move it into common.io (with all the
extras that it requires cause it is a 'new' capability for a provider).
I think I''d support @hussein-aw
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1469303440
##
docs/apache-airflow/core-concepts/xcoms.rst:
##
@@ -56,6 +56,31 @@ XComs are a relative of :doc:`variables`, with the main
difference being that XC
If t
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1469303055
##
airflow/io/xcom.py:
##
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on PR #37058:
URL: https://github.com/apache/airflow/pull/37058#issuecomment-1913716928
> It looks good! I'm just wondering if we should move the introduced XCom
backend and its configurations to the `common.io` provider 🤔
@hussein-awala as mentioned in the topi
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468934584
##
docs/apache-airflow/core-concepts/xcoms.rst:
##
@@ -56,6 +56,31 @@ XComs are a relative of :doc:`variables`, with the main
difference being that XC
If t
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468934439
##
airflow/io/xcom.py:
##
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468934400
##
airflow/io/xcom.py:
##
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
jscheffl commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468932566
##
airflow/io/xcom.py:
##
@@ -0,0 +1,123 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE
hussein-awala commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468932298
##
airflow/models/xcom.py:
##
@@ -641,7 +649,13 @@ def clear(
query = session.query(BaseXCom).filter_by(dag_id=dag_id,
task_id=task_id, run_id=run_id)
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468924474
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468924423
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,21 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917908
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468918879
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,21 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.Bas
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468918789
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917908
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917596
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917399
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOT
bolkedebruin commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468917344
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,21 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xc
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468907283
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468905770
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903437
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903437
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468903325
##
airflow/io/xcom.py:
##
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE fi
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468902862
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,21 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.Bas
potiuk commented on code in PR #37058:
URL: https://github.com/apache/airflow/pull/37058#discussion_r1468901766
##
airflow/config_templates/config.yml:
##
@@ -375,6 +375,21 @@ core:
type: string
example: "path.to.CustomXCom"
default: "airflow.models.xcom.Bas
bolkedebruin opened a new pull request, #37058:
URL: https://github.com/apache/airflow/pull/37058
This adds the possibility to store xcoms on a object storage supported
backend.
This is an initial implementation and I am open for feedback (duh ;-) ). The
BaseXCom is a bit messy and r
69 matches
Mail list logo
