Gah, sorry, I missed the other replies on this one. What they said. :P
________________________________ From: Ping Zhang <[email protected]> Sent: Tuesday, December 20, 2022 10:43 AM To: [email protected] Subject: RE: [EXTERNAL][Discuss]whether fail early for Xcom.set when value exceeds the BLOB limit CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Thanks Jarek and Ash. I will prepare a PR for it. Thanks, Ping On Tue, Dec 20, 2022 at 6:28 AM Ash Berlin-Taylor <[email protected]<mailto:[email protected]>> wrote: +1 -- though I don wonder if that is a behaviour that only MySQL exhibits? Either way, we should throw an exception early. -ash On Dec 20 2022, at 8:04 am, Jarek Potiuk <[email protected]<mailto:[email protected]>> wrote: +1. I am all for failing it. I do not see an immediate issue with it - especially if we also explain how to handle it (i.e. custom XCom, limiting data, truncating data). I assume this is only when "standard" XCom is used. I would be for it, explicit is better than implicit. The XCom limit has always been a bit "vague" - we wrote something alongside "not too big" value and you could check in the code what the limit is but we never explicitly specified that as a limitation, but it was there. For me this looks like even more details of something that we already started discussing separately - i.e. being explicit about the "public interface" of ours and being explicit with that. If our aim is being an "extensible platform", then we have to be very clear about what our users can and cannot do with Airflow. J. On Tue, Dec 20, 2022 at 4:03 AM Ping Zhang <[email protected]<mailto:[email protected]>> wrote: Hi community, I would like to discuss if we want to fail early for Xcom.set when the value exceeds the storage limit. Otherwise, the database (default xcom backend) truncates the value leading to failure when Xcom reads it, (e.g. on UI, Admin -> Xcoms will also fail). But I am not sure whether we want to fail the whole task execution if Xcom.set fails and would like to have your inputs. Thanks, Ping
