potiuk commented on PR #31565:
URL: https://github.com/apache/airflow/pull/31565#issuecomment-1566315042

   > Using two classes (dagrun and DagRunPydantic) requires two instance checks 
more or less everywhere it seems (DagRun | DagRunPydantic) while there seems no 
real reason to do so.
   
   TL;DR; I tried different approaches (also pydantic-sqlalchemy) and I could 
not find a better solution for the needs of Internal DB-API. While there is 
some duplication, I believe it is necessary to keep the DB/NoDB dualism as is a 
result of earlier Airlfow architectural decisions that not only extended some 
of our DB models, but also exposed them to DAG Authors as part of Airflow's 
public interface.
   
   Apologies for the length of it. There are good reasons to do so. I spend 
quite some time not only thinking about it but evaluating and trying different 
solutions and I am out of ideas how to do it better. If someone could actually 
do it - based on the below understanding of the problems we have with the 
DB-less mode, I would appreciate. But I am afraid "just use 
pydantic-sqlalchemy" or just "discover fields" is just scratching the surface 
of the problem and when you actually try to do it (I actually tried both and 
failed). Also I am a bit less available in the coming week, so maybe that will 
give time for others to digest it (if they have the courage to read id) and 
think of another solution.
   
   So please do not treat it is as "preaching" @bolkedebruin - I understand it 
might "seem" as simple as serializing/deserializing fields of ORM objects, but 
(due to Airlfow earler decisions made long time ago and backwards-compatibility 
needs), I believe this is just a first step and it does not solve the problem 
of DB-less mode. So by writing the below I am really asking for help to try to 
find out a better solution that I came up with. Because I tried and failed. But 
maybe others can do it better. So if anything, this explanation is more of a 
"cry for help" - I am desperate to make it easier and more automated, but I 
failed to find a way so far and if anything, I need someone who comes up with 
idea how to do it and proves the idea by a POC based on the below.  
   
   The Dualism of DagRun | DagRunPydantic was very deliberate and a way to 
build of set of common fields.  I am happy if is done differently than I 
proposed, so I would love to hear/see a solution to this (but for now I think 
pydantic-sqlalchemy approach does not solve it - even if pydantic-sqlalchemy 
was maintained. 
   
   Of course - I looked at pydantic-sqlalchemy  at my orm <> noorm journey, 
when attempted to solve the problem (see my detailed explanation here where I 
answered to the same question asked by @pierrejeambrun  
https://github.com/apache/airflow/pull/29776#discussion_r1118339128 in case you 
have not seen it)
   
   But I am happy to repeat it agan and even expand it a bit more after what 
I've learned by not only looking at the code but also making the changes so far 
(and some of that already successfully applied with BaseJob refactoring):
   
   First thing IMHO, we should not use pydantic-sqlalchemy because it just 
scratches the surface of what is needed:
   
   1) last commit and release there were >2 years ago (though the code there is 
very few lines - indeed it uses inspection and code is rather straightforward)
   2) it is deliberately mentioned as experimental by the author
   
   But those two are not as important as the two below:
   
   3) it does not care about relationships between the objects and you have to 
write a custom code to rebuild relationships between then objects.
   4) it only cares about fields in the DB, not the methods in the DB objects 
of ours which IMHO need to be kept due to backwards compaatibility - including 
some DB calls (see later examples)
   
   The points 3) and 4) specifically caused that I tried and rejected it as 
much more complex if we try to automate it rather than repeat the declaration 
of a handful of classes in their Pydantic form (and my personal attempt to 
automatically convert the objects I did suffered from the same problem).
   
   ### Re 3) - relationships:
   
   See the example in the 
(Readme)[https://github.com/tiangolo/pydantic-sqlalchemy], where you have to 
manually create special "UserWithAddress" class when you want to represent the 
relationships:   . In our case we would have to manually recreate 
`TaskInstanceWithDagRunWithConsumedDatasetEventsDatasetsConsumingProducingTasksAndOutletsLinkedByReferences`
 (this is not complete) if we were to serialize TaskInstance object. Because 
yes TaskInstance has relation to DagRun, and DagRun has relation to Datasets it 
consumes.
   
   Why do we need to recreate those? Because currently and DAG developer can do 
(not really useful and not exactly the syntax, but this should explain the 
point):
   
   ```
   
context.ti.dag_run.consumed_dataset_events[0].dataset.producing_tasks[0].is_orphaned
   ```
   
   This works now, and should continue to work after the change in db-less 
mode. We not only use those relationships internally, but also we deliberately 
pass TI and DagRun in the context, and accessing those objects is the only way 
to get information about datasets.
   
   If you use sqlalchemy-pydantic, you need to manually recreate those 
relations. This means that you are not only equally prone to maintenance 
issues, but also you have no way to verify that any future version works before 
and after Pydantic serialization - because you might add another class there, 
and be absolutely no aware that you forgot to add relationship there. Of course 
one could attempt to automatically discover that DagRun object has the 
relationship to Datasets - but there are a number of ways this can be done and 
you you need to scan all other models and build such relationships 
(back-references). It's complex, prone to error. And this case is fully handled 
by one, line in the DAGRunPydantic that was written "by hand".  The 
`TaskInstancePydantic.from_orm(TaskInstance)` **just works** with it (we even 
have a tests confirming it) `test_serializing_pydantic_dataset_event`.
   
   ```
       consumed_dataset_events: List[DatasetEventPydantic]
   ```
   
   ## Re 4) lack of method synchronization in the "automated" objects.
   
   The original assumptions that we will always have a DB access from worker 
has taken a toll here. We made the choice (simlarly as in case of BaseJob 
before) that we added custom methods to the ORM classes - and then exposed 
those ORM classess to the DAG authors. This was not a strange thing to do when 
we always had DB access from the the local workers, but now part of the logic 
for task instances, dag run, datasets lives inside the ORM classes and there is 
no automatically convert the logic when automatically converting the ORM 
classes. Those are few classes that suffer from that - DagRun and TaskInstance 
particularly. But the problem is that our DAG Authors already know and use 
those methods and we need to keep those methods. Otherwise MANY dags will start 
breaking out there. 
   
   For example the DAG Author can currently do:
   
   ```
   context.ti.get_num_running_task_instances()
   ```
   
   This method is actually a DB method that runs a query dynamically when it is 
called. How would you make sure that "automatically" converted 
TaskInstancePydantic works the same way when passed in context? I would love to 
hear a solution to that (without introducing some kind of dualism, duplicating 
the code or applying special cases there.
   
   There are quite a few similar methods (`get_previous_dagrun`, 
`get_previous_execution_date` or even `get_template_context` or even just 
`get_current_state` - which retrieves a LOT of information including dataset 
events, outlets., inlets etc. number of others). DAG authors use them and they 
are even encouraged to do so. This is all part of our "public_interface" 
officially. See for example this description of 
https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/taskinstance/index.html#airflow.models.taskinstance.TaskInstance.current_state
   
   Which is linked from 
https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html
 which is now canonical place which defines what can be used by DAG Authors.
   
   The dualism of ORM/Pydantic classes is one of the ways to solve it. It also 
utilises MyPy to both find out what still needs to be done and to inform the 
user what they can use (TaskInstancePydantic should have a complete set of 
public interface that is usable by the DAGAuthor when completed - parallel to 
TaskInstance, thus making TaskInstance | TaskInstancePydantic a useful union to 
have.
   
   What we plan to do, is to make sure that all such methods that can be used 
by dag author are explicitly defined in TaskInstancePydantic (and other 
classes) and they are proxies to the "real" methods. This has already been 
succesfully applied to a number of methods in Job. We had the same problem 
there, and I solved it by splitting the Job into ORM Job/JobPydantic that only 
contains fields and moving the methods out of `Job' ORM objects when they were 
there initially.
   
   Unfortunately we cannot "remove" those methods from TaskInstance - they need 
to stay there because our users use them. We need to find a way how you can 
have a Pydantic class where when some methods are called will actually reach 
out to the DB of ours and run the query. So what we will need to do is 
similarly to BaseJob, move code of those methods out of the class and run a 
proxy call - either via internal API call (For TaskInstancePydantic) or 
directly (for TaskInstance). 
   
   Also - there is a very nice property of having "TaskInstance | 
TaskInstancePydantic" dualism actually. They implicitly make the "db-less" mode 
as future-proof as possible. Having MyPy checks via TaskInstance | 
TaslInstancePydantic is a great way to make sure that we do not "break" db-less 
mode - it will make sure that we will not add something that is not properly 
handled by the "db-less" mode. If someone adds a new table or new public DB 
method to TaskInstance as reference, it will run fine in DB-less mode, but if 
you don't add it to TaskInstancePydantic, it will fail MyPy. so you need to 
make sure TaskInstancePydantic in this case will make a proxy call over 
internal API. This "future-proofness" is actually improving maintenance of the 
solution, not decreasing it. In this case having duplication is not a problem. 
but forgetting that you should also handle a new DB call in local task job is. 
So in this case the need to have duplication is to our advantage actually, 
because author
  of such a new method will have to also think what to do in case of "db-less" 
mode.
   
   But maybe there is another way?
   
   I would absolutely love to have other solution that would be less complex 
and have less duplication - but I have no better idea how to do it.  Maybe we 
can find one together? 
   
   If we can have a solution for both 3) and 4) without that duplication (and 
with future-proof approach), I am more than happy to use it and I am all ears. 
But for now I cannot see one.  But maybe you or someone else could do a POC of 
doing it differently and showing that it can be done simpler without the 
duplication. I tried several things (including pydantic-sqlalchemy) but could 
not find a different solution (even if there is a degree of duplication there).
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to