cetingokhan commented on PR #62963: URL: https://github.com/apache/airflow/pull/62963#issuecomment-4289550953
Hi @gopidesupavan , Your suggestion was very logical; we sometimes forget the capabilities of an LLM, and I think it's very sensible to ask the LLM for this :) I've structured the entire system to only receive prompts (checks), but I've also left the optional validator choice as optional to prevent the LLM from handling specific needs. If a specific validator isn't specified, we leave the selection of the appropriate validator to the LLM. I didn't include the toolset logic in your suggestion because, since I support custom validation generation, the validation domain is quite large, and I felt that managing it under a toolset would make things even more difficult. However, if we still want to separate it and put it under the toolset, I added it as a separate class in dq_validation.py (if you want, we can separate it and put it under the toolset folder?). After your suggestions, a fairly large refactoring process took place. There might be some areas that need simplification. Before expending more effort, it's more important that we are all on the same page regarding the concept and usage. Therefore, if there are no different suggestions or corrections regarding the use of the operator in the final stage, I will proceed with the simplification phase. Also, as part of the simplification, I plan to create an abstract called BaseOutputGenerator or something similar and move the existing SQL generation under SQLOutputGenerator. This way, extensions can be added for different control methods in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
