GitHub user parshimers added a comment to the discussion: Discuss workflow macro design and lifecycle
I'm glad this got revisited. I didn't follow the earlier discussion. I'm glad it led to form views, but I think macros are a separate and useful thing. I agree with the idea of treating macros as a kind of special workflow, which cannot be run on its own. I think the simple inlining of those workflows into whatever workflow they are used in, as a special kind of operator, is a very common and natural thing. AsterixDB's SQL++ UDFs are implemented in exactly the same way; they are simply queries which get written into some other query where the function appears in the main query. I think a workflow that references a macro should always refer to it with a reference, and only resolve that reference at compile time. It really can only be the compiler that can take ownership of this part. The dependencies are indeed a tricky problem. I think at least for "pure" operators it's fine, but for UDFs it can get complicated. I think it might be easier to constrain those out of the design for now if it complicates it. I don't think a property within a macro should be able to live outside of the context of the macro. It's effectively an operator, after all, and having other workflow side effects seem kind of strange to me. P.S. (Mentor hat): I noticed this thread kind of devolved into a series of offline discussions, and while the latest post did an OK job of trying to share the state and loop everyone else in, the rest of the thread is really an antipattern. The point of discussing it here is to make the discussion entirely open, and not to agree on when closed discussions should or shouldn't take place. It can't be the case that discussions end up as a dumping ground for designs which have already been agreed to a priori by a clique of contributors before ever being brought out into the open, basically as an afterthought. GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18802121 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
