GitHub user mitstake edited a comment on the discussion: Caching roadmap
> In general the goal of the API UX is to make certain things easier (or > harder) to accomplish and trading off what is required To preemptively address @skrawcz comment above from the other thread: this might be another case where the design of how this is done in darl might not fit into the Hamilton philosophy, but will leave that to the Hamilton team to decide. The reason being that darl caching is based off of dependency aware code hashing of functions in the dag. This only works if the functions are perfectly deterministic, and there are no guardrails to ensure/enforce determinism. There is a mechanism to handle some non-determinism but it’s meant to be used sparingly and is not a substitute for careful deterministic definition of all code. This introduces an easy way to shoot yourself in the foot and introduce bad stale cached values. In my experience with a well trained user base it’s not hard to keep all your code deterministic, but for a wide audience and guaranteed accuracy tool this methodology might not fit the Hamilton philosophy. But again I’ll leave it here for the team to take a look and decide for yourselves. (I’m not even sure it’s worth leaning heavily into **cross process** caching without expecting all code to be deterministic. But interested to hear your thoughts on this) GitHub link: https://github.com/apache/hamilton/discussions/1167#discussioncomment-15722954 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
