viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747607411
@HeartSaVioR We can discuss it on
https://issues.apache.org/jira/browse/SPARK-33827 for inactive state store
management.
---
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747325973
I see. It is much clearer now.
This is already over the original scope of this PR. The purpose of this
change is pretty simple: making store unloading more consistent. The
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747232971
> So IMHO the right direction would be either trying our best to unload
inactive state ASAP, or considering the replication as the further improvement.
Not somewhere in between.
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747182349
> The problem explanation sounds me as we should unload ASAP whenever
possible instead of delaying, right?
>
> Providing TTL would delay the unload more than current, even
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747173777
> Could you please elaborate for "stablization", via describing the
situation how things could be broken? It checks with driver that there's
another executor serving state befor
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747155648
> Let's be clear, it is relying on luck "as it is", it requires non-trivial
change to not rely on luck.
> E.g. You can make state store coordinator to track executors being
in
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747148582
> The problem is, this is completely relying on luck - this doesn't give any
help on physical plan. Again the problem exists even without the PR, but then
shouldn't we fix the ro
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747142154
> The problem is, this is completely relying on luck - this doesn't give any
help on physical plan. Again the problem exists even without the PR, but then
shouldn't we fix the ro
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747135903
> Once the state gets huge, the cost to load from state store should be
huge and this PR helps to remedy the situation, but the cost to maintain the
same store across different
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-747093389
> Could you please elaborate the change? I see you're adding `lastAliveTime`
and setting it, but never checking it. `stateStoreKeepAliveTime` isn't
referenced anywhere. Did you m
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-745587886
cc @dongjoon-hyun @HyukjinKwon @HeartSaVioR
This is an automated message from the Apache Git Service.
To respond
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-745439477
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please lo
viirya commented on pull request #30770:
URL: https://github.com/apache/spark/pull/30770#issuecomment-745048298
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please lo
13 matches
Mail list logo
