Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7943#issuecomment-128045237
Hi @tgravescs -- I know there is probably some cleanup to do here, but
before getting into the weeds can I ask two high level questions:
(1) Are you convinced that leveldb is the right way to go? its easy enough
to use, but I don't have confidence around its fail-safe properties, so I will
have to defer to you on that (plus the cost of adding another dependency). I'd
prefer to avoid keeping both of these PRs going as soon as possible :)
(2) Did you see my comment earlier on whether you think its safe to rely on
yarn to call `stopApplication` after an NM comes back? Specifically:
> Just to be clear, I know this leak is most likely very small, but the
problem is how hard it is to ever clean it up. if you always re-registered
everything in that file, then any bogus apps in there can't be removed, unless
you manually go in and delete the file. Even if you restart the NM again, you'd
just re-read the file with that bogus app still in there.
>You will know better than me how strong the guarantees are from Yarn for
calling stopApplication when the NM comes back, vs. how often NMs get restarted
(and thus we increase the chance for an app to stop during an NM restart), vs.
how long till we do a "hard reset" of an NM where the local dir gets cleaned up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
