Github user mallman commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @szhem.
I'm sorry I haven't been more responsive here. I can relate to your
frustration, and I do want to help you make progress on this PR and merge it
in. I have indeed been busy with
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hello @mallman, @sujithjay, @felixcheung, @jkbradley, @mengxr, it's already
about a year passed since this pull request has been opened.
I'm just wondering whether there is any chance to get any
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
I've tested the mentioned checkpointers with
`spark.cleaner.referenceTracking.cleanCheckpoints` set to `true` and without
explicit checkpoint files removal.
It seems that there are somewhere
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @asolimando,
I believe that the solution with weak references will work and probably
with `ContextCleaner` too, but there are some points I'd like to discuss if you
don't mind
-
Github user EthanRock commented on the issue:
https://github.com/apache/spark/pull/19410
I have tried to set graph's storage level to StorageLevel.MEMORY_AND_DISK
in my case and the error still happens.
---
-
To
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @szhem.
I understand you've put a lot of work into this implementation, however I
think you should try a simpler approach before we consider something more
complicated. I believe an
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @mallman,
I believe, that `ContextCleaner` currently does not delete checkpoint data
it case of unexpected failures.
Also as it works at the end of the job then there is still a chance
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @szhem.
Thanks for the information regarding disk use for your scenario. What do
you think about my second point, using the `ContextCleaner`?
---
Github user EthanRock commented on the issue:
https://github.com/apache/spark/pull/19410
Hi, I met the same problem today.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
@mallman
Just my two cents regarding built-in solutions:
Periodic checkpointer deletes checkpoint files not to pollute the hard
drive. Although disk storage is cheap it's not free.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @szhem. I dug deeper and think I understand the problem better.
To state the obvious, the periodic checkpointer deletes checkpoint files of
RDDs that are potentially still accessible. In
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @mallman!
In case of
```
StorageLevel.MEMORY_AND_DISK
StorageLevel.MEMORY_AND_DISK_SER_2
```
... tests pass.
They still fail in case of
```
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/19410
Hi @szhem. Thanks for the kind reminder and thanks for your contribution.
I'm sorry I did not respond sooner.
I no longer work where I regularly used the checkpointing code with large
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Just a kind remainder...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
Hello @viirya, @mallman, @felixcheung,
You were reviewing graph checkpointing, introduced here #15125, and this PR
changes the behaviour a little bit.
Could you please review this PR
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
I would happy if anyone can take a look at this PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19410
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
22 matches
Mail list logo
