Re: Adding Custom finalize method to RDDs.
If you control the codebase, you control when an RDD goes out of scope. Or am I missing something? (Note that finalize will not necessarily executed when an object goes out of scope but when the GC runs at some indeterminate point in the future. Please avoid using finalize for the kind of task you're trying to do. It's not what it was designed for. Better to pay more attention to house-keeping in your own code.) On Wed, Jun 12, 2019 at 9:11 PM Nasrulla Khan Haris < nasrulla.k...@microsoft.com> wrote: > We cannot have control over RDD going out of scope from memory as it was > handled by JVM. Thus I am not sure try and finalize will help. > > Thus I wanted to use some mechanism to cleanup of some temporary data > which is created by RDD immediately as soon as it goes out of scope. > > > > Any ideas ? > > > > Thanks, > > Nasrulla > > > > *From:* Phillip Henry > *Sent:* Tuesday, June 11, 2019 11:28 PM > *To:* Nasrulla Khan Haris > *Cc:* Vinoo Ganesh ; dev@spark.apache.org > *Subject:* Re: Adding Custom finalize method to RDDs. > > > > That's not the kind of thing a finalize method was ever supposed to do. > > > > Use a try/finally block instead. > > > > Phillip > > > > > > On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, < > nasrulla.k...@microsoft.com.invalid> wrote: > > I want to delete some files which I created In my datasource api, as soon > as the RDD is cleaned up. > > > > Thanks, > > Nasrulla > > > > *From:* Vinoo Ganesh > *Sent:* Monday, June 10, 2019 1:32 PM > *To:* Nasrulla Khan Haris ; > dev@spark.apache.org > *Subject:* Re: Adding Custom finalize method to RDDs. > > > > Generally overriding the finalize() method is an antipattern (it was in > fact deprecated in java 11 > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) > . What’s the use case here? > > > > *From: *Nasrulla Khan Haris > *Date: *Monday, June 10, 2019 at 15:44 > *To: *"dev@spark.apache.org" > *Subject: *RE: Adding Custom finalize method to RDDs. > > > > Hello Everyone, > > Is there a way to do it from user-code ? > > > > Thanks, > > Nasrulla > > > > *From:* Nasrulla Khan Haris > *Sent:* Sunday, June 9, 2019 5:30 PM > *To:* dev@spark.apache.org > *Subject:* Adding Custom finalize method to RDDs. > > > > Hi All, > > > > Is there a way to add custom finalize method to RDD objects to add custom > logic when RDDs are destructed by JVM ? > > > > Thanks, > > Nasrulla > > > >
RE: Adding Custom finalize method to RDDs.
We cannot have control over RDD going out of scope from memory as it was handled by JVM. Thus I am not sure try and finalize will help. Thus I wanted to use some mechanism to cleanup of some temporary data which is created by RDD immediately as soon as it goes out of scope. Any ideas ? Thanks, Nasrulla From: Phillip Henry Sent: Tuesday, June 11, 2019 11:28 PM To: Nasrulla Khan Haris Cc: Vinoo Ganesh ; dev@spark.apache.org Subject: Re: Adding Custom finalize method to RDDs. That's not the kind of thing a finalize method was ever supposed to do. Use a try/finally block instead. Phillip On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, mailto:nasrulla.k...@microsoft.com.invalid>> wrote: I want to delete some files which I created In my datasource api, as soon as the RDD is cleaned up. Thanks, Nasrulla From: Vinoo Ganesh mailto:vgan...@palantir.com>> Sent: Monday, June 10, 2019 1:32 PM To: Nasrulla Khan Haris mailto:nasrulla.k...@microsoft.com.INVALID>>; dev@spark.apache.org<mailto:dev@spark.apache.org> Subject: Re: Adding Custom finalize method to RDDs. Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11 https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here? From: Nasrulla Khan Haris mailto:nasrulla.k...@microsoft.com.INVALID>> Date: Monday, June 10, 2019 at 15:44 To: "dev@spark.apache.org<mailto:dev@spark.apache.org>" mailto:dev@spark.apache.org>> Subject: RE: Adding Custom finalize method to RDDs. Hello Everyone, Is there a way to do it from user-code ? Thanks, Nasrulla From: Nasrulla Khan Haris mailto:nasrulla.k...@microsoft.com.INVALID>> Sent: Sunday, June 9, 2019 5:30 PM To: dev@spark.apache.org<mailto:dev@spark.apache.org> Subject: Adding Custom finalize method to RDDs. Hi All, Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ? Thanks, Nasrulla
Re: Adding Custom finalize method to RDDs.
That's not the kind of thing a finalize method was ever supposed to do. Use a try/finally block instead. Phillip On Wed, 12 Jun 2019, 00:01 Nasrulla Khan Haris, wrote: > I want to delete some files which I created In my datasource api, as soon > as the RDD is cleaned up. > > > > Thanks, > > Nasrulla > > > > *From:* Vinoo Ganesh > *Sent:* Monday, June 10, 2019 1:32 PM > *To:* Nasrulla Khan Haris ; > dev@spark.apache.org > *Subject:* Re: Adding Custom finalize method to RDDs. > > > > Generally overriding the finalize() method is an antipattern (it was in > fact deprecated in java 11 > https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) > . What’s the use case here? > > > > *From: *Nasrulla Khan Haris > *Date: *Monday, June 10, 2019 at 15:44 > *To: *"dev@spark.apache.org" > *Subject: *RE: Adding Custom finalize method to RDDs. > > > > Hello Everyone, > > Is there a way to do it from user-code ? > > > > Thanks, > > Nasrulla > > > > *From:* Nasrulla Khan Haris > *Sent:* Sunday, June 9, 2019 5:30 PM > *To:* dev@spark.apache.org > *Subject:* Adding Custom finalize method to RDDs. > > > > Hi All, > > > > Is there a way to add custom finalize method to RDD objects to add custom > logic when RDDs are destructed by JVM ? > > > > Thanks, > > Nasrulla > > >
RE: Adding Custom finalize method to RDDs.
I want to delete some files which I created In my datasource api, as soon as the RDD is cleaned up. Thanks, Nasrulla From: Vinoo Ganesh Sent: Monday, June 10, 2019 1:32 PM To: Nasrulla Khan Haris ; dev@spark.apache.org Subject: Re: Adding Custom finalize method to RDDs. Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11 https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here? From: Nasrulla Khan Haris mailto:nasrulla.k...@microsoft.com.INVALID>> Date: Monday, June 10, 2019 at 15:44 To: "dev@spark.apache.org<mailto:dev@spark.apache.org>" mailto:dev@spark.apache.org>> Subject: RE: Adding Custom finalize method to RDDs. Hello Everyone, Is there a way to do it from user-code ? Thanks, Nasrulla From: Nasrulla Khan Haris mailto:nasrulla.k...@microsoft.com.INVALID>> Sent: Sunday, June 9, 2019 5:30 PM To: dev@spark.apache.org<mailto:dev@spark.apache.org> Subject: Adding Custom finalize method to RDDs. Hi All, Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ? Thanks, Nasrulla
Re: Adding Custom finalize method to RDDs.
Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11 https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here? From: Nasrulla Khan Haris Date: Monday, June 10, 2019 at 15:44 To: "dev@spark.apache.org" Subject: RE: Adding Custom finalize method to RDDs. Hello Everyone, Is there a way to do it from user-code ? Thanks, Nasrulla From: Nasrulla Khan Haris Sent: Sunday, June 9, 2019 5:30 PM To: dev@spark.apache.org Subject: Adding Custom finalize method to RDDs. Hi All, Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ? Thanks, Nasrulla
RE: Adding Custom finalize method to RDDs.
Hello Everyone, Is there a way to do it from user-code ? Thanks, Nasrulla From: Nasrulla Khan Haris Sent: Sunday, June 9, 2019 5:30 PM To: dev@spark.apache.org Subject: Adding Custom finalize method to RDDs. Hi All, Is there a way to add custom finalize method to RDD objects to add custom logic when RDDs are destructed by JVM ? Thanks, Nasrulla
