object MyDatabseSingleton { @transient lazy val dbConn = DB.connect(…)
`transient` marks the variable to be excluded from serialization and `lazy` would open connection only when it's needed and also makes sure that the val is thread-safe http://fdahms.com/2015/10/14/scala-and-the-transient-lazy-val-pattern/ http://code-o-matic.blogspot.com/2009/05/double-checked-locking-idiom-sweet-in.html On Mon, Jul 30, 2018 at 1:32 PM kant kodali <kanth...@gmail.com> wrote: > > Hi Patrick, > > This object must be serializable right? I wonder if I will access to this > object in my driver(since it is getting created on the executor side) so I > can close when I am done with my batch? > > Thanks! > > On Mon, Jul 30, 2018 at 7:37 AM, Patrick McGloin <mcgloin.patr...@gmail.com> > wrote: >> >> You could use an object in Scala, of which only one instance will be created >> on each JVM / Executor. E.g. >> >> object MyDatabseSingleton { >> var dbConn = ??? >> } >> >> On Sat, 28 Jul 2018, 08:34 kant kodali, <kanth...@gmail.com> wrote: >>> >>> Hi All, >>> >>> I understand creating a connection forEachPartition but I am wondering can >>> I create one DB connection per executor and close it after the job is done? >>> any sample code would help. you can imagine I am running a simple batch >>> processing application. >>> >>> Thanks! > > -- Sent from my iPhone --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org