object MyDatabseSingleton {
@transient
lazy val dbConn = DB.connect(…)
`transient` marks the variable to be excluded from serialization
and `lazy` would open connection only when it's needed and also makes
sure that the val is thread-safe
http://fdahms.com/2015/10/14/scala-and-the-transient-lazy-val-pattern/
http://code-o-matic.blogspot.com/2009/05/double-checked-locking-idiom-sweet-in.html
On Mon, Jul 30, 2018 at 1:32 PM kant kodali <kanth...@gmail.com> wrote:
>
> Hi Patrick,
>
> This object must be serializable right? I wonder if I will access to this
> object in my driver(since it is getting created on the executor side) so I
> can close when I am done with my batch?
>
> Thanks!
>
> On Mon, Jul 30, 2018 at 7:37 AM, Patrick McGloin <mcgloin.patr...@gmail.com>
> wrote:
>>
>> You could use an object in Scala, of which only one instance will be created
>> on each JVM / Executor. E.g.
>>
>> object MyDatabseSingleton {
>> var dbConn = ???
>> }
>>
>> On Sat, 28 Jul 2018, 08:34 kant kodali, <kanth...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> I understand creating a connection forEachPartition but I am wondering can
>>> I create one DB connection per executor and close it after the job is done?
>>> any sample code would help. you can imagine I am running a simple batch
>>> processing application.
>>>
>>> Thanks!
>
>
--
Sent from my iPhone
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
