In order to share an HBase connection pool, we create an object

Object Util {
    val HBaseConf = HBaseConfiguration.create
    val Connection= HConnectionManager.createConnection(HBaseConf)
}

which would be shared among tasks on the same executor. e.g.

val result = rdd.map(line => {
  val table = Util.Connection.getTable("user")
  ...
}

However, we don’t how to close the Util.Connection.
If we write Util.Connection.close() in the main function,
it’ll only run on the driver, not the executor.

So, How to make sure every Connection closed before exist?
​

Reply via email to