Re: SessionCatalog lock issue

2021-03-18 Thread Chang Chen
Oh, yeah, moving the call to externalCatalog out of `synchronized` would be better. But we can not do such a simple change in all cases, anyway, we will try. Thanks Wenchen Fan 于2021年3月18日周四 下午9:47写道: > The `synchronized` is needed for getting `currentDb` IIUC. So a small > change is to only

Re: Observable Metrics on Spark Datasets

2021-03-18 Thread Wenchen Fan
I think a listener-based API makes sense for streaming (since you need to keep watching the result), but may not be very reasonable for batch queries (you only get the result once). The idea of Observation looks good, but we should define what happens if `observation.get` is called before the batch

Re: SessionCatalog lock issue

2021-03-18 Thread Wenchen Fan
The `synchronized` is needed for getting `currentDb` IIUC. So a small change is to only wrap `formatDatabaseName(name.database.getOrElse(currentDb))` with `synchronized`. On Thu, Mar 18, 2021 at 3:38 PM Chang Chen wrote: > hi all > > We met an issue which is related with SessionCatalog synchron

SessionCatalog lock issue

2021-03-18 Thread Chang Chen
hi all We met an issue which is related with SessionCatalog synchronized, for example def tableExists(name: TableIdentifier): Boolean = synchronized { val db = formatDatabaseName(name.database.getOrElse(currentDb)) val table = formatTableName(name.table) externalCatalog.tableExists(db, tab