Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22313#discussion_r214528778
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala ---
@@ -55,19 +59,52 @@ import org.apache.spark.sql.types._
* known to be convertible.
*/
private[orc] object OrcFilters extends Logging {
+ case class FilterWithTypeMap(filter: Filter, typeMap: Map[String,
DataType])
+
+ private lazy val cacheExpireTimeout =
+
org.apache.spark.sql.execution.datasources.orc.OrcFilters.cacheExpireTimeout
+
+ private lazy val searchArgumentCache = CacheBuilder.newBuilder()
+ .expireAfterAccess(cacheExpireTimeout, TimeUnit.SECONDS)
+ .build(
+ new CacheLoader[FilterWithTypeMap, Option[Builder]]() {
+ override def load(typeMapAndFilter: FilterWithTypeMap):
Option[Builder] = {
+ buildSearchArgument(
+ typeMapAndFilter.typeMap, typeMapAndFilter.filter,
SearchArgumentFactory.newBuilder())
+ }
+ })
+
+ private def getOrBuildSearchArgumentWithNewBuilder(
--- End diff --
@xuanyuanking . This already reuses `cacheExpireTimeout`.
For the cache value, `SearchArgument`, `SearchArgumentFactory` and
`Builder` are different.
- Here, they comes from `org.apache.hadoop.hive.ql.io.sarg.*`.
- There, they comes from `org.apache.orc.storage.ql.io.sarg.*`.
The only exception I made is `FilterWithTypeMap`. I wanted to keep them
separately since it's also related to cache key.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
