Hi all, Since 1.6.0, low latency query are much slower now. This seems to be connected to the multi-user in the thrift-server. So on any newly created session, jobs are added to fill the session cache with information related to the tables it queries. Here is the details for this job: load at LocalCache.java:3599 org.apache.spark.sql.hive.HiveMetastoreCatalog$$anon$1.load(HiveMetastoreCatalog.scala:124) org.spark-project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) org.spark-project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) org.spark-project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342) org.spark-project.guava.cache.LocalCache$Segment.get(LocalCache.java:2257) org.spark-project.guava.cache.LocalCache.get(LocalCache.java:4000) org.spark-project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004) org.spark-project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874) org.spark-project.guava.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4880) org.spark-project.guava.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4898) org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:387) org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation(HiveContext.scala:457) org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:161) org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation(HiveContext.scala:457) org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:303) org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$9.applyOrElse(Analyzer.scala:315) org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$9.applyOrElse(Analyzer.scala:310) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57) org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:53)
[cid:image001.png@01D15E95.EDDC7080] Any ways to cache this at thrift-server instead? So it's reusable but all sessions? Other than going back to single user ofcourse:) Thanks, Younes
