[ https://issues.apache.org/jira/browse/SPARK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046476#comment-16046476 ]
Chenzhao Guo commented on SPARK-20927: -------------------------------------- What exactly is 'no-op' ? Does that mean scala `NotImplementedError`, or could you give an example? > Add cache operator to Unsupported Operations in Structured Streaming > --------------------------------------------------------------------- > > Key: SPARK-20927 > URL: https://issues.apache.org/jira/browse/SPARK-20927 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.2.0 > Reporter: Jacek Laskowski > Priority: Trivial > Labels: starter > > Just [found > out|https://stackoverflow.com/questions/42062092/why-does-using-cache-on-streaming-datasets-fail-with-analysisexception-queries] > that {{cache}} is not allowed on streaming datasets. > {{cache}} on streaming datasets leads to the following exception: > {code} > scala> spark.readStream.text("files").cache > org.apache.spark.sql.AnalysisException: Queries with streaming sources must > be executed with writeStream.start();; > FileSource[files] > at > org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.org$apache$spark$sql$catalyst$analysis$UnsupportedOperationChecker$$throwError(UnsupportedOperationChecker.scala:297) > at > org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$$anonfun$checkForBatch$1.apply(UnsupportedOperationChecker.scala:36) > at > org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$$anonfun$checkForBatch$1.apply(UnsupportedOperationChecker.scala:34) > at > org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127) > at > org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.checkForBatch(UnsupportedOperationChecker.scala:34) > at > org.apache.spark.sql.execution.QueryExecution.assertSupported(QueryExecution.scala:63) > at > org.apache.spark.sql.execution.QueryExecution.withCachedData$lzycompute(QueryExecution.scala:74) > at > org.apache.spark.sql.execution.QueryExecution.withCachedData(QueryExecution.scala:72) > at > org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:78) > at > org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:78) > at > org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:84) > at > org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:80) > at > org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:89) > at > org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:89) > at > org.apache.spark.sql.execution.CacheManager$$anonfun$cacheQuery$1.apply(CacheManager.scala:104) > at > org.apache.spark.sql.execution.CacheManager.writeLock(CacheManager.scala:68) > at > org.apache.spark.sql.execution.CacheManager.cacheQuery(CacheManager.scala:92) > at org.apache.spark.sql.Dataset.persist(Dataset.scala:2603) > at org.apache.spark.sql.Dataset.cache(Dataset.scala:2613) > ... 48 elided > {code} > It should be included in Structured Streaming's [Unsupported > Operations|http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#unsupported-operations]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org