Alexis Seigneurin created SPARK-20110: -----------------------------------------
Summary: Windowed aggregation do not work when the timestamp is a nested field Key: SPARK-20110 URL: https://issues.apache.org/jira/browse/SPARK-20110 Project: Spark Issue Type: Bug Components: Input/Output Affects Versions: 2.1.0 Reporter: Alexis Seigneurin I am loading data into a DataFrame with nested fields. I want to perform a windowed aggregation on the timestamp from a nested fields: {code} .groupBy(window($"auth.sysEntryTimestamp", "2 minutes")) {code} I get the following error: {quote} org.apache.spark.sql.AnalysisException: Multiple time window expressions would result in a cartesian product of rows, therefore they are not currently not supported. {quote} This works fine if I first extract the timestamp to a separate column: {code} .withColumn("sysEntryTimestamp", $"auth.sysEntryTimestamp") .groupBy( window($"sysEntryTimestamp", "2 minutes") ) {code} Please see the whole sample: - batch: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4683710270868386/4278399007363210/3769253384867782/latest.html - Structured Streaming: https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4683710270868386/4278399007363192/3769253384867782/latest.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org