This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-8344-f30af7e7714fae1f7d55b2c1d4b14c13592329f5 in repository https://gitbox.apache.org/repos/asf/texera.git
commit 004a524c27f7b30a2ed02154e203b6b9cda4fe7c Author: Kary Zheng <[email protected]> AuthorDate: Mon Sep 28 21:26:06 2026 +0000 feat(visualization): export the hierarchy and graph charts as Python (#8344) ### What changes were proposed in this PR? Seven visualizations implement `StandaloneCodeGenerator`: the ones that draw a structure rather than a coordinate system — tree plot, network graph, sankey, hierarchy chart, icicle chart, dendrogram — and the word cloud. A visualization emits a figure rather than a frame, so it is compared as one: number by number after both paths have drawn, which catches a difference in the data behind the picture without failing over a layout detail that carries no meaning. The word cloud renders an image instead, so it is compared as the HTML that carries it. Network Graph and Word Cloud are verified here rather than withheld. Both had been flagged for drawing a different picture on every run, which stopped being true once their placement was seeded, and lifting those two rows showed three things the reason had been hiding. Network Graph never called `manipulateTable`, the method it already had for dropping the rows that name no edge, so a null reached `add_node`, which refuses one outright. Word Cloud counted its words out of whatever column it was given, and the filter that finds them uses pandas' `.str`, which refuses anything but text, so the operator now states that. And the Tree Plot lays itself out without igraph, which is GPL v2 and Category X under the ASF third-party licence policy. Stating it is not enough on its own. An attribute type rule only warns: the property editor prints the warning and lets the workflow run, so a user still reaches the word cloud with a column of numbers, and `.str` ends the run on a pandas error naming an accessor nobody wrote. Both generators now answer that the way they already answer an empty table and a column holding no words, with the operator's own error page. This changes what the engine does with such a workflow: it used to fail, and now it draws the reason. Both sides ask the same question in the same words, so the two paths still agree on what they produce. ### Any related issues, documentation, discussions? Part of #8325, 13 of 27; that issue lists the set in order. It needs #8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here. Closes #7969, the Tree Plot's Category X dependency. Closes #8416, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. Once the verification lands it is also run through the engine and through its generated script, on every configuration its schema offers, and the two answers compared; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI. The word cloud's guard is compared on a table its column knob points at a numeric column, so the two paths are held to the same answer for input the type rule only warned about; that row arrives with the verification harness, like the others. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> --- .../IcicleChart/IcicleChartOpDesc.scala | 50 +++- .../visualization/PlotlyStandaloneCode.scala | 42 ++++ .../dendrogram/DendrogramOpDesc.scala | 61 ++++- .../hierarchychart/HierarchyChartOpDesc.scala | 50 +++- .../networkGraph/NetworkGraphOpDesc.scala | 126 +++++++++- .../sankeyDiagram/SankeyDiagramOpDesc.scala | 53 +++- .../visualization/treeplot/TreePlotOpDesc.scala | 266 +++++++++++++++++++-- .../visualization/wordCloud/WordCloudOpDesc.scala | 73 +++++- .../dendrogram/DendrogramOpDescSpec.scala | 2 +- .../treeplot/TreePlotOpDescSpec.scala | 200 +++++++++++++++- 10 files changed, 892 insertions(+), 31 deletions(-) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/IcicleChart/IcicleChartOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/IcicleChart/IcicleChartOpDesc.scala index 2c90296b9b..faeafba2ab 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/IcicleChart/IcicleChartOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/IcicleChart/IcicleChartOpDesc.scala @@ -22,10 +22,14 @@ package org.apache.texera.amber.operator.visualization.IcicleChart import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle} import org.apache.texera.amber.core.tuple.{AttributeType, Schema} -import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{ + PythonTemplateBuilderStringContext, + pyStringLiteral +} import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.operator.visualization.hierarchychart.HierarchySection @@ -43,7 +47,7 @@ import javax.validation.constraints.{NotEmpty, NotNull} } } """) -class IcicleChartOpDesc extends PythonOperatorDescriptor { +class IcicleChartOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode { @JsonProperty(required = true) @JsonSchemaTitle("Hierarchy Path") @JsonPropertyDescription( @@ -132,4 +136,46 @@ class IcicleChartOpDesc extends PythonOperatorDescriptor { finalCode.encode } + // Output is an HTML chart, not a tabular DataFrame. + // The translator skips it in the leaf-DataFrame print block. + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + val attributes = hierarchy.map(section => pyStringLiteral(section.attributeName)).mkString(", ") + val valueLit = pyStringLiteral(value) + // The error page is written to the same file a plotted chart lands in, so a + // reason for "no chart" is where the reader looks for the chart — printing it + // to the terminal alone left that file absent. render_error's continuation + // line keeps the runtime path's indentation, since the HTML is triple-quoted + // and those spaces reach the browser. + s"""def render_error(error_msg): + | return '''<h1>Icicle chart is not available.</h1> + | <p>Reason is: {} </p> + | '''.format(error_msg) + | + |def fail(error_msg): + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error(error_msg)) + | print(f"Icicle chart error: {error_msg}") + | + |if in1df.empty: + | fail("input table is empty.") + |else: + | # On a copy: the same frame can feed another branch of the plan, and + | # both the assignment and the drop below would otherwise reach it. + | chart_df = in1df.copy() + | chart_df[$valueLit] = chart_df[chart_df[$valueLit] > 0][$valueLit] + | chart_df = chart_df.dropna(subset=[$attributes]) + | if chart_df.empty: + | fail("value column contains only non-positive numbers or nulls.") + | else: + | fig = px.icicle(chart_df, path=[$attributes], values=$valueLit, + | color=$valueLit, hover_data=[$attributes], + | color_continuous_scale='RdBu') + | fig.update_layout(margin=dict(l=0, r=0, b=0, t=0)) + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Icicle chart saved to " + outputHtml)""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/PlotlyStandaloneCode.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/PlotlyStandaloneCode.scala new file mode 100644 index 0000000000..b70bb2327d --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/PlotlyStandaloneCode.scala @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.visualization + +import org.apache.texera.amber.operator.StandaloneCodeGenerator + +/** + * A generator whose emitted code draws with plotly. + * + * Mixed in rather than stated per operator because the three modules are one + * dependency: a chart reaching for `px` today and `go` tomorrow would otherwise + * have to remember to edit a list that nothing checks. What the mixin does say, + * and the reason it is not simply always emitted, is that an operator NOT + * mixing it in draws nothing, so a script built only from those runs wherever + * pandas is installed. + */ +trait PlotlyStandaloneCode extends StandaloneCodeGenerator { + + override def standaloneImports(): Seq[String] = + Seq( + "import plotly.express as px", + "import plotly.graph_objects as go", + "import plotly.io" + ) +} diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDesc.scala index 826cce9061..c8f4d728d7 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDesc.scala @@ -23,10 +23,13 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} import com.fasterxml.jackson.databind.annotation.JsonDeserialize import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle} import org.apache.texera.amber.core.tuple.{AttributeType, Schema} -import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{ + PythonTemplateBuilderStringContext, + pyStringLiteral +} import org.apache.texera.amber.pybuilder.PyStringTypes.{EncodableString, PythonLiteral} import org.apache.texera.amber.core.workflow.PortIdentity -import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.{PythonOperatorDescriptor, StandaloneCodeGenerator} import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.pybuilder.PythonTemplateBuilder @@ -43,7 +46,7 @@ import javax.validation.constraints.NotNull } } """) -class DendrogramOpDesc extends PythonOperatorDescriptor { +class DendrogramOpDesc extends PythonOperatorDescriptor with StandaloneCodeGenerator { @JsonProperty(value = "xVal", required = true) @JsonSchemaTitle("Value X Column") @JsonPropertyDescription("The x values of points in dendrogram") @@ -88,12 +91,15 @@ class DendrogramOpDesc extends PythonOperatorDescriptor { OperatorGroupConstants.VISUALIZATION_SCIENTIFIC_GROUP ) + /** An unset threshold reaches the generated code as Python's `None`, which is + * scipy's own 0.7 * max distance — what a blank field already meant. + */ + private def thresholdExpr: PythonLiteral = threshold.map(_.toString).getOrElse("None") + private def createDendrogram(): PythonTemplateBuilder = { assert(xVal.nonEmpty, "Value X Column cannot be empty") assert(yVal.nonEmpty, "Value Y Column cannot be empty") assert(labels.nonEmpty, "Labels cannot be empty") - // Unset means None, which is scipy's own 0.7 * max distance. - val thresholdExpr: PythonLiteral = threshold.map(_.toString).getOrElse("None") pyb""" | x = np.array(table[$xVal]) | y = np.array(table[$yVal]) @@ -146,4 +152,49 @@ class DendrogramOpDesc extends PythonOperatorDescriptor { |""" finalcode.encode } + + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + + // render_error's continuation line keeps the runtime path's indentation — the + // HTML is triple-quoted, so those spaces reach the browser. + s"""import numpy as np + |import plotly.figure_factory as ff + | + |def render_error(error_msg): + | return '''<h1>Dendrogram is not available.</h1> + | <p>Reason is: {} </p> + | '''.format(error_msg) + | + |def _write_error(message): + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error(message)) + | + |if in1df.empty: + | _write_error("input table is empty.") + |else: + | # A row missing either coordinate has no position to cluster from, and + | # scipy refuses a NaN anywhere in the distance matrix. Bound to a name + | # of its own: the same frame can feed another branch of the plan, which + | # must still see every row. + | chart_df = in1df.dropna(subset=[${pyStringLiteral(xVal)}, ${pyStringLiteral(yVal)}]) + | if chart_df.empty: + | _write_error("input table has no rows with all of the configured columns filled in.") + | # Clustering starts from the distances between rows, so a single row + | # leaves scipy an empty distance matrix and it raises rather than draws. + | elif len(chart_df) < 2: + | _write_error("input table has fewer than two rows to cluster.") + | else: + | x = np.array(chart_df[${pyStringLiteral(xVal)}]) + | y = np.array(chart_df[${pyStringLiteral(yVal)}]) + | data = np.column_stack((x, y)) + | labels = chart_df[${pyStringLiteral(labels)}].tolist() + | fig = ff.create_dendrogram(data, labels=labels, color_threshold=$thresholdExpr) + | fig.update_layout(yaxis_title="Linkage Distance", margin=dict(l=0, r=0, b=0, t=0)) + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Dendrogram saved to " + outputHtml)""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/hierarchychart/HierarchyChartOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/hierarchychart/HierarchyChartOpDesc.scala index 5b052ca59a..b57aa440f9 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/hierarchychart/HierarchyChartOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/hierarchychart/HierarchyChartOpDesc.scala @@ -22,10 +22,14 @@ package org.apache.texera.amber.operator.visualization.hierarchychart import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle} import org.apache.texera.amber.core.tuple.{AttributeType, Schema} -import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{ + PythonTemplateBuilderStringContext, + pyStringLiteral +} import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.pybuilder.PythonTemplateBuilder @@ -42,7 +46,7 @@ import javax.validation.constraints.{NotEmpty, NotNull} } } """) -class HierarchyChartOpDesc extends PythonOperatorDescriptor { +class HierarchyChartOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode { @JsonProperty(required = true) @JsonSchemaTitle("Chart Type") @JsonPropertyDescription("Treemap or Sunburst") @@ -137,4 +141,46 @@ class HierarchyChartOpDesc extends PythonOperatorDescriptor { finalCode.encode } + // Output is an HTML chart, not a tabular DataFrame. + // The translator skips it in the leaf-DataFrame print block. + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + val attributes = hierarchy.map(section => pyStringLiteral(section.attributeName)).mkString(", ") + val valueLit = pyStringLiteral(value) + // The error page is written to the same file a plotted chart lands in, so a + // reason for "no chart" is where the reader looks for the chart — printing it + // to the terminal alone left that file absent. render_error's continuation + // line keeps the runtime path's indentation, since the HTML is triple-quoted + // and those spaces reach the browser. + s"""def render_error(error_msg): + | return '''<h1>Hierarchy chart is not available.</h1> + | <p>Reason is: {} </p> + | '''.format(error_msg) + | + |def fail(error_msg): + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error(error_msg)) + | print(f"Hierarchy chart error: {error_msg}") + | + |if in1df.empty: + | fail("input table is empty.") + |else: + | # On a copy: the same frame can feed another branch of the plan, and + | # both the assignment and the drop below would otherwise reach it. + | chart_df = in1df.copy() + | chart_df[$valueLit] = chart_df[chart_df[$valueLit] > 0][$valueLit] + | chart_df = chart_df.dropna(subset=[$attributes]) + | if chart_df.empty: + | fail("value column contains only non-positive numbers or nulls.") + | else: + | fig = px.${hierarchyChartType.getPlotlyExpressApiName}(chart_df, path=[$attributes], values=$valueLit, + | color=$valueLit, hover_data=[$attributes], + | color_continuous_scale='RdBu') + | fig.update_layout(margin=dict(l=0, r=0, b=0, t=0)) + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Hierarchy chart saved to " + outputHtml)""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/networkGraph/NetworkGraphOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/networkGraph/NetworkGraphOpDesc.scala index 584ee4f5a3..fc6502f19d 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/networkGraph/NetworkGraphOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/networkGraph/NetworkGraphOpDesc.scala @@ -26,13 +26,15 @@ import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBui import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.pybuilder.PythonTemplateBuilder +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral import javax.validation.constraints.NotNull -class NetworkGraphOpDesc extends PythonOperatorDescriptor { +class NetworkGraphOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode { @JsonProperty(required = true) @JsonSchemaTitle("Source Column") @JsonPropertyDescription("Source node for edge in graph") @@ -66,6 +68,10 @@ class NetworkGraphOpDesc extends PythonOperatorDescriptor { OperatorGroupConstants.VISUALIZATION_SCIENTIFIC_GROUP ) + /** An edge needs both of its ends, and networkx refuses a null as a node + * outright. A row missing either one names no edge, so it goes before the + * graph is built rather than reaching add_node. + */ def manipulateTable(): PythonTemplateBuilder = { assert(source.nonEmpty, "Source Column cannot be empty") assert(destination.nonEmpty, "Destination Column cannot be empty") @@ -95,6 +101,7 @@ class NetworkGraphOpDesc extends PythonOperatorDescriptor { | | @overrides | def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]: + |${manipulateTable()} | if not table.empty: | sources = table[$source] | destinations = table[$destination] @@ -199,4 +206,121 @@ class NetworkGraphOpDesc extends PythonOperatorDescriptor { finalCode.encode } + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + val sourceLit = pyStringLiteral(source) + val destinationLit = pyStringLiteral(destination) + // The <br> is part of the emitted title, so it is escaped with the value + // rather than left outside the literal. + val titleLit = pyStringLiteral(s"<br>$title") + s"""import networkx as nx + | + |def render_error(error_msg): + | return '''<h1>Network graph is not available.</h1> + | <p>Reason is: {} </p> + | '''.format(error_msg) + | + |if in1df.empty: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("Table should not have any empty/null values or fields.")) + |else: + | table = in1df.dropna(subset=[$sourceLit]).dropna(subset=[$destinationLit]).copy() + | if table.empty: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("Table should not have any empty/null values or fields.")) + | else: + | sources = table[$sourceLit] + | destinations = table[$destinationLit] + | nodes = list(dict.fromkeys(pd.concat([sources, destinations]).tolist())) + | G = nx.Graph() + | for node in nodes: + | G.add_node(node) + | for _, row in table.iterrows(): + | G.add_edges_from([(row[$sourceLit], row[$destinationLit])]) + | pos = nx.spring_layout(G, k=0.5, iterations=50, seed=0) + | for n, p in pos.items(): + | G.nodes[n]["pos"] = p + | + | edge_trace = go.Scatter( + | x=[], + | y=[], + | name="Edges", + | line=dict(width=0.5, color="#888"), + | hoverinfo="none", + | mode="lines", + | visible=True + | ) + | + | for edge in G.edges(): + | x0, y0 = G.nodes[edge[0]]["pos"] + | x1, y1 = G.nodes[edge[1]]["pos"] + | edge_trace["x"] += tuple([x0, x1, None]) + | edge_trace["y"] += tuple([y0, y1, None]) + | + | node_trace = go.Scatter( + | x=[], + | y=[], + | name="Nodes", + | text=[], + | mode="markers", + | hoverinfo="text", + | visible=True, + | marker=dict( + | showscale=True, + | colorscale="plasma", + | reversescale=True, + | color=[], + | size=15, + | colorbar=dict( + | thickness=10, + | title="Node Connections", + | xanchor="left", + | titleside="right" + | ), + | line=dict(width=0) + | ) + | ) + | + | for node in G.nodes(): + | x, y = G.nodes[node]["pos"] + | node_trace["x"] += tuple([x]) + | node_trace["y"] += tuple([y]) + | + | for _, adjacencies in enumerate(G.adjacency()): + | node_trace["marker"]["color"] += tuple([len(adjacencies[1])]) + | node_info = str(adjacencies[0]) + ": " + str(len(adjacencies[1])) + " connections." + | node_trace["text"] += tuple([node_info]) + | + | fig = go.Figure( + | data=[edge_trace, node_trace], + | layout=go.Layout( + | title=$titleLit, + | hovermode="closest", + | showlegend=False, + | margin=dict(b=20, l=5, r=5, t=40), + | annotations=[ + | dict( + | text="", + | showarrow=False, + | xref="paper", + | yref="paper" + | ) + | ], + | xaxis=dict(showgrid=False, zeroline=False, showticklabels=False), + | yaxis=dict(showgrid=False, zeroline=False, showticklabels=False) + | ) + | ) + | fig.update_layout( + | margin=dict(l=0, r=0, t=0, b=0), + | legend=dict( + | itemclick=False, + | itemdoubleclick=False + | ) + | ) + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Network graph saved to " + outputHtml)""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/sankeyDiagram/SankeyDiagramOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/sankeyDiagram/SankeyDiagramOpDesc.scala index 7e6689bd72..6d8cbdde01 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/sankeyDiagram/SankeyDiagramOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/sankeyDiagram/SankeyDiagramOpDesc.scala @@ -26,9 +26,11 @@ import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBui import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity import org.apache.texera.amber.operator.PythonOperatorDescriptor -import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName +import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode +import org.apache.texera.amber.operator.metadata.annotations.{AutofillAttributeName, SampleColumn} import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.pybuilder.PythonTemplateBuilder +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral import javax.validation.constraints.NotNull @@ -43,12 +45,13 @@ import javax.validation.constraints.NotNull } } """) -class SankeyDiagramOpDesc extends PythonOperatorDescriptor { +class SankeyDiagramOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode { @JsonProperty(value = "Source Attribute", required = true) @JsonSchemaTitle("Source Attribute") @JsonPropertyDescription("The source node of the Sankey diagram") @AutofillAttributeName + @SampleColumn("node_src") @NotNull(message = "Source Attribute cannot be empty") var sourceAttribute: EncodableString = "" @@ -56,6 +59,7 @@ class SankeyDiagramOpDesc extends PythonOperatorDescriptor { @JsonSchemaTitle("Target Attribute") @JsonPropertyDescription("The target node of the Sankey diagram") @AutofillAttributeName + @SampleColumn("node_dst") @NotNull(message = "Target Attribute cannot be empty") var targetAttribute: EncodableString = "" @@ -63,6 +67,7 @@ class SankeyDiagramOpDesc extends PythonOperatorDescriptor { @JsonSchemaTitle("Value Attribute") @JsonPropertyDescription("The value/volume of the flow between source and target") @AutofillAttributeName + @SampleColumn("score") @NotNull(message = "Value Attribute cannot be empty") var valueAttribute: EncodableString = "" @@ -143,4 +148,48 @@ class SankeyDiagramOpDesc extends PythonOperatorDescriptor { |""" finalCode.encode } + + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + val sourceLit = pyStringLiteral(sourceAttribute) + val targetLit = pyStringLiteral(targetAttribute) + val valueLit = pyStringLiteral(valueAttribute) + s"""def render_error(error_msg): + | return '''<h1>Sankey Diagram is not available.</h1> + | <p>Reasons are: {} </p> + | '''.format(error_msg) + | + |if in1df.empty: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("Input table is empty.")) + |else: + | table = in1df.groupby([$sourceLit, $targetLit])[$valueLit].sum().reset_index(name="value") + | if table.empty: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("No valid rows left (every row has at least 1 missing value).")) + | else: + | labels = pd.concat([table[$sourceLit], table[$targetLit]]).unique().tolist() + | table["source_index"] = table[$sourceLit].apply(lambda x: labels.index(x)) + | table["target_index"] = table[$targetLit].apply(lambda x: labels.index(x)) + | fig = go.Figure(data=[go.Sankey( + | node=dict( + | pad=15, + | thickness=20, + | line=dict(color="black", width=0.5), + | label=labels, + | color="blue" + | ), + | link=dict( + | source=table["source_index"].tolist(), + | target=table["target_index"].tolist(), + | value=table["value"].tolist() + | ) + | )]) + | fig.update_layout(title_text="Sankey Diagram", font_size=10) + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Sankey diagram saved to " + outputHtml)""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala index 63fe2cacce..1df3d94e4e 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDesc.scala @@ -26,8 +26,10 @@ import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBui import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity import org.apache.texera.amber.operator.PythonOperatorDescriptor -import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName +import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode +import org.apache.texera.amber.operator.metadata.annotations.{AutofillAttributeName, SampleColumn} import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral import javax.validation.constraints.NotNull @@ -38,12 +40,13 @@ import javax.validation.constraints.NotNull * to construct and visualize an interactive, top-down tree that automatically * sizes itself and supports intuitive scroll/pinch zooming. */ -class TreePlotOpDesc extends PythonOperatorDescriptor { +class TreePlotOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode { @JsonProperty(value = "Edge List Column", required = true) @JsonSchemaTitle("Edge List Column") @JsonPropertyDescription("Column with [parent, child] pairs") @AutofillAttributeName + @SampleColumn("edge_pair") @NotNull(message = "Edge List Column cannot be empty") var edgeListColumn: EncodableString = "" @@ -72,8 +75,6 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | |import plotly.graph_objects as go |import plotly.io - |import igraph - |from igraph import Graph, EdgeSeq |import pandas as pd |import ast | @@ -108,6 +109,66 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | ) | return annotations | + | def build_tree_layout(self, edges): + | # Tidy top-down tree: depth picks the row, a leaf takes the next + | # free column and a parent sits centred over its own children. + | labels = [] + | known = set() + | for parent, child in edges: + | for node in (parent, child): + | if node not in known: + | known.add(node) + | labels.append(node) + | + | children = {label: [] for label in labels} + | has_parent = set() + | seen = set() + | for parent, child in edges: + | if (parent, child) not in seen: + | seen.add((parent, child)) + | children[parent].append(child) + | has_parent.add(child) + | + | depth = {} + | column = {} + | claimed = {} + | placed = set() + | next_column = 0 + | + | def grow(root): + | nonlocal next_column + | placed.add(root) + | stack = [(root, 0, False)] + | while stack: + | node, level, folded = stack.pop() + | if folded: + | kids = claimed[node] + | if kids: + | column[node] = sum(column[kid] for kid in kids) / len(kids) + | else: + | column[node] = next_column + | next_column += 1 + | continue + | depth[node] = level + | # A node belongs to whichever parent reaches it first, so a + | # cycle or a shared child is never laid out twice. + | kids = [kid for kid in children[node] if kid not in placed] + | placed.update(kids) + | claimed[node] = kids + | stack.append((node, level, True)) + | for kid in reversed(kids): + | stack.append((kid, level + 1, False)) + | + | for label in labels: + | if label not in placed and label not in has_parent: + | grow(label) + | # Whatever is left sits in a cycle that no root can reach. + | for label in labels: + | if label not in placed: + | grow(label) + | # The y-axis is inverted here so the tree grows top-down. + | return labels, [(column[label], -depth[label]) for label in labels] + | | @overrides | def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]: | if table.empty: @@ -127,14 +188,10 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | yield {'html-content': self.render_error("No valid [parent, child] pairs found in column " + $edgeListColumn + ".")} | return | - | G = Graph.TupleList(edges, directed=True) - | labels = G.vs['name'] - | - | layout_algorithm = 'rt' | try: - | lay = G.layout(layout_algorithm) + | labels, coords = self.build_tree_layout(edges) | except Exception as e: - | yield {'html-content': self.render_error(f"Layout algorithm '{layout_algorithm}' failed: {e}")} + | yield {'html-content': self.render_error(f"Tree layout failed: {e}")} | return | | HORIZONTAL_DENSITY = 120 @@ -143,8 +200,8 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | MIN_WIDTH = 800 | MIN_HEIGHT = 600 | - | if len(lay.coords) > 1: - | x_coords, y_coords = zip(*lay.coords) + | if len(coords) > 1: + | x_coords, y_coords = zip(*coords) | x_range = max(x_coords) - min(x_coords) | y_range = max(y_coords) - min(y_coords) | plot_width = max(MIN_WIDTH, x_range * HORIZONTAL_DENSITY + PADDING) @@ -153,12 +210,13 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | plot_width = MIN_WIDTH | plot_height = MIN_HEIGHT | - | # Invert the y-axis to make the tree grow top-down. - | position = {k: (lay[k][0], -lay[k][1]) for k in range(len(labels))} + | position = {k: coords[k] for k in range(len(labels))} + | index_of = {label: k for k, label in enumerate(labels)} | | Xe = [] | Ye = [] - | for edge in G.get_edgelist(): + | for parent, child in edges: + | edge = (index_of[parent], index_of[child]) | Xe += [position[edge[0]][0], position[edge[1]][0], None] | Ye += [position[edge[0]][1], position[edge[1]][1], None] | @@ -188,4 +246,182 @@ class TreePlotOpDesc extends PythonOperatorDescriptor { | |""".encode } + + override def producesDataFrame(): Boolean = false + + override def generateStandaloneCode(): String = { + s"""import ast + | + |# Only a layout failure renders an error page; everything else propagates, + |# matching the operator's own error handling. + |class TreeLayoutError(Exception): + | pass + | + |def render_error(error_msg): + | return f'''<h1>Tree Plot is not available.</h1> + | <p>Reason: {error_msg} </p>''' + | + |def make_annotations(pos): + | font_color = 'rgb(250,250,250)' + | node_color = '#6175c1' + | font_size = 10 + | annotations = [] + | for node_name, coords in pos.items(): + | annotations.append( + | dict( + | # The label goes in as it came out of the cell, the way the + | # operator passes it: plotly stringifies it, and a str() here + | # would turn a None node into the text "None". + | text=node_name, + | x=coords[0], + | y=coords[1], + | xref='x1', yref='y1', + | font=dict(color=font_color, size=font_size), + | showarrow=False, + | align='center', + | bordercolor='rgb(50,50,50)', + | borderwidth=1, + | borderpad=5, + | bgcolor=node_color, + | opacity=0.8 + | ) + | ) + | return annotations + | + |def build_tree_layout(edges): + | # Tidy top-down tree: depth picks the row, a leaf takes the next + | # free column and a parent sits centred over its own children. + | labels = [] + | known = set() + | for parent, child in edges: + | for node in (parent, child): + | if node not in known: + | known.add(node) + | labels.append(node) + | + | children = {label: [] for label in labels} + | has_parent = set() + | seen = set() + | for parent, child in edges: + | if (parent, child) not in seen: + | seen.add((parent, child)) + | children[parent].append(child) + | has_parent.add(child) + | + | depth = {} + | column = {} + | claimed = {} + | placed = set() + | next_column = 0 + | + | def grow(root): + | nonlocal next_column + | placed.add(root) + | stack = [(root, 0, False)] + | while stack: + | node, level, folded = stack.pop() + | if folded: + | kids = claimed[node] + | if kids: + | column[node] = sum(column[kid] for kid in kids) / len(kids) + | else: + | column[node] = next_column + | next_column += 1 + | continue + | depth[node] = level + | # A node belongs to whichever parent reaches it first, so a + | # cycle or a shared child is never laid out twice. + | kids = [kid for kid in children[node] if kid not in placed] + | placed.update(kids) + | claimed[node] = kids + | stack.append((node, level, True)) + | for kid in reversed(kids): + | stack.append((kid, level + 1, False)) + | + | for label in labels: + | if label not in placed and label not in has_parent: + | grow(label) + | # Whatever is left sits in a cycle that no root can reach. + | for label in labels: + | if label not in placed: + | grow(label) + | # The y-axis is inverted here so the tree grows top-down. + | return labels, [(column[label], -depth[label]) for label in labels] + | + |def compute_tree_layout(edges): + | try: + | labels, coords = build_tree_layout(edges) + | except Exception as e: + | raise TreeLayoutError(f"Tree layout failed: {e}") + | return {labels[index]: coords[index] for index in range(len(labels))} + | + |if in1df.empty: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("Input table is empty.")) + |else: + | edges = [] + | for item in in1df[${pyStringLiteral(edgeListColumn)}].dropna(): + | try: + | edge = ast.literal_eval(str(item)) + | if isinstance(edge, (list, tuple)) and len(edge) == 2: + | edges.append(list(edge)) + | except (ValueError, SyntaxError): + | pass + | + | if not edges: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error("No valid [parent, child] pairs found in column " + ${pyStringLiteral( + edgeListColumn + )} + ".")) + | else: + | try: + | position = compute_tree_layout(edges) + | + | HORIZONTAL_DENSITY = 120 + | VERTICAL_DENSITY = 120 + | PADDING = 200 + | MIN_WIDTH = 800 + | MIN_HEIGHT = 600 + | + | if len(position) > 1: + | x_coords, y_coords = zip(*position.values()) + | x_range = max(x_coords) - min(x_coords) + | y_range = max(y_coords) - min(y_coords) + | plot_width = max(MIN_WIDTH, x_range * HORIZONTAL_DENSITY + PADDING) + | plot_height = max(MIN_HEIGHT, y_range * VERTICAL_DENSITY + PADDING) + | else: + | plot_width = MIN_WIDTH + | plot_height = MIN_HEIGHT + | + | Xe = [] + | Ye = [] + | for parent, child in edges: + | Xe += [position[parent][0], position[child][0], None] + | Ye += [position[parent][1], position[child][1], None] + | + | fig = go.Figure() + | fig.add_trace(go.Scatter(x=Xe, y=Ye, mode='lines', + | line=dict(color='rgb(210,210,210)', width=1), + | hoverinfo='none')) + | axis = dict(showline=False, zeroline=False, showgrid=False, showticklabels=False) + | fig.update_layout(title='Tree Plot', + | width=int(plot_width), + | height=int(plot_height), + | annotations=make_annotations(position), + | font_size=12, + | showlegend=False, + | xaxis=axis, + | yaxis=axis, + | margin=dict(l=40, r=40, b=85, t=100), + | dragmode='pan', + | hovermode='closest', + | plot_bgcolor='rgb(248,248,248)') + | fig.write_json(outputJson) + | fig.write_html(outputHtml) + | print("Tree plot saved to " + outputHtml) + | except TreeLayoutError as e: + | with open(outputHtml, "w", encoding="utf-8") as output: + | output.write(render_error(str(e)))""".stripMargin + } + } diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/wordCloud/WordCloudOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/wordCloud/WordCloudOpDesc.scala index 2aa5b99d38..8a9fd112e0 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/wordCloud/WordCloudOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/wordCloud/WordCloudOpDesc.scala @@ -29,14 +29,25 @@ import org.apache.texera.amber.core.tuple.{AttributeType, Schema} import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString import org.apache.texera.amber.core.workflow.PortIdentity -import org.apache.texera.amber.operator.PythonOperatorDescriptor +import org.apache.texera.amber.operator.{PythonOperatorDescriptor, StandaloneCodeGenerator} import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} import org.apache.texera.amber.operator.visualization.ImageUtility import org.apache.texera.amber.pybuilder.PythonTemplateBuilder +import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral import javax.validation.constraints.NotNull -class WordCloudOpDesc extends PythonOperatorDescriptor { +// type constraint: the words are counted out of the column's text, and the +// filter that finds them uses pandas' .str accessor, which refuses a column of +// anything else. A word cloud of numbers would say nothing anyway. +@JsonSchemaInject(json = """ +{ + "attributeTypeRules": { + "textColumn": { "enum": ["string"] } + } +} +""") +class WordCloudOpDesc extends PythonOperatorDescriptor with StandaloneCodeGenerator { @JsonProperty(required = true) @JsonSchemaTitle("Text column") @AutofillAttributeName @@ -70,6 +81,21 @@ class WordCloudOpDesc extends PythonOperatorDescriptor { |""" } + // The `.str` accessor raises on anything but text, and the type rule that says + // so is only a warning: a column of numbers reaches here and ends the run with + // a pandas error naming an accessor the user never wrote. Both generators ask + // the same question and answer it in the same words, so the two paths still + // agree on what they produce. + private val NotTextMessage = "text column does not hold text." + + def guardNotText(): PythonTemplateBuilder = { + pyb""" + | if not (pd.api.types.is_object_dtype(table[$textColumn]) or pd.api.types.is_string_dtype(table[$textColumn])): + | yield {'html-content': self.render_error("$NotTextMessage")} + | return + |""" + } + def createWordCloudFigure(): PythonTemplateBuilder = { pyb""" | text = ' '.join(table[$textColumn]) @@ -90,6 +116,7 @@ class WordCloudOpDesc extends PythonOperatorDescriptor { override def generatePythonCode(): String = { pyb""" |from pytexera import * + |import pandas as pd | |class ProcessTableOperator(UDFTableOperator): | @@ -104,6 +131,7 @@ class WordCloudOpDesc extends PythonOperatorDescriptor { | if table.empty: | yield {'html-content': self.render_error("input table is empty.")} | return + | ${guardNotText()} | ${manipulateTable()} | if table.empty: | yield {'html-content': self.render_error("text column does not contain words or contains only nulls.")} @@ -113,4 +141,45 @@ class WordCloudOpDesc extends PythonOperatorDescriptor { | yield {'html-content': html} |""".encode } + + override def producesDataFrame(): Boolean = false + + // The two guards mirror generatePythonCode's: WordCloud.generate raises on a + // wordless string, so without them an input that is empty — or whose text + // column survives neither the dropna nor the word filter — crashes here where + // the runtime path explains itself. + override def generateStandaloneCode(): String = { + val textLit = pyStringLiteral(textColumn) + s"""def render_error(error_msg): + | return '''<h1>Wordcloud is not available.</h1> + | <p>Reason is: {} </p> + | '''.format(error_msg) + | + |table = in1df + |if table.empty: + | with open(outputHtml, "w", encoding="utf-8") as f: + | f.write(render_error("input table is empty.")) + |elif not (pd.api.types.is_object_dtype(table[$textLit]) or pd.api.types.is_string_dtype(table[$textLit])): + | with open(outputHtml, "w", encoding="utf-8") as f: + | f.write(render_error("$NotTextMessage")) + |else: + | table = table.dropna(subset=[$textLit]) + | table = table[table[$textLit].str.contains(r'\\w', regex=True)] + | if table.empty: + | with open(outputHtml, "w", encoding="utf-8") as f: + | f.write(render_error("text column does not contain words or contains only nulls.")) + | else: + | text = ' '.join(table[$textLit]) + | from wordcloud import WordCloud, STOPWORDS + | wordcloud = WordCloud(width=1920, height=1080, stopwords=set(STOPWORDS), max_words=$topN, background_color='white', include_numbers=True, random_state=0).generate(text) + | from io import BytesIO + | image_stream = BytesIO() + | wordcloud.to_image().save(image_stream, format='PNG') + | binary_image_data = image_stream.getvalue() + | import base64 + | encoded_image_str = base64.b64encode(binary_image_data).decode("utf-8") + | html = f'<img src="data:image;base64,{encoded_image_str}" alt="Image" style="max-width: 100vw; max-height: 90vh; width: auto; height: auto;">' + | with open(outputHtml, "w", encoding="utf-8") as f: + | f.write(html)""".stripMargin + } } diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDescSpec.scala index 9851e27596..e0ab0a25af 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/dendrogram/DendrogramOpDescSpec.scala @@ -81,7 +81,7 @@ class DendrogramOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers assert(carries(code, "coord_b")) assert(carries(code, "label_col")) code should include("create_dendrogram") - // empty threshold falls back to color_threshold=None + // An unset threshold arrives as None, scipy's own 0.7 * max distance. code should include("color_threshold=None") } diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDescSpec.scala index 68ea4e6087..f0d9f6f5e0 100644 --- a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDescSpec.scala +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/visualization/treeplot/TreePlotOpDescSpec.scala @@ -19,15 +19,23 @@ package org.apache.texera.amber.operator.visualization.treeplot +import com.typesafe.config.ConfigFactory +import org.apache.texera.amber.operator.tags.IntegrationTest import org.scalatest.BeforeAndAfter +import org.scalatest.Tag import org.scalatest.flatspec.AnyFlatSpec import org.scalatest.matchers.should.Matchers import java.nio.charset.StandardCharsets +import java.nio.file.Files import java.util.Base64 +import java.util.concurrent.TimeUnit +import scala.util.Try class TreePlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers { + private val NeedsPythonPackages = Tag(classOf[IntegrationTest].getName) + var opDesc: TreePlotOpDesc = _ before { @@ -56,6 +64,196 @@ class TreePlotOpDescSpec extends AnyFlatSpec with BeforeAndAfter with Matchers { val code = opDesc.generatePythonCode() assert(carries(code, "edge_pairs")) code should include("class ProcessTableOperator(UDFTableOperator)") - code should include("Graph.TupleList") + code should include("self.build_tree_layout(edges)") + } + + // igraph is GPL v2, Category X under the ASF 3rd party license policy, so the + // layout has to stay something the repository can actually ship. + it should "not reach for igraph in either generated path" in { + opDesc.edgeListColumn = "edge_pairs" + opDesc.generatePythonCode().toLowerCase should not include "igraph" + opDesc.generateStandaloneCode().toLowerCase should not include "igraph" + } + + // Python executable resolution, following FilledAreaPlotOpDescSpec: + // udf.conf python.path (UDF_PYTHON_PATH), then python3 / python / py. + private def resolvePythonExecutable(): Option[String] = { + def fromConfig: Option[String] = { + val configOpt = + Try(ConfigFactory.parseResources("udf.conf").resolve()).toOption + .orElse(Try(ConfigFactory.load()).toOption) + configOpt + .flatMap(c => Try(c.getConfig("python").getString("path")).toOption) + .map(_.trim) + .filter(_.nonEmpty) + } + + def isRunnable(exe: String): Boolean = { + val pTry = Try(new ProcessBuilder(exe, "--version").redirectErrorStream(true).start()) + pTry.toOption.exists { p => + val finished = p.waitFor(5, TimeUnit.SECONDS) + if (!finished) { p.destroyForcibly(); false } + else p.exitValue() == 0 + } + } + + (fromConfig.toList ++ List("python3", "python", "py")).distinct.find(isRunnable) + } + + private def canImportPandasAndPlotly(python: String): Boolean = { + val pTry = Try( + new ProcessBuilder(python, "-c", "import pandas, plotly").redirectErrorStream(true).start() + ) + pTry.toOption.exists { p => + val finished = p.waitFor(60, TimeUnit.SECONDS) + if (!finished) { p.destroyForcibly(); false } + else p.exitValue() == 0 + } + } + + // Driver executed by the runtime test below. It stubs only the pytexera import seam; + // the generated module runs unmodified, and the layout is called directly so the + // positions it computes are what is read, rather than a rendered picture. + private val runtimeDriverScript: String = + """import base64 + |import sys + |import types + |from typing import Iterator, Optional + | + |import pandas as pd + | + |class UDFTableOperator: + | def decode_python_template(self, data): + | return base64.b64decode(data).decode("utf-8") + | + |stub = types.ModuleType("pytexera") + |stub.UDFTableOperator = UDFTableOperator + |stub.overrides = lambda fn: fn + |stub.Table = pd.DataFrame + |stub.TableLike = object + |stub.Iterator = Iterator + |stub.Optional = Optional + |sys.modules["pytexera"] = stub + | + |ns = {"__name__": "generated_tree_plot"} + |with open(sys.argv[1]) as f: + | exec(compile(f.read(), sys.argv[1], "exec"), ns) + |op = ns["ProcessTableOperator"]() + | + |cases = [ + | ("tree", [("a", "b"), ("a", "c"), ("b", "d"), ("b", "e")]), + | ("shared", [("a", "c"), ("b", "c")]), + | ("cycle", [("a", "b"), ("b", "c"), ("c", "a")]), + | ("selfloop", [("a", "a"), ("a", "b")]), + | ("forest", [("a", "b"), ("x", "y")]), + |] + | + |for cid, edges in cases: + | labels, coords = op.build_tree_layout(edges) + | placed = " ".join( + | "%s=%g,%g" % (label, x, y) for label, (x, y) in zip(labels, coords) + | ) + | print("CASE %s %s" % (cid, placed)) + |""".stripMargin + + /** `label -> (x, y)` per case, read out of the driver's output. One run for the + * whole suite: the driver lays every case out in the same process. + */ + private lazy val layouts: Map[String, Map[String, (Double, Double)]] = { + val python = resolvePythonExecutable().getOrElse( + cancel("No runnable python executable (udf.conf python.path, python3, python, py)") + ) + if (!canImportPandasAndPlotly(python)) { + cancel(s"'$python' cannot import pandas and plotly; skipping runtime verification") + } + + opDesc.edgeListColumn = "edge_pairs" + val moduleFile = Files.createTempFile("tree_plot_op_", ".py") + val driverFile = Files.createTempFile("tree_plot_driver_", ".py") + try { + Files.write(moduleFile, opDesc.generatePythonCode().getBytes(StandardCharsets.UTF_8)) + Files.write(driverFile, runtimeDriverScript.getBytes(StandardCharsets.UTF_8)) + + val process = new ProcessBuilder(python, driverFile.toString, moduleFile.toString) + .redirectErrorStream(true) + .start() + val finished = process.waitFor(120, TimeUnit.SECONDS) + if (!finished) { + process.destroyForcibly() + fail("Layout driver timed out after 120s") + } + val output = new String(process.getInputStream.readAllBytes(), StandardCharsets.UTF_8) + withClue(s"Driver output:\n$output\n") { + process.exitValue() shouldBe 0 + } + "CASE (\\S+) (.*)".r + .findAllMatchIn(output) + .map { m => + m.group(1) -> m + .group(2) + .trim + .split(" ") + .filter(_.nonEmpty) + .map { entry => + val Array(label, xy) = entry.split("=", 2) + val Array(x, y) = xy.split(",", 2) + label -> (x.toDouble, y.toDouble) + } + .toMap + } + .toMap + } finally { + Files.deleteIfExists(moduleFile) + Files.deleteIfExists(driverFile) + } + } + + it should "lay a tree out top-down, with every parent centred over its own children" taggedAs NeedsPythonPackages in { + val tree = layouts("tree") + // Depth picks the row and the axis is inverted, so a child sits below its parent. + tree.map { case (label, (_, y)) => label -> y } shouldBe + Map("a" -> 0.0, "b" -> -1.0, "c" -> -1.0, "d" -> -2.0, "e" -> -2.0) + // Leaves take consecutive free columns left to right, in the order they are reached. + tree("d")._1 shouldBe 0.0 + tree("e")._1 shouldBe 1.0 + tree("c")._1 shouldBe 2.0 + // A parent sits at the mean of its own children: b over d and e, a over b and c. + tree("b")._1 shouldBe (tree("d")._1 + tree("e")._1) / 2 + tree("a")._1 shouldBe (tree("b")._1 + tree("c")._1) / 2 + } + + it should "place a shared child once, under the parent that reaches it first" taggedAs NeedsPythonPackages in { + val shared = layouts("shared") + shared.keySet shouldBe Set("a", "b", "c") + // c is claimed by a, so it hangs below a and b is left as a childless root. + shared("c") shouldBe (0.0, -1.0) + shared("a") shouldBe (0.0, 0.0) + shared("b")._2 shouldBe 0.0 + shared("b")._1 should not be shared("a")._1 + } + + it should "terminate on a cycle and still place every node once" taggedAs NeedsPythonPackages in { + val cycle = layouts("cycle") + // No node is a root, so the layout starts from the first label it saw and the + // edge that closes the ring is dropped rather than followed a second time. + cycle shouldBe Map("a" -> (0.0, 0.0), "b" -> (0.0, -1.0), "c" -> (0.0, -2.0)) + } + + it should "survive a self loop" taggedAs NeedsPythonPackages in { + val selfLoop = layouts("selfloop") + selfLoop.keySet shouldBe Set("a", "b") + selfLoop("a")._2 shouldBe 0.0 + selfLoop("b")._2 shouldBe -1.0 + } + + it should "lay each tree of a forest out beside the other" taggedAs NeedsPythonPackages in { + val forest = layouts("forest") + forest.keySet shouldBe Set("a", "b", "x", "y") + // Two roots, each over its own child, in columns that do not overlap. + forest("a")._2 shouldBe 0.0 + forest("x")._2 shouldBe 0.0 + forest("a")._1 should not be forest("x")._1 + forest("b") shouldBe (forest("a")._1, -1.0) + forest("y") shouldBe (forest("x")._1, -1.0) } }
