carloea2 commented on code in PR #8459: URL: https://github.com/apache/texera/pull/8459#discussion_r4310032980
########## docs/contribution-guidelines/guide-to-add-a-new-operator.md: ########## @@ -0,0 +1,303 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +--- +title: "Guide to Add a New Operator" +weight: 60 +--- + +Writing the operator itself is covered by the +[Java/Scala](/docs/contribution-guidelines/guide-to-implement-java-operator/) and +[native Python](/docs/contribution-guidelines/guide-to-implement-python-operator/) guides. +This page lists what else a new operator needs. It is done when all five hold. + +| # | Deliverable | Where | +| --- | --- | --- | +| 1 | Descriptor, executor, registration, icon | `common/workflow-operator/.../operator/<pkg>/` | +| 2 | A stated constraint on every config property | annotations on the descriptor's fields | +| 3 | `generateStandaloneCode()`, or its `inputSchemas` overload | the descriptor, via `StandaloneCodeGenerator` | +| 4 | A green verification run | `OperatorBehaviorSpec` | +| 5 | Unit spec, formatted and linted | the operator's own `*OpDescSpec` | + +## 1. Descriptor, executor, registration + +- Write the `OpDesc` and the `OpExec`. +- Declare the columns each output port carries: `getOutputSchemas` on a Python operator, a + `SchemaPropagationFunc` inside `getPhysicalOp` on a Java or Scala one. A schema that + disagrees with what the executor emits is not caught at compile time, and not always at + run time either. The row is matched to the schema by name, so a column the two name + apart can lose its value without an error. +- Register the descriptor in `LogicalOp`'s `@JsonSubTypes` under a unique name. This is the + only registration there is: the form, the translator and the verification harness all read + that list. +- Add `frontend/src/assets/operator_images/<That Same Name>.png`. + +## 2. Configuration rules + +Every `@JsonProperty` becomes a field in the operator's form, labelled by `@JsonSchemaTitle` +and explained by `@JsonPropertyDescription`. State the operator's real constraints beside it. + +### Columns + +| Annotation | Meaning | +| --- | --- | +| `@AutofillAttributeName` | one column name from input port 0 | +| `@AutofillAttributeNameList` | a list of column names from input port 0 | +| `@AutofillAttributeNameOnPort1` | one column name from input port 1 | +| `@SampleColumn("iso_country")` | test-only: which fixture column should fill this field | + +- Column types go in a class-level `@JsonSchemaInject` carrying `attributeTypeRules`, keyed by + field name. +- A type rule warns, it does not filter. The dropdown still lists every column. +- Give a rule to columns the operator computes on. Leave labels, grouping keys, facets and + hover names unconstrained: that a string column did not crash is not a reason to add one. + +### Numbers + +- Bounds go on the field, as a `@JsonSchemaInject` carrying `minimum` and `maximum`. A lower + bound may come from `@DecimalMin` instead; an upper bound is read from the schema alone. +- An optional numeric field also needs `@JsonDeserialize(contentAs = ...)` naming its boxed + class, or a blank reads as 0. + +### Free-form strings + +- Always give a default value, and an `examples` entry holding one realistic value. The form + does not render `examples`, but the verification generator fills the field with it. +- Add a `pattern` only where the consumer really constrains the input, and copy the rule from + that consumer's source instead of writing one from memory. +- Keep the pattern anchored and loose: rejecting a value the operator would have accepted is + a bug. Two engines read it, the browser's and the verification generator's, and the second + matches whole strings, so an unanchored pattern does not mean the same thing to both. + +### Conditional fields + +- `HideAnnotation` hides a field behind another field's value. +- To constrain what a field may hold given what a sibling holds, use Texera's `valueRules` + key, not a JSON-Schema `allOf`: the form builder merges an `allOf`'s branches into one + field, leaving a single control carrying every branch's constraints at once. The + verification generator reads `valueRules` too, and fills the field from the branch that + applies. + +## 3. Standalone Python code (required) + +Texera exports a workflow as one runnable script, and each operator contributes its own +fragment. Without one, the export emits a `# TODO:` comment in its place. + + + +- Mix in `StandaloneCodeGenerator` and implement `generateStandaloneCode()`. +- Read `in1df`, `in2df`, … and write `out1df`, `out2df`, …, one per declared port. +- Never write to an input frame. The translator names a variable per output port, so two + operators reading one upstream are handed the same name, and `inplace=True` or + `in1df = in1df.dropna()` changes what the other branch goes on to read. +- pandas is imported for every script. Anything else is named by `standaloneImports()`, so a + script that draws nothing does not need a plotting library to start; a chart gets the + plotly imports by mixing in `PlotlyStandaloneCode`. +- Override `producesDataFrame()` to false for a visualization whose output is HTML. +- Put definitions the fragment refers to in `standaloneHelpers()`. They are deduplicated + across the plan. +- Pass every user value through `pyStringLiteral`, or `pyb` with an `EncodableString`. Spliced + in directly, a column named `a"b` closes the literal early and breaks the whole script. +- A `pyb` field has to reach the template whole. Joining it to anything in Scala first — + `s"${attribute}_bin"` — hands `pyb` a plain string and the protection is gone, so derive + such a name in the Python instead. `PythonCodeRawInvalidTextSpec` reports the leak. +- Rendering a column as TEXT takes the declared type, not the value's. A file carries no + types, so a hole makes pandas read an integer column as a float and a boolean one as 1.0 + and 0.0, and 6 renders as "6.0" where the engine wrote "6". Override the + `generateStandaloneCode(inputSchemas)` overload and pass the column through + `renderedAsText`; `None` there means read it as it arrives. +- A TIMESTAMP is a wall clock and carries no zone, so a count from the epoch is read with + UTC arithmetic on both sides. `new Timestamp(millis)` instead reads it in whatever zone the + machine is set to, and pandas reads the same number as UTC, so the two paths part by the + local offset and agree again on a machine set to UTC. `ArrowUtils` states the convention; + follow it wherever an operator turns a number into a moment. Where an existing executor + already uses the machine's zone, as TypeCasting and Aggregate do, call + `_texera_local_wall_clocks` and `_texera_local_zone` rather than `tzlocal()` or `gettz()`, + which know only today's offsets. Java reads a moment before 1900 at the standard offset. Review Comment: [P3] gettz() can use historical rules on Unix, unlike Windows local-zone fallbacks. Please qualify this sentence rather than say it always knows only current offsets. The recommendation to use the shared IANA helper is fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
