sshiv012 commented on code in PR #8737: URL: https://github.com/apache/texera/pull/8737#discussion_r4140963074
########## docs/tutorials/guide-to-use-python-udf/index.md: ########## @@ -0,0 +1,210 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +--- +title: "Guide to Use a Python UDF" +weight: 30 +--- + +A **Python UDF** (user-defined function) is an operator that runs a small piece of Python code you write. Use it when no built-in operator does what you need. + +You don't need to be a programmer. Copy the example below and change one line at a time. + +## Which Python operator should I use? + +| You want to… | Use | +|---|---| +| Add or change a column with a one-line formula, e.g. `tuple_["Units Sold"] * 2` | **Python Lambda Function** (no class to write) | +| Turn a whole table into one summary row, e.g. `table["Units Sold"].mean()` | **Python Table Reducer** (no class to write) | +| Anything else with one or more inputs | **Python UDF** | +| Apply a model (input 0) to data rows (input 1) | **2-in Python UDF** | +| Create data from nothing (no input) | **1-out Python UDF** | + +## Your first UDF in 4 steps + +This example adds a `Revenue` column (units × price) to the sales data from [the dataset tutorial](../create-dataset-upload-data/). + +**1. Add the operator.** Drag **Python UDF** (under *User-defined Functions → Python*) onto the canvas and connect your data to it. + +**2. Write the code.** Click **Edit code content**. The editor opens with a template. Its classes are commented out (their lines start with `#`), so running it unchanged fails. Uncomment one class, or replace everything with: + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + tuple_["Revenue"] = tuple_["Units Sold"] * tuple_["Unit Price"] + yield tuple_ +``` + + + +**3. Declare the new column.** In the property panel, under **Extra output column(s)**, click **+** and add `Revenue` with type `double`. Leave **Retain input columns** checked to keep the original columns. + +<img src="udf-output-columns.png" alt="Retain input columns and one extra output column named Revenue" width="260"> + +**4. Run.** Click the eye icon on the UDF, then **Run**. Click the UDF, then the **Result** tab in the result panel. + +## The 5 rules + +Almost every UDF problem comes from breaking one of these. + +| # | Rule | If you break it | +|---|---|---| +| 1 | The first line is `from pytexera import *` | `name 'UDFOperatorV2' is not defined` | +| 2 | Exactly **one** class. Delete or comment out the others. | `There should be one and only one Operator defined` | +| 3 | The method name must match the class you inherit from (table below) | `... No super class method found`, or rule 2's error | +| 4 | Send results with `yield`, never `return` | ``process_tuple must `yield` results, not `return` them`` | +| 5 | The columns you yield must match the output columns (step 3) | `expected but missing` or `unexpected field` | + +## How your code receives data + +Pick one row of this table. The class name in brackets and the method name must go together. + +| Class `(…)` | Method you write | You receive | Good for | +|---|---|---|---| +| `UDFOperatorV2` | `process_tuple(self, tuple_, port)` | one row at a time (use it like a dict: `tuple_["col"]`) | per-row formulas, filtering | +| `UDFBatchOperator` | `process_batch(self, batch, port)` | `BATCH_SIZE` rows at a time, as a pandas DataFrame | calling an API in chunks | +| `UDFTableOperator` | `process_table(self, table, port)` | **all** rows of one input, as a pandas DataFrame | sorting, statistics, ML training | +| `UDFSourceOperator` (1-out UDF only) | `produce(self)` | nothing; you create the data | generating or downloading data | + +What you can `yield`: + +- **A row**: a `Tuple` or a `dict`, e.g. `yield {"name": "a", "value": 1}`. +- **A table**: a pandas `DataFrame` or `Table`. Yield the DataFrame itself, not a dict that contains DataFrames. +- **`None`**: produce nothing for this input. + +To drop a row, just don't `yield` it. To output several rows, `yield` several times. + +## One-time setup (loading a model, opening a file) + +Put setup code in `open()`, which runs once before any data arrives: + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def open(self): + self.threshold = 100 + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + if tuple_["Units Sold"] > self.threshold: + yield tuple_ +``` + +If you write `__init__` instead, make its first line `super().__init__()`. Older Texera versions fail without it (see the table below). + +To set values like `threshold` from the property panel instead of in code, see [UI parameters](/docs/reference/operators/user-defined-functions/python/#ui-parameters). + +## Output columns + +The UDF's output columns are **Retain input columns** (the input's columns, if checked) plus **Extra output column(s)**. Every row you `yield` must contain exactly these columns. + +- Adding a column? Add it under **Extra output column(s)**. +- Returning a completely new table (e.g. a summary)? Uncheck **Retain input columns** and list every output column. +- With several inputs, **Retain input columns** keeps the columns of input 0 (for the 2-in UDF: the `tuples` input, port 1). + +## Several inputs + +Add input ports with the **+** under the operator's left side. Ports are numbered from 0. Your method's `port` argument tells you which input a row came from. + +- **Inputs arrive in no fixed order.** Don't assume input 0 finishes before input 1. +- To combine inputs safely, use `UDFTableOperator`. It calls `process_table` once per input, when that input is complete. Keep each table and act when you have all of them: + +```python +from pytexera import * + +NUM_INPUTS = 2 + + +class ProcessTableOperator(UDFTableOperator): + + @overrides + def open(self): + self.tables = {} + + @overrides + def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]: + self.tables[port] = table + if len(self.tables) == NUM_INPUTS: + yield self.tables[0].merge(self.tables[1], on="Country") +``` + +- To force an order, click an input port's dot. Under **dependencies**, list the ports that must finish first. For example, port 1 with dependencies `0` waits for port 0. +- The **2-in Python UDF** has this built in: port 0 (`model`) is read completely before port 1 (`tuples`) starts. + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + if port == 0: # model input: remember the model + self.model = tuple_["model"] + else: # tuples input: use it + tuple_["pred"] = self.model.predict([tuple_["text"]])[0] + yield tuple_ +``` + +## Creating data (1-out Python UDF) + +```python +from pytexera import * + + +class GenerateOperator(UDFSourceOperator): + + @overrides + def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]: + for i in range(10): + yield {"number": i} +``` + +List every column (here `number`, type `integer`) under **Columns** in the property panel. More examples: [pytexera examples](https://github.com/apache/texera/tree/main/amber/src/main/python/pytexera/udf/examples). + +## Finding and fixing errors + +- `print(...)` in your code shows up in the **Console** tab. Click the UDF to see it. +- If a row fails, the UDF turns **magenta** (paused) and the error appears in its **Console**. Fix the code, then stop the run (red ⊗ next to *Pause*) and run again. + + Review Comment: Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR ########## docs/tutorials/guide-for-how-to-use-texera/index.md: ########## @@ -0,0 +1,117 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. +--> + +--- +title: "Guide for how to use Texera" +weight: 10 +--- + +Texera lets you analyze data by connecting boxes (**operators**) on a canvas instead of writing a program. Each box does one step, like "read a file", "keep rows where…", or "compute an average". The arrows carry data from one box to the next. + +## Sign in + +Open the Texera address you were given in your browser and sign in. + +On a Texera you installed yourself ([Docker guide](/docs/getting-started/installing-using-docker/)), the built-in admin account is username `texera`, password `texera`. + + + +## The dashboard + +After signing in you see the dashboard. The left menu has: + +- **Hub**: workflows and datasets other people have made public. +- **Your Work** + - **Workflows**: your workflows. Click **Create Workflow** to start one. + - **Datasets**: your data files ([how to upload](../create-dataset-upload-data/)). + - **Compute**: your computing units (see [below](#run-a-workflow)). + - **Environments**: extra Python packages for Python operators. + - **Quota**: how much storage and compute you have used. +- **Admin**: users, executions and site settings. Admins only. + +## The workflow editor + + + +1. **Operators**: all available boxes, grouped by type. If the panel is closed, click the grid icon at the top left of the canvas. Drag a box onto the canvas, or type its name in the search box. Boxes added through search land on top of each other; drag them apart. +2. **Canvas**: connect the output dot on the right of one box to the input dot on the left of the next. +3. **Property panel**: click a box to set its options. A red outline means an option is missing or wrong. +4. **Result panel**: shows a box's output, printed messages and errors. If it's closed, open it with the square icon at the bottom left. + +## Run a workflow + +A **computing unit** is the machine that runs your workflow. Pick one in the top bar, then click **Run**. If the list is empty, click **+ Computing Unit** to create one. + + + +- One computing unit can run all your workflows. Reuse it. +- A new unit usually won't fix an error in the workflow itself (code or settings). It can help when the run needs more memory, CPU or GPU, or different Python packages. +- Stopping (terminating) a unit deletes the results it holds. Your workflows and datasets are safe. + +More in [Computing Units](../computing-units/). + +## Read a running workflow + +Each box's outline color shows its state. The number on each side is how many rows went in (left) and came out (right). + + + +| Color | Meaning | +|---|---| +| Gray | Not started yet | +| Yellow-green | Ready to start | +| Orange | Running | +| Magenta | Paused. You paused it, or a row caused an error | +| Green | Finished | + +Some operators (e.g. Sort, Aggregate, and the build side of Join) must read **all** their input before they output anything. They show 0 rows out until the operator before them finishes. That's normal. See [How Texera runs your workflow](/docs/concepts/how-texera-runs-your-workflow/). + +## See the results + +1. Before running, click the box and turn on its **eye icon** (top toolbar). Texera shows results only for boxes with the eye on. +2. After the run, click the box and open the result panel (square icon, bottom left) if it's closed. The **Result** tab shows its rows. + +The result panel only shows the box you clicked. If it says *No results available to display*, click a box that has the eye on. + +## Where to find errors + +- **Click an empty spot on the canvas.** If any operator failed, the result panel shows a **Static Error** tab with errors from every operator, including code that failed to start. A run that stops right away with every box at 0 rows usually means this. + + Review Comment: Thank you for reviewing this, some images were outdated and wrongly pushed. I will check and update it in the revised PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
