The GitHub Actions job "Build and push images" on texera.git/main has failed. Run started by GitHub user bobbai00 (triggered by bobbai00).
Head commit for run: 604f10967b368a5a34ae41649e5ecc915f2c9a5c / Kary Zheng <[email protected]> feat(sklearn): answer an empty cell by what the estimator can do with it (#7583) ### What changes were proposed in this PR? Most Sklearn operators ended the execution when a cell they read was empty, with `ValueError: Input X contains NaN.` raised by scikit-learn's own input validation rather than by the operator. Some did not: measured against the pinned scikit-learn 1.7.2, twenty of the twenty-six estimators this family wraps refuse a NaN, and six fit on it silently and train on the incomplete rows. So this does not answer with a single rule. What happens to an incomplete row follows what the estimator can do with it, and either way the count is reported. | Operators | Output | Change | | --- | --- | --- | | `SklearnTrainingOpDesc` and `SklearnClassifierOpDesc` subclasses, estimator refuses a NaN | one row holding the model | drop on the whole table, or on the text and target pair when Count Vectorizer is on | | the same, estimator places a NaN itself | one row holding the model | drop on the target alone, keeping the blank features | | `SklearnLinearRegressionOpDesc` | one row holding the model | drop on the whole table | | `SklearnTestingOpDesc` | one row per model, with metric columns | drop before scoring | | `SklearnAdvancedBaseDesc` subclasses | one row per parameter combination | drop on the named features and the ground truth | | `SklearnPredictionOpDesc` | each input row, plus a result column | keep the row, leave the result empty | `handlesMissingValues` on `SklearnModelOpDesc` says which half an estimator is in, and the twelve tree and dummy descriptors override it. The split is not arbitrary. A tree only compares, so it can ask whether a value is present before it asks how large it is and send the whole missing group down one branch. A linear model computes `w1*x1 + w2*x2 + b`, which one NaN poisons end to end. Dropping for the trees threw away rows they could have fitted on, and a blank is often informative rather than noise. Those estimators still drop on the target, because every one of them including a tree refuses a NaN there, and Dummy would otherwise learn the blank as a class of its own. Count Vectorizer still takes the text column whatever the estimator, because it calls `.lower()` on each document. Both outcomes are printed beside the metrics, either the number of rows skipped or the number kept for the model to place. Dropping rows changes the data the model was asked to learn from, and a user who is not told cannot know it happened. Prediction keeps the row deliberately. It adds a column to the user's rows, so dropping would take the row out of the output along with the value the model had nothing to say about. Keeping is also the reversible choice, since a downstream Filter can still remove them. Two further fixes in that operator. It tested the whole tuple for emptiness rather than the features it predicts on, so a blank in the column the user asked it to ignore cost the row its prediction. And it cast the result through `type(ground truth)`, which is `NoneType` when that column is the blank one. `SklearnLinearRegressionOpDesc` builds its own pipeline instead of inheriting the classifier base's, so the first pass over this family missed it entirely and it still ended the run on a blank cell. ### Any related issues, documentation, discussions? Closes #7582 ### How was this PR tested? Each changed operator gained a case in its existing spec asserting the generated Python, next to the cases already asserting on that output, including one that a tree does not drop on its features and one that the emptiness test reads the features rather than the whole row. The `sklearn` and `machineLearning` specs pass, 447 tests. The behaviour was checked against the reproduction in the issue, a four-row CSV with one blank cell. Before this change Bernoulli Naive Bayes ended the execution and Decision Tree trained silently on the blank. The generated Python was printed for a tree, a non-tree, a tree with Count Vectorizer on and the prediction operator, and the generated logic was run against scikit-learn 1.7.2 directly: the tree fits on all four rows and reports one kept, the non-tree fits on three and reports one skipped. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Report URL: https://github.com/apache/texera/actions/runs/33228174811 With regards, GitHub Actions via GitBox
