kz930 opened a new issue, #7582:
URL: https://github.com/apache/texera/issues/7582
### What happened?
Every Sklearn operator ends the execution when any cell it reads is empty.
The error comes from inside scikit-learn's input validation rather than from
the operator: `ValueError: Input X contains NaN.`
No operator in the family looks at missing values. Searching `sklearn/` and
`machineLearning/` for `dropna`, `fillna`, `isna` or `SimpleImputer` returns
nothing. The generated Python takes the columns and fits:
`SklearnTrainingOpDesc.scala:45` is `X = table.drop(target, axis=1)` followed
by `.fit(X, Y)`, and `SklearnClassifierOpDesc.scala:45`,
`SklearnTestingOpDesc.scala:74` and `SklearnAdvancedBaseDesc.scala:158` all do
the same.
An empty value is ordinary input here. A blank CSV cell arrives as null,
since univocity returns null for an empty field and
`AttributeTypeUtils.parseField` passes it through by design.
Two things make this worse than a strict estimator refusing bad input.
The message names nothing the user can act on. `X` is a variable inside
generated code, so the error identifies neither the column nor the row. The
user is told their data contains NaN somewhere.
The blast radius is larger than the user's idea of their model. The training
and classifier operators feed every column except the target into the
estimator, so a blank in a note or source column that has nothing to do with
the model ends the run just the same. Only the Advanced Sklearn operators read
a named list of features, and they crash on those.
Two neighbouring cases fail the same way, for the same reason. A blank in
the target column gives `ValueError: Input y contains NaN.` A blank in the text
column with `Count Vectorizer` enabled gives `AttributeError: 'NoneType' object
has no attribute 'lower'` from inside CountVectorizer.
Expected: a blank cell is skipped, which is what the rest of the codebase
does with a value that is not there. Twenty-four visualization operators open
their generated Python with `dropna(subset=[...]) #remove missing values`,
`COUNT(column)` counts only non-null rows, CONCAT and MIN pass over them, and
`FilterPredicate` answers false for every condition but IS_NULL / IS_NOT_NULL.
The Sklearn family is the only one with no answer at all.
What skipping should mean differs by what the operator emits, and it is
worth settling in one place:
| Operators | Output | Columns read | Skipping means |
| --- | --- | --- | --- |
| `SklearnTrainingOpDesc` subclasses | one row holding the model | every
column but the target | drop rows missing any of them, or the text and target
pair when Count Vectorizer is on |
| `SklearnClassifierOpDesc` subclasses | one row holding the model, metrics
printed | every column but the target, on both ports | the same, on both ports |
| `SklearnTestingOpDesc` | one row per model, with metric columns | every
column but the target | drop rows before scoring |
| `SklearnAdvancedBaseDesc` subclasses | one row per parameter combination |
the named features and the ground truth | drop rows missing any of those |
| `SklearnPredictionOpDesc` | each input row, plus a result column | every
column but the ground truth | keep the row and leave the result empty |
The last row differs deliberately. The prediction operator adds a column to
the user's rows, so dropping would take the user's row out of the output along
with the value the model had nothing to say about. That is what the Hugging
Face inference operators already do with a row they cannot process.
One more detail worth knowing when fixing this: the training and classifier
operators are blocking, so the whole table is consumed before the estimator is
fitted. The error therefore arrives at the end, after every upstream operator
has done its work.
### How to reproduce?
Upload a CSV with a blank cell:
```
x1,x2,y
0.1,0.2,0
0.3,,0
0.5,0.6,1
0.7,0.8,1
```
Build `CSV File Scan` to `Training: Bernoulli Naive Bayes` from the Sklearn
Training group, set Target Attribute to `y`, and run. The execution stops and
the operator's console shows the ValueError. Every other operator in the
Sklearn, Sklearn Training and Advanced Sklearn groups behaves the same way on
the same file.
Note that the scan reads all four rows and hands them on. Nothing goes wrong
until the estimator is fitted.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Relevant log output
```shell
Traceback (most recent call last):
File "core/runnables/data_processor.py", line 77, in
process_internal_marker
self._set_output_tuple(executor.on_finish(port_id))
File "core/runnables/data_processor.py", line 125, in _set_output_tuple
for output in output_iterator:
File "core/models/operator.py", line 280, in on_finish
yield from self.process_table(table, port)
File "udf-v1.py", line 12, in process_table
model = make_pipeline( BernoulliNB()).fit(X, Y)
File "sklearn/pipeline.py", line 663, in fit
self._final_estimator.fit(Xt, y, **last_step_params["fit"])
File "sklearn/naive_bayes.py", line 735, in fit
X, y = self._check_X_y(X, y)
File "sklearn/utils/validation.py", line 169, in
_assert_all_finite_element_wise
ValueError: Input X contains NaN.
BernoulliNB does not accept missing values encoded as NaN natively. For
supervised learning, you might want to consider
sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept
missing values encoded as NaNs natively. Alternatively, it is possible to
preprocess the data, for instance by using an imputer transformer in a pipeline
or drop samples with missing values.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
