kz930 opened a new pull request, #7668:
URL: https://github.com/apache/texera/pull/7668
### What changes were proposed in this PR?
`Text Attribute` named one column, so a dataset whose text is spread over
several could not be used as it stands. A title beside a body, or a description
beside a comment, is ordinary, and the only way through was a Concat upstream,
which merges the columns into one string and gives up which column each word
came from.
Naming several columns was not an alternative either. `CountVectorizer`
takes a flat sequence of documents, so a two-column frame reaches it as two
documents rather than as the rows, and it learns a vocabulary of the column
names:
```
>>> CountVectorizer().fit(df[["title", "body"]]).vocabulary_
{'body': 0, 'title': 1}
```
The field now takes a list, and the generated pipeline opens with a
`ColumnTransformer` that gives each named column its own `CountVectorizer` and
concatenates the results, so a word keeps the column it came from through the
feature's prefix:
```python
make_pipeline(ColumnTransformer([("text0", CountVectorizer(), <col>),
("text1", CountVectorizer(), <col>)]),
TfidfTransformer(), estimator()).fit(X, Y)
```
The steps are named by position rather than after the column, which keeps a
column whose name carries a double underscore away from the separator
`get_feature_names_out` puts between step and feature. The transformer names
the columns it reads, so the frame no longer has to be narrowed before the
pipeline and that line goes.
`Tfidf Transformer` stays one switch over the whole matrix, leaving the
shape of the pipeline as it was. Turning `Count Vectorizer` on still discards
the numeric columns, as before: carrying them along needs a way to say which
columns are features, which these two families do not have, and that is a
separate change.
The operator's output is unchanged, one row of `model_name` and `model`. A
workflow written before this holds a bare string in the field, which
`ACCEPT_SINGLE_VALUE_AS_ARRAY` reads as a list of one, so nothing has to be
migrated.
The field is declared on `SklearnModelOpDesc`, so this reaches all fifty-one
operators of the Sklearn and Sklearn Training groups.
### Any related issues, documentation, discussions?
Closes #7667
This touches the same line of the two codegen templates as #7645, which
rewrites it while this deletes it. Whichever merges second needs that line
resolved by hand: the `else` branch #7645 introduces is kept, and the `if`
branch this removes stays removed.
### How was this PR tested?
`SklearnClassifierOpDescCodegenSpec` and `SklearnTrainingOpDescCodegenSpec`
pinned the generated pipeline and were updated to the new one, each gaining a
case for several columns getting a vectorizer apiece. The fifty-one
per-operator specs that pinned the field's default were updated to the empty
list. `WorkflowOperator/test` passes: 2287 tests.
The emitted line was also run as it stands, with the base64 decoding
stubbed, against a frame of two text columns and a numeric one: it fits, and
the feature names carry the `text0__` and `text1__` prefixes.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
