kz930 opened a new issue, #7667:
URL: https://github.com/apache/texera/issues/7667
### Feature Summary
The Sklearn and Sklearn Training operators vectorize one text column. `Text
Attribute` names a single column, and the generated pipeline hands that column
to a `CountVectorizer`, so a dataset whose text is spread over more than one
column cannot be used as it stands.
A review that carries a title and a body, or a record with a description
beside a comment, is ordinary. Today the only way through is a Concat upstream,
which merges the columns into one string and gives up which column each word
came from. That distinction is usually worth keeping: a word in a title and the
same word in a body are not the same signal.
Naming several columns is not a workaround either. `CountVectorizer` takes a
flat sequence of documents, so a two-column frame reaches it as two documents
rather than as the rows, and it learns a vocabulary of the column names:
```
>>> CountVectorizer().fit(df[["title", "body"]]).vocabulary_
{'body': 0, 'title': 1}
```
The estimator then receives two samples against however many labels, and the
run ends with `Number of labels=15 does not match number of samples=2`.
### Proposed Solution or Design
`Text Attribute` names any number of columns, and the generated pipeline
gives each its own `CountVectorizer`, with the outputs concatenated.
`ColumnTransformer` does this and keeps the columns distinguishable, prefixing
each feature with the column it came from:
```python
ColumnTransformer([
("short_text", CountVectorizer(), "short_text"),
("long_text", CountVectorizer(), "long_text"),
])
```
The operator's output is unchanged: one row of `model_name` and `model`,
with the pipeline pickled into the binary column as before. Only the model's
own input contract widens, from one column to the named several.
The field becomes a list, and old workflows keep loading if it accepts a
single value as one:
```scala
@JsonFormat(with = Array(JsonFormat.Feature.ACCEPT_SINGLE_VALUE_AS_ARRAY))
@AutofillAttributeNameList
var text: List[EncodableString]
```
`Tfidf Transformer` stays one switch over the whole matrix, which keeps the
shape of the generated pipeline as it is:
```python
make_pipeline(CountVectorizer(), TfidfTransformer(), estimator()) #
today
make_pipeline(ColumnTransformer([...]), TfidfTransformer(), estimator()) #
proposed
```
Turning `Count Vectorizer` on still discards the numeric columns, as it does
today. Carrying them along needs a way to say which columns are features, which
the Advanced Sklearn operators have as `Selected Features` and these two
families do not, so it is a separate change and not proposed here.
A smaller alternative is to concatenate the columns into one string before a
single `CountVectorizer`, which is what a Concat operator does today. It merges
the vocabularies, so a word keeps no record of which column it came from, and
the model loses a distinction it could have used.
The field is declared on `SklearnModelOpDesc`, so this reaches all fifty-one
operators of the two groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
