[
https://issues.apache.org/jira/browse/MADLIB-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488086#comment-16488086
]
Frank McQuillan edited comment on MADLIB-1239 at 5/23/18 9:51 PM:
------------------------------------------------------------------
1) The alternative is to create a summary table with 1 row and 1 column in it
{code}
{feat1, feat2,... featn}
{code}
which is a better idea because a string of arrays for a large table is wasting
space. I will update the description above to add a summary table.
2) hardcoding column name to `feature_vector` seems OK to me.
3) Given your comment on arrays, I think we should drop array support for this
initial version. There are a lot of complexity that comes with flattening
arrays, esp since postgres supports things like
{code}
a
-----------------
{1,2}
{1,2,3}
{{1,2,3}{1,2,3}}
{code}
which would make the dictionary a challenge to construct
was (Author: fmcquillan):
1) The alternative is to create a summary table with 1 row and 1 column in it
{code}
{feat1, feat2,... featn}
{code}
which is a better idea because a string of arrays for a large table is wasting
space. I will update the description above to add a summary table.
2) hardcoding column name to `feature_vector` seems OK to me.
3) Given your comment on arrays, I think we should drop array support for this
initial version. There are a lot of complexity that comes with flattening
arrays, esp since postgres supports things like
```
a
-----------------
{1,2}
{1,2,3}
{{1,2,3}{1,2,3}}
```
which would make the dictionary a challenge to construct
> Columns to Vector
> -----------------
>
> Key: MADLIB-1239
> URL: https://issues.apache.org/jira/browse/MADLIB-1239
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Module: Utilities
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v1.15
>
>
> Columns to Vector
> Converts features from multiple columns of an input table into a feature
> array in a single column. Also outputs the names of the features into an
> array in a single column. This process can be reversed using the function
> vec2cols.
> {code}
> cols2vec(
> source_table,
> out_table,
> list_of_features,
> list_of_features_to_exclude,
> cols_to_output
> )
> source_table
> TEXT. Name of the table containing the source data.
> out_table
> TEXT. Name of the generated table containing the output. If a table with the
> same name already exists, an error will be returned.
> list_of_features
> TEXT. Comma-separated string of column names or expressions to put into
> array. Can also be a '*' implying all columns are to be put into array
> (except for the ones included in the next argument that lists exclusions).
> The types of the features should all be the same since PostgreSQL arrays only
> support elements of the same type. If multiple numeric types are present in
> the list of features, they will be cast to DOUBLE PRECISION in the feature
> array.
> Array columns can also be included in the list, and the array will be
> expanded to treat each element of the array as a separate feature.
> list_of_features_to_exclude (optional)
> TEXT, default NULL. Comma-separated string of column names to exclude from
> the feature array. Use only when list_of_features is '*'.
> cols_to_output (optional)
> TEXT, default NULL. Comma-separated string of column names from the source
> table to keep in the output table, in addition to the feature array. To keep
> all columns from the source table, use '*'.
> Output
> The output table produced by the cols2vec function contains the following
> columns:
> <...>
> Columns from source table, depending on which ones are kept (if any).
> feature_vector
> Array of features. Array type will depend on feature type in the source
> table.
> feature_names
> TEXT[] Array of names of features.
> {code}
> Open questions for cols2vec:
> 1) OK to cast to double if there are mixed numeric types? Or should we
> enforce that all feature columns be the exact same type? (Since elements of
> PostgreSQL arrays need to be the same type.)
> Aside
> The function
> http://pivotalsoftware.github.io/PDLTools/group__grp__array__utilities.html#cols2vec_example
> is similar but the proposed MADlib one has more options. To do the
> equivalent of the PDL Tools one in MADlib, you would do:
> {code}
> cols2vec(
> table_name,
> output_table,
> '*',
> exclude_columns,
> '*'
> )
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
