[ https://issues.apache.org/jira/browse/MADLIB-1267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank McQuillan updated MADLIB-1267: ------------------------------------ Fix Version/s: (was: v2.0) v2.1 > General predict function for PL/Python > -------------------------------------- > > Key: MADLIB-1267 > URL: https://issues.apache.org/jira/browse/MADLIB-1267 > Project: Apache MADlib > Issue Type: New Feature > Components: Module: Utilities > Reporter: Frank McQuillan > Priority: Major > Fix For: v2.1 > > > Context > Follow on from https://www.pivotaltracker.com/story/show/158990284 > Story > `As a data scientist` > I want to call a generic PL/Python UDF from SQL to predict > `so that` > I can use the use any code I write or Python libraries for prediction. > Interface > {code} > predict( > model_table, -- model output table > data_table, -- data table > to predict > list_of_columns, -- columns you want in > GD, could be '*' needed??? > list_of_columns_to_exclude, -- columns to explicitly exclude > needed??? > predict_udf, -- plpython UDF to > predict > predict_udf_parameters, -- parameters for UDF, if any > grouping_cols -- groups to build > separate models for (source table distributed by this grouping) needed??? > ); > {code} > Arguments > {code} > source_table > TEXT. Name of the table containing the data to load. > model_table > TEXT. Name of the table containing the model(s), with one row per group. > list_of_columns > TEXT. Comma-separated string of column names or expressions to load. > Can also be '*' implying all columns are to be loaded (except for the ones > included > in the next argument that lists exclusions). The types of the columns can be > mixed. > Array columns can also be included in the list and will be loaded as is > (i.e., not be flattened). (???) > list_of_columns_to_exclude > TEXT. Comma-separated string of column names to exclude from load. > Typically used when 'list_of_columns' is set to '*'. > predict_udf > TEXT. plpython UDF to predict. > predict_udf_parameters (optional) > TEXT. parameters for UDF, if any > grouping_cols (optional) > TEXT, default: NULL. Comma-separated list of column names to group the data > by. > This will produce multiple models, one for each group. > {code} > Open questions > 1) Do we need separate predict functions for R and Python, or can we > autodetect? > If we need separate ones, could call this module `predict_plpythonu' and the > R one would be `predict_plr`. > 2) Do we need `list_of_columns` and `list_of_columns_to_exclude` > or assume it is the same as the training table? > 3) Scoring should be embarrassingly parallel, so do we need `grouping_cols` > in the predict function? > Notes > 1) scikit-learn use the term `predict` and keras uses `evaluate` but I think > `predict` is better. > Acceptance > 1) Generate a model table for sample data set with multiple groups using a > scikit-learn model. Use this predict function to score some test data. > 2) Repeat for Keras/TF. > 3) Repeat for XGBoost. -- This message was sent by Atlassian JIRA (v7.6.3#76005)