Hi Shirish,
Thank you for the explanation. That clears everything up. I misinterpreted
the meaning of the error message.
I'll add that useful table to the documentation.
Deron
On Wed, Dec 9, 2015 at 5:59 PM, Shirish Tatikonda <
shirish.tatiko...@gmail.com> wrote:
> Hi Deron,
>
> As the error said "A column can not be binned and scaled.", no column can
> be subjected to both *binning* and *scaling *because it does not make
> sense. *Binning* turns a scale column with continuous values into a
> categorical column. On the other hand, *Scaling* can only be done on
> continuous values.
>
> The error *does not *mean that *Scaling* is not supported. We do support S
> *caling*.
>
> At some point, I wanted to add the following table (which is currently
> present in Java code as comments) to our documentation to indicate
> transformations that can be used *simultaneously* on a single column. While
> you are at it, could you make sure it is added to the documentation?
>
> x indicates the combination is invalid.
> * indicates the combination is allowed.
> - indicates the combination is not applicable.
>
> OMIT MVI RCD BIN DCD SCL
> OMIT - x * * * *
> MVI x - * * * *
> RCD * * - x * x
> BIN * * x - * x
> DCD * * * * - x
> SCL * * x x x -
>
> OMIT = Missing value handling by *omitting *rows
> MVI = Missing value handling by *imputation*
> RCD = Recoding
> BIN = Binning
> DCD = Dummycoding
> SCL = Scaling
>
> Let me know if you have any further questions.
>
> Thank you,
> Shirish
>
>
> On Wed, Dec 9, 2015 at 4:53 PM, Deron Eriksson <deroneriks...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I'm working on updating the online docs for the DML transform() function
> > since a couple things didn't copy over in the conversion to markdown.
> > However, I've run into an issue when I execute the transform() example.
> In
> > summary, is the "scale" transformation no longer allowed, and "bin" is
> > allowed?
> >
> > I did the following:
> >
> > I created data.csv:
> >
> >
> >
> zipcode,district,sqft,numbedrooms,numbathrooms,floors,view,saleprice,askingprice
> > 95141,south,3002,6,3,2,FALSE,929,934
> > NA,west,1373,,1,3,FALSE,695,698
> > 91312,south,NA,6,2,2,FALSE,902,
> > 94555,NA,1835,3,,3,,888,892
> > 95141,west,2770,5,2.5,,TRUE,812,816
> > 95141,east,2833,6,2.5,2,TRUE,927,
> > 96334,NA,1339,6,3,1,FALSE,672,675
> > 96334,south,2742,6,2.5,2,FALSE,872,876
> > 96334,north,2195,5,2.5,2,FALSE,799,803
> >
> > I created data.csv.mtd:
> >
> > {
> > "data_type": "frame",
> > "format": "csv",
> > "sep": ",",
> > "header": true,
> > "na.strings": [ "NA", "" ]
> > }
> >
> > I created data.spec.json:
> >
> > {
> > "omit": [ "zipcode" ]
> > ,"impute":
> > [ { "name": "district" , "method": "constant", "value": "south" }
> > ,{ "name": "numbedrooms" , "method": "constant", "value": 2 }
> > ,{ "name": "numbathrooms", "method": "constant", "value": 1 }
> > ,{ "name": "floors" , "method": "constant", "value": 1 }
> > ,{ "name": "view" , "method": "global_mode" }
> > ,{ "name": "askingprice" , "method": "global_mean" }
> > ]
> >
> > ,"recode":
> > [ "zipcode", "district", "numbedrooms", "numbathrooms", "floors",
> > "view" ]
> >
> > ,"bin":
> > [ { "name": "saleprice" , "method": "equi-width", "numbins": 3 }
> > ,{ "name": "sqft" , "method": "equi-width", "numbins": 4 }
> > ]
> >
> > ,"dummycode":
> > [ "district", "numbathrooms", "floors", "view", "saleprice", "sqft" ]
> >
> > ,"scale":
> > [ { "name": "sqft", "method": "mean-subtraction" }
> > ,{ "name": "saleprice", "method": "z-score" }
> > ,{ "name": "askingprice", "method": "z-score" }
> > ]
> > }
> >
> > I executed the following DML:
> >
> > D = read("data.csv");
> > tfD = transform(target=D,
> > transformSpec="data.spec.json",
> > transformPath="example-transform");
> > s = sum(tfD);
> > print("Sum = " + s);
> >
> > This generated the following error:
> >
> > java.lang.IllegalArgumentException: Invalid transformations on column ID
> 3.
> > A column can not be binned and scaled.
> >
> > So, I removed the "scale" from data.spec.json:
> >
> > {
> > "omit": [ "zipcode" ]
> > ,"impute":
> > [ { "name": "district" , "method": "constant", "value": "south" }
> > ,{ "name": "numbedrooms" , "method": "constant", "value": 2 }
> > ,{ "name": "numbathrooms", "method": "constant", "value": 1 }
> > ,{ "name": "floors" , "method": "constant", "value": 1 }
> > ,{ "name": "view" , "method": "global_mode" }
> > ,{ "name": "askingprice" , "method": "global_mean" }
> > ]
> >
> > ,"recode":
> > [ "zipcode", "district", "numbedrooms", "numbathrooms", "floors",
> > "view" ]
> >
> > ,"bin":
> > [ { "name": "saleprice" , "method": "equi-width", "numbins": 3 }
> > ,{ "name": "sqft" , "method": "equi-width", "numbins": 4 }
> > ]
> >
> > ,"dummycode":
> > [ "district", "numbathrooms", "floors", "view", "saleprice", "sqft" ]
> >
> > }
> >
> > This generated:
> >
> > java.lang.RuntimeException: Encountered "NA" in column ID "3", when
> > expecting a numeric value. Consider adding "NA" to na.strings, along with
> > an appropriate imputation method.
> >
> > So, I set "sqft" to be "global_mean" in the "impute" section of the spec.
> >
> > {
> > "omit": [ "zipcode" ]
> > ,"impute":
> > [ { "name": "district" , "method": "constant", "value": "south" }
> > ,{ "name": "numbedrooms" , "method": "constant", "value": 2 }
> > ,{ "name": "numbathrooms", "method": "constant", "value": 1 }
> > ,{ "name": "floors" , "method": "constant", "value": 1 }
> > ,{ "name": "view" , "method": "global_mode" }
> > ,{ "name": "askingprice" , "method": "global_mean" }
> > ,{ "name": "sqft" , "method": "global_mean" }
> > ]
> >
> > ,"recode":
> > [ "zipcode", "district", "numbedrooms", "numbathrooms", "floors",
> > "view" ]
> >
> > ,"bin":
> > [ { "name": "saleprice" , "method": "equi-width", "numbins": 3 }
> > ,{ "name": "sqft" , "method": "equi-width", "numbins": 4 }
> > ]
> >
> > ,"dummycode":
> > [ "district", "numbathrooms", "floors", "view", "saleprice", "sqft" ]
> >
> > }
> >
> > This allowed the DML to execute successfully.
> >
> > So, is "scale" not allowed anymore? And "bin" is allowed (despite the
> > message saying it isn't allowed)?
> >
> > Thank you,
> > Deron
> >
>
