While fitting the currently available sparkR models, such as glm for linear and logistic regression, columns which contains strings are one-hot encoded behind the scenes, as part of the parsing of the RFormula. Does that help, or did you have something else in mind?
> Thank you so much for your mail. It is working . > I have another small question in sparkR - can we create dummy > variables for categorical columns ( like in R we have " dummies" package) > eg in iris dataset we have Spieces as a categorical column so 3 dummy > variables columns like setosa, virginica would be created with 0 and 1 as > values On Mon, Jan 25, 2016 at 12:37 PM, Deborah Siegel <deborah.sie...@gmail.com> wrote: > Maybe not ideal, but since read.df is inferring all columns from the csv > containing "NA" as type of strings, one could filter them rather than using > dropna(). > > filtered_aq <- filter(aq, aq$Ozone != "NA" & aq$Solar_R != "NA") > head(filtered_aq) > > Perhaps it would be better to have an option for read.df to convert any > "NA" it encounters into null types, like createDataFrame does for <NA>, and > then one would be able to use dropna() etc. > > > > On Mon, Jan 25, 2016 at 3:24 AM, Devesh Raj Singh <raj.deves...@gmail.com> > wrote: > >> Hi, >> >> Yes you are right. >> >> I think the problem is with reading of csv files. read.df is not >> considering NAs in the CSV file >> >> So what would be a workable solution in dealing with NAs in csv files? >> >> >> >> On Mon, Jan 25, 2016 at 2:31 PM, Deborah Siegel <deborah.sie...@gmail.com >> > wrote: >> >>> Hi Devesh, >>> >>> I'm not certain why that's happening, and it looks like it doesn't >>> happen if you use createDataFrame directly: >>> aq <- createDataFrame(sqlContext,airquality) >>> head(dropna(aq,how="any")) >>> >>> If I had to guess.. dropna(), I believe, drops null values. I suppose >>> its possible that createDataFrame converts R's <NA> values to null, so >>> dropna() works with that. But perhaps read.df() does not convert R <NA>s to >>> null, as those are most likely interpreted as strings when they come in >>> from the csv. Just a guess, can anyone confirm? >>> >>> Deb >>> >>> >>> >>> >>> >>> >>> On Sun, Jan 24, 2016 at 11:05 PM, Devesh Raj Singh < >>> raj.deves...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have applied the following code on airquality dataset available in R >>>> , which has some missing values. I want to omit the rows which has NAs >>>> >>>> library(SparkR) Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" >>>> "com.databricks:spark-csv_2.10:1.2.0" "sparkr-shell"') >>>> >>>> sc <- sparkR.init("local",sparkHome = >>>> "/Users/devesh/Downloads/spark-1.5.1-bin-hadoop2.6") >>>> >>>> sqlContext <- sparkRSQL.init(sc) >>>> >>>> path<-"/Users/devesh/work/airquality/" >>>> >>>> aq <- read.df(sqlContext,path,source = "com.databricks.spark.csv", >>>> header="true", inferSchema="true") >>>> >>>> head(dropna(aq,how="any")) >>>> >>>> I am getting the output as >>>> >>>> Ozone Solar_R Wind Temp Month Day 1 41 190 7.4 67 5 1 2 36 118 8.0 72 5 >>>> 2 3 12 149 12.6 74 5 3 4 18 313 11.5 62 5 4 5 NA NA 14.3 56 5 5 6 28 NA >>>> 14.9 66 5 6 >>>> >>>> The NAs still exist in the output. Am I missing something here? >>>> >>>> -- >>>> Warm regards, >>>> Devesh. >>>> >>> >>> >> >> >> -- >> Warm regards, >> Devesh. >> > >