[
https://issues.apache.org/jira/browse/SPARK-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Grzegorz Chilkiewicz updated SPARK-18226:
-----------------------------------------
Description:
I have encountered a problem with SparkR presenting Spark vectors from
org.apache.spark.mllib.linalg package
* `head(df)` shows in vector column: "<environment: 0x2bd5d10>"
* cast to string does not work as expected, it shows:
"[1,null,null,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@79f50a91]"
* `showDF(df)` work correctly
to reproduce, start SparkR and paste following code (example taken from
https://spark.apache.org/docs/latest/sparkr.html#naive-bayes-model)
{code}
# Fit a Bernoulli naive Bayes model with spark.naiveBayes
titanic <- as.data.frame(Titanic)
titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5])
nbDF <- titanicDF
nbTestDF <- titanicDF
nbModel <- spark.naiveBayes(nbDF, Survived ~ Class + Sex + Age)
# Model summary
summary(nbModel)
# Prediction
nbPredictions <- predict(nbModel, nbTestDF)
#########################################
# My modification to expose the problem #
nbPredictions$rawPrediction_str <- cast(nbPredictions$rawPrediction, "string")
head(nbPredictions)
showDF(nbPredictions)
{code}
was:
I have encountered a problem with SparkR presenting Spark vectors from
org.apache.spark.mllib.linalg package
`head(df)` shows in vector column: "<environment: 0x2bd5d10>"
cast to string does not work as expected, it shows:
"[1,null,null,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@79f50a91]"
`showDF(df)` work correctly
to reproduce, start SparkR and paste following code (example taken from
https://spark.apache.org/docs/latest/sparkr.html#naive-bayes-model)
{code}
# Fit a Bernoulli naive Bayes model with spark.naiveBayes
titanic <- as.data.frame(Titanic)
titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5])
nbDF <- titanicDF
nbTestDF <- titanicDF
nbModel <- spark.naiveBayes(nbDF, Survived ~ Class + Sex + Age)
# Model summary
summary(nbModel)
# Prediction
nbPredictions <- predict(nbModel, nbTestDF)
#########################################
# My modification to expose the problem #
nbPredictions$rawPrediction_str <- cast(nbPredictions$rawPrediction, "string")
head(nbPredictions)
showDF(nbPredictions)
{code}
> SparkR displaying vector columns in incorrect way
> -------------------------------------------------
>
> Key: SPARK-18226
> URL: https://issues.apache.org/jira/browse/SPARK-18226
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 2.0.0
> Reporter: Grzegorz Chilkiewicz
> Priority: Trivial
>
> I have encountered a problem with SparkR presenting Spark vectors from
> org.apache.spark.mllib.linalg package
> * `head(df)` shows in vector column: "<environment: 0x2bd5d10>"
> * cast to string does not work as expected, it shows:
> "[1,null,null,org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@79f50a91]"
> * `showDF(df)` work correctly
> to reproduce, start SparkR and paste following code (example taken from
> https://spark.apache.org/docs/latest/sparkr.html#naive-bayes-model)
> {code}
> # Fit a Bernoulli naive Bayes model with spark.naiveBayes
> titanic <- as.data.frame(Titanic)
> titanicDF <- createDataFrame(titanic[titanic$Freq > 0, -5])
> nbDF <- titanicDF
> nbTestDF <- titanicDF
> nbModel <- spark.naiveBayes(nbDF, Survived ~ Class + Sex + Age)
> # Model summary
> summary(nbModel)
> # Prediction
> nbPredictions <- predict(nbModel, nbTestDF)
> #########################################
> # My modification to expose the problem #
> nbPredictions$rawPrediction_str <- cast(nbPredictions$rawPrediction, "string")
> head(nbPredictions)
> showDF(nbPredictions)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
