[ https://issues.apache.org/jira/browse/SPARK-31466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091196#comment-17091196 ]
Hyukjin Kwon commented on SPARK-31466: -------------------------------------- Can you read the doc for the class? {quote} * Hive varchar type. Similar to other HiveStringType's, these datatypes should only used for * parsing, and should NOT be used anywhere else. Any instance of these data types should be * replaced by a [[StringType]] before analysis. {quote} It's not supposed to use as an API. > String/Int to VarcharType cast not supported in Spark > ------------------------------------------------------- > > Key: SPARK-31466 > URL: https://issues.apache.org/jira/browse/SPARK-31466 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.3.0 > Reporter: Gourav Choubey > Priority: Major > > While casting a string column to varchar it does not do the casting at all > and column remains string. > > I tried to achieve it through VarcharType as below but it errors : > for(i<-colList) > { Df=Df.withColumn(i,Df(i).cast(VarcharType(1000))) } > *org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(`ColName AS > VARCHAR(1000))' due to data type mismatch: cannot cast string to > varchar(1000);;* > > *Also, I tried through selectExpr cast option but no success.* > > While trying to create an empty dataframe with VarcharType also throws an > error > *scala> var empty_df = spark.createDataFrame(sc.emptyRDD[Row], schema_rdd)* > *scala.MatchError: VarcharType(1000) (of class > org.apache.spark.sql.types.VarcharType)* > *at > org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$enco* > > Please suggest a way to cast a string to varchar in spark. As a reading > string column in SAS application has performance implications. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org