I agree. Just pointed out the option, in case you missed it.
Cheers, Shmuel On Mon, Mar 26, 2018 at 10:57 AM, 1427357...@qq.com <1427357...@qq.com> wrote: > Hi, > > Using concat is one of the way. > But the + is more intuitive and easy to understand. > > ------------------------------ > 1427357...@qq.com > > > *From:* Shmuel Blitz <shmuel.bl...@similarweb.com> > *Date:* 2018-03-26 15:31 > *To:* 1427357...@qq.com > *CC:* spark?users <user@spark.apache.org>; dev <d...@spark.apache.org> > *Subject:* Re: the issue about the + in column,can we support the string > please? > Hi, > > you can get the same with: > > import org.apache.spark.sql.functions._ > import sqlContext.implicits._ > import org.apache.spark.sql.types.{IntegerType, StringType, StructField, > StructType} > > val schema = StructType(Array(StructField("name", StringType), > StructField("age", IntegerType) )) > > val lst = List(Row("Shmuel", 13), Row("Blitz", 23)) > val rdd = sc.parallelize(lst) > > val df = sqlContext.createDataFrame(rdd,schema) > > df.withColumn("newName", concat($"name" , lit("abc")) ).show() > > On Mon, Mar 26, 2018 at 6:36 AM, 1427357...@qq.com <1427357...@qq.com> > wrote: > >> Hi all, >> >> I have a table like below: >> >> +---+---------+-----------+ >> | id| name|sharding_id| >> +---+---------+-----------+ >> | 1|leader us| 10000| >> | 3| mycat| 10000| >> +---+---------+-----------+ >> >> My schema is : >> root >> |-- id: integer (nullable = false) >> |-- name: string (nullable = true) >> |-- sharding_id: integer (nullable = false) >> >> I want add a new column named newName. The new column is based on "name" >> and append "abc" after it. My code looks like: >> >> stud_scoreDF.withColumn("newName", stud_scoreDF.col("name") + "abc" >> ).show() >> >> When I run the code, I got the reslult: >> +---+---------+-----------+-------+ >> | id| name|sharding_id|newName| >> +---+---------+-----------+-------+ >> | 1|leader us| 10000| null| >> | 3| mycat| 10000| null| >> +---+---------+-----------+-------+ >> >> >> I checked the code, the key code is in arithmetic.scala. line 165. >> It looks like: >> >> override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = >> dataType match { >> case dt: DecimalType => >> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.$$plus($eval2)") >> case ByteType | ShortType => >> defineCodeGen(ctx, ev, >> (eval1, eval2) => s"(${ctx.javaType(dataType)})($eval1 $symbol >> $eval2)") >> case CalendarIntervalType => >> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1.add($eval2)") >> case _ => >> defineCodeGen(ctx, ev, (eval1, eval2) => s"$eval1 $symbol $eval2") >> } >> >> >> My issue is: >> Can we add case StringType in this class to support string append please? >> >> >> >> ------------------------------ >> 1427357...@qq.com >> > > > > -- > Shmuel Blitz > Big Data Developer > Email: shmuel.bl...@similarweb.com > www.similarweb.com > <https://www.facebook.com/SimilarWeb/> > <https://www.linkedin.com/company/429838/> > <https://twitter.com/similarweb> > > -- Shmuel Blitz Big Data Developer Email: shmuel.bl...@similarweb.com www.similarweb.com <https://www.facebook.com/SimilarWeb/> <https://www.linkedin.com/company/429838/> <https://twitter.com/similarweb>