Re: org.apache.spark.sql.AnalysisException: undefined function lit;
selectExpr just uses the SQL parser to interpret the string you give it. So to get a string literal you would use quotes: df.selectExpr("*", "'" + time.miliseconds() + "' AS ms") On Fri, Feb 12, 2016 at 6:19 PM, Andy Davidson < a...@santacruzintegration.com> wrote: > I am trying to add a column with a constant value to my data frame. Any > idea what I am doing wrong? > > Kind regards > > Andy > > > DataFrame result = … > > String exprStr = "lit(" + time.milliseconds()+ ") as ms"; > > logger.warn("AEDWIP expr: {}", exprStr); > > result.selectExpr("*", exprStr).show(false); > > WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1 > call line:96 AEDWIP expr: lit(1455329175000) as ms > > ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error > running job streaming job 1455329175000 ms.0 > > org.apache.spark.sql.AnalysisException: undefined function lit; > > > >
Re: org.apache.spark.sql.AnalysisException: undefined function lit;
I've never done it that way but you can simply use the withColumn method in data frames to do it. On 13 Feb 2016 2:19 a.m., "Andy Davidson" wrote: > I am trying to add a column with a constant value to my data frame. Any > idea what I am doing wrong? > > Kind regards > > Andy > > > DataFrame result = … > > String exprStr = "lit(" + time.milliseconds()+ ") as ms"; > > logger.warn("AEDWIP expr: {}", exprStr); > > result.selectExpr("*", exprStr).show(false); > > WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1 > call line:96 AEDWIP expr: lit(1455329175000) as ms > > ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error > running job streaming job 1455329175000 ms.0 > > org.apache.spark.sql.AnalysisException: undefined function lit; > > > >
org.apache.spark.sql.AnalysisException: undefined function lit;
I am trying to add a column with a constant value to my data frame. Any idea what I am doing wrong? Kind regards Andy DataFrame result = String exprStr = "lit(" + time.milliseconds()+ ") as ms"; logger.warn("AEDWIP expr: {}", exprStr); result.selectExpr("*", exprStr).show(false); WARN 02:06:17 streaming-job-executor-0 c.p.f.s.s.CalculateAggregates$1 call line:96 AEDWIP expr: lit(1455329175000) as ms ERROR 02:06:17 JobScheduler o.a.s.Logging$class logError line:95 Error running job streaming job 1455329175000 ms.0 org.apache.spark.sql.AnalysisException: undefined function lit;