Hi Chongguang, Thanks for the report!
That makes sense and the proposition should work, or we can add something like `def array_contains(column: Column, value: Column)`. Maybe other functions, such as `array_position`, `element_at`, are the same situation. Could you file a JIRA, and submit a PR if possible? We can have a discussion more about the issue there. Btw, I guess you can use `expr("array_contains(columnA, columnB)")` as a workaround. Thanks. On Thu, Jun 14, 2018 at 2:15 AM, 刘崇光 <lcg31...@gmail.com> wrote: > > ---------- Forwarded message ---------- > From: 刘崇光 <lcg31...@gmail.com> > Date: Thu, Jun 14, 2018 at 11:08 AM > Subject: array_contains in package org.apache.spark.sql.functions > To: user@spark.apache.org > > > Hello all, > > I ran into a use case in project with spark sql and want to share with you > some thoughts about the function array_contains. > > Say I have a Dataframe containing 2 columns. Column A of type "Array of > String" and Column B of type "String". I want to determine if the value of > column B is contained in the value of column A, without using a udf of > course. > The function array_contains came into my mind naturally: > > def array_contains(column: Column, value: Any): Column = withExpr { > ArrayContains(column.expr, Literal(value)) > } > > However the function takes the column B and does a "Literal" of column B, > which yields a runtime exception: RuntimeException("Unsupported literal > type " + v.getClass + " " + v). > > Then after discussion with my friends, we fund a solution without using > udf: > > new Column(ArrayContains(df("ColumnA").expr, df("ColumnB").expr) > > > With this solution, I think of empowering a little bit more the function, > by doing like this: > > def array_contains(column: Column, value: Any): Column = withExpr { > value match { > case c: Column => ArrayContains(column.expr, c.expr) > case _ => ArrayContains(column.expr, Literal(value)) > } > } > > > It does a pattern matching to detect if value is of type Column. If yes, > it will use the .expr of the column, otherwise it will work as it used to. > > Any suggestion or opinion on the proposition? > > > Kind regards, > Chongguang LIU > > -- Takuya UESHIN Tokyo, Japan http://twitter.com/ueshin