Re: Spark streaming dataframe extract message to new columns

2019-08-13 Thread Tianlang
Hi, Do you mean haven a colum A then you want to extract A1 and A2 from A ? like Cloumn A value   123456,2019-08-07 A1 value is 123456 A2 value is 2019-08-07 If that's the case you can use df.select like this df.select(split('A)(0) as "A1", split('A)(1) as "A2") Good Luck 在 2019/8/12

Re: Spark streaming dataframe extract message to new columns

2019-08-12 Thread Gourav Sengupta
Hi, I think that it should be possible to write a query on the streaming data frame and then write the output of the query to S3 or any other sink layer. Regards, Gourav Sengupta On Sat, Aug 10, 2019 at 9:24 AM zenglong chen wrote: > How to extract some message in streaming dataframe and make

Spark streaming dataframe extract message to new columns

2019-08-10 Thread zenglong chen
How to extract some message in streaming dataframe and make a new column? Thanks.