Hi All, I would like to know how and when the types of the result set are figured out in Spark? for example say I have the following dataframe.
*inputdf* col1 | col2 | col3 ------------------- 1 | 2 | 5 2 | 3 | 6 Now say I do something like below (Pseudo sql) resultdf = select col2/2 from inputdf result.writeStream().format("es").start() the first document in ES will be {"col2": 1} and the second document will be {"col2": 1.5} so I would think ES would throw type mismatch error here if dynamic mapping is disabled on ES server. My question really know is from spark perspective when will the types of resultdf will be figured out ? is it before writing to ES(in general any sink) or after writing the first document? Thanks!