> Forward to dev Yes, the databricks runtime 13.0 and 13.1 and 13.2 are all ok and have the same behavior with open source Apache Spark 3.4.x. But I think the docs of databricks need to be updated[1]. It's confusing.
[1] https://docs.databricks.com/en/sql/language-manual/functions/array_insert.html Best Regards, Ran Tao Sean Owen <sro...@gmail.com> 于2023年8月14日周一 11:58写道： > Oh I get it. Let me report to the docs people (I'm at databricks) > > On Sun, Aug 13, 2023, 10:56 PM Ran Tao <chucheng...@gmail.com> wrote: > >> Yes, the databricks runtime 13.0 and 13.1 and 13.2 are all ok and have >> the same behavior with open source Apache Spark 3.4.x. >> But I think the docs of databricks need to be updated. it's confusing. >> >> Thanks owen! >> >> >> Sean Owen <sro...@gmail.com> 于2023年8月14日周一 11:33写道： >> >>> That's just a bug fix, right? I'd imagine. >>> >>> On Sun, Aug 13, 2023, 10:28 PM Ran Tao <chucheng...@gmail.com> wrote: >>> >>>> thanks. it seems that the new Databricks runtime has same behavior with >>>> open source apache spark. >>>> however, the Databricks runtime docs[1] may confuse users, it indicates >>>> can be applicable in Databricks Runtime 13.0 and later, >>>> however it not has this behavior in 13.1 and later version. >>>> >>>> Best Regards, >>>> Ran Tao >>>> https://github.com/chucheng92 >>>> >>>> >>>> Sean Owen <sro...@gmail.com> 于2023年8月14日周一 02:07写道： >>>> >>>>> No, I'm saying I do not see any inconsistency between docs, OSS Spark, >>>>> and Spark in Databricks. All the behavior looks the same and correct. >>>>> I am using the latest Spark and DBR runtime though, and you didn't say >>>>> what version of Spark you use in DB. >>>>> The most likely answer is it's an older version with some kind of bug, >>>>> not sure. >>>>> >>>>> On Sun, Aug 13, 2023 at 1:01 PM Ran Tao <chucheng...@gmail.com> wrote: >>>>> >>>>>> thanks. however I have checked the latest spark version 3.4.0 and >>>>>> 3.4.1 the result is what i listed above. did’u mean the databricks doc is >>>>>> outdated or wrong？which one i should respect？ >>>>>> >>>>>> >>>>>> >>>>>> Sean Owen <sro...@gmail.com>于2023年8月13日 周日22:35写道： >>>>>> >>>>>>> There shouldn't be any difference here. In fact, I get the results >>>>>>> you list for 'spark' from Databricks. It's possible the difference is a >>>>>>> bug >>>>>>> fix along the way that is in the Spark version you are using locally but >>>>>>> not in the DBR you are using. But, yeah seems to work as. you say. >>>>>>> >>>>>>> If you're asking about the Spark semantics being 1-indexed vs >>>>>>> 0-indexed? some comments here: >>>>>>> https://github.com/apache/spark/pull/38867#discussion_r1097054656 >>>>>>> >>>>>>> >>>>>>> On Sun, Aug 13, 2023 at 7:28 AM Ran Tao <chucheng...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, devs. >>>>>>>> >>>>>>>> I found that the ARRAY_INSERT[1] function (from spark 3.4.0) has >>>>>>>> different semantics with databricks[2]. >>>>>>>> >>>>>>>> e.g. >>>>>>>> >>>>>>>> // spark >>>>>>>> SELECT array_insert(array('a', 'b', 'c'), -1, 'z'); >>>>>>>> ["a","b","z","c"] >>>>>>>> >>>>>>>> // databricks >>>>>>>> SELECT array_insert(array('a', 'b', 'c'), -1, 'z'); >>>>>>>> ["a","b","c","z"] >>>>>>>> >>>>>>>> // spark >>>>>>>> SELECT array_insert(array('a', 'b', 'c'), -5, 'z'); >>>>>>>> ["z",null,null,"a","b","c"] >>>>>>>> >>>>>>>> // databricks >>>>>>>> SELECT array_insert(array('a', 'b', 'c'), -5, 'z'); >>>>>>>> ["z",NULL,"a","b","c"] >>>>>>>> >>>>>>>> It looks like that inserting negative index is more reasonable in >>>>>>>> Databricks. >>>>>>>> >>>>>>>> Of cause, I read the source code of spark, and I can understand the >>>>>>>> logic of spark, but my question is whether spark is designed like this >>>>>>>> on >>>>>>>> purpose? >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> https://spark.apache.org/docs/latest/api/sql/index.html#array_insert >>>>>>>> [2] >>>>>>>> https://docs.databricks.com/en/sql/language-manual/functions/array_insert.html >>>>>>>> >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Ran Tao >>>>>>>> https://github.com/chucheng92 >>>>>>>> >>>>>>> -- >>>>>> Best Regards, >>>>>> Ran Tao >>>>>> https://github.com/chucheng92 >>>>>> >>>>>
