Re: Adding JIRA ID as the prefix for the test case name

this is about test description and not test file name right? if yes I don’t see a problem. From: Hyukjin Kwon Sent: Thursday, November 14, 2019 6:03:02 PM To: Shixiong(Ryan) Zhu Cc: dev ; Felix Cheung ; Shivaram Venkataraman Subject: Re: Adding JIRA ID as the

Re: Does StreamingSymmetricHashJoinExec work with watermark? I don't think so

Jacek, would you mind if I ask for the query to reproduce? Not sure I get you without having the example of "not working". Thanks, Jungtaek Lim (HeartSaVioR) On Tue, Nov 12, 2019 at 12:04 AM Jacek Laskowski wrote: > Hi, > > I think watermark does not work for StreamingSymmetricHashJoinExec

Re: [Structured Streaming] Robust watermarking calculation with future timestamps

(dropping user@ as cross-posting mailing lists for mail threads would bother both lists, and it seems more appropriate to dev@) AFAIK there's no API for custom watermark, and you're right picking max timestamp would introduce the issues you provided. Other streaming frameworks may pick min

Re: [build system] Upgrading pyarrow, builds might be temporarily broken

Hi, Bryan, Thank you for your update! Xiao On Thu, Nov 14, 2019 at 8:48 PM Bryan Cutler wrote: > Update: #26133 has been > merged and builds should be passing now, thanks all! > > On Thu, Nov 14, 2019 at 4:12 PM Bryan Cutler wrote: > >> We are in

Re: [build system] Upgrading pyarrow, builds might be temporarily broken

Update: #26133 has been merged and builds should be passing now, thanks all! On Thu, Nov 14, 2019 at 4:12 PM Bryan Cutler wrote: > We are in the process of upgrading pyarrow in the testing environment, > which might cause pyspark test failures until

Re: Ask for ARM CI for spark

Hi, And I found Spark-3.0.0-preview had released, but there is no releasenotes in [1]. So how about to add support ARM notes in the next releasenotes(maybe the releasenotes of Spark-3.0.0-preview). So sorry to raise this, I'm not familiar with this, if any bad from me please feel free to correct.

Re: Ask for ARM CI for spark

Hi @Sean Owen , Thanks for reply. We know that Spark community has own release date and plan. We are happy to follow Spark community. But we think it's great if community could add a sentence into the next releasenotes and claim "Spark can support Arm from this release." after we finish the test

Re: Ask for ARM CI for spark

@Sean, Yes, you are right, we don't have to create a separate release of Spark for ARM, it's enough to add a releasenote to say that Spark supports arm architecture. About the test failure, one or two tests will timeout on our poor performance arm instance sometimes, now we donate a high

Re: Ask for ARM CI for spark

I don't quite understand. You are saying tests don't pass yet, so why would anyone yet run these tests regularly? If it's because the instances aren't fast enough, use bigger instances? I don't think anyone would create a separate release of Spark for ARM, no. But why would that be necessary? On

Re: Adding JIRA ID as the prefix for the test case name

Yeah, sounds good to have it. In case of R, it seems not quite common to write down JIRA ID [1] but looks some have the prefix in its test name in general. In case of Python and Java, seems we time to time write a JIRA ID in the comment right under the test method [2][3]. Given this pattern, I

Re: Ask for ARM CI for spark

Hi Spark team, Any ideas about the above email? Thank you. BR ZhaoBo [image: Mailtrack] Sender notified by Mailtrack

[build system] Upgrading pyarrow, builds might be temporarily broken

We are in the process of upgrading pyarrow in the testing environment, which might cause pyspark test failures until https://github.com/apache/spark/pull/26133 is merged. Apologies for the lack of notice beforehand, but I jumped the gun a little and forgot this would affect other builds too. The

Re: Adding JIRA ID as the prefix for the test case name

Should we also add a guideline for non Scala tests? Other languages (Java, Python, R) don't support using string as a test name. Best Regards, Ryan On Thu, Nov 14, 2019 at 4:04 AM Hyukjin Kwon wrote: > I opened a PR - https://github.com/apache/spark-website/pull/231 > > 2019년 11월 13일 (수) 오전

Re: Adding JIRA ID as the prefix for the test case name

I opened a PR - https://github.com/apache/spark-website/pull/231 2019년 11월 13일 (수) 오전 10:43, Hyukjin Kwon 님이 작성: > > In general a test should be self descriptive and I don't think we should > be adding JIRA ticket references wholesale. Any action that the reader has > to take to understand why a