Hi Chao & DB,
Actually, I cut the RC2 yesterday before your post the Parquet issue:
https://github.com/apache/spark/tree/v3.2.0-rc2
It has been 11 days since RC1. I think we can have RC2 today so that the
community can test and find potential issues earlier.
As for the Parquet issue, we can treat
I think a few lines to add the prompt might be enough. This checks for
interactive()
https://github.com/apache/spark/blob/c6a2021fec5bab9069fbfba33f75d4415ea76e99/R/pkg/R/sparkR.R#L658
On Tue, Aug 31, 2021 at 5:55 PM Hyukjin Kwon wrote:
> Oh I missed this. Yes, can we simply get the user'
Oh I missed this. Yes, can we simply get the user' confirmation when we
install.spark?
IIRC, the auto installation is only triggered by interactive shell so
getting user's confirmation should be fine.
2021년 6월 18일 (금) 오전 2:54, Felix Cheung 님이 작성:
> Any suggestion or comment on this? They are
Hello Xiao, there are multiple patches in Spark 3.2 depending on parquet
1.12, so it might be easier to wait for the fix in parquet community
instead of reverting all the related changes. The fix in parquet community
is very trivial, and we hope that it will not take too long. Thanks.
DB Tsai |
Hi Xiao, I'm still checking with the Parquet community on this. Since the
fix is already +1'd, I'm hoping this won't take long. The delta in
parquet-1.12.x branch is also small with just 2 commits so far.
Chao
On Tue, Aug 31, 2021 at 12:03 PM Xiao Li wrote:
> Hi, Chao,
>
> How long will it
we're having some DNS issues here in the EECS department, and our
crack team is working on getting it resolved asap. until then,
jenkins isn't visible to the outside world.
shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
Hi, Chao,
How long will it take? Normally, in the RC stage, we always revert the
upgrade made in the current release. We did the parquet upgrade multiple
times in the previous releases for avoiding the major delay in our Spark
release
Thanks，
Xiao
On Tue, Aug 31, 2021 at 11:03 AM Chao Sun
The Apache Parquet community found an issue [1] in 1.12.0 which could cause
incorrect file offset being written and subsequently reading of the same
file to fail. A fix has been proposed in the same JIRA and we may have to
wait until a new release is available so that we can upgrade Spark with the
Hi devs,
In current yarn-client mode, we have several problem,
1. When AM lost connection with driver, it will just finish
application with final status of SUCCESS, then
YarnClientSchedulerBackend.MonitorThread will got application status with
SUCCESS final status and then call
