Hi All,
Are there any people interested in adding structured plan logging in Spark?
Currently the logical/physical plan could be logged as plain text via
explain() method, which has some issues, for example, string truncation and
difficult for tool/program to use.
This PR
-1. We have two correctness bugs:
https://issues.apache.org/jira/browse/SPARK-25714 and
https://issues.apache.org/jira/browse/SPARK-25708.
Let us fix all the three issues in ScalaUDF, as mentioned by Sean.
Xiao
Sean Owen 于2018年10月11日周四 上午9:04写道：
> This is a legitimate question about the
Note that, it was deprecated in 2.3.0 already:
https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html
On Fri, Oct 12, 2018 at 12:46 AM Reynold Xin wrote:
> Sounds like a good idea...
>
> > On Oct 11, 2018, at 6:40 PM, Sean Owen wrote:
> >
> > Yep, that already exists as Bahir.
>
Sounds like a good idea...
> On Oct 11, 2018, at 6:40 PM, Sean Owen wrote:
>
> Yep, that already exists as Bahir.
> Also, would anyone object to declaring Flume support at least
> deprecated in 2.4.0?
>> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke wrote:
>>
>> I think it makes sense to remove
Yep, that already exists as Bahir.
Also, would anyone object to declaring Flume support at least
deprecated in 2.4.0?
On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke wrote:
>
> I think it makes sense to remove it.
> If it is not too much effort and the architecture of the flume source is not
>
This is a legitimate question about the behavior of ScalaUDF after the
change to support 2.12:
https://github.com/apache/spark/pull/22259#discussion_r224295469
Not quite a blocker I think, but a potential gotcha we definitely need
to highlight in release notes. There may be an argument for
Hi Hyukjin, can you open a PR to revert it from 2.4? Now I'm kind of
convinced this is too breaking and we need more discussion.
+ Ryan Blue
Hi Ryan,
I think we need to look back at the new write API design and consider data
sources that don't have table concept. We should opt-in for the schema
That's why I initially suggested to revert this part out of Spark 2.4 and
have more discussion at 3.0 since one of the design goal of Data source V2
is no behaviour changes to end users.
2018년 10월 11일 (목) 오후 7:11, Mendelson, Assaf 님이 작성:
> Actually, it is not just a question of a write only data
Forgot to cc dev-list
-- Forwarded message -
From: Wenchen Fan
Date: Thu, Oct 11, 2018 at 10:14 AM
Subject: Re: [VOTE] SPARK 2.4.0 (RC3)
To:
Cc: Sean Owen
Ah sorry guys, I just copy-paste the voting email from the last RC and
forgot to update the date :P
The voting should be
Actually, it is not just a question of a write only data source. The issue is
that in my case (and I imagine this is true for others), the schema is not read
from the database but is understood from the options. This means that I have no
way of understanding the schema without supplying the
See https://github.com/apache/spark/pull/22688
+WEnchen, here looks the problem raised. This might have to be considered
as a blocker ...
On Thu, 11 Oct 2018, 2:48 pm assaf.mendelson,
wrote:
> Hi,
>
> I created a datasource writer WITHOUT a reader. When I do, I get an
> exception:
Hi,
I created a datasource writer WITHOUT a reader. When I do, I get an
exception: org.apache.spark.sql.AnalysisException: Data source is not
readable: DefaultSource
The reason for this is that when save is called, inside the source match to
WriterSupport we have the following code:
val source
So, which date is it?
2018년 10월 11일 (목) 오전 1:48, Garlapati, Suryanarayana (Nokia - IN/Bangalore) <
suryanarayana.garlap...@nokia.com>님이 작성:
> Might be you need to change the date(Oct 1 has already passed).
>
>
>
> >> The vote is open until October 1 PST and passes if a majority +1 PMC
> votes
13 matches
Mail list logo
