I have a scenario similar to yours, but we are using udf to do exactly
that. But you need to get the value of a broadcast variable from the udf.
But it's not clear how to achieve it, does anyone know?
Burak Yavuz 于2019年11月19日周二 下午12:23写道：
> If you store the data that you're going to broadcast
It is still not complete. For example, the supported operators are
different under different outputmodes. For developers, only try.
Jungtaek Lim 于2019年11月29日周五 下午1:36写道：
> Hi,
>
> Unsupported operations in Structured Streaming is explained in the guide
> doc.
>
>
Hi,
Unsupported operations in Structured Streaming is explained in the guide
doc.
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#unsupported-operations
Thanks,
Jungtaek Lim (HeartSaVioR)
On Fri, Nov 29, 2019 at 2:08 PM shicheng31...@gmail.com <
That will heavily depend on your implementation of state functions on
flatMapGroupsWithState, but assuming your state functions respect of the
output mode, chaining flatMapGroupsWithState doesn't work - I don't mean it
doesn't compile or doesn't run. I mean the query will not produce correct
Hi:
Spark Structured Streaming uses the DataFrame API. When programming, there
are no compilation errors, but when running, it will report various unsupported
conditions. The official website does not seem to have a document to list the
unsupported operators. This will Inconvenient when
Ok thanks!
Le jeu. 28 nov. 2019 à 11:27, Phillip Henry a
écrit :
> I saw a large improvement in my GraphX processing by:
>
> - using fewer partitions
> - using fewer executors but with much more memory.
>
> YMMV.
>
> Phillip
>
> On Mon, 25 Nov 2019, 19:14 mahzad kalantari,
> wrote:
>
>> Thanks
I saw a large improvement in my GraphX processing by:
- using fewer partitions
- using fewer executors but with much more memory.
YMMV.
Phillip
On Mon, 25 Nov 2019, 19:14 mahzad kalantari,
wrote:
> Thanks for your answer, my use case is friend recommandation for 200
> million profils.
>
> Le
I wrote few examples chaining flatMapGroupsWithState in append mode and it
worked. Other examples did not.
The question is if the flatMapGroupsWithState chains in append mode are
officially supported.
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
