Re: 退订

please send an empty email to: user-unsubscr...@spark.apache.org for unsubscribing yourself from the list. Thanks On Fri, Mar 5, 2021, at 9:21 PM, 韩天罡 wrote: > 哈哈哈，兄弟你退订成功了吗 > > 在 2021-03-05 15:08:35，"吃完药感觉自己萌萌哒" <1356469...@qq.com> 写道： >> 退订 > > >

Re: [ANNOUNCE] Announcing Apache Spark 3.1.1

Congrats and thanks! From: Hyukjin Kwon Sent: Wednesday, March 3, 2021 4:09:23 PM To: Dongjoon Hyun Cc: Gabor Somogyi ; Jungtaek Lim ; angers zhu ; Wenchen Fan ; Kent Yao ; Takeshi Yamamuro ; dev ; user @spark Subject: Re: [ANNOUNCE] Announcing Apache Spark

Spark streaming with multiple Kafka topics

Hi Team, I have a spark streaming application configured to consume events from 2 Kafka topics. But when I run the application locally, the messages are consumed from either of these topics only and not both. If the first event is published to say topic2 and second message to topic1 then only the

Re: Structured Streaming Microbatch Semantics

Hi Rico, Would it be possible for you to provide a snapshot of Structured Streaming Tab (from Spark GUI) if possible? Thanks Mich LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Structured Streaming Microbatch Semantics

Hi! Thanks for your reply! For several reasons we don't want to "pipe" the real data through Kafka. What may be a problem arising from this approach? Best, Rico. Am 05.03.2021 um 09:18 schrieb Roland Johann: Hi Rico, there is no way to deferr records from one micro batch to the next

Re: Structured Streaming Microbatch Semantics

Hi! As abstract code what I do in my streaming program is: readStream() //from Kafka .flatMap(readIngestionDatasetViaREST) //can return thousands of records for a single event .writeStream.outputMode("append").foreachBatch(upsertIntoDeltaTable).start() I don't use triggers but I limit the

Re:????

?? ?? 2021-03-05 15:08:35??"" <1356469...@qq.com> ??

(无主题)

退订 | | Sophia | | 邮箱：sln-1...@163.com | 签名由 网易邮箱大师 定制

Re: Structured Streaming Microbatch Semantics

Hi Ricco, Just to clarify, your batch interval may have a variable number of rows sent to Kafka topic for each event? In your writeStream code writeStream. \ outputMode('append'). \ option("truncate", "false"). \

Re: Structured Streaming Microbatch Semantics

Hi Rico, there is no way to deferr records from one micro batch to the next one. So it‘s guaranteed that the data and trigger event will be processed within the dame batch. I assume that one trigger event lead to an unknown batch size of actual events pulled via HTTP. This bypasses throughput

Structured Streaming Microbatch Semantics

Hi all! I'm using Spark structured streaming for a data ingestion pipeline. Basically the pipeline reads events (notifications of new available data) from a Kafka topic and then queries a REST endpoint to get the real data (within a flatMap). For one single event the pipeline creates a few