E.g., as per
https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
val df = spark
.read
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1")
.option("startingOffsets", "earliest")
.option("endingOffsets", "latest")
.load()
On Sun, May 3, 2020 at 1:50 AM Liam Clarke-Hutchinson <
[email protected]> wrote:
> Hello 姜戎 ,
>
> Unfortunately there's not enough information in your email for us to help
> you. Are you trying to use Spark Batch to read from Kafka? Have you tried
> setting "endingOffsets" to "latest" instead of an arbitrary number?
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>
>
> On Fri, May 1, 2020 at 2:36 AM 姜戎 <[email protected]> wrote:
>
>> failed to get records for compacted ...after polling for120000
>> partition 0 offset min=0 max=1427265
>> get offsetrange 0 until 500000 to make rdd
>
>
