Re: expected behavior of Kafka dynamic topic subscription
https://issues.apache.org/jira/browse/SPARK-18272 I couldn't speculate on what the issue might be without more info. If you have time to write a test for that ticket, I'd encourage you to do so, I'm not certain how soon I'll be able to get to it. On Sun, Nov 6, 2016 at 7:31 PM, Haopu Wang wrote: > Cody, thanks for the response. Do you think it's a Spark issue or Kafka > issue? Can you please let me know the jira ticket number? > > -Original Message- > From: Cody Koeninger [mailto:c...@koeninger.org] > Sent: 2016年11月4日 22:35 > To: Haopu Wang > Cc: user@spark.apache.org > Subject: Re: expected behavior of Kafka dynamic topic subscription > > That's not what I would expect from the underlying kafka consumer, no. > > But this particular case (no matching topics, then add a topic after > SubscribePattern stream starts) actually isn't part of unit tests for > either the DStream or the structured stream. > > I'll make a jira ticket. > > On Thu, Nov 3, 2016 at 9:43 PM, Haopu Wang wrote: >> I'm using Kafka010 integration API to create a DStream using >> SubscriberPattern ConsumerStrategy. >> >> The specified topic doesn't exist when I start the application. >> >> Then I create the topic and publish some test messages. I can see them in >> the console subscriber. >> >> But the spark application doesn't seem to get the messages. >> >> I think this is not expected, right? What should I check to resolve it? >> >> Thank you very much! - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
RE: expected behavior of Kafka dynamic topic subscription
Cody, thanks for the response. Do you think it's a Spark issue or Kafka issue? Can you please let me know the jira ticket number? -Original Message- From: Cody Koeninger [mailto:c...@koeninger.org] Sent: 2016年11月4日 22:35 To: Haopu Wang Cc: user@spark.apache.org Subject: Re: expected behavior of Kafka dynamic topic subscription That's not what I would expect from the underlying kafka consumer, no. But this particular case (no matching topics, then add a topic after SubscribePattern stream starts) actually isn't part of unit tests for either the DStream or the structured stream. I'll make a jira ticket. On Thu, Nov 3, 2016 at 9:43 PM, Haopu Wang wrote: > I'm using Kafka010 integration API to create a DStream using > SubscriberPattern ConsumerStrategy. > > The specified topic doesn't exist when I start the application. > > Then I create the topic and publish some test messages. I can see them in > the console subscriber. > > But the spark application doesn't seem to get the messages. > > I think this is not expected, right? What should I check to resolve it? > > Thank you very much! - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: expected behavior of Kafka dynamic topic subscription
That's not what I would expect from the underlying kafka consumer, no. But this particular case (no matching topics, then add a topic after SubscribePattern stream starts) actually isn't part of unit tests for either the DStream or the structured stream. I'll make a jira ticket. On Thu, Nov 3, 2016 at 9:43 PM, Haopu Wang wrote: > I'm using Kafka010 integration API to create a DStream using > SubscriberPattern ConsumerStrategy. > > The specified topic doesn't exist when I start the application. > > Then I create the topic and publish some test messages. I can see them in > the console subscriber. > > But the spark application doesn't seem to get the messages. > > I think this is not expected, right? What should I check to resolve it? > > Thank you very much! - To unsubscribe e-mail: user-unsubscr...@spark.apache.org