Quick Question regarding the same topic. If construct the data frame *df*
in the following way
key | value | topic | offset | partition | timestamp
----------------------------------------------------------------
foo1 | value1 | topic1 | .......
foo2 | value2 | topic2 | .......
what happens when I do df.writeStream().format("kafka").option("topic",
"hello_topic").start() ? Will it use the hello_topic specified in the
option or the topic from row ? Sure I don't need to specify topic as an
option if I have it in a row but I wonder what happens internally if this
is the case?
Thanks!
On Thu, Mar 22, 2018 at 7:48 AM, kant kodali <kanth...@gmail.com> wrote:
> Thanks all!
>
> On Thu, Mar 22, 2018 at 2:08 AM, Jorge Machado <jom...@me.com> wrote:
>
>> DataFrames are not mutable.
>>
>> Jorge Machado
>>
>>
>> On 22 Mar 2018, at 10:07, Aakash Basu <aakash.spark....@gmail.com> wrote:
>>
>> Hey,
>>
>> I faced the same issue a couple of days back, kindly go through the mail
>> chain with "*Multiple Kafka Spark Streaming Dataframe Join query*" as
>> subject, TD and Chris has cleared my doubts, it would help you too.
>>
>> Thanks,
>> Aakash.
>>
>> On Thu, Mar 22, 2018 at 7:50 AM, kant kodali <kanth...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Is there a mutable dataframe spark structured streaming 2.3.0? I am
>>> currently reading from Kafka and if I cannot parse the messages that I get
>>> from Kafka I want to write them to say some "dead_queue" topic.
>>>
>>> I wonder what is the best way to do this?
>>>
>>> Thanks!
>>>
>>
>>
>>
>
