Re: producer transaction and threading model

[Edward Capriolo]

Spring framework works around this by providing a method like
doInTransaction( listofstuff).

Behind the scenes it manages a pool of transactional producers with ids
like  transaction_prefix + id.

So each call to dointransaction may initiate a transaction.

On Friday, November 15, 2019, Matthias J. Sax  wrote:

> That's correct. And each would need to use a different `transactional.id`.
>
>
> -Matthias
>
> On 11/14/19 11:17 AM, Anindya Haldar wrote:
> > Thanks for the information. Does that mean that each producer thread, in
> case it wants to have its own transactions, should use its own instance of
> KafkaProducer?
> >
> > Sincerely,
> > Anindya Haldar
> > Oracle Responsys
> >
> >
> >> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax 
> wrote:
> >>
> >> That is not possible. A producer can only have a single open
> >> transaction. If your example, the call of thread-2 to start a new
> >> transactions would fail and an exception would be thrown.
> >>
> >> -Matthias
> >>
> >> On 11/13/19 10:52 AM, Anindya Haldar wrote:
> >>> The producer API doc (https://kafka.apache.org/23/
> javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html <
> https://kafka.apache.org/23/javadoc/index.html?org/apache/
> kafka/clients/producer/KafkaProducer.html>) says that KafkaProducer is
> thread safe.
> >>>
> >>> What I was trying to figure out is what the relationship is between a
> thread context and a transaction initiated in that thread. When multiple
> threads are using a single producer instance and initiating transactions
> are those transaction boundaries scoped to the thread context, or do they
> step on each others’ toes?
> >>>
> >>> Suppose two threads are sharing the same producer instance the
> following way:
> >>>
> >>>thread-1 begins a transaction T1
> >>>thread-1 starts sending messages in T!
> >>>thread-2 begins a transaction T2
> >>>thread-2 sends some messages in T2
> >>>thread-2 gets and error and rolls back its transaction T2
> >>>
> >>> At this point does the rollback of transaction T2 by the thread-2
> affect the transaction T1 remaining open and used by thread-1 in any
> fashion? It is not clear to me from the API documentation. Will very much
> appreciate some insights into the behavior here. Similar questions also
> arise for commits by threads independently of each other.
> >>>
> >>> Sincerely,
> >>> Anindya Haldar
> >>> Oracle Responsys
> >>>
> >>>
> >>
> >
>
>

-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: producer transaction and threading model

[Matthias J. Sax]

That's correct. And each would need to use a different `transactional.id`.


-Matthias

On 11/14/19 11:17 AM, Anindya Haldar wrote:
> Thanks for the information. Does that mean that each producer thread, in case 
> it wants to have its own transactions, should use its own instance of 
> KafkaProducer?
> 
> Sincerely,
> Anindya Haldar
> Oracle Responsys
> 
> 
>> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax  wrote:
>>
>> That is not possible. A producer can only have a single open
>> transaction. If your example, the call of thread-2 to start a new
>> transactions would fail and an exception would be thrown.
>>
>> -Matthias
>>
>> On 11/13/19 10:52 AM, Anindya Haldar wrote:
>>> The producer API doc 
>>> (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>>>  
>>> )
>>>  says that KafkaProducer is thread safe.
>>>
>>> What I was trying to figure out is what the relationship is between a 
>>> thread context and a transaction initiated in that thread. When multiple 
>>> threads are using a single producer instance and initiating transactions 
>>> are those transaction boundaries scoped to the thread context, or do they 
>>> step on each others’ toes?
>>>
>>> Suppose two threads are sharing the same producer instance the following 
>>> way:
>>>
>>>thread-1 begins a transaction T1
>>>thread-1 starts sending messages in T!
>>>thread-2 begins a transaction T2
>>>thread-2 sends some messages in T2
>>>thread-2 gets and error and rolls back its transaction T2
>>>
>>> At this point does the rollback of transaction T2 by the thread-2 affect 
>>> the transaction T1 remaining open and used by thread-1 in any fashion? It 
>>> is not clear to me from the API documentation. Will very much appreciate 
>>> some insights into the behavior here. Similar questions also arise for 
>>> commits by threads independently of each other.
>>>
>>> Sincerely,
>>> Anindya Haldar
>>> Oracle Responsys
>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: producer transaction and threading model

[Anindya Haldar]

Thanks for the information. Does that mean that each producer thread, in case 
it wants to have its own transactions, should use its own instance of 
KafkaProducer?

Sincerely,
Anindya Haldar
Oracle Responsys


> On Nov 13, 2019, at 11:31 PM, Matthias J. Sax  wrote:
> 
> That is not possible. A producer can only have a single open
> transaction. If your example, the call of thread-2 to start a new
> transactions would fail and an exception would be thrown.
> 
> -Matthias
> 
> On 11/13/19 10:52 AM, Anindya Haldar wrote:
>> The producer API doc 
>> (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>>  
>> )
>>  says that KafkaProducer is thread safe.
>> 
>> What I was trying to figure out is what the relationship is between a thread 
>> context and a transaction initiated in that thread. When multiple threads 
>> are using a single producer instance and initiating transactions are those 
>> transaction boundaries scoped to the thread context, or do they step on each 
>> others’ toes?
>> 
>> Suppose two threads are sharing the same producer instance the following way:
>> 
>>thread-1 begins a transaction T1
>>thread-1 starts sending messages in T!
>>thread-2 begins a transaction T2
>>thread-2 sends some messages in T2
>>thread-2 gets and error and rolls back its transaction T2
>> 
>> At this point does the rollback of transaction T2 by the thread-2 affect the 
>> transaction T1 remaining open and used by thread-1 in any fashion? It is not 
>> clear to me from the API documentation. Will very much appreciate some 
>> insights into the behavior here. Similar questions also arise for commits by 
>> threads independently of each other.
>> 
>> Sincerely,
>> Anindya Haldar
>> Oracle Responsys
>> 
>> 
> 

Re: producer transaction and threading model

[Matthias J. Sax]

That is not possible. A producer can only have a single open
transaction. If your example, the call of thread-2 to start a new
transactions would fail and an exception would be thrown.

-Matthias

On 11/13/19 10:52 AM, Anindya Haldar wrote:
> The producer API doc 
> (https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>  
> )
>  says that KafkaProducer is thread safe.
> 
> What I was trying to figure out is what the relationship is between a thread 
> context and a transaction initiated in that thread. When multiple threads are 
> using a single producer instance and initiating transactions are those 
> transaction boundaries scoped to the thread context, or do they step on each 
> others’ toes?
> 
> Suppose two threads are sharing the same producer instance the following way:
> 
> thread-1 begins a transaction T1
> thread-1 starts sending messages in T!
> thread-2 begins a transaction T2
> thread-2 sends some messages in T2
> thread-2 gets and error and rolls back its transaction T2
> 
> At this point does the rollback of transaction T2 by the thread-2 affect the 
> transaction T1 remaining open and used by thread-1 in any fashion? It is not 
> clear to me from the API documentation. Will very much appreciate some 
> insights into the behavior here. Similar questions also arise for commits by 
> threads independently of each other.
> 
> Sincerely,
> Anindya Haldar
> Oracle Responsys
> 
> 



signature.asc
Description: OpenPGP digital signature

producer transaction and threading model

[Anindya Haldar]

The producer API doc 
(https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
 
)
 says that KafkaProducer is thread safe.

What I was trying to figure out is what the relationship is between a thread 
context and a transaction initiated in that thread. When multiple threads are 
using a single producer instance and initiating transactions are those 
transaction boundaries scoped to the thread context, or do they step on each 
others’ toes?

Suppose two threads are sharing the same producer instance the following way:

thread-1 begins a transaction T1
thread-1 starts sending messages in T!
thread-2 begins a transaction T2
thread-2 sends some messages in T2
thread-2 gets and error and rolls back its transaction T2

At this point does the rollback of transaction T2 by the thread-2 affect the 
transaction T1 remaining open and used by thread-1 in any fashion? It is not 
clear to me from the API documentation. Will very much appreciate some insights 
into the behavior here. Similar questions also arise for commits by threads 
independently of each other.

Sincerely,
Anindya Haldar
Oracle Responsys