[jira] [Commented] (KAFKA-4822) Kafka producer implementation without additional threads, similar to sync producer of 0.8.

2017-03-01 Thread Ismael Juma (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890172#comment-15890172
 ] 

Ismael Juma commented on KAFKA-4822:


The data is sent as soon as possible by default (linger.ms=0), so if you wait 
for the Future to complete or use a callback, it should behave the same as 
before (the fact that the send is being done by a background thread is an 
implementation detail.

> Kafka producer implementation without additional threads, similar to sync 
> producer of 0.8.
> --
>
> Key: KAFKA-4822
> URL: https://issues.apache.org/jira/browse/KAFKA-4822
> Project: Kafka
>  Issue Type: New Feature
>  Components: producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Giri
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4822) Kafka producer implementation without additional threads, similar to sync producer of 0.8.

2017-03-01 Thread Giri (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890139#comment-15890139
 ] 

Giri commented on KAFKA-4822:
-

A producer implementation with finer control will be helpful in the above like 
cases.

> Kafka producer implementation without additional threads, similar to sync 
> producer of 0.8.
> --
>
> Key: KAFKA-4822
> URL: https://issues.apache.org/jira/browse/KAFKA-4822
> Project: Kafka
>  Issue Type: New Feature
>  Components: producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Giri
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4822) Kafka producer implementation without additional threads, similar to sync producer of 0.8.

2017-03-01 Thread Giri (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890130#comment-15890130
 ] 

Giri commented on KAFKA-4822:
-

i understand that the sync can be achieved by the above workaround, but still 
another thread will be spawned per producer to send the data in background.

i have a case where a predefined set of threads (40 to 80 depending on the 
machine conf) receive data from a persistent medium and this data is sent to 
kafka (1 producer per thread) and i have to commit the position of the 
persistent medium to protect against restarts. I have achieved this in 0.8 sync 
producer by storing the data from persistent medium in list and after some 
content is cached i sent to kafka as batch and committed the position in the 
persistent medium. But now i do not have explicit control over when the data 
will be sent to kafka as data is completely handled by the new network thread 
and also the batch full and new batch are not visible to user (lost at 
KafkaProducer.doSend()). 

> Kafka producer implementation without additional threads, similar to sync 
> producer of 0.8.
> --
>
> Key: KAFKA-4822
> URL: https://issues.apache.org/jira/browse/KAFKA-4822
> Project: Kafka
>  Issue Type: New Feature
>  Components: producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Giri
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4822) Kafka producer implementation without additional threads, similar to sync producer of 0.8.

2017-03-01 Thread huxi (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889871#comment-15889871
 ] 

huxi commented on KAFKA-4822:
-

For KafkaProducer, all writes are asynchronous by default. Snippets below 
implements synchronous writes:
{code}
Future future = producer.send(record);
RecordMetadata metadata = future.get();
{code}



> Kafka producer implementation without additional threads, similar to sync 
> producer of 0.8.
> --
>
> Key: KAFKA-4822
> URL: https://issues.apache.org/jira/browse/KAFKA-4822
> Project: Kafka
>  Issue Type: New Feature
>  Components: producer 
>Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Giri
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)