[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-06-03 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356292#comment-17356292
 ] 

Sagar Rao commented on KAFKA-9168:
--

hi [~cadonna], Thanks for the feedback. Actually, what I had planned to do was 
probably write some jmh benchmarks for this. There is a ticket that I have 
assigned to myself. Once I see some gains there, then I would proceed with 
asking you to run internal benchmarks. Do you think that's a good idea?

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-06-03 Thread Bruno Cadonna (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356273#comment-17356273
 ] 

Bruno Cadonna commented on KAFKA-9168:
--

Hi [~sagarrao], our internal benchmark did not show any significant performance 
improvements (i.e.. throughput improvements). I left some comments on your PR. 
I think the best thing to move forward here would be to implement a simple 
Kafka Streams app with which you can experiment different usages of the direct 
buffer for different operations on the RocksDB state stores. Then if you have 
found some good use cases for the direct buffers that improve performance in 
Streams, we can run our internal benchmarks again and verify if we see the 
improvements also there. WDYT? 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-31 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354778#comment-17354778
 ] 

Sagar Rao commented on KAFKA-9168:
--

Thanks [~cadonna]! would like to see the results.

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-31 Thread Bruno Cadonna (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354586#comment-17354586
 ] 

Bruno Cadonna commented on KAFKA-9168:
--

[~sagarrao] and [~ableegoldman] I took the liberty to start the internal 
benchmark on the draft PR.

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-31 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354490#comment-17354490
 ] 

Sagar Rao commented on KAFKA-9168:
--

[~ableegoldman] i modified the put API for single column family.. The changes 
are minimal right now but they need to evolve. But, before that I was thinking 
if you could run the internal benchmarking tool just for this API, we can then 
take a call. I will come up with jmh benchmarks as well. Here is the draft PR:

https://github.com/apache/kafka/pull/10798

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-26 Thread A. Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351946#comment-17351946
 ] 

A. Sophie Blee-Goldman commented on KAFKA-9168:
---

Sounds good! This shouldn't need a KIP since the byte[] inputs are only used at 
the lower levels of the state store hierarchy, not the top-level public API. 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-25 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351444#comment-17351444
 ] 

Sagar Rao commented on KAFKA-9168:
--

Sure thanks. Probably, I will send a patch for one of the APIs (let's say 
put()) and you can run benchmarks using the internal repo? Meanwhile, I have 
assigned the other ticket related to benchmarking framework to myself. I can 
start looking into it as well.

BTW, do you think this particular ticket, if the numbers look fine,  would 
warrant a KIP?

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-25 Thread A. Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351377#comment-17351377
 ] 

A. Sophie Blee-Goldman commented on KAFKA-9168:
---

There's nothing available at the moment in AK, I actually just filed 
https://issues.apache.org/jira/browse/KAFKA-12848 for this. If you'd be 
interested in putting together a starter kit of jmh benchmarks that would 
definitely be great – but we do have an internal benchmarking repo so I can 
just run those on your patch if you'd prefer. We'd like to make some of it 
public eventually but it requires some footwork and maybe an audit so no one 
has found the time yet

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-25 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351243#comment-17351243
 ] 

Sagar Rao commented on KAFKA-9168:
--

Got it.. How do you want me to benchmark? Using the rocksdb bechmarking utility 
or is there any other way like jmh within kafka streams?

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-25 Thread A. Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351238#comment-17351238
 ] 

A. Sophie Blee-Goldman commented on KAFKA-9168:
---

Yep, I think that's exactly what we want to do with this ticket. Looking 
forward to the results

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-25 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17351230#comment-17351230
 ] 

Sagar Rao commented on KAFKA-9168:
--

[~ableegoldman], so I went through the github PR link- which honestly i hadn't 
done so far :D - and looks like it enables JNI direct byte buffer for all basic 
operations barring transactions. There is a code snippet explaining put 
operation using this new option:

 
{code:java}
// code placeholder

try (RocksDB db = RocksDB.open(opt, "PerformanceTest");
WriteOptions writeOptions = new WriteOptions()) {

writeOptions.setDisableWAL(true);

ByteBuffer directKeyBuffer = ByteBuffer.allocateDirect(128);
directKeyBuffer.order(ByteOrder.BIG_ENDIAN);

ByteBuffer directValueBuffer = ByteBuffer.allocateDirect(128);
directValueBuffer.order(ByteOrder.BIG_ENDIAN);

for (int i = 0; i < 1_000_000; i++) {
directKeyBuffer.clear();
directValueBuffer.clear();
for (int o = 0; o < 16; o++) {
directKeyBuffer.putLong(i);
directValueBuffer.putLong(i);
}
directKeyBuffer.flip();
directValueBuffer.flip();

db.put(writeOptions, directKeyBuffer, directValueBuffer);
}

{code}
As per the benchmarks released on the link, the iterator performace increased 
by 37% with 0 GC cycles compared to 293 for the byte array based approach. This 
is because there is no referenced memory.

So, looking at these numbers, and to answer to your question about where this 
ticket fits in, maybe we can start with some benchmarking the APIs by changing 
the way rocksdb state store apis are implemented using this new way? What I 
mean is, today if put() API implementation uses the byte[] based APIs, then we 
can benchmark using the ByteBuffer based approach and compare the numbers. Do 
you think that makes sense?

 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-24 Thread A. Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350650#comment-17350650
 ] 

A. Sophie Blee-Goldman commented on KAFKA-9168:
---

Yep, all the rocksdb work waiting on more recent versions has been unblocked. 
Disclaimer: I don't have much context on this particular task or how, or even 
whether, it's something we can do for current features. It may be that the 
direct buffers are only useful for new features that we were considering, and 
not anything in the current codebase. But I really have no idea – just wanted 
to clarify that figuring this out is I guess the first part of this ticket :) 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-05-23 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350118#comment-17350118
 ] 

Sagar Rao commented on KAFKA-9168:
--

[~ableegoldman], looks like Bruno has done his work on 
[Kafka-8897|https://issues.apache.org/jira/browse/KAFKA-8897]? Is this 
something that can be picked up?

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-04-19 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325430#comment-17325430
 ] 

Sagar Rao commented on KAFKA-9168:
--

Sure thanks [~ableegoldman], i have assinged this to myself. Whenever Bruno's 
done with his changes, i can start working on this one..

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Assignee: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-04-19 Thread A. Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325187#comment-17325187
 ] 

A. Sophie Blee-Goldman commented on KAFKA-9168:
---

[~sagarrao] Feel free to pick this up, but you may need to hold off on working 
on it until [~cadonna] has completed KAFKA-8897. I believe he's battling some 
weird runtime issues in RocksDB with the upgrade at the moment, but once we've 
figured that out then this should be unblocked. 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-04-19 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325182#comment-17325182
 ] 

Sagar Rao commented on KAFKA-9168:
--

hey [~guozhang], the benchmarks that I had talked about were for the prefix 
scan changes that i was working on back then. BTW, I can take this up if that's 
ok with you?

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2021-04-18 Thread Guozhang Wang (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324574#comment-17324574
 ] 

Guozhang Wang commented on KAFKA-9168:
--

Hi [~sagarrao] just for my own education could you paste the patch with 
benchmarks to this ticket so that I can understand who would you propose to 
integrate the direct byte buffer once we upgrade? Bruno is working on upgrading 
beyond 6.8.1 so we are on the road to do this soon.

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Sagar Rao
>Priority: Major
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2020-07-07 Thread Adam Bellemare (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152701#comment-17152701
 ] 

Adam Bellemare commented on KAFKA-9168:
---

This would be incredibly useful for improving the performance of KTable-KTable 
foreign key joins.

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Sagar Rao
>Priority: Blocker
>  Labels: perfomance
> Fix For: 3.0.0
>
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2020-05-22 Thread Guozhang Wang (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17114291#comment-17114291
 ] 

Guozhang Wang commented on KAFKA-9168:
--

Just add a reference that this feature is only available in 6.8.1+

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Sagar Rao
>Priority: Blocker
>  Labels: perfomance
> Fix For: 3.0.0
>
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2020-03-12 Thread Sophie Blee-Goldman (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058193#comment-17058193
 ] 

Sophie Blee-Goldman commented on KAFKA-9168:


Thanks [~sagarrao], marking this as a blocker for 3.0 since we unfortunately 
need to wait for the next major release before bumping the major version of 
rocksdb (they made a number of breaking changes that Streams users are 
unfortunately exposed to).

That said, this patch has the potential for some pretty significant performance 
improvements in addition to opening the door for other desirable features that 
aren't available to RocksJava users. Definitely one of the stronger motivations 
to consider a workaround that allows upgrading without introducing breaking 
changes to user code...

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Sagar Rao
>Priority: Blocker
>  Labels: perfomance
> Fix For: 3.0.0
>
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-9168) Integrate JNI direct buffer support to RocksDBStore

2020-02-20 Thread Sagar Rao (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-9168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040766#comment-17040766
 ] 

Sagar Rao commented on KAFKA-9168:
--

hey [~ableegoldman] you had asked me to create this ticket to track the closure 
of the particular PR mentioned in the external Link from RocksDB.

I couldn't find the relevant conversation we had on Slack but this was in 
context of the conversation we had a few months ago around adding Prefix Scan 
behaviour for State-stores(RocksDB for now). I have the changes ready with 
relevant benchmarks created but was waiting for this PR from RocksDB to get 
resolved which happened a couple of days ago.

So, coming back to this ticket, considering that the PR got merged to master in 
RocksDB codebase, is there anything that needs to be done in the streams side 
to incorporate these changes? 

> Integrate JNI direct buffer support to RocksDBStore
> ---
>
> Key: KAFKA-9168
> URL: https://issues.apache.org/jira/browse/KAFKA-9168
> Project: Kafka
>  Issue Type: Task
>  Components: streams
>Reporter: Sagar Rao
>Priority: Minor
>  Labels: perfomance
>
> There has been a PR created on rocksdb Java client to support direct 
> ByteBuffers in Java. We can look at integrating it whenever it gets merged. 
> Link to PR: [https://github.com/facebook/rocksdb/pull/2283]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)