Thanks, Yuriy, That sounds good to me.
-John On Wed, Aug 5, 2020, at 00:24, Yuriy Badalyantc wrote: > Hi John! > > I understand your point. But, I'm still not sure that having a Unit serde > is a good idea because it could be ambiguous for the user. In a KStream > [Long, Unit] valid value could mean one the following: > 1. Empty byte array > 2. null > 3. Any value (value ignored) > > As a user of a library, I think all these options are more or less valid. > That's because the Unit type in my opinion has no clear representation in > terms of values and deserialization. > > About Null serde - I'm ok with this. But, it should be a low priority > implicit to not intersect will other implicits. I will add it. > > -Yuriy > > On Sat, Jul 11, 2020 at 9:18 PM John Roesler <vvcep...@apache.org> wrote: > > > Thanks, Yuriy, > > > > IIRC, I added VoidSerde because there are times when the key or value is > > always null, but since ‘null’ has no type in Java, we had used the > > ByteArraySerde or some other proxy. This is confusing, because then the > > type is ‘byte[]’ although we intended it to always be null. It also delays > > finding bugs because serdes themselves force runtime type checks, but the > > ByteArraySerde accepts all data. > > > > Anyway, Void isn’t the type of null, but it’s close enough. Although > > Scala’s equivalent to “void” method is Unit methods, and like Void, Unit is > > uninstantiable, Unit has a value, also called Unit or “()”, whereas Void > > has no value in Java. But because Void is still a reference type, it can be > > null (I.e., it can only be null), so I’m some sense, it’s closer to the > > Scala type Null. > > > > I guess Scala users would benefit from both a NullSerde and a UnitSerde, > > but not a VoidSerde. > > > > Thanks for bringing it up, Matthias. > > > > -John > > > > On Fri, Jul 10, 2020, at 21:49, Yuriy Badalyantc wrote: > > > Ok, I mentioned adding missing serdes in the Proposed Change paragraph. > > > > > > About VoidSerde. I didn't add it intentionally. The semantic of the Unit > > > (scala's void) type is not clear in terms of the data. If kafka topic > > > contains messages of type Unit, what does it actually means? That there > > is > > > always null? Well, for that we have a Null type. That it's an empty byte > > > array? For that, we have an Array[Byte]. Empty string? No, it's a String. > > > So, I decided to not include Unit serde in the built-in Serdes. And if a > > > user will want to use the Unit type he can implement its own serde. > > > > > > -Yuriy > > > > > > On Fri, Jul 10, 2020 at 11:21 PM Matthias J. Sax <mj...@apache.org> > > wrote: > > > > > > > Thanks Yuriy! > > > > > > > > What about `VoidSerde` ? It's not listed. > > > > > > > > It might also be nice to add a short sentence and state that in > > addition > > > > to fixing the name collisions, the KIP will also close the gap of > > > > out-of-the-box serdes and add missing Serdes that are offered in Java > > to > > > > Scala. > > > > > > > > > > > > -Matthias > > > > > > > > On 7/10/20 7:51 AM, Yuriy Badalyantc wrote: > > > > > Oh, ok. I have done that. Just didn't know that it was necessary. > > > > > > > > > > -Yuriy > > > > > > > > > > On Fri, Jul 10, 2020 at 9:30 PM John Roesler <vvcep...@apache.org> > > > > wrote: > > > > > > > > > >> Ah, thanks Yuriy, > > > > >> > > > > >> Sorry if this wasn't clear, but _all_ public API changes have to > > > > >> be explicitly included in the KIP. Can you just enumerate all > > > > >> the contents of the new API? > > > > >> > > > > >> Thanks, > > > > >> John > > > > >> > > > > >> On Fri, Jul 10, 2020, at 04:54, Yuriy Badalyantc wrote: > > > > >>> Hi, Matthias, > > > > >>> > > > > >>> It's not directly mentioned in the KIP, but I added all missing > > Java > > > > >>> serdes. I mentioned it in the pull request description: > > > > >>> https://github.com/apache/kafka/pull/8955 > > > > >>> > > > > >>> And also, this KIP originally was based on a pull request where I > > added > > > > >>> missing java serdes :) https://github.com/apache/kafka/pull/8049 > > > > >>> > > > > >>> -Yuriy > > > > >>> > > > > >>> On Fri, Jul 10, 2020 at 3:36 AM Matthias J. Sax <mj...@apache.org> > > > > >> wrote: > > > > >>> > > > > >>>> Yuriy, > > > > >>>> > > > > >>>> thanks for the KIP update. I have one follow up thought: I checked > > > > what > > > > >>>> default Serdes we offer in the Java class > > > > >>>> > > > > >>>> `org.apache.kafka.common.serialization.Serdes` > > > > >>>> > > > > >>>> and I think it would be good if we could close the gap between the > > > > Java > > > > >>>> and Scala code and add the missing Java Serdes in Scala, too. > > > > >>>> > > > > >>>> It seems we are missing `Short` (Java and Scala), `Void`, `UUID`, > > and > > > > >>>> `ByterBuffer`. > > > > >>>> > > > > >>>> Can we add those in addition? > > > > >>>> > > > > >>>> > > > > >>>> -Matthias > > > > >>>> > > > > >>>> On 7/8/20 6:45 AM, John Roesler wrote: > > > > >>>>> Hi Yuriy, > > > > >>>>> > > > > >>>>> Once it seems like there’s general agreement in the discussion, > > you > > > > >> can > > > > >>>> start a voting thread. You can find examples on the mailing list > > of > > > > >> what to > > > > >>>> say in the first message. It’s basically just a message with the > > > > >> subject > > > > >>>> line changed from “[DISCUSS]...” to “[VOTE]...”, and then stating > > that > > > > >>>> you’d like to start the vote. It’s nice to link to the kip > > document > > > > >> again. > > > > >>>>> > > > > >>>>> The rules for the vote are at the top of the “Kafka Improvement > > > > >> Process” > > > > >>>> page, but you basically need 3 binding +1 votes and no binding -1 > > > > >> votes. > > > > >>>> You also need to wait at least three days from when you start the > > vote > > > > >>>> before you can declare it accepted. There’s no upper time limit. > > > > >>>>> > > > > >>>>> If you’re unsure of who has a binding vote, it’s just the people > > > > >> listed > > > > >>>> on the Apache Kafka Committers page. > > > > >>>>> > > > > >>>>> If people are slow to vote, feel free to keep bumping the thread, > > > > >> just > > > > >>>> like with the discussion. > > > > >>>>> > > > > >>>>> Thanks again for getting involved! > > > > >>>>> -John > > > > >>>>> > > > > >>>>> On Tue, Jul 7, 2020, at 01:51, Yuriy Badalyantc wrote: > > > > >>>>>> So, what's next? It's my first KIP and I'm not familiar with all > > > > >>>> processes. > > > > >>>>>> > > > > >>>>>> -Yuriy > > > > >>>>>> > > > > >>>>>> On Mon, Jul 6, 2020 at 1:32 AM John Roesler < > > vvcep...@apache.org> > > > > >>>> wrote: > > > > >>>>>> > > > > >>>>>>> Hi Yuriy, > > > > >>>>>>> > > > > >>>>>>> Thanks for the update! It looks good to me. > > > > >>>>>>> > > > > >>>>>>> Thanks, > > > > >>>>>>> John > > > > >>>>>>> > > > > >>>>>>> On Sun, Jul 5, 2020, at 03:27, Yuriy Badalyantc wrote: > > > > >>>>>>>> Hi John. > > > > >>>>>>>> > > > > >>>>>>>> I updated the KIP. An old proposed implementation is now in > > the > > > > >>>> rejected > > > > >>>>>>>> alternatives. > > > > >>>>>>>> > > > > >>>>>>>> - Yuriy > > > > >>>>>>>> > > > > >>>>>>>> On Sun, Jul 5, 2020 at 12:03 AM John Roesler < > > vvcep...@apache.org > > > > >>> > > > > >>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> Hi Yuriy, > > > > >>>>>>>>> > > > > >>>>>>>>> I agree, we can keep them separate. I just wanted to make you > > > > >> aware > > > > >>>> of > > > > >>>>>>> it. > > > > >>>>>>>>> > > > > >>>>>>>>> Thanks for the PR, it looks the way I expected. > > > > >>>>>>>>> > > > > >>>>>>>>> I just read over the KIP document again. I think it needs to > > be > > > > >>>>>>> updated to > > > > >>>>>>>>> the current proposal, and then we’ll be able to start the > > vote. > > > > >>>>>>>>> > > > > >>>>>>>>> Thanks, > > > > >>>>>>>>> John > > > > >>>>>>>>> > > > > >>>>>>>>> On Tue, Jun 30, 2020, at 04:58, Yuriy Badalyantc wrote: > > > > >>>>>>>>>> Hi everybody! > > > > >>>>>>>>>> > > > > >>>>>>>>>> Looks like a discussion about KIP-513 could take a while. I > > > > >> think we > > > > >>>>>>>>> should > > > > >>>>>>>>>> move forward with KIP-616 without waiting for KIP-513. > > > > >>>>>>>>>> > > > > >>>>>>>>>> I created a new pull request for KIP-616: > > > > >>>>>>>>>> https://github.com/apache/kafka/pull/8955. It contains a > > new > > > > >>>>>>>>>> `org.apache.kafka.streams.scala.serialization.Serdes` object > > > > >> without > > > > >>>>>>> name > > > > >>>>>>>>>> clash. An old one was marked as deprecated. This change is > > > > >> backward > > > > >>>>>>>>>> compatible and it could be merged in any further release. > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Wed, Jun 3, 2020 at 12:41 PM Yuriy Badalyantc < > > > > >> lmne...@gmail.com > > > > >>>>> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> Hi, John > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Thanks for pointing that out. I expressed my thoughts about > > > > >>>>>>> KIP-513 and > > > > >>>>>>>>>>> its connection to KIP-616 in the KIP-513 mail list. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> - Yuriy > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On Sun, May 31, 2020 at 1:26 AM John Roesler < > > > > >> vvcep...@apache.org> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> Hi Yuriy, > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> I was just looking back at KIP-513, and I’m wondering if > > > > >> there’s > > > > >>>>>>> any > > > > >>>>>>>>>>>> overlap we should consider here, or if they are just > > > > >> orthogonal. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>> -John > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Thu, May 28, 2020, at 21:36, Yuriy Badalyantc wrote: > > > > >>>>>>>>>>>>> At the current moment, I think John's plan is better > > than the > > > > >>>>>>>>> original > > > > >>>>>>>>>>>> plan > > > > >>>>>>>>>>>>> described in the KIP. I think we should create a new > > > > >> `Serdes` in > > > > >>>>>>>>> another > > > > >>>>>>>>>>>>> package. The old one will be deprecated. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> - Yuriy > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> On Fri, May 29, 2020 at 8:58 AM John Roesler < > > > > >>>>>>> vvcep...@apache.org> > > > > >>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Thanks, Matthias, > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> If we go with the approach Yuriy and I agreed on, to > > > > >>>>>>> deprecate and > > > > >>>>>>>>>>>> replace > > > > >>>>>>>>>>>>>> the whole class and not just a few of the methods, then > > the > > > > >>>>>>>>> timeline > > > > >>>>>>>>>>>> is > > > > >>>>>>>>>>>>>> less of a concern. Under that plan, Yuriy can just > > write the > > > > >>>>>>> new > > > > >>>>>>>>> class > > > > >>>>>>>>>>>>>> exactly the way he wants and people can cleanly swap > > over to > > > > >>>>>>> the > > > > >>>>>>>>> new > > > > >>>>>>>>>>>>>> pattern when they are ready. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> The timeline was more significant if we were just going > > to > > > > >>>>>>>>> deprecate > > > > >>>>>>>>>>>> some > > > > >>>>>>>>>>>>>> methods and add new methods to the existing class. That > > plan > > > > >>>>>>>>> requires > > > > >>>>>>>>>>>> two > > > > >>>>>>>>>>>>>> implementation phases, where we first deprecate the > > existing > > > > >>>>>>>>> methods > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>>>> later swap the implicits at the same time we remove the > > > > >>>>>>> deprecated > > > > >>>>>>>>>>>> members. > > > > >>>>>>>>>>>>>> Aside from the complexity of that approach, it’s not a > > > > >>>>>>> breakage > > > > >>>>>>>>> free > > > > >>>>>>>>>>>> path, > > > > >>>>>>>>>>>>>> as some users would be forced to continue using the > > > > >> deprecated > > > > >>>>>>>>> members > > > > >>>>>>>>>>>>>> until a future release drops them, breaking their source > > > > >>>>>>> code, and > > > > >>>>>>>>>>>> only > > > > >>>>>>>>>>>>>> then can they update their code. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> That wouldn’t be the end of the world, and we’ve had to > > do > > > > >> the > > > > >>>>>>>>> same > > > > >>>>>>>>>>>> thing > > > > >>>>>>>>>>>>>> in the past with the implicit conversations, but this > > is a > > > > >>>>>>> much > > > > >>>>>>>>> wider > > > > >>>>>>>>>>>>>> scope, since it’s all the serdes. I’m happy with the new > > > > >> plan, > > > > >>>>>>>>> since > > > > >>>>>>>>>>>> it’s > > > > >>>>>>>>>>>>>> not only one step, but also it provides everyone a > > > > >>>>>>> breakage-free > > > > >>>>>>>>> path. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> We can still consider dropping the deprecated class in > > 3.0; > > > > >> I > > > > >>>>>>> just > > > > >>>>>>>>>>>> wanted > > > > >>>>>>>>>>>>>> to clarify how the timeline issue has changed. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>>>> John > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> On Thu, May 28, 2020, at 20:34, Matthias J. Sax wrote: > > > > >>>>>>>>>>>>>>> I am not a Scale person, so I cannot really contribute > > > > >> much. > > > > >>>>>>>>>>>> However for > > > > >>>>>>>>>>>>>>> the deprecation period, if we get the change into 2.7, > > it > > > > >>>>>>> might > > > > >>>>>>>>> be > > > > >>>>>>>>>>>> ok to > > > > >>>>>>>>>>>>>>> remove the deprecated classed in 3.0. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> It would only be one minor release in between what is a > > > > >>>>>>> little > > > > >>>>>>>>> bit > > > > >>>>>>>>>>>> short > > > > >>>>>>>>>>>>>>> (we usually prefer at least two minor released, better > > > > >>>>>>> three), > > > > >>>>>>>>> but > > > > >>>>>>>>>>>> if we > > > > >>>>>>>>>>>>>>> have a good reason for it, it might be ok. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> If we cannot remove it in 3.0, it seems there would be > > a > > > > >>>>>>> 4.0 in > > > > >>>>>>>>>>>> about a > > > > >>>>>>>>>>>>>>> year(?) when ZK removal is finished and we can remove > > the > > > > >>>>>>>>> deprecated > > > > >>>>>>>>>>>>>>> code than. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> -Matthias > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On 5/28/20 7:39 AM, John Roesler wrote: > > > > >>>>>>>>>>>>>>>> Hi Yuriy, > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Sounds good to me! I had a feeling we were bringing > > > > >>>>>>> different > > > > >>>>>>>>>>>> context > > > > >>>>>>>>>>>>>>>> to the discussion; thanks for sticking with the > > > > >>>>>>> conversation > > > > >>>>>>>>>>>> until we > > > > >>>>>>>>>>>>>> got > > > > >>>>>>>>>>>>>>>> it hashed out. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> I'm glad you prefer Serde*s*, since having multiple > > > > >>>>>>> different > > > > >>>>>>>>>>>> classes > > > > >>>>>>>>>>>>>> with > > > > >>>>>>>>>>>>>>>> the same name leads to all kinds of trouble. "Serdes" > > > > >>>>>>> seems > > > > >>>>>>>>>>>> relatively > > > > >>>>>>>>>>>>>>>> safe because people in the Scala lib won't be using > > the > > > > >>>>>>> Java > > > > >>>>>>>>>>>> Serdes > > > > >>>>>>>>>>>>>> class, > > > > >>>>>>>>>>>>>>>> and they won't be using the deprecated and > > non-deprecated > > > > >>>>>>> one > > > > >>>>>>>>> at > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> same time. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Thank again, > > > > >>>>>>>>>>>>>>>> -John > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On Thu, May 28, 2020, at 02:21, Yuriy Badalyantc > > wrote: > > > > >>>>>>>>>>>>>>>>> Ok, I understood you, John. I wasn't sure about kafka > > > > >>>>>>>>> deprecation > > > > >>>>>>>>>>>>>> policy > > > > >>>>>>>>>>>>>>>>> and thought that the full cycle could be done with > > 2.7 > > > > >>>>>>>>> version. > > > > >>>>>>>>>>>>>> Waiting for > > > > >>>>>>>>>>>>>>>>> 3.0 is too much, I agree with it. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> So, I think creating one more `Serdes` in another > > > > >>>>>>> package is > > > > >>>>>>>>> our > > > > >>>>>>>>>>>> way. > > > > >>>>>>>>>>>>>> I > > > > >>>>>>>>>>>>>>>>> suggest one of the following: > > > > >>>>>>>>>>>>>>>>> 1. `org.apache.kafka.streams.scala.serde.Serdes` > > > > >>>>>>>>>>>>>>>>> 2. > > `org.apache.kafka.streams.scala.serialization.Serdes` > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> About `Serde` vs `Serdes`. I'm strongly against > > `Serde` > > > > >>>>>>>>> because > > > > >>>>>>>>>>>> it > > > > >>>>>>>>>>>>>> would > > > > >>>>>>>>>>>>>>>>> lead to a new name clash with the > > > > >>>>>>>>>>>>>>>>> `org.apache.kafka.common.serialization.Serde`. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> - Yuriy > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 11:12 AM John Roesler < > > > > >>>>>>>>>>>> vvcep...@apache.org> > > > > >>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Hi Yuriy, > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Thanks for the clarification. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> I guess my concern is twofold: > > > > >>>>>>>>>>>>>>>>>> 1. We typically leave deprecated methods in place > > for at > > > > >>>>>>>>> least a > > > > >>>>>>>>>>>>>> major > > > > >>>>>>>>>>>>>>>>>> release cycle before removing them, so it would seem > > > > >>>>>>> abrupt > > > > >>>>>>>>> to > > > > >>>>>>>>>>>> have a > > > > >>>>>>>>>>>>>>>>>> deprecation period of only one minor release. If we > > > > >>>>>>> follow > > > > >>>>>>>>> the > > > > >>>>>>>>>>>> same > > > > >>>>>>>>>>>>>> pattern > > > > >>>>>>>>>>>>>>>>>> here, it would take over a year to finish this KIP. > > > > >>>>>>>>>>>>>>>>>> 2. It doesn’t seem like there is a nonbreaking > > > > >>>>>>> deprecation > > > > >>>>>>>>> path > > > > >>>>>>>>>>>> at > > > > >>>>>>>>>>>>>> all if > > > > >>>>>>>>>>>>>>>>>> people enumerate their imports (if they don’t use a > > > > >>>>>>>>> wildcard). > > > > >>>>>>>>>>>> In > > > > >>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>> case, they would have no path to implicitly use the > > > > >>>>>>> newly > > > > >>>>>>>>> named > > > > >>>>>>>>>>>>>> serdes, and > > > > >>>>>>>>>>>>>>>>>> therefore they would have no way to avoid > > continuing to > > > > >>>>>>> use > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>> deprecated > > > > >>>>>>>>>>>>>>>>>> ones. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Since you mentioned that your reason is mainly the > > > > >>>>>>>>> preference > > > > >>>>>>>>>>>> for > > > > >>>>>>>>>>>>>> the name > > > > >>>>>>>>>>>>>>>>>> “Serde” or “Serdes”, can we explore just using one > > of > > > > >>>>>>> those? > > > > >>>>>>>>>>>> Would > > > > >>>>>>>>>>>>>> it cause > > > > >>>>>>>>>>>>>>>>>> some kind of conflict to use > > > > >>>>>>>>>>>> org.apache.kafka.streams.scala.Serde or > > > > >>>>>>>>>>>>>> to use > > > > >>>>>>>>>>>>>>>>>> Serdes in a different package, like > > > > >>>>>>>>>>>>>>>>>> org.apache.kafka.streams.scala.implicit.Serdes? > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> I empathize with this desire. I faced the same > > dilemma > > > > >>>>>>> when > > > > >>>>>>>>> I > > > > >>>>>>>>>>>> wanted > > > > >>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> replace Processor but keep the class name in > > KIP-478. I > > > > >>>>>>>>> wound up > > > > >>>>>>>>>>>>>> creating a > > > > >>>>>>>>>>>>>>>>>> new package for the new Processor. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>>>>>>>> John > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> On Wed, May 27, 2020, at 22:20, Yuriy Badalyantc > > wrote: > > > > >>>>>>>>>>>>>>>>>>> Hi John, > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> I'm stick with the > > > > >>>>>>> `org.apache.kafka.streams.scala.Serdes` > > > > >>>>>>>>>>>> because > > > > >>>>>>>>>>>>>> it's > > > > >>>>>>>>>>>>>>>>>>> sort of conventional in the scala community. If you > > > > >>>>>>> have a > > > > >>>>>>>>>>>> typeclass > > > > >>>>>>>>>>>>>>>>>> `Foo`, > > > > >>>>>>>>>>>>>>>>>>> you probably will search `Foo` related stuff in the > > > > >>>>>>> `Foo` > > > > >>>>>>>>> or > > > > >>>>>>>>>>>> maybe > > > > >>>>>>>>>>>>>> `Foos` > > > > >>>>>>>>>>>>>>>>>>> (plural). All other places are far less > > discoverable > > > > >>>>>>> for > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>> developers. > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> I agree that the migration path is a bit complex > > for > > > > >>>>>>> such > > > > >>>>>>>>>>>> change. > > > > >>>>>>>>>>>>>> But I > > > > >>>>>>>>>>>>>>>>>>> think it's more important to provide good developer > > > > >>>>>>>>> experience > > > > >>>>>>>>>>>> than > > > > >>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>>> simplify migration. Also, I think it's debatable > > which > > > > >>>>>>>>>>>> migration > > > > >>>>>>>>>>>>>> path is > > > > >>>>>>>>>>>>>>>>>>> better for library users. If we would create, for > > > > >>>>>>> example, > > > > >>>>>>>>>>>>>> `Serdes2`, > > > > >>>>>>>>>>>>>>>>>>> library users will have to modify their code if > > they > > > > >>>>>>> used > > > > >>>>>>>>> any > > > > >>>>>>>>>>>> part > > > > >>>>>>>>>>>>>> of the > > > > >>>>>>>>>>>>>>>>>>> old `Serde`. With my approach, most of the old code > > > > >>>>>>> will > > > > >>>>>>>>> still > > > > >>>>>>>>>>>> work > > > > >>>>>>>>>>>>>>>>>> without > > > > >>>>>>>>>>>>>>>>>>> changes. Only explicit usage of implicits will > > need to > > > > >>>>>>> be > > > > >>>>>>>>> fixed > > > > >>>>>>>>>>>>>> (because > > > > >>>>>>>>>>>>>>>>>>> names will be changed, and old names will be > > > > >>>>>>> deprecated). > > > > >>>>>>>>>>>> Wildcard > > > > >>>>>>>>>>>>>>>>>> imports > > > > >>>>>>>>>>>>>>>>>>> will work without changes and will not lead to a > > name > > > > >>>>>>>>> clash. > > > > >>>>>>>>>>>>>> Moreover, > > > > >>>>>>>>>>>>>>>>>> many > > > > >>>>>>>>>>>>>>>>>>> users may not notice name clash problems. And with > > my > > > > >>>>>>>>> migration > > > > >>>>>>>>>>>>>> path, > > > > >>>>>>>>>>>>>>>>>> they > > > > >>>>>>>>>>>>>>>>>>> will not notice any changes at all. > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> - Yuriy > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> On Thu, May 28, 2020 at 7:48 AM John Roesler < > > > > >>>>>>>>>>>> vvcep...@apache.org> > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Hi Yuriy, > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I guess I've been out of the > > > > >>>>>>> Scala > > > > >>>>>>>>> game > > > > >>>>>>>>>>>> for a > > > > >>>>>>>>>>>>>>>>>>>> while; all this summoner business is totally new > > to > > > > >>>>>>> me. > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> I think I followed the rationale you provided, > > but I > > > > >>>>>>> still > > > > >>>>>>>>>>>> don't > > > > >>>>>>>>>>>>>> see > > > > >>>>>>>>>>>>>>>>>>>> why you can't implement your whole plan in a new > > > > >>>>>>> class. > > > > >>>>>>>>> What > > > > >>>>>>>>>>>>>>>>>>>> is special about the existing Serdes class? > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> Thanks, > > > > >>>>>>>>>>>>>>>>>>>> -John > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020, at 01:18, Yuriy Badalyantc > > > > >>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>> Hi John, > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Your suggestion looks interesting. I think it's > > > > >>>>>>>>> technically > > > > >>>>>>>>>>>>>> doable. > > > > >>>>>>>>>>>>>>>>>> But > > > > >>>>>>>>>>>>>>>>>>>> I'm > > > > >>>>>>>>>>>>>>>>>>>>> not sure that this is the better solution. I will > > > > >>>>>>> try to > > > > >>>>>>>>>>>> explain. > > > > >>>>>>>>>>>>>>>>>> From > > > > >>>>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>>> scala developers' perspective, `Serde` looks > > really > > > > >>>>>>> like > > > > >>>>>>>>> a > > > > >>>>>>>>>>>>>> typeclass. > > > > >>>>>>>>>>>>>>>>>>>>> Typical typeclass in pure scala will look like > > this: > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> trait Serde[A] { > > > > >>>>>>>>>>>>>>>>>>>>> def serialize(data: A): Array[Byte] > > > > >>>>>>>>>>>>>>>>>>>>> def deserialize(data: Array[Byte]): A > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> object Serde extends DefaultSerdes { > > > > >>>>>>>>>>>>>>>>>>>>> // "summoner" function. With this I can write > > > > >>>>>>>>> `Serde[A]` > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>> serde > > > > >>>>>>>>>>>>>>>>>>>>> will be implicitly summonned. > > > > >>>>>>>>>>>>>>>>>>>>> def apply[A](implicit ev: Serde[A]): Serde[A] > > = ev > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> trait DefaultSerdes { > > > > >>>>>>>>>>>>>>>>>>>>> // default instances here > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Usage example (note, that there are no wildcards > > > > >>>>>>> imports > > > > >>>>>>>>>>>> here): > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App { > > > > >>>>>>>>>>>>>>>>>>>>> import Serde // not wildcard import > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> // explicit summonning: > > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serde[String] // using > > summoner > > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???) > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> // implicit summonning > > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = { > > > > >>>>>>>>>>>>>>>>>>>>> Serde[A].serialize(a) // summoner again > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo") > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Examples are pretty silly, but I just want to > > show > > > > >>>>>>> common > > > > >>>>>>>>>>>>>> patterns of > > > > >>>>>>>>>>>>>>>>>>>>> working with typeclasses in scala. All default > > > > >>>>>>> instances > > > > >>>>>>>>> in > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>> usage > > > > >>>>>>>>>>>>>>>>>>>>> examples are found using implicits searching > > > > >>>>>>> mechanism. > > > > >>>>>>>>> Scala > > > > >>>>>>>>>>>>>>>>>> compiler > > > > >>>>>>>>>>>>>>>>>>>>> searches implicits in a lot of places. Including > > > > >>>>>>>>> companion > > > > >>>>>>>>>>>>>> objects. > > > > >>>>>>>>>>>>>>>>>> In my > > > > >>>>>>>>>>>>>>>>>>>>> examples compiler will found `Serde[String]` > > > > >>>>>>> instance in > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> companion > > > > >>>>>>>>>>>>>>>>>>>>> object of `Serde` typeclass. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Also, I want to pay attention to the summoner > > > > >>>>>>> function. > > > > >>>>>>>>> It > > > > >>>>>>>>>>>> makes > > > > >>>>>>>>>>>>>>>>>> usage of > > > > >>>>>>>>>>>>>>>>>>>>> typeclasses very neat and clear. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> The example above was the example of the perfect > > > > >>>>>>> solution > > > > >>>>>>>>>>>> for the > > > > >>>>>>>>>>>>>>>>>> scala > > > > >>>>>>>>>>>>>>>>>>>>> developers. But this solution requires to create > > > > >>>>>>> separate > > > > >>>>>>>>>>>> `Serde` > > > > >>>>>>>>>>>>>>>>>>>>> typeclass, to make all this implicit searching > > stuff > > > > >>>>>>>>> works. I > > > > >>>>>>>>>>>>>> don't > > > > >>>>>>>>>>>>>>>>>> think > > > > >>>>>>>>>>>>>>>>>>>>> that it worth it, because a lot of code should be > > > > >>>>>>>>>>>> reimplemented > > > > >>>>>>>>>>>>>> using > > > > >>>>>>>>>>>>>>>>>>>> this > > > > >>>>>>>>>>>>>>>>>>>>> new typeclass. But the main point of my example > > is to > > > > >>>>>>>>> show > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> perfect > > > > >>>>>>>>>>>>>>>>>>>>> solution. And I think we should strive to provide > > > > >>>>>>>>> developer > > > > >>>>>>>>>>>>>>>>>> experience > > > > >>>>>>>>>>>>>>>>>>>>> close to this. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> It's a bit out of the scope of my KIP, but I > > have a > > > > >>>>>>> plan > > > > >>>>>>>>> to > > > > >>>>>>>>>>>> make > > > > >>>>>>>>>>>>>>>>>>>>> `org.apache.kafka.streams.scala.Serdes` more > > closer > > > > >>>>>>> to > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>> solution > > > > >>>>>>>>>>>>>>>>>>>> above. > > > > >>>>>>>>>>>>>>>>>>>>> It could be done in 2 steps: > > > > >>>>>>>>>>>>>>>>>>>>> 1. Fix implicit names. > > > > >>>>>>>>>>>>>>>>>>>>> 2. Add summoner function. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> And with this scala developers will be able to > > write > > > > >>>>>>>>> almost > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>> same > > > > >>>>>>>>>>>>>>>>>> code > > > > >>>>>>>>>>>>>>>>>>>>> as in the example above: > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App { > > > > >>>>>>>>>>>>>>>>>>>>> import org.apache.kafka.streams.scala.Serdes > > // not > > > > >>>>>>>>>>>> wildcard > > > > >>>>>>>>>>>>>> import > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = Serdes[String] > > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???) > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = { > > > > >>>>>>>>>>>>>>>>>>>>> Serdes[A].serialize(a) > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo") > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> Of course, wildcard import will still work. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> Other names will make this new entity (containing > > > > >>>>>>> default > > > > >>>>>>>>>>>>>> implicits) > > > > >>>>>>>>>>>>>>>>>> less > > > > >>>>>>>>>>>>>>>>>>>>> discoverable. And summoner usage, in this case, > > will > > > > >>>>>>> look > > > > >>>>>>>>>>>> weird: > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> object Main extends App { > > > > >>>>>>>>>>>>>>>>>>>>> import > > > > >>>>>>> org.apache.kafka.streams.scala.DefaultSerdes // > > > > >>>>>>>>> not > > > > >>>>>>>>>>>>>> wildcard > > > > >>>>>>>>>>>>>>>>>>>> import > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> val stringSerde = DefaultSerdes[String] > > > > >>>>>>>>>>>>>>>>>>>>> stringSerde.serialize(???) > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> def serialize[A: Serde](a: A) = { > > > > >>>>>>>>>>>>>>>>>>>>> DefaultSerdes[A].serialize(a) > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> serialize("foo") > > > > >>>>>>>>>>>>>>>>>>>>> } > > > > >>>>>>>>>>>>>>>>>>>>> ``` > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> So, I think it's more important to provide a > > solid > > > > >>>>>>> and > > > > >>>>>>>>>>>> familiar > > > > >>>>>>>>>>>>>>>>>> developer > > > > >>>>>>>>>>>>>>>>>>>>> experience for the scala developer. And renaming > > (or > > > > >>>>>>>>>>>> creating a > > > > >>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>>>>> version) of `Serdes` will not help here. > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> -Yuriy > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> On Tue, May 19, 2020 at 11:56 AM John Roesler < > > > > >>>>>>>>>>>>>> vvcep...@apache.org> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Yuriy, > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks so much for the KIP! I didn’t anticipate > > the > > > > >>>>>>>>> problem > > > > >>>>>>>>>>>> you > > > > >>>>>>>>>>>>>>>>>> laid > > > > >>>>>>>>>>>>>>>>>>>> out > > > > >>>>>>>>>>>>>>>>>>>>>> in the KIP, but I find it very plausible. > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for pushing back on the “convention” and > > > > >>>>>>> raising > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>>> issue, > > > > >>>>>>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>>>>>> also volunteering a solution! > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> I’m wondering if we can “fix” it in one shot by > > just > > > > >>>>>>>>>>>> deprecating > > > > >>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>>>> whole > > > > >>>>>>>>>>>>>>>>>>>>>> Serdes class and replacing it with a new one > > > > >>>>>>> containing > > > > >>>>>>>>> the > > > > >>>>>>>>>>>> defs > > > > >>>>>>>>>>>>>>>>>> you > > > > >>>>>>>>>>>>>>>>>>>>>> proposed. Then, people could just switch their > > > > >>>>>>> import to > > > > >>>>>>>>>>>> the new > > > > >>>>>>>>>>>>>>>>>> one. > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Of course the new class needs to have a > > different > > > > >>>>>>> name, > > > > >>>>>>>>>>>> which is > > > > >>>>>>>>>>>>>>>>>>>> always a > > > > >>>>>>>>>>>>>>>>>>>>>> challenge in situations like this, so I might > > just > > > > >>>>>>>>> throw out > > > > >>>>>>>>>>>>>>>>>>>> ImplicitSerdes > > > > >>>>>>>>>>>>>>>>>>>>>> as an option. > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Do you think this would work? > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks again, > > > > >>>>>>>>>>>>>>>>>>>>>> John > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> On Mon, May 18, 2020, at 23:35, Yuriy Badalyantc > > > > >>>>>>> wrote: > > > > >>>>>>>>>>>>>>>>>>>>>>> Hi, > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> I would like to propose KIP-616 to fix naming > > > > >>>>>>> clash in > > > > >>>>>>>>> the > > > > >>>>>>>>>>>> kafka > > > > >>>>>>>>>>>>>>>>>>>>>>> streams scala API: > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-616%3A+Rename+implicit+Serdes+instances+in+kafka-streams-scala > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> Looking forward to your feedback. > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>>> -Yuriy > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Attachments: > > > > >>>>>>>>>>>>>>> * signature.asc > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > >