> While doing the conversion, it definitely seemed like it introduced the possibility of more runtime exceptions than I would've expected, and these runtime exceptions were introduced by IntelliJ's conversion and found later when running TR.
That is not true. If you convert a single class to Kotlin, then it will interpret lots of class members and function arguments as nullable, but that is not the way to do it. It is very important to select the *entirety* of the Traffic Router codebase and tell Intellij to convert *that* to Kotlin. If you do this, then the vast majority of those class members and function arguments will *not* be falsely detected as nullable. >From my experience converting classes to Kotlin, the converter is *overly*-sensitive, and rather than fansely detecting variables as nullable, detects many variables as non-nullable, which our code did not previously account for, whichmakes our code safer as a result. > Needless to say, my confidence in IntelliJ's conversion sank after that, so the conversion would require some pretty careful combing over for me to be comfortable with the result. Again, if the conversion is performed using the steps above, it does *not* produce unsafe code. I have opened PR 7071 which adds the ability to use Kotlin in Traffic Router without converting any classes. I would ask that this is merged so that we can start the process of converting TR to Kotlin. A PR that converts many classes at a time would make changes difficult to track, so doing it granularly like this seems preferable. < https://github.com/apache/trafficcontrol/pull/7071> -Z On Tue, Jun 29, 2021 at 9:11 PM Rawlin Peters <raw...@apache.org> wrote: > > Can you explain more about how the conversion process works? > > @zrhoffman and I attempted to figure this out last week, and there are > a number of different ways you could go about it. I attempted to use > IntelliJ to convert the largest classes first (TrafficRouter and > ZoneManager). TrafficRouter was a pretty easy conversion with just a > few manual fix-ups, but converting ZoneManager would've required > hundreds of manual fix-ups, which seemed a bit too daunting for me to > continue onward. @zrhoffman then figured we should try tackling the > smallest packages first and go from there. It was a bit of work to get > the tests passing with Kotlin in the codebase, which led to us > upgrading the Mockito and PowerMock unit testing libraries: > https://github.com/apache/trafficcontrol/pull/5976 (which if nothing > else would help us use a newer Java version or Kotlin in the future). > > > What are the risks associated with it (likelihood of introducing a bug, > risks of having mixed Java+Kotlin, etc...). > > While doing the conversion, it definitely seemed like it introduced > the possibility of more runtime exceptions than I would've expected, > and these runtime exceptions were introduced by IntelliJ's conversion > and found later when running TR. For example, IntelliJ had inserted > some unsafe conversions from nullable to non-nullable types, and those > ended up causing runtime exceptions. Needless to say, my confidence in > IntelliJ's conversion sank after that, so the conversion would require > some pretty careful combing over for me to be comfortable with the > result. > > > Also, how long could the community expect the transition to take and what > are the mechanics (separate branch, etc...)? > > If we were to move forward with this, I think it would have to be done > gradually, starting with the smallest packages and going from there. > Ideally, those conversions would be merged into master so that we > wouldn't end up having to maintain two different versions of TR > indefinitely. > > That said, it seems like most of the risk in converting to Kotlin is > the interoperation with Java. Since basically all Java types are > nullable, you lose a lot of the benefit of Kotlin's non-null types if > you're always getting nullable types from the Java side of the > codebase. Additionally, it seems like IntelliJ has to convert most of > the Java types to nullable by default, and that doesn't exactly lead > to ideal Kotlin code. > > To sum up, here are some key takeaways from our experiment: > - using IntelliJ to convert Java to Kotlin is not as seamless as we'd hoped > - there are definite risks involved in the conversion > - converted code is not really ideal Kotlin code > > I'd say my current view is that I'm not totally opposed to the Java to > Kotlin conversion, but it's definitely more work than I'd personally > want to do. For a theoretical TR 2.0, I still think Kotlin would be a > good choice since it would allow us to easily reuse a lot of the > existing Java code and libraries we depend on, while still giving us > the benefits of modern language features like null safety. > > - Rawlin > > > On Wed, Jun 23, 2021 at 7:44 PM Eric Friedrich <fri...@apache.org> wrote: > > > > Rawlin, ocket8888- > > > > Thanks for pointing out some of the additional benefits Kotlin has over > > Java- I didn't know about the null check differences. > > > > > > Can you explain more about how the conversion process works? > > > > What are the risks associated with it (likelihood of introducing a bug, > > risks of having mixed Java+Kotlin, etc...). > > > > > > Also, how long could the community expect the transition to take and what > > are the mechanics (separate branch, etc...)? > > > > > > --Eric > > > > > > > > On Wed, Jun 23, 2021 at 2:37 PM ocket 8888 <ocket8...@gmail.com> wrote: > > > > > As someone who avoids looking at TR code as much as possible, I'd say > it's > > > certainly true that the codebase is a rat's nest, and switching to > Kotlin > > > won't automagically fix that. I do think, though, that it being Java > > > exacerbates the problem because of how verbose Java is. I, personally, > > > would not want to refactor TR's Java codebase; that just doesn't sound > like > > > fun to me. If it were Kotlin, though, I'd be much more willing to try > to > > > improve it. > > > > > > On Wed, Jun 23, 2021 at 10:15 AM Rawlin Peters <raw...@apache.org> > wrote: > > > > > > > > Rather, I think the TR code itself is badly in need of a clean up, > > > > to improve modularity and to give some thought to extensibility. > > > > > > > > I agree, but it's very difficult to make a business case for "improve > > > > modularity and extensibility" without a concrete use case behind it > > > > that can justify doing that refactoring beforehand. Whereas > converting > > > > from Java to Kotlin can be done fairly easily on a piecemeal basis > > > > over time by devs in their free time. > > > > > > > > > The solution to that is not easy, nor is it fast. It's a process of > > > > making > > > > small, incremental improvements to the cleanliness of TR with minimal > > > risk > > > > to stability. > > > > > > > > That actually sounds like what would happen during a conversion to > > > > Kotlin. In fact, that's basically what I did when I converted > > > > TrafficRouter.java to Kotlin yesterday: > > > > https://github.com/rawlinp/trafficcontrol/commits/kotlin-tr. The > > > > auto-conversion was pretty straightforward -- there were only 3 > errors > > > > I had to fix manually for it to compile. The rest was just fixing > most > > > > of the warnings/suggestions pointed out by IntelliJ, most of which > > > > helped clean things up a bit, making it more readable and less > > > > verbose. Combing over the results of the conversion like that is also > > > > a great way to find even more things that can be cleaned up, which in > > > > most cases would go ignored because no one is just combing over TR > > > > looking for things to clean up. > > > > > > > > > Somewhat less of a concern is shrinking the pool of developers by > > > > choosing > > > > a less mature, less widely adopted language. I understand Kotlin is > on > > > the > > > > rise and is in demand but is still dwarfed by the 20+ years of Java. > > > > > > > > I agree this is also less of a concern -- I think Java devs can learn > > > > Kotlin pretty quickly because Kotlin is basically just an enhanced > > > > version of Java. Also, the community did choose to replace the > > > > original Traffic Ops with a language 20+ years it's junior, and that > > > > was a tremendous amount of work compared to what converting Java to > > > > Kotlin would be. > > > > > > > > One of the main tenets of a CDN is reliability, and all of the > > > > features around null safety in Kotlin > > > > (https://kotlinlang.org/docs/null-safety.html) can seriously help > out > > > > there, especially if the entire codebase was Kotlin. While the > > > > codebase is a mix of the two, there is more chance of a > > > > NullPointerException coming from Java code because all Java > references > > > > are nullable, and Kotlin needs to maintain that interoperability. > Some > > > > of the nastier bugs in TR have stemmed from NullPointerExceptions due > > > > to not knowing that an argument was nullable, and with Kotlin's > > > > built-in nullable vs non-nullable types, that problem is way less > > > > likely to occur. > > > > > > > > If we were to develop TR 2.0 for instance (instead of just converting > > > > TR 1.0), I'd hope we could use a language like Kotlin which has > solved > > > > the Billion Dollar Mistake. > > > > > > > > - Rawlin > > > > > > > > On Tue, Jun 22, 2021 at 9:14 PM Eric Friedrich <fri...@apache.org> > > > wrote: > > > > > > > > > > Zach- > > > > > > > > > > First of all, thanks for trying to build a bigger tent here, I > really > > > > > appreciate that effort. > > > > > > > > > > (Personal comments, not as chair) > > > > > > > > > > I agree that Traffic Router is difficult to work on, and that > presents > > > a > > > > > large barrier to entry. However, I don't think that is a result of > > > > language > > > > > choice. Rather, I think the TR code itself is badly in need of a > clean > > > > up, > > > > > to improve modularity and to give some thought to extensibility. > > > > > > > > > > Recently one of my coworkers added a new feature to TR. He is > > > incredibly > > > > > experienced with Java, but had many problems trying to create a > clean > > > > > integration. > > > > > > > > > > Converting TR to Kotlin wouldn't improve or solve any of these > issues. > > > I > > > > > fear that even as we try to attract new contributors, they would > be > > > > > frustrated with the state of TR and not continue as ongoing > > > contributors. > > > > > > > > > > The solution to that is not easy, nor is it fast. It's a process of > > > > making > > > > > small, incremental improvements to the cleanliness of TR with > minimal > > > > risk > > > > > to stability. > > > > > > > > > > > > > > > Somewhat less of a concern is shrinking the pool of developers by > > > > choosing > > > > > a less mature, less widely adopted language. I understand Kotlin > is on > > > > the > > > > > rise and is in demand but is still dwarfed by the 20+ years of > Java. > > > > > > > > > > --Eric > > > > > > > > > > > > > > > > > > > > On Tue, Jun 22, 2021, 7:14 PM Chris Lemmons <alfic...@gmail.com> > > > wrote: > > > > > > > > > > > I don't have a strong opinion on Kotlin; I haven't used it to any > > > real > > > > > > degree. I do know it interoperates with the Java API and compiles > > > down > > > > > > to the JVM. So in terms of the openness of the platform, it's > not any > > > > > > more "open" than Oracle's OpenJDK. It does compile down to native > > > > > > code, but then we'd need to replace (or convert and maintain) > all our > > > > > > dependent libraries. > > > > > > > > > > > > So if Kotlin helps with maintainability, I say we go for it. I > don't > > > > > > know that we're winning any additional openness we don't already > > > have, > > > > > > though. > > > > > > > > > > > > On Tue, Jun 22, 2021 at 4:44 PM Taylor Frey < > taylor.f...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > I too +1 > > > > > > > > > > > > > > My experience porting a Java project to Kotlin was specific to > > > > Android in > > > > > > > the past (re: years ago). I recall my general impression of > Kotlin > > > to > > > > > > have > > > > > > > some extraneous and unnecessary syntactic sugar (What's up > with the > > > > > > > for/loop keywords?), but it was an overall solid language. I > > > > appreciate > > > > > > the > > > > > > > terseness and brevity of things like variable declarations w/ > > > > implicit > > > > > > data > > > > > > > types and `data classes`, which addresses a long standing > complaint > > > > about > > > > > > > Java's overly verbose syntax. Language aside, the automated > porting > > > > back > > > > > > > then wasn't exactly perfect, but a few things have changed > since > > > > then; 1) > > > > > > > the language specification has solidified over the years, 2) > > > tooling > > > > has > > > > > > > matured, and 3) is manageable to manually intervene if > necessary. > > > > > > > > > > > > > > > As a side note, because of Oracle's aggressive legal > practices, > > > and > > > > > > > because > > > > > > > they really do want you to think that they own Java (see: > > > Google > > > > LLC > > > > > > v. > > > > > > > Oracle America, Inc.), using Java in Apache Traffic > Control is > > > > not > > > > > > > really > > > > > > > in the spirit of free software. Kotlin is an Apache > > > 2.0-licensed > > > > > > > project, > > > > > > > so switching to Kotlin would be a true FOSS win for the ATC > > > > > > community. > > > > > > > > > > > > > > I think the true benefit is moving away from the Java API, > which is > > > > under > > > > > > > such contention, without changing underlying implementation > (such > > > as > > > > a > > > > > > > transition to another, non-JVM, language might incur). > > > > > > > > > > > > > > On Tue, Jun 22, 2021 at 11:42 AM Chatterjee, Srijeet > > > > > > > <srijeet_chatter...@comcast.com.invalid> wrote: > > > > > > > > > > > > > > > +1 > > > > > > > > Sounds like a fun project! > > > > > > > > > > > > > > > > --Srijeet > > > > > > > > > > > > > > > > On 6/21/21, 8:43 PM, "Zach Hoffman" <zrhoff...@apache.org> > > > wrote: > > > > > > > > > > > > > > > > Hi Traffic Control! > > > > > > > > > > > > > > > > As you all may know, the current implementation of > Traffic > > > > Router > > > > > > runs > > > > > > > > on > > > > > > > > Java 8. While coding directly in Java is still done, > there > > > are > > > > some > > > > > > > > reasons > > > > > > > > to be critical of Java. > > > > > > > > > > > > > > > > Java often has very verbose syntax when trying to > accomplish > > > > simple > > > > > > > > things. > > > > > > > > This has made the Traffic Router codebase grow larger > over > > > > time, > > > > > > > > leading to > > > > > > > > maintainability issues. Also, Java suffers from lack of > > > > > > compile-time > > > > > > > > safety > > > > > > > > in some ways that more modern languages do not. For these > > > > reasons, > > > > > > and > > > > > > > > because Java is a very old language, some developers who > may > > > > > > otherwise > > > > > > > > be > > > > > > > > interested in contributing to Apache Traffic Control may > be > > > > > > > > disinterested > > > > > > > > contributing to Traffic Router in particular, simply > because > > > > they > > > > > > want > > > > > > > > to > > > > > > > > work with more modern languages that do not have the > issues > > > > that > > > > > > Java > > > > > > > > does. > > > > > > > > > > > > > > > > Kotlin, a JVM language devised by Jetbrains with the > goal of > > > > > > tackling > > > > > > > > these > > > > > > > > problems, has exploded in popularity in recent years, and > > > > Kotlin is > > > > > > > > becoming a standard way to use the JVM, with Google > choosing > > > > it as > > > > > > the > > > > > > > > preferred language to for developing Android apps. More > > > > > > importantly, > > > > > > > > as the > > > > > > > > Intellij IDEA Kotlin plugin lets you convert Java > sources to > > > > Kotlin > > > > > > > > (either > > > > > > > > at the file level or an entire project, or anywhere in > > > > between), > > > > > > > > Kotlin has > > > > > > > > become a low-effort way to decrease tech debt across > large > > > Java > > > > > > > > projects. > > > > > > > > > > > > > > > > Back to Traffic Router: In order to gauge how we would > > > benefit > > > > from > > > > > > > > converting Traffic Router to Kotlin (it really is a > matter of > > > > > > > > *converting*, > > > > > > > > not rewriting), I converted all of the tests in the > Traffic > > > > Router > > > > > > > > "Core" > > > > > > > > module to Kotlin. You can see the result here (as you > can see > > > > from > > > > > > > > GitHub > > > > > > > > Actions, the TR tests all pass): < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/zrhoffman/trafficcontrol/commits/tr-kotlin__;!!CQl3mcHX2A!UCz457i9stX5A-AP8jvvWo21Rqa43JRsC1jQt_fo5nb07raZxByUR9gAGVPZTI4tGhalAel2$ > > > > > > > > > > > > > > > > > > > > > > > > > Some observations: > > > > > > > > - Without any putting any effort into optimizing the > existing > > > > code, > > > > > > > > lines > > > > > > > > of code shrunk from 10572 in Java to 10217 in Kotlin. > See: < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/zrhoffman/trafficcontrol/commit/5add06bccc__;!!CQl3mcHX2A!UCz457i9stX5A-AP8jvvWo21Rqa43JRsC1jQt_fo5nb07raZxByUR9gAGVPZTI4tGqGQTevX$ > > > > > > > > > > > > > > > > > - Because it now includes the Kotlin standard library, > the > > > > Traffic > > > > > > > > Router > > > > > > > > RPM size increases from 48MB to 58MB. > > > > > > > > - The converted sources benefit from Kotlin's null > safety and > > > > type > > > > > > > > safety > > > > > > > > (avoiding wildcard types). See < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/zrhoffman/trafficcontrol/commit/0c2eec39d9__;!!CQl3mcHX2A!UCz457i9stX5A-AP8jvvWo21Rqa43JRsC1jQt_fo5nb07raZxByUR9gAGVPZTI4tGjjMNj-F$ > > > > > > > > > (Kotlin > > > > > > > > error resolution) and < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/zrhoffman/trafficcontrol/commit/d8a0fd3bca__;!!CQl3mcHX2A!UCz457i9stX5A-AP8jvvWo21Rqa43JRsC1jQt_fo5nb07raZxByUR9gAGVPZTI4tGoysImMb$ > > > > > > > > > (Kotlin > > > > > > > > warning resolution) for the changes involved. > > > > > > > > > > > > > > > > As a side note, because of Oracle's aggressive legal > > > > practices, and > > > > > > > > because > > > > > > > > they really do want you to think that they own Java (see: > > > > Google > > > > > > LLC v. > > > > > > > > Oracle America, Inc.), using Java in Apache Traffic > Control > > > is > > > > not > > > > > > > > really > > > > > > > > in the spirit of free software. Kotlin is an Apache > > > > 2.0-licensed > > > > > > > > project, > > > > > > > > so switching to Kotlin would be a true FOSS win for the > ATC > > > > > > community. > > > > > > > > > > > > > > > > So, converting the rest of Traffic Router would be > relatively > > > > fast > > > > > > and > > > > > > > > easy, would result in *no* functionality gained or lost, > > > would > > > > > > leave TR > > > > > > > > usage totally unchanged, would make development more > > > enjoyable > > > > for > > > > > > > > current > > > > > > > > TR devs, and would maybe catch the interest of additional > > > devs > > > > who > > > > > > > > otherwise may not be interested in Traffic Router. > > > > > > > > > > > > > > > > How does this sound? > > > > > > > > > > > > > > > > -Zach > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
