Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Here you go: https://github.com/apache/incubator-netbeans/pull/537 -Tim On Fri, Apr 27, 2018 at 2:56 AM, Jaroslav Tulach wrote: > Tim, > its 21th century and we are using GitHub. If you want to contribute, please > do it via PR. > Thanks. > -jt > > > 2018-04-25 13:31 GMT+02:00 Tim Boudreau : > > > To follow up on this a bit, since I spent a bit of time attempting to > > optimize this - the two big performance wins are: > > > > 1. Cache a byte[] and reuse it for every JAR entry (pass in a lambda to > > read() rather than get out a Map) > > 2. Maven's DefaultScanner will pass the indexer *every single file* in > the > > repository it's indexing, while NetBeans is likely uninterested in .sha1 > > files and similar; filtering the list of files to things NetBeans is > > likely to be interested offers large benefits > > > > These two optimizations cut scanning time of my ~/.m2/repository dir > (~2700 > > JAR files) from 42304ms to 30100ms, which is a 29% performance > improvement > > with > > > > That said, I have some tests to get passing before this is patch-worthy, > so > > we'll see if those results hold up. > > > > Of course, this only helps local indexing - whatever "Unpacking indexes" > is > > doing with remote repositories won't be helped by this. > > > > Still, it seems like something that could be optimized quite a bit before > > giving up on it. If anyone's interested in poking at this: > > https://timboudreau.com/files/maven-indexer.diff > > > > -Tim > > > -- http://timboudreau.com
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
On Fri, Apr 27, 2018 at 1:56 AM Jaroslav Tulach wrote: > Tim, > its 21th century and we are using GitHub. If you want to contribute, please > do it via PR. > Thanks. As I said, when I have the tests passing I'll do that. -Tim > -jt > > > 2018-04-25 13:31 GMT+02:00 Tim Boudreau : > > > To follow up on this a bit, since I spent a bit of time attempting to > > optimize this - the two big performance wins are: > > > > 1. Cache a byte[] and reuse it for every JAR entry (pass in a lambda to > > read() rather than get out a Map) > > 2. Maven's DefaultScanner will pass the indexer *every single file* in > the > > repository it's indexing, while NetBeans is likely uninterested in .sha1 > > files and similar; filtering the list of files to things NetBeans is > > likely to be interested offers large benefits > > > > These two optimizations cut scanning time of my ~/.m2/repository dir > (~2700 > > JAR files) from 42304ms to 30100ms, which is a 29% performance > improvement > > with > > > > That said, I have some tests to get passing before this is patch-worthy, > so > > we'll see if those results hold up. > > > > Of course, this only helps local indexing - whatever "Unpacking indexes" > is > > doing with remote repositories won't be helped by this. > > > > Still, it seems like something that could be optimized quite a bit before > > giving up on it. If anyone's interested in poking at this: > > https://timboudreau.com/files/maven-indexer.diff > > > > -Tim > > > -- http://timboudreau.com
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Tim, its 21th century and we are using GitHub. If you want to contribute, please do it via PR. Thanks. -jt 2018-04-25 13:31 GMT+02:00 Tim Boudreau : > To follow up on this a bit, since I spent a bit of time attempting to > optimize this - the two big performance wins are: > > 1. Cache a byte[] and reuse it for every JAR entry (pass in a lambda to > read() rather than get out a Map) > 2. Maven's DefaultScanner will pass the indexer *every single file* in the > repository it's indexing, while NetBeans is likely uninterested in .sha1 > files and similar; filtering the list of files to things NetBeans is > likely to be interested offers large benefits > > These two optimizations cut scanning time of my ~/.m2/repository dir (~2700 > JAR files) from 42304ms to 30100ms, which is a 29% performance improvement > with > > That said, I have some tests to get passing before this is patch-worthy, so > we'll see if those results hold up. > > Of course, this only helps local indexing - whatever "Unpacking indexes" is > doing with remote repositories won't be helped by this. > > Still, it seems like something that could be optimized quite a bit before > giving up on it. If anyone's interested in poking at this: > https://timboudreau.com/files/maven-indexer.diff > > -Tim >
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
To follow up on this a bit, since I spent a bit of time attempting to optimize this - the two big performance wins are: 1. Cache a byte[] and reuse it for every JAR entry (pass in a lambda to read() rather than get out a Map) 2. Maven's DefaultScanner will pass the indexer *every single file* in the repository it's indexing, while NetBeans is likely uninterested in .sha1 files and similar; filtering the list of files to things NetBeans is likely to be interested offers large benefits These two optimizations cut scanning time of my ~/.m2/repository dir (~2700 JAR files) from 42304ms to 30100ms, which is a 29% performance improvement with That said, I have some tests to get passing before this is patch-worthy, so we'll see if those results hold up. Of course, this only helps local indexing - whatever "Unpacking indexes" is doing with remote repositories won't be helped by this. Still, it seems like something that could be optimized quite a bit before giving up on it. If anyone's interested in poking at this: https://timboudreau.com/files/maven-indexer.diff -Tim
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Just taking five minutes to glance at ClassDependencyIndexCreator, I see a bunch of possible optimizations: 1. For each JAR file, it creates a Map of className:classBytes, and then iterating it and creating a ClassFile over each one. Garbage-collection wise, it would be considerably cheaper to do the work as the JAR is iterated. 2. A bunch of uses of TreeMap and TreeSet, where it might be cheaper to use HashSet and sort into a list when done - guessing this is done to sort Foo$Bar below Foo? 3. addDependenciesToMap() appears to actually call ClassLoader.getSystemClassLoader().getParent().loadClass() with a bunch of classes found in a random JAR it is scanning. I have trouble imagining why that would ever be a good idea, or is necessary here, and it definitely could seriously impact performance. Unfortunately with a Git checkout, I don't have the history to figure out why. It looks like it is to determine if the class is part of the JDK perhaps? If so, a much faster test would be to look at the package name - yes, theoretically someone can create their own java.lang package, but that's a corner case, and no matter what, actually loading into the JVM every class seen when indexing is costly both in loading things that will never be used by the IDE, and an incredible number of ClassNotFoundExceptions constructed, probably fairly deep in the stack. -Tim On Mon, Apr 23, 2018 at 3:32 AM, Jaroslav Tulach wrote: > 2018-04-23 8:40 GMT+02:00 Laszlo Kishalmi : > > > Dear all, > > > > I just experimented some stuff for that yesterday using bintray API for > > their repositories. I was able to retrieve a list of packages by > > group/artifact ID-s and even query artifacts by package/class names, > though > > the later one might need to use the "I feel lucky!" heuristics, as I > > searched for the new JUnit5 package and that gave me 14000+ matches and > > returned the first 50. > > > Excellent. If you put this query in and change the default for the > download/update of local indexer to "never", I'd be happy. > -jt > > PS: Maybe there will be some bugs, but the current indexer performance is > "buggy" too and a lot! > > > > On 04/22/2018 11:28 PM, Christian Lenz wrote: > > > >> Hey Jaroslav, > >> > >> I think now it is the best time to do the Approach again under Apache 😊 > >> > > > PPS: Probably. It has just one issue: There needs to be somebody to do it. > -- http://timboudreau.com
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
2018-04-23 8:40 GMT+02:00 Laszlo Kishalmi : > Dear all, > > I just experimented some stuff for that yesterday using bintray API for > their repositories. I was able to retrieve a list of packages by > group/artifact ID-s and even query artifacts by package/class names, though > the later one might need to use the "I feel lucky!" heuristics, as I > searched for the new JUnit5 package and that gave me 14000+ matches and > returned the first 50. Excellent. If you put this query in and change the default for the download/update of local indexer to "never", I'd be happy. -jt PS: Maybe there will be some bugs, but the current indexer performance is "buggy" too and a lot! > On 04/22/2018 11:28 PM, Christian Lenz wrote: > >> Hey Jaroslav, >> >> I think now it is the best time to do the Approach again under Apache 😊 >> > PPS: Probably. It has just one issue: There needs to be somebody to do it.
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Yeah, I can confirm setting up debugging (for Maven) in IntelliJ is so complicated... The one thing I think they have right is having multiple configurations. I'd love the ability to save a configuration, either with a name, or hell even as simple as the host, and port number fields remembering previously used entries - Working on a system with multiple Spring boot services means each one of them has different debug ports, and remembering them is getting tougher the more we add... (off to search JIRA for an existing ticket, or to create a new one before I start work ;) ) Regards John On 23 April 2018 at 07:40, Laszlo Kishalmi wrote: > Dear all, > > I just experimented some stuff for that yesterday using bintray API for > their repositories. I was able to retrieve a list of packages by > group/artifact ID-s and even query artifacts by package/class names, though > the later one might need to use the "I feel lucky!" heuristics, as I > searched for the new JUnit5 package and that gave me 14000+ matches and > returned the first 50. > > > > On 04/22/2018 11:28 PM, Christian Lenz wrote: > >> Hey Jaroslav, >> >> I think now it is the best time to do the Approach again under Apache 😊 >> >> >> Cheers >> >> Chris >> >> Von: Jaroslav Tulach >> Gesendet: Montag, 23. April 2018 08:14 >> An: Apache NetBeans >> Betreff: The IDE for DevOps was: IntelliJ IDEA vs Netbeans >> >> I just spent the past 2 weeks using IntelliJ IDEA exclusively (having used >>> it sporatically before). I'm going to share some brief thoughts in the >>> hopes that it helps. >>> >>> As far as I can tell, IntelliJ's killer feature is their debugger (more >>> broadly, their UI). Our killer feature is our profiler, and Maven >>> integration (more broadly, bundling more functionality standard). >>> >>> * Netbeans drives development of Maven projects through Maven. This >>> results in better integration than IntelliJ provides (e.g. good luck >>> trying to start a debugging session through Maven) >>> >>> Yeah, I can confirm setting up debugging (for Maven) in IntelliJ is so >> complicated... >> >> Once I called NetBeans the [IDE for devops and admins]( >> http://wiki.apidesign.org/wiki/DevOps) and this is what I meant. If you >> care about your overall project structure, there shall be benefits of >> using >> NetBeans+Maven. If you just care about the code, the IntelliJ's editor >> focus may give you better experience. >> >> Moreover the NetBeans approach is more fragile. Structures of pom.xml >> files >> differ wildly and when they get out of expectations, things may get broken >> or slow... >> >> indexing and performance levels can be done with the >>> code currently in Apache NetBeans Git. Jaroslav Tulach will have insights >>> as well as gratitude for help in this area >>> >> My thought is simple: there should be no Maven index processing on the >> client (by default). There should be a webservice the IDE would query >> instead. However my idea was rejected by last Oracle NetBeans performance >> team last time I proposed it. It was found too complicated. Anybody wants >> to pick that challenge up now? >> >> -jt >> >> >> > > - > To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org > For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org > > For further information about the NetBeans mailing lists, visit: > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > >
Re: AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Dear all, I just experimented some stuff for that yesterday using bintray API for their repositories. I was able to retrieve a list of packages by group/artifact ID-s and even query artifacts by package/class names, though the later one might need to use the "I feel lucky!" heuristics, as I searched for the new JUnit5 package and that gave me 14000+ matches and returned the first 50. On 04/22/2018 11:28 PM, Christian Lenz wrote: Hey Jaroslav, I think now it is the best time to do the Approach again under Apache 😊 Cheers Chris Von: Jaroslav Tulach Gesendet: Montag, 23. April 2018 08:14 An: Apache NetBeans Betreff: The IDE for DevOps was: IntelliJ IDEA vs Netbeans I just spent the past 2 weeks using IntelliJ IDEA exclusively (having used it sporatically before). I'm going to share some brief thoughts in the hopes that it helps. As far as I can tell, IntelliJ's killer feature is their debugger (more broadly, their UI). Our killer feature is our profiler, and Maven integration (more broadly, bundling more functionality standard). * Netbeans drives development of Maven projects through Maven. This results in better integration than IntelliJ provides (e.g. good luck trying to start a debugging session through Maven) Yeah, I can confirm setting up debugging (for Maven) in IntelliJ is so complicated... Once I called NetBeans the [IDE for devops and admins]( http://wiki.apidesign.org/wiki/DevOps) and this is what I meant. If you care about your overall project structure, there shall be benefits of using NetBeans+Maven. If you just care about the code, the IntelliJ's editor focus may give you better experience. Moreover the NetBeans approach is more fragile. Structures of pom.xml files differ wildly and when they get out of expectations, things may get broken or slow... indexing and performance levels can be done with the code currently in Apache NetBeans Git. Jaroslav Tulach will have insights as well as gratitude for help in this area My thought is simple: there should be no Maven index processing on the client (by default). There should be a webservice the IDE would query instead. However my idea was rejected by last Oracle NetBeans performance team last time I proposed it. It was found too complicated. Anybody wants to pick that challenge up now? -jt - To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
AW: The IDE for DevOps was: IntelliJ IDEA vs Netbeans
Hey Jaroslav, I think now it is the best time to do the Approach again under Apache 😊 Cheers Chris Von: Jaroslav Tulach Gesendet: Montag, 23. April 2018 08:14 An: Apache NetBeans Betreff: The IDE for DevOps was: IntelliJ IDEA vs Netbeans > I just spent the past 2 weeks using IntelliJ IDEA exclusively (having used > it sporatically before). I'm going to share some brief thoughts in the > hopes that it helps. > > As far as I can tell, IntelliJ's killer feature is their debugger (more > broadly, their UI). Our killer feature is our profiler, and Maven > integration (more broadly, bundling more functionality standard). > > * Netbeans drives development of Maven projects through Maven. This >results in better integration than IntelliJ provides (e.g. good luck >trying to start a debugging session through Maven) > Yeah, I can confirm setting up debugging (for Maven) in IntelliJ is so complicated... Once I called NetBeans the [IDE for devops and admins]( http://wiki.apidesign.org/wiki/DevOps) and this is what I meant. If you care about your overall project structure, there shall be benefits of using NetBeans+Maven. If you just care about the code, the IntelliJ's editor focus may give you better experience. Moreover the NetBeans approach is more fragile. Structures of pom.xml files differ wildly and when they get out of expectations, things may get broken or slow... > indexing and performance levels can be done with the > code currently in Apache NetBeans Git. Jaroslav Tulach will have insights > as well as gratitude for help in this area My thought is simple: there should be no Maven index processing on the client (by default). There should be a webservice the IDE would query instead. However my idea was rejected by last Oracle NetBeans performance team last time I proposed it. It was found too complicated. Anybody wants to pick that challenge up now? -jt
