I'd like to point out that tools.deps.alpha can build a differently-ordered classpath from the exact same input files when run on a different machine (i.e., the same source/deps, the same version of t.d.a./CLI).
This has bitten me a couple of times with depstar which I use for building JARs/uberjars (and which I maintain) because a JAR built on one machine can behave differently than a JAR built on another machine -- because of classpath ordering of things that overlap on the classpath. This is completely separate from the :paths issue (with which I have some sympathy). The most recent "gotcha" for me was with log4j2 which has a Log4J2Plugins.dat file containing a cache of the plugins that control formatting in log files, so any library that includes formatter plugins tends to have its own copy of this file. There have been a couple of long-standing bug reports against Log4J2 because of this -- because people in Maven-land keep tripping over classpath ordering issues and finding logging breaks "randomly" in their applications. Years of complaints and the official answer is "We'll stop doing the plugins cache thing in log4j 3". There is a Maven plugin that knows how to merge Log4J2Plugins.dat files but, of course, it is specific to log4j2 and Maven. There is a Leiningen plugin for it as well -- also specific to log4j2 code -- because Leiningen is built on the same "reproducible" base as Maven... So this _is_ an issue in Maven-land. It's an acknowledged problem with libraries that contain overlapping files and has been for years. That t.d.a/CLI makes this a bit more visible is a positive, in my opinion, because you're more likely to realize that you have a problem upstream with badly-behaved dependencies. p.s., My "hack" for depstar was to have it overwrite Log4J2Plugins.dat in the JAR until it found one bigger than ~10K and then ignore subsequent ones (the plugin cache file in the main log4j2 JAR is 20K, the plugin cache files in all the other libs I've seen are 5K or smaller). Sure, that means you don't get _all_ the formatter plugins, you only get the core ones, but in the real world that is almost always going to be the right choice. On Tue, Aug 11, 2020 at 9:34 PM 'bed...@yahoo.com' via Clojure < clojure@googlegroups.com> wrote: > Ok, let's reset as we start talking about different things now. > > None of these libraries are broken. They just include resources. Also, I > don't think it is realistic to tell library authors to please move certain > files out of the way because my build tool randomizes my classpath. That is > not going to happen. People will keep including things like > log4j.properties that are in potential conflict with a local file or files > in other JAR paths. > > The question remains: Should my build tool allow me control over the order > of the classpath? I believe the answer is yes. > (Even gradle gives you that control and/or offers ways to ensure that my > chosen paths and libraries are first on the classpath. > Regarding maven: Even if there would be no documentation at all, their > algorithm has not changed since 2.0.9. It's the Normative Kraft des > Faktischen at play. ) > > If the build tool from version A to version B changes this order, it is > not a usable tool. I can't make this point any clearer. > We might agree to disagree here, but a build tool should not be at liberty > to sort top-level dependencies at will. > > Note that this is completely independent from the thought that order > *shouldn't* matter. > When in fact it does matter. It is an unfortunate reality. That's simply > how resource loading works in Java. > > I understand that building dependency trees is not trivial, but it is > rather trivial to ensure :paths comes before everything else. > Please define some stable order and establish a top-level order for > explicit dependencies. That makes builds deterministic. > > Thanks for looking in this issue, > Jochen > > PS: happy to learn you got in touch with the homebrew team. That is > excellent news. > > > > > On Tuesday, August 11, 2020 at 6:01:18 PM UTC-7 Alex Miller wrote: > >> On Tue, Aug 11, 2020 at 6:41 PM 'bed...@yahoo.com' via Clojure < >> clo...@googlegroups.com> wrote: >> >>> Just 2 quick points before I go back to migrate to shadow-cljs & >>> leiningen ;) >>> >>> "just does not seem well defined " >>> This is not a line of argument you want to pursue when we are talking >>> about maven, who has had a stable order for what feels like decades now. >>> If you need a current link: >>> https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies >>> (search for "first") >>> >> >> I understand it may feel like that to you, but I have been using Maven >> for at least 15 years and have spent a lot of time fighting classpath >> ordering issues while using it. That link says nothing about classpath >> order and I spent some more time today looking for something like >> that, without success. There are plenty of Maven tickets about this, it's >> changed over the years, and a lot of complexity to cover when you start to >> consider parent poms, dependency management, etc. I don't think it's as >> fixed as you think it is, probably more that they just don't touch it much >> (and most differences don't actually matter). It may be stable largely by >> fear. >> >> >>> To your comment re Intellij: you can actually use Intellij to build and >>> it uses its own project configuration to do that and cobble together the >>> classpath (in many cases it merely syncs with maven/lein etc.). In a way, >>> Intellij acts like clj: It builds a classpath and runs stuff. >>> >> >> Yeah, I use it every day. My point was just that it's independent from >> what we're talking about. >> >> >>> Also, arguing that "JARs need to be fixed" is interesting, as I can get >>> *any* resource with `(io/resource...)` and will get different answers >>> depending on CLASSPATH order. >>> >> >> You can only get different answers if two roots have the same resource, >> which again, is bad, and the cause of a big percentage of dependency issues >> I've tracked down at one point or another. Same resource = shadowing = >> eventually you will reorder your deps and see something different. >> >> >>> If that order changes in between minor versions of clj, that makes it an >>> unstable build system. >>> >> >> I'd say if that's coming from dependencies, then it's your build deps >> that are unstable, you just didn't know it. The ordering of project paths >> before deps is a separate issue. >> >> >>> Just try to io-resource '/log4j.properties', for funsies. If in version >>> A of clj I'm getting my own /log4j.properties and in version I'm getting >>> the one from some other JAR, it's a problem. >>> >>> If I run >>> find . -name "*.jar" -exec unzip -l {} \; | awk '{print $4}' | sort | >>> less >>> in ~/.m2 >>> to crudely get all file names for all my cached jars, >>> I can find 4 instances of 'public/index.html' >>> 3 of 'public/css/style.css' etc. etc. >>> >> >> You are proving my point. That's all broken. Those libs cannot control >> where they are placed in an application's classpath and whether they will >> shadow or be shadowed by something else. They are time bombs waiting to >> break your build. You should complain to their creators. >> >> >>> Please return to at least having :paths at the front of the CLASSPATH. >>> >> >> As I said, that's a different issue and I'm inclined to agree with you. >> The details of changing it are more complicated than it appears. >> >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/clojure/81e969d2-91d2-4fc3-bdfd-8e509fc304bdn%40googlegroups.com > <https://groups.google.com/d/msgid/clojure/81e969d2-91d2-4fc3-bdfd-8e509fc304bdn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- https://corfield.org/ World Singles Networks, LLC. -- https://worldsinglesnetworks.com/ "Perfection is the enemy of the good." -- Gustave Flaubert, French realist novelist (1821-1880) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/CAD4thx_xgJWYu2yugv9GGuexkB%2B7Tw7uuBUz%2B3LeomyaL4guog%40mail.gmail.com.
