I had a number of problems getting either Jekyll build to run at all - it kept getting stuck in various places.
I eventually traced the problem to the pygments.rb plugin - I have python, so it didn’t complain about it missing, just kept throwing errors about missing header in a file. But when I took it out everything started working. So I swapped it out for Rouge which is Ruby-based and comes with Jekyll. However, it wasn't really well supported until Asciidoctor 2.x, so we either lose syntax highlighting entirely or upgrade Asciidoctor & jekyll-asciidoc version to get the better support. If we go with the older version of Asciidoctor we also have to downgrade Slim to 3.0.1 since I found a bug a year ago using the 4.0.1 version of Slim with Jekyll-asciidoc, which has since been fixed in the later versions. The whole reason we hadn’t upgraded Asciidoctor was because people needed to install it locally to get our old build to run, and I wasn’t sure how to force folks to update their local version. However, with Gradle we can force the version we want because the build is handling the dependencies. So, the choice is - downgrade the Asciidoctor version and lose syntax highlighting, or I could upgrade Asciidoctor for the project now (in the Ant build & Jenkins jobs) and fix the pages that will fail the validation check. As soon as those pages are fixed, the Gradle build will work fine. Cassandra On Oct 11, 2019, 1:28 PM -0500, Dawid Weiss <dawid.we...@gmail.com>, wrote: > > Sure. Try and see if you can make it work. It is just about the only thing > that still needs to be done, the rest works like a charm. > > > On Fri, Oct 11, 2019, 20:20 Cassandra Targett <casstarg...@gmail.com> wrote: > > > Sorry for the delay getting back to you Dawid. That problem is actually > > > because of the Asciidoctor version. The jekyll-asciidoc plugin will > > > install Asciidoctor if it is not already installed, and it installs a > > > version where the way the links are constructed is different and breaks > > > our validation. > > > > > > More details about why this happens (if you’re curious) is in > > > https://issues.apache.org/jira/browse/SOLR-12786?focusedCommentId=16622115&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16622115 > > > > > > Both the Jenkins script used for Jenkins Ref Guide jobs > > > (./dev-tools/scripts/jenkins.build.ref.guide.sh) and the Ref Guide README > > > (./solr/solr-ref-guide/README.adoc) show examples of how to make sure the > > > right Asciidoctor version is installed - the easiest is to install the > > > Asciidoctor gem version we want first. Let me see if I can insert a line > > > to install before the jekyll-asciidoc gem and see if that fixes it. > > > > > > Cassandra > > > On Oct 10, 2019, 2:22 AM -0500, Dawid Weiss <dawid.we...@gmail.com>, > > > wrote: > > > > Ping, ping. I wanted to finalize the build of solr ref guide since I > > > > started it. Almost everything is working on the branch but I can't get > > > > minor differences to work and I believe they're due to a different > > > > jekyll version (than that mentioned in the docs). > > > > > > > > Specifically, the invalid links are because of asciidoc sections like > > > > this (in the processed resource-and-plugin-loading.adoc): > > > > > > > > === solr_home/lib > > > > > > > > In bare bones html (pure asciidoctor) this gets emitted as: > > > > > > > > <h3 id="solr_home-lib">solr_home/lib</h3> > > > > > > > > but when compiled via jekyll this becomes: > > > > > > > > <h3 id="solr_homelib">solr_home/lib</h3> > > > > > > > > which I can't really explain. > > > > > > > > Cassandra what's the exact version of jekyll that runs the compilation > > > > that is working for you? > > > > > > > > D. > > > > > > > > On Fri, Oct 4, 2019 at 11:42 AM Dawid Weiss <dawid.we...@gmail.com> > > > > wrote: > > > > > > > > > > Seems like pygments is to blame for the python requirement... I didn't > > > > > check but there seem to be ruby-only > > > > > highlighters for jekyll as well: > > > > > > > > > > https://jekyll-windows.juthilo.com/3-syntax-highlighting/ > > > > > > > > > > On Fri, Oct 4, 2019 at 11:39 AM Dawid Weiss <dawid.we...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Hi Cassandra, > > > > > > > > > > > > Apologies this took so long -- I wasn't familiar with these > > > > > > site-generation tools and the whole ecosystem is rather... fragile > > > > > > :) > > > > > > After a few attempts at using gradle plugins I eventually leaned > > > > > > towards using asciidoctor and jekyll explicitly (so that we know > > > > > > which > > > > > > versions are being used and don't have to rely on dependencies). > > > > > > > > > > > > I got bare bone html checking working, PDF generation working and > > > > > > site > > > > > > generation working although the final link check currently fail for > > > > > > me > > > > > > with a bunch of errors. This works for me on Windows... on Linux I > > > > > > get > > > > > > site-generation generate a strange error from within jekyll: > > > > > > > > > > > > Conversion error: Jekyll::AsciiDoc::Converter encountered an error > > > > > > while converting 'about-filters.adoc': > > > > > > Bad file descriptor - /usr/bin/python2 > > > > > > > > > > > > I could install python but I don't see why it'd need it. Perhaps > > > > > > there > > > > > > is something in the docs that would avoid using python altogether > > > > > > but > > > > > > I haven't had the time to look into it. > > > > > > > > > > > > Please feel free to check out the jira/SOLR-13452_gradle_7_refguide > > > > > > branch and try to run: > > > > > > > > > > > > ./gradlew -p solr/solr-ref-guide buildPdf buildSite > > > > > > > > > > > > There is a lot of room for improvement -- from property > > > > > > substitution, > > > > > > through how the "tools" are handled at the moment to task naming > > > > > > but I > > > > > > left this for the future. The initial step would be probably to get > > > > > > the site generation running on Linux/ Macs but I'd gladly hand it > > > > > > over > > > > > > back to you -- I can help with Gradle but a the rest of those tools > > > > > > are a mistery to me. > > > > > > > > > > > > Dawid > > > > > > > > > > > > On Fri, Sep 27, 2019 at 7:53 PM Dawid Weiss <dawid.we...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > No problem. I will get it to work entirely, but not before next > > > > > > > week - I am away for the weekend. > > > > > > > > > > > > > > Dawid > > > > > > > > > > > > > > On Fri, Sep 27, 2019, 16:17 Cassandra Targett > > > > > > > <casstarg...@gmail.com> wrote: > > > > > > > > > > > > > > > > Thanks Dawid for working on this! I’ve been a bit swamped the > > > > > > > > last couple of days but will take a look today at what you’ve > > > > > > > > been able to do so far and see where we might need to go from > > > > > > > > here. > > > > > > > > > > > > > > > > Cassandra > > > > > > > > On Sep 26, 2019, 7:25 AM -0500, Dawid Weiss > > > > > > > > <dawid.we...@gmail.com>, wrote: > > > > > > > > > > > > > > > > I agree. Although I also understand the concern of trying to > > > > > > > > merge the > > > > > > > > changes while we're in the transition period... it'd be hell. > > > > > > > > I'd say > > > > > > > > move as much stuff as possible with the current folder > > > > > > > > structure (and > > > > > > > > ignore what cannot be ported easily) then switch as soon as > > > > > > > > possible > > > > > > > > to gradle and hack the old cruft with a chainsaw... > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Thu, Sep 26, 2019 at 2:13 PM Erick Erickson > > > > > > > > <erickerick...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Of course I’ll completely defer to Dawid and Mark (well and > > > > > > > > anybody else actually, you know, doing _work_), but just can’t > > > > > > > > resist chiming in ;). > > > > > > > > > > > > > > > > My vote would be to “do it the Gradle way”. Yes, it’s a PITA to > > > > > > > > learn new stuff and I won’t like it. Tough. I see no reason to > > > > > > > > carry a bunch of cruft around because “that the way we always > > > > > > > > did it”. > > > > > > > > > > > > > > > > If we lose functionality, that’s a different discussion, > > > > > > > > starting with “do we need that functionality". But jumping > > > > > > > > through hoops and having to maintain that awkwardness forever > > > > > > > > going forward just because we forced the Ant structure on > > > > > > > > Gradle strikes me as a poor trade off. > > > > > > > > > > > > > > > > That said, I’m not doing the work so I really have no vote. But > > > > > > > > don’t strain to do it the old way on my account ;) > > > > > > > > > > > > > > > > Erick > > > > > > > > > > > > > > > > P.S. Thanks Dawid for jumping in! > > > > > > > > > > > > > > > > On Sep 26, 2019, at 3:57 AM, Dawid Weiss > > > > > > > > <dawid.we...@gmail.com> wrote: > > > > > > > > > > > > > > > > I pushed it in to Lucene repo (it's on Cassandra's refguide > > > > > > > > branch > > > > > > > > anyway, so shouldn't interfere with anything else); seems like > > > > > > > > it's in > > > > > > > > better shape than previous code anyway (those questions I asked > > > > > > > > about > > > > > > > > the nature of the gradle port still hold though). > > > > > > > > > > > > > > > > I got as far as building initial bare-bones HTML. > > > > > > > > > > > > > > > > .\gradlew -p solr\solr-ref-guide clean bareBonesHtmlValidation > > > > > > > > > > > > > > > > I don't know anything about the pipeline involved (asciidoctor, > > > > > > > > etc.) > > > > > > > > so it's very likely some attributes will have to be corrected > > > > > > > > later > > > > > > > > on. > > > > > > > > > > > > > > > > Dawid > > > > > > > > > > > > > > > > On Wed, Sep 25, 2019 at 9:14 PM Dawid Weiss > > > > > > > > <dawid.we...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > I looked at the solr ref guide build and started converting it > > > > > > > > to > > > > > > > > Gradle but have a question to Mark (because he coordinates the > > > > > > > > effort). > > > > > > > > > > > > > > > > What immediately jumps into face is the decision problem -- do > > > > > > > > we want > > > > > > > > to emulate what ant does at the moment or do we want to clean > > > > > > > > it up > > > > > > > > (breaking file/ folder structure and causing incompatibility > > > > > > > > with ant > > > > > > > > build). > > > > > > > > > > > > > > > > I went the "compatible" way and started porting ant tasks but > > > > > > > > it's > > > > > > > > quite awkward. For example -- there are template properties > > > > > > > > that refer > > > > > > > > to ivy version properties... we could emulate/ compute these > > > > > > > > but it's > > > > > > > > a pain. The way the module is currently structured is also > > > > > > > > awkward - > > > > > > > > it'd be more natural to have a separate java project with the > > > > > > > > "tools" > > > > > > > > required to compile extra stuff and just reference it from the > > > > > > > > manual > > > > > > > > build (and this would be a plain module, not a java module). > > > > > > > > This > > > > > > > > would limit the need for customizing source sets, classpaths, > > > > > > > > etc. > > > > > > > > > > > > > > > > My few initial tasks syncing sources, setting up infrastructure > > > > > > > > to > > > > > > > > filter templates and compiling the required tools are here: > > > > > > > > https://github.com/apache/lucene-solr/compare/jira/SOLR-13452_gradle_7_refguide...dweiss:jira/SOLR-13452_gradle_7_refguide?expand=1 > > > > > > > > > > > > > > > > I'll stop and wait for feedback (especially on the ivy versions > > > > > > > > issue) > > > > > > > > before I resume. > > > > > > > > > > > > > > > > Dawid > > > > > > > > > > > > > > > > On Wed, Sep 25, 2019 at 6:20 PM Dawid Weiss > > > > > > > > <dawid.we...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Never mind, I've got it. > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Wed, Sep 25, 2019 at 7:59 AM Dawid Weiss > > > > > > > > <dawid.we...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Cassandra, > > > > > > > > > > > > > > > > I’m more than happy to share more details our current build so > > > > > > > > we can replicate some of the above steps, but I’m stuck without > > > > > > > > a lot more basic Gradle skills that I don’t have time to > > > > > > > > acquire with day-job/personal life commitments. I put it into a > > > > > > > > separate branch so we could iterate a little easier, can anyone > > > > > > > > help? > > > > > > > > > > > > > > > > > > > > > > > > Where is this branch you made changes on? If you can point me > > > > > > > > at the > > > > > > > > corresponding ant code I'll try to help you out. > > > > > > > > > > > > > > > > Dawid > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > >
