[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459969298 The following builds have also now passed with no HDFS failures: * Windows, Mac, Solaris, and two more linux builds - jenkins.thetaphi.de. * Linux master and Linux nightly master - https://builds.apache.org I am going to merge the change to 8.x and 8.0 to get some test runs over the weekend. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459880077 Build https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23610/ includes this master commit and was BUILD SUCCESSFUL - Total time: 75 minutes 44 seconds This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459850208 Planning to commit to master (again) and make sure the world doesn't fall apart (again). I'll keep and eye on Jenkins builds. If successful, I'll look at 8.x/8.0 this weekend. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459849996 Squashed commits back down to a single commit to make it easier to commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459801543 Tests have been running across a few machines I have access to for the last few hours. Probably ~10 runs about 50/50 JDK8/JDK11 mix. So far no HDFS test failures with latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459745591 Well naively adding shutdown of the thread pool results in a lot of `RejectedExecutionException`s since the pool is static. Shutting down is per the lifetime of one BlockPoolSlice. The lifetime of the Hadoop `addReplicaThreadPool` needs some more thought. I don't know if it is completely possible to fix this only in BlockPoolSlice. Planning to leave it for now as a bad thread. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459732343 Rebased on master and updated the workaround for BlockPoolSlice. Think that also found the issue with threads not being cleaned up. Running through all tests now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459445817 @uschindler Want to make sure you don't think I am trying to avoid fixing the forbiddenapis/source check stuff. I plan to fix it but since it is test only and want to make sure this cleanup ends up in 8.0 if possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459445398 Pushed to master. Will wait a few hours and then push to other branches. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459441190 > I'll try with Netbeans next. I tried with Netbeans 8.x and tests ran successfully. Netbeans 10.x hung on my machine and didn't track that down. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459435527 Squashed the multiple commits into one and added CHANGES.txt entry. Planning to push this to master and 8.x in a little bit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459352035 @uschindler thanks - any concerns with merging this to the master branch before you have a chance to check? I'd be interested in getting this change into Solr 8.0 if possible. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-459103928 So far I've loaded the project in IntelliJ (`ant intellij`) and Eclipse (`ant eclipse`). In both cases the patched `HttpServer2` class was used to run the tests. I'll try with Netbeans next. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-458978472 > To me it's only important to clarify: If we want to go that route, it's important to have our own test classes BEFORE the hadoop imports on classpath. We have to make sure that this also applies for the Eclipse/NetBeans/... Development integrations! @uschindler any suggestions on how to accomplish this? I understand what you mean but not sure how to go about it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-458978258 @uschindler yea so I tried fixing a lot of the failures. Some up the issues I ran into: * Changing LOG -> log to match the source patterns check broke other Hadoop classes. It turns out Hadoop uses the LOG since it is public outside the HttpServer2 class. I couldn't find a way to suppress just the source check for that file. * Forbiddenapis (even with SuppressForbidden) kept saying that https://github.com/apache/hadoop/blob/branch-3.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java#L1500 was invalid due to https://github.com/apache/lucene-solr/blob/master/lucene/tools/forbiddenApis/servlet-api.txt#L18. I tried to add SuppressForbidden at the class, static class, method, and override methods to fix the issue, but the errors still came back. So with the above two issues that is how I ended up suppressing the checks on the HttpServer2 class. The other thought I had is that it would be nice to not edit HttpServer2 too much in case we need to copy changes back from Apache Hadoop again. I am open to suggestions on how to move forward without suppressing the checks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-458639506 ping @hgadre since you filled SOLR-9761 - I know I didn't get to all the subtasks under SOLR-9761 but would appreciate any thoughts you might have. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-458633207 @uschindler sorry I definitely didn't explain it as well as I could have. Hadoop moved from old mortba Jetty to Jetty 9.3. Solr is on Jetty 9.4. Session management changed between Jetty 9.3 and 9.4. For the integration tests with Hadoop, the classpath has Jetty 9.4. Adding Jetty 9.3 isn't possible since then the new HTTP2 tests don't work. The smallest change I could make was to copy HttpServer2 from Hadoop and fix the session management sections for Jetty 9.4. The changes are similar to https://issues.apache.org/jira/browse/HADOOP-14930 which looked at upgrading Hadoop to Jetty 9.4. So yes your understanding is correct in that we can't have both Jetty 9.3 and 9.4 on the Solr test classpath. The testing I have done shows that the Hadoop integration tests work with Jetty 9.4 with the copied/patched HttpServer2 code. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3
risdenk commented on issue #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#issuecomment-458565430 Would appreciate a review - @markrmiller, @uschindler, @sigram, @ctargett and whoever else is interested. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
