[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253097101 ## File path: lucene/tools/src/groovy/check-source-patterns.groovy ## @@ -149,6 +149,7 @@ ant.fileScanner{ exclude(name: 'lucene/benchmark/temp/**') exclude(name: '**/CheckLoggingConfiguration.java') exclude(name: 'lucene/tools/src/groovy/check-source-patterns.groovy') // ourselves :-) +exclude(name: 'solr/core/src/test/org/apache/hadoop/**') Review comment: Also now catches the BlockPoolSlice code This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253096881 ## File path: solr/core/build.xml ## @@ -25,6 +25,7 @@
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253096125 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -239,4 +253,24 @@ public static String getURI(MiniDFSCluster dfsCluster) { } } + /** + * By default in JDK9+, the ForkJoinWorkerThreadFactory does not give SecurityManager permissions + * to threads that are created. This works around that with a custom thread factory. + * See SOLR-9515 and HDFS-14251 + * Used in org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice + */ + public static class HDFSForkJoinThreadFactory implements ForkJoinPool.ForkJoinWorkerThreadFactory { +@Override +public ForkJoinWorkerThread newThread(ForkJoinPool pool) { + ForkJoinWorkerThread worker = new SecurityManagerWorkerThread(pool); + worker.setName("solr-hdfs-threadpool-" + worker.getPoolIndex()); Review comment: Name used in BadHdfsThreadFilter since this pool is not shutdown correctly by BlockPoolSlice and not easily worked around. See SOLR-9515 and HDFS-14251 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253095650 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -62,73 +63,88 @@ public static MiniDFSCluster setupClass(String dir) throws Exception { return setupClass(dir, true, true); } - + public static MiniDFSCluster setupClass(String dir, boolean haTesting) throws Exception { return setupClass(dir, haTesting, true); } - + + /** + * Checks that commons-lang3 FastDateFormat works with configured locale + */ + @SuppressForbidden(reason="Call FastDateFormat.format same way Hadoop calls it") + private static void checkFastDateFormat() { +try { + FastDateFormat.getInstance().format(System.currentTimeMillis()); +} catch (ArrayIndexOutOfBoundsException e) { + LuceneTestCase.assumeNoException("commons-lang3 FastDateFormat doesn't work with " + + Locale.getDefault().toLanguageTag(), e); +} + } + + /** + * Hadoop fails to generate locale agnostic ids - Checks that generated string matches + */ + private static void checkGeneratedIdMatches() { +// This is basically how Namenode generates fsimage ids and checks that the fsimage filename matches +LuceneTestCase.assumeTrue("Check that generated id matches regex", +Pattern.matches("(\\d+)", String.format(Locale.getDefault(),"%019d", 0))); + } + public static MiniDFSCluster setupClass(String dir, boolean safeModeTesting, boolean haTesting) throws Exception { LuceneTestCase.assumeFalse("HDFS tests were disabled by -Dtests.disableHdfs", - Boolean.parseBoolean(System.getProperty("tests.disableHdfs", "false"))); + Boolean.parseBoolean(System.getProperty("tests.disableHdfs", "false"))); + +checkFastDateFormat(); +checkGeneratedIdMatches(); -savedLocale = Locale.getDefault(); -// TODO: we HACK around HADOOP-9643 -Locale.setDefault(Locale.ENGLISH); - if (!HA_TESTING_ENABLED) haTesting = false; - - -// keep netty from using secure random on startup: SOLR-10098 Review comment: Not needed after netty upgrade. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253095505 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -62,73 +63,88 @@ public static MiniDFSCluster setupClass(String dir) throws Exception { return setupClass(dir, true, true); } - + public static MiniDFSCluster setupClass(String dir, boolean haTesting) throws Exception { return setupClass(dir, haTesting, true); } - + + /** + * Checks that commons-lang3 FastDateFormat works with configured locale + */ + @SuppressForbidden(reason="Call FastDateFormat.format same way Hadoop calls it") + private static void checkFastDateFormat() { +try { + FastDateFormat.getInstance().format(System.currentTimeMillis()); +} catch (ArrayIndexOutOfBoundsException e) { + LuceneTestCase.assumeNoException("commons-lang3 FastDateFormat doesn't work with " + + Locale.getDefault().toLanguageTag(), e); +} + } + + /** + * Hadoop fails to generate locale agnostic ids - Checks that generated string matches + */ + private static void checkGeneratedIdMatches() { Review comment: Hadoop generates filename ids with `String.format` without any Locale set and then uses regex to try to match it. This fails on locales `th-TH-u-nu-thai-x-lvariant-TH` and `hi-IN`. This check matches what Hadoop is doing to make sure there is a match. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253094751 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -62,73 +63,88 @@ public static MiniDFSCluster setupClass(String dir) throws Exception { return setupClass(dir, true, true); } - + public static MiniDFSCluster setupClass(String dir, boolean haTesting) throws Exception { return setupClass(dir, haTesting, true); } - + + /** + * Checks that commons-lang3 FastDateFormat works with configured locale + */ + @SuppressForbidden(reason="Call FastDateFormat.format same way Hadoop calls it") + private static void checkFastDateFormat() { Review comment: commons-lang3 has an issue with the locale `ja-JP-u-ca-japanese-x-lvariant-JP` but this will catch if there are other locales with issues. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253094751 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -62,73 +63,88 @@ public static MiniDFSCluster setupClass(String dir) throws Exception { return setupClass(dir, true, true); } - + public static MiniDFSCluster setupClass(String dir, boolean haTesting) throws Exception { return setupClass(dir, haTesting, true); } - + + /** + * Checks that commons-lang3 FastDateFormat works with configured locale + */ + @SuppressForbidden(reason="Call FastDateFormat.format same way Hadoop calls it") + private static void checkFastDateFormat() { Review comment: commons-lang3 has an issue with the locale `ja-JP-u-ca-japanese-x-lvariant-JP` but this will catch if there are other locales with issues. http://mail-archives.apache.org/mod_mbox/commons-user/201901.mbox/%3CCAJU9nmhqgzh7VcxyhJNfb4czC2SvJzZd4o6ARcuD4msof1U2Zw%40mail.gmail.com%3E This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r253094096 ## File path: solr/core/src/test/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java ## @@ -0,0 +1,1045 @@ +/* Review comment: Copied straight from Hadoop code base with minor modifications to fix SecurityManager and ForkJoinPool integration. Details in SOLR-9515 and HDFS-14251 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r252772727 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -67,6 +68,14 @@ public static MiniDFSCluster setupClass(String dir, boolean safeModeTesting, boo LuceneTestCase.assumeFalse("HDFS tests were disabled by -Dtests.disableHdfs", Boolean.parseBoolean(System.getProperty("tests.disableHdfs", "false"))); +// Checks that commons-lang3 FastDateFormat works with configured locale Review comment: Turns out only locale "ja-JP-u-ca-japanese-x-lvariant-JP" causes this failure. Emailed commons-user list about this. No response yet. I have not seen any failures in the past ~24 hours with this fix put in place. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r252415334 ## File path: solr/core/src/test/org/apache/solr/cloud/hdfs/HdfsTestUtil.java ## @@ -67,6 +68,14 @@ public static MiniDFSCluster setupClass(String dir, boolean safeModeTesting, boo LuceneTestCase.assumeFalse("HDFS tests were disabled by -Dtests.disableHdfs", Boolean.parseBoolean(System.getProperty("tests.disableHdfs", "false"))); +// Checks that commons-lang3 FastDateFormat works with configured locale Review comment: Currently certain locales like "ja-JP-u-ca-japanese-x-lvariant-JP" cause `FastDateFormat` from commons-lang3 to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251859763 ## File path: lucene/tools/src/groovy/check-source-patterns.groovy ## @@ -149,6 +149,7 @@ ant.fileScanner{ exclude(name: 'lucene/benchmark/temp/**') exclude(name: '**/CheckLoggingConfiguration.java') exclude(name: 'lucene/tools/src/groovy/check-source-patterns.groovy') // ourselves :-) +exclude(name: 'solr/core/src/test/org/apache/hadoop/**') Review comment: Needed to skip checking the copied HttpServer2 code from Apache Hadoop. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251859799 ## File path: solr/core/build.xml ## @@ -25,6 +25,7 @@
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251859319 ## File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java ## @@ -29,16 +29,18 @@ public boolean reject(Thread t) { return true; } else if (name.startsWith("org.apache.hadoop.hdfs.PeerCache")) { // SOLR-7288 return true; +} else if (name.endsWith("StatisticsDataReferenceCleaner")) { + return true; } else if (name.startsWith("LeaseRenewer")) { // SOLR-7287 return true; } else if (name.startsWith("org.apache.hadoop.fs.FileSystem$Statistics")) { // SOLR-11261 return true; } else if (name.startsWith("ForkJoinPool.")) { // JVM built in pool return true; +} else if (name.startsWith("ForkJoinPool-")) { // JVM built in pool Review comment: I have not tracked down why this is necessary yet. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251859267 ## File path: solr/test-framework/src/java/org/apache/solr/util/BadHdfsThreadsFilter.java ## @@ -29,16 +29,18 @@ public boolean reject(Thread t) { return true; } else if (name.startsWith("org.apache.hadoop.hdfs.PeerCache")) { // SOLR-7288 return true; +} else if (name.endsWith("StatisticsDataReferenceCleaner")) { Review comment: I have not tracked down why this is necessary yet. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251858965 ## File path: solr/core/src/test/org/apache/solr/security/JWTAuthPluginTest.java ## @@ -44,7 +44,6 @@ import org.junit.Before; import org.junit.BeforeClass; import org.junit.Test; -import org.mortbay.util.ajax.JSON; Review comment: Avoid using old mortbay utilities for converting JSON. Uses existing Solr Utils to convert from JSON string. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251858743 ## File path: solr/core/src/test/org/apache/solr/response/TestCustomDocTransformer.java ## @@ -26,7 +26,6 @@ import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.response.transform.DocTransformer; import org.apache.solr.response.transform.TransformerFactory; -import org.bouncycastle.util.Strings; Review comment: Test used to use old bouncycastle dependency which isn't needed anymore from Hadoop. Switched to use builtin Java split. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251858482 ## File path: solr/core/src/test/org/apache/solr/handler/export/TestExportWriter.java ## @@ -28,14 +28,14 @@ import java.util.Map; import java.util.Set; +import com.fasterxml.jackson.databind.ObjectMapper; import org.apache.lucene.util.TestUtil; import org.apache.solr.SolrTestCaseJ4; import org.apache.solr.common.SolrInputDocument; import org.apache.solr.common.util.SuppressForbidden; import org.apache.solr.common.util.Utils; import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.schema.SchemaField; -import org.codehaus.jackson.map.ObjectMapper; Review comment: Test used to use old Jackson class. Replaced with newer one and removed old Jackson pulled in by Hadoop. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251858165 ## File path: solr/core/src/test/org/apache/hadoop/http/HttpServer2.java ## @@ -0,0 +1,1685 @@ +/* Review comment: @uschindler I know you have a lot of experience with JDK 9+ and packaging. Is this even allowed to have a separate package `org.apache.hadoop...` inside the Solr test directory? Will it cause issues with other JDK versions? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3
risdenk commented on a change in pull request #553: SOLR-9515: Update to Hadoop 3 URL: https://github.com/apache/lucene-solr/pull/553#discussion_r251857862 ## File path: solr/core/src/test/org/apache/hadoop/http/HttpServer2.java ## @@ -0,0 +1,1685 @@ +/* Review comment: This file is copied directly from Apache Hadoop to allow integration tests to run under Jetty 9.4. I couldn't find a way to get the shaded Hadoop dependencies (new in Hadoop 3) to work since they shaded too much (like javax.servlet). This made our integration with hadoop-auth break. I have no idea if copying HttpServer2 in here will break other things. I tested with JDK 8 and so far this looks good. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org