[ https://issues.apache.org/jira/browse/HDFS-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Janus Chow updated HDFS-15669: ------------------------------ Description: In the method of Quota#getQuotaUsage, the result could be different on different versions of Java. The trace route is as follows: {code:java} Quota#getQuotaUsage - Quota#getValidQuotaLocations - Quota#getQuotaRemoteLocations - RouterQuotaManager#getPaths{code} In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in Quota#getValidQuotaLocations, the values in HashSet would be used to check if the paths are parent-child relation. So paths in HashSet could affect the result of getValidQuotaLocations as follows: {code:java} // Situation.1 Locations in HashSet [ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8] // Situation.1 getQuota results {ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100} // Situation.2 Locations in HashSet [ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7] // Situation.2 getQuota results {ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 , ns1->/testdir8= 10 8 100 100 }{code} Situation.1 and Situation.2 happen when the underlying implementation of HashSet is different, that is Situation.2 happens using Java-7, and Situation.2 happens after Java-8. This problem can be solved when we do sorting on the results of Quota#_getQuotaRemoteLocations_, but I'm not sure if we should do it. {code:java} /** * Get all quota remote locations across subclusters under given * federation path. * @param path Federation path. * @return List of quota remote locations. * @throws IOException */ private List<RemoteLocation> getQuotaRemoteLocations(String path) throws IOException { List<RemoteLocation> locations = new LinkedList<>(); RouterQuotaManager manager = this.router.getQuotaManager(); if (manager != null) { Set<String> childrenPaths = manager.getPaths(path); for (String childPath : childrenPaths) { locations.addAll(rpcServer.getLocationsForPath(childPath, true, false)); } } ++ Collections.sort(locations); return locations; } {code} was: In the method of Quota#getQuotaUsage, the result could be different on different versions of Java. The trace route is as follows: {code:java} Quota#getQuotaUsage - Quota#getValidQuotaLocations - Quota#getQuotaRemoteLocations - RouterQuotaManager#getPaths{code} In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in Quota#getValidQuotaLocations, the values in HashSet would be used to check if the paths are parent-child relation. So paths in HashSet could affect the result of getValidQuotaLocations as follows: {code:java} // Situation.1 Locations in HashSet [ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8] // Situation.1 getQuota results {ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100} // Situation.2 Locations in HashSet [ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7] // Situation.2 getQuota results {ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 , ns1->/testdir8= 10 8 100 100 }{code} Situation.1 and Situation.2 happen when the underlying implementation of HashSet is different, that is Situation.2 happens using Java-7, and Situation.2 happens after Java-8. This problem can be solved when we do sorting on the results of Quota#_getQuotaRemoteLocations_. {code:java} /** * Get all quota remote locations across subclusters under given * federation path. * @param path Federation path. * @return List of quota remote locations. * @throws IOException */ private List<RemoteLocation> getQuotaRemoteLocations(String path) throws IOException { List<RemoteLocation> locations = new LinkedList<>(); RouterQuotaManager manager = this.router.getQuotaManager(); if (manager != null) { Set<String> childrenPaths = manager.getPaths(path); for (String childPath : childrenPaths) { locations.addAll(rpcServer.getLocationsForPath(childPath, true, false)); } } ++ Collections.sort(locations); return locations; } {code} > [HDFS] [RBF] Incorrect GetQuota caused by different implementation of HashSet > ------------------------------------------------------------------------------ > > Key: HDFS-15669 > URL: https://issues.apache.org/jira/browse/HDFS-15669 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Janus Chow > Priority: Major > > In the method of Quota#getQuotaUsage, the result could be different on > different versions of Java. > The trace route is as follows: > {code:java} > Quota#getQuotaUsage > - Quota#getValidQuotaLocations > - Quota#getQuotaRemoteLocations > - RouterQuotaManager#getPaths{code} > In RouterQuotaManager#getPaths, the path is stored in a _HashSet_. And in > Quota#getValidQuotaLocations, the values in HashSet would be used to check if > the paths are parent-child relation. So paths in HashSet could affect the > result of getValidQuotaLocations as follows: > {code:java} > // Situation.1 Locations in HashSet > [ns0->/testdir7, ns0->/testdir7/subdir, ns1->/testdir8] > // Situation.1 getQuota results > {ns0->/testdir7= 10 6 100 100 , ns1->/testdir8= 10 8 100 100} > // Situation.2 Locations in HashSet > [ns0->/testdir7/subdir, ns1->/testdir8, ns0->/testdir7] > // Situation.2 getQuota results > {ns0->/testdir7= 10 8 100 100 , ns0->/testdir7/subdir= 10 6 100 100 , > ns1->/testdir8= 10 8 100 100 }{code} > Situation.1 and Situation.2 happen when the underlying implementation of > HashSet is different, that is Situation.2 happens using Java-7, and > Situation.2 happens after Java-8. > This problem can be solved when we do sorting on the results of > Quota#_getQuotaRemoteLocations_, but I'm not sure if we should do it. > {code:java} > /** > * Get all quota remote locations across subclusters under given > * federation path. > * @param path Federation path. > * @return List of quota remote locations. > * @throws IOException > */ > private List<RemoteLocation> getQuotaRemoteLocations(String path) > throws IOException { > List<RemoteLocation> locations = new LinkedList<>(); > RouterQuotaManager manager = this.router.getQuotaManager(); > if (manager != null) { > Set<String> childrenPaths = manager.getPaths(path); > for (String childPath : childrenPaths) { > locations.addAll(rpcServer.getLocationsForPath(childPath, true, false)); > } > } > ++ Collections.sort(locations); > return locations; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org