Hi, > One more thing: > > You said "(although it should not be needed). Can you confirm that? I mean, > if you remove that line, does the test still passes with - > Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into > other reasons, since I did see in many other places file permission on > /home/jenkins/..../morfologik-polish-2.1.0.jar itself is granted.
See my other mail! It should not be needed, because Class#getResource does not throw or document any Security implications. But in fact if you don't wrap a doPrivileged around the call, it will also fail without our extra permission. It was added to work around this issue with 3rd party JARs. I just blew up with the symlink on home dir. Uwe > Thanks > Max > > > > On Oct 17, 2016, at 11:17 PM, Wang Weijun <[email protected]> > wrote: > > > > Yes, this is where the problem is. > > > > So it looks like the permission is granted in a policy file instead of being > granted by the class loader itself. In this case, the path of the permission > must match how you access that file. > > > > I'll think about your suggestion. However, can you guarantee the code > always accesses the file using the canonicalized path? For example, suppose > the actual file is /x, both /a and /b symlink to it, and you grant the > permission > on /a in a policy file. Even if I canonicalize /a to /x and grant permissions > on > both /a and /x, you will still see a failure if the code access /b. > > > > --Max > > > >> On Oct 17, 2016, at 11:03 PM, Uwe Schindler <[email protected]> > wrote: > >> > >> Hi, > >> > >> I attached the debugging logs ("all") for the working (conaonicalized) and > non-working case to this mail. Please keep in mind that around the > getResource() is no doPrivileged, so it looks like it uses privileges of > outer JAR. > >> > >> I also checked, we have the following explicitely in our policy file, which > explicitly allows everything in the IVY cache (although it should not be > needed) - so it should really work anyways!!! > >> > >> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-", > "read"; > >> > >> But I think I know the issue - which is really stupid - but it’s a fact > >> affecting > maaaany people! Here is what the policy file parsing says above the > permission given before (please compare first line and second line): > >> > >> [junit4] 2> Active Principals: [] > >> [junit4] 2> policy: granting ("java.io.FilePermission" > "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read") > >> [junit4] 2> policy: granting ("java.io.FilePermission" > "/var/lib/jenkins/.ivy2/cache/-" "read") > >> [junit4] 2> policy: granting ("java.io.FilePermission" > "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute") > >> > >> Unfortunately, the user changed its home directory, the old one > (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not > updated. So ${user.home} looks different. This is the reason why it did not > happen locally on my windows machine! > >> > >> IMHO: When reading the policy file, it should canonicalize all paths, > especially stuff like ${user.home}. At runtime when actually checking the > permissions it should not canonicalize for performance reasons. Not sure if > this is a bug, but this could hit lots of people. I know that other Jenkins > servers where Lucene runs their tests (not yet on Java 9) are setup in similar > ways! > >> > >> Uwe > >> > >> ----- > >> Uwe Schindler > >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: [email protected] > >> > >>> -----Original Message----- > >>> From: Wang Weijun [mailto:[email protected]] > >>> Sent: Monday, October 17, 2016 4:03 PM > >>> To: [email protected] > >>> Cc: Dalibor Topic <[email protected]>; Balchandra Vaidya > >>> <[email protected]>; Muneer Kolarkunnu > >>> <[email protected]>; Rory O'Donnell > >>> <[email protected]> > >>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) > - > >>> Build # 18064 - Unstable! > >>> > >>> OK, I'll try it tomorrow. > >>> > >>> At the same time can you try -Djava.security.debug=all or - > >>> Djava.security.debug=scl and see if it shows what permission is granted? > >>> > >>> Thanks > >>> Max > >>> > >>>> On Oct 17, 2016, at 9:07 PM, Uwe Schindler <[email protected]> > >>> wrote: > >>>> > >>>> FYI, > >>>> > >>>> With: > >>>> $ ant test -Dtests.useSecurityManager=true -Dargs='- > >>> Djava.security.debug=access - > Djdk.io.permissionsUseCanonicalPath=true' > >>>> > >>>> I can see the following line: > >>>> > >>>> [junit4] 2> access: access allowed ("java.io.FilePermission" > >>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik- > >>> polish/bundles/morfologik-polish-2.1.0.jar" "read") > >>>> > >>>> So there is a difference!!! > >>>> > >>>> Uwe > >>>> > >>>> ----- > >>>> Uwe Schindler > >>>> [email protected] > >>>> ASF Member, Apache Lucene PMC / Committer > >>>> Bremen, Germany > >>>> http://lucene.apache.org/ > >>>> > >>>>> -----Original Message----- > >>>>> From: Uwe Schindler [mailto:[email protected]] > >>>>> Sent: Monday, October 17, 2016 2:44 PM > >>>>> To: [email protected]; 'Wang Weijun' > <[email protected]> > >>>>> Cc: 'Dalibor Topic' <[email protected]>; 'Balchandra Vaidya' > >>>>> <[email protected]>; 'Muneer Kolarkunnu' > >>>>> <[email protected]>; 'Rory O'Donnell' > >>>>> <[email protected]> > >>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9- > ea+140) - > >>>>> Build # 18064 - Unstable! > >>>>> > >>>>> Hi, > >>>>> > >>>>> I attached the log of a test run. As you see it says "access denied" > while > >>>>> reading the morphologic JAR file: > >>>>> > >>>>> jenkins@serv1:~/workspace/Lucene-Solr-master- > >>>>> Linux/lucene/analysis/morfologik$ ant test -Dargs='- > >>>>> Djava.security.debug=access' -Dtestcase=TestMorfologikFilterFactory > >>>>> [...] > >>>>> [junit4] 2> access: access denied ("java.io.FilePermission" > >>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik- > >>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read") > >>>>> [junit4] 2> access: access denied ("java.io.FilePermission" > >>>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik- > >>>>> polish/bundles/morfologik-polish-2.1.0.jar" "read") > >>>>> [...] > >>>>> > >>>>> To reproduce, do the following: > >>>>> Check out Lucene's repository from Github: > >>>>> https://github.com/apache/lucene-solr > >>>>> > >>>>> Go to root directory of it and run (Apache Ant required): "ant ivy- > >>> bootstrap" - > >>>>>> this installs the IVY downloader. Be sure to have internet connection > to > >>>>> access Maven Central! > >>>>> > >>>>> The cd into lucene/analysis/morfologik. From there you can run tests > >>> using > >>>>> "ant test". > >>>>> > >>>>> To pass extra JDK args, pass e.g., $ ant test -Dargs='- > >>>>> Djava.security.debug=access' > >>>>> The test passes with the JDK option canonicalized paths. > >>>>> > >>>>> From the above error message you see that the code tries to read the > JAR > >>> file > >>>>> of morphologic, which is completely outside the build directory in the > IVY > >>>>> cache folder, but it is part of the > >>>>> > >>>>> I have seen that you have a short test for the same in your checkin to > JDK, > >>>>> but there is a slight difference: You are not using a JAR file, just > >>>>> plain > class > >>>>> files in your test. And this works also for Lucene. It is just > >>> JARURLConnection > >>>>> that breaks. The reason why other parts of Lucene don't fail is because > >>>>> Lucene's own tests use class file (not yet packaged). In always fails > when > >>> you > >>>>> have 3rd party JAR files. > >>>>> > >>>>> Uwe > >>>>> > >>>>> ----- > >>>>> Uwe Schindler > >>>>> [email protected] > >>>>> ASF Member, Apache Lucene PMC / Committer > >>>>> Bremen, Germany > >>>>> http://lucene.apache.org/ > >>>>>> -----Original Message----- > >>>>>> From: Wang Weijun [mailto:[email protected]] > >>>>>> Sent: Monday, October 17, 2016 2:20 PM > >>>>>> To: Uwe Schindler <[email protected]> > >>>>>> Cc: [email protected]; Dalibor Topic > <[email protected]>; > >>>>>> Balchandra Vaidya <[email protected]>; Muneer > >>> Kolarkunnu > >>>>>> <[email protected]>; Rory O'Donnell > >>>>>> <[email protected]> > >>>>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9- > ea+140) > >>> - > >>>>>> Build # 18064 - Unstable! > >>>>>> > >>>>>> > >>>>>>> On Oct 17, 2016, at 6:24 PM, Rory O'Donnell > >>>>> <[email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>> Adding Max to discussion. > >>>>>>> > >>>>>>> Rgds,Rory > >>>>>>> > >>>>>>> > >>>>>>> On 17/10/2016 08:26, Dawid Weiss wrote: > >>>>>>>> Yeah, this doesn't look so good. This class and the resource it tries > >>>>>>>> to access are within the same JAR file -- if I understand correctly > >>>>>>>> the JDK issue you quoted as the source of the problem, the code > here > >>>>>>>> should be able to pass regardless of file restrictions/ policies? > >>>>>> > >>>>>> Correct. A class should always be able to read a file that's on its > >>> classpath, > >>>>> no > >>>>>> matter if it's inside a jar or a .class file. > >>>>>> > >>>>>> Can someone tell me how to reproduce this error? I have never > played > >>> with > >>>>>> Lucene before. Also, please try adding -Djava.security.debug=access > and > >>>>> see > >>>>>> if you can get some extra debug info. > >>>>>> > >>>>>> Thanks > >>>>>> Max > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> Dawid > >>>>>>>> > >>>>>>>> On Sun, Oct 16, 2016 at 10:09 PM, Uwe Schindler > <[email protected]> > >>>>>> wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> I reverted the Lucene builds to build Java 9 138 for now. I will > later > >>>>> check > >>>>>> if this also happens with build 139, which I have to download first. I > will > >>>>> also > >>>>>> debug locally. > >>>>>>>>> > >>>>>>>>> The code fails because this code hits "null" on getResource() at > >>>>>> > >>> > morfologik.stemming.polish.PolishStemmer.<init>(PolishStemmer.java:34) > >>>>>>>>> > >>>>>>>>> https://github.com/morfologik/morfologik- > >>>>>> stemming/blob/master/morfologik- > >>>>>> > >>>>> > >>> > polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32 > >>>>>>>>> > >>>>>>>>> This is impossible to happen, because the dict file is in same > package. > >>> I > >>>>>> have no idea why this only fails here and not at other places in > Lucene. > >>> The > >>>>>> main difference looks like the use of URL instead of > >>> getResourceAsStream() > >>>>>> like other places in Lucene. > >>>>>>>>> > >>>>>>>>> So this seems to be a major regression in Java 9 build 140. > >>>>>>>>> > >>>>>>>>> Uwe > >>>>>>>>> > >>>>>>>>> ----- > >>>>>>>>> Uwe Schindler > >>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen > >>>>>>>>> http://www.thetaphi.de > >>>>>>>>> eMail: [email protected] > >>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: Uwe Schindler [mailto:[email protected]] > >>>>>>>>>> Sent: Sunday, October 16, 2016 8:38 PM > >>>>>>>>>> To: [email protected] > >>>>>>>>>> Cc: [email protected]; [email protected]; > >>>>>> 'Muneer > >>>>>>>>>> Kolarkunnu' <[email protected]>; 'Dawid Weiss' > >>>>>>>>>> <[email protected]>; [email protected] > >>>>>>>>>> Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9- > >>>>>> ea+140) - > >>>>>>>>>> Build # 18064 - Unstable! > >>>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>> > >>>>>>>>>> this seems to be a new regression in Java 9 ea build 140. > >>> Interestingly > >>>>>> this > >>>>>>>>>> only affects 2 libraries (morphologic and commons-codec > phonetic). > >>>>> We > >>>>>> use > >>>>>>>>>> loading of resources from classloaders at many places; it is > unclear > >>> to > >>>>>> me, > >>>>>>>>>> why it only fails here. I will look into the code, but this is > >>>>>>>>>> outside > of > >>>>>> Lucene. I > >>>>>>>>>> think it might be some crazyness like using context class loader > in > >>> non- > >>>>>> proper > >>>>>>>>>> ways or similar. > >>>>>>>>>> > >>>>>>>>>> Maybe it is a new bug in JDK 9 build 139 or build 140 (the last > >>> working > >>>>>> one > >>>>>>>>>> was build 138). > >>>>>>>>>> > >>>>>>>>>> Uwe > >>>>>>>>>> > >>>>>>>>>> ----- > >>>>>>>>>> Uwe Schindler > >>>>>>>>>> H.-H.-Meier-Allee 63, D-28213 Bremen > >>>>>>>>>> http://www.thetaphi.de > >>>>>>>>>> eMail: [email protected] > >>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
