Thanks for the tip, Rick. I haven’t actually built Zeppelin yet; I’ve just been using the v0.5.0 binary.
As far as I know I already have all the wisp dependencies locally, in which case z.load-ing them all inside Zeppelin should work. But it doesn’t for reasons unknown. Maybe I don’t have *all* the dependencies; or their order on the classpath is important; or…? Hence the fat jar approach. Thanks for the warning about spark-notebook! I appreciate testing access through a proxy might be difficult (maybe Apache offers one for testing?). Thanks, Lucas. From: Rick Moritz [mailto:rah...@gmail.com] Sent: 22 September 2015 14:23 To: Partridge, Lucas (GE Aviation) Cc: users@zeppelin.incubator.apache.org Subject: Re: Can't load a dependency (because I'm behind a proxy?) I actually got the unit test to pass, by manually downloading the dependency into the local repo. Using an artifact that isn't in the local repo after building Zeppelin makes for an unintuitive unit test. On the other hand, required ones may already be available in the class path and make for a bad test. I'll have to think a bit about what makes sense here. A workaround I can propose to you, is to build dummy poms with your required dependencies and use those to load the deps into the local repo with a mvn resolve-dependencies call. That should get you up and running, if you have the required access. Alternatively, I know that spark-notebook had issues supporting proxy-lookups as well,and the corresponding issue is still pending as well. In that case I used the same strategy of manually pushing dependencies into the local repo, and then adding that as a source (hence my frantic attempts to do this in Zeppelin as well ;) ) I think having a test that needs internet access is pretty much a no-go, and a different issue from the lack of proxy support. I'll definitely keep an eye on ZEPPELIN-169 since it's relevant for our dev environments, and file a new issue once I have a solid proposition on how to go ahead with testing this feature - a unit test should be self-contained. Maybe packaging the dependency as a resource and adding a local repo is a way to test this feature in isolation. at the very least, the dep should be added as a test-scoped dependency in the pom somewhere... Thanks again for your comments @ZEPPELIN-169, at least for my test-case issue that pointed to some magic-string culprits. Best, Rick On Tue, Sep 22, 2015 at 3:05 PM, Partridge, Lucas (GE Aviation) <lucas.partri...@ge.com<mailto:lucas.partri...@ge.com>> wrote: Hi Rick, Ok, I misunderstood your original statement☺. As you know, I’ve already answered some of your questions about repository lookup at https://issues.apache.org/jira/browse/ZEPPELIN-169<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ZEPPELIN-2D169&d=BQMFaQ&c=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI&r=c1CCNND4PG-Q_V2AJWDWrugZAXQ8Y3EE_f_mAHcpXcs&m=LwUjTVxqkwd68xX1N7iKrbopHdYS-VGC36_tgXJYgHE&s=3z6HtPuPfC3sr5Wzbd-hRvkBGyLWoGFxvfBJWKj8k-E&e=>. I’ve tried using multiple z.load() statements to load wisp and all its 30 to 40 dependencies directly from the file system, but although the subsequent import statement succeeds the actual wisp commands fail with things like NoClassDefFoundError. So now I’m trying to package wisp as a fat jar and then try doing a z.load() of that… Please go ahead and raise that issue if you think it will help, although there is already the feature request at ZEPPELIN-169. Regardless of whether this is a feature request or a bug this renders Zeppelin pretty useless to me behind the corporate firewall… Thanks, Lucas. From: Rick Moritz [mailto:rah...@gmail.com<mailto:rah...@gmail.com>] Sent: 21 September 2015 17:00 To: Partridge, Lucas (GE Aviation) Cc: users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org> Subject: Re: Can't load a dependency (because I'm behind a proxy?) Hi, Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala! I was merely referring to the fact, that maven-central is often set as default repository somewhere internally (i.e. source) or externally (Zeppelin config, Maven/aether config...), and figuring out where this setting comes from is going to take some....patience. Nothing scala-specific in this case, but rather something inherited from sonatype-aether. Also, for info/comparison, the trace from the crashing unit tests: Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.584 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkInterpreterTest testZContextDependencyLoading(org.apache.zeppelin.spark.SparkInterpreterTest) Time elapsed: 63.612 sec <<< FAILURE! java.lang.AssertionError: expected:<SUCCESS> but was:<ERROR> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.zeppelin.spark.SparkInterpreterTest.testZContextDependencyLoading(SparkInterpreterTest.java:159) which is followed up by Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.667 sec <<< FAILURE! - in org.apache.zeppelin.spark.DepInterpreterTest testDefault(org.apache.zeppelin.spark.DepInterpreterTest) Time elapsed: 69.528 sec <<< ERROR! java.lang.NullPointerException: null at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352) at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141) at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98) at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189) at org.apache.zeppelin.spark.DepInterpreterTest.testDefault(DepInterpreterTest.java:88) I'll give this a few hours yet, before opening an issue, maybe I'm just "holding it wrong". I doubt an issue will push this along much faster either, unless one of us actually submits a patch/PR to go along with it ;) Best, Rick From: Rick Moritz [mailto:rah...@gmail.com<mailto:rah...@gmail.com>] Sent: 21 September 2015 15:19 To: users@zeppelin.incubator.apache.org<mailto:users@zeppelin.incubator.apache.org> Cc: Partridge, Lucas (GE Aviation) Subject: RE: Can't load a dependency (because I'm behind a proxy?) Hello Lucas, hello list, hopefully this message will thread properly. This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system. Since this actually fails unit tests, I think creating a corresponding issue is a logical next step. I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky. Have you had a look at where this could be overridden yet? Filed an issue already? Unless we get some Progress going in this thread, we should start the usual procedures... Thanks and Best, Rick
