It was an issue in the chair to keyboard interface...just submitted PR 607 <https://github.com/apache/apex-core/pull/607>. The Jenkins build/test/whatever is still running...
On Tue, Jan 15, 2019 at 11:36 PM Pramod Immaneni <pramod.imman...@gmail.com> wrote: > Shouldn't require additional authorization unless something changed > recently. Did you try creating a PR from your branch in your fork on > github.com? > > Thanks > > On Tue, Jan 15, 2019 at 6:59 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > > > When I try to do a pull request from Intellij, I get a permissions > > error...I have not done this much, so it could be something really dumb > on > > my part...do I need you to authorize my account in some way? > > > > On Tue, Jan 15, 2019 at 3:20 PM Aaron Bossert <aa...@punchcyber.com> > > wrote: > > > > > OK, I have my local clone working and updated for Hadoop 3.1... > > > > > > On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com> > > > wrote: > > > > > >> OK, folks. I have imported the Master branch of apex-core....I > updated > > >> the dependency for Hadoop to 3.1.0 and here is first round of > > problems. Of > > >> those listed below, all but the last two constants have equivalents in > > the > > >> newer version that were easy enough to do a find and replace > > >> with...however, the last two are shown in 2.6.1 as deprecated, but no > > >> mention of a replacement...Perhaps someone more familiar with the Apex > > code > > >> base can chime in on these two constants (highlighted in red-ish > toward > > the > > >> bottom and copied just below for convenience)? > > >> > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] > > >> cannot find symbol > > >> symbol: variable HADOOP_SSL_ENABLED_KEY > > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] > > >> cannot find symbol > > >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT > > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > >> > > >> > > >> > > >> [INFO] ------------------------------------------------------------- > > >> [WARNING] COMPILATION WARNING : > > >> [INFO] ------------------------------------------------------------- > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16] > > >> sun.misc.Signal is internal proprietary API and may be removed in a > > future > > >> release > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16] > > >> sun.misc.SignalHandler is internal proprietary API and may be removed > > in a > > >> future release > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27] > > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] > > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] > > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19] > > >> > > > <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T) > > >> in com.datatorrent.api.DAG has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17] > > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11] > > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > > >> com.datatorrent.stram.util.ConfigUtils has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21] > > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) > in > > >> com.datatorrent.api.StatsListener has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47] > > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) > in > > >> com.datatorrent.api.StatsListener has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21] > > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) > in > > >> com.datatorrent.api.StatsListener has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22] > > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) > in > > >> com.datatorrent.api.StatsListener has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22] > > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) > in > > >> com.datatorrent.api.StatsListener has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22] > > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18] > > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35] > > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > > >> com.datatorrent.stram.util.ConfigUtils has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69] > > >> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42] > > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > > >> com.datatorrent.stram.util.ConfigUtils has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62] > > >> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has > > been > > >> deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44] > > >> counters in com.datatorrent.api.Stats.OperatorStats has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74] > > >> counters in com.datatorrent.api.Stats.OperatorStats has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72] > > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43] > > >> com.datatorrent.api.Operator.CheckpointListener in > > >> com.datatorrent.api.Operator has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23] > > >> com.datatorrent.api.Operator.CheckpointListener in > > >> com.datatorrent.api.Operator has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37] > > >> com.datatorrent.api.Operator.CheckpointListener in > > >> com.datatorrent.api.Operator has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17] > > >> com.datatorrent.api.Operator.CheckpointListener in > > >> com.datatorrent.api.Operator has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66] > > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been > > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10] > > >> counters in com.datatorrent.api.Stats.OperatorStats has been > deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26] > > >> counters in com.datatorrent.stram.api.BaseContext has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9] > > >> counters in com.datatorrent.stram.api.BaseContext has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20] > > >> writeStringToFile(java.io.File,java.lang.String,boolean) in > > >> org.apache.commons.io.FileUtils has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16] > > >> copy(java.io.InputStream,java.io.Writer) in > > org.apache.commons.io.IOUtils > > >> has been deprecated > > >> [WARNING] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21] > > >> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has > > been > > >> deprecated > > >> [INFO] 100 warnings > > >> [INFO] ------------------------------------------------------------- > > >> [INFO] ------------------------------------------------------------- > > >> [ERROR] COMPILATION ERROR : > > >> [INFO] ------------------------------------------------------------- > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178] > > >> cannot find symbol > > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT > > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] > > >> cannot find symbol > > >> symbol: variable HADOOP_SSL_ENABLED_KEY > > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > >> [ERROR] > > >> > > > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] > > >> cannot find symbol > > >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT > > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > >> > > >> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com> > > >> wrote: > > >> > > >>> Thanks! > > >>> > > >>> Sent from my iPhone > > >>> > > >>> > On Jan 13, 2019, at 18:00, Thomas Weise <t...@apache.org> wrote: > > >>> > > > >>> > For the upgrade it is probably best when you change the version > > number > > >>> in > > >>> > [1] and then work through the build issues. > > >>> > > > >>> > I would expect a number of dependency changes since the client > > >>> classpath > > >>> > now has the Hadoop transitive dependencies shaded away [2]. > > >>> > > > >>> > Regarding the development process, see [3]. > > >>> > > > >>> > Thomas > > >>> > > > >>> > [1] > > >>> > > > >>> > > > https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109 > > >>> > [2] https://hadoop.apache.org/docs/r3.0.0/ > > >>> > [3] http://apex.apache.org/contributing.html > > >>> > > > >>> > > > >>> > > > >>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert < > aa...@punchcyber.com > > > > > >>> wrote: > > >>> >> > > >>> >> at first blush, I would think, that the YARN node tagging and > native > > >>> Docker > > >>> >> support would be particularly intriguing...the GPU node support > > would > > >>> be a > > >>> >> good implementation trick to be able to put operators that do deep > > >>> learning > > >>> >> tasks... > > >>> >> > > >>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni < > > >>> pramod.imman...@gmail.com > > >>> >>> > > >>> >> wrote: > > >>> >> > > >>> >>> I agree we should just try changing the dependency first. I am in > > >>> favor > > >>> >> of > > >>> >>> upgrading. It will be good to explore if and how we can leverage > > >>> some of > > >>> >>> the new features in 3.x. > > >>> >>> > > >>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vro...@apache.org> > > >>> wrote: > > >>> >>>> > > >>> >>>> Hi Aaron, > > >>> >>>> > > >>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x, > > >>> >>>> recompiling and running unit test. Also, before you put > > significant > > >>> >>> effort > > >>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. > Once > > >>> >>>> everyone agrees and vote passes, please open JIRA for tracking > the > > >>> >>> changes. > > >>> >>>> > > >>> >>>> Thank you, > > >>> >>>> > > >>> >>>> Vlad > > >>> >>>> > > >>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com > > > > >>> >> wrote: > > >>> >>>>> > > >>> >>>>> As requested by Thomas, I am hoping we can kick off a > discussion > > >>> >> about > > >>> >>> a > > >>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I > am > > >>> not > > >>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but > > >>> will > > >>> >>>>> confirm, in which case, the ask is to support 3.1+ > specifically. > > >>> >>>>> > > >>> >>>>> I program in Scala for the most part, but have not problem > > >>> switching > > >>> >> to > > >>> >>>>> pure Java as needed. That being said, I am very unfamiliar > with > > >>> the > > >>> >>> Apex > > >>> >>>>> code base and also unsure of the scope and level of effort > > required > > >>> >> to > > >>> >>>> get > > >>> >>>>> this done. I look to the group for a few pointers on where to > > >>> begin > > >>> >> as > > >>> >>>>> well as code style and such...I don't think I will need a lot > of > > >>> >>>>> hand-holding, just a good shove in the right direction... > > >>> >>>>> > > >>> >>>>> Thanks again for the discussion and support on this...for > > >>> background > > >>> >>> here > > >>> >>>>> is my personal and professional motivation for pressing > forward: > > >>> >>>>> > > >>> >>>>> - Doing research work for clients as well as internal R&D for > > >>> >>> streaming > > >>> >>>>> data processing > > >>> >>>>> - very specific requirement to support the ability to alter > > >>> >>>>> topologies programatically and manually on the fly, hence > the > > >>> >>>> interest in > > >>> >>>>> Apex, which is the only platform I know of that has this > > >>> support > > >>> >>>> built > > >>> >>>>> in...I don't want to re-invent that wheel unless absolutely > > >>> >>>> necessary > > >>> >>>>> - General interest in supporting the open-source community, > both > > >>> >> as a > > >>> >>>>> company and individually > > >>> >>>>> > > >>> >>>>> -- > > >>> >>>>> > > >>> >>>>> M. Aaron Bossert > > >>> >>>>> (202) 594-1703 > > >>> >>>>> Punch Cyber Analytics Group > > >>> >>>> > > >>> >>>> -- > > >>> >>> Thanks, > > >>> >>> Pramod > > >>> >>> http://ts.la/pramod3443 > > >>> >>> > > >>> >> > > >>> >> > > >>> >> -- > > >>> >> > > >>> >> M. Aaron Bossert > > >>> >> (202) 594-1703 > > >>> >> Punch Cyber Analytics Group > > >>> >> > > >>> > > >> > > >> > > >> -- > > >> > > >> M. Aaron Bossert > > >> (202) 594-1703 > > >> Punch Cyber Analytics Group > > >> > > >> > > >> > > > > > > -- > > > > > > M. Aaron Bossert > > > (202) 594-1703 > > > Punch Cyber Analytics Group > > > > > > > > > > > > > -- > > > > M. Aaron Bossert > > (202) 594-1703 > > Punch Cyber Analytics Group > > > > > -- > Thanks, > Pramod > http://ts.la/pramod3443 > -- M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group
