When I try to do a pull request from Intellij, I get a permissions error...I have not done this much, so it could be something really dumb on my part...do I need you to authorize my account in some way?
On Tue, Jan 15, 2019 at 3:20 PM Aaron Bossert <aa...@punchcyber.com> wrote: > OK, I have my local clone working and updated for Hadoop 3.1... > > On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> OK, folks. I have imported the Master branch of apex-core....I updated >> the dependency for Hadoop to 3.1.0 and here is first round of problems. Of >> those listed below, all but the last two constants have equivalents in the >> newer version that were easy enough to do a find and replace >> with...however, the last two are shown in 2.6.1 as deprecated, but no >> mention of a replacement...Perhaps someone more familiar with the Apex code >> base can chime in on these two constants (highlighted in red-ish toward the >> bottom and copied just below for convenience)? >> >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] >> cannot find symbol >> symbol: variable HADOOP_SSL_ENABLED_KEY >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] >> cannot find symbol >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic >> >> >> >> [INFO] ------------------------------------------------------------- >> [WARNING] COMPILATION WARNING : >> [INFO] ------------------------------------------------------------- >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16] >> sun.misc.Signal is internal proprietary API and may be removed in a future >> release >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16] >> sun.misc.SignalHandler is internal proprietary API and may be removed in a >> future release >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27] >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19] >> <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T) >> in com.datatorrent.api.DAG has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17] >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11] >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in >> com.datatorrent.stram.util.ConfigUtils has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21] >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in >> com.datatorrent.api.StatsListener has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47] >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in >> com.datatorrent.api.StatsListener has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21] >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in >> com.datatorrent.api.StatsListener has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22] >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in >> com.datatorrent.api.StatsListener has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22] >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in >> com.datatorrent.api.StatsListener has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22] >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18] >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35] >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in >> com.datatorrent.stram.util.ConfigUtils has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69] >> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42] >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in >> com.datatorrent.stram.util.ConfigUtils has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62] >> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been >> deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44] >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74] >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72] >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43] >> com.datatorrent.api.Operator.CheckpointListener in >> com.datatorrent.api.Operator has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23] >> com.datatorrent.api.Operator.CheckpointListener in >> com.datatorrent.api.Operator has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37] >> com.datatorrent.api.Operator.CheckpointListener in >> com.datatorrent.api.Operator has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17] >> com.datatorrent.api.Operator.CheckpointListener in >> com.datatorrent.api.Operator has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66] >> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10] >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26] >> counters in com.datatorrent.stram.api.BaseContext has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9] >> counters in com.datatorrent.stram.api.BaseContext has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20] >> writeStringToFile(java.io.File,java.lang.String,boolean) in >> org.apache.commons.io.FileUtils has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16] >> copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils >> has been deprecated >> [WARNING] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21] >> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been >> deprecated >> [INFO] 100 warnings >> [INFO] ------------------------------------------------------------- >> [INFO] ------------------------------------------------------------- >> [ERROR] COMPILATION ERROR : >> [INFO] ------------------------------------------------------------- >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178] >> cannot find symbol >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] >> cannot find symbol >> symbol: variable HADOOP_SSL_ENABLED_KEY >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic >> [ERROR] >> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] >> cannot find symbol >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic >> >> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> Thanks! >>> >>> Sent from my iPhone >>> >>> > On Jan 13, 2019, at 18:00, Thomas Weise <t...@apache.org> wrote: >>> > >>> > For the upgrade it is probably best when you change the version number >>> in >>> > [1] and then work through the build issues. >>> > >>> > I would expect a number of dependency changes since the client >>> classpath >>> > now has the Hadoop transitive dependencies shaded away [2]. >>> > >>> > Regarding the development process, see [3]. >>> > >>> > Thomas >>> > >>> > [1] >>> > >>> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109 >>> > [2] https://hadoop.apache.org/docs/r3.0.0/ >>> > [3] http://apex.apache.org/contributing.html >>> > >>> > >>> > >>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> >>> wrote: >>> >> >>> >> at first blush, I would think, that the YARN node tagging and native >>> Docker >>> >> support would be particularly intriguing...the GPU node support would >>> be a >>> >> good implementation trick to be able to put operators that do deep >>> learning >>> >> tasks... >>> >> >>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni < >>> pramod.imman...@gmail.com >>> >>> >>> >> wrote: >>> >> >>> >>> I agree we should just try changing the dependency first. I am in >>> favor >>> >> of >>> >>> upgrading. It will be good to explore if and how we can leverage >>> some of >>> >>> the new features in 3.x. >>> >>> >>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vro...@apache.org> >>> wrote: >>> >>>> >>> >>>> Hi Aaron, >>> >>>> >>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x, >>> >>>> recompiling and running unit test. Also, before you put significant >>> >>> effort >>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once >>> >>>> everyone agrees and vote passes, please open JIRA for tracking the >>> >>> changes. >>> >>>> >>> >>>> Thank you, >>> >>>> >>> >>>> Vlad >>> >>>> >>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> >>> >> wrote: >>> >>>>> >>> >>>>> As requested by Thomas, I am hoping we can kick off a discussion >>> >> about >>> >>> a >>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am >>> not >>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but >>> will >>> >>>>> confirm, in which case, the ask is to support 3.1+ specifically. >>> >>>>> >>> >>>>> I program in Scala for the most part, but have not problem >>> switching >>> >> to >>> >>>>> pure Java as needed. That being said, I am very unfamiliar with >>> the >>> >>> Apex >>> >>>>> code base and also unsure of the scope and level of effort required >>> >> to >>> >>>> get >>> >>>>> this done. I look to the group for a few pointers on where to >>> begin >>> >> as >>> >>>>> well as code style and such...I don't think I will need a lot of >>> >>>>> hand-holding, just a good shove in the right direction... >>> >>>>> >>> >>>>> Thanks again for the discussion and support on this...for >>> background >>> >>> here >>> >>>>> is my personal and professional motivation for pressing forward: >>> >>>>> >>> >>>>> - Doing research work for clients as well as internal R&D for >>> >>> streaming >>> >>>>> data processing >>> >>>>> - very specific requirement to support the ability to alter >>> >>>>> topologies programatically and manually on the fly, hence the >>> >>>> interest in >>> >>>>> Apex, which is the only platform I know of that has this >>> support >>> >>>> built >>> >>>>> in...I don't want to re-invent that wheel unless absolutely >>> >>>> necessary >>> >>>>> - General interest in supporting the open-source community, both >>> >> as a >>> >>>>> company and individually >>> >>>>> >>> >>>>> -- >>> >>>>> >>> >>>>> M. Aaron Bossert >>> >>>>> (202) 594-1703 >>> >>>>> Punch Cyber Analytics Group >>> >>>> >>> >>>> -- >>> >>> Thanks, >>> >>> Pramod >>> >>> http://ts.la/pramod3443 >>> >>> >>> >> >>> >> >>> >> -- >>> >> >>> >> M. Aaron Bossert >>> >> (202) 594-1703 >>> >> Punch Cyber Analytics Group >>> >> >>> >> >> >> -- >> >> M. Aaron Bossert >> (202) 594-1703 >> Punch Cyber Analytics Group >> >> >> > > -- > > M. Aaron Bossert > (202) 594-1703 > Punch Cyber Analytics Group > > > -- M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group
