OK, I have my local clone working and updated for Hadoop 3.1... On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> OK, folks. I have imported the Master branch of apex-core....I updated > the dependency for Hadoop to 3.1.0 and here is first round of problems. Of > those listed below, all but the last two constants have equivalents in the > newer version that were easy enough to do a find and replace > with...however, the last two are shown in 2.6.1 as deprecated, but no > mention of a replacement...Perhaps someone more familiar with the Apex code > base can chime in on these two constants (highlighted in red-ish toward the > bottom and copied just below for convenience)? > > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] > cannot find symbol > symbol: variable HADOOP_SSL_ENABLED_KEY > location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] > cannot find symbol > symbol: variable HADOOP_SSL_ENABLED_DEFAULT > location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > > > [INFO] ------------------------------------------------------------- > [WARNING] COMPILATION WARNING : > [INFO] ------------------------------------------------------------- > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16] > sun.misc.Signal is internal proprietary API and may be removed in a future > release > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16] > sun.misc.SignalHandler is internal proprietary API and may be removed in a > future release > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27] > com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] > com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] > com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19] > <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T) > in com.datatorrent.api.DAG has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17] > setCounters(java.lang.Object) in com.datatorrent.api.Context has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11] > getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > com.datatorrent.stram.util.ConfigUtils has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21] > processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in > com.datatorrent.api.StatsListener has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47] > processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in > com.datatorrent.api.StatsListener has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21] > processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in > com.datatorrent.api.StatsListener has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22] > processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in > com.datatorrent.api.StatsListener has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22] > processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in > com.datatorrent.api.StatsListener has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22] > SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18] > SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35] > getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > com.datatorrent.stram.util.ConfigUtils has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69] > getMemory() in org.apache.hadoop.yarn.api.records.Resource has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42] > getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in > com.datatorrent.stram.util.ConfigUtils has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62] > COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been > deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44] > counters in com.datatorrent.api.Stats.OperatorStats has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74] > counters in com.datatorrent.api.Stats.OperatorStats has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72] > SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43] > com.datatorrent.api.Operator.CheckpointListener in > com.datatorrent.api.Operator has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23] > com.datatorrent.api.Operator.CheckpointListener in > com.datatorrent.api.Operator has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37] > com.datatorrent.api.Operator.CheckpointListener in > com.datatorrent.api.Operator has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17] > com.datatorrent.api.Operator.CheckpointListener in > com.datatorrent.api.Operator has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66] > com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10] > counters in com.datatorrent.api.Stats.OperatorStats has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26] > counters in com.datatorrent.stram.api.BaseContext has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9] > counters in com.datatorrent.stram.api.BaseContext has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20] > writeStringToFile(java.io.File,java.lang.String,boolean) in > org.apache.commons.io.FileUtils has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16] > copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils > has been deprecated > [WARNING] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21] > readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been > deprecated > [INFO] 100 warnings > [INFO] ------------------------------------------------------------- > [INFO] ------------------------------------------------------------- > [ERROR] COMPILATION ERROR : > [INFO] ------------------------------------------------------------- > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94] > cannot find symbol > symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87] > cannot find symbol > symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120] > cannot find symbol > symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173] > cannot find symbol > symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123] > cannot find symbol > symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178] > cannot find symbol > symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT > location: class org.apache.hadoop.yarn.conf.YarnConfiguration > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] > cannot find symbol > symbol: variable HADOOP_SSL_ENABLED_KEY > location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > [ERROR] > /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] > cannot find symbol > symbol: variable HADOOP_SSL_ENABLED_DEFAULT > location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic > > On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> Thanks! >> >> Sent from my iPhone >> >> > On Jan 13, 2019, at 18:00, Thomas Weise <t...@apache.org> wrote: >> > >> > For the upgrade it is probably best when you change the version number >> in >> > [1] and then work through the build issues. >> > >> > I would expect a number of dependency changes since the client classpath >> > now has the Hadoop transitive dependencies shaded away [2]. >> > >> > Regarding the development process, see [3]. >> > >> > Thomas >> > >> > [1] >> > >> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109 >> > [2] https://hadoop.apache.org/docs/r3.0.0/ >> > [3] http://apex.apache.org/contributing.html >> > >> > >> > >> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >> >> >> at first blush, I would think, that the YARN node tagging and native >> Docker >> >> support would be particularly intriguing...the GPU node support would >> be a >> >> good implementation trick to be able to put operators that do deep >> learning >> >> tasks... >> >> >> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni < >> pramod.imman...@gmail.com >> >>> >> >> wrote: >> >> >> >>> I agree we should just try changing the dependency first. I am in >> favor >> >> of >> >>> upgrading. It will be good to explore if and how we can leverage some >> of >> >>> the new features in 3.x. >> >>> >> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vro...@apache.org> >> wrote: >> >>>> >> >>>> Hi Aaron, >> >>>> >> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x, >> >>>> recompiling and running unit test. Also, before you put significant >> >>> effort >> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once >> >>>> everyone agrees and vote passes, please open JIRA for tracking the >> >>> changes. >> >>>> >> >>>> Thank you, >> >>>> >> >>>> Vlad >> >>>> >> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> >> >> wrote: >> >>>>> >> >>>>> As requested by Thomas, I am hoping we can kick off a discussion >> >> about >> >>> a >> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am >> not >> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but will >> >>>>> confirm, in which case, the ask is to support 3.1+ specifically. >> >>>>> >> >>>>> I program in Scala for the most part, but have not problem switching >> >> to >> >>>>> pure Java as needed. That being said, I am very unfamiliar with the >> >>> Apex >> >>>>> code base and also unsure of the scope and level of effort required >> >> to >> >>>> get >> >>>>> this done. I look to the group for a few pointers on where to begin >> >> as >> >>>>> well as code style and such...I don't think I will need a lot of >> >>>>> hand-holding, just a good shove in the right direction... >> >>>>> >> >>>>> Thanks again for the discussion and support on this...for background >> >>> here >> >>>>> is my personal and professional motivation for pressing forward: >> >>>>> >> >>>>> - Doing research work for clients as well as internal R&D for >> >>> streaming >> >>>>> data processing >> >>>>> - very specific requirement to support the ability to alter >> >>>>> topologies programatically and manually on the fly, hence the >> >>>> interest in >> >>>>> Apex, which is the only platform I know of that has this support >> >>>> built >> >>>>> in...I don't want to re-invent that wheel unless absolutely >> >>>> necessary >> >>>>> - General interest in supporting the open-source community, both >> >> as a >> >>>>> company and individually >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> M. Aaron Bossert >> >>>>> (202) 594-1703 >> >>>>> Punch Cyber Analytics Group >> >>>> >> >>>> -- >> >>> Thanks, >> >>> Pramod >> >>> http://ts.la/pramod3443 >> >>> >> >> >> >> >> >> -- >> >> >> >> M. Aaron Bossert >> >> (202) 594-1703 >> >> Punch Cyber Analytics Group >> >> >> > > > -- > > M. Aaron Bossert > (202) 594-1703 > Punch Cyber Analytics Group > > > -- M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group
