OK, folks. I have imported the Master branch of apex-core....I updated the dependency for Hadoop to 3.1.0 and here is first round of problems. Of those listed below, all but the last two constants have equivalents in the newer version that were easy enough to do a find and replace with...however, the last two are shown in 2.6.1 as deprecated, but no mention of a replacement...Perhaps someone more familiar with the Apex code base can chime in on these two constants (highlighted in red-ish toward the bottom and copied just below for convenience)?
[ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] cannot find symbol symbol: variable HADOOP_SSL_ENABLED_KEY location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] cannot find symbol symbol: variable HADOOP_SSL_ENABLED_DEFAULT location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16] sun.misc.Signal is internal proprietary API and may be removed in a future release [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16] sun.misc.SignalHandler is internal proprietary API and may be removed in a future release [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27] com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27] com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19] <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T) in com.datatorrent.api.DAG has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17] setCounters(java.lang.Object) in com.datatorrent.api.Context has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11] getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in com.datatorrent.stram.util.ConfigUtils has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21] processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in com.datatorrent.api.StatsListener has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47] processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in com.datatorrent.api.StatsListener has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21] processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in com.datatorrent.api.StatsListener has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22] processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in com.datatorrent.api.StatsListener has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22] processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in com.datatorrent.api.StatsListener has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22] SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18] SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35] getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in com.datatorrent.stram.util.ConfigUtils has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69] getMemory() in org.apache.hadoop.yarn.api.records.Resource has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42] getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in com.datatorrent.stram.util.ConfigUtils has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62] COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44] counters in com.datatorrent.api.Stats.OperatorStats has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74] counters in com.datatorrent.api.Stats.OperatorStats has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72] SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43] com.datatorrent.api.Operator.CheckpointListener in com.datatorrent.api.Operator has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23] com.datatorrent.api.Operator.CheckpointListener in com.datatorrent.api.Operator has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37] com.datatorrent.api.Operator.CheckpointListener in com.datatorrent.api.Operator has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17] com.datatorrent.api.Operator.CheckpointListener in com.datatorrent.api.Operator has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66] com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10] counters in com.datatorrent.api.Stats.OperatorStats has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26] counters in com.datatorrent.stram.api.BaseContext has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9] counters in com.datatorrent.stram.api.BaseContext has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20] writeStringToFile(java.io.File,java.lang.String,boolean) in org.apache.commons.io.FileUtils has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16] copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils has been deprecated [WARNING] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21] readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been deprecated [INFO] 100 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94] cannot find symbol symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87] cannot find symbol symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120] cannot find symbol symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173] cannot find symbol symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123] cannot find symbol symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178] cannot find symbol symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT location: class org.apache.hadoop.yarn.conf.YarnConfiguration [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71] cannot find symbol symbol: variable HADOOP_SSL_ENABLED_KEY location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic [ERROR] /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125] cannot find symbol symbol: variable HADOOP_SSL_ENABLED_DEFAULT location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com> wrote: > Thanks! > > Sent from my iPhone > > > On Jan 13, 2019, at 18:00, Thomas Weise <t...@apache.org> wrote: > > > > For the upgrade it is probably best when you change the version number in > > [1] and then work through the build issues. > > > > I would expect a number of dependency changes since the client classpath > > now has the Hadoop transitive dependencies shaded away [2]. > > > > Regarding the development process, see [3]. > > > > Thomas > > > > [1] > > > https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109 > > [2] https://hadoop.apache.org/docs/r3.0.0/ > > [3] http://apex.apache.org/contributing.html > > > > > > > >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> > wrote: > >> > >> at first blush, I would think, that the YARN node tagging and native > Docker > >> support would be particularly intriguing...the GPU node support would > be a > >> good implementation trick to be able to put operators that do deep > learning > >> tasks... > >> > >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni < > pramod.imman...@gmail.com > >>> > >> wrote: > >> > >>> I agree we should just try changing the dependency first. I am in favor > >> of > >>> upgrading. It will be good to explore if and how we can leverage some > of > >>> the new features in 3.x. > >>> > >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vro...@apache.org> > wrote: > >>>> > >>>> Hi Aaron, > >>>> > >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x, > >>>> recompiling and running unit test. Also, before you put significant > >>> effort > >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once > >>>> everyone agrees and vote passes, please open JIRA for tracking the > >>> changes. > >>>> > >>>> Thank you, > >>>> > >>>> Vlad > >>>> > >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> > >> wrote: > >>>>> > >>>>> As requested by Thomas, I am hoping we can kick off a discussion > >> about > >>> a > >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am not > >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but will > >>>>> confirm, in which case, the ask is to support 3.1+ specifically. > >>>>> > >>>>> I program in Scala for the most part, but have not problem switching > >> to > >>>>> pure Java as needed. That being said, I am very unfamiliar with the > >>> Apex > >>>>> code base and also unsure of the scope and level of effort required > >> to > >>>> get > >>>>> this done. I look to the group for a few pointers on where to begin > >> as > >>>>> well as code style and such...I don't think I will need a lot of > >>>>> hand-holding, just a good shove in the right direction... > >>>>> > >>>>> Thanks again for the discussion and support on this...for background > >>> here > >>>>> is my personal and professional motivation for pressing forward: > >>>>> > >>>>> - Doing research work for clients as well as internal R&D for > >>> streaming > >>>>> data processing > >>>>> - very specific requirement to support the ability to alter > >>>>> topologies programatically and manually on the fly, hence the > >>>> interest in > >>>>> Apex, which is the only platform I know of that has this support > >>>> built > >>>>> in...I don't want to re-invent that wheel unless absolutely > >>>> necessary > >>>>> - General interest in supporting the open-source community, both > >> as a > >>>>> company and individually > >>>>> > >>>>> -- > >>>>> > >>>>> M. Aaron Bossert > >>>>> (202) 594-1703 > >>>>> Punch Cyber Analytics Group > >>>> > >>>> -- > >>> Thanks, > >>> Pramod > >>> http://ts.la/pramod3443 > >>> > >> > >> > >> -- > >> > >> M. Aaron Bossert > >> (202) 594-1703 > >> Punch Cyber Analytics Group > >> > -- M. Aaron Bossert (202) 594-1703 Punch Cyber Analytics Group
