> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java > > Line 49 (original), 50 (patched) > > <https://reviews.apache.org/r/54705/diff/8-9/?file=1714632#file1714632line50> > > > > Why would this require a ConcurrentHashMap? > > Peter Cseh wrote: > It a precaution to avoid issues when getInstance() is called from > multiple threads and the map is filled up from two threads. It shouldn't > cause any issues with performance or threads this way.
Using a ConcurrentHashMap does not help in anyway. That variable is local to the new CredentialsProviderFactory being created. Two threads cannot fill the same map as the constructors will create two different instances. If two threads call getInstance at same time and if instance is null, both will create a instance and the last one will win. Ideally we should avoid that by synchronizing, but in this case based on the nature of data it is not a problem. Please switch it back to HashMap. > On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote: > > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java > > Line 145 (original), 148 (patched) > > <https://reviews.apache.org/r/54705/diff/8-9/?file=1714649#file1714649line148> > > > > This method can be removed now. It is still not removed. - Rohini ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54705/#review175722 ----------------------------------------------------------- On May 23, 2017, 4:43 p.m., Peter Cseh wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54705/ > ----------------------------------------------------------- > > (Updated May 23, 2017, 4:43 p.m.) > > > Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini > Palaniswamy. > > > Bugs: OOZIE-1770 > https://issues.apache.org/jira/browse/OOZIE-1770 > > > Repository: oozie-git > > > Description > ------- > > Replacing the launcher MR job with a launcher AM > > > Diffs > ----- > > client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f > client/src/main/java/org/apache/oozie/cli/OozieCLI.java > 38fb84e8a8918aee6ae57d19a29cdf181aba1955 > client/src/main/java/org/apache/oozie/client/XOozieClient.java > 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a > core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d > core/src/main/java/org/apache/oozie/WorkflowJobBean.java > 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc > core/src/main/java/org/apache/oozie/action/ActionExecutor.java > 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 > core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java > eadb47b07d157f45f086517ab41c4b9826f003f0 > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java > 6fe22fbd2fca5d873d58ece60f20d0f131371475 > > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java > PRE-CREATION > core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java > 78cd257a836707c1efeac13566d6285d42970545 > core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java > 27654741101d2056b25626fbbe345183eafd612d > core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java > 801bfe6eb212b549888f0e9fb7df7b65bcf59393 > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java > 36ad1df727e869b8051481ad969771e91bc6ae07 > core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java > d7689a924059e553abaeed374eed4072933292cb > core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java > ad2a71d641caa71efc098ad00bfbab3cec4d9506 > core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java > 307f565786ad77f5a35b5790bd5b052b04c6eac8 > core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java > 2aed936af75da38c433be5d202bdc01c2f612c76 > core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java > bd2f1f768c5b04bef4a33613baede9202144c747 > core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java > 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java > 06ae5fd9a52099c5bc63b038d336ce96103c8819 > core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java > 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 > > core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java > 55c937249ac35e14bd63dd0c1fa962ea636afa3a > core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java > 581d3b3219eff882f961c8e495611e873c6a5a52 > core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java > 8b2dc16db80ad602911f1f64e60cfad41b198922 > > core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java > 92e149d5cac6509893c8507b776263a38312a53b > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java > b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java > 1a3197abb52f80b545dcc8634a8cac7a281a9eac > core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java > 22e28746a9a39ae6c1e22326cc4f8f759d820b20 > core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java > fb021bd022d199f93fc5674843514abf45ae12a5 > core/src/main/java/org/apache/oozie/command/XCommand.java > bdf13f65ea25e8e0f5125e2798568a2c03fffe81 > core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java > d2a2742ae418070a1cd0a889f14194224bc944a6 > core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java > cc61d3de813c75c7f1cf4d405fc043253e3ca4ea > > core/src/main/java/org/apache/oozie/command/wf/SubmitScriptLanguageXCommand.java > 9d413059e99bc676feb63e02a2f704048eb4a355 > core/src/main/java/org/apache/oozie/command/wf/SubmitSqoopXCommand.java > 51b739e21c802e09f4110b740138f4ff5c341fe4 > > core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java > ace120d10e3546201aa4145df56abf7933fb24b8 > > core/src/main/java/org/apache/oozie/dependency/hcat/EhcacheHCatDependencyCache.java > 6f0abf6e094ed04158652191f2b8bb3b0cd32e90 > core/src/main/java/org/apache/oozie/service/CallableQueueService.java > a86a8d0889138cd1564ca9253e5b319ea52e55cc > core/src/main/java/org/apache/oozie/service/EventHandlerService.java > 22c6fb0965f45c5cf2322c251d4a8aa77d263458 > core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java > 23a9d92bf56a625469ec247cfb9727972511b101 > core/src/main/java/org/apache/oozie/service/Services.java > 829d5f5cd90c86e6cba9854dc0b4b7df570b8900 > core/src/main/java/org/apache/oozie/service/ShareLibService.java > b79bd37bb2b44348f220128334497d788dd922d4 > core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java > 612302173e0f485345558b0c96a8a75d2f43d238 > core/src/main/java/org/apache/oozie/store/OozieSchema.java > 23dd1e508b6c721a10c9fe828baa5cd0d95064b4 > core/src/main/java/org/apache/oozie/util/ClasspathUtils.java PRE-CREATION > core/src/main/java/org/apache/oozie/util/IOUtils.java > a6c2fdae049d1df936a0345f5ffc6609ea40a5c0 > core/src/main/java/org/apache/oozie/util/JobUtils.java > 63f88acd6fb91668baa6cd0a80b2d45435e7b213 > core/src/main/resources/META-INF/persistence.xml > edda2d5e22856b0250a53bfde7c73fd96f85c6e9 > core/src/main/resources/oozie-default.xml > b22b1ce64e5cdf50359479eec6944caf08f44c7b > core/src/main/resources/oozie-log4j.properties > 05fb37af2a04cfcc464ae98940767d0e5e802880 > core/src/test/java/org/apache/oozie/QueryServlet.java PRE-CREATION > core/src/test/java/org/apache/oozie/action/hadoop/CredentialForTest.java > 06298916e797890731134370412f51f3456aa70c > core/src/test/java/org/apache/oozie/action/hadoop/InsertTestToken.java > 9da8fbe18923899489c71f40c99a01fe6522e143 > core/src/test/java/org/apache/oozie/action/hadoop/LauncherMainTester.java > 4baed6e42beccd2fdb55ab1e6fb150c0d8821a9c > core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java > 8f08ddd9fa077fc9526814fd25e7477850c51737 > > core/src/test/java/org/apache/oozie/action/hadoop/TestDistCpActionExecutor.java > e8a140f330062c5ce8324112aae83b3d0ae1f772 > core/src/test/java/org/apache/oozie/action/hadoop/TestFSPrepareActions.java > 386fef3b89a6342cd09d8e7841d8997e06af46c4 > core/src/test/java/org/apache/oozie/action/hadoop/TestFsActionExecutor.java > 09d723ac96ad8f06e3006a7ded3d9b7f8289bb45 > core/src/test/java/org/apache/oozie/action/hadoop/TestHCatCredentials.java > e6d43ca820330a5f00cc5d82e8cfb88f3a814fb4 > > core/src/test/java/org/apache/oozie/action/hadoop/TestHCatPrepareActions.java > d66d9c9d5c1f84ea11f4bf43e19bd9523933c6eb > > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java > b27b3d8a2b0ce55df70e55c2cb525be0c6573700 > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaMain.java > bb2df43d0d0f9e1630b53ff2c9525e2f949bbcbd > core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java > aa938d01b7273d6db273908e5480b71613598135 > > core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAMCallbackNotifier.java > PRE-CREATION > > core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java > 4cda6151ca5bb0d9007f11bfe712de74b1a0da26 > core/src/test/java/org/apache/oozie/action/hadoop/TestOozieJobInfo.java > 5bc7d00ff6c6c2d29806633761cba4a940ce512a > > core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java > df9e939cb236e8b65e7b3aa9ab683c33f5fa02b7 > > core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java > f12927b89f6e4be36a5d3f6d8677684e503295ae > core/src/test/java/org/apache/oozie/action/hadoop/TestShellMain.java > e757e542cb6256906f43a36f7d30c8e8b8aa04e2 > core/src/test/java/org/apache/oozie/client/TestOozieCLI.java > dbc160f0fa971fdce2a81b073a1af17ada3c6591 > core/src/test/java/org/apache/oozie/client/TestWorkflowXClient.java > 51ae9e87d060c9e276ca74b45b24a5f5ae65e722 > > core/src/test/java/org/apache/oozie/command/coord/TestCoordChangeXCommand.java > c071000b22d87c76fad3f18597ded604f8c2f732 > core/src/test/java/org/apache/oozie/command/wf/HangServlet.java > 3344cf97762c77f5660d94341e72df4243f76eb2 > core/src/test/java/org/apache/oozie/command/wf/TestActionCheckXCommand.java > 5898d1af79631a1b7e5021d9ca05685eab7225a5 > core/src/test/java/org/apache/oozie/command/wf/TestActionStartXCommand.java > ea90c087fc145650dfb77ac35a03ba1203ffb0f2 > core/src/test/java/org/apache/oozie/command/wf/TestSubmitHiveXCommand.java > 43edf5efad0932aa5089ac8a6f6cfcf9f120d966 > core/src/test/java/org/apache/oozie/command/wf/TestSubmitMRXCommand.java > 5bc5747786774766e7156cb241d5444fefe6d7cb > core/src/test/java/org/apache/oozie/command/wf/TestSubmitPigXCommand.java > 5a1de25c85129e0b9567d0fd045133bffc700c5d > core/src/test/java/org/apache/oozie/command/wf/TestSubmitSqoopXCommand.java > 49b5028bcaff62e752a9582f1ddb5fb2ec56433c > > core/src/test/java/org/apache/oozie/command/wf/TestWorkflowActionKillXCommand.java > 72f01148ea457085f293689eda30509a1ca694d6 > core/src/test/java/org/apache/oozie/service/TestConfigurationService.java > 3c6525dc7e1049726eebafe056efdc49851ea62a > core/src/test/java/org/apache/oozie/service/TestHadoopAccessorService.java > bbca4794b693f865290adaa915d60019fc1218dd > core/src/test/java/org/apache/oozie/service/TestRecoveryService.java > 8fd0c2d5080b416470a8d886f69c7e3de8fa3ffc > core/src/test/java/org/apache/oozie/service/TestShareLibService.java > 2d31f9a86c25c15ee50bd420fb7300ee81a20835 > core/src/test/java/org/apache/oozie/test/XDataTestCase.java > ea778bd19f272876f1f8d38614ecba47649526ef > core/src/test/java/org/apache/oozie/test/XFsTestCase.java > 1d399e4ce921af8cb2cb2e6375d425dd5f7780f7 > core/src/test/java/org/apache/oozie/test/XTestCase.java > e1d9068f3ae0ee2ee01eb1e8f6e7164fd3c865b8 > core/src/test/java/org/apache/oozie/util/TestClasspathUtils.java > PRE-CREATION > > core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java > a3c7b142d08101e2bb03737871db939133bd3cd2 > distro/src/main/bin/addtowar.sh 399040936bde3d091d42557f6a64a0aee2b139e2 > docs/src/site/twiki/DG_QuickStart.twiki > 7ea87c7a8d6fdcb3d016b63851c9b75fa80515f1 > docs/src/site/twiki/ENG_Building.twiki > 535f1b04a4331409e0721bf9f1b4946e79443e10 > findbugs-filter.xml PRE-CREATION > hadooplibs/hadoop-auth-1/pom.xml 85e2298c25fc13bfc29ef8e0226d4eda2a95a073 > hadooplibs/hadoop-auth-2/pom.xml 7202b2e036bb49964435d089d9f9ff1e9237a7f7 > hadooplibs/hadoop-distcp-1/pom.xml bbc536b5901540e8f0835f75e185b7bb40e02e3c > hadooplibs/hadoop-distcp-2/pom.xml 2c21b12068ecc9782e2d12c4376ee16bfba16af0 > hadooplibs/hadoop-distcp-3/pom.xml 39cf9f26c71dc5a93bceb1f750de46002d2c7f80 > hadooplibs/hadoop-utils-1/pom.xml a53e9bca3024d924b7e9e79f2199d30b18506132 > > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > c0a237777f44e4d0a9061fc468af98cec2b92aa9 > > hadooplibs/hadoop-utils-1/src/main/java/org/apache/oozie/hadoop/utils/HadoopShims.java > 799dffbd7dc04654e72fa8de484143732f725fa1 > hadooplibs/hadoop-utils-2/pom.xml 7c4a0a10a56a79ce0513bfa2f790ffbbc5eb960a > > hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > a0b7d62c9432614f33baa3631660e6cb7918cd94 > > hadooplibs/hadoop-utils-2/src/main/java/org/apache/oozie/hadoop/utils/HadoopShims.java > c21884c7f94dab78be4a6dea20b698476479dc17 > hadooplibs/hadoop-utils-3/pom.xml 2f36f8bea921ad5ff74663728b377352daf0b2ca > > hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/action/hadoop/LauncherMainHadoopUtils.java > 0cf2e909283712f6e93653241f62ba601ae85706 > > hadooplibs/hadoop-utils-3/src/main/java/org/apache/oozie/hadoop/utils/HadoopShims.java > c21884c7f94dab78be4a6dea20b698476479dc17 > hadooplibs/pom.xml 06801cc6d8dcfe1fa21367c3bb26c3f8dcaf29ec > pom.xml c4a1c25295b264e6cae66db3a3c84fdde0109f13 > release-log.txt ccf8a125df15a1e0140a8c860eb9f0b75080efbb > sharelib/distcp/pom.xml 1f3c0cc6da4e996045fcce39e0e6477df2589f63 > > sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java > 65e7c5ec9eff5e5bb8cb943ca9a8dfc1d82f67d9 > sharelib/hcatalog/pom.xml 6eb88ef85e2f8c9398df64908e7777f251c1dbfa > sharelib/hive/pom.xml 13312192b634be802b3bc52726770d5c7b4057e9 > sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java > 6a600fa0621f5aee697c15d931568742ba5215da > > sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java > 12e1e91cf83a109d77b6a20b022dee851447525d > > sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java > 2ba0da7b1bc342d7d644a914763621940bd34532 > sharelib/hive2/pom.xml e81bfbe0a96a490b349a2660a0b00381d56785e1 > sharelib/hive2/src/main/java/org/apache/oozie/action/hadoop/Hive2Main.java > a3a07bdedfe311fbfb8ec2236d8a7d6db55e82d8 > > sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java > 4818bb6399a6c9d315af8fee7667db3691704840 > sharelib/oozie/pom.xml f3ea0716a6719c44916bb50255f5f31a873fb514 > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMCallBackHandler.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/AMRMClientAsyncFactory.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ActionUtils.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ErrorHolder.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/HdfsOperations.java > PRE-CREATION > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java > 30d68e2a55dd2e6b695c27ec18e5e47a48eb9322 > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAMCallbackNotifier.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java > 9a411acc2adf2907aae75fd02f4010aaec7cf8ef > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java > 8657c678c9786e1af20a2124214bc5106fe31223 > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LocalFsOperations.java > PRE-CREATION > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java > d376057f010dc359b44c6d4fe69ec0493261f037 > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java > 21ae456b768fb1cb5334a9e23e6c6082240f8dff > > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsHandler.java > PRE-CREATION > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java > f109318d949c68d4738e65de5a63de7c950a7888 > > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/LauncherAMTestMainClass.java > PRE-CREATION > > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestHdfsOperations.java > PRE-CREATION > > sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherAM.java > PRE-CREATION > sharelib/pig/pom.xml 99148d73e19a42531275789e4079e11a916bbaeb > sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java > 11cc7ee83af5b7d9f1ea0cc6b6331952711f35cc > > sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java > 503d0ebb8801d8de01a490bce5fd14e7947fb66b > > sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java > 25092ce55d8407384837b07d6d58b0218d1de7ab > > sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigMainWithOldAPI.java > e52e6fddd474da1bd3c4b03376b7b060df2ab3b5 > sharelib/spark/pom.xml abef24fe7f8a95f35347c2515385f9f582864559 > sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java > ffa934aa4d216ad57199cb30f625595228244444 > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestPyspark.java > 458baaa8b6f5bc4ec5dd8395d428a7e4215fb27c > > sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java > 8c77be0bdd7bf8cae07c4d4acd803df5730826e7 > sharelib/sqoop/pom.xml d5afa370b19ac3895e99c4bf594d3d32a4711d5b > sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java > 6672ffbc4460ade84f1a4090fd15217d4881e296 > > sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java > 3dfd606afb9a54791796cb3742998f35cec2f0ce > sharelib/streaming/pom.xml 4f732724846411f13bd0204cdc2bef198b44f0c6 > > sharelib/streaming/src/main/java/org/apache/oozie/action/hadoop/StreamingMain.java > 991bf7e849171cb7847fa6d64781ceb0782dae7c > > sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java > 5f9f38eb2c30edd234b2a523e5ca6c5c2f2ae832 > tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java > 9aa4cb6bf47a632b87b187882a8e105c40f38daf > webapp/pom.xml e4fdfb78589d76b0438e027b51dc563e11f1c58e > > > Diff: https://reviews.apache.org/r/54705/diff/10/ > > > Testing > ------- > > Junit tests are passing. > Submitted jobs to a hadoop cluster successfully. > > > Thanks, > > Peter Cseh > >
