[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883443#comment-15883443 ] Robert Kanter commented on OOZIE-2803: -- +1 > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882821#comment-15882821 ] Hadoop QA commented on OOZIE-2803: -- Testing JIRA OOZIE-2803 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:red}-1{color} the patch contains 11 line(s) longer than 132 characters .{color:green}+1{color} the patch does adds/modifies 3 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:orange}0{color} There are [1] new bugs found in total that would be nice to have fixed. .{color:green}+1{color} There are no new bugs found in [server]. .{color:green}+1{color} There are no new bugs found in [client]. .{color:green}+1{color} There are no new bugs found in [docs]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive]. .{color:green}+1{color} There are no new bugs found in [sharelib/spark]. .{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive2]. .{color:green}+1{color} There are no new bugs found in [sharelib/streaming]. .{color:green}+1{color} There are no new bugs found in [sharelib/pig]. .{color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. .{color:green}+1{color} There are no new bugs found in [sharelib/distcp]. .{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] that would be nice to have fixed. .You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html .{color:green}+1{color} There are no new bugs found in [hadooplibs/hadoop-utils-2]. .{color:green}+1{color} There are no new bugs found in [core]. .{color:green}+1{color} There are no new bugs found in [tools]. .{color:green}+1{color} There are no new bugs found in [examples]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 1886 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3658/ > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it does
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882579#comment-15882579 ] Peter Bacsko commented on OOZIE-2803: - There are no findbugs warnings, Raw patch analysis complains about lines that are too long. But those are just the csv files that contain test data. I changed {{MapReduceMain}} and {{PasswordMasker}} as you requested. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch, OOZIE-2803-004.patch, OOZIE-2803-005.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881715#comment-15881715 ] Robert Kanter commented on OOZIE-2803: -- Looks good overall. A few trivial things: - Can you look into the findbugs warning? - When creating the new {{JobConf}} in {{MapReduceMain}}, I think it would be better to pass {{false}} to make sure it doesn't load any other properties if there's some site/default file on the classpath somehow. All of the properties will be copied in there anyway. {code:java} JobConf maskedJobConf = new JobConf(false); {code} - Shouldn't {{PASSWORD_EXTRACTING_REGEX}} contain a reference to {{PASSWORD_KEY}} instead of hardcoding "pass" there? I can't imagine we'd ever change {{PASSWORD_KEY}}, but that would be more future-proof. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch, OOZIE-2803-004.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878368#comment-15878368 ] Hadoop QA commented on OOZIE-2803: -- Testing JIRA OOZIE-2803 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:red}-1{color} the patch contains 11 line(s) longer than 132 characters .{color:green}+1{color} the patch does adds/modifies 3 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:orange}0{color} There are [1] new bugs found in total that would be nice to have fixed. .{color:green}+1{color} There are no new bugs found in [server]. .{color:green}+1{color} There are no new bugs found in [client]. .{color:green}+1{color} There are no new bugs found in [core]. .{color:green}+1{color} There are no new bugs found in [docs]. .{color:green}+1{color} There are no new bugs found in [hadooplibs/hadoop-utils-2]. .{color:green}+1{color} There are no new bugs found in [tools]. .{color:green}+1{color} There are no new bugs found in [examples]. .{color:green}+1{color} There are no new bugs found in [sharelib/streaming]. .{color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. .{color:green}+1{color} There are no new bugs found in [sharelib/distcp]. .{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] that would be nice to have fixed. .You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html .{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive2]. .{color:green}+1{color} There are no new bugs found in [sharelib/pig]. .{color:green}+1{color} There are no new bugs found in [sharelib/spark]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 1886 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3654/ > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch, OOZIE-2803-004.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878103#comment-15878103 ] Abhishek Bafna commented on OOZIE-2803: --- [~pbacsko] Thanks for pointing to the previous discussion. I think we can keep it same. For the JavaDoc: {{@param String}} and {{@param Input}} there should be parameter names, which is missing. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878078#comment-15878078 ] Peter Bacsko commented on OOZIE-2803: - Thanks for the comment [~abhishekbafna]. 1. What's wrong with the JavaDoc? I think it gives an accurate description of what the methods are doing. Are there spelling or grammatical errors? 2. As for PasswordMasker not being an utility class - I completely agree with you, originally, [~andras.piros] did not want to write static methods, his reasoning was the following: https://issues.apache.org/jira/browse/OOZIE-1814?focusedCommentId=15528763&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15528763 Nevertheless I can make it static, I don't think it's a big deal :) > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877556#comment-15877556 ] Abhishek Bafna commented on OOZIE-2803: --- Few points: There are some unused imports in the {{MapReduceMain}}. Fix the java doc for {{PasswordMasker:maskPasswordsIfNecessary}} and {{PasswordMasker:mask}}. All the field members of {{PasswordMasker}} are declared final and class in itself does not have any state. it can be converted into Utility Class. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876469#comment-15876469 ] Hadoop QA commented on OOZIE-2803: -- Testing JIRA OOZIE-2803 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:red}-1{color} the patch contains 11 line(s) longer than 132 characters .{color:green}+1{color} the patch does adds/modifies 3 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:orange}0{color} There are [1] new bugs found in total that would be nice to have fixed. .{color:green}+1{color} There are no new bugs found in [server]. .{color:green}+1{color} There are no new bugs found in [client]. .{color:green}+1{color} There are no new bugs found in [core]. .{color:green}+1{color} There are no new bugs found in [docs]. .{color:green}+1{color} There are no new bugs found in [hadooplibs/hadoop-utils-2]. .{color:green}+1{color} There are no new bugs found in [tools]. .{color:green}+1{color} There are no new bugs found in [examples]. .{color:green}+1{color} There are no new bugs found in [sharelib/streaming]. .{color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. .{color:green}+1{color} There are no new bugs found in [sharelib/distcp]. .{color:orange}0{color} There are [1] new bugs found in [sharelib/oozie] that would be nice to have fixed. .You can find the FindBugs diff here: sharelib/oozie/findbugs-new.html .{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive2]. .{color:green}+1{color} There are no new bugs found in [sharelib/pig]. .{color:green}+1{color} There are no new bugs found in [sharelib/spark]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 1886 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3653/ > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different a
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876259#comment-15876259 ] Peter Bacsko commented on OOZIE-2803: - Fixing Findbugs problems. The long lines (> 132 chars) occur only in the csv files, so this warning can be ignored. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch, > OOZIE-2803-003.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876043#comment-15876043 ] Hadoop QA commented on OOZIE-2803: -- Testing JIRA OOZIE-2803 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:red}-1{color} the patch contains 12 line(s) longer than 132 characters .{color:green}+1{color} the patch does adds/modifies 3 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} the patch does not seem to introduce new Javadoc warnings {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:red}-1{color} There are [2] new bugs found below threshold in total that must be fixed. .{color:green}+1{color} There are no new bugs found in [server]. .{color:green}+1{color} There are no new bugs found in [client]. .{color:green}+1{color} There are no new bugs found in [core]. .{color:green}+1{color} There are no new bugs found in [docs]. .{color:green}+1{color} There are no new bugs found in [hadooplibs/hadoop-utils-2]. .{color:green}+1{color} There are no new bugs found in [tools]. .{color:green}+1{color} There are no new bugs found in [examples]. .{color:green}+1{color} There are no new bugs found in [sharelib/streaming]. .{color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. .{color:green}+1{color} There are no new bugs found in [sharelib/distcp]. .{color:red}-1{color} There are [2] new bugs found below threshold in [sharelib/oozie] that must be fixed. .You can find the FindBugs diff here (look for the red and orange ones): sharelib/oozie/findbugs-new.html .The most important FindBugs errors are: .At PasswordMasker.java:[lines 85-92]: key must be nonnull but is marked as nullable .At PasswordMasker.java:[lines 85-92]: value must be nonnull but is marked as nullable .{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive]. .{color:green}+1{color} There are no new bugs found in [sharelib/hive2]. .{color:green}+1{color} There are no new bugs found in [sharelib/pig]. .{color:green}+1{color} There are no new bugs found in [sharelib/spark]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 1886 {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/3652/ > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch, OOZIE-2803-002.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs(
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874508#comment-15874508 ] Peter Bacsko commented on OOZIE-2803: - I changed the approach - after talking with [~andras.piros] and [~gezapeti], I decided to re-use {{PasswordMasker}}. I had to enhance it a little bit and move it from {{oozie-core}} to {{sharelib-oozie}} but this solution is better. I also added extra tests. > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: OOZIE-2803-001.patch > > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (OOZIE-2803) Mask passwords when printing out configs/args in MapReduceMain and SparkMain
[ https://issues.apache.org/jira/browse/OOZIE-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871796#comment-15871796 ] Peter Bacsko commented on OOZIE-2803: - In MapReduce, we have to mask the following props (if they contain a password): {{mapred.child.env}} {{yarn.app.mapreduce.am.env}} {{oozie.launcher.mapred.child.env}} > Mask passwords when printing out configs/args in MapReduceMain and SparkMain > > > Key: OOZIE-2803 > URL: https://issues.apache.org/jira/browse/OOZIE-2803 > Project: Oozie > Issue Type: Bug > Components: action >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > > Sometimes passwords are displayed in both MapReduce and Spark action. > *MapReduce*: when using {{HADOOP_CREDSTORE_PASSWORD}}, it must be passed to > some Hadoop-specific config values, like {{mapred.child.env}}. This is easy > to fix because we already have a method {{logMasking()}} where you can define > a {{maskSet}} which contains a list of property keys to be masked. > Note that this is not necessarily the perfect solution, since you can pass > multiple env. vars separated by a colon, and only the password specific parts > should be masked. But we need a working solution relatively quickly - later > we can enhance this, eg. we can re-use {{PasswordMasker}} in some way (right > now it only works with {{Map}}). > *Spark*: for Spark, we have to pass passwords like this: > {{--conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD= password>}} > The Spark arguments are printed in {{SparkMain.run()}}. There is already a > code in {{LauncherMapper.printArgs()}} which deals with situations like this, > but it's not perfect because it only works if the args look something like > {{--password pwd123}}. So if a single arg contains a password, it doesn't > work, therefore we need a different approach here. -- This message was sent by Atlassian JIRA (v6.3.15#6346)