subject:"\[GitHub\] spark issue #17725\: \[SPARK\-20435\]\[CORE\] More thorough redaction of sensitive..."

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Thanks Marcelo!

On Apr 26, 2017 5:06 PM, "Marcelo Vanzin"  wrote:

> Merging to master / 2.2.
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-26 Thread vanzin

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merging to master / 2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Finally it's passing!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76194/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76194 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76194/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76194 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76194/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
This time it finished on regular time (2h 29m) but failed a test. I ran 
that test locally (org.apache.spark.streaming.ReceiverSuite's 'receiver life 
cycle') and it passed for me, so for mostly laziness, I am going to issue 
another run here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76156/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76156 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76156/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76156 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76156/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-25 Thread markgrover

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Kicking off another run here while I am running a run locally.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-25 Thread markgrover

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-25 Thread markgrover

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Hmm, this is weird, it ran for 4h10 mins and got killed due to a timeout. 
Looking...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76143/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76143 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76143/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76143 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76143/testReport)**
 for PR 17725 at commit 
[`80e40ba`](https://github.com/apache/spark/commit/80e40ba57ab6779604fca87cb696d2e889c4ddd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-24 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76114/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-24 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-24 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76114 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76114/testReport)**
 for PR 17725 at commit 
[`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-24 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76114 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76114/testReport)**
 for PR 17725 at commit 
[`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-24 Thread markgrover

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-22 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76055/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-22 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76054/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76054 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76054/testReport)**
 for PR 17725 at commit 
[`0603686`](https://github.com/apache/spark/commit/06036867d96350a51e180565782ffee1515fbea4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76055 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76055/testReport)**
 for PR 17725 at commit 
[`14b0d72`](https://github.com/apache/spark/commit/14b0d72d5fffb69694a2442ade6399161f99545c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

2017-04-21 Thread markgrover

Github user markgrover commented on the issue:

https://github.com/apache/spark/pull/17725

Few decisions that I made here:
* I considered if just `sun.java.command` property should be checked and
redacted but that seemed to specific and likely a bandaid to the current
problem, not a long-term solution, so decided against doing it.
* Redaction for the `SparkListenerEnvironmentUpdate` event was solely being
done on `Spark Properties`, while `sun.java.command` is a part of `System
Properties`. I considered doing redaction for `System Properties` in addition
to `Spark Properties` (that would have gone somewhere around
[here](https://github.com/apache/spark/pull/17725/files#diff-e4a5a68c15eed95d038acfed84b0b66aL258))
but decided against it because that would have even more hardcoding and I
didn't see why these 2 special kinds of properties are special enough to be
redacted but the rest of them. So, decided to redact information from all kinds
of properties.
* One way to redact the property value would have been to redact the
minimum possible set from the value while keeping the rest of the value intact.
For example, if the following were the unredacted case:
`"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf
spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password --conf
spark.other.property=2"`
One option for the redacted output could have been:
`"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf
spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=*(redacted) --conf
spark.other.property=2"`
However, such a redaction is very hard to maintain. For example, we would
had to take the current regex (which is `(?i)secret|password` by default and
add matchers to it like so `(?i)secret|password` like
`"("+SECRET_REDACTION_DEFAULT+"[^ ]*=)[^ ]*"`. That would allow us to squeeze
out and replaced just the matched portion. But this all seemed very fragile and
even worse when the user supplies a non-default regex so I decided it was
easiest to simply replace the entire value, even though only a small part of it
contained `secret` or `password` in it.
* One thing which I didn't explicitly check was the performance
implications of this change. The reason I bring this up is because, previously
we were comparing keys with a regex, now if the key
doesn't match, we match the value with the regex. So, in the worst case, we
are twice as many regex matches as before. Also, before we were simply doing
regex matching on `Spark Properties`, now we do them on all properties - `Spark
Properties`, `System Properties`, `JVM Properties` and `Classpath Properties`.
I don't think this should have a big performance impact so I didn't invest time
in it, mentioning here in interest of full disclosure.

Thanks in advance for reviewing.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76054 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76054/testReport)**
 for PR 17725 at commit 
[`0603686`](https://github.com/apache/spark/commit/06036867d96350a51e180565782ffee1515fbea4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17725
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76051/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17725
  
**[Test build #76051 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76051/testReport)**
 for PR 17725 at commit 
[`2f5148a`](https://github.com/apache/spark/commit/2f5148a2e37d8d36006fb297b28a9e8c21a0026b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17725: [SPARK-20435][CORE] More thorough redaction of sensitive...