[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935590#comment-15935590
 ] 

ASF GitHub Bot commented on TOREE-395:
--

Github user rdblue commented on a diff in the pull request:

https://github.com/apache/incubator-toree/pull/116#discussion_r107310980
  
--- Diff: 
scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
 ---
@@ -48,7 +48,7 @@ class ScalaInterpreter(private val config:Config = 
ConfigFactory.load) extends I
 
protected val lastResultOut = new ByteArrayOutputStream()
 
-   protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))
+   protected val multiOutputStream = MultiOutputStream(List(lastResultOut))
--- End diff --

I think the multiOutputStream can be completely removed rather than using 
it to wrap just one output stream. Instead, lastResultOut should be passed to 
IMain instead of multiOutputStream.


> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935406#comment-15935406
 ] 

ASF GitHub Bot commented on TOREE-395:
--

GitHub user kevin-bates opened a pull request:

https://github.com/apache/incubator-toree/pull/116

[TOREE-395] disable cell result output

This issue - discussed in 
[TOREE-395](https://issues.apache.org/jira/browse/TOREE-395) (and somewhat in 
[TOREE-380](https://issues.apache.org/jira/browse/TOREE-380)) - essentially 
takes the appropriate portion of PR #104 and delivers it here.  This change 
makes the scala interpreter behave like other interpreters (and kernels) by not 
issuing the cell results to the console (where it is typically captured by the 
Jupyter logging) and could contain sensitive information.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kevin-bates/incubator-toree 
remove-results-from-output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-toree/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit aee12381ce23fe7d399aa6488486a0f750027915
Author: Kevin Bates 
Date:   2017-03-21T21:45:43Z

[TOREE-395] disable cell result output




> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (TOREE-393) Upgrade assembly plugin to latest version (0.14.4)

[Luciano Resende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TOREE-393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luciano Resende resolved TOREE-393.
---
Resolution: Fixed

> Upgrade assembly plugin to latest version (0.14.4) 
> ---
>
> Key: TOREE-393
> URL: https://issues.apache.org/jira/browse/TOREE-393
> Project: TOREE
>  Issue Type: Bug
>Reporter: Luciano Resende
>Assignee: Luciano Resende
> Fix For: 0.2.0
>
>
> Trying to produce a shaded jar throws an exception with the current version 
> of assembly plugin.
> [info] Including: classutil_2.11-1.0.12.jar
> [info] Including: akka-slf4j_2.11-2.4.17.jar
> [info] Including: asm-5.1.jar
> [info] Including: scalaz-core_2.11-7.2.7.jar
> [info] Including: commons-logging-1.1.3.jar
> [info] Including: config-1.3.0.jar
> java.util.NoSuchElementException: None.get
>   at scala.None$.get(Option.scala:313)
>   at scala.None$.get(Option.scala:311)
>   at 
> sbtassembly.Assembly$$anonfun$24$$anonfun$25.apply(Assembly.scala:205)
>   at 
> sbtassembly.Assembly$$anonfun$24$$anonfun$25.apply(Assembly.scala:205)
>   at 
> scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
>   at scala.collection.immutable.List.foreach(List.scala:318)
>   at 
> scala.collection.TraversableLike$class.filter(TraversableLike.scala:263)
>   at scala.collection.AbstractTraversable.filter(Traversable.scala:105)
>   at sbtassembly.Assembly$$anonfun$24.apply(Assembly.scala:205)
>   at sbtassembly.Assembly$$anonfun$24.apply(Assembly.scala:202)
>   at 
> scala.collection.parallel.AugmentedIterableIterator$class.map2combiner(RemainsIterator.scala:120)
>   at 
> scala.collection.parallel.immutable.ParVector$ParVectorIterator.map2combiner(ParVector.scala:67)
>   at 
> scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1057)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
>   at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
>   at 
> scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1054)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.internal(Tasks.scala:172)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:514)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:162)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
>   at 
> scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at scala.concurrent.forkjoin.ForkJoinTask.doJoin(ForkJoinTask.java:341)
>   at scala.concurrent.forkjoin.ForkJoinTask.join(ForkJoinTask.java:673)
>   at 
> scala.collection.parallel.ForkJoinTasks$WrappedTask$class.sync(Tasks.scala:444)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:514)
>   at 
> scala.collection.parallel.ForkJoinTasks$class.executeAndWaitResult(Tasks.scala:492)
>   at 
> scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:64)
>   at 
> scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:961)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:54)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
>   at 
> scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:53)
>   at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:56)
>   at 
> scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:956)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:165)
>   at 
> scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:514)
>   at 
> scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> [error] (root/*:assembly::assembledMappings) 
> java.util.NoSuchElementException: None.get
> [error] Total time: 56 s, completed Mar 

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935254#comment-15935254
 ] 

Ryan Blue commented on TOREE-395:
-

Yeah, please do.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: CI Builds

[Luciano Resende]

The Zeppelin folks seem to have a neat way to gather a bunch of files when
the build fails:

https://github.com/apache/zeppelin/blob/master/.travis.yml

Maybe we could do the same?


On Tue, Mar 21, 2017 at 11:27 AM, Corey Stubbs  wrote:

> I know there have been flaky tests in the past due to testing akka actors
> and dealing with the concurrency for those classes. My best guess is that
> current issues relate to similar timing issues.
>
> It might be worth having some record of all the test runs so we can see
> which ones are consistently. Anybody know of a way to do this with travis?
> It basically would entail publishing the test report to some static host.
>
> On Fri, Mar 17, 2017 at 8:18 PM Luciano Resende 
> wrote:
>
> > The current CI builds seems a little flaky, and will usually fail
> unrelated
> > to the changes being proposed. Has anyone investigated these issues and
> has
> > an understanding of what causes them? Is it related to Travis.CI? The
> > actual tests?  Timing?
> >
> > Anyway, we should make the CI build more stable.
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: CI Builds

[Corey Stubbs]

I know there have been flaky tests in the past due to testing akka actors
and dealing with the concurrency for those classes. My best guess is that
current issues relate to similar timing issues.

It might be worth having some record of all the test runs so we can see
which ones are consistently. Anybody know of a way to do this with travis?
It basically would entail publishing the test report to some static host.

On Fri, Mar 17, 2017 at 8:18 PM Luciano Resende 
wrote:

> The current CI builds seems a little flaky, and will usually fail unrelated
> to the changes being proposed. Has anyone investigated these issues and has
> an understanding of what causes them? Is it related to Travis.CI? The
> actual tests?  Timing?
>
> Anyway, we should make the CI build more stable.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Kevin Bates (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934841#comment-15934841
 ] 

Kevin Bates commented on TOREE-395:
---

This is great news Ryan - thanks for the update.  I apologize for not taking a 
closer look at the referenced PR.  The more I thought about things and realized 
that the other interpreters (Toree and otherwise) didn't produce this output, 
the more I leaned to removing this capability entirely.

When do you anticipate this PR getting completed?

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934825#comment-15934825
 ] 

Ryan Blue commented on TOREE-395:
-

Thanks for continuing the discussion on this, Kevin. A few weeks ago, I posted 
a [WIP pull request|https://github.com/apache/incubator-toree/pull/104] with 
what we're currently using that you might be interested in. That addresses a 
few issues, including this one. The approach we took was to get rid of 
multiOutputStream and just use lastResultOut. Normal stderr and stdout printing 
happens just fine, so I think that the only output that gets printed the writer 
passed to IMain is the summary of what ran (e.g. import scala...). Your idea to 
conditionally print the output to the logger should work fine, but is it really 
necessary for people to see this output in the logs? I see no reason to keep it 
this way.

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (TOREE-395) Provide a way to disable automatic printing of results in Toree Scala

[Kevin Bates (JIRA)]

[ 
https://issues.apache.org/jira/browse/TOREE-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933729#comment-15933729
 ] 

Kevin Bates edited comment on TOREE-395 at 3/21/17 3:20 PM:


Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala 
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))}}
to
protected val multiOutputStream = MultiOutputStream(List(new 
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))

appears to do the trick.  My concern is that it would require developers to 
enable debug in order to see the output they see today.  However, I also feel 
its best to error on the side of not logging potentially sensitive information 
by default.

If we took this approach, we could add the following lines to log4j.properties 
so that only the scala interpreter could be enabled for debug.

{{# Change the following to DEBUG to see interpreter results in log}}
{{log4j.logger.org.apache.toree.kernel.interpreter.scala=INFO}}


Other options are to add an interpreter argument to indicate that scala 
interpreter results should be produced (or should be silent - depending on what 
the default should be) or remove the code to post to the console altogether 
(and match the other interpreters).


was (Author: kbates):
Changing the initialization of {{multiOutputStream}} in ScalInterpreter.scala 
from:
{{protected val multiOutputStream = MultiOutputStream(List(Console.out, 
lastResultOut))}}
to
protected val multiOutputStream = MultiOutputStream(List(new 
ConditionalOutputStream(Console.out, logger.isDebugEnabled), lastResultOut))

appears to do the trick.  My concern is that it would require developers to 
enable debug in order to see the output they see today.  However, I also feel 
its best to error on the side of not logging potentially sensitive information 
by default.

If we took this approach, we could add the following lines to log4j.properties 
so that only the scala interpreter could be enabled for debug.

{{# Change the following to DEBUG to see interpreter results in log}}
{{log4j.logger.org.apache.toree.kernel.interpreter.scala=INFO}}


Another option is to add an interpreter argument to indicate that scala 
interpreter results should be produced (or should be silent - depending on what 
the default should be).

> Provide a way to disable automatic printing of results in Toree Scala
> -
>
> Key: TOREE-395
> URL: https://issues.apache.org/jira/browse/TOREE-395
> Project: TOREE
>  Issue Type: Improvement
>Affects Versions: 0.2.0
>Reporter: Kun Liu
>
> Scala REPL supports a flag, ":silent" to enable/disable automatic printing of 
> results (those messages begin with "res"). This is also supported in Spark 
> shell. But for Toree Scala kernel, this flag is not supported. Thus the 
> result of a cell would be always printed.
> But if there is any logging mechanism, the results would be recorded in a log 
> file, while this may not be desirable. For instance, a user may not want any 
> sensitive data logged when running "dataRDD.take(5)" in a cell.
> Also found the possible source codes for this:
> https://github.com/apache/incubator-toree/blob/master/scala-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/scala/ScalaInterpreter.scala
> There are three methods with silent: Boolean = false as parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)