Pasting oddity with Spark 2.0 (scala)

[jggg777]

This one has stumped the group here, hoping to get some insight into why this
error is happening.

I'm going through the  Databricks DataFrames scala docs
<https://docs.cloud.databricks.com/docs/latest/databricks_guide/index.html#04%20SQL,%20DataFrames%20%26%20Datasets/02%20Introduction%20to%20DataFrames%20-%20scala.html>
 
.  Halfway down is the "Flattening" code, which I've copied below:

*The original code*

>>>
import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

implicit class DataFrameFlattener(df: DataFrame) {
  def flattenSchema: DataFrame = {
    df.select(flatten(Nil, df.schema): _*)
  }
  
  protected def flatten(path: Seq[String], schema: DataType): Seq[Column] =
schema match {
    case s: StructType => s.fields.flatMap(f => flatten(path :+ f.name,
f.dataType))
    case other => col(path.map(n =>
s"`$n`").mkString(".")).as(path.mkString(".")) :: Nil
  }
}
>>>

*Pasting into spark-shell with right-click (or pasting into Zeppelin)*


On EMR using Spark 2.0.0 (this also happens on Spark 2.0.1), running
"spark-shell", I right click to paste in the code above.  Here are the
errors I get.  Note that I get the same errors when I paste into Zeppelin on
EMR.

>>>
scala> import org.apache.spark.sql._
import org.apache.spark.sql._

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._

scala>

scala> implicit class DataFrameFlattener(df: DataFrame) {
     |   def flattenSchema: DataFrame = {
     |     df.select(flatten(Nil, df.schema): _*)
     |   }
     |
     |   protected def flatten(path: Seq[String], schema: DataType):
Seq[Column] = schema match {
     |     case s: StructType => s.fields.flatMap(f => flatten(path :+
f.name, f.dataType))
     |     case other => col(path.map(n =>
s"`$n`").mkString(".")).as(path.mkString(".")) :: Nil
     |   }
     | }
<console>:11: error: not found: type DataFrame
       implicit class DataFrameFlattener(df: DataFrame) {
                                             ^
<console>:12: error: not found: type DataFrame
         def flattenSchema: DataFrame = {
                            ^
<console>:16: error: not found: type Column
         protected def flatten(path: Seq[String], schema: DataType):
Seq[Column] = schema match {
                                                                         ^
<console>:16: error: not found: type DataType
         protected def flatten(path: Seq[String], schema: DataType):
Seq[Column] = schema match {
                                                          ^
<console>:17: error: not found: type StructType
           case s: StructType => s.fields.flatMap(f => flatten(path :+
f.name, f.dataType))
                   ^
<console>:18: error: not found: value col
           case other => col(path.map(n =>
s"`$n`").mkString(".")).as(path.mkString(".")) :: Nil
>>>

*Pasting using :paste in spark-shell*


However when I paste the same code into spark-shell using :paste, the code
succeeds.

>>>
scala> :paste
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

implicit class DataFrameFlattener(df: DataFrame) {
  def flattenSchema: DataFrame = {
    df.select(flatten(Nil, df.schema): _*)
  }

  protected def flatten(path: Seq[String], schema: DataType): Seq[Column] =
schema match {
    case s: StructType => s.fields.flatMap(f => flatten(path :+ f.name,
f.dataType))
    case other => col(path.map(n =>
s"`$n`").mkString(".")).as(path.mkString(".")) :: Nil
  }
}

// Exiting paste mode, now interpreting.

import org.apache.spark.sql._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
defined class DataFrameFlattener
>>>


Any idea what's going on here, and how to get this code working in Zeppelin? 
One thing we've found is that providing the full paths for DataFrame,
StructType, etc (for example org.apache.spark.sql.DataFrame) does work, but
it's a painful workaround and we don't know why the imports don't seem to be
working as usual.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Pasting-oddity-with-Spark-2-0-scala-tp28071.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org