*I followed the procedures instructed by
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IntelliJ.
But problems still occurs which has made me a little bit annoyed.
My environment settings are:JAVA 1.7.0 Scala: 2.10.4 Spark:1.2.0, Intellij
Idea 14.0.2, Ubuntu 14.04
Firstly I got the scala plugin correctly installed.
I choosed maven-3, hadoop-2.4, scala-2.10 as my profiles when importing the
project.
After importing, I first turned on "View-Tool Windows-Maven Projects". I see
the "hbase-hadoop1" is selected, but I had not chosen it in the import
process. So I deselected it to leave the hadoop-2.4, maven-3, scala-2.10 to
be the only three selected items in "Maven Projects-Profiles".
According to the Wiki, the next step should be "Generate Sources and Update
Folders For All Projects". I did so, and waited for some minutes to get the
sub-projects prepared.
Then I cleared the "Additional compiler options" in the
"File-Settings-Build, Execution, Deployment-Compiler-Scala Compiler".
Finally I choosed Build-reBuild project.
However, the compiler failed with "value q is not a member of stringcontext"
errors. *
(partial screen shot)
-------------------------------------------------------
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateProjection.scala
Error:(42, 21) value q is not a member of StringContext
val lengthDef = q"final val length = $tupleLength"
^
Error:(54, 7) value q is not a member of StringContext
q"""
^
Error:(66, 9) value q is not a member of StringContext
q"""
^
Error:(83, 9) value q is not a member of StringContext
q"if(isNullAt($iLit)) { null } else { ${newTermName(s"c$i")} }"
^
Error:(85, 7) value q is not a member of StringContext
q"override def iterator = Iterator[Any](..$allColumns)"
^
Error:(88, 27) value q is not a member of StringContext
val accessorFailure = q"""scala.sys.error("Invalid ordinal:" + i)"""
^
Error:(95, 9) value q is not a member of StringContext
q"if(i == $ordinal) { if(isNullAt($i)) return null else return
$elementName }"
^
Error:(97, 7) value q is not a member of StringContext
q"override def apply(i: Int): Any = { ..$cases; $accessorFailure }"
^
Error:(106, 9) value q is not a member of StringContext
q"""
^
Error:(117, 7) value q is not a member of StringContext
q"override def update(i: Int, value: Any): Unit = { ..$cases;
$accessorFailure }"
^
Error:(126, 11) value q is not a member of StringContext
q"if(i == $i) return $elementName" :: Nil
^
Error:(130, 7) value q is not a member of StringContext
q"""
^
Error:(143, 11) value q is not a member of StringContext
q"if(i == $i) { nullBits($i) = false; $elementName = value; return
}" :: Nil
^
Error:(147, 7) value q is not a member of StringContext
q"""
^
Error:(157, 29) value q is not a member of StringContext
case BooleanType => q"if ($elementName) 0 else 1"
^
Error:(158, 52) value q is not a member of StringContext
case ByteType | ShortType | IntegerType => q"$elementName.toInt"
^
Error:(159, 26) value q is not a member of StringContext
case LongType => q"($elementName ^ ($elementName >>> 32)).toInt"
^
Error:(160, 27) value q is not a member of StringContext
case FloatType => q"java.lang.Float.floatToIntBits($elementName)"
^
Error:(162, 11) value q is not a member of StringContext
q"{ val b = java.lang.Double.doubleToLongBits($elementName); (b ^
(b >>>32)).toInt }"
^
Error:(163, 19) value q is not a member of StringContext
case _ => q"$elementName.hashCode"
^
Error:(165, 7) value q is not a member of StringContext
q"if (isNullAt($i)) 0 else $nonNull"
^
Error:(168, 54) value q is not a member of StringContext
val hashUpdates: Seq[Tree] = hashValues.map(v => q"""result = 37 *
result + $v""": Tree)
^
Error:(171, 7) value q is not a member of StringContext
q"""
^
Error:(181, 7) value q is not a member of StringContext
q"if (this.$elementName != specificType.$elementName) return false"
^
Error:(185, 7) value q is not a member of StringContext
q"""
^
Error:(195, 7) value q is not a member of StringContext
q"""
^
Error:(210, 16) value q is not a member of StringContext
val code = q"""
^
---------------------------------------------------------------------
*
Then I tried
http://stackoverflow.com/questions/26995023/errorscalac-bad-option-p-intellij-idea,
which does not clear the "Additional compiler options", but change the -P in
to -Xplugin.
So now my "Additional Compiler Options" is like this
"-Xplugin:/home/yaochunnan/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar"
Then I reBuild again with the following errors: *
(partial screen shot)
----------------------------------------------------------------------
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
Error:(169, 38) not found: value HiveShim
Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
^
Error:(177, 31) not found: value HiveShim
tableParameters.put(HiveShim.getStatsSetupConstTotalSize,
newTotalSize.toString)
^
Error:(292, 36) not found: value HiveShim
val proc: CommandProcessor =
HiveShim.getCommandProcessor(Array(tokens(0)), hiveconf)
^
Error:(304, 25) not found: value HiveShim
val results = HiveShim.createDriverResultsArray
^
Error:(314, 11) not found: value HiveShim
HiveShim.processResults(results)
^
Error:(418, 7) not found: value HiveShim
HiveShim.createDecimal(decimal.toBigDecimal.underlying()).toString
^
Error:(420, 7) not found: value HiveShim
HiveShim.createDecimal(decimal.underlying()).toString
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
Error:(97, 7) not found: value HiveShim
HiveShim.toCatalystDecimal(
^
Error:(123, 46) not found: value HiveShim
case hdoi: HiveDecimalObjectInspector =>
HiveShim.toCatalystDecimal(hdoi, data)
^
Error:(156, 19) not found: value HiveShim
(o: Any) =>
HiveShim.createDecimal(o.asInstanceOf[Decimal].toBigDecimal.underlying())
^
Error:(210, 31) not found: value HiveShim
case b: BigDecimal => HiveShim.createDecimal(b.underlying())
^
Error:(211, 28) not found: value HiveShim
case d: Decimal =>
HiveShim.createDecimal(d.toBigDecimal.underlying())
^
Error:(283, 7) not found: value HiveShim
HiveShim.getStringWritableConstantObjectInspector(value)
^
Error:(285, 7) not found: value HiveShim
HiveShim.getIntWritableConstantObjectInspector(value)
^
Error:(287, 7) not found: value HiveShim
HiveShim.getDoubleWritableConstantObjectInspector(value)
^
Error:(289, 7) not found: value HiveShim
HiveShim.getBooleanWritableConstantObjectInspector(value)
^
Error:(291, 7) not found: value HiveShim
HiveShim.getLongWritableConstantObjectInspector(value)
^
Error:(293, 7) not found: value HiveShim
HiveShim.getFloatWritableConstantObjectInspector(value)
^
Error:(295, 7) not found: value HiveShim
HiveShim.getShortWritableConstantObjectInspector(value)
^
Error:(297, 7) not found: value HiveShim
HiveShim.getByteWritableConstantObjectInspector(value)
^
Error:(299, 7) not found: value HiveShim
HiveShim.getBinaryWritableConstantObjectInspector(value)
^
Error:(301, 7) not found: value HiveShim
HiveShim.getDateWritableConstantObjectInspector(value)
^
Error:(303, 7) not found: value HiveShim
HiveShim.getTimestampWritableConstantObjectInspector(value)
^
Error:(305, 7) not found: value HiveShim
HiveShim.getDecimalWritableConstantObjectInspector(value)
^
Error:(307, 7) not found: value HiveShim
HiveShim.getPrimitiveNullWritableConstantObjectInspector
^
Error:(363, 51) not found: value HiveShim
case w: WritableHiveDecimalObjectInspector =>
HiveShim.decimalTypeInfoToCatalyst(w)
^
Error:(364, 47) not found: value HiveShim
case j: JavaHiveDecimalObjectInspector =>
HiveShim.decimalTypeInfoToCatalyst(j)
^
Error:(393, 30) not found: value HiveShim
case d: DecimalType => HiveShim.decimalTypeInfo(d)
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
Error:(78, 11) not found: value HiveShim
HiveShim.getAllPartitionsOf(client, table).toSeq
^
Error:(205, 7) not found: value HiveShim
HiveShim.setLocation(tbl, crtTbl)
^
Error:(443, 28) not found: value HiveShim
case d: DecimalType => HiveShim.decimalMetastoreString(d)
^
Error:(472, 53) not found: value HiveShim
val totalSize =
hiveQlTable.getParameters.get(HiveShim.getStatsSetupConstTotalSize)
^
Error:(490, 19) not found: value HiveShim
val tableDesc = HiveShim.getTableDesc(
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala
Error:(255, 18) not found: type HiveFunctionWrapper
funcWrapper: HiveFunctionWrapper,
^
Error:(73, 53) not found: type HiveFunctionWrapper
private[hive] case class HiveSimpleUdf(funcWrapper: HiveFunctionWrapper,
children: Seq[Expression])
^
Error:(57, 25) not found: type HiveFunctionWrapper
HiveSimpleUdf(new HiveFunctionWrapper(functionClassName), children)
^
Error:(134, 54) not found: type HiveFunctionWrapper
private[hive] case class HiveGenericUdf(funcWrapper: HiveFunctionWrapper,
children: Seq[Expression])
^
Error:(59, 26) not found: type HiveFunctionWrapper
HiveGenericUdf(new HiveFunctionWrapper(functionClassName), children)
^
Error:(185, 18) not found: type HiveFunctionWrapper
funcWrapper: HiveFunctionWrapper,
^
Error:(62, 27) not found: type HiveFunctionWrapper
HiveGenericUdaf(new HiveFunctionWrapper(functionClassName), children)
^
Error:(214, 18) not found: type HiveFunctionWrapper
funcWrapper: HiveFunctionWrapper,
^
Error:(64, 20) not found: type HiveFunctionWrapper
HiveUdaf(new HiveFunctionWrapper(functionClassName), children)
^
Error:(66, 27) not found: type HiveFunctionWrapper
HiveGenericUdtf(new HiveFunctionWrapper(functionClassName), Nil,
children)
^
Error:(322, 18) not found: type HiveFunctionWrapper
funcWrapper: HiveFunctionWrapper,
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
Error:(1132, 15) not found: type HiveFunctionWrapper
new HiveFunctionWrapper(functionName),
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
Error:(44, 34) object HiveShim is not a member of package
org.apache.spark.sql.hive
import org.apache.spark.sql.hive.HiveShim._
^
Error:(43, 8) object ShimFileSinkDesc is not a member of package
org.apache.spark.sql.hive
import org.apache.spark.sql.hive.{ ShimFileSinkDesc => FileSinkDesc}
^
Error:(76, 21) not found: type FileSinkDesc
fileSinkConf: FileSinkDesc,
^
Error:(142, 23) not found: value HiveShim
val tmpLocation = HiveShim.getExternalTmpPath(hiveContext,
tableLocation)
^
Error:(143, 28) not found: type FileSinkDesc
val fileSinkConf = new FileSinkDesc(tmpLocation.toString, tableDesc,
false)
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
Error:(141, 22) not found: value HiveShim
val partPath = HiveShim.getDataLocationPath(partition)
^
Error:(298, 33) not found: value HiveShim
row.update(ordinal, HiveShim.toCatalystDecimal(oi, value))
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala
Error:(384, 3) not found: value HiveShim
HiveShim.createDefaultDBIfNeeded(this)
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/DescribeHiveTableCommand.scala
Error:(29, 8) object HiveShim is not a member of package
org.apache.spark.sql.hive
import org.apache.spark.sql.hive.HiveShim
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScan.scala
Error:(89, 5) not found: value HiveShim
HiveShim.appendReadColumns(hiveConf, neededColumnIDs,
attributes.map(_.name))
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala
Error:(38, 34) object HiveShim is not a member of package
org.apache.spark.sql.hive
import org.apache.spark.sql.hive.HiveShim._
^
Error:(37, 8) object ShimFileSinkDesc is not a member of package
org.apache.spark.sql.hive
import org.apache.spark.sql.hive.{ShimFileSinkDesc => FileSinkDesc}
^
Error:(174, 19) not found: type FileSinkDesc
fileSinkConf: FileSinkDesc,
^
Error:(46, 19) not found: type FileSinkDesc
fileSinkConf: FileSinkDesc)
^
Error:(220, 33) not found: type FileSinkDesc
val newFileSinkDesc = new FileSinkDesc(
^
/home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala
Warning:(34, 2) @deprecated now takes two arguments; see the scaladoc.
@deprecated("No code should depend on FakeParquetHiveSerDe as it is only
intended as a " +
^
------------------------------------------------------------------------
*I thought it was the problem from Maven Profiles. So I tried reselecting
hbase-hadoop1 or hive or hbase-hadoop2. The error still occurs. Please help
me. This has annoyed me for a whole afternoon!*
-----
Feel the sparking Spark!
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-development-with-IntelliJ-tp10032p10163.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
