HI Ashok
this is not really a spark-related question so i would not use this
mailing list.....
Anyway, my 2 cents here
as outlined by earlier replies, if the class you are referencing is in a
different jar, at compile time you will need to add that dependency to your
build.sbt,
I'd personally leave alone $CLASSPATH...
AT RUN TIME, you have two options:
1 - as suggested by Ted, when yo u launch your app via spark-submit you
can use '--jars utilities-assembly-0.1-SNAPSHOT.jar' to pass the jar.
2 - Use sbt assembly plugin to package your classes and jars into a 'fat
jar', and then at runtime all you need to do is to do
spark-submit --class <your spark app class name> <path to your fat
jar>
I'd personally go for 1 as it is the easiest option. (FYI for 2 you
might encounter situations where you have dependencies referring to same
classes, adn that will require you to define an assemblyMergeStrategy....)
hth
On Mon, Jun 6, 2016 at 8:52 AM, Ashok Kumar <ashok34...@yahoo.com.invalid>
wrote:
> Anyone can help me with this please
>
>
> On Sunday, 5 June 2016, 11:06, Ashok Kumar <ashok34...@yahoo.com> wrote:
>
>
> Hi all,
>
> Appreciate any advice on this. It is about scala
>
> I have created a very basic Utilities.scala that contains a test class and
> method. I intend to add my own classes and methods as I expand and make
> references to these classes and methods in my other apps
>
> class getCheckpointDirectory {
> def CheckpointDirectory (ProgramName: String) : String = {
> var hdfsDir = "hdfs://host:9000/user/user/checkpoint/"+ProgramName
> return hdfsDir
> }
> }
> I have used sbt to create a jar file for it. It is created as a jar file
>
> utilities-assembly-0.1-SNAPSHOT.jar
>
> Now I want to make a call to that method CheckpointDirectory in my app
> code myapp.dcala to return the value for hdfsDir
>
> val ProgramName = this.getClass.getSimpleName.trim
> val getCheckpointDirectory = new getCheckpointDirectory
> val hdfsDir = getCheckpointDirectory.CheckpointDirectory(ProgramName)
>
> However, I am getting a compilation error as expected
>
> not found: type getCheckpointDirectory
> [error] val getCheckpointDirectory = new getCheckpointDirectory
> [error] ^
> [error] one error found
> [error] (compile:compileIncremental) Compilation failed
>
> So a basic question, in order for compilation to work do I need to create
> a package for my jar file or add dependency like the following I do in sbt
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1"
> libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1"
>
>
> Or add the jar file to $CLASSPATH?
>
> Any advise will be appreciated.
>
> Thanks
>
>
>
>
>
>
>
