I was able to get JavaWordCount running with a local instance under
IntelliJ.
In order to do so I needed to use maven to package my code and
call
String[] jars = {
"/SparkExamples/target/word-count-examples_2.10-1.0.0.jar" };
sparkConf.setJars(jars);
After that the sample ran properly and in the debugger I could set break
points in the main.
However when I do
something like
JavaRDD<String> words = lines.flatMap( new WordsMapFunction());
where WordsMapFunction is a separate class like
public static class WordsMapFunction implements FlatMapFunction<String,
String> {
private static final Pattern SPACE = Pattern.compile(" ");
public Iterable<String> call(String s) {
String[] split = SPACE.split(s);
for (int i = 0; i < split.length; i++) {
split[i] = toUpperCase(split[i]);
}
return Arrays.asList(split);
}
}
Breakpoints set in WordsMapFunction are never hit.
Most interesting functionality in the problems I am trying to solve if in
the FlatMapFunction and the Function2 code and this is the functionality I
will need to examine in more detail.
Has anyone figured out how to configure a project to hit breakpoints in
these functions??