Have a look at https://github.com/elastic/elasticsearch-hadoop#apache-spark
You can simply call the .saveToEs function to store your RDD data into ES.
Thanks
Best Regards
On Thu, Oct 29, 2015 at 8:19 PM, Nipun Arora <nipunarora2...@gmail.com>
wrote:
> Hi,
>
> I am sending data to an elasticsearch deployment. The printing to file
> seems to work fine, but I keep getting no-node found for ES when I send
> data to it. I suspect there is some special way to handle the connection
> object? Can anyone explain what should be changed here?
>
> Thanks
> Nipun
>
> The following is the code block where I send the data
>
>
>
>
> addMTSUnmatched.foreach(
> new Function<JavaRDD<String>, Void>() {
> @Override
> public Void call(JavaRDD<String> stringJavaRDD) throws Exception {
> List<String> list = stringJavaRDD.collect();
> for (String str : list){
> if(OnlineUtils.ESFlag) {
> OnlineUtils.printToFile(str, 1, type1_outputFile,
> OnlineUtils.client);
> }else{
> OnlineUtils.printToFile(str, 1, type1_outputFile);
> }
> }
> return null;
> }
> }
> );
>
>
> public static void printToFile(String str, int type, File fileID, Client
> client) throws IOException {
> JSONObject obj = new JSONObject(str);
> if (obj.has("message")) {
> String message = obj.get("message").toString();
> obj.put("message", message.replace("'", ""));
> }
> obj.put("anomaly_type", type);
>
> ESUtils.putMessage(client,"necla","demo",obj.toString());
>
> Files.append(str + "\n", fileID, Charset.defaultCharset());
> }
>
>
