Re: How to send RDD result to REST API?

[Ted Yu]

You can utilize commons-httpclient and push result of RDD to the server.

BTW specify commons-httpclient version which is consistent with the hadoop
version you use.
e.g. the following is brought in along with hadoop 2.6:

[INFO] |  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:compile

Cheers

On Mon, Aug 31, 2015 at 11:45 AM, Cassa L  wrote:

> Hi Ted,
>   My server is  expecting JSON. Can I just write HttpClient in spark job
> and push result of RDD action to the server?  I'm trying to figure out how
> to achieve this.
>
> LCassa
>
> On Fri, Aug 28, 2015 at 9:45 PM, Ted Yu  wrote:
>
>> What format does your REST server expect ?
>>
>> You may have seen this:
>>
>> https://www.paypal-engineering.com/2014/02/13/hello-newman-a-rest-client-for-scala/
>>
>> On Fri, Aug 28, 2015 at 9:35 PM, Cassa L  wrote:
>>
>>> Hi,
>>> If I have RDD that counts something e.g.:
>>>
>>> JavaPairDStream successMsgCounts = successMsgs
>>> .flatMap(buffer -> Arrays.asList(buffer.getType()))
>>> .mapToPair(txnType -> new Tuple2>> Integer>("Success " + txnType, 1))
>>> .reduceByKey((count1, count2) -> count1 + count2);
>>>
>>> successMsgCounts.print();
>>>
>>> Instead of printing this output how can I push it to REST API? I have a
>>> server which needs this information to be fed via REST.
>>>
>>>
>>> thanks
>>> LCassa
>>>
>>
>>
>

Re: How to send RDD result to REST API?

[Cassa L]

Hi Ted,
  My server is  expecting JSON. Can I just write HttpClient in spark job
and push result of RDD action to the server?  I'm trying to figure out how
to achieve this.

LCassa

On Fri, Aug 28, 2015 at 9:45 PM, Ted Yu  wrote:

> What format does your REST server expect ?
>
> You may have seen this:
>
> https://www.paypal-engineering.com/2014/02/13/hello-newman-a-rest-client-for-scala/
>
> On Fri, Aug 28, 2015 at 9:35 PM, Cassa L  wrote:
>
>> Hi,
>> If I have RDD that counts something e.g.:
>>
>> JavaPairDStream successMsgCounts = successMsgs
>> .flatMap(buffer -> Arrays.asList(buffer.getType()))
>> .mapToPair(txnType -> new Tuple2> Integer>("Success " + txnType, 1))
>> .reduceByKey((count1, count2) -> count1 + count2);
>>
>> successMsgCounts.print();
>>
>> Instead of printing this output how can I push it to REST API? I have a
>> server which needs this information to be fed via REST.
>>
>>
>> thanks
>> LCassa
>>
>
>

Re: How to send RDD result to REST API?

[Ted Yu]

What format does your REST server expect ?

You may have seen this:
https://www.paypal-engineering.com/2014/02/13/hello-newman-a-rest-client-for-scala/

On Fri, Aug 28, 2015 at 9:35 PM, Cassa L  wrote:

> Hi,
> If I have RDD that counts something e.g.:
>
> JavaPairDStream successMsgCounts = successMsgs
> .flatMap(buffer -> Arrays.asList(buffer.getType()))
> .mapToPair(txnType -> new Tuple2("Success
> " + txnType, 1))
> .reduceByKey((count1, count2) -> count1 + count2);
>
> successMsgCounts.print();
>
> Instead of printing this output how can I push it to REST API? I have a
> server which needs this information to be fed via REST.
>
>
> thanks
> LCassa
>

How to send RDD result to REST API?

[Cassa L]

Hi,
If I have RDD that counts something e.g.:

JavaPairDStream successMsgCounts = successMsgs
.flatMap(buffer -> Arrays.asList(buffer.getType()))
.mapToPair(txnType -> new Tuple2("Success
" + txnType, 1))
.reduceByKey((count1, count2) -> count1 + count2);

successMsgCounts.print();

Instead of printing this output how can I push it to REST API? I have a
server which needs this information to be fed via REST.


thanks
LCassa