Thank you Akhil and Mayur.
It will be really helpful.
Thanks,
On 27 Aug 2014 13:19, "Akhil Das" <ak...@sigmoidanalytics.com> wrote:
> Like Mayur said, its better to use mapPartition instead of map.
>
> Here's a piece of code which typically reads a text file and inserts each
> raw into the database. I haven't tested it, It might throw up some
> Serialization errors, In that case, you gotta serialize them!
>
> JavaRDD<String> txtRDD = jsc.textFile("/sigmoid/logs.cap");
>>
>> JavaRDD<String> dummyRDD = txtRDD.map(new Function<String,String>(){
>> @Override
>> public String call(String raw) throws Exception {
>>
>> * //Creating JDBC Connection*
>> Class.forName("oracle.jdbc.driver.OracleDriver");
>> Driver myDriver = new oracle.jdbc.driver.OracleDriver();
>> DriverManager.registerDriver( myDriver);
>> String URL = "jdbc:oracle:thin:AkhlD/password@akhldz:1521:EMP";
>> Connection conn = DriverManager.getConnection(URL);
>> Statement stmt = conn.createStatement();
>>
>>
>> * //Do some operations!!* stmt.executeQuery("INSERT INTO logs
>> VALUES(null, '" + raw + "'");
>>
>>
>> * //Now close the connection * conn.close();
>> return raw + " [ Added ]";
>> }
>> });
>
>
> Thanks
> Best Regards
>
>
> On Wed, Aug 27, 2014 at 10:06 AM, Mayur Rustagi <mayur.rust...@gmail.com>
> wrote:
>
>> I would suggest you to use JDBC connector in mappartition instead of maps
>> as JDBC connections are costly & can really impact your performance.
>>
>> Mayur Rustagi
>> Ph: +1 (760) 203 3257
>> http://www.sigmoidanalytics.com
>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>
>>
>>
>> On Tue, Aug 26, 2014 at 6:45 PM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> Yes, you can open a jdbc connection at the beginning of the map method
>>> then close this connection at the end of map() and in between you can use
>>> this connection.
>>>
>>> Thanks
>>> Best Regards
>>>
>>>
>>> On Tue, Aug 26, 2014 at 6:12 PM, Ravi Sharma <raviprincesha...@gmail.com
>>> > wrote:
>>>
>>>> Hello People,
>>>>>
>>>>> I'm using java spark streaming. I'm just wondering, Can I make simple
>>>>> jdbc connection in JavaDStream map() method?
>>>>>
>>>>> Or
>>>>>
>>>>> Do I need to create jdbc connection for each JavaPairDStream, after
>>>>> map task?
>>>>>
>>>>> Kindly give your thoughts.
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Ravi Sharma
>>>>>
>>>>
>>>>
>>>
>>
>
