Like Mayur said, its better to use mapPartition instead of map.
Here's a piece of code which typically reads a text file and inserts each
raw into the database. I haven't tested it, It might throw up some
Serialization errors, In that case, you gotta serialize them!
JavaRDD<String> txtRDD = jsc.textFile("/sigmoid/logs.cap");
>
> JavaRDD<String> dummyRDD = txtRDD.map(new Function<String,String>(){
> @Override
> public String call(String raw) throws Exception {
>
> * //Creating JDBC Connection*
> Class.forName("oracle.jdbc.driver.OracleDriver");
> Driver myDriver = new oracle.jdbc.driver.OracleDriver();
> DriverManager.registerDriver( myDriver);
> String URL = "jdbc:oracle:thin:AkhlD/password@akhldz:1521:EMP";
> Connection conn = DriverManager.getConnection(URL);
> Statement stmt = conn.createStatement();
>
>
> * //Do some operations!!* stmt.executeQuery("INSERT INTO logs
> VALUES(null, '" + raw + "'");
>
>
> * //Now close the connection* conn.close();
> return raw + " [ Added ]";
> }
> });
Thanks
Best Regards
On Wed, Aug 27, 2014 at 10:06 AM, Mayur Rustagi <mayur.rust...@gmail.com>
wrote:
> I would suggest you to use JDBC connector in mappartition instead of maps
> as JDBC connections are costly & can really impact your performance.
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Tue, Aug 26, 2014 at 6:45 PM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Yes, you can open a jdbc connection at the beginning of the map method
>> then close this connection at the end of map() and in between you can use
>> this connection.
>>
>> Thanks
>> Best Regards
>>
>>
>> On Tue, Aug 26, 2014 at 6:12 PM, Ravi Sharma <raviprincesha...@gmail.com>
>> wrote:
>>
>>> Hello People,
>>>>
>>>> I'm using java spark streaming. I'm just wondering, Can I make simple
>>>> jdbc connection in JavaDStream map() method?
>>>>
>>>> Or
>>>>
>>>> Do I need to create jdbc connection for each JavaPairDStream, after
>>>> map task?
>>>>
>>>> Kindly give your thoughts.
>>>>
>>>>
>>>> Cheers,
>>>> Ravi Sharma
>>>>
>>>
>>>
>>
>
