Hi!
Well is seems that you have to contact the team that created the jdbc
storm project. It seems that the issue is deeper than my knowledge.
I hope that my advises helped you. If you have a solution, please share it
here that the others can benefit from your experience.
Good luck!
Regards,
Florin
On Thu, May 5, 2016 at 1:57 PM, Sai Dilip Reddy Kiralam <
dkira...@aadhya-analytics.com> wrote:
> Hi,
>
> my supervisor logs shows the following error :
>
> 2016-05-05T15:45:15.365+0530 b.s.d.executor [INFO] Prepared bolt
> __acker:(67)
> 2016-05-05T15:45:15.369+0530 b.s.d.executor [INFO] Prepared bolt
> __system:(-1)
> 2016-05-05T15:45:18.481+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-0 is starting.
> 2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-1 is starting.
> 2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-3 is starting.
> 2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-2 is starting.
> 2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-3 is starting.
> 2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-4 is starting.
> 2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-5 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-6 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-7 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-8 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-9 is starting.
> 2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-10 is starting.
> 2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-11 is starting.
> 2016-05-05T15:45:18.492+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-12 is starting.
> 2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-13 is starting.
> 2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-4 is starting.
> 2016-05-05T15:45:19.310+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.311+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
>
> do postgres is using all the connections because this error ?
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, May 5, 2016 at 10:57 AM, Sai Dilip Reddy Kiralam <
> dkira...@aadhya-analytics.com> wrote:
>
>> Hi,
>>
>> I'm closing up and opening the connections after execution of insert
>> query by my bolt.Do I need to increase the max connections db to a level of
>> 500 to 1000? Here in log file of supervisor it tells me the connection
>> made by bolt is closed and again taking new connection.I think after
>> closing it is going into ideal and not used for new connections.
>>
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS
>> DONE
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected....
>>
>> if I'm right what is the solution for those ideal connections?
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Thu, May 5, 2016 at 1:01 AM, Spico Florin <spicoflo...@gmail.com>
>> wrote:
>>
>>> hi!
>>> you have 9 bolts with 50 max db connections. so for each bolt you get a
>>> conection pool. try to decresae this number for example to 5 and check if
>>> your performance if fine with your db
>>> regards,
>>> florin
>>>
>>> On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <
>>> dkira...@aadhya-analytics.com> wrote:
>>>
>>>> sorry ! my db is not yet started, so it given me the error ! but when I
>>>> gave that statement and run the topology then it is using more connections
>>>> than the specified number.
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
>>>> dkira...@aadhya-analytics.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I just added
>>>>>
>>>>> a statement
>>>>> * // config.setMaximumPoolSize(50);*
>>>>> public synchronized void prepare() {
>>>>> if(dataSource == null) {
>>>>> Properties properties = new Properties();
>>>>> properties.putAll(configMap);
>>>>> HikariConfig config = new HikariConfig(properties);
>>>>> * config.setMaximumPoolSize(50);*
>>>>> this.dataSource = new HikariDataSource(config);
>>>>> this.dataSource.setAutoCommit(false);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> and I'm getting the following error
>>>>>
>>>>> java.lang.RuntimeException: Fail-fast during pool initialization at
>>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
>>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
>>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
>>>>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
>>>>> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27)
>>>>> at
>>>>> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
>>>>> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
>>>>> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
>>>>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
>>>>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: org.postgresql.util.PSQLException: Connection attempt timed
>>>>> out.
>>>>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
>>>>> org.postgresql.Driver.connect(Driver.java:284) at
>>>>> java.sql.DriverManager.getConnection(DriverManager.java:664) at
>>>>> java.sql.DriverManager.getConnection(DriverManager.java:247) at
>>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
>>>>> at
>>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>>>>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10
>>>>> more
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
>>>>> dkira...@aadhya-analytics.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thank you Spico Florin.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Best regards,*
>>>>>>
>>>>>> *K.Sai Dilip Reddy.*
>>>>>>
>>>>>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <spicoflo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> hi!
>>>>>>> please have a look
>>>>>>> https://github.com/apache/storm/tree/master/external/storm-jdbc
>>>>>>> where you hve to implemnt thw interface connectionprovider interface.
>>>>>>> you
>>>>>>> have to dins a third paty librry that provides conection pooloing for
>>>>>>> postgresql and use that library to implement the interface connection
>>>>>>> provider. i hope that other community member will give you some other
>>>>>>> ideas.
>>>>>>> i hope that these help.
>>>>>>> regards,
>>>>>>> florin
>>>>>>>
>>>>>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>>>>>> dkira...@aadhya-analytics.com> wrote:
>>>>>>>
>>>>>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>>>>>> pooling on codes on topology.please share information on connection
>>>>>>>> pooling
>>>>>>>> used on storm topologies.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Best regards,*
>>>>>>>>
>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>
>>>>>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>>>>>> dkira...@aadhya-analytics.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Spico,
>>>>>>>>>
>>>>>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>>>>>> postgres.Just using jdbc classe examples here is source example
>>>>>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>>>>>> I will check by using the ConnectionPool.
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Best regards,*
>>>>>>>>>
>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>
>>>>>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <
>>>>>>>>> spicoflo...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello!
>>>>>>>>>> How many tasks do you have for inserting the data to your
>>>>>>>>>> database? Are you using ConnectionPool for connecting to Postgres?
>>>>>>>>>> If your
>>>>>>>>>> number of task superseeds the number of max connections provided in
>>>>>>>>>> connection pool then your have a problem.
>>>>>>>>>> Please also check the number of max connections that your db
>>>>>>>>>> accepts.
>>>>>>>>>> I hope that these help.
>>>>>>>>>> Regards,
>>>>>>>>>> Florin
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>>>>>> dkira...@aadhya-analytics.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have topology that make connection with postgresdb and insert
>>>>>>>>>>> the fields into tables of a test db.my topology is working fine but
>>>>>>>>>>> when I
>>>>>>>>>>> submit the topology it is establishing all the connections of
>>>>>>>>>>> db.but I
>>>>>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Best regards,*
>>>>>>>>>>>
>>>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
