Update: This Bandwidth Limiting can be used with any input operator. So you can apply similar changes to AbstractJDBCPollInputOperator shared by Devendra earlier.
-Priyanka On Mon, May 23, 2016 at 4:54 PM, Priyanka Gugale <[email protected]> wrote: > Hi Sunil, > > This > <https://github.com/apache/incubator-apex-malhar/compare/master...DT-Priyanka:jdbc-bandwidth-example?expand=1> > is the updated JDBCPojoInputOperator to use Bandwidth limiting feature > (Just look at JDBCPojoInputOperator changes, other classes are from > bandwidth PR) > > Once you update your code to use extend BandwidthLimitingOperator input > operator and use BandwidthManager (as done in example code), all you need > to do is set bandwidth value. In your properties file by adding, > > <property> > <name>dt.operator.<OperatorName>.prop.bandwidthManager.bandwidth</name> > <value>1073741824</value> > // 1 GB (change > it to desired value) > </property> > > During hours when you want application to stop reading you can modify > value of bandwidth to say -1 using dtcli command: > *set-operator-property operator-name property-name property-value* > > You can write cron jobs to change bandwidth property values. > > *The right solution would be to update BandwidthManager to accept the > bandwidth schedule and update consumption accordingly.* > > Also please note in this solution the application will still be running > and operators will be in deployed state, which means using some of your > cluster resources. > > -Priyanka > > > On Mon, May 23, 2016 at 4:27 PM, Sunil Parmar <[email protected]> > wrote: > >> Thanks Ram ! That’s the precise reason. >> Priyanka, >> Can you elaborate or give more little more guidance to update the >> application property run time ? How to do that ? >> >> -Sunil >> From: Munagala Ramanath <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Monday, May 23, 2016 at 3:42 PM >> To: "[email protected]" <[email protected]> >> Subject: Re: Time controlled ingestion operator >> >> From Sunil's note it sounds like the DB is in production use by other >> systems during certain hours and wants the Apex based migration to happen >> at other (non-overlapping) times. >> >> Ram >> On May 23, 2016 2:32 PM, "Priyanka Gugale" <[email protected]> >> wrote: >> >>> Hi Sunil, >>> >>> I don't think we have exactly pause and resume feature in Malhar yet. >>> But I can think of some other way to achieve this. >>> We are adding bandwidth control feature to Malhar. The pull request >>> <https://github.com/apache/incubator-apex-malhar/pull/279> is already >>> opened for the same. This doesn't have option to configure timed bandwidth >>> consumption (i.e. use x bandwidth during given time hours), but I think >>> it's easy to do that in BandwidthManager. Or quick thing is you can try to >>> update the property "bandwidth" during the time when u don't want operator >>> to work using some external script (this is short term solution). >>> >>> Can you tell the reason why you want to pause the read during certain >>> time hours? >>> >>> -Priyanka >>> >>> >>> On Mon, May 23, 2016 at 2:21 PM, Sunil Parmar <[email protected]> >>> wrote: >>> >>>> All, >>>> We’re writing an app for migration data from legacy JDBC system to >>>> Kafka. Legacy system as it is still being used in production, we would want >>>> to pause the JDBC poll input operator for configured hours. >>>> >>>> Does malhar ibrary has any operator that we can use for time controlled >>>> operator ? >>>> Or is there a way to configure input operators to pause and resume for >>>> given time ? >>>> >>>> -Sunil >>>> >>>> >>>> >>> >
