Hi Sunil,
This
<https://github.com/apache/incubator-apex-malhar/compare/master...DT-Priyanka:jdbc-bandwidth-example?expand=1>
is the updated JDBCPojoInputOperator to use Bandwidth limiting feature
(Just look at JDBCPojoInputOperator changes, other classes are from
bandwidth PR)
Once you update your code to use extend BandwidthLimitingOperator input
operator and use BandwidthManager (as done in example code), all you need
to do is set bandwidth value. In your properties file by adding,
<property>
<name>dt.operator.<OperatorName>.prop.bandwidthManager.bandwidth</name>
<value>1073741824</value>
// 1 GB (change
it to desired value)
</property>
During hours when you want application to stop reading you can modify value
of bandwidth to say -1 using dtcli command:
*set-operator-property operator-name property-name property-value*
You can write cron jobs to change bandwidth property values.
*The right solution would be to update BandwidthManager to accept the
bandwidth schedule and update consumption accordingly.*
Also please note in this solution the application will still be running and
operators will be in deployed state, which means using some of your cluster
resources.
-Priyanka
On Mon, May 23, 2016 at 4:27 PM, Sunil Parmar <[email protected]>
wrote:
> Thanks Ram ! That’s the precise reason.
> Priyanka,
> Can you elaborate or give more little more guidance to update the
> application property run time ? How to do that ?
>
> -Sunil
> From: Munagala Ramanath <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Monday, May 23, 2016 at 3:42 PM
> To: "[email protected]" <[email protected]>
> Subject: Re: Time controlled ingestion operator
>
> From Sunil's note it sounds like the DB is in production use by other
> systems during certain hours and wants the Apex based migration to happen
> at other (non-overlapping) times.
>
> Ram
> On May 23, 2016 2:32 PM, "Priyanka Gugale" <[email protected]>
> wrote:
>
>> Hi Sunil,
>>
>> I don't think we have exactly pause and resume feature in Malhar yet. But
>> I can think of some other way to achieve this.
>> We are adding bandwidth control feature to Malhar. The pull request
>> <https://github.com/apache/incubator-apex-malhar/pull/279> is already
>> opened for the same. This doesn't have option to configure timed bandwidth
>> consumption (i.e. use x bandwidth during given time hours), but I think
>> it's easy to do that in BandwidthManager. Or quick thing is you can try to
>> update the property "bandwidth" during the time when u don't want operator
>> to work using some external script (this is short term solution).
>>
>> Can you tell the reason why you want to pause the read during certain
>> time hours?
>>
>> -Priyanka
>>
>>
>> On Mon, May 23, 2016 at 2:21 PM, Sunil Parmar <[email protected]>
>> wrote:
>>
>>> All,
>>> We’re writing an app for migration data from legacy JDBC system to
>>> Kafka. Legacy system as it is still being used in production, we would want
>>> to pause the JDBC poll input operator for configured hours.
>>>
>>> Does malhar ibrary has any operator that we can use for time controlled
>>> operator ?
>>> Or is there a way to configure input operators to pause and resume for
>>> given time ?
>>>
>>> -Sunil
>>>
>>>
>>>
>>
