Re: Operators in PENDING_DEPLOY

[Pradeep A. Dalvi]

Can you share AppMaster logs? What are container resource requests in
AppMaster logs?

--prad

On Fri, Aug 12, 2016 at 9:28 AM, Ankit Sarraf 
wrote:

> When I deploy my application on DTConsole, all applications are in pending
> deploy state. The App itself is showing the running status.
>
> One of the resources says, state change from ACTIVE to PENDING points to
> an unhealthy state and, if it does not stabilize, your only option is to
> kill the application and reduce the memory needs or acquire more cluster
> resources.
>
> I have no other app running in the QA queue. Also, when i run the app in
> the DEV queue using UCD it works fine without any issues.
>
> Any pointers?
>
> Thanks
> Ankit
>

Re: delegation token not found in cache

[Pradeep A. Dalvi]

Could you run following commands?

klist

klist -kt 

--prad

On Thu, Jul 21, 2016 at 10:14 PM, Raja.Aravapalli <
raja.aravapa...@target.com> wrote:

>
> Hi,
>
> Can someone please help me why my DAG is failing with below exception
> after 8 days running succesfully!!
>
>
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
>  token (HDFS_DELEGATION_TOKEN token 00 for ) can't be found in cache
>   at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>
>
>
>
> Thanks a lot.
>
> Regards,
> Raja.
>
>

Re: Container & memory resource allocation

[Pradeep A. Dalvi]

I've verified 'multiples of minimum-allocation-mb' on latest Apex. However
'increment-allocation-mb' was not set during that exercise.
I shall check that param as well.

Thanks,
--prad

On Wednesday, July 20, 2016, Munagala Ramanath <r...@datatorrent.com> wrote:

> Please note that there are multiple sites making the claim that memory
> allocation
> is in multiples of *yarn.scheduler.minimum-allocation-mb*; this may have
> been true
> at one time but is no longer true (thanks to Sandesh for fact-checking
> this).
>
> There is a (?new?) parameter, *yarn.scheduler.increment-allocation-mb*,
> which serves
> this purpose as discussed here:
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
> Ram
>
> On Tue, Jul 19, 2016 at 11:27 AM, Pradeep A. Dalvi <p...@apache.org
> <javascript:;>> wrote:
>
> > Thanks Chinmay & Ram.
> >
> > Troubleshooting page sounds the appropriate location. I shall raise PR
> with
> > the given suggestions.
> >
> > --prad
> >
> > On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <r...@datatorrent.com
> <javascript:;>>
> > wrote:
> >
> > > There is already a link to a troubleshooting page at bottom of
> > > https://apex.apache.org/docs.html
> > > That page already has some discussion under the section entitled
> > > "Calculating Container Memory"
> > > so adding new content there seems like the right thing to do.
> > >
> > > Ram
> > >
> > > On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar <
> > > chin...@datatorrent.com <javascript:;>
> > > > wrote:
> > >
> > > > Hi Pradeep,
> > > >
> > > > This is a great content to add to the documents. These are the common
> > set
> > > > of errors which might get googled and hence great to get indexed as
> > well.
> > > >
> > > > You can take a look at:
> > > > https://github.com/apache/apex-core/tree/master/docs
> > > >
> > > > The docs for apex reside there in markdown format. Probably its good
> a
> > > > create a troubleshooting page where all such common questions can
> > reside.
> > > >
> > > > After you have the content ready, you can create a pull request to
> > > > apex-core repo which can get merged to apex-core and later deployed
> to
> > > the
> > > > website by committers.
> > > >
> > > > -Chinmay.
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <p...@apache.org
> <javascript:;>>
> > > > wrote:
> > > >
> > > >> Container & memory resource allocation has been a common question
> > around
> > > >> and so I thought it would be good to explain related configuration
> > > >> parameters.
> > > >>
> > > >> Please feel free to let me know your thoughts.
> > > >>
> > > >> Also I'm planning to add following set of information under Apex
> Docs.
> > > How
> > > >> could one add this to Apex Docs?
> > > >>
> > > >> =-=-=-=
> > > >>
> > > >> "Container is running beyond physical memory limits. Current usage:
> X
> > GB
> > > >> of
> > > >> Y GB physical memory used; A GB of B GB virtual memory used. Killing
> > > >> container."
> > > >>
> > > >> This is basically for some better understanding on Application
> > Master's
> > > >> container requests & Resource Manager's memory resource allocation.
> > > Please
> > > >> note that these are individual container request params. All these
> > > >> parameters are in MB i.e. 1024 => 1GB.
> > > >>
> > > >> - AM's container requests to RM shall contain memory in the
> multiples
> > of
> > > >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding
> > > >> *yarn.scheduler.maximum-**allocation-mb*
> > > >>- If *yarn.scheduler.minimum-**allocation-mb *is configured as
> 1024
> > > and
> > > >> container memory requirement is 1025 ( <= 2048 ), container will be
> > > >> allocated with 2048 memory.
> > > >>
> > > >> - With Apex applications, operator memory can be specified by
> property
> > > >> *dt.application..operator..attr.MEMORY_MB*
> > > >>- Please note this parameter is at Operator level and container
> > > memory
> > > >> is calculated based on number of Operators deployed in a container +
> > > >> additional memory required depending on physical deployment
> > requirements
> > > >> e.g. unifier or bufferserver
> > > >>- Wildcard * can be used at APP_NAME and/or OPERATOR_NAME
> > > >>
> > > >> - If container memory is not specified, then AM would request for 1
> > unit
> > > >> of
> > > >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision
> container
> > > >> taking that into consideration.
> > > >>
> > > >> Node Manager monitors memory usage of each of these containers and
> > kills
> > > >> the ones crossing the configured limit.
> > > >>
> > > >> Almost similar stuff is applicable for CPUs.
> > > >>
> > > >> --prad
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Container & memory resource allocation

[Pradeep A. Dalvi]

Thanks Chinmay & Ram.

Troubleshooting page sounds the appropriate location. I shall raise PR with
the given suggestions.

--prad

On Tue, Jul 19, 2016 at 5:49 AM, Munagala Ramanath <r...@datatorrent.com>
wrote:

> There is already a link to a troubleshooting page at bottom of
> https://apex.apache.org/docs.html
> That page already has some discussion under the section entitled
> "Calculating Container Memory"
> so adding new content there seems like the right thing to do.
>
> Ram
>
> On Mon, Jul 18, 2016 at 11:27 PM, Chinmay Kolhatkar <
> chin...@datatorrent.com
> > wrote:
>
> > Hi Pradeep,
> >
> > This is a great content to add to the documents. These are the common set
> > of errors which might get googled and hence great to get indexed as well.
> >
> > You can take a look at:
> > https://github.com/apache/apex-core/tree/master/docs
> >
> > The docs for apex reside there in markdown format. Probably its good a
> > create a troubleshooting page where all such common questions can reside.
> >
> > After you have the content ready, you can create a pull request to
> > apex-core repo which can get merged to apex-core and later deployed to
> the
> > website by committers.
> >
> > -Chinmay.
> >
> >
> >
> >
> > On Tue, Jul 19, 2016 at 10:46 AM, Pradeep A. Dalvi <p...@apache.org>
> > wrote:
> >
> >> Container & memory resource allocation has been a common question around
> >> and so I thought it would be good to explain related configuration
> >> parameters.
> >>
> >> Please feel free to let me know your thoughts.
> >>
> >> Also I'm planning to add following set of information under Apex Docs.
> How
> >> could one add this to Apex Docs?
> >>
> >> =-=-=-=
> >>
> >> "Container is running beyond physical memory limits. Current usage: X GB
> >> of
> >> Y GB physical memory used; A GB of B GB virtual memory used. Killing
> >> container."
> >>
> >> This is basically for some better understanding on Application Master's
> >> container requests & Resource Manager's memory resource allocation.
> Please
> >> note that these are individual container request params. All these
> >> parameters are in MB i.e. 1024 => 1GB.
> >>
> >> - AM's container requests to RM shall contain memory in the multiples of
> >> *yarn.scheduler.minimum-**allocation-mb* & not exceeding
> >> *yarn.scheduler.maximum-**allocation-mb*
> >>- If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024
> and
> >> container memory requirement is 1025 ( <= 2048 ), container will be
> >> allocated with 2048 memory.
> >>
> >> - With Apex applications, operator memory can be specified by property
> >> *dt.application..operator..attr.MEMORY_MB*
> >>- Please note this parameter is at Operator level and container
> memory
> >> is calculated based on number of Operators deployed in a container +
> >> additional memory required depending on physical deployment requirements
> >> e.g. unifier or bufferserver
> >>- Wildcard * can be used at APP_NAME and/or OPERATOR_NAME
> >>
> >> - If container memory is not specified, then AM would request for 1 unit
> >> of
> >> *yarn.scheduler.minimum-**allocation-mb*, RM would provision container
> >> taking that into consideration.
> >>
> >> Node Manager monitors memory usage of each of these containers and kills
> >> the ones crossing the configured limit.
> >>
> >> Almost similar stuff is applicable for CPUs.
> >>
> >> --prad
> >>
> >
> >
>

Container & memory resource allocation

[Pradeep A. Dalvi]

Container & memory resource allocation has been a common question around
and so I thought it would be good to explain related configuration
parameters.

Please feel free to let me know your thoughts.

Also I'm planning to add following set of information under Apex Docs. How
could one add this to Apex Docs?

=-=-=-=

"Container is running beyond physical memory limits. Current usage: X GB of
Y GB physical memory used; A GB of B GB virtual memory used. Killing
container."

This is basically for some better understanding on Application Master's
container requests & Resource Manager's memory resource allocation. Please
note that these are individual container request params. All these
parameters are in MB i.e. 1024 => 1GB.

- AM's container requests to RM shall contain memory in the multiples of
*yarn.scheduler.minimum-**allocation-mb* & not exceeding
*yarn.scheduler.maximum-**allocation-mb*
   - If *yarn.scheduler.minimum-**allocation-mb *is configured as 1024 and
container memory requirement is 1025 ( <= 2048 ), container will be
allocated with 2048 memory.

- With Apex applications, operator memory can be specified by property
*dt.application..operator..attr.MEMORY_MB*
   - Please note this parameter is at Operator level and container memory
is calculated based on number of Operators deployed in a container +
additional memory required depending on physical deployment requirements
e.g. unifier or bufferserver
   - Wildcard * can be used at APP_NAME and/or OPERATOR_NAME

- If container memory is not specified, then AM would request for 1 unit of
*yarn.scheduler.minimum-**allocation-mb*, RM would provision container
taking that into consideration.

Node Manager monitors memory usage of each of these containers and kills
the ones crossing the configured limit.

Almost similar stuff is applicable for CPUs.

--prad

Re: Multiple output and input

[Pradeep A. Dalvi]

In Apex, an app can have multiple pipelines i.e. multiple input operators.
Do you want to read the same set of data from same db table and
duplicate it on 2 different output ports? If yes, then you would need
to extend the input operator and add another output port in extended
class.
Can you share errors?

--prad

On Tue, Jul 12, 2016 at 9:19 AM, Jaikit Jilka  wrote:
> Hello,
>
> In my application I am using JdbcPojooutput operator and JdbcPojoInput 
> operator. I was thinking whether we can have multiple output for 
> JdbcPojoInput operator as currently I am using 2 JdbcPojoInput operator. I 
> tried to do so but I got error. So I want to know can we do that?
>
> Thank You,
>
> Jaikit Jilka

Re: Jdbcoutputoperator implementation

[Pradeep A. Dalvi]

Please share Application Master logs.

On Thu, Jul 7, 2016 at 11:43 AM, Jaikit Jilka <jji...@leadferret.com> wrote:
> Hello,
>
> I completed my application now. Till now I was launching this application in 
> Sandbox and it works perfectly fine. I am getting expected results. But now I 
> tried running it on live cluster and it is not working. It is not giving me 
> any error but all my operators are killed by application master. So do I need 
> to do anything for launching it on live cluster?
>
> - Original Message -
> From: "Jaikit Jilka" <jji...@leadferret.com>
> To: "users" <users@apex.apache.org>
> Sent: Thursday, June 30, 2016 11:29:25 AM
> Subject: Re: Jdbcoutputoperator implementation
>
> Hello,
>
> Now my every operator is working fine. But counter operator does not emit 
> anthing. It is just processing the data. Here is my counter.java operator 
> file.
>
> - Original Message -
> From: "Jaikit Jilka" <jji...@leadferret.com>
> To: "users" <users@apex.apache.org>
> Sent: Thursday, June 30, 2016 10:25:35 AM
> Subject: Re: Jdbcoutputoperator implementation
>
> Hello,
>
> Got it. I have to create a outport of type pojo. Working on it.
>
> - Original Message -
> From: "Jaikit Jilka" <jji...@leadferret.com>
> To: "users" <users@apex.apache.org>
> Sent: Thursday, June 30, 2016 10:11:10 AM
> Subject: Re: Jdbcoutputoperator implementation
>
> Hello,
>
> I can not emit output.emit(pojo) only emits string. It it giving me error as 
> not method found for emit(pojoEvent).
>
> - Original Message -
> From: "Yunhan Wang" <yun...@datatorrent.com>
> To: "users" <users@apex.apache.org>
> Sent: Wednesday, June 29, 2016 2:59:41 PM
> Subject: Re: Jdbcoutputoperator implementation
>
> Hi Jaikit,
>
> I think the main problem here is: your Counter Operator takes String as
> input and emits String as output. However, your JdbcOutputOperator needs
> POJO as input. So you need to modify Counter Operator to emit the POJO
> instead of String. You could do that by writing separate function e.g.
> emitTuples as below:
>
> public void emitTuples(String word)
>
> {
> PojoEvent pojo = new PojoEvent();
> pojo.setName(word);
> output.emit(pojo);
> }
>
> Also, you need to add this to your properties.xml:
>
> 
> 
> dt.application.test.operator.JdbcOutput.port.input.attr.TUPLE_CLASS
> com.mycompany.test.PojoEvent
> 
>
> Hope this can help.
>
> Thanks,
> Yunhan
>
>
>
>
>
> On Wed, Jun 29, 2016 at 1:14 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>
>> This is my POJO class. Can you please tell me what changes should I make
>> into this.
>>
>> Thank you,
>>
>> Jaikit Jilka
>>
>>
>> - Original Message -
>> From: "Pradeep A. Dalvi" <p...@apache.org>
>> To: "users" <users@apex.apache.org>
>> Sent: Tuesday, June 28, 2016 4:56:57 PM
>> Subject: Re: Jdbcoutputoperator implementation
>>
>> By adding a string member variable to POJO class. Use that member
>> variable to send the data to JdbcPOJOOutputOperator.
>>
>> On Tue, Jun 28, 2016 at 4:34 PM, Jaikit Jilka <jji...@leadferret.com>
>> wrote:
>> > So how to convert a java string into POJO?
>> >
>> > - Original Message -
>> > From: "Pradeep A. Dalvi" <p...@apache.org>
>> > To: "users" <users@apex.apache.org>
>> > Sent: Tuesday, June 28, 2016 4:30:09 PM
>> > Subject: Re: Jdbcoutputoperator implementation
>> >
>> > JdbcPOJOOutputOperator expects data as POJO, whose class name is
>> > specified using Context.PortContext.TUPLE_CLASS i.e.
>> > com.mycompany.test.PojoEvent here. Previous operator is probably
>> > sending String to JdbcPOJOOutputOperator.
>> >
>> > On Tue, Jun 28, 2016 at 4:16 PM, Jaikit Jilka <jji...@leadferret.com>
>> wrote:
>> >> Hello,
>> >>
>> >> I am taking data from a .txt file and spliting words and inserting it
>> into database.
>> >>
>> >> - Original Message -
>> >> From: "Pradeep A. Dalvi" <p...@apache.org>
>> >> To: "users" <users@apex.apache.org>
>> >> Sent: Tuesday, June 28, 2016 4:12:31 PM
>> >> Subject: Re: Jdbcoutputoperator implementation
>> >>
>> >> What is the type of data emitted by previous operator to
>&

Re: Jdbcoutputoperator implementation

[Pradeep A. Dalvi]

By adding a string member variable to POJO class. Use that member
variable to send the data to JdbcPOJOOutputOperator.

On Tue, Jun 28, 2016 at 4:34 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
> So how to convert a java string into POJO?
>
> - Original Message -
> From: "Pradeep A. Dalvi" <p...@apache.org>
> To: "users" <users@apex.apache.org>
> Sent: Tuesday, June 28, 2016 4:30:09 PM
> Subject: Re: Jdbcoutputoperator implementation
>
> JdbcPOJOOutputOperator expects data as POJO, whose class name is
> specified using Context.PortContext.TUPLE_CLASS i.e.
> com.mycompany.test.PojoEvent here. Previous operator is probably
> sending String to JdbcPOJOOutputOperator.
>
> On Tue, Jun 28, 2016 at 4:16 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>> Hello,
>>
>> I am taking data from a .txt file and spliting words and inserting it into 
>> database.
>>
>> - Original Message -
>> From: "Pradeep A. Dalvi" <p...@apache.org>
>> To: "users" <users@apex.apache.org>
>> Sent: Tuesday, June 28, 2016 4:12:31 PM
>> Subject: Re: Jdbcoutputoperator implementation
>>
>> What is the type of data emitted by previous operator to 
>> JdbcPOJOOutputOperator?
>>
>> On Tue, Jun 28, 2016 at 3:46 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>>> Hello,
>>>
>>> I am getting an error as follow:
>>>
>>> Stopped running due to an exception. java.lang.ClassCastException: 
>>> java.lang.String cannot be cast to com.mycompany.test.PojoEvent
>>> at SC.get(Unknown Source)
>>> at 
>>> com.datatorrent.lib.db.jdbc.JdbcPOJOOutputOperator.setStatementParameters(JdbcPOJOOutputOperator.java:175)
>>> at 
>>> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.processBatch(AbstractJdbcTransactionableOutputOperator.java:114)
>>> at 
>>> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.endWindow(AbstractJdbcTransactionableOutputOperator.java:93)
>>> at 
>>> com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:145)
>>> at 
>>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:351)
>>> at 
>>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)
>>>
>>> I am not sure where to check the cast mismatch. I am inserting a value in 
>>> the database.
>>>
>>> Thank You,
>>>
>>> Jaikit Jilka
>>>
>>> - Original Message -
>>> From: "Pradeep A. Dalvi" <p...@apache.org>
>>> To: "users" <users@apex.apache.org>
>>> Sent: Friday, June 24, 2016 4:45:18 PM
>>> Subject: Re: Jdbcoutputoperator implementation
>>>
>>> What are errors/exceptions seen in JdbcOutput container logs?
>>> Are you setting Context.PortContext.TUPLE_CLASS to POJO class being used?
>>>
>>> --
>>> prad
>>>
>>> On Fri, Jun 24, 2016 at 2:13 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>>>> Hello,
>>>>
>>>> I resolved that error and now I can launch the application, But now the 
>>>> container which has jdbcoperator is getting killed. I am attaching the 
>>>> dt.log file of container which is running but does not have any operator.
>>>>
>>>> Thank You,
>>>>
>>>> Jaikit Jilka
>>>>
>>>> - Original Message -
>>>> From: "Pradeep A. Dalvi" <p...@apache.org>
>>>> To: "users" <users@apex.apache.org>
>>>> Sent: Friday, June 24, 2016 11:41:47 AM
>>>> Subject: Re: Jdbcoutputoperator implementation
>>>>
>>>> Yes, because it is not yet released. Compile & install latest malhar
>>>> in local ~/.m2 directory.
>>>> For that, git clone Malhar library repository. And do 'mvn clean
>>>> install' in that malhar clone.
>>>>
>>>> On Fri, Jun 24, 2016 at 11:31 AM, Jaikit Jilka <jji...@leadferret.com> 
>>>> wrote:
>>>>> Hello,
>>>>>
>>>>> When I change my malher version in POM.xml to the latest i.e 
>>>>> 3.5.0-SNAPSHOT it gives me an build time error : Failure to find 
>>>>> org.apache.apex:malhar-library:jar:3.5.0-SNAPSHOT in 
>>>>> https://www.datatorrent.com/maven/content/repositories/snapshots/.
>>>>&

Re: Jdbcoutputoperator implementation

[Pradeep A. Dalvi]

JdbcPOJOOutputOperator expects data as POJO, whose class name is
specified using Context.PortContext.TUPLE_CLASS i.e.
com.mycompany.test.PojoEvent here. Previous operator is probably
sending String to JdbcPOJOOutputOperator.

On Tue, Jun 28, 2016 at 4:16 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
> Hello,
>
> I am taking data from a .txt file and spliting words and inserting it into 
> database.
>
> - Original Message -----
> From: "Pradeep A. Dalvi" <p...@apache.org>
> To: "users" <users@apex.apache.org>
> Sent: Tuesday, June 28, 2016 4:12:31 PM
> Subject: Re: Jdbcoutputoperator implementation
>
> What is the type of data emitted by previous operator to 
> JdbcPOJOOutputOperator?
>
> On Tue, Jun 28, 2016 at 3:46 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>> Hello,
>>
>> I am getting an error as follow:
>>
>> Stopped running due to an exception. java.lang.ClassCastException: 
>> java.lang.String cannot be cast to com.mycompany.test.PojoEvent
>> at SC.get(Unknown Source)
>> at 
>> com.datatorrent.lib.db.jdbc.JdbcPOJOOutputOperator.setStatementParameters(JdbcPOJOOutputOperator.java:175)
>> at 
>> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.processBatch(AbstractJdbcTransactionableOutputOperator.java:114)
>> at 
>> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.endWindow(AbstractJdbcTransactionableOutputOperator.java:93)
>> at 
>> com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:145)
>> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:351)
>> at 
>> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)
>>
>> I am not sure where to check the cast mismatch. I am inserting a value in 
>> the database.
>>
>> Thank You,
>>
>> Jaikit Jilka
>>
>> - Original Message -
>> From: "Pradeep A. Dalvi" <p...@apache.org>
>> To: "users" <users@apex.apache.org>
>> Sent: Friday, June 24, 2016 4:45:18 PM
>> Subject: Re: Jdbcoutputoperator implementation
>>
>> What are errors/exceptions seen in JdbcOutput container logs?
>> Are you setting Context.PortContext.TUPLE_CLASS to POJO class being used?
>>
>> --
>> prad
>>
>> On Fri, Jun 24, 2016 at 2:13 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>>> Hello,
>>>
>>> I resolved that error and now I can launch the application, But now the 
>>> container which has jdbcoperator is getting killed. I am attaching the 
>>> dt.log file of container which is running but does not have any operator.
>>>
>>> Thank You,
>>>
>>> Jaikit Jilka
>>>
>>> - Original Message -
>>> From: "Pradeep A. Dalvi" <p...@apache.org>
>>> To: "users" <users@apex.apache.org>
>>> Sent: Friday, June 24, 2016 11:41:47 AM
>>> Subject: Re: Jdbcoutputoperator implementation
>>>
>>> Yes, because it is not yet released. Compile & install latest malhar
>>> in local ~/.m2 directory.
>>> For that, git clone Malhar library repository. And do 'mvn clean
>>> install' in that malhar clone.
>>>
>>> On Fri, Jun 24, 2016 at 11:31 AM, Jaikit Jilka <jji...@leadferret.com> 
>>> wrote:
>>>> Hello,
>>>>
>>>> When I change my malher version in POM.xml to the latest i.e 
>>>> 3.5.0-SNAPSHOT it gives me an build time error : Failure to find 
>>>> org.apache.apex:malhar-library:jar:3.5.0-SNAPSHOT in 
>>>> https://www.datatorrent.com/maven/content/repositories/snapshots/.
>>>>
>>>> Thank You,
>>>>
>>>> Jaikit Jilka
>>>>
>>>> - Original Message -
>>>> From: "Pradeep A. Dalvi" <p...@apache.org>
>>>> To: "users" <users@apex.apache.org>
>>>> Sent: Friday, June 24, 2016 11:13:42 AM
>>>> Subject: Re: Jdbcoutputoperator implementation
>>>>
>>>> This issue was in 3.4 malhar and fixed later on. Fix is present in
>>>> master branch. You can use latest malhar.
>>>> For reference: 
>>>> https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/db/jdbc/AbstractJdbcTransactionableOutputOperator.java#L142
>>>>
>>>> On Fri, Jun 24, 2016 at 10:59 AM, Jaikit Jilka <jji...@leadferret.com> 
>>>>

Re: Jdbcoutputoperator implementation

[Pradeep A. Dalvi]

What is the type of data emitted by previous operator to JdbcPOJOOutputOperator?

On Tue, Jun 28, 2016 at 3:46 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
> Hello,
>
> I am getting an error as follow:
>
> Stopped running due to an exception. java.lang.ClassCastException: 
> java.lang.String cannot be cast to com.mycompany.test.PojoEvent
> at SC.get(Unknown Source)
> at 
> com.datatorrent.lib.db.jdbc.JdbcPOJOOutputOperator.setStatementParameters(JdbcPOJOOutputOperator.java:175)
> at 
> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.processBatch(AbstractJdbcTransactionableOutputOperator.java:114)
> at 
> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.endWindow(AbstractJdbcTransactionableOutputOperator.java:93)
> at 
> com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:145)
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:351)
> at 
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)
>
> I am not sure where to check the cast mismatch. I am inserting a value in the 
> database.
>
> Thank You,
>
> Jaikit Jilka
>
> - Original Message -
> From: "Pradeep A. Dalvi" <p...@apache.org>
> To: "users" <users@apex.apache.org>
> Sent: Friday, June 24, 2016 4:45:18 PM
> Subject: Re: Jdbcoutputoperator implementation
>
> What are errors/exceptions seen in JdbcOutput container logs?
> Are you setting Context.PortContext.TUPLE_CLASS to POJO class being used?
>
> --
> prad
>
> On Fri, Jun 24, 2016 at 2:13 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
>> Hello,
>>
>> I resolved that error and now I can launch the application, But now the 
>> container which has jdbcoperator is getting killed. I am attaching the 
>> dt.log file of container which is running but does not have any operator.
>>
>> Thank You,
>>
>> Jaikit Jilka
>>
>> - Original Message -
>> From: "Pradeep A. Dalvi" <p...@apache.org>
>> To: "users" <users@apex.apache.org>
>> Sent: Friday, June 24, 2016 11:41:47 AM
>> Subject: Re: Jdbcoutputoperator implementation
>>
>> Yes, because it is not yet released. Compile & install latest malhar
>> in local ~/.m2 directory.
>> For that, git clone Malhar library repository. And do 'mvn clean
>> install' in that malhar clone.
>>
>> On Fri, Jun 24, 2016 at 11:31 AM, Jaikit Jilka <jji...@leadferret.com> wrote:
>>> Hello,
>>>
>>> When I change my malher version in POM.xml to the latest i.e 3.5.0-SNAPSHOT 
>>> it gives me an build time error : Failure to find 
>>> org.apache.apex:malhar-library:jar:3.5.0-SNAPSHOT in 
>>> https://www.datatorrent.com/maven/content/repositories/snapshots/.
>>>
>>> Thank You,
>>>
>>> Jaikit Jilka
>>>
>>> - Original Message -
>>> From: "Pradeep A. Dalvi" <p...@apache.org>
>>> To: "users" <users@apex.apache.org>
>>> Sent: Friday, June 24, 2016 11:13:42 AM
>>> Subject: Re: Jdbcoutputoperator implementation
>>>
>>> This issue was in 3.4 malhar and fixed later on. Fix is present in
>>> master branch. You can use latest malhar.
>>> For reference: 
>>> https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/db/jdbc/AbstractJdbcTransactionableOutputOperator.java#L142
>>>
>>> On Fri, Jun 24, 2016 at 10:59 AM, Jaikit Jilka <jji...@leadferret.com> 
>>> wrote:
>>>> Hello,
>>>>
>>>> I am setting UpdateCommand to empty string from properties.xml file still 
>>>> I am getting the same error
>>>>
>>>> 
>>>> 
>>>> dt.application.MyWordCountApplication.operator.JdbcOutput.prop.setupdateCommand
>>>>  
>>>> 
>>>>
>>>> error:
>>>>
>>>> An error occurred trying to launch the application. Server message: 
>>>> javax.validation.ConstraintViolationException: Operator JdbcOutput 
>>>> violates constraints 
>>>> [ConstraintViolationImpl{rootBean=JdbcPOJOOutputOperator{name=null}, 
>>>> propertyPath='updateCommand', message='may not be null', 
>>>> leafBean=JdbcPOJOOutputOperator{name=null}, value=null}] at 
>>>> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1680)
>>>>  at com.datatorrent.stram.StramClient.(Stra