Hello,
I completed my application now. Till now I was launching this application in
Sandbox and it works perfectly fine. I am getting expected results. But now I
tried running it on live cluster and it is not working. It is not giving me any
error but all my operators are killed by application master. So do I need to do
anything for launching it on live cluster?
----- Original Message -----
From: "Jaikit Jilka" <jji...@leadferret.com>
To: "users" <users@apex.apache.org>
Sent: Thursday, June 30, 2016 11:29:25 AM
Subject: Re: Jdbcoutputoperator implementation
Hello,
Now my every operator is working fine. But counter operator does not emit
anthing. It is just processing the data. Here is my counter.java operator file.
----- Original Message -----
From: "Jaikit Jilka" <jji...@leadferret.com>
To: "users" <users@apex.apache.org>
Sent: Thursday, June 30, 2016 10:25:35 AM
Subject: Re: Jdbcoutputoperator implementation
Hello,
Got it. I have to create a outport of type pojo. Working on it.
----- Original Message -----
From: "Jaikit Jilka" <jji...@leadferret.com>
To: "users" <users@apex.apache.org>
Sent: Thursday, June 30, 2016 10:11:10 AM
Subject: Re: Jdbcoutputoperator implementation
Hello,
I can not emit output.emit(pojo) only emits string. It it giving me error as
not method found for emit(pojoEvent).
----- Original Message -----
From: "Yunhan Wang" <yun...@datatorrent.com>
To: "users" <users@apex.apache.org>
Sent: Wednesday, June 29, 2016 2:59:41 PM
Subject: Re: Jdbcoutputoperator implementation
Hi Jaikit,
I think the main problem here is: your Counter Operator takes String as
input and emits String as output. However, your JdbcOutputOperator needs
POJO as input. So you need to modify Counter Operator to emit the POJO
instead of String. You could do that by writing separate function e.g.
emitTuples as below:
public void emitTuples(String word)
{
PojoEvent pojo = new PojoEvent();
pojo.setName(word);
output.emit(pojo);
}
Also, you need to add this to your properties.xml:
<property>
<name>dt.application.test.operator.JdbcOutput.port.input.attr.TUPLE_CLASS</name>
<value>com.mycompany.test.PojoEvent</value>
</property>
Hope this can help.
Thanks,
Yunhan
On Wed, Jun 29, 2016 at 1:14 PM, Jaikit Jilka <jji...@leadferret.com> wrote:
> This is my POJO class. Can you please tell me what changes should I make
> into this.
>
> Thank you,
>
> Jaikit Jilka
>
>
> ----- Original Message -----
> From: "Pradeep A. Dalvi" <p...@apache.org>
> To: "users" <users@apex.apache.org>
> Sent: Tuesday, June 28, 2016 4:56:57 PM
> Subject: Re: Jdbcoutputoperator implementation
>
> By adding a string member variable to POJO class. Use that member
> variable to send the data to JdbcPOJOOutputOperator.
>
> On Tue, Jun 28, 2016 at 4:34 PM, Jaikit Jilka <jji...@leadferret.com>
> wrote:
> > So how to convert a java string into POJO?
> >
> > ----- Original Message -----
> > From: "Pradeep A. Dalvi" <p...@apache.org>
> > To: "users" <users@apex.apache.org>
> > Sent: Tuesday, June 28, 2016 4:30:09 PM
> > Subject: Re: Jdbcoutputoperator implementation
> >
> > JdbcPOJOOutputOperator expects data as POJO, whose class name is
> > specified using Context.PortContext.TUPLE_CLASS i.e.
> > com.mycompany.test.PojoEvent here. Previous operator is probably
> > sending String to JdbcPOJOOutputOperator.
> >
> > On Tue, Jun 28, 2016 at 4:16 PM, Jaikit Jilka <jji...@leadferret.com>
> wrote:
> >> Hello,
> >>
> >> I am taking data from a .txt file and spliting words and inserting it
> into database.
> >>
> >> ----- Original Message -----
> >> From: "Pradeep A. Dalvi" <p...@apache.org>
> >> To: "users" <users@apex.apache.org>
> >> Sent: Tuesday, June 28, 2016 4:12:31 PM
> >> Subject: Re: Jdbcoutputoperator implementation
> >>
> >> What is the type of data emitted by previous operator to
> JdbcPOJOOutputOperator?
> >>
> >> On Tue, Jun 28, 2016 at 3:46 PM, Jaikit Jilka <jji...@leadferret.com>
> wrote:
> >>> Hello,
> >>>
> >>> I am getting an error as follow:
> >>>
> >>> Stopped running due to an exception. java.lang.ClassCastException:
> java.lang.String cannot be cast to com.mycompany.test.PojoEvent
> >>> at SC.get(Unknown Source)
> >>> at
> com.datatorrent.lib.db.jdbc.JdbcPOJOOutputOperator.setStatementParameters(JdbcPOJOOutputOperator.java:175)
> >>> at
> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.processBatch(AbstractJdbcTransactionableOutputOperator.java:114)
> >>> at
> com.datatorrent.lib.db.jdbc.AbstractJdbcTransactionableOutputOperator.endWindow(AbstractJdbcTransactionableOutputOperator.java:93)
> >>> at
> com.datatorrent.stram.engine.GenericNode.processEndWindow(GenericNode.java:145)
> >>> at
> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:351)
> >>> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1388)
> >>>
> >>> I am not sure where to check the cast mismatch. I am inserting a value
> in the database.
> >>>
> >>> Thank You,
> >>>
> >>> Jaikit Jilka
> >>>
> >>> ----- Original Message -----
> >>> From: "Pradeep A. Dalvi" <p...@apache.org>
> >>> To: "users" <users@apex.apache.org>
> >>> Sent: Friday, June 24, 2016 4:45:18 PM
> >>> Subject: Re: Jdbcoutputoperator implementation
> >>>
> >>> What are errors/exceptions seen in JdbcOutput container logs?
> >>> Are you setting Context.PortContext.TUPLE_CLASS to POJO class being
> used?
> >>>
> >>> --
> >>> prad
> >>>
> >>> On Fri, Jun 24, 2016 at 2:13 PM, Jaikit Jilka <jji...@leadferret.com>
> wrote:
> >>>> Hello,
> >>>>
> >>>> I resolved that error and now I can launch the application, But now
> the container which has jdbcoperator is getting killed. I am attaching the
> dt.log file of container which is running but does not have any operator.
> >>>>
> >>>> Thank You,
> >>>>
> >>>> Jaikit Jilka
> >>>>
> >>>> ----- Original Message -----
> >>>> From: "Pradeep A. Dalvi" <p...@apache.org>
> >>>> To: "users" <users@apex.apache.org>
> >>>> Sent: Friday, June 24, 2016 11:41:47 AM
> >>>> Subject: Re: Jdbcoutputoperator implementation
> >>>>
> >>>> Yes, because it is not yet released. Compile & install latest malhar
> >>>> in local ~/.m2 directory.
> >>>> For that, git clone Malhar library repository. And do 'mvn clean
> >>>> install' in that malhar clone.
> >>>>
> >>>> On Fri, Jun 24, 2016 at 11:31 AM, Jaikit Jilka <jji...@leadferret.com>
> wrote:
> >>>>> Hello,
> >>>>>
> >>>>> When I change my malher version in POM.xml to the latest i.e
> 3.5.0-SNAPSHOT it gives me an build time error : Failure to find
> org.apache.apex:malhar-library:jar:3.5.0-SNAPSHOT in
> https://www.datatorrent.com/maven/content/repositories/snapshots/.
> >>>>>
> >>>>> Thank You,
> >>>>>
> >>>>> Jaikit Jilka
> >>>>>
> >>>>> ----- Original Message -----
> >>>>> From: "Pradeep A. Dalvi" <p...@apache.org>
> >>>>> To: "users" <users@apex.apache.org>
> >>>>> Sent: Friday, June 24, 2016 11:13:42 AM
> >>>>> Subject: Re: Jdbcoutputoperator implementation
> >>>>>
> >>>>> This issue was in 3.4 malhar and fixed later on. Fix is present in
> >>>>> master branch. You can use latest malhar.
> >>>>> For reference:
> https://github.com/apache/apex-malhar/blob/master/library/src/main/java/com/datatorrent/lib/db/jdbc/AbstractJdbcTransactionableOutputOperator.java#L142
> >>>>>
> >>>>> On Fri, Jun 24, 2016 at 10:59 AM, Jaikit Jilka <
> jji...@leadferret.com> wrote:
> >>>>>> Hello,
> >>>>>>
> >>>>>> I am setting UpdateCommand to empty string from properties.xml file
> still I am getting the same error
> >>>>>>
> >>>>>> <property>
> >>>>>>
>
> <name>dt.application.MyWordCountApplication.operator.JdbcOutput.prop.setupdateCommand</name>
> >>>>>> <value> </value>
> >>>>>> </property>
> >>>>>>
> >>>>>> error:
> >>>>>>
> >>>>>> An error occurred trying to launch the application. Server message:
> javax.validation.ConstraintViolationException: Operator JdbcOutput violates
> constraints
> [ConstraintViolationImpl{rootBean=JdbcPOJOOutputOperator{name=null},
> propertyPath='updateCommand', message='may not be null',
> leafBean=JdbcPOJOOutputOperator{name=null}, value=null}] at
> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1680)
> at com.datatorrent.stram.StramClient.<init>(StramClient.java:161) at
> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:509)
> at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2050)
> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3456) at
> com.datatorrent.stram.cli.DTCli.access$7100(DTCli.java:106) at
> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1895) at
> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449)
> >>>>>>
> >>>>>> Thank You,
> >>>>>>
> >>>>>> Jaikit Jilka
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>> From: "Munagala Ramanath" <r...@datatorrent.com>
> >>>>>> To: "users" <users@apex.apache.org>
> >>>>>> Sent: Thursday, June 23, 2016 3:32:42 PM
> >>>>>> Subject: Re: Jdbcoutputoperator implementation
> >>>>>>
> >>>>>> Please look at:
> >>>>>>
> >>>>>> http://docs.datatorrent.com/beginner/ and
> >>>>>> http://docs.datatorrent.com/application_packages/
> >>>>>>
> >>>>>> for examples on how to set properties from XML files.
> >>>>>>
> >>>>>> Ram
> >>>>>>
> >>>>>> On Thu, Jun 23, 2016 at 3:25 PM, Jaikit Jilka <
> jji...@leadferret.com> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> How to pass an empty string for updatecommand in properties.xml.
> >>>>>>> When I change my malhar version to 3.5.0-SNAPSHOT it gives me a
> complie
> >>>>>>> time error -Failure to find
> >>>>>>> org.apache.apex:malhar-library:jar:3.5.0-SNAPSHOT in
> >>>>>>> https://www.datatorrent.com/maven/content/repositories/snapshots/
> >>>>>>>
> >>>>>>> Thank You,
> >>>>>>>
> >>>>>>> Jaikit Jilka
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>> From: "Pradeep A. Dalvi" <p...@apache.org>
> >>>>>>> To: "users" <users@apex.apache.org>
> >>>>>>> Sent: Thursday, June 23, 2016 12:43:40 PM
> >>>>>>> Subject: Re: Jdbcoutputoperator implementation
> >>>>>>>
> >>>>>>> Hi Jaikit,
> >>>>>>>
> >>>>>>> This constraint violation exception was enforced from
> >>>>>>> AbstractJdbcTransactionableOutputOperator. In 3.3/3.4 malhar
> library,
> >>>>>>> updateCommand parameter has constraint set to @NotNull on getter
> >>>>>>> method.
> >>>>>>>
> >>>>>>> You could do either of following to solve this:
> >>>>>>> 1. Set updateCommand from properties.xml to empty string
> >>>>>>> 2. Use master branch of Malhar
> >>>>>>> 3. Remove @NotNull on getUpdateCommand in
> >>>>>>> AbstractJdbcTransactionableOutputOperator
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Pradeep A. Dalvi
> >>>>>>>
> >>>>>>> On Thu, Jun 23, 2016 at 11:36 AM, Jaikit Jilka <
> jji...@leadferret.com>
> >>>>>>> wrote:
> >>>>>>> > Hi Devendra,
> >>>>>>> >
> >>>>>>> > I tried to run your application. I followed all steps mentioned
> but I am
> >>>>>>> getting following error.
> >>>>>>> >
> >>>>>>> > An error occurred trying to launch the application. Server
> message:
> >>>>>>> javax.validation.ConstraintViolationException: Operator JdbcOutput
> violates
> >>>>>>> constraints
> >>>>>>>
> [ConstraintViolationImpl{rootBean=JdbcPOJOOutputOperator{name=null},
> >>>>>>> propertyPath='updateCommand', message='may not be null',
> >>>>>>> leafBean=JdbcPOJOOutputOperator{name=null}, value=null}] at
> >>>>>>>
> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1680)
> >>>>>>> at com.datatorrent.stram.StramClient.<init>(StramClient.java:161)
> at
> >>>>>>>
> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:509)
> >>>>>>> at
> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2050)
> >>>>>>> at
> com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3456) at
> >>>>>>> com.datatorrent.stram.cli.DTCli.access$7100(DTCli.java:106) at
> >>>>>>>
> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1895) at
> >>>>>>> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449)
> >>>>>>> >
> >>>>>>> > Thank you,
> >>>>>>> >
> >>>>>>> > Jaikit Jilka
> >>>>>>> >
> >>>>>>> > ----- Original Message -----
> >>>>>>> > From: "Devendra Tagare" <devend...@datatorrent.com>
> >>>>>>> > To: "users" <users@apex.apache.org>
> >>>>>>> > Sent: Thursday, June 23, 2016 10:32:17 AM
> >>>>>>> > Subject: Re: Jdbcoutputoperator implementation
> >>>>>>> >
> >>>>>>> > Hi Jalkit,
> >>>>>>> >
> >>>>>>> > The JdbcPOJOOutputOperator is being configured with the
> >>>>>>> > JdbcTransactionalStore which requires an additional dt_meta
> table.
> >>>>>>> >
> >>>>>>> > The table schema is here -
> >>>>>>> >
> >>>>>>>
> https://github.com/devtagare/examples/blob/SPOI-8251/tutorials/jdbcToJdbc/src/test/resources/example.sql
> >>>>>>> >
> >>>>>>> > You can find an example application which uses the
> JdbcPOJOOutputOperator
> >>>>>>> > here -
> >>>>>>> >
> >>>>>>>
> https://github.com/devtagare/examples/tree/SPOI-8251/tutorials/jdbcToJdbc
> >>>>>>> >
> >>>>>>> > Thanks,
> >>>>>>> > Dev
> >>>>>>> >
> >>>>>>> > On Thu, Jun 23, 2016 at 10:17 AM, Jaikit Jilka <
> jji...@leadferret.com>
> >>>>>>> > wrote:
> >>>>>>> >
> >>>>>>> >> Hello,
> >>>>>>> >>
> >>>>>>> >> I am currently working on an application which simply read data
> from
> >>>>>>> .txt
> >>>>>>> >> file and slpits the words and inserts them into mysql. But I am
> getting
> >>>>>>> >> error. I am using jdbcpojooutputoperator from malhar library.
> But I am
> >>>>>>> >> getting an error on ActiveFieldInfos which says it is protected
> in
> >>>>>>> >> jdbcpojoinputoperator and also I do not know what to do in
> >>>>>>> Application.java
> >>>>>>> >> file. I am attaching both the file here. Kindly help me resolve
> the
> >>>>>>> issue.
> >>>>>>> >>
> >>>>>>> >> Thank You,
> >>>>>>> >>
> >>>>>>> >> Jaikit Jilka
> >>>>>>>
>
