[ANNOUNCE] Apache Apex Core 3.4.0 released

2016-05-12 Thread Thomas Weise
The Apache Apex community is pleased to announce release 3.4.0 (Apex Core). This is the first release after graduation. It adds the support for (anti-)affinity of operators, several documentation and other improvements and important bug fixes. The release has 63 resolved JIRAs. There is a backwar

Re: YARN memory settings and the Apex memory model

2016-05-12 Thread Thomas Weise
Ananth, Please have a look at: http://docs.datatorrent.com/troubleshooting/#configuring-memory Thanks, Thomas On Thu, May 12, 2016 at 4:00 AM, Ananth Gundabattula < agundabatt...@gmail.com> wrote: > Thanks Shubham. I shall bump up the memory a bit more. > > I was wondering how the operator me

Re: javax.persistence.Persistence classNotFoundError

2016-05-07 Thread Thomas Weise
gt; On Thu, May 5, 2016 at 5:35 AM, Ananth Gundabattula < > agundabatt...@gmail.com> wrote: > >> Hello Thomas, >> >> Thanks for the input. >> >> I tried both the UI and the command line dtcli client . Both of them are >> giving the same exact stack trace error

Re: Set schemaRequired differently on different operator implementations

2016-05-05 Thread Thomas Weise
different in case of ports in Apex, then it might solve the problem. >>> I'll try it out. >>> >>> Thanks. >>> ~Bhupesh >>> >>> On Mon, May 2, 2016 at 5:09 PM, Thomas Weise >>> wrote: >>> >>>> The port is called by t

Re: javax.persistence.Persistence classNotFoundError

2016-05-04 Thread Thomas Weise
Hi Ananth, One possibility is the bval resolver not considering the context class loader when processing the annotation. Do you see the same issue when launching the application through the UI? Thanks, Thomas On Mon, May 2, 2016 at 2:35 PM, Ananth Gundabattula wrote: > Hello Siyuan, > > Pleas

Re: Set schemaRequired differently on different operator implementations

2016-05-02 Thread Thomas Weise
are used by both parent and child. > > ~Bhupesh > > On Mon, May 2, 2016 at 5:00 PM, Thomas Weise > wrote: > >> Yes, you can override the port with a new port field with the same name. >> >> >> On Mon, May 2, 2016 at 4:22 PM, Bhupesh Chawda >> wrot

Re: Set schemaRequired differently on different operator implementations

2016-05-02 Thread Thomas Weise
Yes, you can override the port with a new port field with the same name. On Mon, May 2, 2016 at 4:22 PM, Bhupesh Chawda wrote: > Hi All, > > I have a base operator which is parametrized: > > > > *class OperatorBase extends BaseOperator{* > > *public final transient DefaultOutputPort output

Apache Apex announced as Top-Level Project

2016-04-25 Thread Thomas Weise
Dear Community, Apex graduation was approved by the ASF board last week and the announcement went out this morning: https://blogs.apache.org/foundation/entry/the_apache _software_foundation_announces90 https://twitter.com/TheASF/status/724538689993474048 Congratulations everyone and we are look

Re: Apex engine and malhar maven versions for Kafka 0.9 version operator

2016-04-23 Thread Thomas Weise
Ram added an example app for the new Kafka input operator: https://github.com/DataTorrent/examples/tree/master/tutorials/kafka Thanks, Thomas On Sat, Apr 16, 2016 at 10:47 PM, Ananth Gundabattula < agundabatt...@gmail.com> wrote: > Thanks for the clarification Siyuan. > > On Sun, Apr 17, 2016

Re: Out of order processing

2016-03-18 Thread Thomas Weise
Currently event time can be handled on the operator level with time bucketing (which allows for late / out of order arrival). We are planning to generalize this in the future. Thomas On Fri, Mar 18, 2016 at 12:00 PM, Christian Hellström wrote: > Ok, thanks. Are there plans to support event-tim

Re: DAG JSON Import

2016-03-14 Thread Thomas Weise
There is an example here: https://github.com/apache/incubator-apex-malhar/blob/master/demos/pi/src/main/resources/app/PiJsonDemo.json On Mon, Mar 14, 2016 at 5:32 PM, Aaron Bawcom wrote: > I've been trying to find the documentation for pupulating a DAG instance > using JSON. Is there formal

Re: resources @ incubator website

2016-03-07 Thread Thomas Weise
+1 On Mon, Mar 7, 2016 at 8:21 PM, Amol Kekre wrote: > > This is a good idea. > > Thks > Amol > > On Mon, Mar 7, 2016 at 7:05 PM, Sandesh Hegde > wrote: > >> Hello Team, >> >> How about having a single page which links to all the resources available >> for learning Apex? >> >> For examples: >>

Re: SMTP Output Operator using mailgun

2016-03-03 Thread Thomas Weise
Hi Jim, Thanks for the feedback. Would you like to contribute these changes to make the operator more extensible? http://apex.incubator.apache.org/contributing.html Thomas -- sent from mobile On Mar 3, 2016 12:43 AM, "Jim" wrote: > Yes, we have it all figured out already. > > The SmtpOuputOpe

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-03-01 Thread Thomas Weise
ect template to build the apex > application which is adding these jars to application package.. So do you > know how to avoid it? > > Regards, > —Suhas > > From: Thomas Weise > Reply-To: "users@apex.incubator.apache.org" < > users@apex.incubator.apache.org> &g

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-03-01 Thread Thomas Weise
change my pom.xml to use hadoop 2.7.1 jars? > > From: Thomas Weise > Reply-To: "users@apex.incubator.apache.org" < > users@apex.incubator.apache.org> > Date: Tuesday, March 1, 2016 at 1:28 PM > > To: "users@apex.incubator.apache.org" > Su

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-03-01 Thread Thomas Weise
Can you please check if there is a hadoop jar file inside the .apa ? On Tue, Mar 1, 2016 at 1:18 PM, Suhas Gogate wrote: > Thanks Ram, I could launch the application after correcting the property > names but just after launching application it fails as AppMaster fails two > times with following

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-02-29 Thread Thomas Weise
e .apa > package. So not sure how to add the malhar-contrib library in the > application package? > > Thanks for the help > > —Suhas > > From: Thomas Weise > Reply-To: "users@apex.incubator.apache.org" < > users@apex.incubator.apache.org> > Da

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-02-29 Thread Thomas Weise
: lib/activation-1.1.jar OK > > testing: lib/commons-compiler-2.7.8.jar OK > > testing: conf/OK > > testing: conf/my-app-conf1.xmlOK > > testing: META-INF/properties.xml OK > > No errors detected in compressed data of AdStream-1.0-S

Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

2016-02-29 Thread Thomas Weise
Suhas, Do not copy any operator libraries into the lib folder. These dependencies need to be packaged with the application into the .apa application package. The .apa will also need to contain the Kafka dependencies. Can you please share what .jar files are in your app package (unzip -l yourapp.a

Re: Kinesis Operator Help

2016-02-16 Thread Thomas Weise
Ram, The recovery path, when under the application directory, will be automatically copied to the new app directory when relaunch option is used. This is how the previous instance data is available to the new app. Thomas On Tue, Feb 16, 2016 at 5:23 PM, Munagala Ramanath wrote: > Ah, I underst

[ANNOUNCE] Apache Apex Core 3.3.0-incubating released

2016-02-08 Thread Thomas Weise
The Apache Apex community is pleased to announce release 3.3.0-incubating (Apex Core). Changes: https://github.com/apache/incubator-apex-core/blob/v3.3.0-incubating/CHANGELOG.md Apache Apex is an enterprise grade native YARN big data-in-motion platform that unifies stream and batch processing. Ap

Re: Possibility of saving checkpoints on other distributed filesystems

2016-02-02 Thread Thomas Weise
Functionality can additionally be made available as a CLI command. Just need to ensure it works correctly with YARN application status and security.

Re: Upcoming meetups

2016-02-01 Thread Thomas Weise
Another meetup with presentation about Apex: http://www.meetup.com/IMC-Pune-Meetup/events/228287594/ On Mon, Feb 1, 2016 at 1:04 PM, amol kekre wrote: > > We have a series of meetups on Apache Apex in Feb. Here is list > > @GE - > http://www.meetup.com/IoT-Industrial-Internet-Bay-Area-Meetup/ev

Re: What is the backpressure story ?

2016-01-31 Thread Thomas Weise
That's incorrect. Backpressure works when spooling is enabled (which is default). It's not handled only when you turn spooling off explicitly. On Sun, Jan 31, 2016 at 3:50 PM, Sandesh Hegde wrote: > According to Vlad, disabling the spooling will crash the buffer server > after it runs out of mem

Re: Archetype for 3.2.0

2016-01-29 Thread Thomas Weise
Ram, Please go by release tag: https://github.com/apache/incubator-apex-core/tree/v3.2.0-incubating Thanks, Thomas On Fri, Jan 29, 2016 at 9:47 AM, Munagala Ramanath wrote: > Looking at apex-app-archetype/README.md in branch release-3.2 of > incubator-apex-core > and running the following ma

Re: read from multiple kafka topics

2016-01-27 Thread Thomas Weise
, Ashwin Chandra Putta < ashwinchand...@gmail.com> wrote: > Thanks Thomas. How do I specify multiple topics on the operator? > > On Wed, Jan 27, 2016 at 5:15 PM, Thomas Weise > wrote: > >> The new operator for Kafka 0.9 can read multiple topics. >> >> >&

Re: read from multiple kafka topics

2016-01-27 Thread Thomas Weise
The new operator for Kafka 0.9 can read multiple topics. On Wed, Jan 27, 2016 at 5:02 PM, Munagala Ramanath wrote: > Since a single AbstractKafkaInputOperator has a single KafkaConsumer and > the latter has only a single topic, > it appears that the only option is to have one input operator per

Re: Stateless operator and Properties

2016-01-22 Thread Thomas Weise
Properties are part of the operator state. So if the intention is to change properties after the application was launched the operator cannot be marked stateless. On Fri, Jan 22, 2016 at 9:11 AM, Sandesh Hegde wrote: > Hello Team, > > Is it possible mark the operators stateless but the properti

Re: Making inherited operator port optional

2016-01-21 Thread Thomas Weise
You should be able to override the port. On Thu, Jan 21, 2016 at 7:18 AM, Munagala Ramanath wrote: > In OutputPortFieldAnnotation.java we have this: > > public boolean optional() default true; > > So looks like output ports are optional by default. If A annotates the > port as "optional = true

[ANNOUNCE] Apache Apex Malhar Release 3.3.0-incubating released

2016-01-18 Thread Thomas Weise
The Apache Apex community is pleased to announce release 3.3.0-incubating of the Malhar Library. Changes: https://github.com/apache/incubator-apex-malhar/blob/v3.3.0-incubating/CHANGELOG.md Apache Apex is an enterprise grade native YARN big data-in-motion platform that unifies stream processing a

Re: Questions regarding committing code to Malhar

2015-12-29 Thread Thomas Weise
Yes, they are configured to be skipped as they require manual setup. You can run them locally with -DskipTests=false Also, when you submit your pull request Travis CI will automatically run all the tests that are enabled. You would see any related problems in the build status. Thomas On Tue, Dec

Re: Lost my Code

2015-12-22 Thread Thomas Weise
https://github.com/DataTorrent/Malhar-old/pull/1496 See last comment about the branch also. On Tue, Dec 22, 2015 at 11:05 AM, David Yan wrote: > You can try searching for your commits on all branches: > > git log --all --author= > > > On Tue, Dec 22, 2015 at 10:59 AM, Perumal, Nithin > wrot