Re: Stray folders under Malhar/lib

Chetan, I think you have mis understood my comments. Since I proposed a change in StramLocalCluster which is in platform it is quite evident that I do not want the application test developer to be concerned about StorageAgent. I dont see a use of async writer in local mode. In fact 2 step write

Re: Stray folders under Malhar/lib

The problem is the same as BufferServer (BS) needing to write the temp files. The BS does not write the temp files in stray locations even though each invocation of StramLocalCluster triggers BS initialization. It checks with the context for a temporary location automatically to decide a temporary

Re: Stray folders under Malhar/lib [was Re: Vagrant folders under Malhar/lib]

Just changing the subject here. -- Chetan On Thu, Sep 3, 2015 at 3:35 PM, Thomas Weise wrote: > Ram, > > Are you referring to running your app from the dtcli? > > That's one more item to check to not end up with stray files. There is > nothing stopping us from using a different default for tha

Re: Vagrant folders under Malhar/lib

Yes Thomas, when I run my app with *dtcli *I'm seeing all those stray directories. I agree that re-configuring a whole bunch of existing tests is not something we want to do. The key question seems to be: *Can we do something in the platform so that both objectives are achieved*, namely: (a) Exis

Re: Incubator PMC/Board report for Sep 2015 ([ppmc])

Digged into a few earlier filed INFRA jiras. More info: - https://blogs.apache.org/infra/entry/git_based_websites_available - Example jira:https://issues.apache.org/jira/browse/INFRA-9559 A questions for the community: I assume there is a single website for apex, where would you prefer t

Re: Vagrant folders under Malhar/lib

Ram, Are you referring to running your app from the dtcli? That's one more item to check to not end up with stray files. There is nothing stopping us from using a different default for that use case. My take is that we should structure this in a way where by default unit tests have minimum things

Re: Incubator PMC/Board report for Sep 2015 ([ppmc])

Thanks a lot Hitesh! I have opened up INFRA-10249 . Hopefully I can get some response there and continue my effort. Best, Andy On Thu, Sep 3, 2015 at 3:01 PM, Hitesh Shah wrote: > For Tez, the general approach we followed was using svnpubsub.

Re: Vagrant folders under Malhar/lib

It's not just unit tests. An app developer is likely to run a random app in LM to uncover bugs before hitting the cluster. The closer the LM setup is to the cluster setup (i.e. running as much of the same code as reasonably possible) the higher the probability that bugs will be hit in LM. Ram On

Re: Vagrant folders under Malhar/lib

Yes Chetan, I am claiming that :-) I still don't understand the need for having two step checkpointing in LocalMode by default. StramLocalCluster should simplify test execution environment as pointed out by Thomas. Async checkpoint should have its own test cases using StramLocalCluster that shoul

Re: Incubator PMC/Board report for Sep 2015 ([ppmc])

For Tez, the general approach we followed was using svnpubsub. What this entails is the following: - use an svn location to host the content of the website. e.g http://svn.apache.org/viewvc/tez/site/. If the apex svn dir needs creation, one of the mentors can probably create one within the

Re: Vagrant folders under Malhar/lib

Changing the storage agent is one of the ways to address the symptoms of the problem. But it's not treating the problem. In this case - change the basePath to a location under target and all the opinions are moot. And someone is claiming that we should not do it. Not sure why. Or is anyone claimin

Re: Incubator PMC/Board report for Sep 2015 ([ppmc])

I created APEX-90 to track progress on the incubator website. Thomas, I read through http://incubator.apache.org/guides/sites.html, however it is not entirely clear to me how to "initialize" the podling site. I have a feeling that a new svn repo needs t

Re: Can Partitioners/StatsListeners be shared between Operators?

Yes! As a matter of fact - we do use this pattern frequently as stats frequently influence need for partitioning. -- Chetan On Thu, Sep 3, 2015 at 11:54 AM, Ganelin, Ilya wrote: > Hello – I’m wondering if the following code is safe. Are there any > potential issues with sharing a partitioner or

Re: Modules support in Apex

Atri, Dynamic changes were in the plans. We missed putting it in open source jira. Please open a jira in APEX for dynamic changes to DAG. Thks, Amol On Thu, Sep 3, 2015 at 1:09 PM, Atri Sharma wrote: > > On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma > wrote: > > > > > Thanks for the detailed ex

Re: Possible to gracefully shutdown local cluster?

Hello, Here is an example of how a test that uses LocalMode can be shutdown gracefully. https://github.com/chandnisingh/Malhar/blob/examples/library/src/test/java/com/datatorrent/lib/io/fs/ApplicationTest.java Thanks, Chandni On Thu, Sep 3, 2015 at 1:40 PM, Chetan Narsude wrote: > Hi Ilya, >

Re: Vagrant folders under Malhar/lib

Good point regarding the coverage. These JUnit tests are supposed to test individual components and all the tests collectively should strive to achieve high coverage. There are tests in Apex to cover storage agents, recovery semantics etc. Components that fall outside of the test scope are reduced

Re: Visualize custom metrics in the dashboard

Got it - thanks for the in-depth explanation. I’ll work through this and let you know how it goes. Thanks! On 9/3/15, 1:26 PM, "Timothy Farkas" wrote: >Hey Ilya, > >We can discuss the simple AppData operators present in Malhar in this >thread. Operators which can perform aggregations over time b

Re: Possible to gracefully shutdown local cluster?

Hi Ilya, It looks like the input operator is taking too long to do finish emitTuples. You can look at StreamingContainer.undeploy call to see what's happening soon after lc.shutdown is called. From the error it looks like the input operator is so busy emitting that it's even ignoring the interru

Re: Visualize custom metrics in the dashboard

Hey Ilya, We can discuss the simple AppData operators present in Malhar in this thread. Operators which can perform aggregations over time buckets, store results to hdfs using hdht, and make the aggregations available for visualization are not open source and will have to be discussed offline if y

Re: Possible to gracefully shutdown local cluster?

Ilya, In your code there is a hard stop after 15s. There are other options to do this: - The application itself exits when it is done, that is if the input operator raises the ShutdownException which leads to graceful termination. - In your test code, let the cluster run asynchronously,

Re: Modules support in Apex

> On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma wrote: > > > Thanks for the detailed explanation! > > > > I have few small questions. Please find them inline. > > > > On 3 Sep 2015 21:03, "Amol Kekre" wrote: > > > > > Design time -> design, try, iterate => Re-use of IP saves a lot of time > > > he

Re: Modules support in Apex

Atri, I added my comments inline Thks, Amol On Thu, Sep 3, 2015 at 11:55 AM, Atri Sharma wrote: > Thanks for the detailed explanation! > > I have few small questions. Please find them inline. > > On 3 Sep 2015 21:03, "Amol Kekre" wrote: > > > Design time -> design, try, iterate => Re-use of IP

Possible to gracefully shutdown local cluster?

Hello all – I’m using the following code to execute a topology locally: try { LocalMode lma = LocalMode.newInstance(); Configuration conf = new Configuration(false); conf.addResource(this.getClass().getResourceAsStream("/META-INF/properties.xml")); lma.prepareDAG(new Application()

Re: Modules support in Apex

Thanks for the detailed explanation! I have few small questions. Please find them inline. On 3 Sep 2015 21:03, "Amol Kekre" wrote: > Design time -> design, try, iterate => Re-use of IP saves a lot of time > here. Operators help as leaf level IP, Modules help as higher level IP that > can be mad

Can Partitioners/StatsListeners be shared between Operators?

Hello – I’m wondering if the following code is safe. Are there any potential issues with sharing a partitioner or StatsListener in this manner?: FeatureGeneration featureGenOne = dag.addOperator("FeatureGenerator I", FeatureGeneration.class); // setup dynamic partitioning for the FeatureG

Re: Vagrant folders under Malhar/lib

I think Ram explained in a little more detail on what I am thinking. Tests are supposed to provide code coverage. Having localcluster is already a variable, it's not what runs in production. Having a different storage agent is another variable and it misses out on testing the asynchronous flow. Th

Re: Vagrant folders under Malhar/lib

The local mode was so far using FSStorageAgent which was used in production. In production using Async is needed because hdfs writes are slow but is that the case with LocalMode? In local mode if we use Async we are creating checkpoints under one local directory and then copying it to another loca

Re: POJO operators

Thanks Chinmay, these operators will be really useful. On Thu, Sep 3, 2015 at 1:25 AM, Chinmay Kolhatkar wrote: > Hi Sandesh, > > I've created a JIRA for the same which I'm working on. > https://malhar.atlassian.net/browse/MLHR-1838 > > I'll be uploading the details about design soon on JIRA. >

Re: Vagrant folders under Malhar/lib

That sounds a lot like self contradicting reason; Let's make a change because we don't want to make change. :-) The code is in certain state. This certain state is consistent with how things run in production. In test environment there is a problem that stray files are created. It's a small fix to

Re: Vagrant folders under Malhar/lib

There is something to be said for the argument that Local Mode (LM) should, as far as possible, use the same code as non-LM so that more bugs are uncovered earlier. On the other hand I'm seeing stray folders 1, 2, 3, etc. and target/com.datatorrent.stram.StramLocalCluster/ when running in LM and t

Re: Vagrant folders under Malhar/lib

There is no need to configure anything extra with the proposed change, it just brings back LM to how it worked before. There is no point modifying n tests for extra setup with no gain. Thomas On Thu, Sep 3, 2015 at 9:14 AM, Chetan Narsude wrote: > Why does it matter that AsyncFSStorageAgent is

Re: Vagrant folders under Malhar/lib

Why does it matter that AsyncFSStorageAgent is being used with LocalCluster? It using the localfs and hence no gain is the implementation detail that's abstracted out by FileSystem already. If there is a problem of random artifacts left behind after the test, there is a reason and most likely it's

Re: Vagrant folders under Malhar/lib

Clean up container files left over should be a distributed OS task. Clean up, back up, archive, ... all is for the OS (aka YARN). We must assume kill -9. The only thing where the operator comes into play is "teardown()", which is business logic (not Apex engine) issue. This could be db connection

Re: Vagrant folders under Malhar/lib

BTW if anyone can find a solution that will detect JUnit environment and automatically set the application path to a test specific directory under target without the test having to do anything extra, that would be great. On Thu, Sep 3, 2015 at 8:42 AM, Thomas Weise wrote: > It makes sense to use

Re: Vagrant folders under Malhar/lib

When the container gets killed, we should not assume anything about cleanup. It can be a kill -9. Any related "cleanup" falls under nice to have, no guarantees. On Thu, Sep 3, 2015 at 8:49 AM, Chandni Singh wrote: > I have a question regarding what Gaurav mentioned > > When container runs i

Re: Vagrant folders under Malhar/lib

I have a question regarding what Gaurav mentioned When container runs in cluster, "." specifies the containers local path on the node where container specific jars and other resources resides. It creates a folder under that which is live as long as container lives. So there are no vagrant fold

Re: Vagrant folders under Malhar/lib

It makes sense to use the synchronous checkpointing for the local mode. LM is meant to simplify dependencies and setup. The default for execution on YARN remains async. Thomas On Thu, Sep 3, 2015 at 8:34 AM, Chandni Singh wrote: > APPLICATION_PATH isn't related to local base dir of Async as fa

Re: Vagrant folders under Malhar/lib

APPLICATION_PATH isn't related to local base dir of Async as far as I know. StramLocalCluster sets the APP_PATH to "target/...". StramLocalCluster should use FSStorageAgent. - Chandni On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta wrote: > As Thomas mentioned as default remains to be async. You c

Re: Modules support in Apex

Atri, Great questions. Module flattening will mainly happen at launch time. Here is the flow Design time -> Launch time -> Run Time // QA, tests, benchmarking, etc. is orthogonal to these, as each of them will walk through the same flow Design time -> design, try, iterate => Re-use of IP saves a

Re: Vagrant folders under Malhar/lib

As Thomas mentioned as default remains to be async. You can either change the storage agent or set the APPLICATION_PATH. When container runs in cluster, "." specifies the containers local path on the node where container specific jars and other resources resides. It creates a folder under that whi

RE: Visualize custom metrics in the dashboard

Correct Thank you, Ilya Ganelin -Original Message- From: Timothy Farkas [t...@datatorrent.com] Sent: Thursday, September 03, 2015 03:41 AM Eastern Standard Time To: dev@apex.incubator.apache.org Subject: Re: Visualize custom metrics in the dashboard Cool

Re: APEX-67 issue

Hi Anuj, I've very basic questions. As httpReq.getCookies() returned null, I suspect HTTP Client is neither creating cookies nor forwarding them. >From request header it appears that Apache HTTP client (Apache-HttpClient/4.3.5 (java 1.5)) is being used to fire HTTP request. Are there any cookies

Re: POJO operators

Hi Sandesh, I've created a JIRA for the same which I'm working on. https://malhar.atlassian.net/browse/MLHR-1838 I'll be uploading the details about design soon on JIRA. -Chinmay. On Wed, Sep 2, 2015 at 11:51 PM, Sandesh Hegde wrote: > Hello Team, > > Do we have the following operators > > C

Re: Modules support in Apex

Amol. For my understanding, when you mention launch time/code generation time, are you referring to generation of physical plan, please? Regards, Atri On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre wrote: > Atri, > For a lot of operations module should be treated as a black box. It is just > ano

Re: Visualize custom metrics in the dashboard

Cool just to check though do you have the enterprise license for 3.0? 2.0 does not include the visualization tools. Thanks, Tim On Wed, Sep 2, 2015 at 11:57 PM, Ganelin, Ilya wrote: > Hi Timothy - we do actually have an enterprise license and would > appreciate more details on how to use those

Re: Modules support in Apex

Hi, Thanks for the explanation. Makes a lot more sense now. I like the idea of Modules especially their abstraction for APEX-3. I have few questions which I have marked inline please. On Thu, Sep 3, 2015 at 12:48 PM, Amol Kekre wrote: > Atri, > For a lot of operations module should be treated

Re: Modules support in Apex

Atri, For a lot of operations module should be treated as a black box. It is just another reusable IP. The flattening should happen at launch time. If we think of Apex as a compiler, then all the compile time checks (ports connectivity, matching types/schema, properties, attributes, ...) are as ap

Re: Modules support in Apex

So the idea around our APEX-3 work will be that we will implement Module interface to build a class that adds operators at runtime? Sounds like a good idea, if Module is essentially a set of operators plugged in DAG. Is Module to be treated like a black box with input and output ports, and the int