[GitHub] incubator-apex-core pull request: App not launching with HA and se...

2015-09-30 Thread vrozov
Github user vrozov commented on a diff in the pull request: https://github.com/apache/incubator-apex-core/pull/83#discussion_r40880696 --- Diff: engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java --- @@ -284,6 +279,20 @@ public void addRMDelegationToken(final S

[GitHub] incubator-apex-malhar pull request: File input operator that emits...

2015-09-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/38 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the f

Re: Import order enforced by checkstyle

2015-09-30 Thread Priyanka Gugale
Mine also +1 for single commit for import order changes. Doing this there will be clear distinction of changes made for formatting (import order change) and actual code change. On Thu, Oct 1, 2015 at 9:07 AM, Vlad Rozov wrote: > +1 to change import order in one single commit as Jenkins CI. The l

Re: October Apex Incubator Report

2015-09-30 Thread Chetan Narsude
Thomas, PPMC members include all the initial committers. -- Chetan On Wed, Sep 30, 2015 at 7:46 PM, Thomas Weise wrote: > Just to clarify, the pull requests were all from the set of initial > committers, not limited to PPMC. We did not have non-committer > contributions so far. > > Thomas > >

[GitHub] incubator-apex-core pull request: App not launching with HA and se...

2015-09-30 Thread PramodSSImmaneni
Github user PramodSSImmaneni commented on the pull request: https://github.com/apache/incubator-apex-core/pull/83#issuecomment-144621482 Vlad can you review this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

Re: Import order enforced by checkstyle

2015-09-30 Thread Vlad Rozov
+1 to change import order in one single commit as Jenkins CI. The less violations we have in the build the easier is to find a new one introduced and resolve it. It will be also less confusing for new developers to use the right import order. Thank you, Vlad On 9/30/15 12:29, Chandni Singh w

Re: [malhar-users] Re: How to use FileSplitter to read huge file

2015-09-30 Thread Chandni Singh
Hi Vk, Please find a CSV block reader here and let me know if you have questions. I have also added a test and it seems to be working fine. https://github.com/chandnisingh/Malhar/tree/examples/demos/examples/src/main/java/com/datatorrent/examples/reader Please note that the BlockReader api has

Re: October Apex Incubator Report

2015-09-30 Thread Thomas Weise
Just to clarify, the pull requests were all from the set of initial committers, not limited to PPMC. We did not have non-committer contributions so far. Thomas On Wed, Sep 30, 2015 at 3:07 PM, Chetan Narsude wrote: > Hi Justin, > > So far there has not been a pull request from anyone other th

Re: October Apex Incubator Report

2015-09-30 Thread Chris Nauroth
Yes, I agree. Thank you for following up. --Chris Nauroth On 9/30/15, 4:42 PM, "Chetan Narsude" wrote: >Chris, > > Considering that Marvin sent out an email saying that the report is not >due until another week, I think it's safe to postpone it to next >Wednesday. >This will also give us m

Re: October Apex Incubator Report

2015-09-30 Thread Chetan Narsude
Chris, Considering that Marvin sent out an email saying that the report is not due until another week, I think it's safe to postpone it to next Wednesday. This will also give us more time to absorb feedback on it. What do you think? -- Chetan On Tue, Sep 29, 2015 at 4:48 PM, Chetan Narsude

[GitHub] incubator-apex-core pull request: APEX-164 - Changed log4j level t...

2015-09-30 Thread 243826
Github user 243826 commented on the pull request: https://github.com/apache/incubator-apex-core/pull/92#issuecomment-144575089 The discussion here applies to archetype log4j.properties and not to this pull request. I'll give it until Friday morning PST before pulling it in. Please giv

Re: StreamingAppMasterService Unit testing

2015-09-30 Thread Isha Arkatkar
I was thinking of mocking the calls that we make during allocation. E.g. RegisterApplicationMasterResponse, AMRMClient.allocate call, AllocateResponse returned etc.. But yes, if there is any supporting library for the same, it would be very useful! Isha On Wed, Sep 30, 2015 at 1:26 PM, Pramod Imm

Re: [GitHub] incubator-apex-core pull request: APEX-164 - Changed log4j level t...

2015-09-30 Thread Chetan Narsude
I have a comment on David's comment. Moving this discussion to JIRA. Please use that going forward. On Wed, Sep 30, 2015 at 3:12 PM, vrozov wrote: > Github user vrozov commented on the pull request: > > > https://github.com/apache/incubator-apex-core/pull/92#issuecomment-144559147 > > Please

[GitHub] incubator-apex-core pull request: APEX-174 #resolve #comment modif...

2015-09-30 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-core/pull/95 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

[GitHub] incubator-apex-core pull request: APEX-164 - Changed log4j level t...

2015-09-30 Thread vrozov
Github user vrozov commented on the pull request: https://github.com/apache/incubator-apex-core/pull/92#issuecomment-144559147 Please don't use this pull request to discuss and vote on archetype log level. It is not part of the pull request anymore. See https://malhar.atlassian.net/br

[GitHub] incubator-apex-core pull request: APEX-164 - Changed log4j level t...

2015-09-30 Thread davidyan74
Github user davidyan74 commented on the pull request: https://github.com/apache/incubator-apex-core/pull/92#issuecomment-144555663 +1 for leaving it as DEBUG in the log4j.properties in the archetype. This is because in most cases, the archetype is used by someone who wants to try out

[GitHub] incubator-apex-core pull request: APEX-174 #resolve #comment modif...

2015-09-30 Thread chandnisingh
Github user chandnisingh commented on the pull request: https://github.com/apache/incubator-apex-core/pull/95#issuecomment-144538686 Have fixed the import order. Please review and merge --- If your project is set up for it, you can reply to this email and have your reply appear on Git

Re: StreamingAppMasterService Unit testing

2015-09-30 Thread Pramod Immaneni
Would going to Mockito route mean we have to keep up with the RMClient protocol and handle generation of responses ourselves, are there any supporting libraries in hadoop that would help us with that. On Wed, Sep 30, 2015 at 12:36 PM, Isha Arkatkar wrote: > Hi all, > >I want to add some

Re: October Apex Incubator Report

2015-09-30 Thread Hitesh Shah
The site should also have a disclaimer notice and the incubator logo ( as per http://incubator.apache.org/guides/branding.html ) thanks — Hitesh On Sep 30, 2015, at 12:52 PM, Andy Perlitch wrote: > Site has been updated with the feather and the changed copyright notice. > > Thanks, > Andy >

Re: October Apex Incubator Report

2015-09-30 Thread Andy Perlitch
Site has been updated with the feather and the changed copyright notice. Thanks, Andy On Tue, Sep 29, 2015 at 8:43 PM, Justin Mclean wrote: > Hi, > > > The community voted and selected the APEX logo which is put on the > > brand new project website at http://apex.incubator.apache.org/ > > The

Re: Thread and Container locality

2015-09-30 Thread Pramod Immaneni
With Mpmc I get upto 20M for container local. Lets sync up for doing across nodes. Thanks On Wed, Sep 30, 2015 at 8:31 AM, Vlad Rozov wrote: > Blog and presentation on algorithms behind JCTools: > > http://psy-lob-saw.blogspot.com/p/lock-free-queues.html > https://vimeo.com/100197431 > > Thank

StreamingAppMasterService Unit testing

2015-09-30 Thread Isha Arkatkar
Hi all, I want to add some unit testing for container allocation flow in StreamingAppMasterService class. Some of the scenarios like consecutive node failures etc. are hard to simulate manually and would be nice to cover that through unit testing. I am right now exploring 2 options: Mo

Re: Import order enforced by checkstyle

2015-09-30 Thread Chandni Singh
I can fix them but that should be a different change and the user to which that change is attributed should be either jenkins/ci. On Wed, Sep 30, 2015 at 12:07 PM, Vlad Rozov wrote: > How many import order violations exist now in Apex-core and Apex-malhar? > Is there a way to fix them as one sin

Re: October Apex Incubator Report

2015-09-30 Thread Chetan Narsude
Hi Justin, So far there has not been a pull request from anyone other than PPMC members. -- Chetan On Tue, Sep 29, 2015 at 8:21 PM, Justin Mclean wrote: > Hi, > > > Various metrics are as follows: > > > > +---+ > > | Metric

Re: Import order enforced by checkstyle

2015-09-30 Thread Vlad Rozov
How many import order violations exist now in Apex-core and Apex-malhar? Is there a way to fix them as one single batch using maven plugin or IDE auto format? Thank you, Vlad On 9/30/15 07:19, Chandni Singh wrote: Actually keeping static imports at the bottom didn't increase rather decreased

Re: Import order enforced by checkstyle

2015-09-30 Thread Chetan Narsude
This looks great. On Wednesday, September 30, 2015, Chandni Singh wrote: > Yes. So I think we need to change the import order so that they each IDE > can support it. > > New order: > > all java/javax imports > > org.* imports > > org.apache.* imports > > com.* > > com.datatorrent.* imports > > *

[GitHub] incubator-apex-core pull request: APEX-174 #resolve #comment modif...

2015-09-30 Thread ilooner
Github user ilooner commented on the pull request: https://github.com/apache/incubator-apex-core/pull/95#issuecomment-144469949 @chandnisingh I mispoke, the unmatched imports are actually put at the top in Netbeans. And they are not separated into a separate group by a newline they ar

Re: Thread and Container locality

2015-09-30 Thread Vlad Rozov
Blog and presentation on algorithms behind JCTools: http://psy-lob-saw.blogspot.com/p/lock-free-queues.html https://vimeo.com/100197431 Thank you, Vlad On 9/29/15 21:14, Vlad Rozov wrote: I guess yes, it should show improvement every time there is consumer/producer contention on a resource fr

Re: [malhar-users] Re: How to use FileSplitter to read huge file

2015-09-30 Thread Chandni Singh
Hi vk, I think you don't need to override readBlock() in AbstractBlockReader. A simpler way to do this will be using ReadAheadLineReaderContext as the readerContext and providing implementation of converting bytes to the CSV bean. public class CSVBeanReader extends AbstractFSBlockReader { publ

[GitHub] incubator-apex-core pull request: APEX-174 #resolve #comment modif...

2015-09-30 Thread chandnisingh
Github user chandnisingh commented on the pull request: https://github.com/apache/incubator-apex-core/pull/95#issuecomment-144431177 The import order has a change because Netbeans Order Imports plugin doesn't allow to put * imports in middle of a group. It puts them at the bottom ---

Re: Import order enforced by checkstyle

2015-09-30 Thread Chandni Singh
Yes. So I think we need to change the import order so that they each IDE can support it. New order: all java/javax imports org.* imports org.apache.* imports com.* com.datatorrent.* imports * imports all static imports I will update APEX-174. On Wed, Sep 30, 2015 at 4:38 AM, Thomas Weise

Re: Import order enforced by checkstyle

2015-09-30 Thread Chandni Singh
Actually keeping static imports at the bottom didn't increase rather decreased the existing violation in Apex-core. We have not followed any consistent style so far so this is the result of that. As and when a person touches a file they can correct the import order. Chandni On Wed, Sep 30, 2015

Re: [malhar-users] HDFS file read

2015-09-30 Thread Tushar Gosavi
Hi, Moving this thread to dev@apex. Which operator are you using for reading HDFS files? If you have written your own operator for parsing, then can you please check your parsing logic separately and make sure that it works before adding it into the operator. - Tushar. On Wed, Sep 30, 2015 at

Re: Import order enforced by checkstyle

2015-09-30 Thread Priyanka Gugale
No, there are few classes in Apex-cor and Apex-malhar e.g. StreamPersistanceTests, PojoUtilsTest etc. Even for Ingestion we used to keep static imports on top. We might have to change them once we add these templates. -Priyanka On Wed, Sep 30, 2015 at 5:08 PM, Thomas Weise wrote: > Is it a bina

Re: Import order enforced by checkstyle

2015-09-30 Thread Thomas Weise
Is it a binary file? If so, it does not belong into this repo. On Wed, Sep 30, 2015 at 2:31 AM, Chandni Singh wrote: > Pramod, > > Please advise. > > Chandni > > On Tue, Sep 29, 2015 at 10:47 PM, Chandni Singh > wrote: > > > Tim fixed the plugin. We can add the nbm file at the same location whe

Re: Import order enforced by checkstyle

2015-09-30 Thread Priyanka Gugale
Now many classes in our code base has highest priority to static imports (i.e. all static imports at top). It should be fine to move them to bottom, but it's good to keep all static imports together. +1 for proposed order. -Priyanka On Wed, Sep 30, 2015 at 12:01 PM, Chandni Singh wrote: > Pra