Write access to Apex/Malhar GH

2017-05-05 Thread Ganelin, Ilya
Hello, all – I was attempting to close out one of the open PRs and it seems I may not have write access to the repos on GH. How do I resolve that? Do folks typically merge patches through the GH UI or just use the GIT command line? - Ilya Ganelin [id:image001.png@01D1F7A4.F3D42980] _

Re: Spurious test failures in master

2017-04-28 Thread Ganelin, Ilya
Minor clarification: failures are in apex-malhar: - Ilya Ganelin [id:image001.png@01D1F7A4.F3D42980] From: "Ganelin, Ilya" Reply-To: "dev@apex.apache.org" Date: Friday, April 28, 2017 at 11:18 PM To: "dev@apex.apache.org" Subject: Spurious test failures in mas

Spurious test failures in master

2017-04-28 Thread Ganelin, Ilya
Jenkins is failing on two PRs with seemingly unrelated errors to the contents of the PR. Are the following already tracked somewhere in JIRA? 1) Error Message Unable to load AWS credentials from the /home/jenkins/jenkins-slave/workspace/Apex_Malhar_PR%402/library/target/test-classes/sqstestCreds

Dusting off old PRs

2017-04-28 Thread Ganelin, Ilya
Hi all – have been reviewing some long stagnant code now that I’ve had a moment to breathe and return to OSS ☺ Would be much obliged if folks could take a glance at the following PRs, both of which should be in a near ready state, complete with tests. https://github.com/apache/apex-malhar/pull/

Re: PR merge policy

2017-04-28 Thread Ganelin, Ilya
Guess we can all go home then. Our work here is done: [cid:image001.png@01D2C024.5F2A5090] W.R.T the discussion below, I think rolling back an improperly reviewed PR could be considered disrespectful to the committer who merged it in the first place. I think that such situations, unless they tr

Re: Why does emit require current thread to be the operator thread?

2017-04-10 Thread Ganelin, Ilya
> > > On Mon, Apr 10, 2017 at 1:01 PM, Ganelin, Ilya < > ilya.gane...@capitalone.com> > wrote: > > > Thanks, Amol – that makes sense and was the solution I’d arrived at. I > > just was trying to avoid the delay between the data bei

Re: Why does emit require current thread to be the operator thread?

2017-04-10 Thread Ganelin, Ilya
amolhkekre* www.datatorrent.com On Mon, Apr 10, 2017 at 12:39 PM, Ganelin, Ilya wrote: > Hello – I’ve got an operator that runs a cleanup thread (separate from the > main event loop) and triggers a callback when an item is removed from an > internal data stru

Why does emit require current thread to be the operator thread?

2017-04-10 Thread Ganelin, Ilya
Hello – I’ve got an operator that runs a cleanup thread (separate from the main event loop) and triggers a callback when an item is removed from an internal data structure. I would like for this callback to emit data from one of the operator’s ports, but I run into the following Exception: (Fro

Re: [GitHub] apex-malhar pull request #587: APEXMALHAR-2453 Sort Accumulation for Windowe...

2017-03-21 Thread Ganelin, Ilya
To unsubscribe please follow the links here: https://apex.apache.org/community.html - Ilya Ganelin On 3/21/17, 1:29 PM, "Herger, Brendan" wrote: unsubscribe Thanks, Brendan Herger Machine Learning Engineer Center for Machine Learning @ Capital One 415.582.7457 (cel

Re: NO_LOCAL_WRITE Error from Stram

2017-03-03 Thread Ganelin, Ilya
Minor amendment: hadoop-2.6.0+cdh5.8.0+1592 (2.6 vs 2.7) - Ilya Ganelin [id:image001.png@01D1F7A4.F3D42980] From: "Ganelin, Ilya" Reply-To: "dev@apex.apache.org" Date: Friday, March 3, 2017 at 1:59 PM To: "dev@apex.apache.org" Subject: NO_LOCAL_WRITE Error from

NO_LOCAL_WRITE Error from Stram

2017-03-03 Thread Ganelin, Ilya
Hello, all – new error cropping up at Application start and Google does not offer any helpful guidance. 2017-03-03 16:49:02,558 INFO com.datatorrent.common.util.AsyncFSStorageAgent: using /opt/cloudera/hadoop/1/yarn/nm/usercache/XX/appcache/application_1483979920683_0448/container_e97905_14

Re: FailoverProxyProvider Error on Launch

2017-03-03 Thread Ganelin, Ilya
h Please check that your application package does not include hadoop jars. Thank you, Vlad Join us at Apex Big Data World-San Jose<http://www.apexbigdata.com/san-jose.html>, April 4, 2017 [ttp://www.apexbigdata.com/san-jose-register.html]<http://www.apexbigdata.com/san-jose-register.h

FailoverProxyProvider Error on Launch

2017-03-03 Thread Ganelin, Ilya
Hi, all – my application submits to the gateway, is accepted, but then fails with: 017-03-03 14:30:50,505 INFO org.apache.hadoop.service.AbstractService: Service com.datatorrent.stram.StreamingAppMasterService failed in state INITED; cause: java.lang.RuntimeException: java.lang.reflect.Invocati

Re: Local mode execution

2017-03-03 Thread Ganelin, Ilya
t and I'll copy it here. Ram On Fri, Mar 3, 2017 at 9:10 AM, Ganelin, Ilya wrote: > Hey all - any word on this? Would love to be able to test locally using > the new framework. Thanks! > > > > > ______

RE: Local mode execution

2017-03-03 Thread Ganelin, Ilya
Hey all - any word on this? Would love to be able to test locally using the new framework. Thanks! From: Ganelin, Ilya Sent: Wednesday, March 1, 2017 12:23:06 PM To: dev@apex.apache.org Subject: Local mode execution Hi all, I’ve returned to writing Apex apps

RE: Operator Node Affinity

2017-03-02 Thread Ganelin, Ilya
: http://www.apexbigdata.com/san-jose-register.html] <http://www.apexbigdata.com/san-jose-register.html> On Thu, Mar 2, 2017 at 8:59 AM, Ganelin, Ilya wrote: > Thanks – the solution I’m leaning towards is to deploy a load balancer > with a list of the nodes in the cluster, once Apex spins up

Re: Operator Node Affinity

2017-03-02 Thread Ganelin, Ilya
et a container on that host, but there > isn't a guarantee. Generally, services running on the cluster shouldn't > make assumptions about hosts and ports and use discovery instead. > > HTH, > Thomas > > > On Wed, Mar 1, 2017 at 7:53 PM,

Operator Node Affinity

2017-03-01 Thread Ganelin, Ilya
Hello, all – is there any way to deploy a given operator to a specific Node? E.g. if I’m trying to create a listener for a TCP socket that can then pipe data to a DAG, is there any way for the location of that listener to be deterministic so an upstream entity knows what to connect to? - Ilya G

Local mode execution

2017-03-01 Thread Ganelin, Ilya
Hi all, I’ve returned to writing Apex apps after a hiatus, it seems that the LocalMode is now deprecated, having been replaced by the Launcher interface. Is there an example or documentation anywhere of using this new API? Please let me know, thanks! - Ilya Ganelin [id:image001.png@01D1F7A4.F3D

Running application built against custom Malhar version

2016-07-11 Thread Ganelin, Ilya
I’m attempting to build a new application leveraging a custom Malhar library. I’ve compiled the updated Malhar library, confirmed that the classes I’ve added are present, and added this specific library as a system dependency in my pom of the newly created application: org.apache.apex malh

Proposed update to partitioning logic

2016-05-27 Thread Ganelin, Ilya
Hello all, as part of investigation https://issues.apache.org/jira/browse/APEXCORE-332 I dove pretty deep into the present implementation of the Partitioner, and the classes that leverage it including the PartitionAwareSink and DefaultPartition. Given that this is a rather core utility of the A

RE: A proposal for Malhar

2016-05-27 Thread Ganelin, Ilya
who pulls in the code posts the checklist? Thks, Amol On Fri, May 27, 2016 at 8:22 AM, Ganelin, Ilya wrote: > If we're going to adopt partially completed code, I propose that every new > Malhar operator then contain a checklist as a comment within the class. > > This che

RE: A proposal for Malhar

2016-05-27 Thread Ganelin, Ilya
If we're going to adopt partially completed code, I propose that every new Malhar operator then contain a checklist as a comment within the class. This checklist would be defined by the community and would track the current development state of the operator. That way there are no unexpected surp

Re: IntelliJ and Netbeans code styles are missing

2016-05-26 Thread Ganelin, Ilya
Another question - are CheckStyle settings packaged anywhere? There is obviously a CheckStyle run by Jenkins - is this CheckStyle published for Apex or Malhar? On 5/24/16, 2:19 PM, "Ganelin, Ilya" wrote: >I just realized that these were moved to subdirectories but t

Re: Get ApplicationId from inside Application

2016-05-25 Thread Ganelin, Ilya
>> String appid = dag.getValue(DAGContext.APPLICATION_ID); >> >> David >> >> On Thu, May 19, 2016 at 10:12 AM, Sandesh Hegde >> wrote: >> >> > Yes, use the conf object passed to StreamingApplication. >> > >> > On Thu, May 19, 2016

Re: IntelliJ and Netbeans code styles are missing

2016-05-24 Thread Ganelin, Ilya
I just realized that these were moved to subdirectories but the associated documentation was never updated. I’ll fix that. On 5/24/16, 2:16 PM, "Ganelin, Ilya" wrote: >Hi all – I’ve been setting up a new dev environment and noticed that the >apex-style.jar and apex-style

IntelliJ and Netbeans code styles are missing

2016-05-24 Thread Ganelin, Ilya
Hi all – I’ve been setting up a new dev environment and noticed that the apex-style.jar and apex-style.zip are missing from: https://github.com/apache/incubator-apex-core/tree/master/misc/ide-templates/intellij https://github.com/apache/incubator-apex-core/tree/master/misc/ide-templates/netbeans

Re: Apache Beam Integration

2016-05-23 Thread Ganelin, Ilya
input and also some reviews from a beam >translation perspective. > >And also anyone who wants to work on the Operator side can are more than >welcome to give feedback/contribute to the common interface. > > >Thanks, >Siyuan > > > > > > >On Fri, May 2

Re: [VOTE] Apache Apex Malhar Release 3.4.0 (RC1)

2016-05-23 Thread Ganelin, Ilya
+1 (Binding) Comments: Apache release tests failed on my machine with a multitude of errors related to @param notation. java version “1.8.0_77" OS X 10.11.4 PASSED: * Verified md5 and SHA512 for apache-apex-malhar-3.4.0-source-release.tar.gz * Verified md5 and SHA512 for apache-apex-malhar-3.4