Hi Ashwin,
The recovery checkpoint for operator A is computed by taking the checkpoint
with the largest window id that is less than or equal to the checkpoint
with the largest window id among all the operators down stream to A. The
output operators in a dag will always recover to their most recent
When we are dealing with secured data, the usual scenarios are that you get
encrypted data.
This data need to decrypt and then perform other functions on it. The
output of the dag is then encrypted.
In the past we have solved these use cases by performing
decryption/encryption in the operator.
IMO
Github user ilooner closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/130
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
In the apex architecture there is concept of checkpointing and concept of
committed when all operator have crossed a common checkpoint.
So, in which scenarios does a given operator recover at last checkpoint
window vs last committed window vs some other checkpoint window in between?
--
Regards,
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/148#discussion_r47602363
--- Diff:
engine/src/main/java/com/datatorrent/stram/webapp/StramWebServices.java ---
@@ -765,10 +856,10 @@ public JSONObject
getPhysicalOperator
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/148#discussion_r47601043
--- Diff:
engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java ---
@@ -1067,28 +1096,186 @@ public int hashCode()
publi
Well we have committers from bank, their feedback will be really valuable.
On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale
wrote:
> Sounds good. This is good feature for banks and security domain.
> One suggestion: We can do key management ourself at application (may be by
> providing default k
Sounds good. This is good feature for banks and security domain.
One suggestion: We can do key management ourself at application (may be by
providing default keys) and there should be an option to override keys if
user really want to do so.
-Priyanka
On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolh
We need to identify the operators and components that are stable if we want
to go with semver check of only Stable classes.
I can create an initial list.
Thanks,
Chandni
On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar wrote:
> Yep, That's what I am doing now :)
>
> Thanks,
> Isha
>
> On Mon, De
Hi All,
I wanted to propose an idea using which one can have encrypted stream
flowing in a DAG.
Basically, the idea is to create a new EncryptedInputPort which will extend
from DefaultInputPort and will return a StreamCodec object which will take
care of encryption/decryption.
As the same StreamC
GitHub user ilooner opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/130
MLHR-1946 #resolve #comment Marked WindowUtils as unstable.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilooner/incubator-ape
Yep, That's what I am doing now :)
Thanks,
Isha
On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh
wrote:
> Isha,
>
> I think for now you can configure the japicmp plugin to exclude the package
> as follows in the pom.xml.
>
>
>
> com.datatorrent.lib.parser.*
>
>
>
> This is an example where we
Isha,
I think for now you can configure the japicmp plugin to exclude the package
as follows in the pom.xml.
com.datatorrent.lib.parser.*
This is an example where we can benefit from inclusion approach with
japicmp 0.7 version.
Thanks,
Chandni
On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkat
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/148#discussion_r47597681
--- Diff:
engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java ---
@@ -2246,6 +2306,50 @@ public LogicalOperatorInfo
getLogi
Github user tushargosavi commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/148#discussion_r47596699
--- Diff:
engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java ---
@@ -2246,6 +2306,50 @@ public LogicalOperatorInfo
g
+1
When 0.7 version of japicmp is available, we can add exclusions for
@Evolving or inclusions for @Stable, whichever way is finalized.
But before that should we add package exclusions individually if all the
operators inside the package are marked Evolving?
I wanted to make changes to some of the
Thanks David.
-Original Message-
From: David Yan [mailto:da...@datatorrent.com]
Sent: Monday, December 14, 2015 6:58 PM
To: dev@apex.incubator.apache.org
Subject: Re: configure filesplitter to read batch data from different folders
Hi Venkatesh,
The POST command you listed actually acce
Github user tweise commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/184#issuecomment-164631996
When all other comments are addressed, please squash the commits. See (9) in
http://apex.incubator.apache.org/contributing.html#opening-pull-request
Github user tweise commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/184#issuecomment-164631402
A possible future enhancement could be support for the xxxAware interface
for all attribute values. But this can be taken up later also.
---
If your projec
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/184#discussion_r47592207
--- Diff:
api/src/main/java/com/datatorrent/api/ApplicationAwareStorageAgent.java ---
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Sof
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/184#discussion_r47592139
--- Diff:
api/src/main/java/com/datatorrent/api/ApplicationAwareStorageAgent.java ---
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Sof
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/184#discussion_r47592052
--- Diff:
api/src/main/java/com/datatorrent/api/ApplicationAwareStorageAgent.java ---
@@ -0,0 +1,38 @@
+/**
+ * Licensed to the Apache Sof
Hi Venkatesh,
The POST command you listed actually accepts the following as payload:
{
"{propertyName}" : "{propertyValue}",
...
}
The propertyName can be
"dt.application.app_name.operator.fileSplitter.prop.scanner.files" and the
propertyValue can be "/user/data/date=12-14-2015/ver=6/".
Da
Hi Chandini,
It is not from a running application. I am aware of that way but it doesn’t
suit my use case.
Here is the example.
dt.application.app_name.operator.fileSplitter.prop.scanner.files
/user/data/date=12-14-2015/ver=6/
Now when I
There is a way to dynamically change properties of an operator in a running
application. Is that what you are looking for?
On Mon, Dec 14, 2015 at 6:25 PM, Kottapalli, Venkatesh <
vkottapa...@directv.com> wrote:
> Hi,
>
> I am trying to configure filesplitter to read files from
>
Hi,
I am trying to configure filesplitter to read files from
different directories based on parameters that I wish to provide as input to
the application?
* One way of doing it is to read data from a properties file on server
into application.java and set those values
Github user tweise commented on a diff in the pull request:
https://github.com/apache/incubator-apex-core/pull/148#discussion_r47588531
--- Diff:
engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java ---
@@ -2246,6 +2306,50 @@ public LogicalOperatorInfo
getLogi
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47588391
--- Diff: library/pom.xml ---
@@ -310,6 +310,11 @@
commons-beanutils
1.8.3
+
+ net.sf.trove4j
Github user davidyan74 commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47588137
--- Diff: library/pom.xml ---
@@ -310,6 +310,11 @@
commons-beanutils
1.8.3
+
+ net.sf.trove
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47584078
--- Diff:
library/src/main/java/com/datatorrent/lib/dimensions/aggregator/Aggregator.java
---
@@ -0,0 +1,14 @@
+package com.datatorrent.l
Github user brightchen commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47580926
--- Diff:
library/src/main/java/com/datatorrent/lib/dimensions/aggregator/Aggregator.java
---
@@ -0,0 +1,14 @@
+package com.datatorren
Github user ilooner commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47580479
--- Diff:
library/src/main/java/com/datatorrent/lib/dimensions/aggregator/Aggregator.java
---
@@ -0,0 +1,14 @@
+package com.datatorrent.l
Github user chandnisingh commented on a diff in the pull request:
https://github.com/apache/incubator-apex-malhar/pull/129#discussion_r47579533
--- Diff:
library/src/main/java/com/datatorrent/lib/dimensions/aggregator/Aggregator.java
---
@@ -0,0 +1,14 @@
+package com.datatorr
GitHub user brightchen opened a pull request:
https://github.com/apache/incubator-apex-malhar/pull/129
MLHR-1943 #resolve #comment copied from Megh
@ilooner Could you please review and merge?
You can merge this pull request into a Git repository by running:
$ git pull https://
Sorry, meant to say 4-6GB, not MB.
Ram
On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath
wrote:
> I'm OK with increasing it to 1280MB.
>
> I agree with Sandeep about new people trying out Apex,
> often on limited environments like VMs limited to 4-6MB. With a default of
> 0.5GB per Buffer Serv
I'm OK with increasing it to 1280MB.
I agree with Sandeep about new people trying out Apex,
often on limited environments like VMs limited to 4-6MB. With a default of
0.5GB per Buffer Server
and 1GB per operator, things start to fail when the user has more than a
handful of operators.
I'm guessin
Hi Sandeep,
I have seen the issue even with small apps like Twitter Demo. It happens
less frequently in very small apps but the issue is still there. It happens
frequently in medium size apps (10 containers). Could we increase the
MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get great
Hey Siyuan and Brennon,
Just an Idea, not sure how relevant it is to the work you guys are doing. I
have a use case for alerts where I want to have a proxy operator receive an
alert, then based on the communication channels listed in the alert, I want
the alert to be forwarded to one or more opera
Github user davidyan74 closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/109
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/117
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-apex-malhar/pull/120
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Everybody, please vote as soon as you can. TOMORROW is the deadline! We
need to have Apex included in Hadoop Summit!
On Mon, Dec 14, 2015 at 10:46 AM, Sandesh Hegde
wrote:
> Tomorrow (Dec 15) is the deadline for the voting. Does anybody know how
> many sessions will be selected ? Currently w
Github user ashishtadose commented on the pull request:
https://github.com/apache/incubator-apex-core/pull/184#issuecomment-164523414
@tweise suggested changes are incorporated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Tomorrow (Dec 15) is the deadline for the voting. Does anybody know how
many sessions will be selected ? Currently we are at 8th position.
On Fri, Dec 11, 2015 at 2:30 PM Amol Kekre wrote:
> Please vote :)
>
>
> https://hadoopsummit.uservoice.com/forums/332079-the-future-of-apache-hadoop/sugges
I agree with Tim. WindowGenerator is as its name suggests is not a general
purpose utility class that will be used by operator developer.
On Mon, Dec 14, 2015 at 10:24 AM, Timothy Farkas
wrote:
> Hi Siyuan
>
> The ticket is here
>
> https://malhar.atlassian.net/browse/MLHR-1940
>
> A small set o
Hi Siyuan
The ticket is here
https://malhar.atlassian.net/browse/MLHR-1940
A small set of utility methods have already been added to Malhar library
https://github.com/apache/incubator-apex-malhar/blob/devel-3/library/src/main/java/com/datatorrent/lib/util/time/WindowUtils.java
If you have any
Similar utilities are already in WindowGenerator in apex core. Maybe add
to that class instead of creating a separate class?
David
On Dec 11, 2015 4:13 PM, "Timothy Farkas" wrote:
> I'd like to propose creating an operator utility in
> com.datatorrent.lib.operatorutils to convert time into numb
Hey Tim,
+1
Please add me to watcher list to the ticket for this. The highlevel API may
need this work.
For example, stream.window(TimeUnit.SECOND, 10) etc.
Thanks!
On Sat, Dec 12, 2015 at 5:36 PM, Timothy Farkas wrote:
> will do
>
> On Sat, Dec 12, 2015 at 4:36 PM, Chandni Singh
> wrote:
>
48 matches
Mail list logo
