+1
On Mon, Apr 9, 2018 at 11:31 AM Pramod Immaneni
wrote:
> Also, since we will be changing min jdk version to 8 for next release, I am
> fine with thomas's suggestion to change next version to 4.0. If there are
> no objections I will update the version on master.
>
> Thanks
>
> On Mon, Apr 9, 2
Using all the technicalities and loop holes, we can declare many votes
invalid. What purpose does it solve? This thread is dividing the community,
instead of recognizing the difference if we move forward with this, there
is a chance that Apex will alienate many contributors. What's the end game
her
Today, I saw the below -1 by Thomas,
https://github.com/apache/apex-malhar/pull/666 without the technical
justification.
Saumya, PR Author, has created a mail thread to discuss the justification,
but there was no comment in the mail thread.
So should we consider this as invalid -1?
On Thu, Aug 2
Ambarish, those operators still need some work to make it user friendly.
GenericRecord is not Kryo serializable, so AvroReader and GenericToPojo
converter needs to be CONTAINER/THREAD local.
This combo of 2 operators is best suitable for creation of Avro Module.
On Mon, Aug 7, 2017 at 7:58 AM Vla
Why is there a urgency, why cant this go into 4.0 Malhar with possibly
other breaking changes?
On Sat, Jul 15, 2017 at 7:57 AM Thomas Weise wrote:
> Discussing what in the future might become stable needs to be a separate
> thread, it will be a much bigger discussion.
>
> The topic here is to rel
+1
On Thu, Jul 13, 2017 at 2:00 PM Amol Kekre wrote:
> me too
>
> Thks
> Amol
>
>
> E:a...@datatorrent.com | M: 510-449-2606 <(510)%20449-2606> | Twitter:
> @*amolhkekre*
>
> www.datatorrent.com
>
>
> On Thu, Jul 13, 2017 at 1:31 PM, Pramod Immaneni
> wrote:
>
> > https://git-wip-us.apache.org/
My vote is to make the new proposal as the default behavior. Is there a use
case for the current behavior? If not then no need to add the configuration
setting.
On Thu, May 18, 2017 at 3:47 PM Pramod Immaneni
wrote:
> Sorry typo in sentence "as we are not asking for permissions for a lower
> pri
I will register my idea, which requires more work.
Expose a metric port, which can be aggregated at another operator. Default
operator could be AppDataPushAgent operator.
Users can plugin their own operators to send the metric to different types
of sinks.
On Tue, Mar 14, 2017 at 11:17 PM Chinmay
Please check the Hadoop dependency version in your POM. Also we need to
move these discussions to users@
On Fri, Mar 3, 2017 at 2:14 PM Ganelin, Ilya
wrote:
> Minor amendment: hadoop-2.6.0+cdh5.8.0+1592 (2.6 vs 2.7)
>
>
>
>
>
> - Ilya Ganelin
>
> [image: id:image001.png@01D1F7A4.F3D42980]
>
>
>
Instead of treating the stateless operator in a special way and missing
corner cases, just have a dummy checkpoint, then there is no need to handle
corner cases.
There is a name for this solution,
https://en.wikipedia.org/wiki/Null_Object_pattern
On Wed, Mar 1, 2017 at 2:52 PM Pramod Immaneni
1. Create an empty checkpoint file for the stateless operators.
2. Remove the logic to treat stateless operators as a special case.
Rest of the design remains as is.
On Wed, Mar 1, 2017 at 11:18 AM Amol Kekre wrote:
> The third option should be it.
> 1. On relaunch the DAG should start at commi
Settings mentioned by Sanjay, will only guarantee exactly once for Windows,
but not for partial window processed by the operator, in a way that setting
is a misnomer.
To achieve Exactly once, there are some precoditions that need to be met
along with the support in the output operator. Here is a bl
Java serializer comes with a big performance cost, so it is better to
reduce it's usage.
Can you please give more detail about your use case?
On Sun, Feb 5, 2017 at 10:05 PM Hitesh Kapoor
wrote:
Hi Ambarish,
Yes you can plug in your own serializer. You will have to set the
"STREAM_CODEC" port a
Prefer it to be made part of Apex core release. No need for another vote.
On Fri, Jan 20, 2017 at 1:53 PM Amol Kekre wrote:
> +1
>
> Thks
> Amol
>
>
> On Thu, Jan 19, 2017 at 8:55 PM, Priyanka Gugale >
> wrote:
>
> > +1
> >
> > On Fri, Jan 20, 2017 at 9:25 AM, Chinmay Kolhatkar <
> > chin...@da
I do see value in having the review PR along with the discussion in
the mailing list/jira.
It shows the commitment of the Author to the idea and to the project.
There is some truth in what Linus Torvalds says
https://lkml.org/lkml/2000/8/25/132
Thanks
On Mon, Jan 16, 2017 at 8:11 AM Amol Kekre
Kryo is used in a default implementation of the StreamCodec interface.
Ideally, if the StreamCodec interface itself allows the buffer to be passed
then we can also send the buffer from the BufferServer in future.
On Mon, Jan 9, 2017 at 4:10 PM Bright Chen wrote:
> Hi,
>
> The kryo Output has som
+1
Followed the steps in mentioned in http://apex.apache.org/verification.html
Verified the launch of an application.
On Tue, Dec 6, 2016 at 10:55 PM Thomas Weise wrote:
> Dear Community,
>
> Please vote on the following Apache Apex Core 3.5.0 release candidate.
>
> This is a source release wit
Yarn allows the AppMaster to run on the selected node, Apex shouldn't
select the blacklisted nodes, so it is possible to achieve not running the
Apex containers on certain nodes.
http://stackoverflow.com/questions/29302659/run-my-own-application-master-on-a-specific-node-in-a-yarn-cluster
On Thu
then let's fix that scenario instead of creating a new option.
>
> On Thu, Dec 1, 2016 at 11:17 AM, Sandesh Hegde
> wrote:
>
> > I have created a jira, for adding the list of blacklisted nodes,
> > https://issues.apache.org/jira/browse/APEXCORE-584
> >
> > On Wed
good to have this feature.
>
> I understand that Stram should not get into resourcing and still rely
> on
> Yarn, however, as the App Master it should have the right to reject the
> nodes offered by Yarn and request for other resources.
>
> Regards,
>
>
> David
>
> On Mon, Nov 28, 2016 at 5:22 PM, Bhupesh Chawda
> wrote:
>
> > +1 for the plan.
> >
> > I would be interested in contributing to this feature.
> >
> > ~ Bhupesh
> >
> > On Nov 29, 2016 03:26, "Sandesh Hegde" w
Apex has automatic blacklisting of the troublesome nodes, please take a
look at the following attributes,
MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST
https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Context.DAGContext.html#MAX_CONSECUTIVE_CONTAINER_FAILURES_FOR_BLACKLIST
BLACKLIS
I am interested in contributing to this feature.
On Mon, Nov 28, 2016 at 1:54 PM David Yan wrote:
> I think we should probably go ahead with option 1 since this works with
> most use cases and prevents developers from shooting themselves in the foot
> in terms of idempotency.
>
> We can have a c
Do we have any projects today that can benefit from this setup?
Earlier in this mail thread, we discussed "contrib (low bar) & graduation"
in Malhar, that is not sufficient?
On Wed, Nov 16, 2016 at 11:19 AM Chinmay Kolhatkar
wrote:
> @sanjay, yes we can define the process around this.
>
> @pramo
I have the PR open for the following issue,
https://issues.apache.org/jira/browse/APEXMALHAR-2298
https://github.com/apache/apex-malhar/pull/492
This change was done after a user feedback. Should we get this in for 3.6?
Thanks
On Tue, Nov 15, 2016 at 3:45 PM Thomas Weise wrote:
> It has bee
Good work Bhupesh.
On Mon, Nov 7, 2016 at 11:17 AM David Yan wrote:
> It took perseverance to get this merged, Good work Bhupesh!
>
> On Mon, Nov 7, 2016 at 1:25 AM, Bhupesh Chawda
> wrote:
>
> > Hi All,
> >
> > The PR for making Apex a runner for SAMOA has been merged.
> >
> > Apache SAMOA now
Here is a one more jira for 3.6, I need few more days to open the PR.
https://issues.apache.org/jira/browse/APEXMALHAR-2298?filter=-1
Thanks
On Thu, Oct 27, 2016 at 10:28 AM Vlad Rozov wrote:
+1. It will be nice to have
https://issues.apache.org/jira/browse/APEXMALHAR-2178 fixed in 3.6.
Thank
Hi All,
Operators that are not making progress are killed after
TIMEOUT_WINDOW_COUNT. But this doesn't take the operator recovery into
account. Operators that are recovering may need more time than what is set
in TIMEOUT_WINDOW_COUNT.
The new attribute, RECOVERY_WINDOW_TIMEOUT_COUNT, will be used
Hi All,
When an app package is compiled with Java 1.8 and Apex is using 1.7, we see
the following behaviour with Apex CLI
1. launch
"No applications in Application Package", ideally it should point out the
exact error message
2. get-app-package-operators
Throws the following exception java.lan
+1 for 2.6
On Mon, Oct 3, 2016, 2:06 PM Sasha Parfenov wrote:
> +1 for Hadoop 2.6 upgrade.
>
> Thanks,
> Sasha
>
> On Monday, October 3, 2016, Thomas Weise wrote:
>
> > +1 for 2.6 upgrade
> >
> >
> > On Mon, Oct 3, 2016 at 1:47 PM, David Yan > > wrote:
> >
> > > Hi all,
> > >
> > > Thomas crea
Travis is lot faster than the Apache Jenkins, should we just stick with
that?
On Sat, Oct 1, 2016 at 9:46 AM Chinmay Kolhatkar
wrote:
> Hi Vlad,
>
> In one of the PR the travis build is passing but Jenkins build is failing.
> https://github.com/apache/apex-malhar/pull/409
>
> Is it observed some
BigFoot?
On Wed, Sep 28, 2016 at 1:20 PM Thomas Weise wrote:
> What about the name "CI Support"? Does not look like best fit either. Any
> better ideas or keep it?
>
> I will document the outcome in the contributor guidelines.
>
> On Wed, Sep 28, 2016 at 11:13 AM, Pramod Immaneni
> wrote:
>
> >
Say it takes x MB size and y seconds to do the checkpoint. What does the
user do with that information?
On Sun, Sep 25, 2016, 6:51 AM Tushar Gosavi wrote:
> +1
>
> -Tushar
>
> On Sun, Sep 25, 2016, 8:54 AM Sanjay Pujare
> wrote:
>
> > +1
> >
> > Sanjay
> >
> >
> > On Sun, Sep 25, 2016 at 7:06 A
blem remains the same when we want to start from
> savepoint, where we will need to copy state from
> savepoint to application.
>
> -Tushar.
>
>
>
> On Tue, Sep 20, 2016 at 8:34 PM, Sandesh Hegde
> wrote:
> > How about re-launching the app from the same location?
>
How about re-launching the app from the same location?
If at all they want to store the state we can provide savepoint feature.
On Tue, Sep 20, 2016 at 4:39 AM Tushar Gosavi
wrote:
> We have observed that application relaunch takes long time.
> The one major reason for delay in application star
One point to my previous mail, Yarn tags are supported from Hadoop 2.4
release onwards. Apex supports Hadoop 2.2, that is why Yarn feature cannot
be leveraged.
On Tue, Sep 13, 2016 at 1:24 PM Sandesh Hegde
wrote:
> Hi All,
>
> I am proposing a new attribute “Tags”, on the similar line
Hi All,
I am proposing a new attribute “Tags”, on the similar lines as Yarn
application tags. (https://issues.apache.org/jira/browse/YARN-1461)
This is useful in a case where an Admin wants to attach an extra
information to Apex applications launched by various users/departments in
the company.
Congratulations Chandni
On Mon, Sep 12, 2016 at 10:46 AM Shubham Pathak
wrote:
> Congratulations Chandni !!
>
> Thanks,
> Shubham
>
> On Mon, Sep 12, 2016 at 10:29 AM, Amol Kekre wrote:
>
> > Chandni,
> > Congrats
> >
> > Thks
> > Amol
> >
> > On Mon, Sep 12, 2016 at 10:22 AM, Chanchal Singh <
+1
Did the tests mentioned below
http://apex.apache.org/verification.html
On Wed, Aug 31, 2016 at 4:06 PM David Yan wrote:
> +1 (binding)
>
> Downloaded the source, built with "mvn clean apache-rat:check verify
> -Dlicense.skip=false -Pall-modules install" successfully
> Checked LICENSE, NOTI
We can have categorized highlights, individuals can update it in the proper
categories.
New Operators
New Features
Major Bug Fixes
Miscellaneous
On Fri, Aug 26, 2016 at 2:04 PM Thomas Weise wrote:
> There are 2 issues left which unless they get resolved today I suggest we
> move to 3.6
>
> Th
n the DAG” will be available thru this listener?
>
>
>
> On 8/10/16, 8:20 AM, "Sandesh Hegde" wrote:
>
> Any operators can subscribe to Stram Events affecting the DAG.
>
> Implementation will most probably use heartbeat.
>
> On Tue, Aug 9, 2016 at
u suggesting an RPC interface or operator request for sending
> this information
> from Stram to all partitions of the interested operator?
>
> - Tushar.
>
>
>
> On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde
> wrote:
> > Hi All,
> >
> > As we add mo
commit.
> during restart the storage agent can get required checkpoint from its
> store, and which checkpoints to load will be available in
> savepoint metadata file.
>
> - Tushar.
>
>
>
> On Mon, Aug 8, 2016 at 8:49 PM, Sandesh Hegde
> wrote:
> > The idea here w
Hi All,
As we add more features to support batch use cases, there will be a need to
access more information about the DAG from an operator. One example is the need
to access the operator name while using Batched StatsListener.
The idea here is to implement DAG Listener ( similar to StatsListener
;> than
> > >>the mechanism of actually creating named checkpoints, it means
> having
> > >> the
> > >>ability for operators to move forward (a.k.a committed and dropping
> > >> committed states and buffer data) while still having the
ckup strategies.
>
>
> Thanks,
>
> A
>
> _
> Sent with difficulty, I mean handheld ;)
> On 4 Aug 2016 8:03 pm, "Munagala Ramanath" wrote:
>
> > +1
> >
> > Ram
> >
> > On Thu, Aug 4, 2016 at 12
Hello Team,
This thread is to discuss the Named Checkpoint feature for Apex. (
https://issues.apache.org/jira/browse/APEXCORE-498)
Named checkpoints allow following workflow,
1. Users can trigger a checkpoint and give it a name
2. Relaunch the application from the named checkpoint.
3. These chec
be backward incompatible change.
>
> - Tushar.
>
> On Tue, Jul 26, 2016 at 12:09 AM, Sandesh Hegde
> wrote:
> > How about making the mapping of "OperatorId" to "OperatorName" ( and
> other
> > extra information) as a part of the DAG conte
How about making the mapping of "OperatorId" to "OperatorName" ( and other
extra information) as a part of the DAG context?
On Mon, Jul 18, 2016 at 10:53 PM Tushar Gosavi
wrote:
> Hi All,
>
> We support shared stats listener, But user do not have any way to
> identify the operator for which th
Or
> something else?
>
> Thanks,
> Sasha
>
>
>
> On Tue, Jul 19, 2016 at 5:37 PM, Sandesh Hegde
> wrote:
>
> > Hi All,
> >
> > Apex supports configuration package, separates application package from
> the
> > actual configuration. (
> http://do
to. BTW, High-level API can definitely benefit from java 8. :)
>
> Regards,
> Siyuan
>
> On Wed, Jul 20, 2016 at 8:30 AM, Sandesh Hegde
> wrote:
>
> > Our current model of supporting the oldest supported Hadoop, penalizes
> the
> > users of latest Hadoop versions
Our current model of supporting the oldest supported Hadoop, penalizes the
users of latest Hadoop versions by favoring the slow movers.
Also, we won't benefit from the increased maturity of the Hadoop platform,
as we will be working on the many years old version of Hadoop.
We also need to incentivi
Hi All,
Apex supports configuration package, separates application package from the
actual configuration. (http://docs.datatorrent.com/configuration_packages/)
We want to enhance the configuration package by adding support to "add
Apps" (json format).
UseCase: Multiple users sharing the same app
Was this resolved?
My understanding is that, Kafka Input operator doesn't support the changes
in Kafka partitions after the initial launch.
On Mon, Jul 18, 2016 at 1:54 AM Chaitanya Chebolu
wrote:
> Hi All,
>
>I am facing dynamic partition issues in 0.8 version of Kafka Input
> Operator. My
+1 with some variation
Support next version, compared to one supported by the Apex main, of the
Hadoop instead of the latest Hadoop. This makes moving the Apex main to
next version of the Hadoop easy.
On Mon, Jul 11, 2016 at 10:33 AM Sanjay Pujare
wrote:
> strong +1 (will be nice to have some
Created a jira for this issue,
https://issues.apache.org/jira/browse/APEXCORE-482
On Thu, Jul 7, 2016 at 9:22 PM Amol Kekre wrote:
> +1. Makes sense. Do need to allow users to override if they want.
>
> Thks
> Amol
>
>
> On Thu, Jul 7, 2016 at 6:53 PM, Sandesh Hegde
Hi All,
Unifier's are deployed as CONTAINER_LOCAL with the downstream operator(
except in the corner case of Mx1 ). Default Unifiers are essentially doing
buffer to buffer copy, so instead it should be THREAD_LOCAL to improve the
performance.
Let me know your thoughts on this.
Thanks
Most of the time download goes to mirrored copy on the 3rd party site, so
it may be tough to get the stats.
http://www.apache.org/dyn/closer.cgi
On Thu, Jun 30, 2016 at 5:40 PM Sasha Parfenov wrote:
> I saw that https://repository.apache.org/ might provide download stats
> under "Views/Reposit
://issues.apache.org/jira/browse/STORM-1157
On Mon, May 23, 2016 at 11:31 AM Sandesh Hegde
wrote:
> After connecting to the app user will run the following command.
>
> Users will select the container id, jdk tool and the arguments to the tool.
>
> Apex CLI api
> run-jdkTools "C
Why restrict the control tuples to input operators?
On Sat, Jun 25, 2016 at 9:07 AM Amol Kekre wrote:
> David,
> We should avoid control tuple within the window by simply restricting it
> through API. This can be done by calling something like "sendControlTuple"
> between windows, notably in inp
It is a backward incompatible change. That's used to list Apex apps.
On Thu, Jun 23, 2016, 4:08 AM Sandeep Deshmukh
wrote:
> Yes, good idea.
>
> Will that be a backward compatible change? I think, we should wait for 4.0.
>
> On Thu, Jun 23, 2016 at 1:44 PM, Aniruddha Thombare <
> anirud...@datat
For the usecase 1, is it possible to avoid changing the Context? Can we
have something along the lines of "StramToNodeRequest" ?
On Tue, Jun 21, 2016 at 11:09 AM Tushar Gosavi
wrote:
> Hi All,
>
> We have seen few use cases in field which require Apex application
> scheduling based on some condi
Hi All,
This change has been merged, will be part of 3.5.
New API in the DAG, setOperatorAttribute(Operator operator, Attribute key,
T value) )
Another API was deprecated, "setAttribute(Operator ... "
Thanks
On Tue, Jun 7, 2016 at 1:08 PM Sandesh Hegde
wrote:
> I have created
Hi All,
Our current design pattern for using Window Data Manager is to purge the
Window Data in the committed window call back.
In the case of Output operators, all we care is the Last (/window)
snapshot, that is because we have already written the tuples before that
snapshot so those can be safe
Congratulations!!!
On Thu, Jun 16, 2016 at 10:06 AM Bright Chen wrote:
> Siyuan, Congratulations!
>
> thanks
> Bright
>
> > On Jun 16, 2016, at 9:27 AM, Pradeep A. Dalvi wrote:
> >
> > Congrats Siyuan!
> >
> > -prad
> >
> > On Thursday, June 16, 2016, Dongming Liang
> wrote:
> >
> >> Congrats,
Hello Team,
Purging of the Checkpoints is done in Stram. Why not do that from the
StreamingContainers?
Committed window information is already available in StreamingContainers
and it will also distribute the computation across the containers.
Corner cases can still be handled in Stram. Example:
There is a parallel between StreamCodec and Unifier. As StreamCodec can be
applied on the input ports as an Attribute, having Unifier as an attribute
for the output ports completes the similarity, making it easy for app
developers.
Another reason for making it an Attribute : Unifier can be applied
eratorAttribute*.
>
> Ram
>
> On Mon, Jun 6, 2016 at 1:39 PM, Sandesh Hegde
> wrote:
>
> > Currently, *setAttribute* is used to set the operator attributes. Other 2
> > Attribute setting APIs are specific to input ports
> > (*setInputPortAttributes*) and output port
Currently, *setAttribute* is used to set the operator attributes. Other 2
Attribute setting APIs are specific to input ports
(*setInputPortAttributes*) and output ports (*setOutputPortsAttributes*).
Proposal is to have *SetOperatorAttribute*
api, which will clearly indicate that user wants set att
+1 for removing the not-used operators.
So we are creating a process for operator writers who don't want to
understand the platform, yet wants to contribute? How big is that set?
If we tell the app-user, here is the code which has not passed all the
checklist, will they be ready to use that in pro
https://github.com/sandeshh/myapexapp/blob/master/src/main/java/com/example/myapexapp/KafkaOutput.java
f it’s logic
> depended on key.
>
> thanks
> Bright
>
> > On May 13, 2016, at 12:37 PM, Sandesh Hegde
> wrote:
> >
> > Hi All,
> >
> > I am working on Kafka 0.9 output operator and one of the requirement is
> to
> > implement Exac
t; Thanks,
> Thomas
>
>
>
> On Mon, May 23, 2016 at 11:27 AM, Sandesh Hegde
> wrote:
>
> > Users can pass the arguments to the JDK tools. So it exposes all the
> power
> > of those tools. If we have to write the code we are doing the duplicate
> > work.
>
, May 23, 2016 at 11:15 AM Vlad Rozov wrote:
> What is the purpose of the new process? Why that information can't be
> collected directly from JVM and passed to app master using heartbeat?
>
> Thank you,
> Vlad
>
> On 5/23/16 10:57, Sandesh Hegde wrote:
> > Hello All
Hello All,
Getting various information from the StreamingConatainers is a useful
feature to have.
As StreamingContainers are JVMs, various JDK tools can be used to get the
information.
So the idea is to spawn the new process from the streaming containers and
return the information via Stram.
Rec
75 matches
Mail list logo
