+1. This is a really good starting point to cleanup malhar.
On Wed, Jul 13, 2016 at 3:06 AM, David Yan wrote:
> Hi Lakshmi,
>
> Thanks for volunteering.
>
> I think Pramod's suggestion of putting the operators into 3 buckets and
> Siyuan's suggestion of starting a shared Google Sheet that tracks
I don't know the relation between the DAG size and the AppMaster memory
yet. Maybe others can fill in.
When the situation you mentioned happens, I just raise the memory of the
AppMaster by few GBs.
On Tue, Jul 12, 2016 at 2:33 PM Raja.Aravapalli
wrote:
>
> Sure Sandesh Thanks.
>
> Also, one quic
Hi Lakshmi,
Thanks for volunteering.
I think Pramod's suggestion of putting the operators into 3 buckets and
Siyuan's suggestion of starting a shared Google Sheet that tracks
individual operators are both good, with the exception that lib/streamquery
is one unit and we probably do not need to loo
Sure Sandesh Thanks.
Also, one quick question,
When will the size/memory of the Application Master grows ?
Does the memory of AM depends on the no.of operators in the pipeline ?
One issue I observed with my DAG is,
Memory of the application master is growing for my DAG and after reaching max.
I am interested to work on this.
Regards,
Lakshmi prasanna
On Tue, Jul 12, 2016 at 1:55 PM, hsy...@gmail.com wrote:
> Why not have a shared google sheet with a list of operators and options
> that we want to do with it.
> I think it's case by case.
> But retire unused or obsolete operators is i
Why not have a shared google sheet with a list of operators and options
that we want to do with it.
I think it's case by case.
But retire unused or obsolete operators is important and we should do it
sooner rather than later.
Regards,
Siyuan
On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre wrote:
>
I would suggest we go through the operators in those packages on an
individual basis and grade them into 3 buckets, those that meet the level
we expect from the operators (could be few of them), those that are
potentially useful but need additional work and those that we don't think
would be useful
My vote is to do 2&3
Thks
Amol
On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh <
vkottapa...@directv.com> wrote:
> +1 for deprecating the packages listed below.
>
> -Original Message-
> From: hsy...@gmail.com [mailto:hsy...@gmail.com]
> Sent: Tuesday, July 12, 2016 12:01 PM
>
>
+1
On Tue, Jul 12, 2016 at 11:53 AM, David Yan wrote:
> Hi all,
>
> I would like to renew the discussion of retiring operators in Malhar.
>
> As stated before, the reason why we would like to retire operators in
> Malhar is because some of them were written a long time ago before Apache
> incuba
Hi all,
I would like to renew the discussion of retiring operators in Malhar.
As stated before, the reason why we would like to retire operators in
Malhar is because some of them were written a long time ago before Apache
incubation, and they do not pertain to real use cases, are not up to par in
To unsubscribe, please send an email to users-unsubscr...@apex.apache.org
Ref: http://apex.apache.org/community.html
--prad
On Tue, Jul 12, 2016 at 11:38 AM, ganesh borate
wrote:
> Please unsubscribe me
> And stop sending mail related to this domain because right now I am not
> working with this
Please unsubscribe me
And stop sending mail related to this domain because right now I am not
working with this technology
So stop sending mails
Thank you
This is a mailing list Ganesh, that’s not how it works. To stop getting these
emails, unsubscribe: users-unsubscr...@apex.apache.org
From: ganesh borate
Reply-To: "users@apex.apache.org"
Date: Tuesday, July 12, 2016 at 2:31 PM
To: "users@apex.apache.org"
Subject: Re: ApacheCon Europe Call For
Please stop sending mail for me
Because these are irrelative to my field for now
Whenever I need help about this topic I will make contact with you
For now please don't send mails
Thank you
On 12 Jul 2016 11:55 p.m., "Rich Bowen" wrote:
> As you are no doubt already aware, we will be holding Apac
As you are no doubt already aware, we will be holding ApacheCon in
Seville, Spain, the week of November 14th, 2016. The call for papers
(CFP) for this event is now open, and will remain open until
September 9th.
The event is divided into two parts, each with its own CFP. The first
part of the eve
In Apex, an app can have multiple pipelines i.e. multiple input operators.
Do you want to read the same set of data from same db table and
duplicate it on 2 different output ports? If yes, then you would need
to extend the input operator and add another output port in extended
class.
Can you share
I request you sir for the being I don't want this kind of mails which is
related to your topic
So please stop sending mail
In future if I need your help about Apache apex or Hadoop I will contact
you but for now please stop sending mails
On 7 Jul 2016 1:56 a.m., "Mukkamula, Suryavamshivardhan (
Hi,
What memory does the “allocated mem.” refers to on UI for a DAG ? Application
Master OR Containers memory of an operators ?
[cid:B61FE0C9-4767-4FF8-9E23-454CB502C53C]
I included below properties as well and re-triggered the DAG, still it is
showing 32GB only!!
dt.application..at
Looks like you are trying to create an Operator that has multiple input
sources and multiple output sinks. For this specific question, I guess you
want to have 2 DefaultOutputPort. So you can try the following in the
JdbcPojooutput operator.
final transient DefaultOutputPort op1 = new DefaultOutp
Raja,
Please increase the container size and launch the app again. yarn.scheduler
.maximum-allocation-mb is for the container and not for the DAG and the
error message showed by you is for the container.
Here is one quick way, use the following attribute.
dt.operator.*.attr.MEMORY_MB
4096
Hi Ram,
Sorry I did not share that details of 32gb with you.
I am saying 32gb is allocated because, I observed the same on UI, when the
application is running. But now, as the DAG is failed, I cannot take a
screenshot and send!!
Regards,
Raja.
From: Munagala Ramanath mailto:r...@datatorrent
Hello,
In my application I am using JdbcPojooutput operator and JdbcPojoInput
operator. I was thinking whether we can have multiple output for JdbcPojoInput
operator as currently I am using 2 JdbcPojoInput operator. I tried to do so but
I got error. So I want to know can we do that?
Thank You,
Hi Raja,
Try if this helps. Setting up this property in
yarn-site.xml to more than 10. It specifies the maximum percent of
resources in the cluster which can be used to run the application The
default will be 0.1
yarn.scheduler.capacity.maximum-am-resource-percent
100
Thanks
You can increase the maximum allocation MB but it will require a resource
manager restart.
Thanks,
Dev
On Jul 12, 2016 9:01 AM, "Raja.Aravapalli"
wrote:
>
> Thanks for the response Sandesh.
>
> Since our yarn-site is configured with value *32768* for the property *
> yarn.scheduler.maximum-all
How do you know it is allocating 32GB ? The diagnostic message you posted
does not show
that.
Ram
On Tue, Jul 12, 2016 at 8:51 AM, Raja.Aravapalli wrote:
>
> Thanks for the response Sandesh.
>
> Since our yarn-site is configured with value *32768* for the property *
> yarn.scheduler.maximum-all
Thanks for the response Sandesh.
Since our yarn-site is configured with value 32768 for the property
yarn.scheduler.maximum-allocation-mb, it is allocating a max of 32gb and not
more than that!!
Wish to know, is there a way I can increase the max allowed value ? OR, since
it is configured in
It looks like the current allocation is the default of 1GB; please increase
it to, say, 4GB and see
if the problem is resolved. The max appears to be 32GB.
Also checkout the "Advanced Features" section of the Top N Words tutorial (
http://docs.datatorrent.com/tutorials/topnwords-c7/)
where memory
Quoting from the doc shared by the Ram, those parameters control operator
memory size.
actual container memory allocated by RM has to lie between
[yarn.scheduler.minimum-allocation-mb, yarn.scheduler.maximum-allocation-mb]
On Tue, Jul 12, 2016 at 8:38 AM Raja.Aravapalli
wrote:
>
> Hi Ram,
>
Hi Ram,
I see in the cluster yarn-site.xml, below two properties are configured with
below settings..
yarn.scheduler.minimum-allocation-mb ===> 1024
yarn.scheduler.maximum-allocation-mb ===> 32768
So with the above settings at cluster level, I can’t increase the memory
allocated for my DAG ?
Thank you Sandesh.
Regards,
Surya Vamshi
From: Sandesh Hegde [mailto:sand...@datatorrent.com]
Sent: 2016, July, 12 11:24 AM
To: users@apex.apache.org
Subject: Re: Inputs needed on File Writer
When you launch the application using Apex cli, it returns the application id.
Here is the sample outpu
When you launch the application using Apex cli, it returns the application
id.
Here is the sample output,
{"appId": "application_1467701377054_12860"}
On Tue, Jul 12, 2016 at 7:36 AM Mukkamula, Suryavamshivardhan (CWM-NR) <
suryavamshivardhan.mukkam...@rbc.com> wrote:
> Thank you ram, this is
Thank you ram, this is really helpful.
Regards,
Surya Vamshi
From: Munagala Ramanath [mailto:r...@datatorrent.com]
Sent: 2016, July, 12 10:30 AM
To: users@apex.apache.org
Subject: Re: Inputs needed on File Writer
Please take a look at the Python script under
https://github.com/DataTorrent/exampl
Also from a previous thread on users:
-- Forwarded message --
From: Shubham Pathak
Date: Wed, Jun 15, 2016 at 1:57 AM
Subject: Re: Containers Not getting Allocated.
To: users@apex.apache.org
Hello,
This was a Hadoop configuration issue. yarn.nodemanager.resource.cpu-vcores
was s
Please see: http://docs.datatorrent.com/troubleshooting/#configuring-memory
Ram
On Tue, Jul 12, 2016 at 6:57 AM, Raja.Aravapalli wrote:
>
> Hi,
>
> My DAG is failing with memory issues for container. Seeing below
> information in the log.
>
>
>
> Diagnostics: Container [pid=xxx,containerID=cont
Please take a look at the Python script under
https://github.com/DataTorrent/examples/tree/master/tools
It uses the Gateway REST API to retrieve application info given the name;
the id is part of that JSON object.
Ram
On Tue, Jul 12, 2016 at 6:58 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <
surya
Hi,
Can you check your YARN memory configuration?
This may help you:
http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_ig_yarn_tuning.html
Thanks,
A
_
Sent with difficulty, I mean handheld ;)
On 12 Jul 2016 7:53 pm, "Raja.Aravapalli"
wrote:
Hi,
My DAG is failing with memory issues for container. Seeing below information in
the log.
Diagnostics: Container [pid=xxx,containerID=container_xyclksdjf] is running
beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory
used; 2.9 GB of 2.1 GB virtual memory used. K
Hi Tushar,
This is very helpful information, thank you.
In the below cases, when I want to schedule the kill of the application , How
would I know the App ID at run time to initiate the kill.
I would like to kill the application may be after an hour after launch in the
oozie scheduler.
Regard
Hi Surya,
Launching Apex application.
1) Using apex cli, you can write a script to launch an Application
apex -e "launch -conf "
Kill an application.
apex -e "kill-app "
This could be integrated with oozie shell action to launch and kill the app.
2) Using REST api.
If you are using RTS relea
39 matches
Mail list logo
