Programmatic log4j appender in Apex

[Sergey Golovko]

Hi All,

I'd like to add supporting of a custom defined log4j appender that can be
added to Apex Application Master and Containers and be configurable
programmatically.

Sometimes it is not trivial to control log4j configuration via log4j
properties. And I think the having of the approach to add a log4j appender
programmatically will allow the customers and developers to plugin their
own custom defined log4j appenders and be much flexible for streaming and
collection of Apex log events.

I assume to provide generic approach for definition of the programmatic
log4j appender and to pass all configuration parameters including a name of
the Java class with implementation of the log4j appender via system and/or
command line properties.

Thanks,
Sergey

Re: Apex Sandbox in Apex Dev Setup documentation

[Sasha Parfenov]

+1 on listing various sandboxes on download page.  As Vlad pointed out,
there are examples in Hadoop and other Apache projects with vendor
sponsored/provided sandboxes:

Kudu - http://kudu.apache.org/docs/quickstart.html
Drill - https://drill.apache.org/docs/installing-the-apache-drill-sandbox/
Eagle - http://eagle.apache.org/docs/deployment-in-sandbox.html

If there is a vendor willing to contribute time and effort to make Apache
Apex easier to consume, I think it benefits all Apex users.

Thanks,
Sasha

On Fri, Apr 7, 2017 at 1:27 PM, Vlad Rozov  wrote:

> I would argue that a link to a vendor sandbox or a download that makes it
> easier to jump start with Apex and promotes adoption is OK. For example
> Hadoop provides links to HDP and CDH sandboxes:
> https://wiki.apache.org/hadoop/QuickStart. Additionally, bigtop sandbox
> does not seem to support VirtualBox out of the box, so why not to list both?
>
> Thank you,
>
> Vlad
>
>
> On 4/7/17 09:09, Thomas Weise wrote:
>
>> I disagree. Apex documentation is not the place to promote vendor
>> offerings.
>>
>> I suggested to refer to the download page, which already contains the
>> DataTorrent link.
>>
>> Thomas
>>
>>
>> On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
>> wrote:
>>
>> Hi Dean,
>>>
>>> There aren't many good options out there for users to get an environment
>>> up
>>> with apex apps running easily or quickly. We all know how difficult it
>>> would be for someone new to get an application up and running even if
>>> they
>>> had a hadoop sandbox with apex. Also, for many when they download a
>>> sandbox
>>> they not only want to be able to run something in a few steps, but also
>>> have easy to use tools and see something visually. My suggestion is to
>>> keep
>>> both the sandboxes. You can put the bigtop sandbox first in the list as
>>> it
>>> is vendor neutral.
>>>
>>> Thanks
>>>
>>> On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard >> >
>>> wrote:
>>>
>>> All,

 In the Sandbox section of the Apache Apex Development Environment Setup
 documentation (
 https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
 instructions are provided for a vendor-specific Sandbox.

 I would like to propose that these instructions be changed to reference

>>> the
>>>
 Apache Apex Sandbox instead, which is an Apache Bigtop build and

>>> available
>>>
 via docker at https://hub.docker.com/r/apacheapex/sandbox.

 Regards,
 Dean


>

Re: Apex Sandbox in Apex Dev Setup documentation

[Vlad Rozov]

+1 to have a link from doc to downloads page. It is also necessary to 
limit downloads page to links to various binary distributions and move 
the detailed steps from the download page to the site where downloads 
point to.


Thank you,

Vlad

On 4/7/17 16:54, Amol Kekre wrote:

We should just point it to apex download page in the docs ->
http://apex.apache.org/downloads.html; I will add my comments to the jira

Thks
Amol


E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*

www.datatorrent.com


On Fri, Apr 7, 2017 at 1:33 PM, Dean Lockgaard 
wrote:


Thanks for the input, Thomas and Pramod.

I've created the following two tickets to track this:

https://issues.apache.org/jira/browse/APEXCORE-692
https://issues.apache.org/jira/browse/APEXCORE-693

Regards,
Dean




On Fri, Apr 7, 2017 at 9:22 AM, Pramod Immaneni 
wrote:


Agreed, didn't realize this was in the documentation page.

On Fri, Apr 7, 2017 at 9:09 AM, Thomas Weise  wrote:


I disagree. Apex documentation is not the place to promote vendor
offerings.

I suggested to refer to the download page, which already contains the
DataTorrent link.

Thomas


On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni <

pra...@datatorrent.com>

wrote:


Hi Dean,

There aren't many good options out there for users to get an

environment

up

with apex apps running easily or quickly. We all know how difficult

it

would be for someone new to get an application up and running even if

they

had a hadoop sandbox with apex. Also, for many when they download a

sandbox

they not only want to be able to run something in a few steps, but

also

have easy to use tools and see something visually. My suggestion is

to

keep

both the sandboxes. You can put the bigtop sandbox first in the list

as

it

is vendor neutral.

Thanks



On Thu, Apr 6, 2017 at 7:18 PM, Thomas Weise  wrote:


+1 this should be tracked in a JIRA.

There are also some improvements that can be done to the instructions on
the docker hub (separate activity).

I would also suggest to list the Apex binary build on the downloads page
for users that have an existing cluster:

https://github.com/atrato/apex-cli-package/releases

And perhaps mention in the setup tutorial that these other download

options

are listed on the website.

Thanks,
Thomas






On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard <

dean.lockga...@gmail.com

wrote:


All,

In the Sandbox section of the Apache Apex Development Environment

Setup

documentation (
https://apex.apache.org/docs/apex/apex_development_setup/#sandbox

),

instructions are provided for a vendor-specific Sandbox.

I would like to propose that these instructions be changed to

reference

the

Apache Apex Sandbox instead, which is an Apache Bigtop build and

available

via docker at https://hub.docker.com/r/apacheapex/sandbox.

Regards,
Dean

[jira] [Resolved] (APEXMALHAR-2465) Travis build is failing because of excessive logging

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXMALHAR-2465.
--
   Resolution: Fixed
Fix Version/s: 3.8.0

> Travis build is failing because of excessive logging
> 
>
> Key: APEXMALHAR-2465
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2465
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Velineni Lakshmi Prasanna
>Assignee: Velineni Lakshmi Prasanna
>Priority: Critical
> Fix For: 3.8.0
>
>
> Recently many examples were added from datatorrent/examples. This seems to 
> have pushed the amount of logs been generated during running of tests in the 
> build, beyond the limit of what Travis allows. This is causing Travis build 
> to fail. This needs to be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-malhar pull request #602: Inserted "threshold" log property in Log4j.pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-malhar/pull/602


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Apex Sandbox in Apex Dev Setup documentation

[Amol Kekre]

We should just point it to apex download page in the docs ->
http://apex.apache.org/downloads.html; I will add my comments to the jira

Thks
Amol


E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*

www.datatorrent.com


On Fri, Apr 7, 2017 at 1:33 PM, Dean Lockgaard 
wrote:

> Thanks for the input, Thomas and Pramod.
>
> I've created the following two tickets to track this:
>
> https://issues.apache.org/jira/browse/APEXCORE-692
> https://issues.apache.org/jira/browse/APEXCORE-693
>
> Regards,
> Dean
>
>
>
>
> On Fri, Apr 7, 2017 at 9:22 AM, Pramod Immaneni 
> wrote:
>
> > Agreed, didn't realize this was in the documentation page.
> >
> > On Fri, Apr 7, 2017 at 9:09 AM, Thomas Weise  wrote:
> >
> > > I disagree. Apex documentation is not the place to promote vendor
> > > offerings.
> > >
> > > I suggested to refer to the download page, which already contains the
> > > DataTorrent link.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni <
> pra...@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Dean,
> > > >
> > > > There aren't many good options out there for users to get an
> > environment
> > > up
> > > > with apex apps running easily or quickly. We all know how difficult
> it
> > > > would be for someone new to get an application up and running even if
> > > they
> > > > had a hadoop sandbox with apex. Also, for many when they download a
> > > sandbox
> > > > they not only want to be able to run something in a few steps, but
> also
> > > > have easy to use tools and see something visually. My suggestion is
> to
> > > keep
> > > > both the sandboxes. You can put the bigtop sandbox first in the list
> as
> > > it
> > > > is vendor neutral.
> > > >
> > > > Thanks
> > > >
> >
>
>
> On Thu, Apr 6, 2017 at 7:18 PM, Thomas Weise  wrote:
>
> > +1 this should be tracked in a JIRA.
> >
> > There are also some improvements that can be done to the instructions on
> > the docker hub (separate activity).
> >
> > I would also suggest to list the Apex binary build on the downloads page
> > for users that have an existing cluster:
> >
> > https://github.com/atrato/apex-cli-package/releases
> >
> > And perhaps mention in the setup tutorial that these other download
> options
> > are listed on the website.
> >
> > Thanks,
> > Thomas
>
>
>
>
>
> > > > On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard <
> > dean.lockga...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > In the Sandbox section of the Apache Apex Development Environment
> > Setup
> > > > > documentation (
> > > > > https://apex.apache.org/docs/apex/apex_development_setup/#sandbox
> ),
> > > > > instructions are provided for a vendor-specific Sandbox.
> > > > >
> > > > > I would like to propose that these instructions be changed to
> > reference
> > > > the
> > > > > Apache Apex Sandbox instead, which is an Apache Bigtop build and
> > > > available
> > > > > via docker at https://hub.docker.com/r/apacheapex/sandbox.
> > > > >
> > > > > Regards,
> > > > > Dean
> > > > >
> > > >
> > >
> >
>

[jira] [Resolved] (APEXCORE-677) Avoid starting StramLocalCluster in StreamingContainerManagerTest.testAppDataSources

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov resolved APEXCORE-677.
-
   Resolution: Fixed
Fix Version/s: 3.6.0

> Avoid starting StramLocalCluster in 
> StreamingContainerManagerTest.testAppDataSources
> 
>
> Key: APEXCORE-677
> URL: https://issues.apache.org/jira/browse/APEXCORE-677
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
> Fix For: 3.6.0
>
>
> testAppDataSources does not wait for the StramLocalCluster to be initialized, 
> so it does not need to start it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apex Sandbox in Apex Dev Setup documentation

[Pramod Immaneni]

Wouldn't it then be better then if we make this more flexible and
dynamic. The documentation can contain some basic description of a sandbox
is for and point to the apex download page instead. In the download page we
can list the sandboxes. For the vendor specific sandbox today in question,
there is already a link that points to the general download page. I don't
think another link would be necessary. The verbiage of the text can be
changed to include the text sandbox.

On Fri, Apr 7, 2017 at 1:27 PM, Vlad Rozov  wrote:

> I would argue that a link to a vendor sandbox or a download that makes it
> easier to jump start with Apex and promotes adoption is OK. For example
> Hadoop provides links to HDP and CDH sandboxes:
> https://wiki.apache.org/hadoop/QuickStart. Additionally, bigtop sandbox
> does not seem to support VirtualBox out of the box, so why not to list both?
>
> Thank you,
>
> Vlad
>
>
> On 4/7/17 09:09, Thomas Weise wrote:
>
>> I disagree. Apex documentation is not the place to promote vendor
>> offerings.
>>
>> I suggested to refer to the download page, which already contains the
>> DataTorrent link.
>>
>> Thomas
>>
>>
>> On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
>> wrote:
>>
>> Hi Dean,
>>>
>>> There aren't many good options out there for users to get an environment
>>> up
>>> with apex apps running easily or quickly. We all know how difficult it
>>> would be for someone new to get an application up and running even if
>>> they
>>> had a hadoop sandbox with apex. Also, for many when they download a
>>> sandbox
>>> they not only want to be able to run something in a few steps, but also
>>> have easy to use tools and see something visually. My suggestion is to
>>> keep
>>> both the sandboxes. You can put the bigtop sandbox first in the list as
>>> it
>>> is vendor neutral.
>>>
>>> Thanks
>>>
>>> On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard >> >
>>> wrote:
>>>
>>> All,

 In the Sandbox section of the Apache Apex Development Environment Setup
 documentation (
 https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
 instructions are provided for a vendor-specific Sandbox.

 I would like to propose that these instructions be changed to reference

>>> the
>>>
 Apache Apex Sandbox instead, which is an Apache Bigtop build and

>>> available
>>>
 via docker at https://hub.docker.com/r/apacheapex/sandbox.

 Regards,
 Dean


>

[jira] [Commented] (APEXCORE-677) Avoid starting StramLocalCluster in StreamingContainerManagerTest.testAppDataSources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961407#comment-15961407
 ] 

ASF GitHub Bot commented on APEXCORE-677:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/492


> Avoid starting StramLocalCluster in 
> StreamingContainerManagerTest.testAppDataSources
> 
>
> Key: APEXCORE-677
> URL: https://issues.apache.org/jira/browse/APEXCORE-677
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Trivial
>
> testAppDataSources does not wait for the StramLocalCluster to be initialized, 
> so it does not need to start it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-core pull request #492: APEXCORE-677 Avoid starting StramLocalCluster i...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Design] - Kudu Output Operator

[Amol Kekre]

Ananth,
This is good proposal. We will work with you.

Thks
Amol


E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*

www.datatorrent.com


On Sat, Apr 1, 2017 at 4:29 PM, ananth  wrote:

> Hello All,
>
> I would like to the community's opinion on the implementation of Kudu
> output operator.  A first cut implementation was made available in November
> last year but I guess we did not get time to discuss this thoroughly on the
> mailing list and hence the PR did not get merged.
>
> This operator would allow Apex to stream data into Kudu. A brief
> description of Kudu is here : https://kudu.apache.org/. This would allow
> at a high level the following use cases from Apex point of view:
>
> - Low latency writes into Kudu store that allows SQL queries on the Kudu
> store. This essentially means sub-second data updates available for SQL
> querying. As opposed to parquet styled data dumps which would ideally need
> a few minutes to accumulate data to take advantage of Parquet formats, this
> would make same second queries on very large datasets on Kudu with Impala.
>
> - Another very interesting use cases would be to allow Kudu as a source
> store to stream based on SQL queries. The kudu input operator is another
> JIRA(https://issues.apache.org/jira/browse/APEXMALHAR-2472) and would be
> covering mechanisms to stream data from Kudu into Apex. This will bring in
> interesting use cases like de-dupe and selective streaming and out of band
> data in a different way if Kudu is part of the eco system in a given setup.
>
> Here is the design of the Kudu output operator:
>
>
> 1. The operator would be an AbstractOperator and would allow the concrete
> implementations to set a few behavioral aspects of the operator.
>
> 2. The following are the major phases of the operator:
>
> During activate() phase of the operator : Establish a connection to the
> cluster and get the metadata about the table that is being used as the sink.
> During setup() phase of the operator: Fetch the current window information
> and use it decide if we are recovering from a failure mode. (See point 8
> below )
> During process() of Input port : Inspect the incoming ExecutionContext (
> see below ) tuple and perform one of the operations (
> Insert/Update/Delete/Upsert)
> 3. The following parameters are tunable while establishing a Kudu
> connection:
> Table name, Boss worker threads, Worker threads, Socket read time outs and
> External Consistency mode.
> 4. The user need not specify any schema outright. The pojo fields are
> automatically mapped to the table column names as identified in the schema
> parse in the activate phase.
> 5. Allow the concrete implementation of the operator to override the Pojo
> field name to the table schema column name. This would allow flexibility in
> use cases like table schema column names are not compatible with java bean
> frameworks or in situations when column names cant be controlled as POJO is
> coming from an upstream operator.
> 6. The input tuple that is to be supplied to this operator is of type
> "Kudu Execution Context". This tuple encompasses the actual Pojo that is
> going to be persisted to the Kudu store. Additionally it allows the
> upstream operator to specify the operation that needs to be performed. One
> of the following operations is permitted as part of the context : Insert,
> Upsert, Update and delete on the Pojo that is acting as the payload in the
> Execution Context.
> 7. The concrete implementation of the operator would allow the user to
> specify the actual POJO class definition that would be used to the write to
> the table. The execution context would contain this POJO as well as the
> metadata that defines the behavior of the processing that needs to be done
> on that tuple.
> 8. The operator would allow for a special case of execution mode for the
> first window that is being processed as the operator gets activated. There
> are two modes for the first window of processing of the operator :
> a. Safe Mode : Safe mode is the "happy path execution" as in no extra
> processing is required to perform the Kudu mutation.
> b. Reconciling Mode: There is an additional function that would be
> called to see if the user would like the tuple to be used for mutation.
> This mode is automatically set when OperatorContext.ACTIVATION_WINDOW_ID
> != Stateless.WINDOW_ID during the first window of processing by the
> operator.
>
> This feature is deemed to be useful when an operator is recovering from a
> crash instance of the application and we do not want to perform multiple
> mutations of the same tuple given ATLEAST_ONCE is the default semantics.
>
> 9. The operator is a stateless operator.
> 10. The operator would generate the following autometrics :
>  a. Counts of Inserts, Upserts, Deletes and Updates (separate counters
> for each mutation) for a given window
>  b. Bytes written in a given window
>  c. Write RPCs in the given 

Re: Apex Sandbox in Apex Dev Setup documentation

[Dean Lockgaard]

Thanks for the input, Thomas and Pramod.

I've created the following two tickets to track this:

https://issues.apache.org/jira/browse/APEXCORE-692
https://issues.apache.org/jira/browse/APEXCORE-693

Regards,
Dean




On Fri, Apr 7, 2017 at 9:22 AM, Pramod Immaneni 
wrote:

> Agreed, didn't realize this was in the documentation page.
>
> On Fri, Apr 7, 2017 at 9:09 AM, Thomas Weise  wrote:
>
> > I disagree. Apex documentation is not the place to promote vendor
> > offerings.
> >
> > I suggested to refer to the download page, which already contains the
> > DataTorrent link.
> >
> > Thomas
> >
> >
> > On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
> > wrote:
> >
> > > Hi Dean,
> > >
> > > There aren't many good options out there for users to get an
> environment
> > up
> > > with apex apps running easily or quickly. We all know how difficult it
> > > would be for someone new to get an application up and running even if
> > they
> > > had a hadoop sandbox with apex. Also, for many when they download a
> > sandbox
> > > they not only want to be able to run something in a few steps, but also
> > > have easy to use tools and see something visually. My suggestion is to
> > keep
> > > both the sandboxes. You can put the bigtop sandbox first in the list as
> > it
> > > is vendor neutral.
> > >
> > > Thanks
> > >
>


On Thu, Apr 6, 2017 at 7:18 PM, Thomas Weise  wrote:

> +1 this should be tracked in a JIRA.
>
> There are also some improvements that can be done to the instructions on
> the docker hub (separate activity).
>
> I would also suggest to list the Apex binary build on the downloads page
> for users that have an existing cluster:
>
> https://github.com/atrato/apex-cli-package/releases
>
> And perhaps mention in the setup tutorial that these other download options
> are listed on the website.
>
> Thanks,
> Thomas





> > > On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard <
> dean.lockga...@gmail.com
> > >
> > > wrote:
> > >
> > > > All,
> > > >
> > > > In the Sandbox section of the Apache Apex Development Environment
> Setup
> > > > documentation (
> > > > https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
> > > > instructions are provided for a vendor-specific Sandbox.
> > > >
> > > > I would like to propose that these instructions be changed to
> reference
> > > the
> > > > Apache Apex Sandbox instead, which is an Apache Bigtop build and
> > > available
> > > > via docker at https://hub.docker.com/r/apacheapex/sandbox.
> > > >
> > > > Regards,
> > > > Dean
> > > >
> > >
> >
>

Re: Apex Sandbox in Apex Dev Setup documentation

[Vlad Rozov]

I would argue that a link to a vendor sandbox or a download that makes 
it easier to jump start with Apex and promotes adoption is OK. For 
example Hadoop provides links to HDP and CDH sandboxes: 
https://wiki.apache.org/hadoop/QuickStart. Additionally, bigtop sandbox 
does not seem to support VirtualBox out of the box, so why not to list 
both?


Thank you,

Vlad

On 4/7/17 09:09, Thomas Weise wrote:

I disagree. Apex documentation is not the place to promote vendor
offerings.

I suggested to refer to the download page, which already contains the
DataTorrent link.

Thomas


On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
wrote:


Hi Dean,

There aren't many good options out there for users to get an environment up
with apex apps running easily or quickly. We all know how difficult it
would be for someone new to get an application up and running even if they
had a hadoop sandbox with apex. Also, for many when they download a sandbox
they not only want to be able to run something in a few steps, but also
have easy to use tools and see something visually. My suggestion is to keep
both the sandboxes. You can put the bigtop sandbox first in the list as it
is vendor neutral.

Thanks

On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard 
wrote:


All,

In the Sandbox section of the Apache Apex Development Environment Setup
documentation (
https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
instructions are provided for a vendor-specific Sandbox.

I would like to propose that these instructions be changed to reference

the

Apache Apex Sandbox instead, which is an Apache Bigtop build and

available

via docker at https://hub.docker.com/r/apacheapex/sandbox.

Regards,
Dean

[jira] [Assigned] (APEXCORE-693) Add Apex binary build download for existing clusters

[Dean Lockgaard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dean Lockgaard reassigned APEXCORE-693:
---

Assignee: Dean Lockgaard

> Add Apex binary build download for existing clusters
> 
>
> Key: APEXCORE-693
> URL: https://issues.apache.org/jira/browse/APEXCORE-693
> Project: Apache Apex Core
>  Issue Type: Improvement
>  Components: Website
>Reporter: Dean Lockgaard
>Assignee: Dean Lockgaard
>Priority: Minor
>
> Add the following Apex binary build on the website downloads page for users 
> with an existing cluster:
> https://github.com/atrato/apex-cli-package/releases



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXCORE-693) Add Apex binary build download for existing clusters

[Dean Lockgaard (JIRA)]

Dean Lockgaard created APEXCORE-693:
---

 Summary: Add Apex binary build download for existing clusters
 Key: APEXCORE-693
 URL: https://issues.apache.org/jira/browse/APEXCORE-693
 Project: Apache Apex Core
  Issue Type: Improvement
  Components: Website
Reporter: Dean Lockgaard
Priority: Minor


Add the following Apex binary build on the website downloads page for users 
with an existing cluster:
https://github.com/atrato/apex-cli-package/releases




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (APEXCORE-692) Apex Dev Setup doc should reference Apex Sandbox

[Dean Lockgaard (JIRA)]

Dean Lockgaard created APEXCORE-692:
---

 Summary: Apex Dev Setup doc should reference Apex Sandbox
 Key: APEXCORE-692
 URL: https://issues.apache.org/jira/browse/APEXCORE-692
 Project: Apache Apex Core
  Issue Type: Documentation
  Components: Documentation
Reporter: Dean Lockgaard
Priority: Minor


In the Sandbox section of the Apache Apex Development Environment Setup 
documentation 
(https://apex.apache.org/docs/apex/apex_development_setup/#sandbox), 
instructions are provided for a vendor-specific Sandbox.

These instructions should be changed to reference the Apache Apex Sandbox 
instead (https://hub.docker.com/r/apacheapex/sandbox).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-malhar pull request #599: Apexmalhar 2465

[prasannapramod]

Github user prasannapramod closed the pull request at:

https://github.com/apache/apex-malhar/pull/599


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXCORE-691) Use type inference for generic instance creation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961129#comment-15961129
 ] 

ASF GitHub Bot commented on APEXCORE-691:
-

GitHub user vrozov opened a pull request:

https://github.com/apache/apex-core/pull/505

APEXCORE-691 Use type inference for generic instance creation

@tweise Please merge

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vrozov/apex-core APEXCORE-691

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit d15c0781c11c3b673d4af05a855b17a9b3efa625
Author: Apex Dev 
Date:   2017-04-07T17:08:45Z

APEXCORE-691 Use type inference for generic instance creation




> Use type inference for generic instance creation
> 
>
> Key: APEXCORE-691
> URL: https://issues.apache.org/jira/browse/APEXCORE-691
> Project: Apache Apex Core
>  Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
>
> The current Java language level allows using type inference for generic 
> instance creation making constructs like
> {noformat}
> Map  map = new HashMap();
> {noformat}
> unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-core pull request #505: APEXCORE-691 Use type inference for generic ins...

[vrozov]

GitHub user vrozov opened a pull request:

https://github.com/apache/apex-core/pull/505

APEXCORE-691 Use type inference for generic instance creation

@tweise Please merge

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vrozov/apex-core APEXCORE-691

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-core/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit d15c0781c11c3b673d4af05a855b17a9b3efa625
Author: Apex Dev 
Date:   2017-04-07T17:08:45Z

APEXCORE-691 Use type inference for generic instance creation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (APEXCORE-691) Use type inference for generic instance creation

[Vlad Rozov (JIRA)]

Vlad Rozov created APEXCORE-691:
---

 Summary: Use type inference for generic instance creation
 Key: APEXCORE-691
 URL: https://issues.apache.org/jira/browse/APEXCORE-691
 Project: Apache Apex Core
  Issue Type: Task
Reporter: Vlad Rozov
Assignee: Vlad Rozov
Priority: Minor


The current Java language level allows using type inference for generic 
instance creation making constructs like
{noformat}
Map  map = new HashMap();
{noformat}
unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apex Sandbox in Apex Dev Setup documentation

[Pramod Immaneni]

Agreed, didn't realize this was in the documentation page.

On Fri, Apr 7, 2017 at 9:09 AM, Thomas Weise  wrote:

> I disagree. Apex documentation is not the place to promote vendor
> offerings.
>
> I suggested to refer to the download page, which already contains the
> DataTorrent link.
>
> Thomas
>
>
> On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
> wrote:
>
> > Hi Dean,
> >
> > There aren't many good options out there for users to get an environment
> up
> > with apex apps running easily or quickly. We all know how difficult it
> > would be for someone new to get an application up and running even if
> they
> > had a hadoop sandbox with apex. Also, for many when they download a
> sandbox
> > they not only want to be able to run something in a few steps, but also
> > have easy to use tools and see something visually. My suggestion is to
> keep
> > both the sandboxes. You can put the bigtop sandbox first in the list as
> it
> > is vendor neutral.
> >
> > Thanks
> >
> > On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard  >
> > wrote:
> >
> > > All,
> > >
> > > In the Sandbox section of the Apache Apex Development Environment Setup
> > > documentation (
> > > https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
> > > instructions are provided for a vendor-specific Sandbox.
> > >
> > > I would like to propose that these instructions be changed to reference
> > the
> > > Apache Apex Sandbox instead, which is an Apache Bigtop build and
> > available
> > > via docker at https://hub.docker.com/r/apacheapex/sandbox.
> > >
> > > Regards,
> > > Dean
> > >
> >
>

Re: Apex Sandbox in Apex Dev Setup documentation

[Thomas Weise]

I disagree. Apex documentation is not the place to promote vendor
offerings.

I suggested to refer to the download page, which already contains the
DataTorrent link.

Thomas


On Fri, Apr 7, 2017 at 8:52 AM, Pramod Immaneni 
wrote:

> Hi Dean,
>
> There aren't many good options out there for users to get an environment up
> with apex apps running easily or quickly. We all know how difficult it
> would be for someone new to get an application up and running even if they
> had a hadoop sandbox with apex. Also, for many when they download a sandbox
> they not only want to be able to run something in a few steps, but also
> have easy to use tools and see something visually. My suggestion is to keep
> both the sandboxes. You can put the bigtop sandbox first in the list as it
> is vendor neutral.
>
> Thanks
>
> On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard 
> wrote:
>
> > All,
> >
> > In the Sandbox section of the Apache Apex Development Environment Setup
> > documentation (
> > https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
> > instructions are provided for a vendor-specific Sandbox.
> >
> > I would like to propose that these instructions be changed to reference
> the
> > Apache Apex Sandbox instead, which is an Apache Bigtop build and
> available
> > via docker at https://hub.docker.com/r/apacheapex/sandbox.
> >
> > Regards,
> > Dean
> >
>

Re: Apex Sandbox in Apex Dev Setup documentation

[Pramod Immaneni]

Hi Dean,

There aren't many good options out there for users to get an environment up
with apex apps running easily or quickly. We all know how difficult it
would be for someone new to get an application up and running even if they
had a hadoop sandbox with apex. Also, for many when they download a sandbox
they not only want to be able to run something in a few steps, but also
have easy to use tools and see something visually. My suggestion is to keep
both the sandboxes. You can put the bigtop sandbox first in the list as it
is vendor neutral.

Thanks

On Thu, Apr 6, 2017 at 1:16 PM, Dean Lockgaard 
wrote:

> All,
>
> In the Sandbox section of the Apache Apex Development Environment Setup
> documentation (
> https://apex.apache.org/docs/apex/apex_development_setup/#sandbox),
> instructions are provided for a vendor-specific Sandbox.
>
> I would like to propose that these instructions be changed to reference the
> Apache Apex Sandbox instead, which is an Apache Bigtop build and available
> via docker at https://hub.docker.com/r/apacheapex/sandbox.
>
> Regards,
> Dean
>

[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960602#comment-15960602
 ] 

ASF GitHub Bot commented on APEXCORE-408:
-

Github user tushargosavi closed the pull request at:

https://github.com/apache/apex-core/pull/410


> Ability to schedule Sub-DAG from running application
> 
>
> Key: APEXCORE-408
> URL: https://issues.apache.org/jira/browse/APEXCORE-408
> Project: Apache Apex Core
>  Issue Type: Sub-task
>Reporter: Thomas Weise
>Assignee: Tushar Gosavi
>
> Today it is possible to add operators to a running application from the 
> CLI/client. It should be possible to do this from within the application 
> also. This can be used to expand/remove a Sub-DAG on demand, triggered by 
> application specific logic. It will enable use cases such as batch 
> applications that perform multiple stages of processing where not all 
> resources are required at the same time or the logic of subsequent steps 
> depends on the execution of previous steps.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] apex-core pull request #410: APEXCORE-408: Ability to schedule Sub-DAG from ...

[tushargosavi]

Github user tushargosavi closed the pull request at:

https://github.com/apache/apex-core/pull/410


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (APEXMALHAR-2296) Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2296:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add support for SQL GROUP BY (Aggregate RelNode)
> --
>
> Key: APEXMALHAR-2296
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2296
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Add support for SQL GROUP BY (Aggregate RelNode)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2295) Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2295:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Replace POJOInnerJoinOperator with WindowedJoinOperator
> -
>
> Key: APEXMALHAR-2295
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2295
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Replace POJOInnerJoinOperator with WindowedJoinOperator



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2294) Apex SQL: Make operator names user friendly for user to configure via properties.xml

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2294:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Make operator names user friendly for user to configure via 
> properties.xml
> 
>
> Key: APEXMALHAR-2294
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2294
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Make operator names user friendly for user to configure via properties.xml



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2293) Apex SQL: Add sql line cli to sql support of apex

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2293:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add sql line cli to sql support of apex
> -
>
> Key: APEXMALHAR-2293
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2293
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>  Components: sql
>Reporter: Chinmay Kolhatkar
>
> Add cli like support to malhar-sql.
> The support can be done similar to: 
> https://github.com/apache/storm/blob/master/external/sql/README.md
> It should support following type of statements:
> 1. CREATE EXTERNAL TABLE with kafka, file, jdbc specifications
> 2. DROP EXTERNAL TABLE
> 3. CREATE FUNCTION
> 4. DELETE FUNCTION
> 5. INSERT and SELECT statements



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (APEXMALHAR-2292) Apex SQL: Add generic endpoint to malhar-sql

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned APEXMALHAR-2292:
-

Assignee: (was: Chinmay Kolhatkar)

> Apex SQL: Add generic endpoint to malhar-sql
> 
>
> Key: APEXMALHAR-2292
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2292
> Project: Apache Apex Malhar
>  Issue Type: Sub-task
>  Components: sql
>Affects Versions: 3.5.0
>Reporter: Chinmay Kolhatkar
>
> This includes exploring generalization of the endpoint concept as a follow-up 
> to APEXMALHAR-1818. Right now Kafka is hard-wired, but there are many other 
> connectors and they may be custom. Perhaps something like a registry and 
> generic configuration through both, table properties and the Apex 
> configuration file?
> Once exploration is done, implement it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)