[DISCUSS] Quarks proposal

[Katherine Marsden]

Find below a draft proposal for a new incubator project, Quarks for 
discussion.  Quarks is seeking experienced mentors  as well as 
contributors to the project.  Please discuss and provide feedback. The 
proposal is also available on the Wiki at 
https://wiki.apache.org/incubator/QuarksProposal 



Thanks

Kathey Marsden

= Quarks Proposal =
=== Abstract ===
Quarks is a a stream processing programming model and lightweight 
runtime to execute analytics at devices on the edge or at the gateway.


=== Proposal ===
 Quarks  is a programming model and runtime for streaming analytics at 
the edge.   Applications are developed using a functional flow api to  
define operations on data streams that is executed as a graph of 
"oplets" in a lightweight embeddable runtime.   The SDK provides 
capabilities like windowing, aggregation  and connectors with an 
extensible model for the community to expand its capabilities.


=== Background ===
 Stream processing systems are commonly used to process  data from edge 
devices and there is a need to push some of the  streaming analytics to 
the edge to reduce communication costs, react  locally and offload 
processing from the central systems.  Quarks was developed by IBM as an 
entirely new project to provide an SDK  and lightweight embeddable 
runtime for streaming analytics at the edge.   Quarks was created to be 
an open source project that could provide edge  analytics to a broad 
community and foster collaboration on common  analytics and connectors 
across a broad ecosystem of devices.


=== Rationale ===
 With the growth in number of connected devices (Internet of Things)  
there is a need to execute analytics at the edge in order to take local  
actions based upon sensor information and/or reduce the volume of data  
sent to back-end analytic systems to reduce communication cost.
 Quarks rationale is to provide  consistent and easy to use programming 
models to allow application developers to focus on their  application 
rather than issues like device connectivity, threading etc.   Quarks' 
functional data flow programming model is similar to systems  like 
Apache Flink, Google DataFlow, Java 8 Streams & Apache Spark.   The API 
currently has language bindings for Java8, Java7 and Android. Quarks was 
developed to address requirements for analytics at the  edge for IoT use 
cases that were not addressed by central analytic  solutions.  We 
believe that these capabilities will be useful to many  organizations 
and that the diverse nature of edge devices and use cases  is best 
addressed by an open community. Therefore, we would like to contribute 
Quarks to the ASF as an open source project and begin developing a 
community of developers and users within Apache.


=== Initial Goals ===
 Quarks initial code contribution provides:

 * APIs for developing applications that execute  analytics using a 
per-event (data item) streaming paradigm including  support for windows  
against a stream for aggregation

 * A micro-kernel style runtime for execution.
 * Connectors for MQTT, HTTP, JDBC, File, Apache Kafka &  IBM Watson 
IoT Platform

 * Simple analytics aimed at device sensors (using Apache Common Math)
 * Development mode including a web-console to view the graph of 
running applications
 * Testing mechanism for Quarks applications that integrates with 
assertion based testing systems like JUnit
 * Android specific functionality such as producing a stream that 
contains a phone's sensor events (e.g. ambient temperature, pressure)

 * JUnit tests

 All of the initial code is implemented using Java 8 and when built 
produces jars that can execute on Java 8, Java 7 and Android. The goal 
is to encourage community contributions in any area of Quarks, to  
expand the community (including new committers) and use of Quarks. We  
expect contributions will be driven by real-world use of Quarks by  
anyone active in the IoT space such as auto manufactures, insurance  
companies, etc. as well as individuals experimenting with devices such  
as Raspberry Pis, Arduinos and/or smart phone apps etc. Contributions 
would be welcomed in any aspect of Quarks including:


 * Support for additional programming languages used in devices such as 
C, OpenSwift, Python etc.
 * Specific device feature (e.g. Raspberry Pi, Android) or protocol 
(e.g. OBD-2) support
 * Connectors for device to device (e.g. AllJoyn), device local data 
sources,  or to back-end systems (e.g. a IoT cloud service)
 * Additional analytics, either exposing more functionality from Apache 
Common Math, other libraries or hand-coded analytics.
 * Improvements to the development console, e.g. additional 
visualizations of running applications
 * Documentation, improving existing documentation or adding new guides 
etc.

 * Sample applications
 * Testing

 The  code base has been designed to be modular so that additional 
functionality can be added without having to learn

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/15/2016 4:59 PM, Katherine Marsden wrote:
Find below a draft proposal for a new incubator project, Quarks for 
discussion.  Quarks is seeking experienced mentors  as well as 
contributors to the project.  Please discuss and provide feedback. The 
proposal is also available on the Wiki at 
https://wiki.apache.org/incubator/QuarksProposal 



Please forgive. The JoshuaProposal link was erroneous. I am not sure how 
that got in there.

Only https://wiki.apache.org/incubator/QuarksProposal is relevant.


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Sandeep Deshmukh]

Hi All,

I would be happy to be part of this interesting project. Processing at the
edge nodes is going to help a lot in analyzing @ scaling in  IoT space.

While going through the proposal, I noticed following:

   1. Email id of Daniel Debrunner is djd at *apache* dot *com*. Should
   that be dot *org* ?
   2. Same with email id of Dale LaBossiere

Regards,
Sandeep

On Tue, Feb 16, 2016 at 6:35 AM, Katherine Marsden 
wrote:

> On 2/15/2016 4:59 PM, Katherine Marsden wrote:
>
>> Find below a draft proposal for a new incubator project, Quarks for
>> discussion.  Quarks is seeking experienced mentors  as well as contributors
>> to the project.  Please discuss and provide feedback. The proposal is also
>> available on the Wiki at https://wiki.apache.org/incubator/QuarksProposal
>> 
>>
>> Please forgive. The JoshuaProposal link was erroneous. I am not sure how
> that got in there.
> Only https://wiki.apache.org/incubator/QuarksProposal is relevant.
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/17/2016 9:52 PM, Sandeep Deshmukh wrote:

I would be happy to be part of this interesting project.

Wonderful! Welcome and thank you for reviewing the proposal!
I created an Additional Interested Contributors section and added you 
and May Wone who expressed interest in contributing as well.



While going through the proposal, I noticed following:

1. Email id of Daniel Debrunner is djd at *apache* dot *com*. Should
that be dot *org* ?

Fixed.


2. Same with email id of Dale LaBossiere
I think Dale's email is ok. It is a gmail dot com email as he doesn't 
have an apache account yet.


One additional edit I made was to change DataFlow references to Beam to 
reflect the name change.


Updated proposal is available at: 
https://wiki.apache.org/incubator/QuarksProposal



Best

Kathey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Luciano Resende]

On Thursday, February 18, 2016, Katherine Marsden 
wrote:

> On 2/17/2016 9:52 PM, Sandeep Deshmukh wrote:
>
>> I would be happy to be part of this interesting project.
>>
> Wonderful! Welcome and thank you for reviewing the proposal!
> I created an Additional Interested Contributors section and added you and
> May Wone who expressed interest in contributing as well.
>
>
What is the intent of the new section ? I would say, either add possible
contributors as initial committers or don't add at all as it will cause
confusion.


>
>

-- 
Sent from my Mobile device

Re: [DISCUSS] Quarks proposal

[Bhupesh Chawda]

Hi All,

Seems to be a nice idea for offloading processing from the centralized
systems.
I am just trying to understand the need for streaming analytics engine at
the edge devices. An example use case justifying the need for such systems
would definitely help.

Thanks.
-Bhupesh

On Fri, Feb 19, 2016 at 10:27 AM, Luciano Resende 
wrote:

> On Thursday, February 18, 2016, Katherine Marsden 
> wrote:
>
> > On 2/17/2016 9:52 PM, Sandeep Deshmukh wrote:
> >
> >> I would be happy to be part of this interesting project.
> >>
> > Wonderful! Welcome and thank you for reviewing the proposal!
> > I created an Additional Interested Contributors section and added you and
> > May Wone who expressed interest in contributing as well.
> >
> >
> What is the intent of the new section ? I would say, either add possible
> contributors as initial committers or don't add at all as it will cause
> confusion.
>
>
> >
> >
>
> --
> Sent from my Mobile device
>



-- 
Regards,
Bhupesh Chawda

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/18/2016 8:57 PM, Luciano Resende wrote:

On Thursday, February 18, 2016, Katherine Marsden 
wrote:

I created an Additional Interested Contributors section



What is the intent of the new section ? I would say, either add possible
contributors as initial committers or don't add at all as it will cause
confusion.
I don't think it is confusing, but rather highlights the project's 
potential for growth in numbers and diversity and recognizes folks who 
have expressed interest in contributing, but for whom we don't have 
details yet.  I don't feel super strongly about keeping the section in 
though,  so can take it out if you or others  feel strongly about it.


Best

Kathey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Seetharam Venkatesh]

Apache Beam (incubating) did just this and there were a lot of 'em who
signed up as interested contributors. Its not clear as to what it means
though.

Thanks!

On Fri, Feb 19, 2016 at 9:09 AM Katherine Marsden 
wrote:

> On 2/18/2016 8:57 PM, Luciano Resende wrote:
> > On Thursday, February 18, 2016, Katherine Marsden 
> > wrote:
> >> I created an Additional Interested Contributors section
> >>
> >>
> > What is the intent of the new section ? I would say, either add possible
> > contributors as initial committers or don't add at all as it will cause
> > confusion.
> I don't think it is confusing, but rather highlights the project's
> potential for growth in numbers and diversity and recognizes folks who
> have expressed interest in contributing, but for whom we don't have
> details yet.  I don't feel super strongly about keeping the section in
> though,  so can take it out if you or others  feel strongly about it.
>
> Best
>
> Kathey
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Quarks proposal

[Henry Saputra]

Officially it means nothing =P

But it is nice way to express interest and support to the proposal.

- Henry

On Fri, Feb 19, 2016 at 9:48 AM, Seetharam Venkatesh <
venkat...@innerzeal.com> wrote:

> Apache Beam (incubating) did just this and there were a lot of 'em who
> signed up as interested contributors. Its not clear as to what it means
> though.
>
> Thanks!
>
> On Fri, Feb 19, 2016 at 9:09 AM Katherine Marsden 
> wrote:
>
> > On 2/18/2016 8:57 PM, Luciano Resende wrote:
> > > On Thursday, February 18, 2016, Katherine Marsden  >
> > > wrote:
> > >> I created an Additional Interested Contributors section
> > >>
> > >>
> > > What is the intent of the new section ? I would say, either add
> possible
> > > contributors as initial committers or don't add at all as it will cause
> > > confusion.
> > I don't think it is confusing, but rather highlights the project's
> > potential for growth in numbers and diversity and recognizes folks who
> > have expressed interest in contributing, but for whom we don't have
> > details yet.  I don't feel super strongly about keeping the section in
> > though,  so can take it out if you or others  feel strongly about it.
> >
> > Best
> >
> > Kathey
> >
> > -
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > For additional commands, e-mail: general-h...@incubator.apache.org
> >
> >
>

Re: [DISCUSS] Quarks proposal

[Luciano Resende]

On Fri, Feb 19, 2016 at 10:13 AM, Henry Saputra 
wrote:

> Officially it means nothing =P
>
>
Exactly, this is what worried my a little bit.


> But it is nice way to express interest and support to the proposal.
>
>
Agree, so it's ok to leave there, as long as we understand is just a way to
express support and officially it does not mean anything on the proposal
content.


> - Henry
>
> On Fri, Feb 19, 2016 at 9:48 AM, Seetharam Venkatesh <
> venkat...@innerzeal.com> wrote:
>
> > Apache Beam (incubating) did just this and there were a lot of 'em who
> > signed up as interested contributors. Its not clear as to what it means
> > though.
> >
> > Thanks!
> >
> > On Fri, Feb 19, 2016 at 9:09 AM Katherine Marsden 
> > wrote:
> >
> > > On 2/18/2016 8:57 PM, Luciano Resende wrote:
> > > > On Thursday, February 18, 2016, Katherine Marsden <
> kmars...@apache.org
> > >
> > > > wrote:
> > > >> I created an Additional Interested Contributors section
> > > >>
> > > >>
> > > > What is the intent of the new section ? I would say, either add
> > possible
> > > > contributors as initial committers or don't add at all as it will
> cause
> > > > confusion.
> > > I don't think it is confusing, but rather highlights the project's
> > > potential for growth in numbers and diversity and recognizes folks who
> > > have expressed interest in contributing, but for whom we don't have
> > > details yet.  I don't feel super strongly about keeping the section in
> > > though,  so can take it out if you or others  feel strongly about it.
> > >
> > > Best
> > >
> > > Kathey
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>



-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/18/2016 10:19 PM, Bhupesh Chawda wrote:

Hi All,

Seems to be a nice idea for offloading processing from the centralized
systems.
I am just trying to understand the need for streaming analytics engine at
the edge devices. An example use case justifying the need for such systems
would definitely help.

One such example might be engine or equipment monitoring.  You want to 
transmit data if parameters are out of range, (for example too hot), but 
don't want to communicate if things are operating within normal parameters.


The Quarks overview gives more details at:
http://quarks-edge.github.io/quarks.documentation/docs/quarks/overview/


Best

Kathey


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Luciano Resende]

On Fri, Feb 19, 2016 at 11:01 AM, Katherine Marsden 
wrote:

> On 2/18/2016 10:19 PM, Bhupesh Chawda wrote:
>
>> Hi All,
>>
>> Seems to be a nice idea for offloading processing from the centralized
>> systems.
>> I am just trying to understand the need for streaming analytics engine at
>> the edge devices. An example use case justifying the need for such systems
>> would definitely help.
>>
>> One such example might be engine or equipment monitoring.  You want to
> transmit data if parameters are out of range, (for example too hot), but
> don't want to communicate if things are operating within normal parameters.
>
>
Particularly if the sensors for example are in limited bandwidth
connections, etc


> The Quarks overview gives more details at:
> http://quarks-edge.github.io/quarks.documentation/docs/quarks/overview/
>
>
>
> Best
>
> Kathey
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [DISCUSS] Quarks proposal

[Dan Debrunner]

On Thursday, February 18, 2016 10:19 PM, Bhupesh Chawda 
 wrote:

I am just trying to understand the need for streaming analytics engine at
the edge devices. An example use case justifying the need for such systems
would definitely help.

There are at least a couple of drivers:
1) Intelligently reducing the volume of data sent from edge devices to central 
systems, for example transportation companies wanting to reduce their big 
monthly cellular bill.
2) Being able to locally detect & react to anomalies even when connectivity is 
not available.
Dan.

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

I really appreciate the input so far on this proposal.  I was wondering...

Is anyone else willing to be a mentor?  We have three volunteers, but 
would appreciate more.
Does anyone have any concerns that they have not expressed?  If not, I 
would like to call a vote Wednesday, February 24 or as soon as the IPMC 
approves Dan and I joining (we have both submitted member requests) , 
whichever comes later.


https://wiki.apache.org/incubator/QuarksProposal

Best

Kathey

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Luciano Resende]

On Fri, Feb 19, 2016 at 2:20 PM, Katherine Marsden 
wrote:

> I really appreciate the input so far on this proposal.  I was wondering...
>
> Is anyone else willing to be a mentor?  We have three volunteers, but
> would appreciate more.
> Does anyone have any concerns that they have not expressed?  If not, I
> would like to call a vote Wednesday, February 24 or as soon as the IPMC
> approves Dan and I joining (we have both submitted member requests) ,
> whichever comes later.
>
>
Just FYI, we don't need to wait for you and Dan to be officially part of
IPMC, vote can be started sooner (we had similar issue with couple mentors
on SystemML)


> https://wiki.apache.org/incubator/QuarksProposal
>
> Best
>
> Kathey
>
>
>
-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [DISCUSS] Quarks proposal

[Justin Mclean]

Hi,

>> Is anyone else willing to be a mentor?  We have three volunteers, but
>> would appreciate more.

You add add me if you want, I’ve a little busy right now so may no be so active 
initially.

I’m one f the mentors on several other incubating projects (including IoT 
projects) and a couple of those should graduate soon.

Thanks,
Justin


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/19/2016 3:36 PM, Luciano Resende wrote:

Just FYI, we don't need to wait for you and Dan to be officially part of
IPMC, vote can be started sooner (we had similar issue with couple mentors
on SystemML)

Thank you Luciano. That's good news! I will call the vote Wednesday 
then, unless something comes up.



-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Bhupesh Chawda]

Thanks for all the replies.
This definitely looks interesting, particularly how a Quarks application is
modeled as a streaming topology.

I would also like to be part of this project and contribute to it.

Thanks.
-Bhupesh

On Sat, Feb 20, 2016 at 12:31 AM, Katherine Marsden 
wrote:

> On 2/18/2016 10:19 PM, Bhupesh Chawda wrote:
>
>> Hi All,
>>
>> Seems to be a nice idea for offloading processing from the centralized
>> systems.
>> I am just trying to understand the need for streaming analytics engine at
>> the edge devices. An example use case justifying the need for such systems
>> would definitely help.
>>
>> One such example might be engine or equipment monitoring.  You want to
> transmit data if parameters are out of range, (for example too hot), but
> don't want to communicate if things are operating within normal parameters.
>
> The Quarks overview gives more details at:
> http://quarks-edge.github.io/quarks.documentation/docs/quarks/overview/
>
>
>
> Best
>
> Kathey
>
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Regards,
Bhupesh Chawda

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/19/2016 3:49 PM, Justin Mclean wrote:
You add add me if you want, I’ve a little busy right now so may no be 
so active initially.


Thank you Justin for volunteering.  It will be great to have a mentor 
with your experience on the project.



Best

Kathey



-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: [DISCUSS] Quarks proposal

[Katherine Marsden]

On 2/19/2016 4:25 PM, Bhupesh Chawda wrote:

Thanks for all the replies.
This definitely looks interesting, particularly how a Quarks application is
modeled as a streaming topology.

I would also like to be part of this project and contribute to it.

Thank you Bhupesh and welcome!


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org