date:20170330

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread Xin Wang

Jungtaek,

Currently no patches, the work is in progress. I will submit several days
later.

2017-03-31 14:18 GMT+08:00 Jungtaek Lim :

> Xin,
>
> Do you already have patches? Or just waiting for this before doing some
> works?
>
> 2017년 3월 31일 (금) 오후 3:10, Xin Wang 님이 작성:
>
> > +1 on keeping the current external/storm-* in place and move flux, sql,
> > storm-submit-tools into top-level.
> >
> > BTW, I am waiting for the discuss result before submitting several SQL
> > related PRs. :)
> >
> > - Xin
> >
> > 2017-03-31 10:41 GMT+08:00 Jungtaek Lim :
> >
> > > Yeah sure I'm OK to just apply it for master branch.
> > > Are you okay for moving them to root directory without renaming? Or do
> > you
> > > want to rename or suggest other base directory?
> > >
> > > 2017년 3월 31일 (금) 오전 11:36, P. Taylor Goetz 님이 작성:
> > >
> > > > I'm hesitant to change the layout of the 1.x release. People do some
> > > crazy
> > > > things when it comes to operations that we can't predict. I'm okay
> with
> > > > doing this on the master/2.0 branch, but I'm hesitant on the 1.x
> > branch.
> > > >
> > > > -Taylor
> > > >
> > > > > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim 
> wrote:
> > > > >
> > > > > Once we just released Storm 1.1.0, I guess we can continue
> discussion
> > > > here.
> > > > >
> > > > > I checked the PR list, and there're no open PRs for among Flux,
> Storm
> > > > SQL,
> > > > > and Storm submit tool. So it's good to go.
> > > > >
> > > > > I think we have consensus to move non-connectors modules to out of
> > > > > external. There're also some interest about renaming "external" to
> > > > > "connectors", but given that "external" is chosen by community and
> > has
> > > > been
> > > > > the norm for years, so I agree it would be better not to rename it.
> > We
> > > > can
> > > > > do it later when there's another chance of doing it.
> > > > >
> > > > > We didn't decide where to move, but most of us seems to be OK to
> move
> > > to
> > > > > the top directory.
> > > > > Taylor, any further opinion regarding this?
> > > > > Suppose we're moving them to the top directory (or whatever the
> same
> > > base
> > > > > directory), what would be good names for them? Flux doesn't have
> > prefix
> > > > > 'storm' so a bit different, but if we're OK we skip renaming it.
> > > > >
> > > > > I'll do the work when Taylor is OK for changing this.
> > > > >
> > > > > - Jungtaek Lim (HeartSaVioR)
> > > > >
> > > > > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:
> > > > >
> > > > >> 1. apply versions
> > > > >>
> > > > >> First plan was applying this only for master, but realized that
> > > > >> contributors should make two patches for every PRs when we apply
> > this
> > > to
> > > > >> only master.
> > > > >>
> > > > >> So in order to make less inconvenience, it would be better to
> apply
> > > this
> > > > >> for both 1.x and master. It also affects opened pull requests so
> we
> > > > would
> > > > >> like to check that relevant PRs are open, and apply it later than
> > > > reviewing
> > > > >> them.
> > > > >>
> > > > >> I agree with Harsha. No need to make change for current release
> > vote.
> > > > >>
> > > > >> 2. naming issue for "external"
> > > > >>
> > > > >> "external" makes me feel that it's related to "external"
> component,
> > > say,
> > > > >> outside of Storm. That's why I suggest moving non-connectors to
> out
> > of
> > > > >> "external". IMHO "connector" is still more intuitive and
> > > > self-describing,
> > > > >> but I understand that renaming the directory structure would be
> > > painful,
> > > > >> and "storm-kafka-monitor" is an example of what it's not a
> > "connector"
> > > > but
> > > > >> an "external". So I'm OK to keep it as "external".
> > > > >>
> > > > >> 3. where to move non-connectors
> > > > >>
> > > > >> Except Flux they're directly supported by Storm. I mean "storm.py"
> > is
> > > > >> aware of them and supports them, so for me they are eligible to
> move
> > > to
> > > > the
> > > > >> top directory. I'm open to other suggestion as well.
> > > > >>
> > > > >> - Jungtaek Lim (HeartSaVioR)
> > > > >>
> > > > >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani  >님이
> > > 작성:
> > > > >>
> > > > >> We should this in next release of 1.x or 2.0. I am +1 on continue
> > with
> > > > >> current release.
> > > > >> -Harsha
> > > > >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <
> ptgo...@gmail.com
> > >
> > > > wrote:
> > > > >>>
> > > > >>> The question remains if we want to do this in the 1.1.0 release,
> or
> > > > >> later.
> > > > >>>
> > > > >>> If it's the 1.1.0 release we need to make the changes and cut
> > another
> > > > RC.
> > > > >>> I'm fine with that, but want to make sure we have consensus
> before
> > > > going
> > > > >>> down that road.
> > > > >>>
> > > > >>> -Taylor
> > > > >>>
> > > >  On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <
> > st...@harsha.io>
> > > > >>> wrote:
> > > > 
> > > >  Agree on change like this would be confusing to the users

[GitHub] storm issue #2034: STORM-2441 Break down 'storm-core' to extract client (wor...

2017-03-30 Thread HeartSaVioR

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2034
  
Travis build fails intermittently. Travis build for fork passes (without 
integration test).
https://travis-ci.org/HeartSaVioR/storm/builds/217048964



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread Jungtaek Lim

Xin,

Do you already have patches? Or just waiting for this before doing some
works?

2017년 3월 31일 (금) 오후 3:10, Xin Wang 님이 작성:

> +1 on keeping the current external/storm-* in place and move flux, sql,
> storm-submit-tools into top-level.
>
> BTW, I am waiting for the discuss result before submitting several SQL
> related PRs. :)
>
> - Xin
>
> 2017-03-31 10:41 GMT+08:00 Jungtaek Lim :
>
> > Yeah sure I'm OK to just apply it for master branch.
> > Are you okay for moving them to root directory without renaming? Or do
> you
> > want to rename or suggest other base directory?
> >
> > 2017년 3월 31일 (금) 오전 11:36, P. Taylor Goetz 님이 작성:
> >
> > > I'm hesitant to change the layout of the 1.x release. People do some
> > crazy
> > > things when it comes to operations that we can't predict. I'm okay with
> > > doing this on the master/2.0 branch, but I'm hesitant on the 1.x
> branch.
> > >
> > > -Taylor
> > >
> > > > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim  wrote:
> > > >
> > > > Once we just released Storm 1.1.0, I guess we can continue discussion
> > > here.
> > > >
> > > > I checked the PR list, and there're no open PRs for among Flux, Storm
> > > SQL,
> > > > and Storm submit tool. So it's good to go.
> > > >
> > > > I think we have consensus to move non-connectors modules to out of
> > > > external. There're also some interest about renaming "external" to
> > > > "connectors", but given that "external" is chosen by community and
> has
> > > been
> > > > the norm for years, so I agree it would be better not to rename it.
> We
> > > can
> > > > do it later when there's another chance of doing it.
> > > >
> > > > We didn't decide where to move, but most of us seems to be OK to move
> > to
> > > > the top directory.
> > > > Taylor, any further opinion regarding this?
> > > > Suppose we're moving them to the top directory (or whatever the same
> > base
> > > > directory), what would be good names for them? Flux doesn't have
> prefix
> > > > 'storm' so a bit different, but if we're OK we skip renaming it.
> > > >
> > > > I'll do the work when Taylor is OK for changing this.
> > > >
> > > > - Jungtaek Lim (HeartSaVioR)
> > > >
> > > > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:
> > > >
> > > >> 1. apply versions
> > > >>
> > > >> First plan was applying this only for master, but realized that
> > > >> contributors should make two patches for every PRs when we apply
> this
> > to
> > > >> only master.
> > > >>
> > > >> So in order to make less inconvenience, it would be better to apply
> > this
> > > >> for both 1.x and master. It also affects opened pull requests so we
> > > would
> > > >> like to check that relevant PRs are open, and apply it later than
> > > reviewing
> > > >> them.
> > > >>
> > > >> I agree with Harsha. No need to make change for current release
> vote.
> > > >>
> > > >> 2. naming issue for "external"
> > > >>
> > > >> "external" makes me feel that it's related to "external" component,
> > say,
> > > >> outside of Storm. That's why I suggest moving non-connectors to out
> of
> > > >> "external". IMHO "connector" is still more intuitive and
> > > self-describing,
> > > >> but I understand that renaming the directory structure would be
> > painful,
> > > >> and "storm-kafka-monitor" is an example of what it's not a
> "connector"
> > > but
> > > >> an "external". So I'm OK to keep it as "external".
> > > >>
> > > >> 3. where to move non-connectors
> > > >>
> > > >> Except Flux they're directly supported by Storm. I mean "storm.py"
> is
> > > >> aware of them and supports them, so for me they are eligible to move
> > to
> > > the
> > > >> top directory. I'm open to other suggestion as well.
> > > >>
> > > >> - Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani 님이
> > 작성:
> > > >>
> > > >> We should this in next release of 1.x or 2.0. I am +1 on continue
> with
> > > >> current release.
> > > >> -Harsha
> > > >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz  >
> > > wrote:
> > > >>>
> > > >>> The question remains if we want to do this in the 1.1.0 release, or
> > > >> later.
> > > >>>
> > > >>> If it's the 1.1.0 release we need to make the changes and cut
> another
> > > RC.
> > > >>> I'm fine with that, but want to make sure we have consensus before
> > > going
> > > >>> down that road.
> > > >>>
> > > >>> -Taylor
> > > >>>
> > >  On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <
> st...@harsha.io>
> > > >>> wrote:
> > > 
> > >  Agree on change like this would be confusing to the users. Lets
> keep
> > > >> the
> > >  original plan of moving non-connectors modules of external instead
> > of
> > >  introducing new changes
> > >  that are not in scope of this discussion.
> > >  My +1 still stands on keeping the current external/storm-* in
> place
> > > and
> > >  move just sql and storm-perf into top-level. We can have
> discussion
> > > for
> > >  storm 2.0 if we want to do
> > >  more changes.
> > > 
> > > >>

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

2017-03-30 Thread Jungtaek Lim

Bobby,

I've worked on separating worker and daemon classpath.

- Issue: STORM-2441: Break down 'storm-core' to extract client (worker)
artifacts 
- PR: https://github.com/apache/storm/pull/2034

I don't address your suggestion about "classpath selection" and "hiding
local mode". Please file issues if you would like to address.

Btw, I exclude artifacts from shade & relocation list so still need to
address dependency issue.

Folks,

any other ideas or opinions around dependency issue?

IMHO Option 2 is clearer but not sure where we can create a new git repo
(ASF git or even outside), and also it's not against LICENSEs to repackage
shade & relocated artifacts to Maven.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 3월 29일 (수) 오후 10:42, Bobby Evans 님이 작성:

I am fine with those changes so long as we finish the separation of worker
and daemon classpaths.  Otherwise we have made some very big changes for
our end users that are going to have a hard time upgrading.
If all we support is the option to run an old worker version with a new
supervisor/nimbus I think that would be good enough, although I would like
to see a full separation of the classpaths.


- Bobby

On Tuesday, March 28, 2017, 6:03:26 PM CDT, Jungtaek Lim 
wrote:Just FYI:
I've worked with minimal patch for 3, though I still don't like such
workaround:
https://github.com/HeartSaVioR/storm/commit/d3122faa7ae182915242b979beaac156f91fe3b2

It excludes 'libthrift', 'jetty', 'codahale metrics' from relocation
targets. I can see IDEA is OK to build the project, and Maven build passing.

- Jungtaek Lim (HeartSaVioR)

2017년 3월 29일 (수) 오전 11:02, Jungtaek Lim 님이 작성:

> Back to origin issue (before breaking down 'storm-core'), turned out
> IntelliJ doesn't recognize relocated classes within project. That's why
> build (via Maven) for master branch succeeds but IDEA compile doesn't.
>
> There're some issues filed but no action has been made.
> https://youtrack.jetbrains.com/issue/IDEA-93855
> https://youtrack.jetbrains.com/issue/IDEA-126596
>
> So suppose we have two modules A and B within project, and A relocates L
> to Lr.
> B relies on A's method which returns a class of Lr or has parameters for a
> class of LR, B needs to use Lr rather than L, and Lr is not recognized
from
> B.
>
> Moving 'storm-drpc-server' to 'storm-core' may help but it's not a nice
> solution though. (think about why we add new module 'storm-drpc-server')
> To minimize dependency for worker (which actually affects end users) we
> should break down 'storm-core' and it will remain to be headache.
>
> There seemed to be little workarounds.
>
> 1. Guide IDEA users to take hacky workaround.
>
> Quoting https://youtrack.jetbrains.com/issue/IDEA-93855#comment=27-1838157
> :
> "A hacky workaround is to make the module in intellij with the dependency
> depend on the jar explicitly in target/. This at least allows things to
> compile and tests to run."
>
> That is really bad and annoying, but we might have no choice when we don't
> want to take other workarounds.
>
> 2. Maintaining separate project for relocated dependencies.
>
> This avoids contributors to take hacky workaround so good to go, but
> maintaining relocated artifacts might be another headache, and I'm not
sure
> ASF (or LICENSE of relocated targets) allows to do that.
>
> 3. Minimize (or remove) relocate targets and/or don't relocate troublesome
> targets.
>
> For 'storm-drpc-server', there seems to be three troublesome targets:
>
> - 'thrift'
> - 'codahale metrics'
> - 'jetty server' (We may be able to move this to 'storm-drpc-server' when
> another webapp port is done.)
>
> If we are OK to give up relocating those things we might be OK for now. We
> may want to extend the list when we break down more modules from
> 'storm-core'.
>
> Btw, IMHO relocating is not a good option. Elastic gives up shading
> anything for 2.0. (https://www.elastic.co/blog/to-shade-or-not-to-shade)
> Someone might feel that it's a regression, but we need to decide to do it
> when it can provide better shape.
>
> Please add ideas if you have any, and give your opinions about above
> options.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 28일 (화) 오후 10:23, Bobby Evans 님이 작성:
>
> Sure I am happy to help out how I can.  I really would like to spend more
> time on storm, but sadly work has shifted and my team got 2 new projects
> recently, but we have not increased the head count to cover it yet, so I
am
> swamped.  But if you do need help with some of these let me know and I'll
> see what I can do in my spare time.
>
>
> - Bobby
>
> On Tuesday, March 28, 2017, 2:10:46 AM CDT, Jungtaek Lim <
> kabh...@gmail.com> wrote:Bobby,
>
> I just tried to follow your suggestion and found it's less error-prone
> compared to my approach, and has lots of benefits. (I am seeing the great
> chance to minimize dependencies for 'storm-client', say, Worker.)
>
> Thanks for the suggestion. I'm working on this now. I'll mention you

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread Xin Wang

+1 on keeping the current external/storm-* in place and move flux, sql,
storm-submit-tools into top-level.

BTW, I am waiting for the discuss result before submitting several SQL
related PRs. :)

- Xin

2017-03-31 10:41 GMT+08:00 Jungtaek Lim :

> Yeah sure I'm OK to just apply it for master branch.
> Are you okay for moving them to root directory without renaming? Or do you
> want to rename or suggest other base directory?
>
> 2017년 3월 31일 (금) 오전 11:36, P. Taylor Goetz 님이 작성:
>
> > I'm hesitant to change the layout of the 1.x release. People do some
> crazy
> > things when it comes to operations that we can't predict. I'm okay with
> > doing this on the master/2.0 branch, but I'm hesitant on the 1.x  branch.
> >
> > -Taylor
> >
> > > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim  wrote:
> > >
> > > Once we just released Storm 1.1.0, I guess we can continue discussion
> > here.
> > >
> > > I checked the PR list, and there're no open PRs for among Flux, Storm
> > SQL,
> > > and Storm submit tool. So it's good to go.
> > >
> > > I think we have consensus to move non-connectors modules to out of
> > > external. There're also some interest about renaming "external" to
> > > "connectors", but given that "external" is chosen by community and has
> > been
> > > the norm for years, so I agree it would be better not to rename it. We
> > can
> > > do it later when there's another chance of doing it.
> > >
> > > We didn't decide where to move, but most of us seems to be OK to move
> to
> > > the top directory.
> > > Taylor, any further opinion regarding this?
> > > Suppose we're moving them to the top directory (or whatever the same
> base
> > > directory), what would be good names for them? Flux doesn't have prefix
> > > 'storm' so a bit different, but if we're OK we skip renaming it.
> > >
> > > I'll do the work when Taylor is OK for changing this.
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:
> > >
> > >> 1. apply versions
> > >>
> > >> First plan was applying this only for master, but realized that
> > >> contributors should make two patches for every PRs when we apply this
> to
> > >> only master.
> > >>
> > >> So in order to make less inconvenience, it would be better to apply
> this
> > >> for both 1.x and master. It also affects opened pull requests so we
> > would
> > >> like to check that relevant PRs are open, and apply it later than
> > reviewing
> > >> them.
> > >>
> > >> I agree with Harsha. No need to make change for current release vote.
> > >>
> > >> 2. naming issue for "external"
> > >>
> > >> "external" makes me feel that it's related to "external" component,
> say,
> > >> outside of Storm. That's why I suggest moving non-connectors to out of
> > >> "external". IMHO "connector" is still more intuitive and
> > self-describing,
> > >> but I understand that renaming the directory structure would be
> painful,
> > >> and "storm-kafka-monitor" is an example of what it's not a "connector"
> > but
> > >> an "external". So I'm OK to keep it as "external".
> > >>
> > >> 3. where to move non-connectors
> > >>
> > >> Except Flux they're directly supported by Storm. I mean "storm.py" is
> > >> aware of them and supports them, so for me they are eligible to move
> to
> > the
> > >> top directory. I'm open to other suggestion as well.
> > >>
> > >> - Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani 님이
> 작성:
> > >>
> > >> We should this in next release of 1.x or 2.0. I am +1 on continue with
> > >> current release.
> > >> -Harsha
> > >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz 
> > wrote:
> > >>>
> > >>> The question remains if we want to do this in the 1.1.0 release, or
> > >> later.
> > >>>
> > >>> If it's the 1.1.0 release we need to make the changes and cut another
> > RC.
> > >>> I'm fine with that, but want to make sure we have consensus before
> > going
> > >>> down that road.
> > >>>
> > >>> -Taylor
> > >>>
> >  On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani 
> > >>> wrote:
> > 
> >  Agree on change like this would be confusing to the users. Lets keep
> > >> the
> >  original plan of moving non-connectors modules of external instead
> of
> >  introducing new changes
> >  that are not in scope of this discussion.
> >  My +1 still stands on keeping the current external/storm-* in place
> > and
> >  move just sql and storm-perf into top-level. We can have discussion
> > for
> >  storm 2.0 if we want to do
> >  more changes.
> > 
> >  -Harsha
> > 
> > > On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz  >
> > >>> wrote:
> > >
> > > If we decide to change the structure of the distribution like
> this, I
> > > think we should do it in masrwe/2.0. If we want this for 1.1.0 we
> > need
> > >>> to
> > > cut a new release candidate.
> > >
> > > Changing the structure of the distribution file structure can be
> > > disruptive for users.

[GitHub] storm pull request #2034: STORM-2441 Break down 'storm-core' to extract clie...

2017-03-30 Thread HeartSaVioR

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/2034

STORM-2441 Break down 'storm-core' to extract client (worker) artifacts

* Issue link: http://issues.apache.org/jira/browse/STORM-2441

Sorry for the huge diff. I've `generated` classes from storm.thrift from 
new module, and git doesn't recognize (or IDEA doesn't support) moving files.
(I'll try to rework via `git mv` if we really want to minimize the 
changeset.)

Please read issue link and following discussion to see rationale of this 
issue before reviewing.

This PR breaks down 'storm-core' into multiple artifacts.

* `storm-client`: topology and worker, play as client SDK
* `storm-client-misc`: client SDK misc which is not added due to dependency 
addition
  * depends on `storm-client`
* `storm-server`: LocalCluster and related daemons (most of them, without 
webapp)
  * depends on `storm-client`
* `storm-core`: Clojure things and unported daemons (webapp) and tests, and 
commands for storm.py
  * depends on `storm-server` (hence `storm-client`, too)
  * we could get rid of this after porting work is done, or at least rename 
to other name so that end users can notice that they should use `storm-client` 
instead of `storm-core`
* `storm-webapp`: DRPC HTTP server, Logviewer and UI will be placed here
  * depends on `storm-core` (hence `storm-client` and `storm-server`, too)

After applying this patch, binary dist. will have three library 
directories, `lib` for `storm-core` (daemons), `lib-worker` for `storm-client`, 
`lib-webapp` for `storm-webapp`.

We could change lib to refer `storm-server` after porting is done, or even 
`storm-webapp` and remove `lib-webapp`.

Please note that there is a precondition to finish this work.

[ ] - Decide how to get over dependency issue (need discussion)

This patch doesn't relocate what we have been relocating. Relocation 
doesn't work when we have multiple modules, and we already have multiple 
modules for storm-core and storm-drpc-server which shows build failure from 
IntelliJ IDEA.

Unit tests and manual tests passed.

commit log


* break down 'storm-core' into multiple artifacts
  * 'storm-client': topology and worker, play as client SDK
  * 'storm-client-misc': client SDK misc which is not added due to 
dependency addition
  * 'storm-server': LocalCluster and related daemons
  * 'storm-core': Clojure things and unported daemons (webapp) and tests
  * 'storm-webapp': DRPC HTTP server, Logviewer and UI will be placed here
* Supervisor will set classpath for worker to just ensure 'storm-client', 
not 'storm-core'
* address documentation
* apply the change to all modules's pom.xml
* add new modules into Travis build
* NOTE: we don't shade and relocate what we have been relocated
  * we should discuss how to overcome dependency issue



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2034.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2034


commit a6337ebfa6ed775a36e82cee47d71d57eca9b74b
Author: Jungtaek Lim 
Date:   2017-03-28T23:08:54Z

STORM-2441 Break down 'storm-core' to extract client (worker) artifacts

* break down 'storm-core' into multiple artifacts
  * 'storm-client': topology and worker, play as client SDK
  * 'storm-client-misc': client SDK misc which is not added due to 
dependency addition
  * 'storm-server': LocalCluster and related daemons
  * 'storm-core': Clojure things and unported daemons (webapp) and tests
  * 'storm-webapp': DRPC HTTP server, Logviewer and UI will be placed here
* Supervisor will set classpath for worker to just ensure 'storm-client', 
not 'storm-core'
* addresse documentation
* apply the change to all modules's pom.xml
* add new modules into Travis build
* NOTE: we don't shade and relocate what we have been relocated
  * we should discuss how to overcome dependency issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread Jungtaek Lim

Yeah sure I'm OK to just apply it for master branch.
Are you okay for moving them to root directory without renaming? Or do you
want to rename or suggest other base directory?

2017년 3월 31일 (금) 오전 11:36, P. Taylor Goetz 님이 작성:

> I'm hesitant to change the layout of the 1.x release. People do some crazy
> things when it comes to operations that we can't predict. I'm okay with
> doing this on the master/2.0 branch, but I'm hesitant on the 1.x  branch.
>
> -Taylor
>
> > On Mar 30, 2017, at 9:01 PM, Jungtaek Lim  wrote:
> >
> > Once we just released Storm 1.1.0, I guess we can continue discussion
> here.
> >
> > I checked the PR list, and there're no open PRs for among Flux, Storm
> SQL,
> > and Storm submit tool. So it's good to go.
> >
> > I think we have consensus to move non-connectors modules to out of
> > external. There're also some interest about renaming "external" to
> > "connectors", but given that "external" is chosen by community and has
> been
> > the norm for years, so I agree it would be better not to rename it. We
> can
> > do it later when there's another chance of doing it.
> >
> > We didn't decide where to move, but most of us seems to be OK to move to
> > the top directory.
> > Taylor, any further opinion regarding this?
> > Suppose we're moving them to the top directory (or whatever the same base
> > directory), what would be good names for them? Flux doesn't have prefix
> > 'storm' so a bit different, but if we're OK we skip renaming it.
> >
> > I'll do the work when Taylor is OK for changing this.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:
> >
> >> 1. apply versions
> >>
> >> First plan was applying this only for master, but realized that
> >> contributors should make two patches for every PRs when we apply this to
> >> only master.
> >>
> >> So in order to make less inconvenience, it would be better to apply this
> >> for both 1.x and master. It also affects opened pull requests so we
> would
> >> like to check that relevant PRs are open, and apply it later than
> reviewing
> >> them.
> >>
> >> I agree with Harsha. No need to make change for current release vote.
> >>
> >> 2. naming issue for "external"
> >>
> >> "external" makes me feel that it's related to "external" component, say,
> >> outside of Storm. That's why I suggest moving non-connectors to out of
> >> "external". IMHO "connector" is still more intuitive and
> self-describing,
> >> but I understand that renaming the directory structure would be painful,
> >> and "storm-kafka-monitor" is an example of what it's not a "connector"
> but
> >> an "external". So I'm OK to keep it as "external".
> >>
> >> 3. where to move non-connectors
> >>
> >> Except Flux they're directly supported by Storm. I mean "storm.py" is
> >> aware of them and supports them, so for me they are eligible to move to
> the
> >> top directory. I'm open to other suggestion as well.
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani 님이 작성:
> >>
> >> We should this in next release of 1.x or 2.0. I am +1 on continue with
> >> current release.
> >> -Harsha
> >>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz 
> wrote:
> >>>
> >>> The question remains if we want to do this in the 1.1.0 release, or
> >> later.
> >>>
> >>> If it's the 1.1.0 release we need to make the changes and cut another
> RC.
> >>> I'm fine with that, but want to make sure we have consensus before
> going
> >>> down that road.
> >>>
> >>> -Taylor
> >>>
>  On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani 
> >>> wrote:
> 
>  Agree on change like this would be confusing to the users. Lets keep
> >> the
>  original plan of moving non-connectors modules of external instead of
>  introducing new changes
>  that are not in scope of this discussion.
>  My +1 still stands on keeping the current external/storm-* in place
> and
>  move just sql and storm-perf into top-level. We can have discussion
> for
>  storm 2.0 if we want to do
>  more changes.
> 
>  -Harsha
> 
> > On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz 
> >>> wrote:
> >
> > If we decide to change the structure of the distribution like this, I
> > think we should do it in masrwe/2.0. If we want this for 1.1.0 we
> need
> >>> to
> > cut a new release candidate.
> >
> > Changing the structure of the distribution file structure can be
> > disruptive for users. Even the change to no longer include connector
> > binaries, as we've learned, will be a headache for some users.
> >
> > IMHO, from an ops perspective, changes like this should be handled
> >> like
> > API changes.
> >
> > -Taylor
> >
> >> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <
> >>> hlo...@hortonworks.com>
> > wrote:
> >>
> >> Another possibility is to keep the ‘external' module, and create sub
> > modules under it. The legacy structure would remain i

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread P. Taylor Goetz

I'm hesitant to change the layout of the 1.x release. People do some crazy 
things when it comes to operations that we can't predict. I'm okay with doing 
this on the master/2.0 branch, but I'm hesitant on the 1.x  branch.

-Taylor

> On Mar 30, 2017, at 9:01 PM, Jungtaek Lim  wrote:
> 
> Once we just released Storm 1.1.0, I guess we can continue discussion here.
> 
> I checked the PR list, and there're no open PRs for among Flux, Storm SQL,
> and Storm submit tool. So it's good to go.
> 
> I think we have consensus to move non-connectors modules to out of
> external. There're also some interest about renaming "external" to
> "connectors", but given that "external" is chosen by community and has been
> the norm for years, so I agree it would be better not to rename it. We can
> do it later when there's another chance of doing it.
> 
> We didn't decide where to move, but most of us seems to be OK to move to
> the top directory.
> Taylor, any further opinion regarding this?
> Suppose we're moving them to the top directory (or whatever the same base
> directory), what would be good names for them? Flux doesn't have prefix
> 'storm' so a bit different, but if we're OK we skip renaming it.
> 
> I'll do the work when Taylor is OK for changing this.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:
> 
>> 1. apply versions
>> 
>> First plan was applying this only for master, but realized that
>> contributors should make two patches for every PRs when we apply this to
>> only master.
>> 
>> So in order to make less inconvenience, it would be better to apply this
>> for both 1.x and master. It also affects opened pull requests so we would
>> like to check that relevant PRs are open, and apply it later than reviewing
>> them.
>> 
>> I agree with Harsha. No need to make change for current release vote.
>> 
>> 2. naming issue for "external"
>> 
>> "external" makes me feel that it's related to "external" component, say,
>> outside of Storm. That's why I suggest moving non-connectors to out of
>> "external". IMHO "connector" is still more intuitive and self-describing,
>> but I understand that renaming the directory structure would be painful,
>> and "storm-kafka-monitor" is an example of what it's not a "connector" but
>> an "external". So I'm OK to keep it as "external".
>> 
>> 3. where to move non-connectors
>> 
>> Except Flux they're directly supported by Storm. I mean "storm.py" is
>> aware of them and supports them, so for me they are eligible to move to the
>> top directory. I'm open to other suggestion as well.
>> 
>> - Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani 님이 작성:
>> 
>> We should this in next release of 1.x or 2.0. I am +1 on continue with
>> current release.
>> -Harsha
>>> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz  wrote:
>>> 
>>> The question remains if we want to do this in the 1.1.0 release, or
>> later.
>>> 
>>> If it's the 1.1.0 release we need to make the changes and cut another RC.
>>> I'm fine with that, but want to make sure we have consensus before going
>>> down that road.
>>> 
>>> -Taylor
>>> 
 On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani 
>>> wrote:
 
 Agree on change like this would be confusing to the users. Lets keep
>> the
 original plan of moving non-connectors modules of external instead of
 introducing new changes
 that are not in scope of this discussion.
 My +1 still stands on keeping the current external/storm-* in place and
 move just sql and storm-perf into top-level. We can have discussion for
 storm 2.0 if we want to do
 more changes.
 
 -Harsha
 
> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz 
>>> wrote:
> 
> If we decide to change the structure of the distribution like this, I
> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need
>>> to
> cut a new release candidate.
> 
> Changing the structure of the distribution file structure can be
> disruptive for users. Even the change to no longer include connector
> binaries, as we've learned, will be a headache for some users.
> 
> IMHO, from an ops perspective, changes like this should be handled
>> like
> API changes.
> 
> -Taylor
> 
>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <
>>> hlo...@hortonworks.com>
> wrote:
>> 
>> Another possibility is to keep the ‘external' module, and create sub
> modules under it. The legacy structure would remain intact, while
>> making
> things more modular. An idea would be:
>> 
>> + external
>>   + connectors
>>   + tools
>>   + monitoring
>>   + etc
>> 
>> Hugo
>> 
>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz 
> wrote:
>>> 
>>> For the background on why “external” was selected, you have to go
>> back
> to a lengthy discussion in Feb. 2014.
>>> 
>>> Here’s the start of the thread:
>>>

[GitHub] storm issue #2026: Eventhub3

2017-03-30 Thread SreeramGarlapati

Github user SreeramGarlapati commented on the issue:

https://github.com/apache/storm/pull/2026
  
:shipit:


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2026: Eventhub3

2017-03-30 Thread SreeramGarlapati

Github user SreeramGarlapati commented on a diff in the pull request:

https://github.com/apache/storm/pull/2026#discussion_r109074402
  
--- Diff: 
external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/StringEventDataScheme.java
 ---
@@ -44,21 +38,32 @@
 public class StringEventDataScheme implements IEventDataScheme {
 
   private static final long serialVersionUID = 1L;
+  private static final Logger logger = 
LoggerFactory.getLogger(StringEventDataScheme.class);
 
   @Override
-  public List deserialize(Message message) {
+  public List deserialize(EventData eventData) {
 final List fieldContents = new ArrayList();
-
-for (Section section : message.getPayload()) {
-  if (section instanceof Data) {
-Data data = (Data) section;
-fieldContents.add(new String(data.getValue().getArray()));
-  } else if (section instanceof AmqpValue) {
-AmqpValue amqpValue = (AmqpValue) section;
-fieldContents.add(amqpValue.getValue().toString());
+String messageData = "";
+if (eventData.getBytes()!=null) {
+  messageData = new String(eventData.getBytes());
+}
+/*Will only serialize AMQPValue type*/
+else if (eventData.getObject()!=null) {
+  try {
+if (!(eventData.getObject() instanceof List)) {
+  messageData = eventData.getObject().toString();
+} else {
+  throw new RuntimeException("Cannot serialize the given AMQP 
type.");
--- End diff --

>serialize [](start = 45, length = 9)

cannot deserialize...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2026: Eventhub3

2017-03-30 Thread SreeramGarlapati

Github user SreeramGarlapati commented on a diff in the pull request:

https://github.com/apache/storm/pull/2026#discussion_r109074150
  
--- Diff: 
external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/SerializeDeserializeUtil.java
 ---
@@ -0,0 +1,35 @@

+/***
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ 
***/
+
+package org.apache.storm.eventhubs.spout;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+
+public class SerializeDeserializeUtil {
+public static byte[] serialize(Object obj) throws IOException { 
+try (ByteArrayOutputStream b = new ByteArrayOutputStream()) {
+try (ObjectOutputStream o = new ObjectOutputStream(b)) {
+o.writeObject(obj);
+o.close();
--- End diff --

>o.close(); [](start = 16, length = 10)

not needed...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2026: Eventhub3

2017-03-30 Thread SreeramGarlapati

Github user SreeramGarlapati commented on a diff in the pull request:

https://github.com/apache/storm/pull/2026#discussion_r109073792
  
--- Diff: 
external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/PartitionManager.java
 ---
@@ -41,29 +41,29 @@ public PartitionManager(
 
 super(spoutConfig, partitionId, stateStore, receiver);
 
-this.pending = new LinkedHashMap();
-this.toResend = new TreeSet();
+this.pending = new LinkedHashMap();
+this.toResend = new TreeSet();
--- End diff --

>() [](start = 46, length = 2)

order by sequence number


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #2026: Eventhub3

2017-03-30 Thread SreeramGarlapati

Github user SreeramGarlapati commented on a diff in the pull request:

https://github.com/apache/storm/pull/2026#discussion_r109072842
  
--- Diff: 
external/storm-eventhubs/src/main/java/org/apache/storm/eventhubs/spout/EventDataScheme.java
 ---
@@ -44,27 +41,31 @@
 public class EventDataScheme implements IEventDataScheme {
 
private static final long serialVersionUID = 1L;
-
+   private static final Logger logger = 
LoggerFactory.getLogger(EventDataScheme.class);
@Override
-   public List deserialize(Message message) {
+   public List deserialize(EventData eventData) {
final List fieldContents = new ArrayList();
-
-   Map metaDataMap = new HashMap();
String messageData = "";
-
-   for (Section section : message.getPayload()) {
-   if (section instanceof Data) {
-   Data data = (Data) section;
-   messageData = new 
String(data.getValue().getArray());
-   } else if (section instanceof AmqpValue) {
-   AmqpValue amqpValue = (AmqpValue) section;
-   messageData = amqpValue.getValue().toString();
-   } else if (section instanceof ApplicationProperties) {
-   final ApplicationProperties 
applicationProperties = (ApplicationProperties) section;
-   metaDataMap = applicationProperties.getValue();
+   if (eventData.getBytes()!=null) {
+   messageData = new String(eventData.getBytes());
+   }
+   /*Will only serialize AMQPValue type*/
+   else if (eventData.getObject()!=null) {
+   try {
+   if (!(eventData.getObject() instanceof List)) {
+   messageData = 
eventData.getObject().toString();
+   } else {
+   throw new RuntimeException("Cannot 
serialize the given AMQP type");
+   }
+   } catch (RuntimeException e) {
+   logger.error("Failed to serialize EventData 
payload class"
+   + 
eventData.getObject().getClass());
+   logger.error("Exception encountered while 
serializing EventData payload is"
+   + e.toString());
+   throw e;
}
}
-
+   Map metaDataMap = eventData.getProperties().size() > 0 ? 
eventData.getProperties() : null;
fieldContents.add(messageData);
fieldContents.add(metaDataMap);
--- End diff --

>fieldContents.add(metaDataMap); [](start = 2, length = 31)

Do you still have to add this if the properties are null ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Storm 2.0 Roadmap

2017-03-30 Thread Jungtaek Lim

Maybe related to or not, but would we want to create a new branch
"1.1.x-branch", and make "1.x-branch" target for 1.2?

I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence the
question.

- Jungtaek Lim (HeartSaVioR)

2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro 님이 작성:

> +1 for finishing the porting to Java ahead of anything else - it will be a
> significant milestone. I have a JIRA assigned concerning to the porting. I
> will work on it for the 2.0 release.
>
> it’s a priority to guarantee no performance regressions. As part of this
> endeavor, explore an automated (or easy) way to run and assert major
> performance benchmarks. Ideally any contributor should be able to fairly
> easily test the impact of changes under certain performance test scenarios.
>
> Beam Runner work should take into account the impact of incorporating new
> JStorm features and Storm Worker Redesign<
> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient to
> start doing it, to  find out that it will have to chance in face of Storm
> and worker redesign. That is, it should be done after it’s building blocks
> are stable.
>
> Thanks,
> Hugo
>
> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan  ar...@apache.org>> wrote:
>
> +1 to release with the porting completed. I think its mainly the UI server
> and log viewer that’s pending.
>
> We can start doing the regression and performance tests for whatever is
> already ported.
>
> If anyone is running the master branch in their pre-prod / prod
> environments, it will be good to know and give us more confidence.
>
> The other features can be added in follow up releases.
>
> Regards,
> Arun
>
>
> On 3/24/17, 11:47 AM, "Satish Duggana"  satish.dugg...@gmail.com>> wrote:
>
> +1 to have 2.0 with porting and performance(it should be at least as good
> as 1.x release) issues addressed
>
> We can target other tasks(mentioned by Taylor and Jungtaek) for 2.x-branch.
>
>
> Exactly-once support:
> While thinking through the exactlyonce support design, it is realized
> better to avoid acking tuples and implement exactly once by snapshotting
> barriers. It seems JStorm folks followed similar design, they claim it
> gives better performance. This feature is essential for beam runner and we
> can decide on respective approaches though.
>
> Beam Runner
> Lets hold on this for now and keep it in Storm till 2.x. We should avoid
> having a minimal beam runner in haste. It is better to address STORM-2284,
> exactly-once and other windowing enhancements to enable beam runner.
>
> JStorm
> Agree with Jungtaek on looking at the latest JStorm and align/scope with
> the features for 2.x.
>
> STORM-2284
> We may want to look at JStorm worker before working on respective
> components in this epic to pull appropriate enhancements.
>
> YARN/MESOS
> Supporting Storm on YARN/Mesos for 2.x.
>
> Thanks,
> Satish.
>
>
> On Fri, Mar 24, 2017 at 9:09 AM, Jungtaek Lim  kabh...@gmail.com>> wrote:
>
> First of all, +1 to complete only port work and do sanity check (including
> performance regression), and release.
>
> If we can get STORM-2284 within deterministic time frame (say 2~3 months)
> that should be great, but if not I'd in favor of postponing that to later
> 2.x release.
>
> JStorm released their new versions after code donation. So there're more
> things we could get ideas from, or even adopt from.
> https://github.com/alibaba/jstorm/blob/master/history.md
> As you noticed from release note link, we also need to update phase 2 since
> they already changed what we're planning to do in phase 2. For example,
> they changed backpressure to end-to-end, and changed to use snapshot rather
> than acker.
> May be sure, JStorm pulled many features from today's Storm, like Flux,
> Windowing, more shuffle groupings, log search, log level change, and so on.
>
> STORM-2426  is due to
> the
> limitation of Spout lifecycle (all the things are done in single thread),
> and STORM-1358 (JStorm's
> multi-thread Spout) can remedy this (despite that Spout implementation may
> need to guarantee thread-safety later). It's not a just improvement but
> close to design concern so would like to address sooner than other things
> in phase 2.
>
> For Storm SQL side, I've lost progress but major work would be adopting
> group by with windowing. It was not available from Calcite but will be
> available at next release (1.12.0).
> I've filed this to STORM-2405
> , but windowing & micro
> batch is not intuitive, so I would like to change the underlying API to
> stream API in SQL. Also filed this to STORM-2406
> .
>
> Just 2 cents btw, hopefully I would like to see metrics V2 sooner since we
> lost metrics even when doing normal operation like restarting worker,
> rebalancing, and so on. Eventually we need to fight with dyn

Re: [DISCUSS] Move non-connectors modules to out of external

2017-03-30 Thread Jungtaek Lim

Once we just released Storm 1.1.0, I guess we can continue discussion here.

I checked the PR list, and there're no open PRs for among Flux, Storm SQL,
and Storm submit tool. So it's good to go.

I think we have consensus to move non-connectors modules to out of
external. There're also some interest about renaming "external" to
"connectors", but given that "external" is chosen by community and has been
the norm for years, so I agree it would be better not to rename it. We can
do it later when there's another chance of doing it.

We didn't decide where to move, but most of us seems to be OK to move to
the top directory.
Taylor, any further opinion regarding this?
Suppose we're moving them to the top directory (or whatever the same base
directory), what would be good names for them? Flux doesn't have prefix
'storm' so a bit different, but if we're OK we skip renaming it.

I'll do the work when Taylor is OK for changing this.

- Jungtaek Lim (HeartSaVioR)

2017년 3월 27일 (월) 오전 11:16, Jungtaek Lim 님이 작성:

> 1. apply versions
>
> First plan was applying this only for master, but realized that
> contributors should make two patches for every PRs when we apply this to
> only master.
>
> So in order to make less inconvenience, it would be better to apply this
> for both 1.x and master. It also affects opened pull requests so we would
> like to check that relevant PRs are open, and apply it later than reviewing
> them.
>
> I agree with Harsha. No need to make change for current release vote.
>
> 2. naming issue for "external"
>
> "external" makes me feel that it's related to "external" component, say,
> outside of Storm. That's why I suggest moving non-connectors to out of
> "external". IMHO "connector" is still more intuitive and self-describing,
> but I understand that renaming the directory structure would be painful,
> and "storm-kafka-monitor" is an example of what it's not a "connector" but
> an "external". So I'm OK to keep it as "external".
>
> 3. where to move non-connectors
>
> Except Flux they're directly supported by Storm. I mean "storm.py" is
> aware of them and supports them, so for me they are eligible to move to the
> top directory. I'm open to other suggestion as well.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 26일 (일) 오후 12:14, Harsha Chintalapani 님이 작성:
>
> We should this in next release of 1.x or 2.0. I am +1 on continue with
> current release.
> -Harsha
> On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz  wrote:
>
> > The question remains if we want to do this in the 1.1.0 release, or
> later.
> >
> > If it's the 1.1.0 release we need to make the changes and cut another RC.
> > I'm fine with that, but want to make sure we have consensus before going
> > down that road.
> >
> > -Taylor
> >
> > > On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani 
> > wrote:
> > >
> > > Agree on change like this would be confusing to the users. Lets keep
> the
> > > original plan of moving non-connectors modules of external instead of
> > > introducing new changes
> > > that are not in scope of this discussion.
> > > My +1 still stands on keeping the current external/storm-* in place and
> > > move just sql and storm-perf into top-level. We can have discussion for
> > > storm 2.0 if we want to do
> > > more changes.
> > >
> > > -Harsha
> > >
> > >> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz 
> > wrote:
> > >>
> > >> If we decide to change the structure of the distribution like this, I
> > >> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need
> > to
> > >> cut a new release candidate.
> > >>
> > >> Changing the structure of the distribution file structure can be
> > >> disruptive for users. Even the change to no longer include connector
> > >> binaries, as we've learned, will be a headache for some users.
> > >>
> > >> IMHO, from an ops perspective, changes like this should be handled
> like
> > >> API changes.
> > >>
> > >> -Taylor
> > >>
> > >>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <
> > hlo...@hortonworks.com>
> > >> wrote:
> > >>>
> > >>> Another possibility is to keep the ‘external' module, and create sub
> > >> modules under it. The legacy structure would remain intact, while
> making
> > >> things more modular. An idea would be:
> > >>>
> > >>> + external
> > >>>+ connectors
> > >>>+ tools
> > >>>+ monitoring
> > >>>+ etc
> > >>>
> > >>> Hugo
> > >>>
> >  On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz 
> > >> wrote:
> > 
> >  For the background on why “external” was selected, you have to go
> back
> > >> to a lengthy discussion in Feb. 2014.
> > 
> >  Here’s the start of the thread:
> > 
> > 
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e
> > >> <
> > >>
> >
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E
> > >>>
> > 
> >  It continues into March:
> > 
> > 
> > >>

[ANNOUNCE] Apache Storm 1.1.0 Released

2017-03-30 Thread P. Taylor Goetz

The Apache Storm community is pleased to announce the release of Apache Storm 
version 1.1.0.

Storm is a distributed, fault-tolerant, and high-performance realtime 
computation system that provides strong guarantees on the processing of data. 
You can read more about Storm on the project website:

http://storm.apache.org

Downloads of source and binary distributions are listed in our download
section:

http://storm.apache.org/downloads.html

You can read more about this release in the following blog post:

http://storm.apache.org/2017/03/29/storm110-released.html

Distribution artifacts are available in Maven Central at the following 
coordinates:

groupId: org.apache.storm
artifactId: storm-core
version: 1.1.0

The full list of changes is available here[1]. Please let us know [2] if you 
encounter any problems.

Regards,

The Apache Storm Team

[1]: https://github.com/apache/storm/blob/v1.1.0/CHANGELOG.md
[2]: https://issues.apache.org/jira/browse/STORM

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

2017-03-30 Thread ppoulosk

Github user ppoulosk commented on a diff in the pull request:

https://github.com/apache/storm/pull/1785#discussion_r108951566
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/utils/ArtifactoryConfigLoader.java
 ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.scheduler.utils;
+
+import org.apache.storm.utils.Time;
+import org.apache.storm.Config;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Map;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.ParseException;
+import org.json.simple.parser.JSONParser;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.config.RequestConfig; 
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.HttpClientBuilder; 
+import org.apache.http.client.HttpClient; 
+import org.apache.http.util.EntityUtils;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * ArtifactoryConfigLoader.java - A dynamic loader for that can load
+ *scheduler configurations for user
+ *resource guarantees from Artifactory.
+ *
+ * 
+ * Configuration items for this config loader are passed in via config 
settings in 
+ * each scheduler that has a configurable loader.
+ *
+ * 
+ * For example, the resource aware scheduler has configuration items 
defined in Config.java
+ * that allow a user to configure which implementation of IConfigLoader to 
use to load
+ * specific scheduler configs as well as any parameters to pass into the 
prepare method of
+ * that configuration.
+ *
+ * 
+ * resource.aware.scheduler.user.pools.loader can be set to 
org.apache.storm.scheduler.utils.ArtifactoryConfigLoader
+ *
+ * 
+ * and then
+ *
+ * 
+ * resource.aware.scheduler.user.pools.loader.params can be set to any of 
the following
+ *
+ * 
+ * 
+ * {"artifactory.config.loader.uri": 
"http://artifactory.example.org:9989/artifactory/confs/my_cluster/mt_user_pool"}
+ *
+ * {"artifactory.config.loader.uri": 
"file:///confs/my_cluster/mt_user_pool"}
+ *
+ * {"artifactory.config.loader.uri": 
"file:///confs/my_cluster/mt_user_pool", 
"artifactory.config.loader.timeout.secs" : "60"}
+ * 
+ * 
+ */
+public class ArtifactoryConfigLoader implements IConfigLoader {
+protected static final String ARTIFACTORY_URI = 
"artifactory.config.loader.uri";
+protected static final String 
ARTIFACTORY_TIMEOUT_SECS="artifactory.config.loader.timeout.secs";
+protected static final String 
ARTIFACTORY_POLL_TIME_SECS="artifactory.config.loader.polltime.secs";
+protected static final String 
ARTIFACTORY_SCHEME="artifactory.config.loader.scheme";
+protected static final String 
ARTIFACTORY_BASE_DIRECTORY="artifactory.config.loader.base.directory";
+protected static final String LOCAL_ARTIFACT_DIR="scheduler_artifacts";
+static final String cacheFilename = "latest.yaml";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(ArtifactoryConfigLoader.class);
+
+@SuppressWarnings("rawtypes")
+private Map _conf;
+private int _artifactoryPollTimeSecs = 600;
+private boolean _cacheInitialized = false;
+// Location of the file in the artifactory archive.  Also used to name 
file in cache.
+private String _localCacheDir;
+private String _artifactoryScheme = "http";
+private St

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

2017-03-30 Thread ppoulosk

Github user ppoulosk commented on a diff in the pull request:

https://github.com/apache/storm/pull/1785#discussion_r108951633
  
--- Diff: 
storm-core/src/jvm/org/apache/storm/scheduler/utils/ArtifactoryConfigLoader.java
 ---
@@ -0,0 +1,429 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.scheduler.utils;
+
+import org.apache.storm.utils.Time;
+import org.apache.storm.Config;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Map;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.ParseException;
+import org.json.simple.parser.JSONParser;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.ClientProtocolException;
+import org.apache.http.client.ResponseHandler;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.config.RequestConfig; 
+import org.apache.http.client.utils.URIBuilder;
+import org.apache.http.impl.client.HttpClientBuilder; 
+import org.apache.http.client.HttpClient; 
+import org.apache.http.util.EntityUtils;
+import org.yaml.snakeyaml.Yaml;
+import org.yaml.snakeyaml.constructor.SafeConstructor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * ArtifactoryConfigLoader.java - A dynamic loader for that can load
+ *scheduler configurations for user
+ *resource guarantees from Artifactory.
+ *
+ * 
+ * Configuration items for this config loader are passed in via config 
settings in 
+ * each scheduler that has a configurable loader.
+ *
+ * 
+ * For example, the resource aware scheduler has configuration items 
defined in Config.java
+ * that allow a user to configure which implementation of IConfigLoader to 
use to load
+ * specific scheduler configs as well as any parameters to pass into the 
prepare method of
+ * that configuration.
+ *
+ * 
+ * resource.aware.scheduler.user.pools.loader can be set to 
org.apache.storm.scheduler.utils.ArtifactoryConfigLoader
+ *
+ * 
+ * and then
+ *
+ * 
+ * resource.aware.scheduler.user.pools.loader.params can be set to any of 
the following
+ *
+ * 
+ * 
+ * {"artifactory.config.loader.uri": 
"http://artifactory.example.org:9989/artifactory/confs/my_cluster/mt_user_pool"}
+ *
+ * {"artifactory.config.loader.uri": 
"file:///confs/my_cluster/mt_user_pool"}
+ *
+ * {"artifactory.config.loader.uri": 
"file:///confs/my_cluster/mt_user_pool", 
"artifactory.config.loader.timeout.secs" : "60"}
+ * 
+ * 
+ */
+public class ArtifactoryConfigLoader implements IConfigLoader {
+protected static final String ARTIFACTORY_URI = 
"artifactory.config.loader.uri";
+protected static final String 
ARTIFACTORY_TIMEOUT_SECS="artifactory.config.loader.timeout.secs";
+protected static final String 
ARTIFACTORY_POLL_TIME_SECS="artifactory.config.loader.polltime.secs";
+protected static final String 
ARTIFACTORY_SCHEME="artifactory.config.loader.scheme";
+protected static final String 
ARTIFACTORY_BASE_DIRECTORY="artifactory.config.loader.base.directory";
+protected static final String LOCAL_ARTIFACT_DIR="scheduler_artifacts";
+static final String cacheFilename = "latest.yaml";
+
+private static final Logger LOG = 
LoggerFactory.getLogger(ArtifactoryConfigLoader.class);
+
+@SuppressWarnings("rawtypes")
+private Map _conf;
+private int _artifactoryPollTimeSecs = 600;
+private boolean _cacheInitialized = false;
+// Location of the file in the artifactory archive.  Also used to name 
file in cache.
+private String _localCacheDir;
+private String _artifactoryScheme = "http";
+private St

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

2017-03-30 Thread ppoulosk

Github user ppoulosk commented on a diff in the pull request:

https://github.com/apache/storm/pull/1785#discussion_r108951217
  
--- Diff: docs/IConfigLoader.md ---
@@ -0,0 +1,58 @@
+---
+title: IConfigLoader
+layout: documentation
+documentation: true
+---
+
+
+### Introduction
+IConfigLoader is an interface designed to allow way to dynamically load 
scheduler resource constraints into scheduler implementations. Currently, the 
MultiTenant scheduler uses this interface to dynamically load the number of 
isolated nodes a given user has been guaranteed, and the ResoureAwareScheduler 
uses the interface to dynamically load per user resource guarantees.
--- End diff --

Much better wording.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

2017-03-30 Thread ppoulosk

Github user ppoulosk commented on a diff in the pull request:

https://github.com/apache/storm/pull/1785#discussion_r108951166
  
--- Diff: pom.xml ---
@@ -220,6 +220,7 @@
 0.3.1
 1.1.9
 0.3.6
+1.2
--- End diff --

Fixing this.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Is there any document explaining Storm Built-in Metrics

2017-03-30 Thread Zhechao Ma

Hi

I'm working on tuning my topology by adding metrics in it, but I can't find
document that explain the meanings of the built-in metrics. There're some
metrics I'm not much clear about, like arrival_rate_secs,
__skipped-throttle and so on.

-- 
Thanks
Zhechao Ma

Re: [DISCUSS] Move non-connectors modules to out of external

[GitHub] storm issue #2034: STORM-2441 Break down 'storm-core' to extract client (wor...

Re: [DISCUSS] Move non-connectors modules to out of external

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

Re: [DISCUSS] Move non-connectors modules to out of external

[GitHub] storm pull request #2034: STORM-2441 Break down 'storm-core' to extract clie...

Re: [DISCUSS] Move non-connectors modules to out of external

Re: [DISCUSS] Move non-connectors modules to out of external

[GitHub] storm issue #2026: Eventhub3

[GitHub] storm pull request #2026: Eventhub3

[GitHub] storm pull request #2026: Eventhub3

[GitHub] storm pull request #2026: Eventhub3

[GitHub] storm pull request #2026: Eventhub3

Re: [DISCUSS] Storm 2.0 Roadmap

Re: [DISCUSS] Move non-connectors modules to out of external

[ANNOUNCE] Apache Storm 1.1.0 Released

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

[GitHub] storm pull request #1785: [STORM-2201] Add dynamic scheduler configuration l...

Is there any document explaining Storm Built-in Metrics

21 matches

Site Navigation

Mail list logo

Footer information