[GitHub] storm issue #1682: STORM-2089 Replace Consumer of ISqlTridentDataSource with...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1682
  
@arunmahadevan Thanks for the review. I addressed your comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1682: STORM-2089 Replace Consumer of ISqlTridentDataSour...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/1682#discussion_r80415924
  
--- Diff: 
external/sql/storm-sql-runtime/src/jvm/org/apache/storm/sql/runtime/ISqlTridentDataSource.java
 ---
@@ -17,14 +17,62 @@
  */
 package org.apache.storm.sql.runtime;
 
-import org.apache.storm.trident.operation.Function;
-import org.apache.storm.trident.spout.IBatchSpout;
 import org.apache.storm.trident.spout.ITridentDataSource;
+import org.apache.storm.trident.state.StateFactory;
+import org.apache.storm.trident.state.StateUpdater;
 
 /**
  * A ISqlTridentDataSource specifies how an external data source produces 
and consumes data.
  */
 public interface ISqlTridentDataSource {
+  /**
+   * SqlTridentConsumer is a data structure containing StateFactory and 
StateUpdater for consuming tuples with State.
+   *
+   * Please note that StateFactory and StateUpdater should use same class 
which implements State.
+   *
+   * @see org.apache.storm.trident.state.StateFactory
+   * @see org.apache.storm.trident.state.StateUpdater
+   */
+  class SqlTridentConsumer {
--- End diff --

Good point. Will address.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1682: STORM-2089 Replace Consumer of ISqlTridentDataSour...

2016-09-25 Thread arunmahadevan 
Github user arunmahadevan commented on a diff in the pull request:

https://github.com/apache/storm/pull/1682#discussion_r80413517
  
--- Diff: 
external/sql/storm-sql-runtime/src/jvm/org/apache/storm/sql/runtime/ISqlTridentDataSource.java
 ---
@@ -17,14 +17,62 @@
  */
 package org.apache.storm.sql.runtime;
 
-import org.apache.storm.trident.operation.Function;
-import org.apache.storm.trident.spout.IBatchSpout;
 import org.apache.storm.trident.spout.ITridentDataSource;
+import org.apache.storm.trident.state.StateFactory;
+import org.apache.storm.trident.state.StateUpdater;
 
 /**
  * A ISqlTridentDataSource specifies how an external data source produces 
and consumes data.
  */
 public interface ISqlTridentDataSource {
+  /**
+   * SqlTridentConsumer is a data structure containing StateFactory and 
StateUpdater for consuming tuples with State.
+   *
+   * Please note that StateFactory and StateUpdater should use same class 
which implements State.
+   *
+   * @see org.apache.storm.trident.state.StateFactory
+   * @see org.apache.storm.trident.state.StateUpdater
+   */
+  class SqlTridentConsumer {
--- End diff --

Its better to have this as an interface with getStateFactory and 
getStateUpdater.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1715: Move system test (VM based integration test) to an...

2016-09-25 Thread HeartSaVioR 
GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1715

Move system test (VM based integration test) to another build matrix

* this test adds more and 20 mins in each build and sometimes makes build 
failing due to long running
* making it to separate build helps reducing build time

I and Bobby saw some builds failing due to long running. This fix should 
help that situation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm 
move-system-test-to-another-build-matrix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1715


commit 92df7044846e6f296ecab6282503dd0fac6875e2
Author: Jungtaek Lim 
Date:   2016-09-26T01:57:36Z

Move system test (VM based integration test) to another build matrix

* this test adds more and 20 mins in each build and sometimes makes build 
failing due to long running
* making it to separate build helps reducing build time




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1714: STORM-2125 Use Calcite's implementation of Rex Compiler

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1714
  
There's an another issue: when to initialize DataContext.

Calcite utilizes the variables in DataContext. and there're datetime 
related variables in DataContext.

The thing is, SQL standard says that functions such as CURRENT_TIMESTAMP 
return the same value throughout the query. Since we're querying in stream, 
when to initialize DataContext defines the boundary to show same datetime 
across rows. With joining stream, the issue is more complicated.

For now I just initialize DataContext at the query compilation phase and 
pass it to components, so that the value of current datetime is static (don't 
change) across workers in topology. If we would like to make it dynamic, we 
need to discuss.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1709: STORM-2116 [Storm SQL] Support 'CASE' statement

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR closed the pull request at:

https://github.com/apache/storm/pull/1709


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1704: STORM-2111 [Storm SQL] support 'LIKE' and 'SIMILAR...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR closed the pull request at:

https://github.com/apache/storm/pull/1704


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1707: STORM-2113 [Storm SQL] fix 'OR' and 'AND' operators handl...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1707
  
Closing this since STORM-2125 (#1714) covers this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1709: STORM-2116 [Storm SQL] Support 'CASE' statement

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1709
  
Closing this since STORM-2125 (#1714) covers this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1707: STORM-2113 [Storm SQL] fix 'OR' and 'AND' operator...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR closed the pull request at:

https://github.com/apache/storm/pull/1707


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1704: STORM-2111 [Storm SQL] support 'LIKE' and 'SIMILAR TO'

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1704
  
Closing this since STORM-2125 (#1714) covers this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1714: STORM-2125 Use Calcite's implementation of Rex Com...

2016-09-25 Thread HeartSaVioR 
GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1714

STORM-2125 Use Calcite's implementation of Rex Compiler

Note: This patch is on top of STORM-2089.

This can cover STORM-2111(#1704), STORM-2113(#1707), STORM-2114, 
STORM-2116(#1709).

* Upgrade Calcite to latest 1.9.0
* borrow JaninoRexCompiler and modify to return Expression instead of 
Scalar (RexNodeToBlockStatementCompiler)
  * since we need to pass code block to Bolts, not executable method 
implementation on classloader
* modify codebase to use RexNodeToBlockStatementCompiler
* add some feature validation unit tests
* remove ExprCompiler and its test

Btw, Calcite Rex Compiler assigns the result of ITEM() to Object, so 
comparing result to other constant just fails. CAST() works but if any of the 
result of ITEM() is null or out of bound (array), it throws Exception at 
runtime. So only using them to select field would be safe.

I sent a mail to Calcite dev to ask this strange behavior, but I think it's 
not a big matter so even though it can't be fixed, I would like to document it 
as limitation and drop that feature.

@arunmahadevan Please review and comment, and also please comment what do 
you think about above behavior. Thanks in advance.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2125

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1714


commit 4b69c077a1f477e06049aa97bd127fc1b2786c21
Author: Jungtaek Lim 
Date:   2016-09-13T02:40:13Z

STORM-2089 Replace Consumer of ISqlTridentDataSource with SqlTridentConsumer

* SqlTridentConsumer contains StateFactory and StateUpdater which is needed 
to store tuples to State via batch
* Apply the change to storm-sql-kafka
* move out JsonScheme and JsonSerializer to runtime
  * it will be used from other external sql modules
* add javadoc to ISqlTridentDataSource

commit 2fa4c95ff1ecb3cab9c09c25ef08a4faa21905e2
Author: Jungtaek Lim 
Date:   2016-09-22T06:38:44Z

STORM-2125 Use Calcite's implementation of Rex Compiler

* Upgrade Calcite to latest 1.9.0
* borrow JaninoRexCompiler and modify to return Expression instead of 
Scalar (RexNodeToBlockStatementCompiler)
  * since we need to pass code block to Bolts, not executable method 
implementation on classloader
* modify codebase to use RexNodeToBlockStatementCompiler
* add some feature validation unit tests
* remove ExprCompiler and its test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Strange behavior on MAP and ARRAY type

2016-09-25 Thread Jungtaek Lim 
Sorry wrong recipients. It should be sent to Calcite dev. mailing list.

2016년 9월 24일 (토) 오후 3:51, Jungtaek Lim 님이 작성:

> Hi devs,
>
> I'm experimenting with ARRAY and MAP type with Calcite's JaninoRexCompiler.
> (with Calcite 1.9.0)
>
> While testing I found some behaviors, so would like to see they're
> intentional
> or bugs we want to resolve.
>
> 1. Even though I set map's key/value type or array's type explicitly,
> compiler ignores the value type and create assignment to Object.
>
> This makes extracting value from nested collection not possible. Also
> comparing extracted value with numeric constant, too.
> (For example, suppose there's a row which MAPFIELD['a'] has 1 as value.
> Writing MAPFIELD['a'] = 1 on where statement throwing an Exception saying
> there's no SqlFunctions.eq(Object, int).
>
> Btw, I took a breakpoint on BinaryImplementor.implement() to see variables
> just
> before compiler finds SqlFunctions.eq().
> The type of expression is Object which is not generic, but seems like
> matching
> RexCall operand have value type information. We might create a quick fix
> from
> here (handling type conversion) but I'm not sure this can clearly resolve
> this.
>
> 2. I just replace their type to ANY, and wrap expression with CAST like
> MAPFIELD['a'] -> CAST(MAPFIELD['a'] AS INTEGER) = 1 and succeed.
> (While using MAP type even I wrap to CAST, above error happens.)
>
> But it doesn't work when MAPFIELD['a'] is null because CAST AS INTEGER
> calls
> SqlFunctions.toInt(null) which throws RuntimeException.
> (It happens MAPFIELD['a'] doesn't have item which has 'a' as key. I guess
> same
> applies to null value. And also same applies to ARRAY, and
> ARRAYFIELD[].)
>
> CAST(COALESCE(MAPFIELD['a'], -1) AS INTEGER) also throws Exception since
> MAPFIELD['a'] is Object and -1 is int.
> (CompileException : Incompatible expression types "java.lang.Object" and
> "int")
>
> 3. ArrayOutOfIndexException is thrown when query is trying to access the
> array
> with out of index. I don't know there's a standard rule on this, but at
> least
> PostgreSQL doesn't throw an Exception but just treats it as null.
> https://www.postgresql.org/docs/9.1/static/arrays.html
> (This means the result of ITEM operator is nullable.)
>
> Any ideas on these behaviors?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

[GitHub] storm issue #1697: STORM-2018: Supervisor V2

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1697
  
@revans2 Relevant PRs (#1699 #1700 #1705 #1712) are all merged to master. 
Please pull them here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1604: [STORM-2013] Upgrade Netty to 3.10.6

2016-09-25 Thread darionyaphet 
Github user darionyaphet commented on the issue:

https://github.com/apache/storm/pull/1604
  
@HeartSaVioR  OK maybe move to netty 4 is better thanks :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1604: [STORM-2013] Upgrade Netty to 3.10.6

2016-09-25 Thread darionyaphet 
Github user darionyaphet closed the pull request at:

https://github.com/apache/storm/pull/1604


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1699: STORM-2109: Treat Supervisor CPU/MEMORY Configs as...

2016-09-25 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1699


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1699: STORM-2109: Treat Supervisor CPU/MEMORY Configs as Number...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1699
  
+1 again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1706: [STORM-2118] A few fixes for storm-sql standalone ...

2016-09-25 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1706


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1708: [STORM-2119] - bug in log message printing to stdo...

2016-09-25 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1708


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1712: STORM-2122: Cache dependency data, and serialize r...

2016-09-25 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1712


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1712: STORM-2122: Cache dependency data, and serialize reading ...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1712
  
I've tested with --jars and --artifacts options. Works like a charm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1712: STORM-2122: Cache dependency data, and serialize reading ...

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1712
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1698: Storm 2105 cluster supervisor total and avail reso...

2016-09-25 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1698


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1698: Storm 2105 cluster supervisor total and avail resources

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1698
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1713: Storm 2124 show requested cpu mem for each component

2016-09-25 Thread HeartSaVioR 
Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1713
  
Travis CI terminated because of exceeding the maximum time limit for jobs. 
https://travis-ci.org/apache/storm/jobs/162370376
We need to find alternative way to run Integration test.
@raghavgautam I'll disable the integration test now. Please find the way to 
run integration test in parallel.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1679: STORM-2087: storm-kafka-client - tuples not always being ...

2016-09-25 Thread srdo 
Github user srdo commented on the issue:

https://github.com/apache/storm/pull/1679
  
Thought about it a little more. The spout should probably just check when 
it polls messages that the messages are not behind the committedOffset. If they 
are, the committedOffset needs to move to the left.

It's a little irritating that the consumer doesn't expose an API to let the 
client know that an unclean leader election happened. It would be nice if for 
example ConsumerRebalanceListener had a method to tell the client to roll back 
the committed offset to the new LEO after an unclean leader was elected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1679: STORM-2087: storm-kafka-client - tuples not always being ...

2016-09-25 Thread srdo 
Github user srdo commented on the issue:

https://github.com/apache/storm/pull/1679
  
@jfenc91 Hope it works for you now. You're right though that the spout 
needs to handle unclean leader elections better. It's not great that the 
consumer group will remain "ahead" of the LEO and the spout will fail to commit 
offsets until it has caught up to where the unclean leader election happened. I 
think an option for a solution would be to have the spout check if the current 
LEO is behind committedOffset when it hits the else case in 
findNextCommitOffsets, and if so, it should reset the consumer group's 
committed offset to the either the LEO or the beginning of the partition, 
probably depending on a new configuration parameter. It's never going to be a 
nice situation to be in, but at least the spout shouldn't break when it 
happens, especially since it can happen with the default Kafka configuration.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---