[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/104#discussion_r85693032
--- Diff: project/BuildExample.scala ---
@@ -57,6 +57,19 @@ object BuildExample extends sbt.Build {
)
) dependsOn(streaming % "test->test; provided")
+ lazy val hbase = Project(
--- End diff --
`hbase-example` would be better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/104#discussion_r85692924
--- Diff:
examples/streaming/hbase/src/main/scala/org/apache/gearpump/streaming/hbase/InsertToHBase.scala
---
@@ -0,0 +1,51 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this fileenv
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package org.apache.gearpump.streaming.hbase
+
+import java.time.Instant
+
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.external.hbase.HBaseSink
+import org.apache.gearpump.streaming.source.DataSource
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.hadoop.hbase.util.Bytes
+
+class InsertToHBase extends DataSource {
+
+ private var x = 1
+ lazy val hBaseSink = HBaseSink(UserConfig.empty, "sss")
+
+ override def open(context: TaskContext, startTime: Instant): Unit = {}
+
+ override def read(): Message = {
+hBaseSink.insert(Bytes.toBytes(s"row$x"), Bytes.toBytes("group"),
--- End diff --
It's strange that the DataSource is acting like a DataSink.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/104#discussion_r85692981
--- Diff: project/BuildExample.scala ---
@@ -57,6 +57,19 @@ object BuildExample extends sbt.Build {
)
) dependsOn(streaming % "test->test; provided", daemon % "test->test;
provided")
+ lazy val hbase = Project(
--- End diff --
`hbase-example` would be better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Assigned] (GEARPUMP-103) Support finite stream, add stream-completion capability
[ https://issues.apache.org/jira/browse/GEARPUMP-103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang reassigned GEARPUMP-103: --- Assignee: Manu Zhang > Support finite stream, add stream-completion capability > --- > > Key: GEARPUMP-103 > URL: https://issues.apache.org/jira/browse/GEARPUMP-103 > Project: Apache Gearpump > Issue Type: New Feature > Components: streaming >Reporter: Manu Zhang >Assignee: Manu Zhang >Priority: Minor > > imported from [https://github.com/gearpump/gearpump/issues/1230] on behalf of > [~clockfly], > Whether we need to support finite stream that we allow a stream to complete? > It may be useful if we want to handle some short living tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Consider highlighting your podling at ApacheCON in Seville
Hi! with ApacheCON in Seville right around the corner chances are you've made your travel arrangements already. If you happen to be around the conference on Fri, November 18th, 16:30 - 17:20 why not use that time to highlight your favorite podling? https://s.apache.org/aceu2016_shark_tank Podling Shark Tank is back and it is a perfect opportunity to give a short, 5 min pitch in a fun environment. You don't have to be a committer on the project, or even a developer! All you have to have is a bit of free time and a lot of passion to let the world know how awesome this project is! If you feel interested -- please either sign up on the wiki page (it also provides more details on the event) https://wiki.apache.org/apachecon/Seville16PodlingSharkTank or just reply to this thread. Either way -- hope to see you at Podling Shark Tank in Seville! Thanks, Roman.
[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/104#discussion_r85692859
--- Diff:
examples/streaming/hbase/src/main/scala/org/apache/gearpump/streaming/hbase/HBaseConn.scala
---
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gearpump.streaming.hbase
+
+
+import akka.actor.ActorSystem
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.cluster.client.ClientContext
+import org.apache.gearpump.cluster.main.{ArgumentsParser, CLIOption,
ParseResult}
+import org.apache.gearpump.streaming.StreamApplication
+import org.apache.gearpump.streaming.source.DataSourceProcessor
+import org.apache.gearpump.util.Graph.Node
+import org.apache.gearpump.util.{AkkaApp, Graph, LogUtil}
+import org.slf4j.Logger
+
+object HBaseConn extends AkkaApp with ArgumentsParser {
+ private val LOG: Logger = LogUtil.getLogger(getClass)
+ val RUN_FOR_EVER = -1
+
+ override val options: Array[(String, CLIOption[Any])] = Array(
+"splitNum" -> CLIOption[Int]("", required = false,
defaultValue = Some(1))
+ )
+
+ def application(config: ParseResult, system: ActorSystem):
StreamApplication = {
+implicit val actorSystem = system
+
+val splitNum = config.getInt("splitNum")
+
+val split = new InsertToHBase
+val sourceProcessor = DataSourceProcessor(split, splitNum, "Split")
--- End diff --
It's weird. The `InsertToHBase` is actually a hbase sink that writes data
to HBase, not a DataSource.
Why not create a `Split` class that will generate messages and then pass
them to a `HBaseSink`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump issue #102: [GEARPUMP-228] Message's timestamp should be ...
Github user darionyaphet commented on the issue: https://github.com/apache/incubator-gearpump/pull/102 thanks @huafengw --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump issue #104: Add InsertToHBase example
Github user codecov-io commented on the issue: https://github.com/apache/incubator-gearpump/pull/104 ## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/104?src=pr) is 71.42% (diff: 100%) > Merging [#104](https://codecov.io/gh/apache/incubator-gearpump/pull/104?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will decrease coverage by **0.23%** ```diff @@ master #104 diff @@ == Files 186186 Lines 5985 5985 Methods5456 5456 Messages 0 0 Branches529529 == - Hits 4289 4275-14 - Misses 1696 1710+14 Partials 0 0 ```  > Powered by [Codecov](https://codecov.io?src=pr). Last update [7bc5fba...68d9027](https://codecov.io/gh/apache/incubator-gearpump/compare/7bc5fba5b5c5166992e04218071a3da4931e47a9...68d9027656fd7470c6389cb85ee63181fe64d0f5?src=pr) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #102: [GEARPUMP-228] Message's timestamp sho...
Github user darionyaphet closed the pull request at: https://github.com/apache/incubator-gearpump/pull/102 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/104#discussion_r85692424
--- Diff: conf/gear.conf ---
@@ -243,7 +243,7 @@ gearpump {
###
executor {
vmargs = "-server -Xms512M -Xmx1024M -Xss1M
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
-Djava.rmi.server.hostname=localhost"
-extraClasspath = ""
+extraClasspath =
"/home/manuzhang/hbase-1.2.2/lib/*:/home/manuzhang/hbase-1.2.2/conf"
--- End diff --
irrelevant change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-gearpump pull request #104: Add InsertToHBase example
GitHub user Roshanson opened a pull request: https://github.com/apache/incubator-gearpump/pull/104 Add InsertToHBase example You can merge this pull request into a Git repository by running: $ git pull https://github.com/Roshanson/incubator-gearpump InHBase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-gearpump/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit e70cb18ec8dff8e59448853e32cb9acbf1696c81 Author: Roshanson <736781...@qq.com> Date: 2016-08-26T07:08:07Z refactor example sources task to use DataSourceAPI commit ff3d1401c307d661307138ff05c29c9f03bc3961 Author: Roshanson <736781...@qq.com> Date: 2016-08-29T11:32:16Z refactor example sources task to use DataSourceAPI commit 4ff8453c494773b39acadceb9848aecbdd432859 Author: Roshanson <736781...@qq.com> Date: 2016-09-02T11:17:33Z refactor example sources task to use DataSourceAPI commit d8f0838ed3e31b844db46e122df56658a0a698a9 Author: 736781...@qq.com Date: 2016-09-02T12:24:30Z refactor example sources task to use DataSourceAPI commit bc9a2ee818278abc7b0b4bd8b3a42026f3905236 Author: 736781...@qq.com Date: 2016-09-02T12:26:48Z refactor example sources task to use DataSourceAPI commit 7191000bd4a97edb6575b092265ca886a61e07b0 Author: 736781...@qq.com Date: 2016-09-02T12:31:28Z refactor example sources task to use DataSourceAPI commit f4c6737198cc0a0a9a1a4ae42b6aece1e6a4eb3c Author: 736781...@qq.com Date: 2016-09-02T13:16:16Z [GEARPUMP-192] Refactor example sources task to use DataSourceAPI commit 84e884c42196a091d170a175600a91a4858d8ad4 Author: Roshanson <736781...@qq.com> Date: 2016-09-02T13:25:43Z [GEARPUMP-192] Refactor example sources task to use DataSourceAPI commit ee7d2acdb50686e781b31c283bf651eeab64b155 Author: Roshanson <736781...@qq.com> Date: 2016-09-13T13:30:57Z [GEARPUMP-204]Add unit test for external_hbase module commit 65e13bde0d728da62396ac59c73d01b3c5dc1320 Author: roshanson Date: 2016-09-19T09:00:49Z connection to hbase commit bc5c21e52fcdc46f1f4b6b350d4351bc2e1134f7 Author: Roshanson <736781...@qq.com> Date: 2016-09-02T11:17:33Z refactor example sources task to use DataSourceAPI commit 5db634a5f6cfa2ff198a46019bb8a4fa6b63cfeb Author: 736781...@qq.com Date: 2016-09-02T12:26:48Z refactor example sources task to use DataSourceAPI commit 57c7e8817ba9c8683039e050a217904d065f7a13 Author: 736781...@qq.com Date: 2016-09-02T12:31:28Z refactor example sources task to use DataSourceAPI commit 0097e2527783114a7d8890eca7173e5fb55a6194 Author: Roshanson <736781...@qq.com> Date: 2016-09-13T13:30:57Z [GEARPUMP-204]Add unit test for external_hbase module commit 99257d02e54bd0fa7ad530487aa1b2282964ca85 Author: roshanson Date: 2016-09-19T09:00:49Z connection to hbase commit a14df0d10d03219800a6b1a166ec9d773cfa76b3 Author: roshanson Date: 2016-09-21T04:33:57Z [GEARPUMP-204]add unit test for external_hbase module commit 8ef7dbb0d9b816a595441e66b0f71a041fc5b9f8 Author: roshanson Date: 2016-09-21T05:57:24Z [GEARPUMP-204]add unit test for external_hbase module commit 87364c049fd283c5da6898abf61973eae031a8a7 Author: roshanson Date: 2016-09-22T03:36:25Z [GEARPUMP-204]add unit test for external_hbase module commit 3086e4535fbbaa1c6e652f8e3cc3fdeb7bf24ead Author: roshanson Date: 2016-09-22T05:12:49Z [GEARPUMP-204]add unit test for external_hbase module commit ff7b930a6d2cd8081bdf2be1f5c2d69c2d144fd8 Author: roshanson Date: 2016-09-22T06:59:19Z [GEARPUMP-204]add unit test for external_hbase module commit f07f7feba571ba4a2ba830f8ae794b9dcd8f1e54 Author: roshanson Date: 2016-09-22T07:16:31Z [GEARPUMP-204]add unit test for external_hbase module commit 4920f2598721beede24e182205a87b72b9f13afb Author: roshanson Date: 2016-09-22T08:46:08Z [GEARPUMP-204]add unit test for external_hbase module commit 2e6ea2c33fdacddff283ea25d93493d1ef30bb4d Author: Roshanson Date: 2016-09-22T08:49:29Z [GEARPUMP-204]add unit test for external_hbase module commit 1caf488e88253380c01e319d59957f36694784a3 Author: Roshanson <736781...@qq.com> Date: 2016-09-22T11:41:31Z [GEARPUMP-204]Add unit test for external_hbase module commit 52d4b8d6ff4e1720147b9c088412625f5312f0a0 Author: Roshanson Date: 2016-09-23T02:41:37Z [GEARPUMP-204]add unit test for external_hbase module commit d52f51da4ccb797c7ef954b396b88735a2bf969e Author: Roshanson <736781...@qq.com> Date: 2016-09-23T03:33:33Z [GEARPUMP-204]add unit test for external_hbase module commit 501dca4861549e27beeb9a6ef89f4bb9902a3249 Author: Roshanson <736781...@qq.com> Date: 2016-09-27T10:53:25Z [GEARPUMP-204]Add unit test for external_hbase module commit c8120978b8609b41012823707ff04b830f1dfb50 Author: Roshans
[jira] [Updated] (GEARPUMP-230) Removed unused code from AppMasterService
[ https://issues.apache.org/jira/browse/GEARPUMP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated GEARPUMP-230: Assignee: Christoph Seifert > Removed unused code from AppMasterService > - > > Key: GEARPUMP-230 > URL: https://issues.apache.org/jira/browse/GEARPUMP-230 > Project: Apache Gearpump > Issue Type: Task > Components: restapi >Affects Versions: 0.8.2 >Reporter: Christoph Seifert >Assignee: Christoph Seifert >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (GEARPUMP-230) Removed unused code from AppMasterService
Christoph created GEARPUMP-230: -- Summary: Removed unused code from AppMasterService Key: GEARPUMP-230 URL: https://issues.apache.org/jira/browse/GEARPUMP-230 Project: Apache Gearpump Issue Type: Task Components: restapi Affects Versions: 0.8.2 Reporter: Christoph Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-228) Message's timestamp should be current time stamp
[ https://issues.apache.org/jira/browse/GEARPUMP-228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15621514#comment-15621514 ] ASF GitHub Bot commented on GEARPUMP-228: - Github user darionyaphet commented on the issue: https://github.com/apache/incubator-gearpump/pull/102 thanks @huafengw > Message's timestamp should be current time stamp > > > Key: GEARPUMP-228 > URL: https://issues.apache.org/jira/browse/GEARPUMP-228 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: darion yaphet >Assignee: darion yaphet > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-228) Message's timestamp should be current time stamp
[ https://issues.apache.org/jira/browse/GEARPUMP-228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15621515#comment-15621515 ] ASF GitHub Bot commented on GEARPUMP-228: - Github user darionyaphet closed the pull request at: https://github.com/apache/incubator-gearpump/pull/102 > Message's timestamp should be current time stamp > > > Key: GEARPUMP-228 > URL: https://issues.apache.org/jira/browse/GEARPUMP-228 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: darion yaphet >Assignee: darion yaphet > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
