[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859899#comment-15859899 ] Hudson commented on PHOENIX-3214: - SUCCESS: Integrated in Jenkins build Phoenix-master #1549 (See [https://builds.apache.org/job/Phoenix-master/1549/]) PHOENIX-3214 Addendum to add Apache Software License to kakfa props (jmahonin: rev 41d6349bd50de1ef13a52a8b146c01b504e60aeb) * (edit) phoenix-kafka/src/it/resources/consumer.props * (edit) phoenix-kafka/src/it/resources/producer.props > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Josh Mahonin > Fix For: 4.10.0 > > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214_license-2.patch, > PHOENIX-3214_license.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859783#comment-15859783 ] Josh Elser commented on PHOENIX-3214: - bq. Definitely would appreciate a second set of eyes on it though. An ounce of prevention is worth at least a pound of cure when it comes to potential classpath issues. I took a look at the dependency list and I don't see anything outwardly bad that Kafka brought in which is good. Sadly, I saw other things that were inherited that we were already doing wrong :). I think we're good for this contribution. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Josh Mahonin > Fix For: 4.10.0 > > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214_license-2.patch, > PHOENIX-3214_license.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858331#comment-15858331 ] James Taylor commented on PHOENIX-3214: --- +1 to commit. Good catch, [~kliew] and thanks for the quick follow up, [~jmahonin]! > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Josh Mahonin > Fix For: 4.10.0 > > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214_license-2.patch, > PHOENIX-3214_license.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858328#comment-15858328 ] Josh Mahonin commented on PHOENIX-3214: --- Just gave it a whirl, let's see what happens. If it doesn't trigger by EOD I'll commit it regardless. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Josh Mahonin > Fix For: 4.10.0 > > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214_license-2.patch, > PHOENIX-3214_license.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858298#comment-15858298
]
Kevin Liew commented on PHOENIX-3214:
-
[~jmahonin], Hudson can usually be triggered to run maven tests and release
auditing if you click {{Submit patch}} before uploading the patch file. You may
be able to re-open this issue to do this.
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3214.addendum-0.98.patch,
> PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch,
> PHOENIX-3214-final.patch, PHOENIX-3214_license.patch, PHOENIX-3214.patch,
> PHOENIX-3214-updated-1.patch, PHOENIX-3214-updated-2.patch,
> PHOENIX-3214-updated.patch
>
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858017#comment-15858017 ] Josh Mahonin commented on PHOENIX-3214: --- Hi [~kliew] I've attached a file 'PHOENIX-3214_license.patch' which I think should take care of this. I'm not sure how to trigger the license check, however. If you can let me know how to verify it, I can put it in. Thanks! Josh > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Fix For: 4.10.0 > > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214_license.patch, PHOENIX-3214.patch, > PHOENIX-3214-updated-1.patch, PHOENIX-3214-updated-2.patch, > PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856348#comment-15856348
]
Kevin Liew commented on PHOENIX-3214:
-
Thanks for the contribution [~kalyanhadoop].
Lets also add the Apache software license to {{consumer.props}} and
{{producer.props}} as it is causing release audit warnings.
https://builds.apache.org/job/PreCommit-PHOENIX-Build/755//artifact/patchprocess/patchReleaseAuditWarnings.txt
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
> Fix For: 4.10.0
>
> Attachments: PHOENIX-3214.addendum-0.98.patch,
> PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch,
> PHOENIX-3214-final.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch,
> PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch
>
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854439#comment-15854439 ] Josh Mahonin commented on PHOENIX-3214: --- Thanks for fixing my mess [~elserj], sorry again. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854361#comment-15854361 ] Josh Elser commented on PHOENIX-3214: - bq. Thanks for your support. I also verified your patches .. working fine. Good enough for me. We can get [~jmahonin] to take a look when he's eyes-on again :) Will push these. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854323#comment-15854323 ] Kalyan commented on PHOENIX-3214: - Hi [~elserj], Thanks for your support. I also verified your patches .. working fine. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.addendum-0.98.patch, > PHOENIX-3214.addendum-1.1.patch, PHOENIX-3214-docs.patch, > PHOENIX-3214-final.patch, PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854287#comment-15854287 ] Josh Elser commented on PHOENIX-3214: - bq. i am working on this. let me try to fix .. problem only in pom.xml file in parent. For 4.x-HBase-1.1, yes, but it's a little more involved for 4.x-HBase-0.98 :) > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854284#comment-15854284 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin], [~elserj], [~ankit.singhal], i am working on this. let me try to fix .. problem only in pom.xml file in parent. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854282#comment-15854282 ] Hudson commented on PHOENIX-3214: - FAILURE: Integrated in Jenkins build Phoenix-master #1547 (See [https://builds.apache.org/job/Phoenix-master/1547/]) PHOENIX-3214 Kafka Phoenix Consumer (Kalyan Hadoop) (jmahonin: rev ad113c8351cee509e915420d0d8e8f2e22b0945c) * (add) phoenix-kafka/src/it/resources/producer.props * (edit) phoenix-assembly/pom.xml * (add) phoenix-kafka/src/main/java/org/apache/phoenix/kafka/consumer/PhoenixConsumer.java * (add) phoenix-kafka/src/it/resources/consumer.props * (add) phoenix-kafka/src/main/java/org/apache/phoenix/kafka/consumer/PhoenixConsumerTool.java * (add) phoenix-kafka/pom.xml * (add) phoenix-kafka/src/main/java/org/apache/phoenix/kafka/KafkaConstants.java * (edit) pom.xml * (edit) phoenix-assembly/src/build/components/all-common-jars.xml * (add) phoenix-kafka/src/it/java/org/apache/phoenix/kafka/PhoenixConsumerIT.java > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854175#comment-15854175 ] Josh Mahonin commented on PHOENIX-3214: --- [~kalyanhadoop] If you have any cycles to look at working the patch into the 4.x-HBase-0.98 (and presumably 4.x-HBase-1.1 branch), that'd be great. Thanks! > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854164#comment-15854164 ] Josh Mahonin commented on PHOENIX-3214: --- Sorry about that. I'm seeing the same thing. I neglected to update the phoenix-kafka/pom.xml parent version. It looks like there's also some other dependencies that might need to be updated too. I'm booked up for the next few hours. If this is totally blocking, let's revert it, otherwise I'll attempt to fix as soon as I can today. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854135#comment-15854135
]
Ankit Singhal commented on PHOENIX-3214:
ping [~jmahonin], it seems it broke 4.x-HBase-0.98 build. Not sure if it is the
problem with my local setup but it was running fine before the update of this
commit.
{code}
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project
org.apache.phoenix:phoenix-kafka:4.10.0-HBase-0.98-SNAPSHOT
(/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml) has 8 errors
[ERROR] 'dependencies.dependency.version' for
org.apache.tephra:tephra-hbase-compat-1.1:jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 68, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.htrace:htrace-core:jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 171, column 15
[ERROR] 'dependencies.dependency.version' for io.netty:netty-all:jar is
missing. @ org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 175, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.hbase:hbase-annotations:jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 207, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.hbase:hbase-common:test-jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 215, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.hbase:hbase-server:test-jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 239, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.hbase:hbase-hadoop2-compat:jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 255, column 15
[ERROR] 'dependencies.dependency.version' for
org.apache.hbase:hbase-hadoop2-compat:test-jar is missing. @
org.apache.phoenix:phoenix-kafka:[unknown-version],
/Users/asinghal/git/apache/phoenix/phoenix-kafka/pom.xml, line 260, column 15
{code}
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
> Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch,
> PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch,
> PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch
>
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853270#comment-15853270 ] James Taylor commented on PHOENIX-3214: --- No objections from me. Thanks for following up, [~jmahonin] and thanks for the good work, [~kalyanhadoop]. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853266#comment-15853266 ] Josh Mahonin commented on PHOENIX-3214: --- Thanks for the updated patch [~kalyanhadoop] Verified unit tests run and pass locally. LGTM. Anyone have any objections to getting this in? > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, > PHOENIX-3214-updated-2.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840080#comment-15840080
]
Josh Mahonin commented on PHOENIX-3214:
---
Overall the patch looks good and applies cleanly.
However, I'm unable to successfully run the integration tests using 'mvn
verify'. Exceptions logged below. This could very well just be my environment,
but [~kalyanhadoop] can you describe how you run the integration tests, and on
what environment? Thanks!
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m;
support was removed in 8.0
Running org.apache.phoenix.kafka.PhoenixConsumerIT
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 sec <<<
FAILURE! - in org.apache.phoenix.kafka.PhoenixConsumerIT
org.apache.phoenix.kafka.PhoenixConsumerIT Time elapsed: 0.004 sec <<< ERROR!
java.lang.RuntimeException: java.io.IOException: Failed to save in any storage
directories while saving namespace.
Caused by: java.io.IOException: Failed to save in any storage directories while
saving namespace.
Results :
Tests in error:
PhoenixConsumerIT>BaseHBaseManagedTimeIT.doSetup:57->BaseTest.setUpTestDriver:557->BaseTest.setUpTestDriver:561->BaseTest.checkClusterInitialized:483->BaseTest.setUpTestCluster:509->BaseTest.initMiniCluster:591
ยป Runtime
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] --- maven-failsafe-plugin:2.19.1:integration-test
(NeedTheirOwnClusterTests) @ phoenix-kafka ---
{noformat}
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
> Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch,
> PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, PHOENIX-3214-updated.patch
>
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838261#comment-15838261 ] James Taylor commented on PHOENIX-3214: --- [~jmahonin] - any chance for a review on this one? > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214.patch, PHOENIX-3214-updated-1.patch, PHOENIX-3214-updated.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823355#comment-15823355
]
Josh Mahonin commented on PHOENIX-3214:
---
Hi [~kalyanhadoop]
I've just tried applying the {{PHOENIX-3214-updated.patch}} file to the master
branch, but I end up with a build failure:
{noformat}
[ERROR]
/Users/jmahonin/devel/phoenix/phoenix-kafka/src/main/java/org/apache/phoenix/kafka/consumer/PhoenixConsumer.java:[276,2]
class, interface, or enum expected
[ERROR]
/Users/jmahonin/devel/phoenix/phoenix-kafka/src/main/java/org/apache/phoenix/kafka/consumer/PhoenixConsumerTool.java:[107,2]
class, interface, or enum expected
[ERROR]
/Users/jmahonin/devel/phoenix/phoenix-kafka/src/main/java/org/apache/phoenix/kafka/KafkaConstants.java:[52,2]
class, interface, or enum expected
{noformat}
Please provide an updated patch. Thanks!
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
> Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch,
> PHOENIX-3214.patch, PHOENIX-3214-updated.patch
>
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814968#comment-15814968 ] Josh Mahonin commented on PHOENIX-3214: --- Hi [~kalyanhadoop] this is definitely on my radar. I've had to do some short-notice travel recently, but will try get to this in the next day or two. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810215#comment-15810215 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin], can you please review it > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15776569#comment-15776569 ] Kalyan commented on PHOENIX-3214: - Thanks [~giacomotaylor] , [~jmahonin] > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15775660#comment-15775660 ] James Taylor commented on PHOENIX-3214: --- This is awesome, [~kalyanhadoop]. Nice work on the docs. I'm sure [~jmahonin will review once the holidays are done. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15775627#comment-15775627 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin] can please review it .. it is completed with test cases and document. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-docs.patch, PHOENIX-3214-final.patch, > PHOENIX-3214-updated.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767196#comment-15767196 ] Kalyan commented on PHOENIX-3214: - Hi [~jmahonin] This integration internal uses `phoenix-flume plugin` only. I am converting streaming `Kakfa messages` as a `Flume events` then using total `phoenix-flume plugin` to insert into Phoenix Tables. I made as a separate `phoenix-kafka` module and provided the dependency as a `phoenix-flume` module. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766061#comment-15766061 ] Josh Mahonin commented on PHOENIX-3214: --- Hi [~kalyanhadoop] Overall the patch looks good, though I'm curious if it uses the phoenix-flume serializers anyway, what's the advantage of using this instead of just combining the Kafka consumer built-in to Flume with the phoenix-flume producer? As well, would you be willing to include a README file with a high level overview, and maybe with some example usage and / or code samples? Ideally we'd end up using that as the basis of documentation on the website as well with the other 'Integration' pages. You can find the source repository for the website here: https://phoenix.apache.org/building_website.html > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761938#comment-15761938 ] Kalyan commented on PHOENIX-3214: - Hi Josh Mahonin, Sorry i missed those files in previous patch. Updated final patch. Thanks > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214-final.patch, PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761890#comment-15761890 ] Josh Mahonin commented on PHOENIX-3214: --- Hi [~kalyanhadoop]. I think that patch is missing the files under 'kafka/consumer'. That patch just has the POM and PhoenixConsumerIT files. > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761866#comment-15761866 ] Kalyan commented on PHOENIX-3214: - Hi Josh Mahonin, Please review the patch. Thanks > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > Attachments: PHOENIX-3214.patch > > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761640#comment-15761640 ] Josh Mahonin commented on PHOENIX-3214: --- This looks pretty cool [~kalyanhadoop]. Would it be possible for you to either create a Github pull request, or attach a patchfile against the master branch? I'm finding it difficult to track the commits / changes. Thanks! > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760418#comment-15760418 ] Kalyan commented on PHOENIX-3214: - Kevin Liew, thanks for review. Updated as per suggestion. https://github.com/kalyanhadooptraining/phoenix/commit/d6d8061f554e52ab692414affc115386a006a647 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[
https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759801#comment-15759801
]
Kevin Liew commented on PHOENIX-3214:
-
Can we import and use those macros from ConsumerConfig instead of redefining
the same thing in KafkaConstants?
ie. instead of using
{code:java}
producerProperties.setProperty(KafkaConstants.KEY_SERIALIZER,
KafkaConstants.DEFAULT_KEY_SERIALIZER);
{code}
use this instead
{code:java}
producerProperties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
KafkaConstants.DEFAULT_KEY_SERIALIZER);
{code}
or in KafkaConstants
{code:java}
public static final String KEY_SERIALIZER =
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG;
{code}
> Kafka Phoenix Consumer
> --
>
> Key: PHOENIX-3214
> URL: https://issues.apache.org/jira/browse/PHOENIX-3214
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Kalyan
>Assignee: Kalyan
>
> Providing a new feature to Phoenix.
> Directly ingest Kafka messages to Phoenix.
> Similar to flume phoenix integration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759792#comment-15759792 ] Kalyan commented on PHOENIX-3214: - we can reuse it ... the below line will take care of it. this.consumer = new KafkaConsumer<>(properties); https://github.com/kalyanhadooptraining/phoenix/commit/b35a7fb8ffb1fd7745a0467a4cea3b761459e463#diff-475a67c1e9b3f68ef5d95f4ce536bb04R121 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759762#comment-15759762 ] Kevin Liew commented on PHOENIX-3214: - Many of the macros in KafkaConstants are already defined in org.apache.kafka.clients.consumer.ConsumerConfig. Can we reuse some of those? > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759737#comment-15759737 ] Kalyan commented on PHOENIX-3214: - Can any one please look into the patch ... any suggestions ? > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PHOENIX-3214) Kafka Phoenix Consumer
[ https://issues.apache.org/jira/browse/PHOENIX-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448913#comment-15448913 ] Kalyan commented on PHOENIX-3214: - Providing a new feature to Phoenix. Directly ingest Kafka messages to Phoenix. Similar to flume phoenix integration. Implementation is done using phoenix-flume solution for inserting the kafka messages. Please review the patch: https://github.com/kalyanhadooptraining/phoenix/commit/b35a7fb8ffb1fd7745a0467a4cea3b761459e463 > Kafka Phoenix Consumer > -- > > Key: PHOENIX-3214 > URL: https://issues.apache.org/jira/browse/PHOENIX-3214 > Project: Phoenix > Issue Type: New Feature >Reporter: Kalyan >Assignee: Kalyan > > Providing a new feature to Phoenix. > Directly ingest Kafka messages to Phoenix. > Similar to flume phoenix integration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
