Re: [SAMZA-423] Integrate Lucene into Samza
Hi Robert, Thank you for the contribution and very sorry for the late replay. :( Left some comments in the JIRA in terms of the design doc. Hope that can help you polish your design. Thanks, Fang, Yan yanfang...@gmail.com On Thu, Aug 13, 2015 at 12:29 AM, Robert Zuljevic wrote: > Hello all, > > > > I’ve added a design document to task SAMZA-423. If anyone is interested > please take a look : ) > > > > I have some outstanding questions that I would like to resolve so I could > continue working on this as soon as possible: > > > > 1. Should the API support bulk indexing/removing/matching? (note > that in some cases this might end up being only cosmetical)? > > 2. Should the match API return the objects representing indexed > elements or their wrappers (as suggested in the design document)? > > 3. Should this task renamed to something like “Create Document > Store” and create two other tasks dealing with concrete implementations in > Luwak and Lucene (which is the endgoal). > > > > As stated in the ticket any and all suggestions/comments/criticisms are > welcome : ) > > > > Met vriendelijke groet / Kind regards, > > Robert Žuljević > > Software Developer > > [image: Title: Levi9 IT Services] > -- > > Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia > > Tel.: +31 20 6701 947 | +381 21 2155 500 > > Mobile: +381 64 428 28 46 > > Skype: robert.zuljevic > > Internet: www.levi9.com > > > > Chamber of commerce Levi9 Holding: 34221951 > > Chamber of commerce Levi9 IT Services BV: 34224746 > -- > > This e-mail may contain confidential or privileged information. If you are > not (one of) the intended recipient(s), please notify the sender > immediately by reply e-mail and delete this message and any attachments > permanently without retaining a copy. Any review, disclosure, copying, > distribution or taking any action in reliance on the contents of this > e-mail by persons or entities other than the intended recipient(s) is > strictly prohibited and may be unlawful. > > The services of Levi9 are exclusively subject to its general terms and > conditions. These general terms and conditions can be found on > www.levi9.com and a copy will be promptly submitted to you on your > request and free of charge. > > >
Re: remote kafka producer -- kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries
Hi Selina,
I guess you can post this question in the Kafka mailing list if this is
"pure" kafka, there are more experts in that community, though there are
some Kafka experts here as well. :)
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, Aug 13, 2015 at 9:56 PM, Job-Selina Wu
wrote:
> Dear All:
>
>I got kafka.common.FailedToSendMessageException: Failed to send
> messages after 3 tries as below. When I have a remote java Kafka producer
> try to produce message to Kafka broker Server. Both Producer and Broker are
> at AWS cloud. BTW, I tried my code first at local machine and Virtual
> machine first, It did not work either.(advertised.host.name was set to
> the ip address of the kafka server*)*
>
> *-**This is my KafkaProducer at remote Producer for producer
> configuration**-*
> public class KafkaProducer {
>
> Properties props = new Properties();
>
> private final Producer producer;
> private final String kafkaServerIP = "52.19.2.74:9092";
>
> public KafkaProducer() {
>
>
> props.put("metadata.broker.list", kafkaServerIP);
> //props.put("bootstrap.servers", "localhost:9092 ");
> props.put("serializer.class", "kafka.serializer.StringEncoder");
> props.put("advertised.host.name", "localhost");
> props.put("request.required.acks", "0");
>
> ProducerConfig config = new ProducerConfig(props);
>
> producer = new Producer(config);
> }
>
> public Producer getProducer() {
>
> return this.producer;
> }
> }
>
>
> *The configs/server.properties at Kafka Server at AWS*-
>
> zookeeper.connect=localhost:2181
> zookeeper.connection.timeout.ms=6000
>
> delete.topic.enable=true
>
> broker.id=0
> port=9092
> host.name=localhost
> *advertised.host.name *=
> ec2-51-18-21-235.us-west-1.compute.amazonaws.com
>
> # below is same as default
> #advertised.port=
> #advertised.port=
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/tmp/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=1
> #log.flush.interval.messages=1
> #log.flush.interval.ms=1000
> log.retention.hours=168
> #log.retention.bytes=1073741824
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=30
> log.cleaner.enable=false
>
>
> - - --- - - - - --
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at kafka.producer.Producer.send(Producer.scala:77)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at com.cinarra.kafka.Main.main(Main.java:21)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
>
>
> reference:
>
> http://stackoverflow.com/questions/30217255/cant-connect-to-a-remote-kafka-producer-from-windows-through-java-code
>
> Your help is highly appreciated,
> Selina
>
>
Re: KIP-28 kafka processor
+1 Any feedback would be appreciated! -Jay On Sat, Aug 15, 2015 at 3:55 PM, Chris Riccomini wrote: > Hey all, > > I wanted to call attention to KIP-28: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client > > This is the result of the last conversation that we had about > samza's future direction. > > It would be good to have the samza community involved in this. > > Cheers, > Chris >
Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37506/ --- (Updated Aug. 16, 2015, 3:57 p.m.) Review request for samza, Yi Pan (Data Infrastructure) and Navina Ramesh. Bugs: SAMZA-552 https://issues.apache.org/jira/browse/SAMZA-552 Repository: samza Description (updated) --- New proposal for TopologuBuilder API proposed in rb34500 (https://reviews.apache.org/r/34500/). * Created a new class called TopologyBuilderV2 instead of changing existing TopologyBuilder * org.apache.samza.sql.operators.factory.TestTopologyBuilderV2 contains two tests which demonstrate the basic usage of the new API * Window and aggregate related draft APIs are not done yet * This is a WIP, please feel free to comment on the APIs * This contains Yi's changes from RB 34500 Diffs - samza-sql-core/src/main/java/org/apache/samza/sql/api/data/EntityName.java 80ba455 samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Schema.java 1e8f192 samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java 7b4d984 samza-sql-core/src/main/java/org/apache/samza/sql/api/expressions/ScalarExpression.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/expressions/TupleExpression.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java d6f6b57 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorCallback.java fb2aa89 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorRouter.java 0759638 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSink.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSource.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SimpleOperator.java c49a822 samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java 72a59f2 samza-sql-core/src/main/java/org/apache/samza/sql/operators/NoopOperatorCallback.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/OperatorTopology.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleOperatorImpl.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleOperatorSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleRouter.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/NoopOperatorCallback.java c3d2266 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java cbc84d0 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorImpl.java e66451f samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java 56753b6 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleRouter.java e570897 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilder.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilderException.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilderV2.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/filter/FilterOp.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/filter/FilterSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/JoinSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/JoinType.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamRelationJoin.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java 2854aeb samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java cc0aca0 samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/InsertToStreamOp.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/Operation.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/StreamModifySpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/TableModifySpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/FieldBasedPartitionKeyGenerator.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java b93d789 samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java c47eed9 samza-sql-core/src/main/java/org/apache/samza/sql/operators/project/ProjectOp.java PRE-CRE
Re: Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37506/ --- (Updated Aug. 16, 2015, 3:56 p.m.) Review request for samza, Yi Pan (Data Infrastructure) and Navina Ramesh. Bugs: SAMZA-552 https://issues.apache.org/jira/browse/SAMZA-552 Repository: samza Description --- New proposal for TopologuBuilder API proposed in rb34500 (https://reviews.apache.org/r/34500/). * Created a new class called TopologyBuilderV2 instead of changing existing TopologyBuilder * org.apache.samza.sql.operators.factory.TestTopologyBuilderV2 contains two tests which demonstrate the basic usage of the new API * Window and aggregate related draft APIs are not done yet * This is a WIP, please feel free to comment on the APIs Diffs (updated) - samza-sql-core/src/main/java/org/apache/samza/sql/api/data/EntityName.java 80ba455 samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Schema.java 1e8f192 samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java 7b4d984 samza-sql-core/src/main/java/org/apache/samza/sql/api/expressions/ScalarExpression.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/expressions/TupleExpression.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java d6f6b57 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorCallback.java fb2aa89 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorRouter.java 0759638 samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSink.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSource.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SimpleOperator.java c49a822 samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java 72a59f2 samza-sql-core/src/main/java/org/apache/samza/sql/operators/NoopOperatorCallback.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/OperatorTopology.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleOperatorImpl.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleOperatorSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleRouter.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/NoopOperatorCallback.java c3d2266 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java cbc84d0 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorImpl.java e66451f samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java 56753b6 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleRouter.java e570897 samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilder.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilderException.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilderV2.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/filter/FilterOp.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/filter/FilterSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/JoinSpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/JoinType.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamRelationJoin.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java 2854aeb samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java cc0aca0 samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/InsertToStreamOp.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/Operation.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/StreamModifySpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/modify/TableModifySpec.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/FieldBasedPartitionKeyGenerator.java PRE-CREATION samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java b93d789 samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java c47eed9 samza-sql-core/src/main/java/org/apache/samza/sql/operators/project/ProjectOp.java PRE-CREATION samza-sql-core/src/main/java/org/
Review Request 37506: WIP: SAMZA-552 Operator API change: New Builder API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/37506/ --- Review request for samza, Yi Pan (Data Infrastructure) and Navina Ramesh. Bugs: SAMZA-552 https://issues.apache.org/jira/browse/SAMZA-552 Repository: samza Description --- New proposal for TopologuBuilder API proposed in rb34500 (https://reviews.apache.org/r/34500/). * Created a new class called TopologyBuilderV2 instead of changing existing TopologyBuilder * org.apache.samza.sql.operators.factory.TestTopologyBuilderV2 contains two tests which demonstrate the basic usage of the new API * Window and aggregate related draft APIs are not done yet * This is a WIP, please feel free to comment on the APIs Diffs - samza-sql-calcite/src/main/java/org/apache/samza/sql/calcite/schema/AvroSchemaConverter.java 705c0ffadd0982defae686bb0df178e9a7cbef8c samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java 20dc70179e74070e8feb617b6a7b3b62ef5c1156 samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java 9124e3c7191aac8a738345c02b260baae811c5c5 samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java 96e96c363e723c20c30622d1253588f66166b206 Diff: https://reviews.apache.org/r/37506/diff/ Testing --- ./gradlew :samza-sql-core:test passed Thanks, Milinda Pathirage
