[
https://issues.apache.org/jira/browse/BEAM-7427?focusedWorklogId=375651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-375651
]
ASF GitHub Bot logged work on BEAM-7427:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Jan/20 15:33
Start Date: 22/Jan/20 15:33
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #10644: [BEAM-7427]
Refactore JmsCheckpointMark to be usage via Coder
URL: https://github.com/apache/beam/pull/10644#discussion_r369619236
##########
File path: sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
##########
@@ -404,9 +408,96 @@ private JmsIO() {}
T mapMessage(Message message) throws Exception;
}
+ /**
+ * Checkpoint for an unbounded JMS source. Consists of the JMS messages
waiting to be acknowledged
+ * and oldest pending message timestamp.
+ */
+ @VisibleForTesting
+ static class JmsCheckpointMark implements UnboundedSource.CheckpointMark,
Serializable {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(JmsCheckpointMark.class);
+
+ @VisibleForTesting Instant oldestMessageTimestamp = Instant.now();
+ @VisibleForTesting transient List<Message> messages = new ArrayList<>();
+
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+
+ public JmsCheckpointMark() {}
+
+ public void add(Message message) throws Exception {
+ lock.writeLock().lock();
+ try {
+ Instant currentMessageTimestamp = new
Instant(message.getJMSTimestamp());
+ if (currentMessageTimestamp.isBefore(oldestMessageTimestamp)) {
+ oldestMessageTimestamp = currentMessageTimestamp;
+ }
+ messages.add(message);
+ } finally {
+ lock.writeLock().unlock();
+ }
+ }
+
+ public Instant getOldestMessageTimestamp() {
+ lock.readLock().lock();
+ try {
+ return this.oldestMessageTimestamp;
+ } finally {
+ lock.readLock().unlock();
+ }
+ }
+
+ /**
+ * Acknowledge all outstanding message. Since we believe that messages
will be delivered in
+ * timestamp order, and acknowledged messages will not be retried, the
newest message in this
+ * batch is a good bound for future messages.
+ */
+ @Override
+ public void finalizeCheckpoint() {
+ lock.writeLock().lock();
+ try {
+ for (Message message : messages) {
+ try {
+ message.acknowledge();
+ Instant currentMessageTimestamp = new
Instant(message.getJMSTimestamp());
+ if (currentMessageTimestamp.isAfter(oldestMessageTimestamp)) {
+ oldestMessageTimestamp = currentMessageTimestamp;
+ }
+ } catch (Exception e) {
+ LOG.error("Exception while finalizing message: {}", e);
+ }
+ }
+ messages.clear();
+ } finally {
+ lock.writeLock().unlock();
+ }
+ }
+
+ // set an empty list to messages when deserialize
+ private void readObject(java.io.ObjectInputStream stream)
+ throws IOException, ClassNotFoundException {
+ stream.defaultReadObject();
+ messages = new ArrayList<>();
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (other instanceof JmsCheckpointMark) {
+ JmsCheckpointMark that = (JmsCheckpointMark) other;
+ return Objects.equals(this.oldestMessageTimestamp,
that.oldestMessageTimestamp);
Review comment:
We need to add messages to the equals too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 375651)
Time Spent: 4h 20m (was: 4h 10m)
> JmsCheckpointMark Avro Serialization issue with UnboundedSource
> ---------------------------------------------------------------
>
> Key: BEAM-7427
> URL: https://issues.apache.org/jira/browse/BEAM-7427
> Project: Beam
> Issue Type: Bug
> Components: io-java-jms
> Affects Versions: 2.12.0
> Environment: Message Broker : solace
> JMS Client (Over AMQP) : "org.apache.qpid:qpid-jms-client:0.42.0
> Reporter: Mourad
> Assignee: Mourad
> Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> I get the following exception when reading from unbounded JMS Source:
>
> {code:java}
> Caused by: org.apache.avro.SchemaParseException: Illegal character in: this$0
> at org.apache.avro.Schema.validateName(Schema.java:1151)
> at org.apache.avro.Schema.access$200(Schema.java:81)
> at org.apache.avro.Schema$Field.<init>(Schema.java:403)
> at org.apache.avro.Schema$Field.<init>(Schema.java:396)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:622)
> at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:740)
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:604)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:218)
> at org.apache.avro.specific.SpecificData$2.load(SpecificData.java:215)
> at
> avro.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
> at
> avro.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
> {code}
>
> The exception is thrown by Avro when introspecting {{JmsCheckpointMark}} to
> generate schema.
> JmsIO config :
>
> {code:java}
> PCollection<DFAMessage> messages = pipeline.apply("read messages from the
> events broker", JmsIO.<DFAMessage>readMessage()
> .withConnectionFactory(jmsConnectionFactory) .withTopic(options.getTopic())
> .withMessageMapper(new DFAMessageMapper())
> .withCoder(AvroCoder.of(DFAMessage.class)));
> {code}
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
