[jira] [Created] (APEXMALHAR-2195) LineReaderContext gives incorrect results for files not ending with the newline
Yogi Devendra created APEXMALHAR-2195: - Summary: LineReaderContext gives incorrect results for files not ending with the newline Key: APEXMALHAR-2195 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2195 Project: Apache Apex Malhar Issue Type: Bug Reporter: Yogi Devendra Assignee: Yogi Devendra Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2195) LineReaderContext gives incorrect results for files not ending with the newline
[ https://issues.apache.org/jira/browse/APEXMALHAR-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422467#comment-15422467 ] Yogi Devendra commented on APEXMALHAR-2195: --- ReaderContext.LineReaderContext not able to correctly read last record of the files not ending with the newline character. > LineReaderContext gives incorrect results for files not ending with the > newline > --- > > Key: APEXMALHAR-2195 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2195 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Yogi Devendra >Assignee: Yogi Devendra >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-malhar pull request #372: APEXMALHAR-2195 - Fixing LineReaderContext Is...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/372 APEXMALHAR-2195 - Fixing LineReaderContext Issue 2. Changes in the test app You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2195-LineReaderContext-last-record Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit accc093ddb70faecc5b6a7342ad52bbaf010405b Author: yogidevendra Date: 2016-08-14T17:45:30Z Fixing ReaderContext Issue Changes in the test app --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2195) LineReaderContext gives incorrect results for files not ending with the newline
[ https://issues.apache.org/jira/browse/APEXMALHAR-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422472#comment-15422472 ] ASF GitHub Bot commented on APEXMALHAR-2195: GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/372 APEXMALHAR-2195 - Fixing LineReaderContext Issue 2. Changes in the test app You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2195-LineReaderContext-last-record Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit accc093ddb70faecc5b6a7342ad52bbaf010405b Author: yogidevendra Date: 2016-08-14T17:45:30Z Fixing ReaderContext Issue Changes in the test app > LineReaderContext gives incorrect results for files not ending with the > newline > --- > > Key: APEXMALHAR-2195 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2195 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Yogi Devendra >Assignee: Yogi Devendra >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXMALHAR-2196) S3 Record reader module
Yogi Devendra created APEXMALHAR-2196: - Summary: S3 Record reader module Key: APEXMALHAR-2196 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2196 Project: Apache Apex Malhar Issue Type: New Feature Reporter: Yogi Devendra Assignee: Yogi Devendra -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/361 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2176-filter-expressionFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 083bef378df3839513b49fd91f975a4824042b98 Author: yogidevendra Date: 2016-08-04T14:31:11Z APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method 2. incorporating review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds
[
https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422505#comment-15422505
]
ASF GitHub Bot commented on APEXMALHAR-2176:
Github user yogidevendra closed the pull request at:
https://github.com/apache/apex-malhar/pull/361
> expressionFunctions for FilterOperator throws IndexOutOfBounds
> --
>
> Key: APEXMALHAR-2176
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2176
> Project: Apache Apex Malhar
> Issue Type: Bug
>Reporter: Yogi Devendra
>Assignee: Yogi Devendra
>Priority: Minor
>
> If expressionFunctions are added through xml conf file as follows
> {code}
>
>
> dt.application.FilterExample.operator.filterOperator.prop.expressionFunctions[5]
> org.apache.commons.lang3.BooleanUtils.*
>
> {code}
> This gives IndexOutOfBoundsException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds
[
https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422506#comment-15422506
]
ASF GitHub Bot commented on APEXMALHAR-2176:
GitHub user yogidevendra reopened a pull request:
https://github.com/apache/apex-malhar/pull/361
APEXMALHAR-2176 expressionFunctions for FilterOperator
1. added setExpressionFunctionsItem method
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yogidevendra/apex-malhar
APEXMALHAR-2176-filter-expressionFunctions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-malhar/pull/361.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #361
commit 083bef378df3839513b49fd91f975a4824042b98
Author: yogidevendra
Date: 2016-08-04T14:31:11Z
APEXMALHAR-2176 expressionFunctions for FilterOperator
1. added setExpressionFunctionsItem method
2. incorporating review comments.
> expressionFunctions for FilterOperator throws IndexOutOfBounds
> --
>
> Key: APEXMALHAR-2176
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2176
> Project: Apache Apex Malhar
> Issue Type: Bug
>Reporter: Yogi Devendra
>Assignee: Yogi Devendra
>Priority: Minor
>
> If expressionFunctions are added through xml conf file as follows
> {code}
>
>
> dt.application.FilterExample.operator.filterOperator.prop.expressionFunctions[5]
> org.apache.commons.lang3.BooleanUtils.*
>
> {code}
> This gives IndexOutOfBoundsException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2196) S3 Record reader module
[ https://issues.apache.org/jira/browse/APEXMALHAR-2196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422518#comment-15422518 ] Yogi Devendra commented on APEXMALHAR-2196: --- This module will be similar in functionality as what FSRecordReader achieves for HDFS and other filesystems. Difference being FSRecordReader reads using hadoop FS interface. This module will read using amazon S3client interface. > S3 Record reader module > --- > > Key: APEXMALHAR-2196 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2196 > Project: Apache Apex Malhar > Issue Type: New Feature >Reporter: Yogi Devendra >Assignee: Yogi Devendra > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/361 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds
[
https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422549#comment-15422549
]
ASF GitHub Bot commented on APEXMALHAR-2176:
Github user yogidevendra closed the pull request at:
https://github.com/apache/apex-malhar/pull/361
> expressionFunctions for FilterOperator throws IndexOutOfBounds
> --
>
> Key: APEXMALHAR-2176
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2176
> Project: Apache Apex Malhar
> Issue Type: Bug
>Reporter: Yogi Devendra
>Assignee: Yogi Devendra
>Priority: Minor
>
> If expressionFunctions are added through xml conf file as follows
> {code}
>
>
> dt.application.FilterExample.operator.filterOperator.prop.expressionFunctions[5]
> org.apache.commons.lang3.BooleanUtils.*
>
> {code}
> This gives IndexOutOfBoundsException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2176-filter-expressionFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 083bef378df3839513b49fd91f975a4824042b98 Author: yogidevendra Date: 2016-08-04T14:31:11Z APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method 2. incorporating review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXMALHAR-2176) expressionFunctions for FilterOperator throws IndexOutOfBounds
[
https://issues.apache.org/jira/browse/APEXMALHAR-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422550#comment-15422550
]
ASF GitHub Bot commented on APEXMALHAR-2176:
GitHub user yogidevendra reopened a pull request:
https://github.com/apache/apex-malhar/pull/361
APEXMALHAR-2176 expressionFunctions for FilterOperator
1. added setExpressionFunctionsItem method
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yogidevendra/apex-malhar
APEXMALHAR-2176-filter-expressionFunctions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-malhar/pull/361.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #361
commit 083bef378df3839513b49fd91f975a4824042b98
Author: yogidevendra
Date: 2016-08-04T14:31:11Z
APEXMALHAR-2176 expressionFunctions for FilterOperator
1. added setExpressionFunctionsItem method
2. incorporating review comments.
> expressionFunctions for FilterOperator throws IndexOutOfBounds
> --
>
> Key: APEXMALHAR-2176
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2176
> Project: Apache Apex Malhar
> Issue Type: Bug
>Reporter: Yogi Devendra
>Assignee: Yogi Devendra
>Priority: Minor
>
> If expressionFunctions are added through xml conf file as follows
> {code}
>
>
> dt.application.FilterExample.operator.filterOperator.prop.expressionFunctions[5]
> org.apache.commons.lang3.BooleanUtils.*
>
> {code}
> This gives IndexOutOfBoundsException
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Planning to add an InputOperator for gRPC and Protobuf
Sanjay, Good proposal. Does gRPC have an asynchronous API or do you need the separate thread to make blocking calls? It may also be interesting to further abstract the RPC interaction to possibly reuse the base operator for similar IO pattern such as HTTP. Thomas On Mon, Aug 15, 2016 at 3:26 PM, Sanjay Pujare wrote: > I am thinking of adding an input operator to Apex Malhar that allows gRPC > based message streams to be consumed by an Apex system. > > > > gRPC (http://www.grpc.io/posts/principles) is a recent open source RPC > framework that started at Google and is becoming popular. It is typically > used with Protobuf (a serialization framework also developed at Google, see > https://developers.google.com/protocol-buffers/docs/overview). > > > > In this proposal I will create an AbstractGrpcInputOperator that will > behave somewhat like the Http input operator in the sense that it will > generate a request to the Grpc service and will process the response to > parse the individual messages and emit tuples. Of course the operator will > have support for idempotency and exception handling. We will also try to > add support for partitionability and dynamic scalability based on their > applicability to the Grpc input operator. Similarly we will > opportunistically add support for Client interceptors ( > http://www.grpc.io/grpc-java/javadoc/io/grpc/ClientInterceptor.html) and > other gRPC usage models (e.g. unary vs streaming). > > > > A developer uses the “protoc” compiler and an input “proto” file to > generate Java classes that define the client “stubs” and serialized message > classes that correspond to the RPC definition in the proto file. Hence > AbstractGrpcInputOperator is a generic class requiring the request and > response type arguments: > > > > abstract class AbstractGrpcInputOperator GeneratedMessageV3, ResponseType extends GeneratedMessageV3> > > > > All Protobuf (version 3) generated protocol message classes extend class > com.google.protobuf.GeneratedMessageV3. This class implements most of the > Message and Builder interfaces using Java reflection. > > > > The operator also needs an “AbstractStub” instance that is generated by > “protoc”. AbstractStub is the common base type for client stub > implementations. It encapsulates things such as remote host+port, TLS vs > TCP transport, and trust store in case of TLS. > > > > The operator also needs a MethodDescriptor object (which encapsulates the > name of the operation to execute as well as Marshaller instances used to > parse and serialize request and response messages) and a RequestType object > that contains the RPC/Request arguments. > > > > The operator will create a separate thread to asynchronously post gRPC > requests in an infinite loop and the same thread will process the response > for received messages (ResponseType objects). These ResponseType objects > will be added to an ArrayBlockingQueue and the emitTuple() will read this > queue to generate the tuples (similar to the logic in > AbstractJMSInputOperator of Malhar). > > > > The class will go in the package org.apache.apex.malhar.lib.io.grpc . > User will need to subclass this class and provide the actual types for > RequestType and ResponseType as well as the properties described above. > > > > Comments/feedback welcome. > > > > Sanjay > > > >
Re: Planning to add an InputOperator for gRPC and Protobuf
సెలవు > On Aug 16, 2016, at 9:02 AM, Thomas Weise wrote: > > Sanjay, > > Good proposal. Does gRPC have an asynchronous API or do you need the > separate thread to make blocking calls? > > It may also be interesting to further abstract the RPC interaction to > possibly reuse the base operator for similar IO pattern such as HTTP. > > Thomas > > > On Mon, Aug 15, 2016 at 3:26 PM, Sanjay Pujare > wrote: > >> I am thinking of adding an input operator to Apex Malhar that allows gRPC >> based message streams to be consumed by an Apex system. >> >> >> >> gRPC (http://www.grpc.io/posts/principles) is a recent open source RPC >> framework that started at Google and is becoming popular. It is typically >> used with Protobuf (a serialization framework also developed at Google, see >> https://developers.google.com/protocol-buffers/docs/overview). >> >> >> >> In this proposal I will create an AbstractGrpcInputOperator that will >> behave somewhat like the Http input operator in the sense that it will >> generate a request to the Grpc service and will process the response to >> parse the individual messages and emit tuples. Of course the operator will >> have support for idempotency and exception handling. We will also try to >> add support for partitionability and dynamic scalability based on their >> applicability to the Grpc input operator. Similarly we will >> opportunistically add support for Client interceptors ( >> http://www.grpc.io/grpc-java/javadoc/io/grpc/ClientInterceptor.html) and >> other gRPC usage models (e.g. unary vs streaming). >> >> >> >> A developer uses the “protoc” compiler and an input “proto” file to >> generate Java classes that define the client “stubs” and serialized message >> classes that correspond to the RPC definition in the proto file. Hence >> AbstractGrpcInputOperator is a generic class requiring the request and >> response type arguments: >> >> >> >> abstract class AbstractGrpcInputOperator> GeneratedMessageV3, ResponseType extends GeneratedMessageV3> >> >> >> >> All Protobuf (version 3) generated protocol message classes extend class >> com.google.protobuf.GeneratedMessageV3. This class implements most of the >> Message and Builder interfaces using Java reflection. >> >> >> >> The operator also needs an “AbstractStub” instance that is generated by >> “protoc”. AbstractStub is the common base type for client stub >> implementations. It encapsulates things such as remote host+port, TLS vs >> TCP transport, and trust store in case of TLS. >> >> >> >> The operator also needs a MethodDescriptor object (which encapsulates the >> name of the operation to execute as well as Marshaller instances used to >> parse and serialize request and response messages) and a RequestType object >> that contains the RPC/Request arguments. >> >> >> >> The operator will create a separate thread to asynchronously post gRPC >> requests in an infinite loop and the same thread will process the response >> for received messages (ResponseType objects). These ResponseType objects >> will be added to an ArrayBlockingQueue and the emitTuple() will read this >> queue to generate the tuples (similar to the logic in >> AbstractJMSInputOperator of Malhar). >> >> >> >> The class will go in the package org.apache.apex.malhar.lib.io.grpc . >> User will need to subclass this class and provide the actual types for >> RequestType and ResponseType as well as the properties described above. >> >> >> >> Comments/feedback welcome. >> >> >> >> Sanjay >> >> >> >>
Re: Join Support
Hi Shunxin, One problem with join support using WindowedOperator is that Apex operator does not support variable number of ports so we might have to limit the join operator to, say, 5 input ports. Implementing join support for WindowedOperator should not be difficult, but might be a little messy because we will need to have a watermark control port for each regular input port, making it 10 total input ports if we support a maximum of 5 join inputs. Please take a look at the JoinAccumulation template interface. That was there for the future join support I planned to add. Also, pay a bit of attention on how you process watermarks from each input, and let me know if you need help. David On Fri, Aug 12, 2016 at 11:03 AM, Shunxin Lu wrote: > Hello there, > > I am planning to add join support in Windowed Operator, but need some > advice on how to start. > Currently I am thinking to add a new subclass inheriting > AbstractWindowedOperator and do all the work we need in that class (add > more input ports, do join accumulation, etc.), but I am experiencing some > difficulties doing so. Or should I directly change the codes in > AbstractWindowedOperator? > > Thanks, > Shunxin >
[GitHub] apex-core pull request #368: APEXCORE-448 made operator name available in th...
Github user asfgit closed the pull request at: https://github.com/apache/apex-core/pull/368 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXCORE-448) Make operator name available in OperatorContext
[ https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423279#comment-15423279 ] ASF GitHub Bot commented on APEXCORE-448: - Github user asfgit closed the pull request at: https://github.com/apache/apex-core/pull/368 > Make operator name available in OperatorContext > --- > > Key: APEXCORE-448 > URL: https://issues.apache.org/jira/browse/APEXCORE-448 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Chandni Singh >Assignee: Chandni Singh > > Need name of the logical operator in the OperatorContext which can be used by > WindowDataManager to create a unique path per logical operator . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Join Support
Also, regarding the difficulties you mentioned about a new subclass inheriting AbstractWindowedOperator, what specifically are they? David On Tue, Aug 16, 2016 at 12:31 PM, David Yan wrote: > Hi Shunxin, > > One problem with join support using WindowedOperator is that Apex operator > does not support variable number of ports so we might have to limit the > join operator to, say, 5 input ports. Implementing join support for > WindowedOperator should not be difficult, but might be a little messy > because we will need to have a watermark control port for each regular > input port, making it 10 total input ports if we support a maximum of 5 > join inputs. > > Please take a look at the JoinAccumulation template interface. That was > there for the future join support I planned to add. > > Also, pay a bit of attention on how you process watermarks from each > input, and let me know if you need help. > > David > > On Fri, Aug 12, 2016 at 11:03 AM, Shunxin Lu wrote: > >> Hello there, >> >> I am planning to add join support in Windowed Operator, but need some >> advice on how to start. >> Currently I am thinking to add a new subclass inheriting >> AbstractWindowedOperator and do all the work we need in that class (add >> more input ports, do join accumulation, etc.), but I am experiencing some >> difficulties doing so. Or should I directly change the codes in >> AbstractWindowedOperator? >> >> Thanks, >> Shunxin >> > >
[jira] [Created] (APEXCORE-507) Expose firstWindowMillis (the timestamp when the first window is generated) in the API
David Yan created APEXCORE-507: -- Summary: Expose firstWindowMillis (the timestamp when the first window is generated) in the API Key: APEXCORE-507 URL: https://issues.apache.org/jira/browse/APEXCORE-507 Project: Apache Apex Core Issue Type: New Feature Reporter: David Yan Assignee: David Yan This is useful if you want to get the timestamp when the application started, and is required if the user code wants to extract the exact timestamp from the window id. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Join Support
Hi David, Thanks for the reply. I think I will need to reconsider the whole situation again base on your input. The main problem that I had was, every input port has its own type, how can I write methods that can handle all of them? Thanks, Shunxin On Tue, Aug 16, 2016 at 12:49 PM, David Yan wrote: > Also, regarding the difficulties you mentioned about a new subclass > inheriting AbstractWindowedOperator, what specifically are they? > > David > > On Tue, Aug 16, 2016 at 12:31 PM, David Yan wrote: > > > Hi Shunxin, > > > > One problem with join support using WindowedOperator is that Apex > operator > > does not support variable number of ports so we might have to limit the > > join operator to, say, 5 input ports. Implementing join support for > > WindowedOperator should not be difficult, but might be a little messy > > because we will need to have a watermark control port for each regular > > input port, making it 10 total input ports if we support a maximum of 5 > > join inputs. > > > > Please take a look at the JoinAccumulation template interface. That was > > there for the future join support I planned to add. > > > > Also, pay a bit of attention on how you process watermarks from each > > input, and let me know if you need help. > > > > David > > > > On Fri, Aug 12, 2016 at 11:03 AM, Shunxin Lu > wrote: > > > >> Hello there, > >> > >> I am planning to add join support in Windowed Operator, but need some > >> advice on how to start. > >> Currently I am thinking to add a new subclass inheriting > >> AbstractWindowedOperator and do all the work we need in that class (add > >> more input ports, do join accumulation, etc.), but I am experiencing > some > >> difficulties doing so. Or should I directly change the codes in > >> AbstractWindowedOperator? > >> > >> Thanks, > >> Shunxin > >> > > > > >
[GitHub] apex-malhar pull request #324: Spillable Datastructures PR for review only
Github user ilooner closed the pull request at: https://github.com/apache/apex-malhar/pull/324 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Join Support
Hi Shunxin,
How about declaring the JoinWindowedOperator interface something like this:
public interface JoinWindowedOperator
extends WindowedOperator
{
void accumulateTuple2(Tuple.WindowedTuple tuple);
void accumulateTuple3(Tuple.WindowedTuple tuple);
void accumulateTuple4(Tuple.WindowedTuple tuple);
void accumulateTuple5(Tuple.WindowedTuple tuple);
void processWatermark2(ControlTuple.Watermark watermark);
void processWatermark3(ControlTuple.Watermark watermark);
void processWatermark4(ControlTuple.Watermark watermark);
void processWatermark5(ControlTuple.Watermark watermark);
}
then have the AbstractJoinWindowedOperator clared like this:
public abstract class AbstractJoinWindowedOperator
extends AbstractWindowedOperator
implements JoinWindowedOperator
{
...
}
David
On Tue, Aug 16, 2016 at 1:19 PM, Shunxin Lu wrote:
> Hi David,
>
> Thanks for the reply. I think I will need to reconsider the whole situation
> again base on your input.
> The main problem that I had was, every input port has its own type, how can
> I write methods that can handle all of them?
>
> Thanks,
> Shunxin
>
> On Tue, Aug 16, 2016 at 12:49 PM, David Yan wrote:
>
> > Also, regarding the difficulties you mentioned about a new subclass
> > inheriting AbstractWindowedOperator, what specifically are they?
> >
> > David
> >
> > On Tue, Aug 16, 2016 at 12:31 PM, David Yan
> wrote:
> >
> > > Hi Shunxin,
> > >
> > > One problem with join support using WindowedOperator is that Apex
> > operator
> > > does not support variable number of ports so we might have to limit the
> > > join operator to, say, 5 input ports. Implementing join support for
> > > WindowedOperator should not be difficult, but might be a little messy
> > > because we will need to have a watermark control port for each regular
> > > input port, making it 10 total input ports if we support a maximum of 5
> > > join inputs.
> > >
> > > Please take a look at the JoinAccumulation template interface. That was
> > > there for the future join support I planned to add.
> > >
> > > Also, pay a bit of attention on how you process watermarks from each
> > > input, and let me know if you need help.
> > >
> > > David
> > >
> > > On Fri, Aug 12, 2016 at 11:03 AM, Shunxin Lu
> > wrote:
> > >
> > >> Hello there,
> > >>
> > >> I am planning to add join support in Windowed Operator, but need some
> > >> advice on how to start.
> > >> Currently I am thinking to add a new subclass inheriting
> > >> AbstractWindowedOperator and do all the work we need in that class
> (add
> > >> more input ports, do join accumulation, etc.), but I am experiencing
> > some
> > >> difficulties doing so. Or should I directly change the codes in
> > >> AbstractWindowedOperator?
> > >>
> > >> Thanks,
> > >> Shunxin
> > >>
> > >
> > >
> >
>
Re: Join Support
Thanks David. That's very helpful! I will continue to work on that and let
you know once I encounter more problems.
On Tue, Aug 16, 2016 at 2:02 PM, David Yan wrote:
> Hi Shunxin,
>
> How about declaring the JoinWindowedOperator interface something like this:
>
> public interface JoinWindowedOperator InputT5>
> extends WindowedOperator
> {
> void accumulateTuple2(Tuple.WindowedTuple tuple);
> void accumulateTuple3(Tuple.WindowedTuple tuple);
> void accumulateTuple4(Tuple.WindowedTuple tuple);
> void accumulateTuple5(Tuple.WindowedTuple tuple);
>
> void processWatermark2(ControlTuple.Watermark watermark);
>
> void processWatermark3(ControlTuple.Watermark watermark);
>
> void processWatermark4(ControlTuple.Watermark watermark);
>
> void processWatermark5(ControlTuple.Watermark watermark);
>
> }
>
> then have the AbstractJoinWindowedOperator clared like this:
>
> public abstract class AbstractJoinWindowedOperator InputT3, InputT4, InputT5, OutputT, DataStorageT extends WindowedStorage,
> RetractionStorageT extends WindowedStorage, AccumulationT extends
> JoinAccumulation>
> extends AbstractWindowedOperator RetractionStorageT, AccumulationT>
> implements JoinWindowedOperator InputT5>
> {
> ...
> }
>
> David
>
> On Tue, Aug 16, 2016 at 1:19 PM, Shunxin Lu wrote:
>
> > Hi David,
> >
> > Thanks for the reply. I think I will need to reconsider the whole
> situation
> > again base on your input.
> > The main problem that I had was, every input port has its own type, how
> can
> > I write methods that can handle all of them?
> >
> > Thanks,
> > Shunxin
> >
> > On Tue, Aug 16, 2016 at 12:49 PM, David Yan
> wrote:
> >
> > > Also, regarding the difficulties you mentioned about a new subclass
> > > inheriting AbstractWindowedOperator, what specifically are they?
> > >
> > > David
> > >
> > > On Tue, Aug 16, 2016 at 12:31 PM, David Yan
> > wrote:
> > >
> > > > Hi Shunxin,
> > > >
> > > > One problem with join support using WindowedOperator is that Apex
> > > operator
> > > > does not support variable number of ports so we might have to limit
> the
> > > > join operator to, say, 5 input ports. Implementing join support for
> > > > WindowedOperator should not be difficult, but might be a little messy
> > > > because we will need to have a watermark control port for each
> regular
> > > > input port, making it 10 total input ports if we support a maximum
> of 5
> > > > join inputs.
> > > >
> > > > Please take a look at the JoinAccumulation template interface. That
> was
> > > > there for the future join support I planned to add.
> > > >
> > > > Also, pay a bit of attention on how you process watermarks from each
> > > > input, and let me know if you need help.
> > > >
> > > > David
> > > >
> > > > On Fri, Aug 12, 2016 at 11:03 AM, Shunxin Lu
> > > wrote:
> > > >
> > > >> Hello there,
> > > >>
> > > >> I am planning to add join support in Windowed Operator, but need
> some
> > > >> advice on how to start.
> > > >> Currently I am thinking to add a new subclass inheriting
> > > >> AbstractWindowedOperator and do all the work we need in that class
> > (add
> > > >> more input ports, do join accumulation, etc.), but I am experiencing
> > > some
> > > >> difficulties doing so. Or should I directly change the codes in
> > > >> AbstractWindowedOperator?
> > > >>
> > > >> Thanks,
> > > >> Shunxin
> > > >>
> > > >
> > > >
> > >
> >
>
Malhar Newbie / Beginner tasks
Hey, There is only one Newbie / Beginner task for Malhar. Are there any other tasks that could be labeled Newbie / Beginner? Thanks, Brendan Herger The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
[jira] [Assigned] (APEXCORE-502) Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
[
https://issues.apache.org/jira/browse/APEXCORE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-502:
---
Assignee: Vlad Rozov
> Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
> -
>
> Key: APEXCORE-502
> URL: https://issues.apache.org/jira/browse/APEXCORE-502
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>
> {noformat}
> slice = new Slice(os.toByteArray(), 0, os.toByteArray().length);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] apex-core pull request #370: APEXCORE-502 Unnecessary byte array copy in Def...
GitHub user vrozov opened a pull request: https://github.com/apache/apex-core/pull/370 APEXCORE-502 Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray @tweise Please review You can merge this pull request into a Git repository by running: $ git pull https://github.com/vrozov/apex-core APEXCORE-502 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-core/pull/370.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #370 commit c210dc212a8ac2ecd3d83ce21e72960f35c2e9db Author: Vlad Rozov Date: 2016-08-17T01:46:14Z APEXCORE-502 Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXCORE-502) Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
[
https://issues.apache.org/jira/browse/APEXCORE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423739#comment-15423739
]
ASF GitHub Bot commented on APEXCORE-502:
-
GitHub user vrozov opened a pull request:
https://github.com/apache/apex-core/pull/370
APEXCORE-502 Unnecessary byte array copy in
DefaultKryoStreamCodec.toByteArray
@tweise Please review
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vrozov/apex-core APEXCORE-502
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-core/pull/370.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #370
commit c210dc212a8ac2ecd3d83ce21e72960f35c2e9db
Author: Vlad Rozov
Date: 2016-08-17T01:46:14Z
APEXCORE-502 Unnecessary byte array copy in
DefaultKryoStreamCodec.toByteArray
> Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
> -
>
> Key: APEXCORE-502
> URL: https://issues.apache.org/jira/browse/APEXCORE-502
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>
> {noformat}
> slice = new Slice(os.toByteArray(), 0, os.toByteArray().length);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
