[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188807#comment-15188807 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55640357 --- Diff: library/src/test/java/com/datatorrent/lib/io/fs/FileSplitterInputTest.java --- @@ -110,8 +110,7 @@ protected void starting(

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188800#comment-15188800 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188797#comment-15188797 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55640205 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55640120 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188795#comment-15188795 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55640050 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188794#comment-15188794 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639986 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188792#comment-15188792 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639847 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to th

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188790#comment-15188790 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639806 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to th

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188789#comment-15188789 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639760 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to th

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188776#comment-15188776 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188775#comment-15188775 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639379 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to th

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread DT-Priyanka
Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639373 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java --- @@ -384,7 +399,14 @@ protected void scanIterationComp

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188772#comment-15188772 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639221 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to t

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188769#comment-15188769 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188767#comment-15188767 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639110 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to t

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55639045 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to t

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55638987 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to t

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188765#comment-15188765 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188766#comment-15188766 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chinmaykolhatkar

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chinmaykolhatkar
Github user chinmaykolhatkar commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55638951 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to t

Re: Projection Operator in Malhar

2016-03-09 Thread Bhupesh Chawda
+1 for the Projection operator and for the name. One question: Will the two output POJO classes be user supplied? Or will they be dynamically generated? In other words, will the user, in addition to specifying the list of selected and dropped fields from the incoming POJO, also define the correspo

Re: Projection Operator in Malhar

2016-03-09 Thread Mohit Jotwani
+1 Regards, Mohit On Thu, Mar 10, 2016 at 11:07 AM, Yogi Devendra wrote: > @Priyanka > > Projection is standard term from relational algebra > https://en.wikipedia.org/wiki/Projection_(relational_algebra) > > +1 for the name projection. > > ~ Yogi > > On 10 March 2016 at 11:01, Priyanka Gugale

Re: HDFS file copy module for Malhar

2016-03-09 Thread Chaitanya Chebolu
+1 We can extend this design and will support other file systems like S3, FTP, etc. On Thu, Mar 10, 2016 at 10:10 AM, Priyanka Gugale wrote: > +1, this would be a very helpful module. > > -Priyanka > On Mar 10, 2016 9:16 AM, "Mohit Jotwani" wrote: > > > +1 > > > > Regards, > > Mohit > > On 9 Ma

Re: Projection Operator in Malhar

2016-03-09 Thread Yogi Devendra
@Priyanka Projection is standard term from relational algebra https://en.wikipedia.org/wiki/Projection_(relational_algebra) +1 for the name projection. ~ Yogi On 10 March 2016 at 11:01, Priyanka Gugale wrote: > +1. > > This name is not self explanatory, I can't think of anything better as of

RE: Operator ID Overlap

2016-03-09 Thread Ganelin, Ilya
I just realized on second glance that I do have a naming conflict! Thanks! Sent with Good (www.good.com) From: Tushar Gosavi Sent: Thursday, March 10, 2016 12:07:33 AM To: dev@apex.incubator.apache.org Subject: Re: Operator ID Overlap Hi, >From the stack trace

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188704#comment-15188704 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user yogidevendra com

Re: Projection Operator in Malhar

2016-03-09 Thread Priyanka Gugale
+1. This name is not self explanatory, I can't think of anything better as of now. If someone have suggestions please add it here. I will also try to find some options. -Priyanka On Thu, Mar 10, 2016 at 10:17 AM, Sandeep Deshmukh wrote: > + 1 on projection w/o adding or updating fields. IMO, t

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread yogidevendra
Github user yogidevendra commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55635731 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apa

Re: Operator ID Overlap

2016-03-09 Thread Tushar Gosavi
Hi, >From the stack trace it seems that you are adding two or more operator in the DAG using same name (Records_HDHT). Adding multiple operators to logical DAG using same name is not allowed. Can you check if this is the case, else there might be the problem in the platform. - Tushar. On Thu, M

Re: Get partition ID

2016-03-09 Thread Thomas Weise
They are unique but you should not make assumptions about the sequence. On Wed, Mar 9, 2016 at 9:01 PM, Sandeep Deshmukh wrote: > By default, are the IDs returned by getId() method sequential like 0..(N-1) > where there are N instances of the operator or these IDs are some unique > numbers? > >

Re: Get partition ID

2016-03-09 Thread Sandeep Deshmukh
By default, are the IDs returned by getId() method sequential like 0..(N-1) where there are N instances of the operator or these IDs are some unique numbers? Regards, Sandeep On Thu, Mar 10, 2016 at 4:41 AM, Thomas Weise wrote: > You can also assign a custom ID or other any other per-partition

Re: Projection Operator in Malhar

2016-03-09 Thread Sandeep Deshmukh
+ 1 on projection w/o adding or updating fields. IMO, the purpose is to split the record into something that is useful and something that is not. Regards, Sandeep On Wed, Mar 9, 2016 at 9:08 PM, Chinmay Kolhatkar wrote: > +1 for projection operator. > > @Sandesh, Adding of new fields and updati

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread sandeepdeshmukh
Github user sandeepdeshmukh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55633512 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188667#comment-15188667 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user sandeepdeshmukh

Re: HDFS file copy module for Malhar

2016-03-09 Thread Priyanka Gugale
+1, this would be a very helpful module. -Priyanka On Mar 10, 2016 9:16 AM, "Mohit Jotwani" wrote: > +1 > > Regards, > Mohit > On 9 Mar 2016 21:09, "Chinmay Kolhatkar" wrote: > > > +1. > > > > On Wed, Mar 9, 2016 at 8:38 PM, Yogi Devendra > > wrote: > > > > > Hi, > > > > > > I mentioned earlie

Re: HDFS file copy module for Malhar

2016-03-09 Thread Mohit Jotwani
+1 Regards, Mohit On 9 Mar 2016 21:09, "Chinmay Kolhatkar" wrote: > +1. > > On Wed, Mar 9, 2016 at 8:38 PM, Yogi Devendra > wrote: > > > Hi, > > > > I mentioned earlier here, > > > > > http://mail-archives.apache.org/mod_mbox/apex-dev/201602.mbox/%3CCAHekGF9xNa6qvvt4ySGBC4SmCN7_Hn2r9rj2SQSV%2BE

Re: Operator ID Overlap

2016-03-09 Thread Ganelin, Ilya
Additional note: I am extracting operatorId from the context in setup: @Override public void setup(Context.OperatorContext context) { super.setup(context); operatorId = context.getId(); operatorUniquePath = new Path(getOutputFileName() + idDelim + operatorId + ".txt").toSt

Operator ID Overlap

2016-03-09 Thread Ganelin, Ilya
Hi all – I’ve created some helper functions to efficiently create HDFS Output operators (which require common configurations). However, I run into a duplicate operator ID error when attempting to run this, error and code are below: Why would the DAG be assigning a duplicate operator ID? For the

[GitHub] incubator-apex-core pull request: Use artifactId for the git prope...

2016-03-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-core/pull/267 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

[GitHub] incubator-apex-core pull request: Use artifactId for the git prope...

2016-03-09 Thread tweise
GitHub user tweise opened a pull request: https://github.com/apache/incubator-apex-core/pull/267 Use artifactId for the git properties resource. @davidyan74 please merge You can merge this pull request into a Git repository by running: $ git pull https://github.com/tweise/incub

[GitHub] incubator-apex-core pull request: Make the version retrieval reusa...

2016-03-09 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-core/pull/266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

[GitHub] incubator-apex-core pull request: Make the version retrieval reusa...

2016-03-09 Thread tweise
Github user tweise commented on the pull request: https://github.com/apache/incubator-apex-core/pull/266#issuecomment-194581862 @davidyan74 Fixed it, please check again when build is complete. --- If your project is set up for it, you can reply to this email and have your reply appea

[GitHub] incubator-apex-core pull request: Make the version retrieval reusa...

2016-03-09 Thread davidyan74
Github user davidyan74 commented on a diff in the pull request: https://github.com/apache/incubator-apex-core/pull/266#discussion_r55612060 --- Diff: engine/src/main/java/com/datatorrent/stram/util/VersionInfo.java --- @@ -53,16 +54,16 @@ Attributes mainAttribs = mf.ge

[GitHub] incubator-apex-core pull request: Make the version retrieval reusa...

2016-03-09 Thread tweise
GitHub user tweise opened a pull request: https://github.com/apache/incubator-apex-core/pull/266 Make the version retrieval reusable. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tweise/incubator-apex-core master Alternativel

Re: Get partition ID

2016-03-09 Thread Thomas Weise
You can also assign a custom ID or other any other per-partition properties in the partitioner. On Wed, Mar 9, 2016 at 3:09 PM, Ashwin Chandra Putta < ashwinchand...@gmail.com> wrote: > Use the getId() method in OperatorContext. > > > https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/C

Re: Get partition ID

2016-03-09 Thread Ashwin Chandra Putta
Use the getId() method in OperatorContext. https://www.datatorrent.com/docs/apidocs/com/datatorrent/api/Context.OperatorContext.html#getId() Regards, Ashwin. On Wed, Mar 9, 2016 at 3:05 PM, Ganelin, Ilya wrote: > Given that any single logical operator may have multiple physical > partitions, i

Get partition ID

2016-03-09 Thread Ganelin, Ilya
Given that any single logical operator may have multiple physical partitions, is there any way for an operator to know something akin to a partition ID? E.g. part0, part1, etc. The information contained in this e-mail is confidential and/o

Re: Projection Operator in Malhar

2016-03-09 Thread Chinmay Kolhatkar
+1 for projection operator. @Sandesh, Adding of new fields and updating will be done as a part of Enricher and Transform operator respectively. I've sent another mail about Transform operator before. On Wed, Mar 9, 2016 at 8:54 PM, Sandesh Hegde wrote: > +1 > > Adding of new fields and updatin

Re: HDFS file copy module for Malhar

2016-03-09 Thread Chinmay Kolhatkar
+1. On Wed, Mar 9, 2016 at 8:38 PM, Yogi Devendra wrote: > Hi, > > I mentioned earlier here, > > http://mail-archives.apache.org/mod_mbox/apex-dev/201602.mbox/%3CCAHekGF9xNa6qvvt4ySGBC4SmCN7_Hn2r9rj2SQSV%2BE1Cc5A0fQ%40mail.gmail.com%3E > > I am proposing HDFS file copy module. > JIRA created for

Re: Projection Operator in Malhar

2016-03-09 Thread Sandesh Hegde
+1 Adding of new fields and updating of fields can be part of this? On Wed, Mar 9, 2016, 5:50 AM Pradeep A. Dalvi wrote: > Dear Folks, > > We are creating Projection Operator which will allow apex users to project > (select/drop) certain fields from the incoming tuples. This operation might > b

HDFS file copy module for Malhar

2016-03-09 Thread Yogi Devendra
Hi, I mentioned earlier here, http://mail-archives.apache.org/mod_mbox/apex-dev/201602.mbox/%3CCAHekGF9xNa6qvvt4ySGBC4SmCN7_Hn2r9rj2SQSV%2BE1Cc5A0fQ%40mail.gmail.com%3E I am proposing HDFS file copy module. JIRA created for this work is available here : https://issues.apache.org/jira/browse/APEXM

[jira] [Updated] (APEXMALHAR-2013) HDFS output module for file copy

2016-03-09 Thread Yogi Devendra (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yogi Devendra updated APEXMALHAR-2013: -- Description: To write files to HDFS using block-by-block approach. Main use-case be

[jira] [Created] (APEXMALHAR-2013) HDFS output module for file copy

2016-03-09 Thread Yogi Devendra (JIRA)
Yogi Devendra created APEXMALHAR-2013: - Summary: HDFS output module for file copy Key: APEXMALHAR-2013 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2013 Project: Apache Apex Malhar

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187156#comment-15187156 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user yogidevendra com

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread yogidevendra
Github user yogidevendra commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55525483 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apa

Projection Operator in Malhar

2016-03-09 Thread Pradeep A. Dalvi
Dear Folks, We are creating Projection Operator which will allow apex users to project (select/drop) certain fields from the incoming tuples. This operation might be done unconditionally or based on certain condition. Use case: - Not all fields of tuples are of interest for the downst

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187034#comment-15187034 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chaithu14 commen

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chaithu14
Github user chaithu14 commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55512047 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,236 @@ +/** + * Licensed to the Apache

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187031#comment-15187031 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user sandeepdeshmukh

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread sandeepdeshmukh
Github user sandeepdeshmukh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55511842 --- Diff: library/src/test/java/com/datatorrent/lib/io/fs/FileSplitterInputTest.java --- @@ -215,7 +213,7 @@ public void testTimeScan(

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187028#comment-15187028 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user sandeepdeshmukh

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread sandeepdeshmukh
Github user sandeepdeshmukh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55511718 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java --- @@ -384,7 +399,14 @@ protected void scanIteration

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187021#comment-15187021 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chaithu14 commen

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chaithu14
Github user chaithu14 commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55511575 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockMetadata.java --- @@ -0,0 +1,63 @@ +/** + * Licensed to the Ap

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187022#comment-15187022 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user chaithu14 commen

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread chaithu14
Github user chaithu14 commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55511560 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockMetadata.java --- @@ -0,0 +1,63 @@ +/** + * Licensed to the Ap

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187020#comment-15187020 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user sandeepdeshmukh

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread sandeepdeshmukh
Github user sandeepdeshmukh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55511509 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/HDFSBlockReader.java --- @@ -0,0 +1,78 @@ +/** + * Licensed to th

Re: Avro to POJO & POJO to Avro transformation operators

2016-03-09 Thread Chinmay Kolhatkar
+1 On Wed, Mar 9, 2016 at 2:17 PM, Mohit Jotwani wrote: > +1 > > Regards, > Mohit > > On Wed, Mar 9, 2016 at 1:37 PM, Devendra Tagare > > wrote: > > > Hi, > > > > I am working on developing a record converter which would take a POJO as > an > > input and emit a Generic record as the output base

Re: Avro to POJO & POJO to Avro transformation operators

2016-03-09 Thread Mohit Jotwani
+1 Regards, Mohit On Wed, Mar 9, 2016 at 1:37 PM, Devendra Tagare wrote: > Hi, > > I am working on developing a record converter which would take a POJO as an > input and emit a Generic record as the output based on the given Avro > schema. > > > JIRA : https://issues.apache.org/jira/browse/APE

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

2016-03-09 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186748#comment-15186748 ] ASF GitHub Bot commented on APEXMALHAR-2008: GitHub user DT-Priyanka open

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

2016-03-09 Thread DT-Priyanka
GitHub user DT-Priyanka opened a pull request: https://github.com/apache/incubator-apex-malhar/pull/207 APEXMALHAR-2008: Create HDFS File Reader module Code to add HDFS file reader module. 1. The module reads file/list of files (directory is also accepted) and emit the file blo

Avro to POJO & POJO to Avro transformation operators

2016-03-09 Thread Devendra Tagare
Hi, I am working on developing a record converter which would take a POJO as an input and emit a Generic record as the output based on the given Avro schema. JIRA : https://issues.apache.org/jira/browse/APEXMALHAR-2011 Additionally, I would develop a record converter which would take an Avro Ge