Re: Aligning FileSplitter and BlocReader with hadoop.mapreduce InputFormats

2016-03-24 Thread Shubham Pathak
+1 . Will certainly be a good addition. On Fri, Mar 25, 2016 at 9:31 AM, Chandni Singh wrote: > +1 for the idea > On Mar 24, 2016 8:41 PM, "Thomas Weise" wrote: > > > +1 for the idea in general and extending existing implementation. > > > > In case this introduces a MapReduce dependency we will

Re: Aligning FileSplitter and BlocReader with hadoop.mapreduce InputFormats

2016-03-24 Thread Chandni Singh
+1 for the idea On Mar 24, 2016 8:41 PM, "Thomas Weise" wrote: > +1 for the idea in general and extending existing implementation. > > In case this introduces a MapReduce dependency we will also need to > consider a separate module. > > Thomas > > > On Thu, Mar 24, 2016 at 2:35 AM, Devendra Tagar

Re: Aligning FileSplitter and BlocReader with hadoop.mapreduce InputFormats

2016-03-24 Thread Thomas Weise
+1 for the idea in general and extending existing implementation. In case this introduces a MapReduce dependency we will also need to consider a separate module. Thomas On Thu, Mar 24, 2016 at 2:35 AM, Devendra Tagare wrote: > Hi, > > We are thinking of extending the FileSplitter and BlockRea

RE: [VOTE] Graduate Apex from the Incubator

2016-03-24 Thread York, Brennon
+1 (binding) From: Gaurav Gupta Sent: Thursday, March 24, 2016 12:57:55 PM To: dev@apex.incubator.apache.org Subject: Re: [VOTE] Graduate Apex from the Incubator +1 On Tue, Mar 22, 2016 at 10:36 AM, Pramod Immaneni wrote: > Dear Community, > > Based on the pos

[jira] [Created] (APEXMALHAR-2027) Add Timer Scheduler

2016-03-24 Thread Timothy Farkas (JIRA)
Timothy Farkas created APEXMALHAR-2027: -- Summary: Add Timer Scheduler Key: APEXMALHAR-2027 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2027 Project: Apache Apex Malhar Issue Ty

[jira] [Created] (APEXMALHAR-2026) Spooled Datastructures

2016-03-24 Thread Timothy Farkas (JIRA)
Timothy Farkas created APEXMALHAR-2026: -- Summary: Spooled Datastructures Key: APEXMALHAR-2026 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2026 Project: Apache Apex Malhar Issue

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2025 Move FileLineI...

2016-03-24 Thread chandnisingh
Github user chandnisingh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/220#discussion_r57414100 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/LineByLineFileInputOperator.java --- @@ -0,0 +1,55 @@ +package com.datator

[jira] [Commented] (APEXMALHAR-2025) Move FileLineInputOperator out of AbstractFileInputOperator

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211326#comment-15211326 ] ASF GitHub Bot commented on APEXMALHAR-2025: Github user chandnisingh com

[jira] [Commented] (APEXMALHAR-2025) Move FileLineInputOperator out of AbstractFileInputOperator

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211321#comment-15211321 ] ASF GitHub Bot commented on APEXMALHAR-2025: Github user chandnisingh com

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2025 Move FileLineI...

2016-03-24 Thread chandnisingh
Github user chandnisingh commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/220#discussion_r57413886 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/LineByLineFileInputOperator.java --- @@ -0,0 +1,55 @@ +package com.datator

[jira] [Commented] (APEXMALHAR-2025) Move FileLineInputOperator out of AbstractFileInputOperator

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211251#comment-15211251 ] ASF GitHub Bot commented on APEXMALHAR-2025: GitHub user PramodSSImmaneni

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2025 Move FileLineI...

2016-03-24 Thread PramodSSImmaneni
GitHub user PramodSSImmaneni opened a pull request: https://github.com/apache/incubator-apex-malhar/pull/220 APEXMALHAR-2025 Move FileLineInputOperator out of AbstractFileInputOperator @chandnisingh please review You can merge this pull request into a Git repository by running:

[jira] [Commented] (APEXMALHAR-2011) POJO to Avro record converter

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211064#comment-15211064 ] ASF GitHub Bot commented on APEXMALHAR-2011: GitHub user devtagare reopen

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2011-2012 Avro to P...

2016-03-24 Thread devtagare
GitHub user devtagare reopened a pull request: https://github.com/apache/incubator-apex-malhar/pull/211 APEXMALHAR-2011-2012 Avro to Pojo & Pojo to Avro converter Please review changes for Avro to Pojo conversion & back. Thanks, Dev You can merge this pull request into

[jira] [Resolved] (APEXCORE-406) Instructions in PodlingNameSearch comments have not yet been followed

2016-03-24 Thread Sasha Parfenov (JIRA)
[ https://issues.apache.org/jira/browse/APEXCORE-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sasha Parfenov resolved APEXCORE-406. - Resolution: Fixed Added trademark symbols in prominent places. http://apex.incubator.ap

[jira] [Commented] (APEXMALHAR-2011) POJO to Avro record converter

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211020#comment-15211020 ] ASF GitHub Bot commented on APEXMALHAR-2011: Github user devtagare closed

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2011-2012 Avro to P...

2016-03-24 Thread devtagare
Github user devtagare closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/211 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if t

[jira] [Commented] (APEXMALHAR-2011) POJO to Avro record converter

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210981#comment-15210981 ] ASF GitHub Bot commented on APEXMALHAR-2011: GitHub user devtagare reopen

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2011-2012 Avro to P...

2016-03-24 Thread devtagare
GitHub user devtagare reopened a pull request: https://github.com/apache/incubator-apex-malhar/pull/211 APEXMALHAR-2011-2012 Avro to Pojo & Pojo to Avro converter Please review changes for Avro to Pojo conversion & back. Thanks, Dev You can merge this pull request into

[jira] [Commented] (APEXMALHAR-2011) POJO to Avro record converter

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210979#comment-15210979 ] ASF GitHub Bot commented on APEXMALHAR-2011: Github user devtagare closed

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2011-2012 Avro to P...

2016-03-24 Thread devtagare
Github user devtagare closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/211 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if t

Re: Adaptive SPIN_MILLIS attributes

2016-03-24 Thread Amol Kekre
This is a good way, and it fits well with platform self-adjusting. We may need to still look at a way to allow users to keep SPIN_MILLIS fixed in case they depend on it for downstream intake. Thks Amol On Thu, Mar 24, 2016 at 12:45 PM, Sandesh Hegde wrote: > That is cool, we need to something

Re: [VOTE] Graduate Apex from the Incubator

2016-03-24 Thread Gaurav Gupta
+1 On Tue, Mar 22, 2016 at 10:36 AM, Pramod Immaneni wrote: > Dear Community, > > Based on the positive discussion, I'm calling for the community VOTE > to graduate Apache Apex (incubating) to top level project (TLP). > > Please vote on the resolution pasted below. > > [ ] +1 Graduate Apache Ape

Re: Adaptive SPIN_MILLIS attributes

2016-03-24 Thread Sandesh Hegde
That is cool, we need to something similar for input operators. On Thu, Mar 24, 2016 at 12:32 PM Thomas Weise wrote: > Please see: > > https://issues.apache.org/jira/browse/APEXCORE-380 > https://github.com/apache/incubator-apex-core/pull/271 > > > On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde

Re: Adaptive SPIN_MILLIS attributes

2016-03-24 Thread Thomas Weise
Please see: https://issues.apache.org/jira/browse/APEXCORE-380 https://github.com/apache/incubator-apex-core/pull/271 On Thu, Mar 24, 2016 at 12:27 PM, Sandesh Hegde wrote: > Hello Team, > > The attributes SPIN_MILLIS is the amount of time the input operator sleeps > while there are no inputs

Adaptive SPIN_MILLIS attributes

2016-03-24 Thread Sandesh Hegde
Hello Team, The attributes SPIN_MILLIS is the amount of time the input operator sleeps while there are no inputs to process and it's default value is 10ms. Recently while I was doing performance experiments, reducing the SPIN_MILLIS to 1ms increased the number of tuples processed by the input ope

Re: Contributing to Apache Apex

2016-03-24 Thread Siyuan Hua
Rodrigo, You are very welcome to the community. We definitely want to have some CEP model supported. Translate a CEP model explained here( https://people.cs.umass.edu/~yanlei/publications/sase-sigmod08.pdf) or Oracle CEP to Apex DAG would make many developers' life easier to use Apex Please feel f

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2013 : HDFS output ...

2016-03-24 Thread yogidevendra
Github user yogidevendra closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/216 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or i

[jira] [Commented] (APEXMALHAR-2013) HDFS output module for file copy

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210445#comment-15210445 ] ASF GitHub Bot commented on APEXMALHAR-2013: Github user yogidevendra clo

[jira] [Commented] (APEXMALHAR-2013) HDFS output module for file copy

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210446#comment-15210446 ] ASF GitHub Bot commented on APEXMALHAR-2013: GitHub user yogidevendra reo

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2013 : HDFS output ...

2016-03-24 Thread yogidevendra
GitHub user yogidevendra reopened a pull request: https://github.com/apache/incubator-apex-malhar/pull/216 APEXMALHAR-2013 : HDFS output module for file copy 1. Added operators required for file copy 2. Added junit tests 3. Added sample app for HDFS to HDFS file copy app You

[jira] [Commented] (APEXCORE-406) Instructions in PodlingNameSearch comments have not yet been followed

2016-03-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/APEXCORE-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210430#comment-15210430 ] Thomas Weise commented on APEXCORE-406: --- {quote} Please be sure to update at least

[jira] [Updated] (APEXCORE-406) Instructions in PodlingNameSearch comments have not yet been followed

2016-03-24 Thread Thomas Weise (JIRA)
[ https://issues.apache.org/jira/browse/APEXCORE-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated APEXCORE-406: -- Assignee: Sasha Parfenov > Instructions in PodlingNameSearch comments have not yet been follow

[jira] [Commented] (APEXMALHAR-1897) Create ManagedState

2016-03-24 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/APEXMALHAR-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210416#comment-15210416 ] ASF GitHub Bot commented on APEXMALHAR-1897: Github user amberarrow comme

Re: Contributing to Apache Apex

2016-03-24 Thread Rodrigo Abreu
Thanks Amol for the directions! I'm excited to join the Apex community. I look forward to coding with you guys. Thanks, Rodrigo Abreu. On Thu, Mar 24, 2016 at 2:47 AM Amol Kekre wrote: > Rodrigo, > First and foremost welcome aboard. It is great to have someone of your > calibre join our commu

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-1897 added managed ...

2016-03-24 Thread amberarrow
Github user amberarrow commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/145#discussion_r57335712 --- Diff: library/src/main/java/com/datatorrent/lib/state/managed/AbstractManagedStateImpl.java --- @@ -0,0 +1,499 @@ +/** + * Lic

[jira] [Updated] (APEXCORE-406) Instructions in PodlingNameSearch comments have not yet been followed

2016-03-24 Thread Hitesh Shah (JIRA)
[ https://issues.apache.org/jira/browse/APEXCORE-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated APEXCORE-406: - Priority: Critical (was: Blocker) > Instructions in PodlingNameSearch comments have not yet bee

[jira] [Created] (APEXCORE-406) Instructions in PodlingNameSearch comments have not yet been followed

2016-03-24 Thread Hitesh Shah (JIRA)
Hitesh Shah created APEXCORE-406: Summary: Instructions in PodlingNameSearch comments have not yet been followed Key: APEXCORE-406 URL: https://issues.apache.org/jira/browse/APEXCORE-406 Project: Apa

Re: Adding features to HBase Input Operators in Malhar-contrib

2016-03-24 Thread Bhupesh Chawda
Dear Community, Can anyone help review the pull request: https://github.com/apache/incubator-apex-malhar/pull/212 Thanks. ~Bhupesh On Thu, Mar 17, 2016 at 4:16 PM, Bhupesh Chawda wrote: > Hi, > > I have opened a pull request for the changes as described in the previous > emails. Here is the p

Re: Aligning FileSplitter and BlocReader with hadoop.mapreduce InputFormats

2016-03-24 Thread Devendra Tagare
Hi, We are thinking of extending the FileSplitter and BlockReader . Changing the existing code could have side effects. Thanks, Dev On Mar 24, 2016 1:16 AM, "Tushar Gosavi" wrote: > My suggestion is to extend from FileSplitter and BlockReader without > changing them, and add support for InputFo

Re: S3 Input Module

2016-03-24 Thread Chaitanya Chebolu
@Ashwin: Please find the below comments: 1) No. 2) Yes. All the blocks of a file read the same reader instance in the case of Hadoop version < 2.6 If we won't copy s3a files then missing the parallel reader functionality for Hadoop version < 2.6. Regards, Chaitanya On Thu, Mar 24, 2016 at 12

Re: Aligning FileSplitter and BlocReader with hadoop.mapreduce InputFormats

2016-03-24 Thread Tushar Gosavi
My suggestion is to extend from FileSplitter and BlockReader without changing them, and add support for InputFormat in derived classes. FileSplitter and BlockReader already provides enough hooks to define splits and read records. - Tushar. On Thu, Mar 24, 2016 at 11:17 AM, Yogi Devendra wrote: