[ANNOUNCE] New Apache Apex Committer: Ashish Tadose

The Project Management Committee (PPMC) for Apache Apex has asked Ashish Tadose to become a committer and we are pleased to announce that he has accepted. Ashish contributed the Apache Geode (incubating) integration and presented at meetups and recently at the Geode Summit. Ashish is affiliated wi

[GitHub] incubator-apex-site pull request: resources.md

Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-site/pull/18 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

Re: Stack overflow errors when launching job

Ilya, Realized it is a StackOverflow so increasing the memory size may not help. So maybe increasing the stack size may help: The attribute to specify the JVM options is CONTAINER_JVM_OPTIONS The setting for stack size is -Xss Chandni On Fri, Mar 11, 2016 at 3:46 PM, Chandni Singh wrote: > He

Re: Stack overflow errors when launching job

Hey Ilya, Can you please remove the duplicate output port from your implementation of NewLineFileInputOperator as well. Thanks, Chandni On Fri, Mar 11, 2016 at 3:42 PM, Ashwin Chandra Putta < ashwinchand...@gmail.com> wrote: > Why do you want to have thread locality and partitioning together, y

Re: Stack overflow errors when launching job

Why do you want to have thread locality and partitioning together, you will lose parallel processing. What is the use case? Regards, Ashwin. On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya wrote: > Now with files: > https://gist.github.com/ilganeli/7f770374113b40ffa18a > > From: "Ganelin, Ilya"

[GitHub] incubator-apex-site pull request: resources.md

Github user sashadt commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55904530 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26

[GitHub] incubator-apex-site pull request: resources.md

Github user tweise commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55904534 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26/

[GitHub] incubator-apex-site pull request: resources.md

Github user sashadt commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55904332 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26

[GitHub] incubator-apex-site pull request: resources.md

Github user sashadt commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55904543 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26

[GitHub] incubator-apex-site pull request: resources.md

Github user sashadt commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55904465 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26

Re: Stack overflow errors when launching job

The DAG attribute to do so is MASTER_MEMORY_MB and by default it is 1GB. Can you please increase it? On Fri, Mar 11, 2016 at 3:18 PM, Chandni Singh wrote: > Hey Ilya, > > Can you please assign more memory to the App Master and check? > > Chandni > > On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ily

Re: Stack overflow errors when launching job

Hey Ilya, Can you please assign more memory to the App Master and check? Chandni On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya wrote: > Now with files: > https://gist.github.com/ilganeli/7f770374113b40ffa18a > > From: "Ganelin, Ilya" ilya.gane...@capitalone.com>> > Reply-To: "dev@apex.incuba

[GitHub] incubator-apex-site pull request: resources.md

Github user sandeshh commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55903316 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/2

Re: Stack overflow errors when launching job

Now with files: https://gist.github.com/ilganeli/7f770374113b40ffa18a From: "Ganelin, Ilya" mailto:ilya.gane...@capitalone.com>> Reply-To: "dev@apex.incubator.apache.org" mailto:dev@apex.incubator.apache.org>> Date: Friday, March 11, 2016 at 3:02 PM To: "dev

[GitHub] incubator-apex-site pull request: resources.md

Github user tweise commented on a diff in the pull request: https://github.com/apache/incubator-apex-site/pull/18#discussion_r55902290 --- Diff: src/md/resources.md --- @@ -0,0 +1,71 @@ + +## Presentations + +- Ilya Genelin - "Next gen decision making < 2ms" - 02/26/

Stack overflow errors when launching job

Hi guys – I’m running into a very frustrating issue where certain DAG configurations cause the following error log (attached). When this happens, my application even fails to launch. This does not seem to be a YARN issue since this occurs even with a relatively small number of partitions/memory.

RE: Long-running HDFS Write errors

If I see this error again then I will do so. I've been running many jobs. Thanks. Sent with Good (www.good.com) From: Chandni Singh Sent: Friday, March 11, 2016 4:04:43 PM To: dev@apex.incubator.apache.org Subject: Re: Long-running HDFS Write errors Hi Ilya,

Re: Long-running HDFS Write errors

Hi Ilya, Can you please share the log files for this container? Is the log level set to 'DEBUG'? Thanks, Chandni On Fri, Mar 11, 2016 at 8:57 AM, Chaitanya Chebolu < chaita...@datatorrent.com> wrote: > I think rolling is not happening and this depends on "rollingFile" > property. > By defaul

[GitHub] incubator-apex-malhar pull request: Apexcore 293.migrate docs.v3

Github user asfgit closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/208 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-apex-site pull request: resources.md

Github user sandeshh commented on the pull request: https://github.com/apache/incubator-apex-site/pull/18#issuecomment-195513810 @sashadt please review. This page is essentially *everything under the sun - apex* page. Only selected Apex blogs are added. --- If your project i

[GitHub] incubator-apex-site pull request: resources.md

GitHub user sandeshh opened a pull request: https://github.com/apache/incubator-apex-site/pull/18 resources.md You can merge this pull request into a Git repository by running: $ git pull https://github.com/sandeshh/incubator-apex-site patch-1 Alternatively you can review and

Re: Long-running HDFS Write errors

I think rolling is not happening and this depends on "rollingFile" property. By default, value of rollingFile = false. Property "rollingFile" is true only if one of the below condition satisfies: - maxLength < Long.MAX_VALUE - rotationWindows > 0. Please check by setting one of the above pr

Re: Long-running HDFS Write errors

This is happening after some time but file roll-over appears to be working well with this approach in other instances. On 3/11/16, 8:02 AM, "Sandeep Deshmukh" wrote: >Is this happening for the first itself or after some time? > >May be the file is getting rolled over to the next file but as

Re: Long-running HDFS Write errors

Is this happening for the first itself or after some time? May be the file is getting rolled over to the next file but as you are overriding the default file naming policy, the rollover is also trying to write to the same file. Regards, Sandeep On Fri, Mar 11, 2016 at 9:21 PM, Ganelin, Ilya wro

Re: Long-running HDFS Write errors

I explicitly assign a different name for each partition of the operator as well based on the context ID. On 3/11/16, 7:34 AM, "Sandeep Deshmukh" wrote: >The AbstractFileOutputOperator creates file with timestamp in the file >name. So, if there is conflict in the name prompts that the same op

Re: Long-running HDFS Write errors

The AbstractFileOutputOperator creates file with timestamp in the file name. So, if there is conflict in the name prompts that the same operator could be trying to write to same file. Does this happen after operator recovery or before any other failure occurs? Is it possible that multiple partitio

RE: Long-running HDFS Write errors

Multiple partitions do write to the same directory but they write to different files in that directory. I didn't see other failures in the log - as an aside, is it possible to increase the length of the log in the DT UI itself? Sent with Good (www.good.com) Fr

Re: Long-running HDFS Write errors

Does this happen after operator recovery or before any other failure occurs? Is it possible that multiple partitions write to the same directory? On Fri, Mar 11, 2016 at 7:12 AM, Ganelin, Ilya wrote: > This is 3.0.0. > > > > Sent with Good (www.good.com) > > F

RE: Long-running HDFS Write errors

This is 3.0.0. Sent with Good (www.good.com) From: Thomas Weise Sent: Friday, March 11, 2016 2:02:13 AM To: dev@apex.incubator.apache.org Subject: Re: Long-running HDFS Write errors Which version of Malhar is this? On Thu, Mar 10, 2016 at 10:56 PM, Ganelin, I

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-1985: Using startRo...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/182#discussion_r55825610 --- Diff: contrib/src/test/java/com/datatorrent/contrib/cassandra/CassandraOperatorTest.java --- @@ -333,26 +333,26 @@ public void testCa

[jira] [Commented] (APEXMALHAR-1985) Cassandra Input Oeprator: startRow set incorrectly

[ https://issues.apache.org/jira/browse/APEXMALHAR-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190877#comment-15190877 ] ASF GitHub Bot commented on APEXMALHAR-1985: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-1985: Using startRo...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/182#discussion_r55825428 --- Diff: contrib/src/test/java/com/datatorrent/contrib/cassandra/CassandraOperatorTest.java --- @@ -333,26 +333,26 @@ public void testCa

[jira] [Commented] (APEXMALHAR-1985) Cassandra Input Oeprator: startRow set incorrectly

[ https://issues.apache.org/jira/browse/APEXMALHAR-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190874#comment-15190874 ] ASF GitHub Bot commented on APEXMALHAR-1985: Github user DT-Priyanka comm

[jira] [Commented] (APEXMALHAR-1985) Cassandra Input Oeprator: startRow set incorrectly

[ https://issues.apache.org/jira/browse/APEXMALHAR-1985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190875#comment-15190875 ] ASF GitHub Bot commented on APEXMALHAR-1985: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-1985: Using startRo...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/182#discussion_r55825355 --- Diff: contrib/src/main/java/com/datatorrent/contrib/cassandra/CassandraPOJOInputOperator.java --- @@ -359,24 +373,43 @@ public void e

Re: Adding features to HBase Input Operators in Malhar-contrib

Hi All, In the current design of HBase input and output operators, the row key is hard-coded to be of String type. I foresee the following issue: - In case of numeric keys which are type casted to String, *incremental read* is problematic. For example, after reading key = 9, we may not be

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2004: Add file's mo...

Github user tushargosavi commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/203#discussion_r55806762 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/FileSplitterInput.java --- @@ -439,12 +439,12 @@ protected ScannedFileInfo crea

[jira] [Commented] (APEXMALHAR-2004) TimeBasedDirectoryScanner keep reading same file

[ https://issues.apache.org/jira/browse/APEXMALHAR-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190697#comment-15190697 ] ASF GitHub Bot commented on APEXMALHAR-2004: Github user tushargosavi com

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190641#comment-15190641 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55801007 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,253 @@ +/** + * Licensed to the Apac

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190618#comment-15190618 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190619#comment-15190619 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55800213 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to the Apa

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55800249 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSInputModule.java --- @@ -0,0 +1,253 @@ +/** + * Licensed to the Apac

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55800146 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,180 @@ +/** + * Licensed to the Apa

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190617#comment-15190617 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190614#comment-15190614 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55800097 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,184 @@ +/** + * Licensed to the Apa

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190611#comment-15190611 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55799956 --- Diff: library/src/main/java/com/datatorrent/lib/io/fs/HDFSFileSplitter.java --- @@ -0,0 +1,184 @@ +/** + * Licensed to the Apa

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190608#comment-15190608 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[jira] [Commented] (APEXMALHAR-2008) Create hdfs file input module

[ https://issues.apache.org/jira/browse/APEXMALHAR-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190606#comment-15190606 ] ASF GitHub Bot commented on APEXMALHAR-2008: Github user DT-Priyanka comm

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55799890 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/BlockReader.java --- @@ -0,0 +1,81 @@ +/** + * Licensed to the Apache

[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2008: Create HDFS F...

Github user DT-Priyanka commented on a diff in the pull request: https://github.com/apache/incubator-apex-malhar/pull/207#discussion_r55799843 --- Diff: library/src/main/java/com/datatorrent/lib/io/block/BlockReader.java --- @@ -0,0 +1,81 @@ +/** + * Licensed to the Apache