FINAL REMINDER: CFP for ApacheCon closes February 11th
Dear Apache Enthusiast, This is your FINAL reminder that the Call for Papers (CFP) for ApacheCon Miami is closing this weekend - February 11th. This is your final opportunity to submit a talk for consideration at this event. This year, we are running several mini conferences in conjunction with the main event, so if you're submitting for one of those events, please pay attention to the instructions below. Apache: Big Data * Event information: http://events.linuxfoundation.org/events/apache-big-data-north-america * CFP: http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp Apache: IoT (Internet of Things) * Event Information: http://us.apacheiot.org/ * CFP - http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp (Indicate 'IoT' in the Target Audience field) CloudStack Collaboration Conference * Event information: http://us.cloudstackcollab.org/ * CFP - http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp (Indicate 'CloudStack' in the Target Audience field) FlexJS Summit * Event information - http://us.apacheflexjs.org/ * CFP - http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp (Indicate 'Flex' in the Target Audience field) TomcatCon * Event information - https://tomcat.apache.org/conference.html * CFP - http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp (Indicate 'Tomcat' in the Target Audience field) All other topics and projects * Event information - http://events.linuxfoundation.org/events/apachecon-north-america/program/about * CFP - http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp Admission to any of these events also grants you access to all of the others. Thanks, and we look forward to seeing you in Miami! -- Rich Bowen VP Conferences, Apache Software Foundation rbo...@apache.org Twitter: @apachecon (You are receiving this email because you are subscribed to a dev@ or users@ list of some Apache Software Foundation project. If you do not wish to receive email from these lists any more, you must follow that list's unsubscription procedure. View the headers of this message for unsubscription instructions.)
[jira] [Assigned] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-643: --- Assignee: Vlad Rozov (was: Sandesh) > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-643: Component/s: Buffer Server > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-643: Priority: Minor (was: Major) > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858207#comment-15858207 ] Vlad Rozov commented on APEXCORE-643: - [~sandesh] Please test against https://github.com/apache/apex-core/pull/436 > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-438: --- Assignee: Vlad Rozov (was: Sandesh) > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-438: Component/s: Buffer Server > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-438: Priority: Minor (was: Major) > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-643. - Resolution: Duplicate > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-642: Priority: Minor (was: Major) > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858243#comment-15858243 ] Vlad Rozov commented on APEXCORE-642: - [~sanjaypujare] Is not the exception already logged in StreamingAppMaster.main()? > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-558) Do not use yellow color to display command strings in help output
[ https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-558: --- Assignee: Sanjay M Pujare > Do not use yellow color to display command strings in help output > - > > Key: APEXCORE-558 > URL: https://issues.apache.org/jira/browse/APEXCORE-558 > Project: Apache Apex Core > Issue Type: Bug > Environment: MacOS, Linux >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI Help output (at least on MacOS and Linux terminals) shows command > strings in yellow which is extremely hard to read and reduces usability of > the CLI. Should be changed to red or purple or just be bolded in black. > In the following output "alias" and "begin-macro" are in yellow and hard to > read. > apex> help > GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN: > alias alias-name command > Create a command alias > begin-macro name > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] apex-core pull request #468: SPOI-10695: get-app-package-operators with pare...
GitHub user sgolovko reopened a pull request: https://github.com/apache/apex-core/pull/468 SPOI-10695: get-app-package-operators with parent option does not work There are two apex command line operators that have the option "-parent" (get-app-package-operators and get-jar-operator-classes). And both of them have the mandatory argument . The implementation of the class GetOperatorClassesCommandLineOptions that builds command line parser options created the descriptor of the option "-parent" without an argument. And as the result, the argument of the option was ignored and it was moved to the list of the regular command line parameters. The bug fix is trivial. It defines the option "-parent" as an option with an argument. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sgolovko/apex-core master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-core/pull/468.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #468 commit 5220a3b7a4de24357d13c753067efae3c23ab9d5 Author: Sergey Golovko Date: 2017-02-08T04:00:02Z SPOI-10695: get-app-package-operators with parent option does not work There are two apex command line operators that have the option "-parent" (get-app-package-operators and get-jar-operator-classes). And both of them have the mandatory argument . The implementation of the class GetOperatorClassesCommandLineOptions that builds command line parser options created the descriptor of the option "-parent" without an argument. And as the result, the argument of the option was ignored and it was moved to the list of the regular command line parameters. The bug fix is trivial. It defines the option "-parent" as an option with an argument. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (APEXCORE-631) Improving BufferServer tests
[ https://issues.apache.org/jira/browse/APEXCORE-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh updated APEXCORE-631: - Component/s: Buffer Server > Improving BufferServer tests > > > Key: APEXCORE-631 > URL: https://issues.apache.org/jira/browse/APEXCORE-631 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-633) Add unit tests for DataList
[ https://issues.apache.org/jira/browse/APEXCORE-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh updated APEXCORE-633: - Component/s: Buffer Server > Add unit tests for DataList > --- > > Key: APEXCORE-633 > URL: https://issues.apache.org/jira/browse/APEXCORE-633 > Project: Apache Apex Core > Issue Type: Sub-task > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-632) Fix the existing unit test and move them to JUnit from testNg
[ https://issues.apache.org/jira/browse/APEXCORE-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh updated APEXCORE-632: - Component/s: Buffer Server > Fix the existing unit test and move them to JUnit from testNg > - > > Key: APEXCORE-632 > URL: https://issues.apache.org/jira/browse/APEXCORE-632 > Project: Apache Apex Core > Issue Type: Sub-task > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-613) Fix/Remove the LogicalNode's javadoc.
[ https://issues.apache.org/jira/browse/APEXCORE-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandesh updated APEXCORE-613: - Component/s: Buffer Server > Fix/Remove the LogicalNode's javadoc. > - > > Key: APEXCORE-613 > URL: https://issues.apache.org/jira/browse/APEXCORE-613 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > > LogicalNode doesn't represent Logical Node in the DAG anymore, it represents > a physical instance of the operator. Update the JavaDoc accordingly. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] apex-core pull request #468: SPOI-10695: get-app-package-operators with pare...
Github user sgolovko closed the pull request at: https://github.com/apache/apex-core/pull/468 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858551#comment-15858551 ] Sanjay M Pujare commented on APEXCORE-642: -- It is logged to the log file (dt.log) but no StramEvent is created. But you have a good point - in the same place in main() we can also raise the StramEvent (assuming the init required for StramEvent has been done) which the event monitoring systems can catch and display to aid with debugging > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
Sanjay M Pujare created APEXMALHAR-2403: --- Summary: Finalization is not invoked for AbstractFileOutputOperator Key: APEXMALHAR-2403 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 Project: Apache Apex Malhar Issue Type: Bug Reporter: Sanjay M Pujare Priority: Minor I used an example/template program for Database to HDFS data copy. While copying the table, I see "temp" files created on the output HDFS side which I expected to be "finalized" to non-temp names when the app is shut down. But that did not happen. AbstractFileOutputOperator and its subclasses should do proper finalization via the deactivate() method when an app is properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858779#comment-15858779 ] Sanjay M Pujare commented on APEXMALHAR-2403: - [~devendra] [~chaithu] pls let me know if you need more info > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858779#comment-15858779 ] Sanjay M Pujare edited comment on APEXMALHAR-2403 at 2/9/17 12:41 AM: -- [~devendra] [~chaithu] [~sandesh] pls let me know if you need more info or have comments was (Author: sanjaypujare): [~devendra] [~chaithu] pls let me know if you need more info > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXMALHAR-2404) Avro file input operator should call beginWindow of AbstractFileInputOperator
Sandesh created APEXMALHAR-2404: --- Summary: Avro file input operator should call beginWindow of AbstractFileInputOperator Key: APEXMALHAR-2404 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2404 Project: Apache Apex Malhar Issue Type: Bug Reporter: Sandesh Assignee: devendra tagare Avro file input operator is atmost once as it is not calling beginWindow of abstractFileInputOperator and recovery happens only in AbstractFileInputOperator -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858835#comment-15858835 ] Thomas Weise commented on APEXMALHAR-2403: -- This has to do with data, not with operator deployment. The correct way to handle it is a control tuple / watermark that indicates completeness of input. > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858851#comment-15858851 ] Sanjay M Pujare commented on APEXMALHAR-2403: - That is or cannot be done here. The user has decided to shut down the application (thru Apex CLI) because he has noticed that the app has finished reading the whole SQL table and has determined that no more records are going to be added. Shouldn't the shutdown command tell the FileOutoutOperator to finalize the files i.e. rename from temp to actual names? This is an example of a batch application where the completion of batch is determined externally (not thru EOF on the input) and it will be good to support such use cases. > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858851#comment-15858851 ] Sanjay M Pujare edited comment on APEXMALHAR-2403 at 2/9/17 1:58 AM: - That is (or can) not be done here. The user has decided to shut down the application (thru Apex CLI) because he has noticed that the app has finished reading the whole SQL table and has determined that no more records are going to be added. Shouldn't the shutdown command tell the FileOutoutOperator to finalize the files i.e. rename from temp to actual names? This is an example of a batch application where the completion of batch is determined externally (not thru EOF on the input) and it will be good to support such use cases. was (Author: sanjaypujare): That is or cannot be done here. The user has decided to shut down the application (thru Apex CLI) because he has noticed that the app has finished reading the whole SQL table and has determined that no more records are going to be added. Shouldn't the shutdown command tell the FileOutoutOperator to finalize the files i.e. rename from temp to actual names? This is an example of a batch application where the completion of batch is determined externally (not thru EOF on the input) and it will be good to support such use cases. > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858859#comment-15858859 ] Thomas Weise commented on APEXMALHAR-2403: -- It does not matter how it is determined, the point is that this should be a control tuple that is handled consistent with all other data. I think similar use cases were already discussed in that context. > Finalization is not invoked for AbstractFileOutputOperator > - > > Key: APEXMALHAR-2403 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sanjay M Pujare >Priority: Minor > > I used an example/template program for Database to HDFS data copy. While > copying the table, I see "temp" files created on the output HDFS side which I > expected to be "finalized" to non-temp names when the app is shut down. But > that did not happen. AbstractFileOutputOperator and its subclasses > should do proper finalization via the deactivate() method when an app is > properly shut down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858863#comment-15858863 ] Sanjay M Pujare commented on APEXCORE-642: -- Looking at the code, currently it is easy (trivial) to do it in StreamingAppMasterService.serviceInit() rather than in StreamingAppMaster.main() because serviceInit can call StreamingContainerManager.recordEventAsync(StramEvent) on this.dnmgr whereas StreamingAppMaster doesn't have access to dnmgr in StreamingContainerManager (unless we change it from private). > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXMALHAR-2404) Avro file input operator should call beginWindow of AbstractFileInputOperator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859011#comment-15859011 ] devendra tagare commented on APEXMALHAR-2404: - Good catch. Will fix it. Thanks, Dev > Avro file input operator should call beginWindow of AbstractFileInputOperator > - > > Key: APEXMALHAR-2404 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2404 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Sandesh >Assignee: devendra tagare > > Avro file input operator is atmost once as it is not calling beginWindow of > abstractFileInputOperator and recovery happens only in > AbstractFileInputOperator -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-644) get-app-package-operators with parent option does not work
Yatin Chaubal created APEXCORE-644:
--
Summary: get-app-package-operators with parent option does not work
Key: APEXCORE-644
URL: https://issues.apache.org/jira/browse/APEXCORE-644
Project: Apache Apex Core
Issue Type: Bug
Affects Versions: 3.5.0
Reporter: Yatin Chaubal
Issue: get-app-package-operators with -parent option doesnot work
Steps:
1) Start dtcli/apex
2) Run get-app-package-operators -parent com.datatorrent.demos.pi
/home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
Expected out output: valid JSON
Actual output: com.datatorrent.stram.cli.ApexCli$CliException:
/home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
at
com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
Reference:
Without -parent option this work fine
apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
{
"operatorClasses": [
{
"name": "com.datatorrent.common.util.DefaultDelayOperator",
"properties": [],
"portTypeInfo": [
{
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-644:
---
Assignee: Sergey Golovko
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output: com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
