[GitHub] incubator-ariatosca pull request #15: ARIA-14 Implement initial engine tests

[dankilman]

GitHub user dankilman opened a pull request:

https://github.com/apache/incubator-ariatosca/pull/15

ARIA-14 Implement initial engine tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-ariatosca 
ARIA-14-workflow-engine-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-ariatosca/pull/15.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #15


commit f87d50aa2b2885782ce51a283609eb7342f9935a
Author: Dan Kilman 
Date:   2016-11-01T14:42:34Z

ARIA-14 Implement initial engine tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-ariatosca git commit: ARIA-14 Implement initial engine tests [Forced Update!]

[dankilman]

Repository: incubator-ariatosca
Updated Branches:
  refs/heads/ARIA-14-workflow-engine-tests d08040bd6 -> f87d50aa2 (forced 
update)


ARIA-14 Implement initial engine tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/commit/f87d50aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/tree/f87d50aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/diff/f87d50aa

Branch: refs/heads/ARIA-14-workflow-engine-tests
Commit: f87d50aa2b2885782ce51a283609eb7342f9935a
Parents: c0bf347
Author: Dan Kilman 
Authored: Tue Nov 1 16:42:34 2016 +0200
Committer: Dan Kilman 
Committed: Thu Nov 3 00:25:27 2016 +0200

--
 aria/contexts.py |   4 +-
 aria/events/__init__.py  |   1 +
 aria/events/builtin_event_handler.py |  15 ++-
 aria/storage/models.py   |   6 +-
 aria/tools/application.py|  10 +-
 aria/workflows/core/engine.py|  23 ++--
 aria/workflows/core/tasks.py |  33 +++--
 tests/.pylintrc  |   2 +-
 tests/workflows/test_engine.py   | 203 ++
 9 files changed, 268 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/f87d50aa/aria/contexts.py
--
diff --git a/aria/contexts.py b/aria/contexts.py
index ae7fc66..fdd26a2 100644
--- a/aria/contexts.py
+++ b/aria/contexts.py
@@ -201,11 +201,11 @@ class OperationContext(LoggerMixin):
 """
 The model operation
 """
-return self.storage.operation.get(self.id)
+return self.model.operation.get(self.id)
 
 @operation.setter
 def operation(self, value):
 """
 Store the operation in the model storage
 """
-self.storage.operation.store(value)
+self.model.operation.store(value)

http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/f87d50aa/aria/events/__init__.py
--
diff --git a/aria/events/__init__.py b/aria/events/__init__.py
index 6b07213..74f3e22 100644
--- a/aria/events/__init__.py
+++ b/aria/events/__init__.py
@@ -39,6 +39,7 @@ from blinker import signal
 from ..tools.plugin import plugin_installer
 
 # workflow engine task signals:
+sent_task_signal = signal('sent_task_signal')
 start_task_signal = signal('start_task_signal')
 on_success_task_signal = signal('success_task_signal')
 on_failure_task_signal = signal('failure_task_signal')

http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/f87d50aa/aria/events/builtin_event_handler.py
--
diff --git a/aria/events/builtin_event_handler.py 
b/aria/events/builtin_event_handler.py
index ec3238f..2dfbd00 100644
--- a/aria/events/builtin_event_handler.py
+++ b/aria/events/builtin_event_handler.py
@@ -27,12 +27,21 @@ from . import (
 start_workflow_signal,
 on_success_workflow_signal,
 on_failure_workflow_signal,
+sent_task_signal,
 start_task_signal,
 on_success_task_signal,
 on_failure_task_signal,
 )
 
 
+@sent_task_signal.connect
+def _task_sent(task, *args, **kwargs):
+operation_context = task.context
+operation = operation_context.operation
+operation.status = operation.SENT
+operation_context.operation = operation
+
+
 @start_task_signal.connect
 def _task_started(task, *args, **kwargs):
 operation_context = task.context
@@ -62,7 +71,7 @@ def _task_succeeded(task, *args, **kwargs):
 
 @start_workflow_signal.connect
 def _workflow_started(workflow_context, *args, **kwargs):
-execution_cls = workflow_context.storage.execution.model_cls
+execution_cls = workflow_context.model.execution.model_cls
 execution = execution_cls(
 id=workflow_context.execution_id,
 deployment_id=workflow_context.deployment_id,
@@ -80,7 +89,7 @@ def _workflow_failed(workflow_context, exception, *args, 
**kwargs):
 execution = workflow_context.execution
 execution.error = str(exception)
 execution.status = execution.FAILED
-execution.ended_at = datetime.utcnow(),
+execution.ended_at = datetime.utcnow()
 workflow_context.execution = execution
 
 
@@ -88,5 +97,5 @@ def _workflow_failed(workflow_context, exception, *args, 
**kwargs):
 def _workflow_succeeded(workflow_context, *args, **kwargs):
 execution = workflow_context.execution
 execution.status = execution.TERMINATED
-execution.ended_at = datetime.utcnow(),
+execution.ended_at = datetime.utcnow()
 workflow_context.execution = execution

Re: Podling Report Reminder - November 2016

[John D. Ament]

Please note - the board report is due tonight.

John

On Wed, Nov 2, 2016 at 1:42 PM Arthur Berezin  wrote:

> Thanks John, I'll update this soon.
>
> On Wed, Nov 2, 2016, 04:30 John D. Ament  wrote:
>
> > I created a template of the page here:
> >
> >
> https://cwiki.apache.org/confluence/display/ARIATOSCA/Podling+Report+2016-11
> >
> > Please fill in as appropriate :-)
> >
> > John
> >
> > On Sat, Oct 29, 2016 at 11:48 PM Suneel Marthi 
> > wrote:
> >
> > > Nope I did not do it, please go ahead and create it.
> > >
> > > Thanx
> > >
> > > Sent from my iPhone
> > >
> > > > On Oct 29, 2016, at 9:50 PM, Arthur Berezin 
> > > wrote:
> > > >
> > > > Suneel, I'll take care of this month's podling report.
> > > > Have you already created this in confluence?
> > > >
> > > >
> > > >
> > > > On Thu, Oct 27, 2016 at 2:52 AM Suneel Marthi <
> suneel.mar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Will do.
> > > >>
> > > >> On Wed, Oct 26, 2016 at 7:51 PM, John D. Ament <
> johndam...@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >>> Can the drafts be moved to confluence?
> > > >>>
> > >  On Oct 26, 2016 19:30, "Suneel Marthi" 
> wrote:
> > > 
> > >  Here's a Draft of November Podling Report, @Ran and others please
> > fill
> > > >>> in /
> > >  modify as you see fit.
> > > 
> > >  https://docs.google.com/document/d/1QqShfZUBf5Awzw_
> > >  cued-79kHAFjEtW5A6EM_8aHwlRI/edit?usp=sharing
> > > 
> > > 
> > > 
> > > > On Wed, Oct 26, 2016 at 7:01 PM,  wrote:
> > > >
> > > > Dear podling,
> > > >
> > > > This email was sent by an automated system on behalf of the
> Apache
> > > > Incubator PMC. It is an initial reminder to give you plenty of
> time
> > > >> to
> > > > prepare your quarterly board report.
> > > >
> > > > The board meeting is scheduled for Wed, 16 November 2016, 10:30
> am
> > > >> PDT.
> > > > The report for your podling will form a part of the Incubator PMC
> > > > report. The Incubator PMC requires your report to be submitted 2
> > > >> weeks
> > > > before the board meeting, to allow sufficient time for review and
> > > > submission (Wed, November 02).
> > > >
> > > > Please submit your report with sufficient time to allow the
> > Incubator
> > > > PMC, and subsequently board members to review and digest. Again,
> > the
> > > > very latest you should submit your report is 2 weeks prior to the
> > > >> board
> > > > meeting.
> > > >
> > > > Thanks,
> > > >
> > > > The Apache Incubator PMC
> > > >
> > > > Submitting your Report
> > > >
> > > > --
> > > >
> > > > Your report should contain the following:
> > > >
> > > > *   Your project name
> > > > *   A brief description of your project, which assumes no
> knowledge
> > > >> of
> > > >the project or necessarily of its field
> > > > *   A list of the three most important issues to address in the
> > move
> > > >towards graduation.
> > > > *   Any issues that the Incubator PMC or ASF Board might
> wish/need
> > to
> > > >>> be
> > > >aware of
> > > > *   How has the community developed since the last report
> > > > *   How has the project developed since the last report.
> > > >
> > > > This should be appended to the Incubator Wiki page at:
> > > >
> > > > http://wiki.apache.org/incubator/November2016
> > > >
> > > > Note: This is manually populated. You may need to wait a little
> > > >> before
> > > > this page is created from a template.
> > > >
> > > > Mentors
> > > > ---
> > > >
> > > > Mentors should review reports for their project(s) and sign them
> > off
> > > >> on
> > > > the Incubator wiki page. Signing off reports shows that you are
> > > > following the project - projects that are not signed may raise
> > alarms
> > > > for the Incubator PMC.
> > > >
> > > > Incubator PMC
> > > >
> > > 
> > > >>>
> > > >>
> > >
> >
>

incubator-ariatosca git commit: fixes

[dankilman]

Repository: incubator-ariatosca
Updated Branches:
  refs/heads/ARIA-14-workflow-engine-tests ff5098ca8 -> 1efdef505


fixes


Project: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/commit/1efdef50
Tree: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/tree/1efdef50
Diff: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/diff/1efdef50

Branch: refs/heads/ARIA-14-workflow-engine-tests
Commit: 1efdef505ee82fa609a594664d1b370f3fc15a89
Parents: ff5098c
Author: Dan Kilman 
Authored: Wed Nov 2 17:47:15 2016 +0200
Committer: Dan Kilman 
Committed: Wed Nov 2 17:47:15 2016 +0200

--
 aria/storage/models.py| 2 +-
 aria/workflows/core/engine.py | 3 ++-
 aria/workflows/core/tasks.py  | 4 ++--
 3 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/1efdef50/aria/storage/models.py
--
diff --git a/aria/storage/models.py b/aria/storage/models.py
index eb6b8e8..e5fc1ac 100644
--- a/aria/storage/models.py
+++ b/aria/storage/models.py
@@ -237,7 +237,7 @@ class Operation(Model):
 id = Field(type=basestring, default=uuid_generator)
 status = Field(type=basestring, choices=STATES, default=PENDING)
 execution_id = Field(type=basestring)
-eta = Field(type=datetime, default=0)
+eta = Field(type=datetime, default=datetime.now)
 started_at = Field(type=datetime, default=None)
 ended_at = Field(type=datetime, default=None)
 max_retries = Field(type=int, default=0)

http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/1efdef50/aria/workflows/core/engine.py
--
diff --git a/aria/workflows/core/engine.py b/aria/workflows/core/engine.py
index 7a2431c..5cd4604 100644
--- a/aria/workflows/core/engine.py
+++ b/aria/workflows/core/engine.py
@@ -18,6 +18,7 @@ The workflow engine. Executes workflows
 """
 
 import time
+from datetime import datetime
 
 import networkx
 
@@ -63,7 +64,7 @@ class Engine(logger.LoggerMixin):
 raise
 
 def _executable_tasks(self):
-now = time.time()
+now = datetime.now()
 return (task for task in self._tasks_iter()
 if task.status == models.Operation.PENDING and
 task.eta <= now and

http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/1efdef50/aria/workflows/core/tasks.py
--
diff --git a/aria/workflows/core/tasks.py b/aria/workflows/core/tasks.py
index 625802a..98d7c13 100644
--- a/aria/workflows/core/tasks.py
+++ b/aria/workflows/core/tasks.py
@@ -17,7 +17,7 @@
 Workflow tasks
 """
 
-import time
+from datetime import datetime
 
 from aria import logger
 from aria.storage import models
@@ -64,7 +64,7 @@ class BaseWorkflowTask(BaseTask):
 def __init__(self, *args, **kwargs):
 super(BaseWorkflowTask, self).__init__(*args, **kwargs)
 self.status = models.Operation.PENDING
-self.eta = time.time()
+self.eta = datetime.now()
 
 
 class StartWorkflowTask(BaseWorkflowTask):

incubator-ariatosca git commit: clean and add a test

[dankilman]

Repository: incubator-ariatosca
Updated Branches:
  refs/heads/ARIA-14-workflow-engine-tests 5a77c4716 -> ff5098ca8


clean and add a test


Project: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/commit/ff5098ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/tree/ff5098ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/diff/ff5098ca

Branch: refs/heads/ARIA-14-workflow-engine-tests
Commit: ff5098ca8e09f4a5a72d7b160d0c08902c282849
Parents: 5a77c47
Author: Dan Kilman 
Authored: Wed Nov 2 17:25:31 2016 +0200
Committer: Dan Kilman 
Committed: Wed Nov 2 17:25:31 2016 +0200

--
 aria/workflows/core/engine.py  |   3 +-
 tests/workflows/test_engine.py | 251 
 2 files changed, 138 insertions(+), 116 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/ff5098ca/aria/workflows/core/engine.py
--
diff --git a/aria/workflows/core/engine.py b/aria/workflows/core/engine.py
index 5853c61..7a2431c 100644
--- a/aria/workflows/core/engine.py
+++ b/aria/workflows/core/engine.py
@@ -36,10 +36,9 @@ class Engine(logger.LoggerMixin):
 def __init__(self, executor, workflow_context, tasks_graph, **kwargs):
 super(Engine, self).__init__(**kwargs)
 self._workflow_context = workflow_context
-self._tasks_graph = tasks_graph
 self._execution_graph = networkx.DiGraph()
 self._executor = executor
-translation.build_execution_graph(task_graph=self._tasks_graph,
+translation.build_execution_graph(task_graph=tasks_graph,
   workflow_context=workflow_context,
   
execution_graph=self._execution_graph)
 

http://git-wip-us.apache.org/repos/asf/incubator-ariatosca/blob/ff5098ca/tests/workflows/test_engine.py
--
diff --git a/tests/workflows/test_engine.py b/tests/workflows/test_engine.py
index 6b37e7f..2488264 100644
--- a/tests/workflows/test_engine.py
+++ b/tests/workflows/test_engine.py
@@ -29,98 +29,164 @@ from aria.workflows.core import engine
 logging.basicConfig()
 
 
-class TestEngine(object):
+global_test_holder = {}
+
 
-def test_empty_graph_execution(self, workflow_context):
-self.executor = thread.ThreadExecutor()
+class TestEngine(object):
 
+def test_empty_graph_execution(self, workflow_context, executor):
 @workflow
 def mock_workflow(context, graph):
-return graph
-
-graph = mock_workflow(context=workflow_context)
-eng = engine.Engine(executor=self.executor,
-workflow_context=workflow_context,
-tasks_graph=graph)
-eng.execute()
+pass
+self._execute(workflow_func=mock_workflow,
+  workflow_context=workflow_context,
+  executor=executor)
 assert workflow_context.states == ['start', 'success']
 
-def test_single_task_successful_execution(self, workflow_context):
-self.executor = thread.ThreadExecutor()
-
+def test_single_task_successful_execution(self, workflow_context, 
executor):
 @workflow
 def mock_workflow(context, graph):
-graph.add_task(context.operation(
-name='failing',
-operation_details={'operation': 
'tests.workflows.test_engine.{name}'.format(
-name=mock_success_task.__name__)},
-node_instance=None
-))
-return graph
-
-graph = mock_workflow(context=workflow_context)
-eng = engine.Engine(executor=self.executor,
-workflow_context=workflow_context,
-tasks_graph=graph)
-execution_tasks = [data['task'] for _, data in 
eng._execution_graph.nodes_iter(data=True)]
-eng.execute()
+graph.add_task(_op(mock_success_task, context))
+execution_tasks = self._execute(
+workflow_func=mock_workflow,
+workflow_context=workflow_context,
+executor=executor)
 assert workflow_context.states == ['start', 'success']
 assert any(hasattr(t, 'sent') and t.sent is True for t in 
execution_tasks)
 
-def test_single_task_failed_execution(self, workflow_context):
-self.executor = thread.ThreadExecutor()
-
+def test_single_task_failed_execution(self, workflow_context, executor):
 @workflow
 def mock_workflow(context, graph):
-graph.add_task(context.operation(
-name='failing',
-   

[jira] [Updated] (ARIATOSCA-10) Automatic-retry for failed tasks

[Ran Ziv (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ARIATOSCA-10?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ran Ziv updated ARIATOSCA-10:
-
Assignee: Dan Kilman

> Automatic-retry for failed tasks
> 
>
> Key: ARIATOSCA-10
> URL: https://issues.apache.org/jira/browse/ARIATOSCA-10
> Project: AriaTosca
>  Issue Type: Story
>Reporter: Ran Ziv
>Assignee: Dan Kilman
>
> Create an automatic-retry mechanism for failed tasks in the workflow engine



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)