Boy, this sure escalated quickly.
Long ago we had similar discussions (for months on end) and decided on the
Project Lifecycle hierarchy which is the following:
- A "Project Lifecycle" which has "Project Lifecycle Environments"
- A "Project lifecycle Environment" which is linked to a single "Project".
The terminology was picked from a standard definition on Wikipedia
somewhere.
Environments and Projects were exposed in the configuration and user
interface. Lifecycles themselves are not yet exposed.
The terminology matters. You can find elsewhere on the Internet that folks
call these environments "project phases" and some other variants but
that's not what we picked and it's not accurate either I think.
A lot of environments are IMO not part of a phase but often serve
completely separate purposes.
A practical example of a Lifecyle would be a project A with a development,
test and production environment.
These 3 form the lifecycle of project A. The name is on any level intended
to be something you can choose and that is the case right now.
That's all this is really. Please note that this is a separate thing from
the physical reality of these concepts.
The project lifecycle could be somehow physically represented on the same
machine, server, container, ... but likely not.
This is not to say that this could not be the case in the future.
Now you can mean "Project Lifecyle" when you say "Project" but that's not
what we mean in Hop right now and actually it's kind of inaccurate I think.
So when Peter says that he might want to have a single name represent
A/Development and A/Test on a single machine and call it "A" then what
we're talking about is the implementation of the Project Lifecycle.
Typical tools that come with this are comparisons, diff extraction (for
operators), promotion of artifacts from one environment to the next (a
project lifecycle typically has an order) and so on.
So far we haven't had that use-case yet and so we haven't focussed on this
but that doesn't mean that class ProjectLifecycle doesn't exist :-)
In the same way I think that some people simply didn't understand the
(quite simple) hierarchy despite our best attempts at explaining and
documenting this.
Cheers,
Matt
On Tue, Nov 2, 2021 at 10:52 PM Brandon Jackson<[email protected]>
wrote:
I wanted to put a perspective out there on environments. What I think we
would want to end up with are small collections of environments.
For example, at Neo Corp, perhaps they have a single collection for their
company consisting of
1) Production
2) Development
3) Test
For a consultant type user, they work with multiple corporate entities
and
have a collection each.
Nord Ltd. may have
1) Production
2) Test
Taho US Inc.
1) Prod
2) Dev
3) Test
etc.
What each user would want to do is have the ability to tell an ETL
project
which collection and specific environment profile to use.
It would probably be good to separate out the environment support and
config files to hop/config/environment/(collection name)/(supporting flat
files or configs).
There could be a special folder hop/config/environment/relations.json
which
could keep track of client project / collection ties. This means you
could
ship projects with environment collections separately, but the Hop UI
could
parse and manage some of these relationships separate of the environment
and projects themselves.
The above would allow Hop to scan for these things.
One other requirement is I would want some way to scan and report on a
project to understand where all the environmental tentacles are touching.
If I take any project right now, variables could be used or consumed
anywhere. Only design best practices lead one to have a tendency of
defining and using variables in particular places.
It would be cool if Matt's fun reporting project might highlight those
variables in a color, especially for variables with the same name.
(patterned usage).
Hope this contributes positively to the direction of decision-making.
Another thing that would make Hop infinitely more useful is the return of
syntax highlighting in steps. It is hard to overstate how difficult it
is
to write any kind of script in Hop at the moment. I have had to resort
to
outside tools for preparing and formatting scripts and queries. A prior
ETL tool spoiled me.
On Tue, Nov 2, 2021 at 3:19 AM Hans Van Akelyen <
[email protected]>
wrote:
Hi Hop enthusiasts, developers and philosophers
I have had an interesting discussion with Sergio on Jira ticket
HOP-3471
[1] but we would like to get a bit of a broader consensus.
So the main question is: do we see an environment as something linked
to
a
single project or are they objects that can be reused for several
projects?
Both approaches have pro's and con's but we need to agree on this as it
would change user experience and maybe how they are stored.
Currently an environment is a semi-standalone object, it is not linked
to a
project in the hop-project.json but it is linked to a project in the
hop-config.json. In the GUI all environments are shown, not only the
environments linked to a project. When using hop-run I *think* you can
point to an environment that is not linked to the project in the
hop-config.json.
My personal preference would be to unlink them completely and have
environments that could be used by multiple projects. This gives
greater
flexibility when you would want to divide your projects on a lower
level.
con would be that you would always see all available environments
defined
in your hop-config.json.
I would love for some opinions on the matter so it can be set in stone
once
and for all. And we can update our docs to reflect this decision
[2][3].
Cheers,
Hans
[1]https://issues.apache.org/jira/browse/HOP-3471
[2]
https://hop.apache.org/manual/latest/projects/projects-environments.html#top
[3]
https://hop.apache.org/manual/latest/projects/index.html#_environments
--
Neo4j Chief Solutions Architect
*✉ *[email protected]
