I didn't expect submodules to be a part of the Swift 4 discussion. When it
came up I was pleasantly surprised. I have been thinking about the design of a
submodule system for quite a while but was planning to wait until it was
clearly in scope to draft a proposal. Now that the topic has been introduced I
decide to write down the design I've been thinking about, along with the
motivation and goals that underly it. I understand submodules may not be in
scope for Swift 4 but wanted to contribute this design while the discussion is
fresh in everyone's mind.
I am including the contents of the proposal below. You can also find it on
Github:
https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md
<https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md>
I am very much looking forward to everyone's feedback!
Matthew
Scope-based submodules
Proposal: SE-NNNN
<file:///Users/Matthew/Dropbox/Matthew/Development/__notes/__swift/evolution/NNNN-scope-based-submodules.md>
Authors: Matthew Johnson <https://github.com/anandabits>
Review Manager: TBD
Status: Awaiting review
Introduction
This proposal describes a submodule system based on the principle of strictly
nested scopes. The design strives to follow the Swift philosophy of offering
good defaults while progressively disclosing very powerful tools that can be
used by experts to solve complex problems.
Motivation
Swift currently provides two kinds of entities that provide system boundaries
without directly introducing a symbol: modules and files*.
Modules introduce an ABI boundary, a name boundary, and a scope boundary.
Files introduce a scope boundary that carries no additional semantics.
Swift currently lacks the ability to introduce a name and scope boundary
without also introducing an ABI boundary. Such a boundary would be naturally
situated halfway between in terms of strength and ceremony. The lack of such a
boundary significantly limits our abiltiy to structure a large Swift program.
Introducing a way to form this kind of boundary will provide a powerful tool
for giving internal structure to a module.
*The important aspect of a file in Swift is the logical scope boundary it
introduces. The physical file system representation is incidental to this. The
appendix on file system independence discusses this in more detail.
Goals
Any discussion of submodules inevitably reveals that there are very different
perspectives of what a submodule is, what problems submodules should be able to
solve, etc. This section describes the design goals of this proposal in order
to facilitate evaluation of both the goals themselves as well as how well the
solution accomplishes those goals.
The primary goal of this proposal are to introduce a unit of encapsulation
within a module that is larger than a file as a means of adding explicit
structure to a large program. All other goals are subordinate to this goal and
should be considered in light of it.
Some other goals of this proposal are:
Submodules should help us to manage and understand the internal dependencies of
a large, complex system.
Submodules should be able to collaborate with peer submodules without
necessarily being exposed to the rest of the module.
A module should not be required to expose its internal submodule structure to
users when symbols are exported.
It should be possible to extract a submodule from existing code with minimal
friction. The only difficulty should be breaking any circular dependencies.
Some additional non-functional requirements for the solution are:
Submodules should not negatively impact runtime performance. WMO should be able
to see across submodule boundaries.
Submodules should not negatively impact build performance. Ideally they will
improve build performance by giving the compiler more high-level information
about internal dependencies.
Deferred goal:
It is not an immediate goal to support submodules in single-file scripts. The
appendix discussing file system independence discusses some ideas that could be
used to support single-file scripts in the future.
Proposed solution
There are several relatively orthogonal aspects to the design of a submodule
system. A design must answer the following questions:
How is code placed in a submodule?
How are symbols in one submodule made available to another submodule or module?
How do submodules interact with access control?
This proposal diverages a little bit from the usual proposal format to
faciliate discussion of alternatives within the context of each aspect of the
chosen design. In each case an alternative could be substituted without
compromising the overall design.
Note: This proposal uses the term “top-level submodule” to distinguish the
scope of code that is not explicitly placed in a submodule from the module as a
whole. The top-level submodule is in most respects identical to any other
parent submodule. There are two differences: 1) it is the only submodule that
can export symbols outside of the module and 2) any open or public symbols it
declares are automatically exported according to their access modifier.
Placing code in a submodule
Each file is part of the top level submodule by default.
A submodule declaration may be used at the top of a file:
Only the first non-comment, non-whitespace line may contain a submodule
declaration.
The submodule decalaration looks like this: submodule MySubmoduleName
A submodule declaration may not be prefixed with the module name.
Submodule names form a hierarchical path:
The fully qualified name of the submodule specified by Submodule.InnerSubmodule
is: MyModuleName.Submodule.InnerSubmodule.
In this example, InnerSubmodule is a child of Submodule.
A submodule may not have the same name as any of its ancestors. This follows
the rule used by types.
Submodules may not be extended. They form strictly nested scopes.
The only way to place code in a submodule is with a submodule declaration at
the top of a file.
All code in a file exists in a single submodule.
A module is made up of strictly nested scoped that look like this:
The hierarchy of nested scopes in scope-based submodules
Alternatives
Grouping mechanisms
There are several other ways to specify which submodule the top-level scope of
a file is in. All of these alternatives share a crucial problem: you can’t tell
what submodule your code is in by looking at the file.
The alternatives are:
Use a manifest file. This would be painful to maintain.
Use file system paths. This is too tightly coupled to physical organization.
Appendix A discusses file system independence in more detail.
Leave this up to the build system. This makes it more difficult for a module to
support multiple build systems.
Require all files to include a submodule declaration
We could require all files to include an explicit submodule declaration.
However, that would be a breaking change and would violate the principle of
progressive disclosure. Users should not need to know about submodules if they
don’t use them.
Allow submodule references to explicitly state the name of the module
The module name is implicit throughout the scope of the entire module.
Specifying it explicitly is redundant. Prohibiting explicit mention of the
module name offers more flexibility for combining submodules into build
products.
Visibility of submodules
The export statement is used to modifiy the visibiltiy of a submodule within
the module. It is also used by the top-level module to publish submodules to
clients of the module.
All submodules are implicitly exported with module-wide visibility by default
(and hidden outside of the module by default*).
All submodules are implicitly available for export outside the module.
A submodule may use an explicit export statement to modify the visibility of a
descendent submodule.
export statements are only allowed at the top level of a file.
*The exception to this is that open and public symbols in the top-level
submodule are always exported exactly as declared.
Top-level export
All export statements consist of an access modifier, the export keyword, and a
submodule name:
open export ChildSubmodule
When this export statement appears in the top-level submodule, ChildSubmodule
becomes available for import by clients of the submodule with the fully
qualified name Module.ChildSubmodule. public exports are also available. When
public is used all published symbols in the exported submodule have a maximum
visiblity of publicregardless of how they were declared.
Top-level public and open export statement may be modified with the following
options:
A submodule may be published under a different external name using the export
as NewName syntax*.
@implicit causes symbols from the submodule to be implicitly imported when the
module is imported.
@inline causes the symbols from the submodule to appear as if they had been
declared directly within the top-level submodule.
@implicit may be combined with renaming but @inline may not appear along with
either of them.
*When a submodule is renamed for export with the as clause its internal name
does not change. A submodule always has the same fully qualfied name everywhere
within its module.
Here are some example export statements:
// All symbols in `Child1` are available for import by clients as `Module.Foo`
// These symbols are *not* imported automatically when a client imports the
module with `import Module`
public export Child1 as Foo
// All symbols in `Child2` are available for explicit import by clients as
`Module.Child2`
// The symbols are also automatically imported when a client imports the module
with `import Module`
// When the symbols are imported implicitly they retain the fully qualified
name prefix of `Module.Child2`
@implicit open export Child2
// All symbols in `Child3` are available for explicit import by clients as
`Module.Foo.Bar`
// The symbols are also automatically imported when a client imports the module
with `import Module`
// When the symbols are imported implicitly they retain the fully qualified
name prefix of `Module.Foo.Bar`
@implicit open export Child3 as Foo.Bar
// All symbols in `Child4.Grandchild` appear to clients as if they had been
declared
// directly in the top-level submodule.
// If the process of inlining the symbols produces duplicate symbols a compiler
error is produced
// at the site of one or both of the `export` statements.
@inline public export Child4.Grandchild
// All symbols in `Child5.Grandchild` are available for explicit import by
clients as `Module.Foo`
// along with the symbols declared in `Child`.
// The symbols are also automatically imported when a client imports the module
with `import Module`
// When the symbols are imported implicitly they retain the fully qualified
name prefix of `Module.Foo`
// As with `@inline`, when two submodules are given the same external name a
duplicate symbol error may occur.
@implicit public export Child5.Grandchild as Foo
One interesting observation is that both Child1 and Child5.Grandchild are
renamed to Foo. The symbols declared former is not implicitly imported by
import Module but the latter is, despite having the same fully qualified name
prefix.
Exports within the module
A submodule may bound the maximum visibility of any of its descendent
submodules by explicitly exporting it:
// `Child1.Grandchild` may be exported by the top-level module, but only with
`public` visibility.
// `Child1.Grandchild` may not be exported as `open`.
public export Child1.Grandchild
// `Child2` is exported with `internal` visibility.
// Because `internal` is scoped to the submodule level *only* the parent
submodule can see `Child2`.
// No submodules except the direct parent of `Child2` (the current submodule)
are allowed to `import Child2`.
// This also implies that the `Child2` may not be exported to clients because
the top-level
// submodule is not able to see or reference `Child2` at all.
internal export Child2
The access modifier may specify a scope internal or greater.
Only the direct parent of a submodule may specify the internal modifier. A
grandparent cannot hide a grandchild from its parent.
If a descendent includes an export statement for the same submodule, the access
modifier must be no greater than the access modifier specified by the
descendent. An ancestor may provide a tighter bound to visibility but may not
increase visibility. An attempt to increase visibility results in an error.
Note: If a submodule is not visible none of its descendents is visible either.
Alternatives
Use import access modifiers to export a submodule
The semantics of placing a bound on the visibility of a descendent submodule is
significantly different than the semantics of importing symbols from a
submodule into the current lexical scope. Mixing the semantics of the two is
confusing.
Restrict the visibility of a submodule to its parent unless the parent
explicitly exports it.
Users should be able to use submodules without needing to place export
statements in every parent submodule. Module-wide default visibility for
submodules is analagous to internal default visibility for symbols.
Require all submodules to be visible module-wide.
This removes an important tool for bounded collaboration within a complex
system. A parent submodule should be allowed to have a child submodule(s) which
are implementation details of the parent and not exposed to the rest of the
module.
Allow renaming to be used by export statements within the module.
A submodule should have the same fully qualified name everywhere it is used
within a single module, whether that be the declaring module or a client
module. The declaring module and client modules may see different names, but
each sees a name that is consistent fully qualified name everywhere the
submodule is referenced.
Allow @inline to be used by export statements within the module.
As with renaming, a symbol should have a single fully qualified name everywhere
within a single module.
Allow @implicit to be used by export statements within the module.
This would reduce the visibility of internal dependencies. If we find that
import-per-submodule becomes boilerplate-y this is an easy feature to add later.
Importing submodules
Submodules are imported in exactly the same way as an external module by using
an import statement. There are a few additional details that are not applicable
for external modules:
Circular imports are not allowed.
A submodule may not import any of its ancestors.
Relative child and sibling names are allowed using the same rules that apply to
nested types.
Access control
An access modifier applies an upper bound to the scope in which a symbol or
submodule is visible. With the introduction of submodules, internal now applies
at the level of a submodule: only the current submodule may see an internal
entity. We need a new way to specify module-wide visibility.
This proposal builds on Option 2 in the proposal Fix Private Access Levels
which reverts private to the Swift 2 meaning (equivalent to fileprivate) and
uses scoped for the Swift 3 scoped access feature. It does this by allowing the
scoped access modifier to be parameterized with a scope reference. By defaults
it references the scope in which it appears, but any ancestor scope may be
specified as a parameter.
The paremeterization of the scoped access modifier provides a simple yet
powerful way for a submodule to bound the visibility of a descendent.
Some examples of using scoped exports are:
submodule Parent
// `Grandparent` and all of its descendents can see `Child1` (fully qualified:
`Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.
scoped(Grandparent) export Child1
// `Child2` is visible throughout the module but may not be exported for use by
clients.
// This reads: `Child2` is scoped to the module.
scoped(module) export Child2
With parameterization, scoped has the power to specify all access levels that
Swift has today:
`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) ==
`fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`
This design is a direct generalization of the principle underlying Swift’s
existing access control system. It unifies the semantics of the system under
the single elegant mechanism of ancestor scope references.
While it is possible to specify all access levels using scoped that is not
recommended. The aliases public, private(Swift 2) and internal provide
excellent default access levels that don’t require a user to think about scope
hierarchies. Using the default access levels when possible calls extra
attention to cases where a different choice was made.
*This is a conceptual model. This proposal does not introduce the inherit or
override parameter to access modifiers. It could be added in the future as a
way to bound inheritance within a module. It would work similarly to
private(set) does in Swift today.
Aside
The parameterization of scoped also allows us to reference other scopes that we
cannot in today’s system, specifically extensions: scoped(extension) and outer
types: scoped(TypeName).
Alternatives
If we don’t adopt the approach of parameterizing scoped our options for access
control include:
Submodules are only allowed to see public and open symbols from other submodules
A module-wide scope is highly desirable. People might avoid using submodules if
this is not available.
This approach also creates a lot more friction when refactoring. A possible
workaround to the lack of a module-wide scope in this system is to place code
in a non-exported submodule and declare symbols public. Even with the
workaround, extracting a submodule may not always be possible or desirable and
the public access modifiers required would be misleading. It would be much
better to be able to state our intent directly.
Use internal to cover the whole module and private to cover a submodule
One suggestion that has appeared is the idea of removing fileprivate and making
private be submodule-wide. internal would remain module-wide. This is too
coarse - many people want a file-level scope.
internal is Swift’s default access modifier. A symbol with default access
modifier should not be able to cross a submodule boundary implicitly.
Add the moduleinternal access modifier
This is about as ugly as fileprivate.
Detailed design
Export errors
Multiple exports of the same submodule
If a submodule exports the same descendent more than once and the semantics of
the declarations are not identical an error is produced.
Symbol flattening
When a submodule is exported by the top-level module using the @inline
attribute it is possible that there will be conflicting symbol definitions in
the child and the top-level submodule (or other inlined submodules). This
results in a compiler error at the site of the conflicing @inline export
statements.
Overlapping renames
As with flattening, when two or more submodules are given the same external
name symbol conflicts are possible. This also results in a compiler error at
the site of the conflicting export as statements.
Access errors during export if the specified access modifier exceeds maximum
An error is produced when an export statement includes an access modifier
greater than the bound provided for the exported submodule by a descendent of
the exporting submodule.
Source compatibility
This proposal is purely additive. That said, it would be a breaking change for
the standard library to move existing API into an externally visible submodule.
Effect on ABI stability
This proposal is purely additive. That said, it would be a breaking change for
the standard library to move existing API into an externally visible submodule.
Effect on API resilience
This proposal is purely additive.
Future directions
Selective export and import
The ability to import and export individual symbols would be a very nice to
have.
Scoped import
The ability to import modules, submodules, and symbols into any lexical scope
would be nice to have.
Bounded inheritance
It could be useful to have the ability to bound inheritance within a module.
This could be accomplished by inroducing inherit and override parameters for
access modifiers (which would work similarly to the existing set parameter).
Appendix A: file system independence
The submodule design specified by this proposal is file system independent. The
only relationship it has with the physical file system is that a file
introduces an anonymous scope boundary which is referenced by scoped(file) or
fileprivate (Swift 3) or private (Swift 2 and 4?).
The logical role of a “file” in this design is to provide a boundary of
encapsulation that is even lighter weight than a submodule: it doesn’t hide
names. All declarations are implicitly available not only within the file but
also across the file boundary (modulo access control). Files are to submodules
as submodules are to modules.
If a future version of Swift were to eliminate files in favor of some kind of
code browser it would still be very useful to have the ability to form a pure
scope boundary (with no additional semantics). A scope declaration could be
used to do this. Scope declarations could have an optional (or required) name
and could even be nested. In this system privatewould reference the nearest
anonymous ancestor scope declaration (or would be removed if we don’t allow
anonymous scopes).
The logical structure of this design can be directly translated into a grammar
that could be represented directly with syntax. Such a grammer could be used to
support scripts with submodules. An example follows:
// A module contans a single implicit, anonymous submodule.
// submodule {
// A submodule may contain `scope` declarations (i.e. files) as well as other
submodules.
// An anonymous scope is equivalent to a file in current Swift.
// If we introduce lexical scopes we would probably require them to be named
explicitly.
// This example uses the anonymous scope in order to most closely match the
role files play in the current system.
// Because `scope` does not provide a name boundary all names declared in one
scope
// are visible in other scopes (modulo access control)
scope {
// Top-level declarations go here.
// This is equivalent to the top level of a file in Swift today.
// It is also equivalent to the top level of a file that does not contain
// a `submodule` declaration in this proposal.
// It would be possible to allow nested, named scopes.
// A scope name participates in the scope and name hierachies.
// However, it does not form a name boundary like a submodule does.
scope Named {
// This declares the static variable `Named.foo`
// `scoped(file)` references the nearest anonymous ancestor scope.
// It is used in this example for specificity.
// Real code would use the alias `private` or `fileprivate`
// If we introduce explicit scope syntax we would probably want a
better name to refer
// to the nearest anonymous scope than `file` or we may just require
all scopes to have a name.
scoped(file) var foo: String
}
// `Named.foo` is visible here
}
// `Named.foo` is not visible here.
submodule Foo {}
submodule Baz {}
submodule Buzz {
// Equivalient to a file in current Swift.
scope {
// submodule declarations go here.
// This is equivalent to the top level scope of a file that contains the
`submodule Foo` declaration.
}
scope {}
submodule Baz {}
}
//}
Appendix B: namespace style submodules
It is possible to design a system that allows a name boundary to be formed
without also forming a scope boundary. A natural consequence of this is that
symbols may be placed into a namespace-style submodule in many (unlimited)
scopes via extension (even extension outside the module is theoretically
possible). Allowing this runs contray to both of the two primary goals of this
proposal (encapsulation and structure).
Allowing a submodule to be extended in multiple scopes precludes the
possibility of submodule internal visibility. A submodule internal access
modifier could still be defined but it would not provide the guarantee it
purports to. The submodule can be opened by extension anywhere within the
module. If a lazy developer wants to access a submodule internal symbol from a
distant subsytem all they need to do is add an extension and wrap the submodule
internal symbol with a new symbol offering higher visibility*. In such a system
there is the same wide gap between file scope and module scope that exists
today.
Allowing a submodule to be extended in multiple scopes precludes the ability to
introduce real structure to a module. We are able to introduce structure to the
names but not the module itself. The structure of a submodule in such a system
may be widely dispersed throughout the module. It is not part of a strictly
hierarchical structure of scopes which each having a single designated location
within the larger structure.
What you do get from name boundaries that do not also form a scope boundary is
a soft form of symbol hiding (soft because all submodules are available for
import or extension anywhere within the program). This does provide some value,
but not nearly as much value as is provided by a name boundary that is
accompanied by a scope boundary.
Another downside to namespace-style submodules that are open to extension is
that they are much less likely to facilitate improved build performance because
they don’t add any physical structure to the system.
Finally, lexical submodules have the potential to be confusing. If submodules
form a name boundary (even a soft one) an import statement is required to
access the symbols declared inside a submodule. Is code that surrounds a
lexical submodule declaration able to see the symbols it declares without
importing them? Most developers will expect the symbols to be available. It is
probably necessary to make an exception to name boundary for the surrounding
lexical context. However, if an exception is made then this system relies
heavily on a file to provide a bound to the implicit symbol import.
*It is worth observing that the ability to violate encapsulation via extension
(or subclassing) is one of the primary reasons Swift does not offer type-based
access modifiers such as typeprivate or protected. The do not offer true
encapsulation at all. They are a statement of intent that cannot really be
verified in the way that is desired. They form a permeable rather than a hard
boundary.
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution