This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/daffodil-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bed108a Publishing from 17f2e9c58ab3fb775a9adb8c8252309147ba4c8e
bed108a is described below
commit bed108a9fec46ac4143783b25982369b6e23b35f
Author: Apache Daffodil Site Autobuild <[email protected]>
AuthorDate: Tue Nov 4 17:20:31 2025 +0000
Publishing from 17f2e9c58ab3fb775a9adb8c8252309147ba4c8e
---
content/binary-large-objects/index.html | 281 ++++++++++++++++++++++
content/dfdl-extensions/index.html | 407 ++++++++++++++++++++++++++++----
content/layers/index.html | 68 ++++--
content/unsupported/index.html | 2 +-
4 files changed, 690 insertions(+), 68 deletions(-)
diff --git a/content/binary-large-objects/index.html
b/content/binary-large-objects/index.html
new file mode 100644
index 0000000..4783e41
--- /dev/null
+++ b/content/binary-large-objects/index.html
@@ -0,0 +1,281 @@
+<!DOCTYPE html>
+<html lang="en">
+ <head>
+ <meta charset="utf-8">
+ <title>Apache Daffodil | Binary Large Objects Feature</title>
+ <meta name="description" content="Binary Large Objects Feature">
+ <meta name="author" content="">
+
+ <!-- Enable responsive viewport -->
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <!-- HTML5 shim, for IE6-8 support of HTML elements -->
+ <!--[if lt IE 9]>
+ <script
src="http://html5shim.googlecode.com/svn/trunk/html5.js";></script>
+ <![endif]-->
+
+ <link href="/assets/themes/apache/img/apache-daffodil-icon.png" rel="icon"
type="image/png">
+
+ <link href="/assets/themes/apache/bootstrap/css/bootstrap.css"
rel="stylesheet">
+ <link href="/assets/themes/apache/css/style.css?body=1" rel="stylesheet"
type="text/css">
+ <link href="/assets/themes/apache/css/syntax.css" rel="stylesheet"
type="text/css" media="screen" />
+
+ </head>
+
+ <body>
+
+ <div class="navbar navbar-inverse" role="navigation">
+ <div class="container">
+ <div class="navbar-header"><a class="navbar-brand" href="/"><img
src="/assets/themes/apache/img/apache-daffodil-logo.png" alt="Apache
Daffodil"/></a></div>
+ <nav role="navigation">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="/releases">Releases</a></li>
+ <li id="extensions">
+ <a href="#" data-toggle="dropdown"
class="dropdown-toggle">Extensions<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/vscode">VS Code</a></li>
+ <li><a href="/sbt">SBT</a></li>
+ </ul>
+ </li>
+ <li id="documentation">
+ <a href="#" data-toggle="dropdown"
class="dropdown-toggle">Docs<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/getting-started/">Getting Started</a></li>
+ <li><a href="/examples/">Examples</a></li>
+ <li><a href="/docs/latest/javadoc/">API</a></li>
+ <li><a href="/docs/dfdl/">DFDL Specification</a></li>
+ <li><a href="/unsupported/">Unsupported Features</a></li>
+ <li><a href="/faq/">Frequently Asked Questions</a></li>
+ <li><a href="/dfdl-extensions/">Daffodil DFDL Language
Extensions</a></li>
+ </ul>
+ </li>
+ <li id="community">
+ <a href="#" data-toggle="dropdown"
class="dropdown-toggle">Community<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a href="/community">Get Involved</a></li>
+ <li><a href="/people">People</a></li>
+ </ul>
+ </li>
+ <li id="development">
+ <a href="#" data-toggle="dropdown"
class="dropdown-toggle">Development<b class="caret"></b></a>
+ <ul class="dropdown-menu dropdown-left">
+ <li><a class="external"
href="https://cwiki.apache.org/confluence/display/DAFFODIL/";>Wiki</a></li>
+ <li><a class="external"
href="https://github.com/apache/?q=daffodil";>GitHub</a></li>
+ <li><a class="external"
href="https://issues.apache.org/jira/projects/DAFFODIL/";>JIRA</a></li>
+ </ul>
+ </li>
+ <li id="apache">
+ <a href="#" data-toggle="dropdown"
class="dropdown-toggle">Apache<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a class="external"
href="https://www.apache.org/";>Foundation</a></li>
+ <li><a class="external"
href="https://www.apache.org/licenses/";>License</a></li>
+ <li><a class="external"
href="https://www.apache.org/events/current-event";>Events</a></li>
+ <li><a class="external"
href="https://www.apache.org/security";>Security</a></li>
+ <li><a class="external"
href="https://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li>
+ <li><a class="external"
href="https://www.apache.org/foundation/thanks.html";>Thanks</a></li>
+ <li><a class="external"
href="https://privacy.apache.org/policies/privacy-policy-public.html";>Privacy
Policy</a></li>
+ </ul>
+ </li>
+ </ul>
+ </nav>
+ </div>
+ </div>
+
+
+
+
+<div class="title">
+ <div class="container"></div>
+</div>
+<div class="container">
+ <h1>Binary Large Objects Feature</h1>
+</div>
+
+
+
+
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+ <!--
+
+-->
+<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
+
+<ol id="markdown-toc">
+ <li><a href="#introduction"
id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#type-xsanyuri-and-property-dfdlxobjectkind"
id="markdown-toc-type-xsanyuri-and-property-dfdlxobjectkind">Type <code
class="language-plaintext highlighter-rouge">xs:anyURI</code> and Property
<code class="language-plaintext
highlighter-rouge">dfdlx:objectKind</code></a></li>
+ <li><a href="#daffodil-blob-api"
id="markdown-toc-daffodil-blob-api">Daffodil BLOB API</a></li>
+ <li><a href="#dfdl-expressions" id="markdown-toc-dfdl-expressions">DFDL
Expressions</a></li>
+ <li><a href="#testing-dfdl-schemas-using-blobs-via-the-tdml-runner"
id="markdown-toc-testing-dfdl-schemas-using-blobs-via-the-tdml-runner">Testing
DFDL Schemas using BLOBs via the TDML Runner</a></li>
+ <li><a href="#command-line-interface"
id="markdown-toc-command-line-interface">Command Line Interface</a></li>
+</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
+
+<!--
+This page is linked from https://s.apache.org/daffodil-blob-feature.
+ If this page content moves, please update that link from https://s.apache.org.
+-->
+
+<h2 id="introduction">Introduction</h2>
+
+<p>Daffodil has implemented a DFDL extension that allows data much larger than
memory to be manipulated.</p>
+
+<p>A variety of data formats, such as for image and video files, consist of
fields of what is effectively metadata, surrounding large blocks of data
containing compressed image or video data.</p>
+
+<p>An important use case for DFDL is to expose this metadata for easy use, and
to provide access to
+the large data via a streaming mechanism akin to opening a file, thereby
avoiding
+large <code class="language-plaintext highlighter-rouge">xs:hexBinary</code>
strings in the infoset.</p>
+
+<p>In RDBMS systems, BLOB (Binary Large Object) is the type used when the data
row returned from an SQL query will not contain the actual value data, but
rather a handle that can be used to open/read/write/close the BLOB.</p>
+
+<p>Daffodil has an analogous BLOB capability.
+This enables processing of images or video of arbitrary size without the need
to ever hold all the data in memory.</p>
+
+<p>This also bypasses the limitation on object size.</p>
+
+<h2 id="type-xsanyuri-and-property-dfdlxobjectkind">Type <code
class="language-plaintext highlighter-rouge">xs:anyURI</code> and Property
<code class="language-plaintext highlighter-rouge">dfdlx:objectKind</code></h2>
+
+<p>DFDL is extended to allow simple types to have the <code
class="language-plaintext highlighter-rouge">xs:anyURI</code> type.
+Elements with this type will be treated as BLOB objects.</p>
+
+<p>The <code class="language-plaintext
highlighter-rouge">dfdlx:objectKind</code> property is added to define what
type of object it is.
+The valid value for this property is only <code class="language-plaintext
highlighter-rouge">"bytes"</code> specifying binary large objects.
+All other values reserved for future extensions of this feature.</p>
+
+<p>An example of this usage in a DFDL schema may look something like this:</p>
+
+<pre><code class="language-xsd"><xs:schema
+ xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions";
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/">
+
+ <xs:element name="data" type="xs:anyURI"
+ dfdlx:objectKind="bytes"
+ dfdl:lengthUnits="bytes"
+ dfdl:length="1024" />
+
+</xs:schema>
+</code></pre>
+
+<p>The resulting infoset (as XML) will look something like this:</p>
+
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span
class="nt"><data></span>file:///path/to/blob/data<span
class="nt"></data></span>
+</code></pre></div></div>
+
+<p>With the 1024 bytes of data being written to a file at location <code
class="language-plaintext highlighter-rouge">/path/to/blob/data</code>.</p>
+
+<p>The BLOB URI must always use the <em>file scheme</em> and must be
absolute.</p>
+
+<h2 id="daffodil-blob-api">Daffodil BLOB API</h2>
+
+<p>API calls are used to specify where Daffodil should write the BLOB
files.</p>
+
+<p>Two functions are used on the Daffodil <code class="language-plaintext
highlighter-rouge">InfosetOutputter</code>.</p>
+
+<p>The first API function allows a way to set the properties used when
+creating BLOB files, including the output directory, and prefix/suffix
+for the BLOB file.</p>
+
+<div class="language-scala highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="cm">/**
+ * Set the attributes for how to create blob files.
+ *
+ * @param dir the Path the the directory to create files. If the directory
+ * does not exist, Daffodil will attempt to create it before
+ * writing a blob.
+ * @param prefix the prefix string to be used in generating a blob file name
+ * @param suffix the suffix string to be used in generating a blob file name
+ */</span>
+<span class="k">final</span> <span class="k">def</span> <span
class="nf">setBlobAttributes</span><span class="o">(</span><span
class="n">directory</span><span class="k">:</span> <span
class="kt">Path</span><span class="o">,</span> <span
class="n">prefix</span><span class="k">:</span> <span
class="kt">String</span><span class="o">,</span> <span
class="n">suffix</span><span class="k">:</span> <span
class="kt">String</span><span class="o">)</span>
+</code></pre></div></div>
+
+<p>The second API function allows a way for the API user to get a list of
+all BLOB files that were created during <code class="language-plaintext
highlighter-rouge">parse()</code>.</p>
+
+<div class="language-scala highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="cm">/**
+ * Get the list of blob paths that were output in the infoset.
+ *
+ * This is the same as what would be found by iterating over the infoset.
+ */</span>
+<span class="k">final</span> <span class="k">def</span> <span
class="nf">getBlobFiles</span><span class="o">()</span><span class="k">:</span>
<span class="kt">Seq</span><span class="o">[</span><span
class="kt">Path</span><span class="o">]</span>
+</code></pre></div></div>
+
+<p>Note that no changes to the <code class="language-plaintext
highlighter-rouge">unparse()</code> API are required, since the BLOB URI
provides
+all the necessary information to retrieve files containing BLOB data.</p>
+
+<p>BLOB files are not automatically deleted.
+It is the responsibility of the API user to determine when files are no
+longer needed and remove them.</p>
+
+<h2 id="dfdl-expressions">DFDL Expressions</h2>
+
+<p>Any expression access to the <em>data</em> of a BLOB element will result in
a
+Schema Definition Error during schema compilation.</p>
+
+<p>The <em>length</em> of a BLOB element is available since it is very common
in
+data formats to include both a BLOB payload and the length of that
+payload. On unparse, we can calculate the length of the BLOB data so
+that the value can be output in a length field in the data. This is
+done using the regular <code class="language-plaintext
highlighter-rouge">dfdl:contentLength()</code> and <code
class="language-plaintext highlighter-rouge">dfdl:valueLength()</code>
+functions.</p>
+
+<h2 id="testing-dfdl-schemas-using-blobs-via-the-tdml-runner">Testing DFDL
Schemas using BLOBs via the TDML Runner</h2>
+
+<p>The TDML language is extended to support the <code
class="language-plaintext highlighter-rouge">xsi:type="xs:anyURI"</code>
annotation on XML data elements.</p>
+
+<p>For example:</p>
+
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="nt"><tdml:dfdlInfoset></span>
+ <span class="nt"><data</span> <span class="na">xsi:type=</span><span
class="s">"xs:anyURI"</span><span class="nt">></span>path/to/blob/data<span
class="nt"></data></span>
+<span class="nt"></tdml:dfdlInfoset></span>
+</code></pre></div></div>
+
+<p>The path provided as the URI value can be, and usually will be, a relative
path within the
+<code class="language-plaintext highlighter-rouge">src/test/resources</code>
directory of the DFDL schema project.
+During Infoset comparisons the TDML Runner will compare the contents of this
file
+with the BLOB file in the corresponding element (having type <code
class="language-plaintext highlighter-rouge">xs:anyURI</code>) of the
infoset.</p>
+
+<p>BLOB files created when running the tests are deleted when the test
completes.</p>
+
+<h2 id="command-line-interface">Command Line Interface</h2>
+
+<p>The CLI supports ad-hoc testing of the use of BLOBs.
+BLOBs are written to the directory given by the JVM <em>System Property</em>
<code class="language-plaintext highlighter-rouge">user.dir</code> into
+a subdirectory of it named <code class="language-plaintext
highlighter-rouge">daffodil-blobs</code>.
+If it does not exist, Daffodil will attempt to create the <code
class="language-plaintext highlighter-rouge">daffodil-blobs</code> directory.
+The CLI does not delete any BLOB files.</p>
+
+ </div>
+</div>
+
+
+ <footer>
+ <footer class="site-footer">
+ <div class="wrapper">
+ <div class="footer-col-wrapper" style="font-size: .85em;">
+ <hr>
+ <div>
+ <div style="text-align: center;">
+ Copyright © 2025 <a href="https://www.apache.org";>The
Apache Software Foundation</a>.
+ Licensed under the <a
href="https://www.apache.org/licenses/LICENSE-2.0";>Apache License, Version
+ 2.0</a>.
+ <br>
+ Apache, Apache Daffodil, Daffodil, and the Apache Daffodil
logo
+ are trademarks of The Apache Software Foundation.
+ </div>
+ </div>
+ </div>
+ </div>
+</footer>
+
+ </footer>
+ </div>
+
+ <script src="/assets/themes/apache/jquery/jquery-2.1.1.min.js"></script>
+
+ <script src="/assets/themes/apache/bootstrap/js/bootstrap.min.js"></script>
+
+
+ </body>
+</html>
+
diff --git a/content/dfdl-extensions/index.html
b/content/dfdl-extensions/index.html
index 5b2bcb6..17b8125 100644
--- a/content/dfdl-extensions/index.html
+++ b/content/dfdl-extensions/index.html
@@ -2,7 +2,7 @@
<html lang="en">
<head>
<meta charset="utf-8">
- <title>Apache Daffodil | DFDL Extensions</title>
+ <title>Apache Daffodil | Daffodil Extensions to the DFDL Language</title>
<meta name="author" content="">
@@ -88,7 +88,7 @@
<div class="container"></div>
</div>
<div class="container">
- <h1>DFDL Extensions</h1>
+ <h1>Daffodil Extensions to the DFDL Language</h1>
</div>
@@ -101,35 +101,96 @@
-->
+<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
+
+<ol id="markdown-toc">
+ <li><a href="#introduction"
id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#binary-large-objects-blob-feature"
id="markdown-toc-binary-large-objects-blob-feature">Binary Large Objects (BLOB)
Feature</a></li>
+ <li><a href="#expression-functions"
id="markdown-toc-expression-functions">Expression Functions</a> <ol>
+ <li><a href="#dfdlxtracevalue-label"
id="markdown-toc-dfdlxtracevalue-label"><code class="language-plaintext
highlighter-rouge">dfdlx:trace(value, label)</code></a></li>
+ <li><a href="#dfdlxlookaheadoffset-bitsize"
id="markdown-toc-dfdlxlookaheadoffset-bitsize"><code class="language-plaintext
highlighter-rouge">dfdlx:lookAhead(offset, bitSize)</code></a> <ol>
+ <li><a href="#examples-of-dfdlxlookahead"
id="markdown-toc-examples-of-dfdlxlookahead">Examples of <code
class="language-plaintext highlighter-rouge">dfdlx:lookAhead</code></a></li>
+ </ol>
+ </li>
+ <li><a
href="#bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift"
id="markdown-toc-bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift">Bitwise
Functions: <code class="language-plaintext highlighter-rouge">bitAnd</code>,
<code class="language-plaintext highlighter-rouge">bitOr</code>, <code
class="language-plaintext highlighter-rouge">bitXor</code>, <code
class="language-plaintext highlighter-rouge">bitNot</code>, <code
class="language-plaintext highlig [...]
+ <li><a href="#dfdlxbitandarg1-arg2"
id="markdown-toc-dfdlxbitandarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitAnd(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitorarg1-arg2"
id="markdown-toc-dfdlxbitorarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitOr(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitxorarg1-arg2"
id="markdown-toc-dfdlxbitxorarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitXor(arg1, arg2)</code></a></li>
+ <li><a href="#dfdlxbitnotarg" id="markdown-toc-dfdlxbitnotarg"><code
class="language-plaintext highlighter-rouge">dfdlx:bitNot(arg)</code></a></li>
+ <li><a href="#dfdlxleftshiftvalue-shiftcount"
id="markdown-toc-dfdlxleftshiftvalue-shiftcount"><code
class="language-plaintext highlighter-rouge">dfdlx:leftShift(value,
shiftCount)</code></a></li>
+ <li><a href="#dfdlxrightshiftvalue-shiftcount"
id="markdown-toc-dfdlxrightshiftvalue-shiftcount"><code
class="language-plaintext highlighter-rouge">dfdlx:rightShift(value,
shiftCount)</code></a></li>
+ </ol>
+ </li>
+ <li><a
href="#dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg"
id="markdown-toc-dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg"><code
class="language-plaintext
highlighter-rouge">dfdlx:doubleFromRawLong(longArg)</code> and <code
class="language-plaintext
highlighter-rouge">dfdlx:doubleToRawLong(doubleArg)</code></a></li>
+ </ol>
+ </li>
+ <li><a href="#properties" id="markdown-toc-properties">Properties</a> <ol>
+ <li><a href="#dfdlxalignmentkind"
id="markdown-toc-dfdlxalignmentkind"><code class="language-plaintext
highlighter-rouge">dfdlx:alignmentKind</code></a></li>
+ <li><a href="#dfdlxparseunparsepolicy"
id="markdown-toc-dfdlxparseunparsepolicy"><code class="language-plaintext
highlighter-rouge">dfdlx:parseUnparsePolicy</code></a></li>
+ <li><a href="#dfdlxlayer" id="markdown-toc-dfdlxlayer"><code
class="language-plaintext highlighter-rouge">dfdlx:layer</code></a></li>
+ <li><a href="#dfdlxdirection" id="markdown-toc-dfdlxdirection"><code
class="language-plaintext highlighter-rouge">dfdlx:direction</code></a></li>
+ <li><a href="#enumerations-dfdlxreptype-dfdlxrepvalues"
id="markdown-toc-enumerations-dfdlxreptype-dfdlxrepvalues">Enumerations: <code
class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code></a>
<ol>
+ <li><a href="#dfdlxreptype" id="markdown-toc-dfdlxreptype"><code
class="language-plaintext highlighter-rouge">dfdlx:repType</code></a></li>
+ <li><a href="#dfdlxrepvalues" id="markdown-toc-dfdlxrepvalues"><code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code></a></li>
+ <li><a href="#examples-of-enumerations-in-daffodil-dfdl"
id="markdown-toc-examples-of-enumerations-in-daffodil-dfdl">Examples of
Enumerations in Daffodil DFDL</a></li>
+ </ol>
+ </li>
+ </ol>
+ </li>
+ <li><a href="#extended-behaviors-for-dfdl-types"
id="markdown-toc-extended-behaviors-for-dfdl-types">Extended Behaviors for DFDL
Types</a> <ol>
+ <li><a href="#type-xshexbinary" id="markdown-toc-type-xshexbinary">Type
<code class="language-plaintext highlighter-rouge">xs:hexBinary</code></a></li>
+ </ol>
+ </li>
+</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
+
+<h2 id="introduction">Introduction</h2>
+
<p>Daffodil provides extensions to the DFDL specification.
-These properties are in the namespace defined by the URI
+These functions and properties are in the namespace defined by the URI
<code class="language-plaintext
highlighter-rouge">http://www.ogf.org/dfdl/dfdl-1.0/extensions</code> which is
normally bound to the <code class="language-plaintext
highlighter-rouge">dfdlx</code> prefix
like so:</p>
-<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="nt"><xs:schema</span> <span
class="na">xmlns:xs=</span><span
class="s">"http://www.w3.org/2001/XMLSchema";</span>
- <span class="na">xmlns:dfdl=</span><span
class="s">"http://www.ogf.org/dfdl/dfdl-1.0/";</span>
- <span class="na">xmlns:dfdlx=</span><span
class="s">"http://www.ogf.org/dfdl/dfdl-1.0/extensions";</span>
-<span class="nt">></span>
+<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="nt"><schema</span> <span
class="na">xmlns=</span><span
class="s">"http://www.w3.org/2001/XMLSchema";</span>
+ <span class="na">xmlns:dfdl=</span><span
class="s">"http://www.ogf.org/dfdl/dfdl-1.0/";</span>
+ <span class="na">xmlns:dfdlx=</span><span
class="s">"http://www.ogf.org/dfdl/dfdl-1.0/extensions";</span><span
class="nt">></span>
</code></pre></div></div>
-<p>The following symbols defined in this namespace are described below.</p>
+<p>The DFDL language extensions described below have Long Term Support (LTS)
in Daffodil
+going forward, and are proposed for inclusion in a future revision of the DFDL
+standard.
+DFDL schema authors can depend on the features and behaviors defined here
without fear
+that these extensions will be withdrawn in the future.</p>
-<h2 id="expression-functions">Expression Functions</h2>
+<h2 id="binary-large-objects-blob-feature">Binary Large Objects (BLOB)
Feature</h2>
-<h3 id="daferror"><code class="language-plaintext
highlighter-rouge">daf:error()</code></h3>
+<p>Daffodil supports processing data that contains large opaque binary objects,
+also known as <em>BLOBs</em>.
+These enable processing of data types such as images, audio, or video where
the
+data content is surrounded by important metadata.
+The DFDL Schema can expose the metadata fields for processing and carry
+along the opaque BLOB data in files.</p>
-<p>A function that can be used in DFDL expressions. This functions does not
return a value or accept any arguments. When called, it causes a Parse Error or
Unparse Error.</p>
+<p>There is
+<a href="/binary-large-objects">separate documentation for the Binary Large
Object (BLOB) feature</a>.</p>
-<p><em>This function is deprecated as of Daffodil 2.0.0. Use the <code
class="language-plaintext highlighter-rouge">fn:error(...)</code> function
instead.</em></p>
+<h2 id="expression-functions">Expression Functions</h2>
-<h3 id="dfdlxtracevalue-label"><code class="language-plaintext
highlighter-rouge">dfdlx:trace($value, $label)</code></h3>
+<h3 id="dfdlxtracevalue-label"><code class="language-plaintext
highlighter-rouge">dfdlx:trace(value, label)</code></h3>
-<p>A function that can be used in DFDL expressions, similar to the <code
class="language-plaintext highlighter-rouge">fn:trace()</code> function. This
logs the string <code class="language-plaintext
highlighter-rouge">$label</code> followed by <code class="language-plaintext
highlighter-rouge">$value</code> converted to a string and returns <code
class="language-plaintext highlighter-rouge">$value</code>. The second argument
must be of type <code class="language-plaintext highlighter-rou [...]
+<p>A function that can be used to debug DFDL expressions, similar to
+the <a href="https://www.w3.org/TR/xpath-functions-31/#func-trace";>XPath <code
class="language-plaintext highlighter-rouge">fn:trace(value, label)</code></a>
+function.
+This creates a message from the string argument <code
class="language-plaintext highlighter-rouge">label</code> followed by <code
class="language-plaintext highlighter-rouge">value</code> converted to a
+string and logs the message.
+The function returns the <code class="language-plaintext
highlighter-rouge">value</code>.
+The second <code class="language-plaintext highlighter-rouge">label</code>
argument must be of type <code class="language-plaintext
highlighter-rouge">xs:string</code>.</p>
<h3 id="dfdlxlookaheadoffset-bitsize"><code class="language-plaintext
highlighter-rouge">dfdlx:lookAhead(offset, bitSize)</code></h3>
<p>Read <code class="language-plaintext highlighter-rouge">bitSize</code>
bits, where the first bit is located at an <code class="language-plaintext
highlighter-rouge">offset</code> (in bits)
-from the current location. The result is a <code class="language-plaintext
highlighter-rouge">xs:nonNegativeInteger</code>. Restrictions:</p>
+ from the current location. The result is a <code class="language-plaintext
highlighter-rouge">xs:nonNegativeInteger</code>. Restrictions:</p>
<ul>
<li>offset >=0</li>
@@ -148,39 +209,127 @@ the data being read will not be used.</li>
<h4 id="examples-of-dfdlxlookahead">Examples of <code
class="language-plaintext highlighter-rouge">dfdlx:lookAhead</code></h4>
<p>The following two elements both populate element <code
class="language-plaintext highlighter-rouge">a</code> with the value of the
next 3 bits as an
-unsignedInt. They are not completely equivalent because the first will consume
3 bits of the
+unsignedInt.
+They are not completely equivalent because the first will consume 3 bits of
the
input stream where the second will not advance the input stream.</p>
+
<div class="language-xml highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><span class="nt"><xs:element</span> <span
class="na">name=</span><span class="s">"a"</span> <span
class="na">type=</span><span class="s">"xs:unsignedInt"</span> <span
class="na">dfdl:length=</span><span class="s">"3"</span> <span
class="na">dfdl:lengthUnits=</span><span class="s">"bits"</span> <span
class="nt">/></span>
<span class="nt"><xs:element</span> <span class="na">name=</span><span
class="s">"a"</span> <span class="na">type=</span><span
class="s">"xs:unsignedInt"</span> <span
class="na">dfdl:inputValueCalc=</span><span class="s">"{ dfdlx:lookAhead(0,3)
}"</span> <span class="nt">/></span>
</code></pre></div></div>
<p>The following example demonstrates using lookAhead to branch based on a
field in the future.
In this case the choice of elements <code class="language-plaintext
highlighter-rouge">a</code> vs. <code class="language-plaintext
highlighter-rouge">b</code> depends on the value of the <code
class="language-plaintext highlighter-rouge">tag</code> field which is
-found after fields <code class="language-plaintext highlighter-rouge">a</code>
and <code class="language-plaintext highlighter-rouge">b</code>:
-```</p>
-<xs:choice dfdl:choiceDispatchKey="{ dfdlx:lookAhead(16,8) }">
-<xs:element name="a" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="1" />
-<xs:element name="b" type="xs:int" dfdl:length="16" dfdl:choiceBranchKey="2" />
-</xs:choice>
-<xs:element name="tag" type="xs:int" dfdl:length="8" />
-
-<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre
class="highlight"><code> ``` # Bitwise Functions
+found after fields <code class="language-plaintext highlighter-rouge">a</code>
and <code class="language-plaintext highlighter-rouge">b</code>:</p>
+<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre
class="highlight"><code><xs:choice dfdl:choiceDispatchKey="{
dfdlx:lookAhead(16,8) }">
+ <xs:element name="a" type="xs:int" dfdl:length="16"
dfdl:choiceBranchKey="1"/>
+ <xs:element name="b" type="xs:int" dfdl:length="16"
dfdl:choiceBranchKey="2"/>
+</xs:choice>
+<xs:element name="tag" type="xs:int" dfdl:length="8" />
</code></pre></div></div>
-<p>TBD, but the complete list (all <code class="language-plaintext
highlighter-rouge">dfdlx</code>) is <code class="language-plaintext
highlighter-rouge">BitAnd</code>, <code class="language-plaintext
highlighter-rouge">BitNot</code>, <code class="language-plaintext
highlighter-rouge">BitOr</code>, <code class="language-plaintext
highlighter-rouge">BitXor</code>, <code class="language-plaintext
highlighter-rouge">LeftShift</code>,
-<code class="language-plaintext highlighter-rouge">RightShift</code></p>
+<h3
id="bitwise-functions-bitand-bitor-bitxor-bitnot-leftshift-rightshift">Bitwise
Functions: <code class="language-plaintext highlighter-rouge">bitAnd</code>,
<code class="language-plaintext highlighter-rouge">bitOr</code>, <code
class="language-plaintext highlighter-rouge">bitXor</code>, <code
class="language-plaintext highlighter-rouge">bitNot</code>, <code
class="language-plaintext highlighter-rouge">leftShift</code>, <code
class="language-plaintext highlighter-rouge">rightShift</code></h3>
+
+<p>These functions are defined on types <code class="language-plaintext
highlighter-rouge">long</code>, <code class="language-plaintext
highlighter-rouge">int</code>, <code class="language-plaintext
highlighter-rouge">short</code>, <code class="language-plaintext
highlighter-rouge">byte</code>, <code class="language-plaintext
highlighter-rouge">unsignedLong</code>,
+<code class="language-plaintext highlighter-rouge">unsignedInt</code>, <code
class="language-plaintext highlighter-rouge">unsignedShort</code>, and <code
class="language-plaintext highlighter-rouge">unsignedByte</code></p>
+
+<h4 id="dfdlxbitandarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitAnd(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise AND of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted
into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does
sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitorarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitOr(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise OR of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted
into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does
sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitxorarg1-arg2"><code class="language-plaintext
highlighter-rouge">dfdlx:bitXor(arg1, arg2)</code></h4>
+
+<p>This computes the bitwise Exclusive OR of two integers.</p>
+
+<ul>
+ <li>Both arguments must be signed, or both must be unsigned.</li>
+ <li>If the two arguments are not the same type the smaller one is converted
into the type of the
+larger one.</li>
+ <li>If the smaller argument is signed, this conversion does
sign-extension.</li>
+ <li>The result type is the that of the largest argument.</li>
+</ul>
+
+<h4 id="dfdlxbitnotarg"><code class="language-plaintext
highlighter-rouge">dfdlx:bitNot(arg)</code></h4>
+
+<p>This computes the bitwise NOT of an integer. Every bit is inverted. The
result type is the same
+as the argument type.</p>
+
+<h4 id="dfdlxleftshiftvalue-shiftcount"><code class="language-plaintext
highlighter-rouge">dfdlx:leftShift(value, shiftCount)</code></h4>
-<h3 id="dfdlxdoublefromrawlong-and-dfdlxdoubletorawlong"><code
class="language-plaintext highlighter-rouge">dfdlx:doubleFromRawLong</code> and
<code class="language-plaintext
highlighter-rouge">dfdlx:doubleToRawLong</code></h3>
+<p>This is the <em>logical</em> shift left, meaning that bits are shifted from
less-significant positions
+to more-significant positions.</p>
-<p>Converting binary floating point numbers to/from base 10 text can result in
lost information.
-The base 10 representation, converted back to binary representation, may not
be bit-for-bit
-identical. These functions can be used to carry 8-byte double precision IEEE
floating point
-numbers as type <code class="language-plaintext
highlighter-rouge">xs:long</code> so that no information is lost. The DFDL
schema can still obtain
-and operate on the floating point value by converting these <code
class="language-plaintext highlighter-rouge">xs:long</code> values into type
-<code class="language-plaintext highlighter-rouge">xs:double</code>, and back
if necessary for unparsing a new value.</p>
+<ul>
+ <li>The left-most bits shifted out are discarded.</li>
+ <li>Zeros are shifted in for the right-most bits.</li>
+ <li>The result type is the same as the <code class="language-plaintext
highlighter-rouge">value</code> argument type.</li>
+ <li>It is a processing error if the <code class="language-plaintext
highlighter-rouge">shiftCount</code> argument is < 0.</li>
+ <li>It is a processing error if the <code class="language-plaintext
highlighter-rouge">shiftCount</code> argument is greater than the number of
+bits in the type of the value argument.</li>
+</ul>
+
+<h4 id="dfdlxrightshiftvalue-shiftcount"><code class="language-plaintext
highlighter-rouge">dfdlx:rightShift(value, shiftCount)</code></h4>
+
+<p>This is the <em>arithmetic</em> shift right, meaning bits move from
most-significant to
+less-significant positions.
+If <em>logical</em> (zero-filling) shift right is needed, you must use
unsigned types.</p>
+
+<ul>
+ <li>The <code class="language-plaintext highlighter-rouge">value</code>
argument is shifted by the <code class="language-plaintext
highlighter-rouge">shiftCount</code>.</li>
+ <li>The right-most bits shifted out are discarded.</li>
+ <li>If the <code class="language-plaintext highlighter-rouge">value</code>
is signed, then the sign bit is shifted in for the left-most bits.</li>
+ <li>If the <code class="language-plaintext highlighter-rouge">value</code>
is unsigned, then zeros are shifted in for the left-most bits.</li>
+ <li>The result type is the same as the <code class="language-plaintext
highlighter-rouge">value</code> argument type.</li>
+ <li>It is a processing error if the <code class="language-plaintext
highlighter-rouge">shiftCount</code> argument is < 0.</li>
+ <li>It is a processing error if the <code class="language-plaintext
highlighter-rouge">shiftCount</code> argument is greater than the number of
+bits in the type of the value argument.</li>
+</ul>
+
+<h3 id="dfdlxdoublefromrawlonglongarg-and-dfdlxdoubletorawlongdoublearg"><code
class="language-plaintext
highlighter-rouge">dfdlx:doubleFromRawLong(longArg)</code> and <code
class="language-plaintext
highlighter-rouge">dfdlx:doubleToRawLong(doubleArg)</code></h3>
+
+<p>IEEE binary float and double values that are not NaN will parse to base 10
text and unparse back
+to the same exact IEEE binary bits.
+However, the same cannot be said for NaN (not a number) values, of which there
are many bit
+patterns.
+To preserve float and double NaN values bit for bit you can use these
functions to compute
+<code class="language-plaintext highlighter-rouge">xs:long</code> values that
enable the DFDL Infoset to preserve the bits of a float or double value
+even if it is a NaN.</p>
<h2 id="properties">Properties</h2>
+<h3 id="dfdlxalignmentkind"><code class="language-plaintext
highlighter-rouge">dfdlx:alignmentKind</code></h3>
+
+<p>Valid values for this property are <code class="language-plaintext
highlighter-rouge">manual</code> or <code class="language-plaintext
highlighter-rouge">automatic</code> with <code class="language-plaintext
highlighter-rouge">automatic</code> being the default
+behavior.
+When specified, the <code class="language-plaintext
highlighter-rouge">manual</code> value turns off all automatic alignment based
on the
+<code class="language-plaintext highlighter-rouge">dfdl:alignment</code> and
<code class="language-plaintext highlighter-rouge">dfdl:alignmentUnits</code>
properties.
+The schema author must use <code class="language-plaintext
highlighter-rouge">dfdl:leadingSkip</code>, <code class="language-plaintext
highlighter-rouge">dfdl:trailingSkip</code>, or just ensure all the
+elements/terms are aligned based on their length.</p>
+
+<p>This property is sometimes needed to facilitate creation of schemas where
interactions occur
+between computed lengths (that is, stored length fields) and
+alignment regions that are automatically being inserted.
+It can be easier to do all alignment manually than to debug these
interactions.</p>
+
<h3 id="dfdlxparseunparsepolicy"><code class="language-plaintext
highlighter-rouge">dfdlx:parseUnparsePolicy</code></h3>
<p>A property applied to simple and complex elements, which specifies whether
the element supports only parsing, only unparsing, or both parsing and unparse.
Valid values for this property are <code class="language-plaintext
highlighter-rouge">parse</code>, <code class="language-plaintext
highlighter-rouge">unparse</code>, or <code class="language-plaintext
highlighter-rouge">both</code>. This allows one to leave off properties that
are required for only parse or only unparse, such as <c [...]
@@ -191,24 +340,200 @@ and operate on the floating point value by converting
these <code class="languag
<h3 id="dfdlxlayer"><code class="language-plaintext
highlighter-rouge">dfdlx:layer</code></h3>
-<p><a href="/layers">Layers</a> provide algorithmic capabilities for
decoding/encoding data or computing
-checksums. Some are built-in to Daffodil. New layers can be created in
Java/Scala and
-plugged-in to Daffodil dynamically.</p>
+<p><em>Layers</em> provide algorithmic capabilities for decoding/encoding data
or computing
+ checksums. Some are built-in to Daffodil. New layers can be created in
Java/Scala and
+ plugged-in to Daffodil dynamically.
+There is <a href="/layers">separate Layer documentation</a>.</p>
<h3 id="dfdlxdirection"><code class="language-plaintext
highlighter-rouge">dfdlx:direction</code></h3>
-<p>TBD</p>
+<p>This property can appear only on DFDL <code class="language-plaintext
highlighter-rouge">defineVariable</code> statement annotations.
+This property has possible values <code class="language-plaintext
highlighter-rouge">both</code> (the default), <code class="language-plaintext
highlighter-rouge">parseOnly</code>, or <code class="language-plaintext
highlighter-rouge">unparseOnly</code>.
+It declares
+whether the variable is to be available for only parsing, only unparsing, or
both.
+Since this is a newly introduced extension property and existing schemas won't
contain a definition
+for it, it has a default value of <code class="language-plaintext
highlighter-rouge">both</code>.</p>
+
+<p>This property can conflict with the <code class="language-plaintext
highlighter-rouge">dfdlx:parseUnparsePolicy</code> property which takes the
same
+values (<code class="language-plaintext highlighter-rouge">both</code>, <code
class="language-plaintext highlighter-rouge">parseOnly</code>, and <code
class="language-plaintext highlighter-rouge">unparseOnly</code>).
+If <code class="language-plaintext
highlighter-rouge">dfdlx:parseUnparsePolicy='parseOnly'</code> then it is a
Schema Definition Error if
+variables in the DFDL schema have <code class="language-plaintext
highlighter-rouge">dfdlx:direction='unparseOnly'</code>.
+Similarly if <code class="language-plaintext
highlighter-rouge">dfdlx:parseUnparsePolicy='unparseOnly'</code> then it is a
Schema Definition Error if
+variables in the DFDL schema have <code class="language-plaintext
highlighter-rouge">dfdlx:direction='parseOnly'</code>.</p>
+
+<p>It is a Schema Definition Error if a variable defined with direction <code
class="language-plaintext highlighter-rouge">parseOnly</code> is accessed
+from an expression used by the unparser.
+Symmetrically, it is a Schema Definition Error if a variable defined with
direction
+<code class="language-plaintext highlighter-rouge">unparseOnly</code> is
accessed from an expression used by the parser.
+This error is detected at DFDL schema compilation time, not runtime.</p>
+
+<p>These properties take expressions for their values and are generally
evaluated at both parse and
+unparse time.
+Hence, unless the whole schema is constrained by <code
class="language-plaintext highlighter-rouge">dfdlx:parseUnparsePolicy</code>,
any expressions for
+these properties<sup id="fnref:moreProps"><a href="#fn:moreProps"
class="footnote" rel="footnote" role="doc-noteref">1</a></sup> cannot<br />
+cannot reference DFDL variables with <code class="language-plaintext
highlighter-rouge">dfdlx:direction</code> of <code class="language-plaintext
highlighter-rouge">parseOnly</code> or <code class="language-plaintext
highlighter-rouge">unparseOnly</code>.</p>
+
+<ul>
+ <li><code class="language-plaintext highlighter-rouge">byteOrder</code></li>
+ <li><code class="language-plaintext highlighter-rouge">encoding</code></li>
+ <li><code class="language-plaintext highlighter-rouge">initiator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">terminator</code></li>
+ <li><code class="language-plaintext highlighter-rouge">separator</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">escapeCharacter</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">escapeEscapeCharacter</code></li>
+ <li><code class="language-plaintext highlighter-rouge">length</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">occursCount</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">textStandardDecimalSeparator</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">textStandardGroupingSeparator</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">textStandardExponentRep</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">binaryFloatRep</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">textBooleanTrueRep</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">textbooleanFalseRep</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">calendarLanguage</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext
highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or
a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code>
default value expression when
+that variable being set/defaulted is itself referenced from a another
expression and the variable
+being set/defaulted has <code class="language-plaintext
highlighter-rouge">dfdlx:direction</code> of <code class="language-plaintext
highlighter-rouge">both</code> (the default)</li>
+</ul>
-<h3 id="dfdlxreptype-dfdlxrepvalues-and-dfdlxrepvalueranges"><code
class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code>, and <code
class="language-plaintext highlighter-rouge">dfdlx:repValueRanges</code></h3>
+<!-- footnotes must be all one big long line -->
-<p>TBD</p>
+<p>Parser-specific expressions include</p>
+
+<ul>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:inputValueCalc</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:length</code>
(when dfdl:lengthKind='explicit')</li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:occursCount</code> (when
`dfdl:occursCountKind='expression')</li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:choiceDispatchKey</code></li>
+ <li>the <code class="language-plaintext highlighter-rouge">message</code>
and <code class="language-plaintext highlighter-rouge">test</code> attributes
of the <code class="language-plaintext highlighter-rouge">dfdl:assert</code>
and <code class="language-plaintext
highlighter-rouge">dfdl:discriminator</code> statement annotations</li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext
highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or
a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code>
default value expression when
+that variable being set/defaulted is itself referenced from a another
expression being
+accessed at parser creation time, and the variable being set/defaulted has
<code class="language-plaintext highlighter-rouge">dfdlx:direction</code>
+of <code class="language-plaintext highlighter-rouge">parseOnly</code></li>
+</ul>
+
+<p>Unparser-specific expressions include:</p>
+
+<ul>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:outputValueCalc</code></li>
+ <li><code class="language-plaintext highlighter-rouge">dfdl:length</code>
(when `dfdl:lengthKind='explicit')</li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:outputNewLine</code></li>
+ <li><code class="language-plaintext
highlighter-rouge">dfdl:setVariable</code>, a <code class="language-plaintext
highlighter-rouge">dfdl:newVariableInstance</code> default value expression, or
a
+<code class="language-plaintext highlighter-rouge">dfdl:defineVariable</code>
default value expression when
+that variable being set/defaulted is itself referenced from a another
expression being
+accessed at unparser creation time, and the variable being set/defaulted has
<code class="language-plaintext highlighter-rouge">dfdlx:direction</code>
+of <code class="language-plaintext highlighter-rouge">unparseOnly</code></li>
+</ul>
-<h2 id="extended-behaviors">Extended Behaviors</h2>
+<h3 id="enumerations-dfdlxreptype-dfdlxrepvalues">Enumerations: <code
class="language-plaintext highlighter-rouge">dfdlx:repType</code>, <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code></h3>
+
+<p>These properties work together to allow DFDL schemas to define
<em>enumerations</em>;
+that is, symbolic representations for integer constants.
+When parsing, Daffodil will convert these integers into the corresponding
string values.
+When unparsing, Daffodil will convert strings into the corresponding
integers.</p>
+
+<p>An element of type (or derived from) <code class="language-plaintext
highlighter-rouge">xs:string</code> can be defined using XSD <code
class="language-plaintext highlighter-rouge">enumeration</code> facets
+which constrain the valid values of this string.
+These enumeration values are effectively symbolic constants.
+The <code class="language-plaintext highlighter-rouge">dfdlx:repType</code>
and <code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
properties are then used to define the correspondence of
+the symbolic strings to the corresponding integer values.</p>
+
+<h4 id="dfdlxreptype"><code class="language-plaintext
highlighter-rouge">dfdlx:repType</code></h4>
+
+<p>The value of this property is an XSD QName of a simple type definition that
must be derived
+from <code class="language-plaintext highlighter-rouge">xs:int</code>, or
<code class="language-plaintext highlighter-rouge">xs:unsignedInt</code>.
+A simple type definition for a string can be annotated with <code
class="language-plaintext highlighter-rouge">dfdlx:repType</code>
+in order to declare that the representation of the string is not as text
characters but is a
+numeric integer value.
+The type referenced from <code class="language-plaintext
highlighter-rouge">dfdlx:repType</code> is usually a fixed length binary
integer, but can be any
+DFDL type derived from <code class="language-plaintext
highlighter-rouge">xs:int</code> or <code class="language-plaintext
highlighter-rouge">xs:unsignedInt</code>, with any DFDL representation
properties.</p>
+
+<p>The mapping between the representation integer and the symbolic constants
is specified using the
+<code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
property.</p>
+
+<h4 id="dfdlxrepvalues"><code class="language-plaintext
highlighter-rouge">dfdlx:repValues</code></h4>
+
+<p>The value of this property is one or more integer values within
+the numeric range defined for the type referenced by <code
class="language-plaintext highlighter-rouge">dfdlx:repType</code>. When more
than one value
+is specified, they are in a whitespace separated list.</p>
+
+<p>This property is placed on the <code class="language-plaintext
highlighter-rouge">xs:enumeration</code> facets of a symbolic string constant
having a
+<code class="language-plaintext highlighter-rouge">dfdlx:repType</code>.
+At parse time, if the value of the <code class="language-plaintext
highlighter-rouge">dfdlx:repType</code> integer is found within the <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
+list, then the infoset value for the symbolic string gets the corresponing
enumeration facet value.
+It is a parse error if the <code class="language-plaintext
highlighter-rouge">dfdlx:repType</code> integer is not found in any of the
<code class="language-plaintext highlighter-rouge">dfdlx:repValues</code>
+lists of the <code class="language-plaintext
highlighter-rouge">xs:enumeration</code> facets.
+At unparse time, the symbolic constant is mapped to the first integer in the
<code class="language-plaintext highlighter-rouge">dfdlx:repValues</code> list.
+It is an unparse error if the symbolic string value is not found among the
<code class="language-plaintext highlighter-rouge">xs:enumeration</code>
+facet values of the symbolic string type.</p>
+
+<h4 id="examples-of-enumerations-in-daffodil-dfdl">Examples of Enumerations in
Daffodil DFDL</h4>
+
+<p>A simple example of a basic enum is:</p>
+
+<pre><code class="language-xsd"> <simpleType name="rep3Bit"
dfdl:lengthUnits="bits" dfdl:length="3" dfdl:lengthKind="explicit">
+ <restriction base="xs:unsignedInt"/>
+ </simpleType>
+
+ <simpleType name="precedenceEnum" dfdlx:repType="pre:rep3Bit">
+ <restriction base="xs:string">
+ <enumeration value="Reserved_0" dfdlx:repValues="0"/>
+ <enumeration value="Reserved_1" dfdlx:repValues="1"/>
+ <enumeration value="Emergency" dfdlx:repValues="2"/>
+ <enumeration value="Reserved_3" dfdlx:repValues="3"/>
+ <enumeration value="Flash" dfdlx:repValues="4"/>
+ <enumeration value="Immediate" dfdlx:repValues="5"/>
+ <enumeration value="Priority" dfdlx:repValues="6"/>
+ <enumeration value="Routine" dfdlx:repValues="7"/>
+ </restriction>
+ </simpleType>
+</code></pre>
+
+<p>Above we see the <code class="language-plaintext
highlighter-rouge">dfdlx:repType</code> is <code class="language-plaintext
highlighter-rouge">rep3Bit</code> which is a 3 bit <code
class="language-plaintext highlighter-rouge">xs:unsignedInt</code>. This can
+represent the values 0 to 7 which one can see are the <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code> of the
<code class="language-plaintext highlighter-rouge">xs:enumeration</code>
+facets for this enumeration string type which is named <code
class="language-plaintext highlighter-rouge">precedenceEnum</code>.</p>
+
+<p>In the above you can also see that the symbolic strings are in one-to-one
correspondence with
+every possible value of the 3-bit representation integer.
+This one-to-one correspondence assures that data that is first parsed and then
unparsed will
+recreate the exact numeric bits used.</p>
+
+<p>However, in data security applications the following may be preferred:</p>
+<pre><code class="language-xsd"> <simpleType name="precedenceEnum"
dfdlx:repType="pre:rep3Bit">
+ <restriction base="xs:string">
+ <enumeration value="Reserved" dfdlx:repValues="0 1 3"/>
+ <enumeration value="Emergency" dfdlx:repValues="2"/>
+ <enumeration value="Flash" dfdlx:repValues="4"/>
+ <enumeration value="Immediate" dfdlx:repValues="5"/>
+ <enumeration value="Priority" dfdlx:repValues="6"/>
+ <enumeration value="Routine" dfdlx:repValues="7"/>
+ </restriction>
+ </simpleType>
+</code></pre>
+
+<p>In the above we see that three numeric values, 0, 1, and 3 are the <code
class="language-plaintext highlighter-rouge">dfdlx:repValues</code> mapped to
+the symbolic string <code class="language-plaintext
highlighter-rouge">Reserved</code>.
+This technique has the advantage of blocking covert signals being transmitted
by use of the
+different reserved values since when unparsed, the constant string <code
class="language-plaintext highlighter-rouge">Reserved</code> will always be
+<em>canonicalized</em> to integer 0.
+Putting data into canonical form when unparsing generally improves data
security.</p>
+
+<h2 id="extended-behaviors-for-dfdl-types">Extended Behaviors for DFDL
Types</h2>
<h3 id="type-xshexbinary">Type <code class="language-plaintext
highlighter-rouge">xs:hexBinary</code></h3>
<p>Daffodil allows <code class="language-plaintext
highlighter-rouge">dfdlx:lengthUnits='bits'</code> for this simple type.</p>
+<hr />
+<div class="footnotes" role="doc-endnotes">
+ <ol>
+ <li id="fn:moreProps">
+ <p>New properties added as part of errata corrections to the DFDL v1.0
standard which take expressions for their values will need to be added to this
list or those for parser-specific or unparser-specific properties. <a
href="#fnref:moreProps" class="reversefootnote"
role="doc-backlink">↩</a></p>
+ </li>
+ </ol>
+</div>
+
</div>
</div>
diff --git a/content/layers/index.html b/content/layers/index.html
index 87511d6..41258fb 100644
--- a/content/layers/index.html
+++ b/content/layers/index.html
@@ -100,16 +100,18 @@
<!--
-->
-
<h3 class="no_toc" id="table-of-contents">Table of Contents</h3>
+<!-- The {: .no_toc } excludes the above heading from the ToC -->
<ol id="markdown-toc">
- <li><a href="#introduction"
id="markdown-toc-introduction">Introduction</a></li>
- <li><a href="#built-in-layers" id="markdown-toc-built-in-layers">Built-in
Layers</a></li>
- <li><a href="#custom-plug-in-layers"
id="markdown-toc-custom-plug-in-layers">Custom Plug-In Layers</a></li>
- <li><a href="#layer-kinds-transforming-layers-and-checksum-layers"
id="markdown-toc-layer-kinds-transforming-layers-and-checksum-layers">Layer
Kinds: Transforming Layers and Checksum Layers</a> <ol>
- <li><a href="#transforming-layers"
id="markdown-toc-transforming-layers">Transforming Layers</a></li>
- <li><a href="#checksum-layers"
id="markdown-toc-checksum-layers">Checksum Layers</a></li>
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a>
<ol>
+ <li><a href="#built-in-layers"
id="markdown-toc-built-in-layers">Built-in Layers</a></li>
+ <li><a href="#custom-plug-in-layers"
id="markdown-toc-custom-plug-in-layers">Custom Plug-In Layers</a></li>
+ <li><a href="#layer-kinds-transforming-layers-and-checksum-layers"
id="markdown-toc-layer-kinds-transforming-layers-and-checksum-layers">Layer
Kinds: Transforming Layers and Checksum Layers</a> <ol>
+ <li><a href="#transforming-layers"
id="markdown-toc-transforming-layers">Transforming Layers</a></li>
+ <li><a href="#checksum-layers"
id="markdown-toc-checksum-layers">Checksum Layers</a></li>
+ </ol>
+ </li>
</ol>
</li>
<li><a href="#using-layers" id="markdown-toc-using-layers">Using Layers</a>
<ol>
@@ -121,11 +123,8 @@
</li>
</ol>
</li>
- <li><a href="#examples" id="markdown-toc-examples">Examples</a> <ol>
- <li><a href="#line-folding" id="markdown-toc-line-folding">Line
Folding</a></li>
- <li><a href="#base64-gzip-and-boundarymark-layers-used-together"
id="markdown-toc-base64-gzip-and-boundarymark-layers-used-together">Base64,
GZip, and BoundaryMark Layers used Together</a></li>
- </ol>
- </li>
+ <li><a href="#example-line-folding"
id="markdown-toc-example-line-folding">Example: Line Folding</a></li>
+ <li><a href="#example-base64-gzip-and-boundarymark-layers-used-together"
id="markdown-toc-example-base64-gzip-and-boundarymark-layers-used-together">Example:
Base64, GZip, and BoundaryMark Layers used Together</a></li>
<li><a href="#using-custom-plug-in-layers"
id="markdown-toc-using-custom-plug-in-layers">Using Custom Plug-In
Layers</a></li>
<li><a href="#daffodil-built-in-layer-documentation"
id="markdown-toc-daffodil-built-in-layer-documentation">Daffodil Built-In Layer
Documentation</a> <ol>
<li><a href="#base64-mime-layer"
id="markdown-toc-base64-mime-layer">Base64 MIME Layer</a></li>
@@ -142,6 +141,7 @@
</ol>
</li>
</ol>
+<!-- note the above line {:toc} cannot have whitespace at the start -->
<h2 id="introduction">Introduction</h2>
@@ -170,7 +170,7 @@ There is no limit to this depth.</p>
<p>In the section on <a href="#UsingLayers">Using Layers</a> below we will
look at an example that uses
multiple layers together.</p>
-<h2 id="built-in-layers">Built-in Layers</h2>
+<h3 id="built-in-layers">Built-in Layers</h3>
<p>Daffodil includes several built-in layers:</p>
<ul>
@@ -194,20 +194,20 @@ These are:</p>
<a href="#daffodil-built-in-layer-documentation">documented separately
below</a> with examples of their
usage.</p>
-<h2 id="custom-plug-in-layers">Custom Plug-In Layers</h2>
+<h3 id="custom-plug-in-layers">Custom Plug-In Layers</h3>
<p>Additional layers can be written in Java or Scala and deployed as
<em>plug-ins</em> for Daffodil.
These are generally packaged as DFDL <em>layer schemas</em>, a kind of
<em>component schema</em>,
that provide the layer packaged for import by other DFDL <em>assembly</em>
schemas that use the
layer in the data format they describe.</p>
-<h2 id="layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds:
Transforming Layers and Checksum Layers</h2>
+<h3 id="layer-kinds-transforming-layers-and-checksum-layers">Layer Kinds:
Transforming Layers and Checksum Layers</h3>
<p>There are two different kinds of layers, though they share many
characteristics. They are
<em>transforming</em> layers, and <em>checksum</em> layers. Both run small
algorithms over part (or all) of
the data stream. The difference is the purpose of the algorithm and its
output.</p>
-<h3 id="transforming-layers">Transforming Layers</h3>
+<h4 id="transforming-layers">Transforming Layers</h4>
<p>These layers decode data (when parsing), and encode data (when unparsing).
The simplest example of a transforming layer is the <code
class="language-plaintext highlighter-rouge">base64_MIME</code> layer which
@@ -222,7 +222,7 @@ They can also assign computed result values to DFDL
variables, though this is un
<a
href="/docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/Layer.html"><code
class="language-plaintext highlighter-rouge">Layer</code></a> class
which is introduced in a later section.</p>
-<h3 id="checksum-layers">Checksum Layers</h3>
+<h4 id="checksum-layers">Checksum Layers</h4>
<p>Checksum layers are a simplified kind of layer which do not decode or
encode data, they simply
pass through the data unmodified, but while doing so they compute a checksum,
hash, or Cyclic
@@ -245,6 +245,8 @@ DFDL schema, which uses a Daffodil layer to describe the
IPv4 packet header chec
html"><code class="language-plaintext
highlighter-rouge">ChecksumLayer</code></a>
class, which is introduced in a later section.</p>
+<hr />
+
<h2 id="using-layers">Using Layers</h2>
<p>To use a layer you must know</p>
@@ -320,9 +322,7 @@ use the DFDL <code class="language-plaintext
highlighter-rouge">layerLength</cod
<p>Layers may specify restrictions on the minimum and maximum allowed values
of these lengths,
and passing an out-of-range value for the variable is a processing error.</p>
-<h2 id="examples">Examples</h2>
-
-<h3 id="line-folding">Line Folding</h3>
+<h2 id="example-line-folding">Example: Line Folding</h2>
<p>Consider the line folding layer, specifically the <code
class="language-plaintext highlighter-rouge">lineFolded_IMF</code> layer,
which is built-in to Daffodil.</p>
@@ -338,7 +338,7 @@ a limited line length.</p>
</code></pre></div></div>
<p>This data has been <em>line folded</em> at roughly 72 characters by
inserting a CRLF
before an existing space in the data.
-Each line ends with a CRLF (\r\n) and the second through fourth lines begin
+Each line ends with a CRLF (<code class="language-plaintext
highlighter-rouge">\r\n</code>) and the second through fourth lines begin
with a space as a way of indicating that they are extension lines.
This data is supposed to be reassembled to form a long single-line string by
removing
all CRLF pairs.</p>
@@ -379,7 +379,7 @@ Other examples will show how the layer length can be
limited to a sub-region of
<p>More detailed documentation for the <a href="#line-folded-layers">Line
Folded Layers</a> is below.</p>
-<h3 id="base64-gzip-and-boundarymark-layers-used-together">Base64, GZip, and
BoundaryMark Layers used Together</h3>
+<h2 id="example-base64-gzip-and-boundarymark-layers-used-together">Example:
Base64, GZip, and BoundaryMark Layers used Together</h2>
<p>In this example, the data consists of a preliminary string, a section of
CSV-like data, and a
final string element.
@@ -539,6 +539,8 @@ This group definition is the last thing in the schema:</p>
</code></pre></div></div>
<p>The above schema works both to parse, but also to unparse this data.</p>
+<hr />
+
<h2 id="using-custom-plug-in-layers">Using Custom Plug-In Layers</h2>
<p>A custom plug-in layer is used in the same manner as the built-in Daffodil
layers with just a few
@@ -571,6 +573,9 @@ base class.</p>
<p>Further details on how to define custom plug-in layers is in the Javadoc
for the
<a
href="/docs/latest/javadoc/org/apache/daffodil/runtime1/layers/api/package-summary.html">Layer
API</a></p>
+<hr />
+<hr />
+
<h2 id="daffodil-built-in-layer-documentation">Daffodil Built-In Layer
Documentation</h2>
<p>Each of the layers built-in to the Daffodil implementation are documented
in a section below
@@ -586,6 +591,8 @@ which gives the name, namespace, variables, and some usage
notes.</p>
<li><a href="#line-folded-layers">lineFolded_iCalendar</a></li>
</ul>
+<hr />
+
<h3 id="base64-mime-layer">Base64 MIME Layer</h3>
<ul>
@@ -603,11 +610,13 @@ which gives the name, namespace, variables, and some
usage notes.</p>
<p>This is specified by <a href="https://www.ietf.org/rfc/rfc2045.txt";>RFC
2045</a>.
The encoded output must be represented in lines of no more than 76 characters
-each and uses a carriage return '\r' followed immediately by a linefeed '\n'
as the line separator.
+each and uses a carriage return <code class="language-plaintext
highlighter-rouge">\r</code> followed immediately by a linefeed <code
class="language-plaintext highlighter-rouge">\n</code> as the line separator.
No line separator is added to the end of the encoded output.
All line separators or other characters not found in the base64 alphabet table
are ignored in
decoding operation.</p>
+<hr />
+
<h3 id="boundarymark-layer">BoundaryMark Layer</h3>
<ul>
@@ -657,6 +666,8 @@ of any child element enclosed within the layer, or even the
lengths of other lay
within the scope of this boundary mark layer are not considered and do not
disrupt the search
for the boundary mark string.</p>
+<hr />
+
<h3 id="byte-swapping-layers">Byte-Swapping Layers</h3>
<ul>
@@ -698,6 +709,8 @@ order 2 1 4 3 6 5 8 7 10 9.</p>
<p>If <code class="language-plaintext
highlighter-rouge">requireLengthInWholeWords</code> is bound to "yes", then if
the length is not a multiple of the
word size a processing error occurs.</p>
+<hr />
+
<h3 id="fixedlength-layer">FixedLength Layer</h3>
<ul>
@@ -721,6 +734,8 @@ The entire fixed length region of the data will be pulled
into a byte buffer in
</li>
</ul>
+<hr />
+
<h3 id="gzip-layer">GZIP Layer</h3>
<ul>
@@ -745,6 +760,8 @@ depending on the Java version used.
To avoid inconsistent behavior of test failures that expect a certain byte
value this layer
always writes a consistent header (header byte 9 of 255) regardless of the
Java version.</p>
+<hr />
+
<h3 id="line-folded-layers">Line Folded Layers</h3>
<ul>
@@ -762,18 +779,17 @@ always writes a consistent header (header byte 9 of 255)
regardless of the Java
<div class="language-plaintext highlighter-rouge"><div
class="highlight"><pre class="highlight"><code> <xs:import
namespace="urn:org.apache.daffodil.layers.lineFolded"
schemaLocation="/org/apache/daffodil/layers/xsd/lineFoldedLayer.dfdl.xsd"/>
</code></pre></div> </div>
+ <h4 id="general-usage">General Usage</h4>
</li>
</ul>
-<h4 id="general-usage">General Usage</h4>
-
<p>There is a limitation on the compatibility of line folding of data
with adjacent parts of the format which also use line-endings.
For example, line folding can interact badly with surrounding elements of
<code class="language-plaintext highlighter-rouge">dfdl:lengthKind
'pattern'</code> if the pattern is, for example <code
class="language-plaintext highlighter-rouge">".*?\\r\\n(?!(?:\\t|\\ ))"</code>
which is anything up to
and including a CRLF not followed by a space or tab.
The problem is that line folding
-converts isolated \n or \r into \r\n, and if this just happens to be followed
by a
+converts isolated <code class="language-plaintext highlighter-rouge">\n</code>
or <code class="language-plaintext highlighter-rouge">\r</code> into <code
class="language-plaintext highlighter-rouge">\r\n</code>, and if this just
happens to be followed by a
non space/tab character this will have inserted an end-of-data in the middle
of the
data.</p>
diff --git a/content/unsupported/index.html b/content/unsupported/index.html
index 677a54e..b681e7e 100644
--- a/content/unsupported/index.html
+++ b/content/unsupported/index.html
@@ -133,7 +133,7 @@ but may return in a future version of the DFDL spec.)</li>
<ul>
<li>fixed [<a
href="https://issues.apache.org/jira/browse/DAFFODIL-117";>DAFFODIL-117</a>]</li>
- <li>default [<a
href="https://issues.apache.org/jira/browse/DAFFODIL-115";>DAFFODIL-115</a>] [<a
href="https://issues.apache.org/jira/browse/DAFFODIL-1277";>DAFFODIL-1277</a>]</li>
+ <li>default [<a
href="https://issues.apache.org/jira/browse/DAFFODIL-115";>DAFFODIL-115</a>]</li>
</ul>
<h2 id="properties-and-property-enumerations">Properties and Property
Enumerations</h2>
