View Issue Details

IDProjectCategoryView StatusLast Update
000711910000-006: MappingsApi Changepublic2021-08-31 16:41
ReporterDavid Levine Assigned ToRandy Armstrong  
PriorityurgentSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Summary0007119: When a server contains DataType definitions using both a DataTypeDictionary (v1.03) and a DataTypeDefinition attribute (v1.04),
Description

For compatibility with old and new clients, servers will provide definitions using both v1.03 and v1.04 definitions. Even though the mechanics of accessing the definitions are different, the encodings provide the same field information except when a field is a matrix. v1.03 only provided the metadata for a 1 dimension array (length) and v1.04 provides the definition for a matrix (contains the ArrayDimensions). The value is encoded differently for each.

There is no API a client stack can use to choose the version it wants the server to use and a default choice is not defined, thus there are no rules a decoder can rely on to determine how it should decode a value other than to use the encoding NodeId that is sent with the value and use the appropriate definition. Even if the stack supports both types of encodings, this requires that the stack store both definitions in its memory. The server could switch between definitions with each value and still be in compliance with the spec.

If the stack the client is using makes any assumptions at all about which definition it should use, it may "guess wrong" and either generate decoding errors or (worse) incorrectly decode the value and assume it worked. If the client only supports one type of definition it may be different than what the server is encoding the value as, creating a compatibility issue.

Desired outcome: when both versions are supported by the server, the client can choose the version of the encoding the server shall use.

1) A server can report which style encodings it supports: v1.03, v1.04, v1.xx (future) or a combination. This can be done a number of ways, one being to create a node below the Server Capabilities folder which contains the version information.
2) a client can select which version of the encoding it wishes the server to use. This can be part of the open session request, a method call, part of a MonitoredItem request, or something else.
3) If a server does not support a client's requested encoding version, the server returns an error, An alternative is the server returns the version number that it will use. If it is different than what the client requested, the client can choose to use that version or terminate the session.

Use case (actual):
A server contains both v1.03 and v1.04 definitions. The server contains a custom type which has a field that is a matrix. The v1.04 definition includes the ArrayDimension of the matrix, while the v1.03 definition defined it as an array whose length matches the sum of the length of all dimensions. The stack only used the v1.04 definitions.

The values are transmitted using v1.03 encodings (the array dimensions are not sent). The stack tries to decode it using v1.04 definition because that was what it expected. This fails because the length of the transmitted data did not match the expected length (this is good because erroneous data is not forwarded to the application, but bad because it did not work.).

End result: client and server are incompatible.

Even if the client stack switched to using the v1.03 definition it would still be "wrong" because the client application expects a matrix (i.e. its notation for an element uses matrix notation) but the server sent an array (1D). This is a mismatch between the design-time information and the runtime value.

It is possible to convert between the two forms, but this is an additional step that must be taken and which can be done incorrectly. To do the conversion from a flat buffer (i.e. a 1D array) to a matrix the array dimensions must be known.

TagsNo tags attached.
Commit Version
Fix Due Date

Relationships

related to 0007148 closedRandy Armstrong Encoding of multidimensional arrays is not clear 

Activities

Jim Luth

2021-07-20 17:11

administrator   ~0014696

Randy to work on example of a 1.04 Definition that uses the new first-class Matrix concept and how that would look if described by the 1.03 DataTypeDictionary.

David Levine

2021-08-02 16:20

developer   ~0014713

Randy is working on a POC to show how to encode a matrix field in v1.03 type definition dictionary. At a high level, applications must be able to interpret the metadata from the type definition as matrix field.
v1.04 type definitions support matrices in structured fields, but v1.03 does not have a defined way to support this.

Desired outcome:
1) If POC shows this is possible, then define normative requirements for clients and servers on how to support matrices in structured fields using v1.03 type dictionaries.
or
2) If POC shows it is not possible, then define normative (testable) requirements for v1.03 type definitions (e.g. a type with matrix field shall not be included in v1.03 dictionary).
or
a) If support for this is needed in v1.03 type definitions then this must be added to the spec.

IMO acceptance criteria for a successful POC that demonstrates how to define a matrix field in a v1.03 type dictionary:

  • For both v1.03 and v1.04 type definitions, the metadata used by applications shall unambiguously identify the field as a matrix. i.e. the array dimensions of each rank shall be identified.
  • The encoded value on-the-wire must be identical (bit-wise) regardless of which definition is used, v1.03 or v1.04. This allows a mix of servers & clients using different type definitions to successfully exchange data.
  • The encoded value can be decoded by both v1.03 and v1.04 decoders

This is a high priority for us:
1) there are shipping toolkits and UA servers that have implemented this, perhaps incorrectly.
2) Our toolkit vendor is currently adding support for matrix fields and correct behavior is TBD for v1.03 type defs.

Use cases:

  • Server supports both v1.03 and v1.04 type definitions.
  • Server supports only v1.03 or only v1.04 type definitions
  • Client only supports v1.03
  • Client only supports v1.04

Randy Armstrong

2021-08-14 11:17

administrator   ~0014746

It appears that there is no definition in the specification that explains how to encode an inline Matrix.

There is a Matrix that can only be used within a Variant and it could be adapted for inline use but the specification does not say you should do that.

This issue cannot be closed until we agree on a binary/XML/JSON representation for an inline matrix.

Jim Luth

2021-08-17 19:21

administrator   ~0014754

We discussed Randy’s findings and agreed that Randy will Modify Part 6 to state that the 1.04 inline matrix encoding is incompatible with previous versions and if used the structure shall not be described in the Data Dictionary. If compatibility with previous versions is desired, a Variant should be used instead of the 1.04 inline matrix since a Variant can already contain a matrix and is compatible with all previous versions.

Randy Armstrong

2021-08-18 05:08

administrator   ~0014756

Added inline matrix with explanation of backward compatibility issues.

Jim Luth

2021-08-31 16:41

administrator   ~0014792

Agreed to changes in edited 1.05.01 Draft 4.

Issue History

Date Modified Username Field Change
2021-07-19 15:35 David Levine New Issue
2021-07-20 17:09 Jim Luth Assigned To => Randy Armstrong
2021-07-20 17:09 Jim Luth Status new => assigned
2021-07-20 17:11 Jim Luth Note Added: 0014696
2021-07-20 17:11 Jim Luth Project Feature Requests => 10000-006: Mappings
2021-07-20 17:11 Jim Luth Category IOP Problem => Api Change
2021-08-02 16:20 David Levine Note Added: 0014713
2021-08-14 11:17 Randy Armstrong Note Added: 0014746
2021-08-17 19:21 Jim Luth Note Added: 0014754
2021-08-18 05:08 Randy Armstrong Status assigned => resolved
2021-08-18 05:08 Randy Armstrong Resolution open => fixed
2021-08-18 05:08 Randy Armstrong Note Added: 0014756
2021-08-18 09:10 Randy Armstrong Relationship added related to 0007148
2021-08-31 16:41 Jim Luth Status resolved => closed
2021-08-31 16:41 Jim Luth Note Added: 0014792