View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
000580010000-014: PubSubSpecpublic2020-07-15 16:262022-06-22 06:50
ReporterMartin Regen Assigned ToMatthias Damm  
PrioritynormalSeverityminorReproducibilityN/A
Status closedResolutionfixed 
Target Version1.05.02 RC1Fixed in Version1.05.02 RC1 
Summary0005800: For the MQTT PubSub encoding, specify a gzipped version of the body/payload for huge cost savings
Description

Cost is an issue when sending data to the cloud. Especially sending JSON can become quite costly when data is metered at 4kb message boundaries.
For publishing large number of nodes, it would be highly beneficial to allow to send the JSON message body or the UADP message body as a gzipped version.

In the latest spec with MQTT Version 5.0, to support the gzipped payload , it would be possible to add the ContentType 'application/json+gzip' for JSON and application/opcua+uadp+gzip for UADP.

The saving as measured with JSON were seen between 2 and 15 times smaller payload with gzip.
For JSON it could be a huge cost factor when ten thousands of nodes are being monitored.

For binary still saving by a factor of two might be possible, but we need to measure the compression factor with some real payload. It may even make things worse, need to verify.

RFC 1952 --> GZIP compression

7.3.5.8 Message body
7.3.5.8.1 General
When sending a MQTT Version 5.0 message the UAMessageType property shall be set to uadata for data messages or ua-metadata for metadata messages.
7.3.5.8.2 JSON message mapping
A JSON body is encoded as defined for the JSON message mapping defined in 7.2.3.
When sending a MQTT Version 5.0 message the MQTT ContentType property shall be set to application/json. <-- add gzip content type
7.3.5.8.3 UADP message mapping
A UADP body is encoded as defined for the UADP message mapping defined in 7.2.2.
It is expected that the software used to receive UADP NetworkMessage can process the body without needing to know how it was transported.
If the encoded MQTT message size exceeds the Broker limits, it is broken into multiple chunks as described in 7.2.2.4.4.
When sending such message over MQTT Version 5.0 the ContentType property shall be set to application/opcua+uadp. <-- add gzip content type
It is recommended that the MetaDataQueueName as described in 6.4.2.5.5 is configured as a sub-topic of the related QueueName with the name “$Metadata”. The MQTT RETAIN flag shall be set for metadata messages.

Additional Information

Note --> the numbers seen need to be verified with an optimized JSON or UADP payload. I'm working with the team on getting some good numbers for discussion, e.g. network message with 1000 nodes.

Update: I got some numbers with simulated JSON data as RawValues and DataValues because our test server created nodes with almost identical names.
Test payloads can be provided on demand.

From lcantero@microsoft.com
Here are the result for rawvalues (worst-case, compressed by ~2.2x):
Compression ratio: 220 % Original size: 19001 bytes Result size: 8627 bytes

Here are the results for pubsub with changed ids and random values (compressed by ~10x):
Compression ratio: 1026 % Original size: 159021 bytes Result size: 15502 bytes
(Here I changed the Ids to be increasing numbers and randomized the values. I did not change the timestamps as an OPC reading will probably include similar timestamps. What do you think?)

Here are the (similar) results for PubSub mode + Json encoding:
Compression ratio: 1526 % Original size: 163137 bytes Result size: 10690 bytes

So I think we could compress by 2 to 15 times and be somewhere in the middle in average, which is still a lot of saving.

Thanks,
Luis

TagsNo tags attached.
Commit Version
Fix Due Date

Activities

Martin Regen

2020-12-09 18:07

developer   ~0013432

Document as discussed in the WG meeting is here: https://opcfoundation.sharepoint.com/UA/work/Meetings/2020/2020-12-07%20F2F%20Virtual/OPC%2010000-14%20-%20UA%20Specification%20Part%2014%20-%20PubSub%201.05.0%20mregen%20jsongzip.docx

Addition was accepted but a new TransportProfileUri is required to select the json+gzip format

The deflat and brotli formats were dropped as they just increase the IOP test matrix.

Matthias Damm

2022-01-17 15:46

developer   ~0015732

There is also a requirement to batch multiple NetworkMessages and to compress them.
See PubSub prototyping meeting minutes 2022-01-17

Martin Regen

2022-01-18 07:22

developer   ~0015761

Adding the text as it was accepted in the WG meeting on 12/9/2020 from the word doc:

7.3.5.8.2 JSON message mapping
A JSON body is encoded as defined for the JSON message mapping defined in 7.2.3.
When sending a MQTT Version 5.0 message the MQTT ContentType property shall be set to application/json.
JSON messages can become quite large, in order to save bandwidth and to reduce message size, on MQTT Version 5.0 the MQTT ContentType property allows to select a compression type as encoding for a JSON message. Applying a compression on a message is optional and can be chosen by the publisher application at send time, to ensure that the applied compression is not increasing the size of a message. The application programs responsible for sending and receiving messages are responsible for supporting the compression type at both ends.
A broker may provide services with topics for compressed and uncompressed JSON messages to support application programs which do not support message compression.
When sending a gzip (RFC 1952) compressed JSON message on MQTT Version 5.0 the MQTT ContentType property shall be set to application/json+gzip.

Matthias Damm

2022-02-18 16:12

developer   ~0016058

Add following text to 1.05.02
7.3.5.8.2 JSON message mapping

JSON messages can become quite large, in order to save bandwidth and to reduce message size, on MQTT Version 5.0 the MQTT ContentType property allows to select a compression type as encoding for a JSON message. Applying a compression on a message is optional and can be chosen by the publisher application at send time, to ensure that the applied compression is not increasing the size of a message. The application programs responsible for sending and receiving messages are responsible for supporting the compression type at both ends.
A broker may provide services with topics for compressed and uncompressed JSON messages to support application programs which do not support message compression.
When sending a gzip (RFC 1952) compressed JSON message on MQTT Version 5.0 the MQTT ContentType property shall be set to application/json+gzip.

Matthias Damm

2022-02-23 13:52

developer   ~0016078

Added proposed text to
OPC 10000-14 - UA Specification Part 14 - PubSub 1.05.02 Draft1.docx

Jim Luth

2022-06-22 06:50

administrator   ~0016965

Agreed to changes edited in Munich F2F.

Issue History

Date Modified Username Field Change
2020-07-15 16:26 Martin Regen New Issue
2020-07-15 16:26 Martin Regen Status new => assigned
2020-07-15 16:26 Martin Regen Assigned To => Matthias Damm
2020-07-16 06:39 Martin Regen Description Updated
2020-07-16 06:41 Martin Regen Additional Information Updated
2020-07-23 06:29 Martin Regen Description Updated
2020-07-23 06:29 Martin Regen Additional Information Updated
2020-07-23 06:30 Martin Regen Additional Information Updated
2020-12-09 18:07 Martin Regen Note Added: 0013432
2022-01-17 15:46 Matthias Damm Note Added: 0015732
2022-01-18 07:22 Martin Regen Note Added: 0015761
2022-02-18 16:12 Matthias Damm Note Added: 0016058
2022-02-18 16:12 Matthias Damm Target Version => 1.05.02 RC1
2022-02-23 13:52 Matthias Damm Status assigned => resolved
2022-02-23 13:52 Matthias Damm Resolution open => fixed
2022-02-23 13:52 Matthias Damm Fixed in Version => 1.05.02 RC1
2022-02-23 13:52 Matthias Damm Note Added: 0016078
2022-06-22 06:50 Jim Luth Status resolved => closed
2022-06-22 06:50 Jim Luth Note Added: 0016965