Investigation continues, however as a first step, ensure that the settings \Advanced\Test Tool\Address Space Model\MaxNodeInAddressSpaceCache is set to a value that is large enough to gather all alarm information.

The default value is often not large enough to discover all alarms in the model.

The CTT is working to expectations.

Issue 1.
Not necessarily true.
The first error for Test_002 states that there were 15 errors for OffNormalAlarmType, however there were 3 tests that passed.
Likewise for NonExclusiveLevel alarm there were two tests that passed, and for ExclusiveLevel, there was one test that passed.
The Script Output should contain more details about the tests.

Issue 2.
The conformance units depend on a constant stream of alarms. Considering that the potential problem is on the first test of the Acknowledge conformance unit, there is no way this could be blocked by another test, it appears that there are no alarms happening during the Acknowledge conformance unit.

Issue 3.
Basic Test_002 does not look at an instance, it can only look at the event that was generated. It looks at all the expected fields, and compares it to the specification in \library\Information\NodeSet2.xml\Opc.Ua.NodeSet2.PartX.js. If a mandatory field has a datatype different than the one expected in the specification, an error would be generated (\library\AlarmsAndConditions\AlarmUtilities.js, function TestMandatoryProperty).

Firstly, thank you for your quick response.

For your notes:
1) Issue 1: yes, you're right. Alarms below AlarmsWithNodes are also detected.
2) Issue 2: I re-checked today and get more information. Will explain below and list my new findings.*
3) Issue 3: Guess you're correct. My understanding is wrong. I expected CTT looks at the instance.

My new findings and doubts:
1) Inconsistent test conclusions in Results and Script Output.
I compared all the tests that are shown as "Skipped" in Result with their corresponding outputs in Script Output.
The Script Output can be divided into three categories and you can find the details with the attached Excel and twice Script Output.

• Tests passed, no tests skipped. For example, script output for A and C Basic Test_003. ---> Does this mean the test is passed actually?
• Tests skipped. ---> Because no alarms found? I used automatic script to trigger alarms constantly so why would no alarms be detected…
• No summary. Only description, like "CheckResults Test_003: key i=2782". For example, script output for A and C Alarm Test_003. ---> So how should testers handle these outputs? Treat it as passed or failed?
  2) Inconsistent script outputs for the twice compliance tests.
  The only potential cause is for the 2nd compliance test, the automatic alarm triggering is started about 30 seconds later then the 1st. For both times, the automatic alarm triggering is started a little bit after compliance test is started.
  3) Test result for A and C Acknowledge Test_001
  If script outputs can be regarded as correct test results, Test_001 "Invoke Acknowledge method passing eventId as received and a unique comment" is skipped while other tests to invoke with empty comment or NULL comment are passed. I tried manually to acknowledge alarms with different kinds of comments and alarms are acknowledged successfully. So have no idea why this test are skipped. Because no alarms detected for this test?

Sorry that I have so many issues. The test setup requirements are complicated than the test for DA server.
Sorry for any inconvenience caused for your daily work.

New Findings One

• Tests passed, no tests skipped. For example, script output for A and C Basic Test_003. ---> Does this mean the test is passed actually?

Yes, this would mean that the test passed.

In initialization, the object model is read, and each alarm type in the type system is noted. For the majority of tests, the infrastructure will expect to test an alarm of each alarm type defined in the type system. If there is not a specific type of alarm, for example, a LimitAlarmType, the test would try to test for a LimitAlarmType. If it does not get any of this alarm type, the test is marked as skipped to show the user that this type cannot be tested.

The infrastructure is such that a success is determined automatically, and if there are skips, warnings or errors, the test itself cannot be marked as successful.

To show the actual results, each test that uses the TestEvent algorithm will have a CheckResult function that does show the results for each alarm type in the script output.

In Script output_CTT_1.txt, for Basic Test_003, the output is:

CheckResults Test_003: key i=2782
CheckResults Test_003: key i=2881
CheckResults Test_003: key i=2915
CheckResults Test_003: key i=10523
CheckResults Test_003: key i=10637
29 tests passed for OffNormalAlarmType 0 tests skipped
CheckResults Test_003: key i=11753
CheckResults Test_003: key i=13225
CheckResults Test_003: key i=19297
CheckResults Test_003: key i=2955
CheckResults Test_003: key i=9906
CheckResults Test_003: key i=10060
28 tests passed for NonExclusiveLevelAlarmType 0 tests skipped
CheckResults Test_003: key i=9341
CheckResults Test_003: key i=9482
30 tests passed for ExclusiveLevelAlarmType 0 tests skipped

• Tests skipped. ---> Because no alarms found? I used automatic script to trigger alarms constantly so why would no alarms be detected…

I can't know what is happening in the server, but the test will monitor the Server object for events.

There is a line in .\libraries\AlarmsAndConditions\AlarmCollector.js in the function this.SingleLoop that you could temporarily uncomment
print( "Single Loop Event Type " + eventType.toString() + " message " + this.GetSelectField( eventFields, "Message" ).toString() );

This will output every Event/Alarm that is received by the test. If this doesn't show the alarms that you are expecting to see, then I would verify that the server is actually sending them.

• No summary. Only description, like "CheckResults Test_003: key i=2782". For example, script output for A and C Alarm Test_003. ---> So how should testers handle these outputs? Treat it as passed or failed?

There is a missing status message that would say "No alarms discovered for SomeDefinedAlarmType"

Until this addressed, assume that no message after CheckResults Test_XXX: key i=yyyy means that no alarms were found for the alarm type.

New Findings Two

This is quite feasible. For most alarm tests, it is assumed that alarms will cycle through all alarm conditions including inactive on a regular cycle. There is a setting on Server Test\Alarms and Conditions\Alarm Cycle Time. Err on the side of caution. For example, if it takes 45 seconds to cycle all states and return to the original state, set this variable to more than 45 seconds.

New Findings Three

There are two reasons why this test may be skipped, one is the alarm is not acknowledgeable. This only affects ConditionType, so it is unlikely this is the culprit. The other would be that the alarm state is changing too fast, and a read of an instance has a modified state, which would cause a skip. There would be an activity message that describes this
"ConditionId" Has Changed. Unable to do test.

The test only expects changing alarm conditions, so it is not necessary to manually acknowledge alarms, it would disturb the test operations.

Thanks for the update. Now I can summarize the issues that I still get:
1) I still get many test cases skipped and have no idea why.
I attached the picture "Triggering alarms for CT" to show the structure of triggering DiscreteAlarmType, ExclusiveAlarmType, NonExclusiveAlarmType automatically and cyclically that I am using during CT.
I'm not sure if improper cyclic interval or delay between alarms caused test case skipped.
I tried with different intervals and got different results.
If too fast, as you said, would cause problem for alarm acknowledgement, but if too slow, like making interval too long, more test cases will be skipped with the result No alarms found.
For example, I tried with 30000 and 10000 for the interval of one cycle and get different results shown in the picture "Different results with different interval".
I wonder if you could give some recommendation for the alarm interval and I would use the result as baseline, or I cannot know whether I get the proper result for our compliance test.

2) As you mentioned, if the results are passed in Script output, then I can treat the test as Passed.
As required, I need to integrate the test result to DevOps pipeline with the file xxxx.results.xml.
The result file collects results from CTT Result, not Script Output, will lead to that the result does not looks good as most cases are marked as Skipped.
If there are any solutions or workarounds to optimize the CT result?
Thanks a lot. Looking forward to your reply.

Different results with different interval.png (110,363 bytes)   

Different results with different interval.png (110,363 bytes)   

Triggering alarms for CT.png (106,329 bytes)   

Triggering alarms for CT.png (106,329 bytes)   

Modified the AlarmCollector::CheckResults to addLog for visibility into how many tests passed and skipped.

Requires that Project setting Advanced\Test Tool\SuppressLogEntries be unchecked, as it is checked by default.

This modification is integrated in next version of CTT, is it right? Is it possible for me to modify something and use it with version 1.4.9.398? Thanks.

Reduced the number of skip messages based on the configured alarm source nodes at \Server Test\Alarms and Conditions\Supported ConditionTypes.

reviewed issue in call, agreed and closed