Types of purple teaming

PT types can vary in their purpose, learning experience, form, level of BT involvement and specificities. Most importantly, proper consideration needs to be given to when (test phase) and why (specific situation) they might be applied.

This section describes some possible types and examples of such PT, which might be used alone or in combination. However, since each TIBER test is different, they might be expanded or adapted according to the specific circumstances of a given test. The best approach when deciding which type of PT to select is for the relevant stakeholders to openly discuss the different alternatives. A detailed description of each type of PT is provided in Sections 5.1 and 5.2.

Because the selection of PT type is strongly dependent on the nature of the TIBER test in question, some of these types may only be suitable in specific situations. Used elsewhere, they might lead to invalidation of the test. This is crucial for the testing phase, as the TIBER-EU Framework requires the BT to be unaware of the engagement during this phase.

When planning PT, the following general aspects should be considered:

the purpose and learning experience for the entity, bearing in mind all the specificities of the entity and the test;
clear agreement on how to carry out PT;
the BT’s level of knowledge about the test;
the level of communication and bilateral channels between RT and BT;
the involvement of other stakeholders, such as the WT, TCT or even (additional) board members of the entity concerned.

Types of purple teaming in the testing phase

The types of PT during the testing phase are typically limited to activities conducted on the technical systems themselves, since tabletop activities are usually conducted in the closure phase and do not necessarily justify informing the BT about the ongoing test. Potential types of PT specific to the testing phase are described below.

Catch-and-release

“Catch-and-release” is a useful way of testing an entity’s defensive capabilities when there have been repeated detections by the BT during the final stage of a test. Because of such detection and consequent blocking of the accounts and tools involved, further progress in testing activities might not be possible without a significant change in TTPs. While different TTPs, such as choosing alternative routes into or through a network or different attack techniques, could be employed in the early stages of a test, this might not be feasible during the final stage. For example, if production systems have been successfully breached but exfiltration or modification of data is still outstanding, or when a route to the critical functions cannot be identified by the RT, a sudden change in TTPs might be counterproductive. In such situations, a catch-and-release approach might be carried out, enabling lessons to be learnt about very specific aspects of an attack that could not otherwise be achieved.

Catch-and-release is initiated by revealing to the BT that a test is being performed on its systems and installing a dedicated communication channel between the RT, BT and WT. In the event of any further detection within well-defined boundaries (e.g. certain machines or subnets), the BT uses this channel to report the detected Indicators of Compromise (IoC) to the RT, which confirms or refutes those as being part of their test. Note that special care should be taken by the WT to ensure that the BT blocks out confidential information that might not be related to the TIBER test. Should the identified IoCs indeed be part of the TIBER test, the BT will then perform the agreed measures to allow the test to continue (e.g. releasing an isolated machine or account). Alternatively, other previously agreed actions might also be taken, such as shutting down a machine, escalating the incident or starting a forensic investigation with the aim of evaluating these processes as part of the test.

The BT documents the events, together with all countermeasures or releases, for later reproduction and discussion. Importantly, specific guardrails should be defined in advance, specifying which machines (or subnets) are in scope of such types of PT as well as which responses actions cannot or should not be skipped.

Although the BT is aware of a test being performed on its production systems, this does not necessarily mean that the BT is aware that it is a TIBER test or what scenarios are covered by the test. During the complete PT phase, regular updates should take place to ensure adequate risk control and facilitate mitigation measures.

Collaborative proof-of-concept

“Proof-of-concept” can be a very helpful activity to provide evidence of a weakness discovered during TIBER tests in situations where practical testing on the production systems by the RT alone is not feasible (e.g. because of being out of scope, unjustified high risk of impacting critical systems, etc.). In some settings, a proof-of- concept might require the explicit involvement of the BT to provide a particular part of infrastructure expertise or risk control, for example. A collaborative proof-of-concept offers a very detailed learning experience related to a particular attack step or vulnerability.

Executing a collaborative proof-of-concept includes collecting all the evidence required to illustrate the feasibility of a certain attack vector without actually fully performing such an attack. This might include a theoretical discussion of the expected outcome of executing a given step as well as the protective measures in

place. A practical test of partial aspects of the attack is usually also carried out (e.g. sub-steps, using dummy data, execution on testing systems, etc.).

As for other, similar activities, close cooperation between RT and BT (also including relevant business areas where appropriate) is required to consider all offensive and defensive aspects of an attack. In most cases, proofs-of-concept are conducted during the closure phase to avoid undesired effects during the test itself, such as putting the BT on high alert. However, at a very late stage of testing, the remaining activity might depend on a particular proof-of-concept. BT staff might thus be informed about the test and asked to contribute when drawing up the proof-of- concept.

Other stakeholders, such as the WT and the TCT, should be closely involved to ensure the proof does indeed provide evidence of a weakness.

War game

A war game is an activity in which RT and BT are fully aware of each other’s respective goal to capture the respective flags (RT) or to protect the entity’s critical assets and terminate the attackers’ access to the network (BT). As such, war games differ from the spirit of a normal TIBER test in the sense that the BT knows what the RT flags are. If a war game is performed, flags are usually placed in systems underpinning critical functions included in the test scope.

In very particular situations, such as when a TIBER test is known to the BT at a very early stage of the RT, a war game might be a suitable option to continue the test and still enable a learning experience.

Types of purple teaming in the closure phase

Under the TIBER-EU Framework, the mandatory replay workshop is a chronological walk-through of each scenario. It serves to detail and discuss the actions of the RT, together with the relevant responses of the BT, step by step at an appropriate level of technical detail based on the RT and BT reports and logs, but without necessarily actively using the systems themselves (i.e. the replay is often conducted in the form of a tabletop discussion, see Figure 3).

In practice, the replay w orkshop is often complemented by additional types of PT to investigate selected aspects of a test in more detail and hence maximise the learning experience.¹ As such, different types of PT can either be combined with the mandatory replay workshop or carried out in separate workshops.

PT activities in the closure phase might focus on attack scenarios planned and executed during the testing phase or on alternative, more explorative approaches.

They can take the form of tabletop discussions as well as activities on the technical systems.

Deciding on which types of PT are best suited for a particular test depends on many factors, such as the results of the test, risk considerations, the intended type of learning experience and available resources. The types of PT described below can serve as a reference to ensure the consistent use of terminology by all parties involved. These are likely to be combined and blended in a way that best fits the situation at hand and the specificities of the entity being tested. Additional stakeholders, such as the internal RT of the entity tested, might also be included in a passive or active role. This can lead to additional learnings for all parties.

Alternative scenario: tabletop discussion

A tabletop discussion can provide valuable learning experiences where technical systems are not required or available for the review. This type of activity allows a less technical audience, including management, to be included more widely in the discussion. While the planned attack vectors have already been analysed in the mandatory replay workshop, a tabletop discussion is a great way to investigate alternative attack vectors and discuss or simulate the “what ifs” without a strict focus on technical systems. For example, a simulation might be used to discuss the entity’s response in case the method or pathway used by the attacker to infiltrate the system was successful. Such tabletop discussions offer high flexibility with regard to the tools used.

A tabletop discussion can be carried out in a variety of ways. These might include:

a role play to discuss and simulate alternative offensive and defensive measures and their consequences;
the theoretical evaluation of scenarios that are closely related but out of scope of a TIBER test;
the simulation of potential consequences reaching far beyond the test (e.g. restoring business continuity after a successful ransomware attack);
the inclusion of senior management.

Tabletop discussions might include many different stakeholders (such as business process owners) and therefore require thorough planning and moderation. However, they facilitate an all-round view of the wider aspects of an entity’s security and can even go far beyond the initial scope of the test.

Re-exploration of planned scenarios on live systems

A technical re-exploration of the attack vectors planned and executed during a TIBER test is an effective way to combine the expertise of the RT and the BT to practically show the offensive and defensive potential of an attack step or attack chain. Although this type of activity is quite resource-intensive in terms of preparation, it can deliver a highly comprehensive and detailed hands-on learning experience for the BT. For example, it might be very helpful in cases where the BT struggled to detect RT activities during a test. In this case, the RT and BT might engage in a collaborative attack recollection. Running through a chosen attack sequence step by step, the RT executes the respective TTPs while the BT can simultaneously provide corresponding defensive information (e.g. event notifications received, alerts, blocked executions, etc.). The two teams can together discuss (and document) the consequences of such an attack sequence and draw up preventive and/or reactive measures from their respective points of view.

This type of activity might include:

walking through an RT activity that was not visible in the BT logs during the testing phase;
walking through an RT activity that was visible in log entries, but the malicious activity was not detected by the BT during the testing phase;
walking through an RT activity that triggered an alert during testing but was not triaged properly;
walking through activities to which defensive responses were ineffective during testing;
walking through activities that triggered a defensive response that effectively closed the attack vector but was unable to prevent the attackers from meeting their objectives.

Alternative defensive measures and potential offensive countermeasures might be discussed and practically evaluated to achieve a good understanding of the different possibilities of attack and defence. This type of activity requires close cooperation between RT and BT as well as an open and explorative spirit.

Alternative scenario: exploration on live systems

Technical explorations of relevant attack scenarios during the closure phase might not be limited to merely evaluating scenarios conducted during the test phase. Variations of tested attack scenarios as well as novel or more elaborate scenarios are constantly inspired during a TIBER test. These can often not be comprehensively evaluated during the testing phase due to time and other constraints. For example, there might not be sufficient time to test a potential attack vector due to its late point of discovery during the testing period or a high risk of system damage or BT detection.

This attack vector might be explored during the closure phase, since more time is available and the close cooperation with the BT could significantly reduce the identified risks. Although such reviews could be done in a theoretical manner, more detailed insights are often gained when testing is carried out on the actual systems. Alternative scenario explorations on live systems might contain:

a technical exploration of attack scenarios deviating from those conducted during the testing phase;
a technical exploration of tested attack scenarios applied to alternative target environments (e.g. execution in a Citrix environment instead of on a company laptop);
proof-of-concept (e.g. scenarios not conducted during testing due to a high risk of BT detection or system damage);
a technical exploration of novel TTPs which have emerged but could not be tested on the technical systems during the testing phase.
The technical exploration of alternative attack scenarios makes it possible to take an in-depth look at the consequences, potential detection and response measures for novel kinds of attacks. Since this is done in a collaborative manner, even aspects entailing a high degree of risk can be investigated to obtain very realistic results with a high level of technical detail.

In some jurisdictions, PT is specified as a mandatory requirement in some national TIBER implementation guide. ↩