Gender Equality Impact Drivers Revisited: Assessing Institutional Capacity in Research and Higher Education Institutions

Abstract

This article presents the development and piloting of an innovative tool to assess the sustainability and impact of institutional change towards gender equality, termed the Impact Driver model. It provides a description of the model and the resulting tool, as well as how it has been developed, based on earlier models. It also presents the revised model following a pilot test and accompanying workshop, which were carried out to gather feedback on the use and potential of the tool. In conclusion, the article provides recommendations for the use of the tool, considering the EU context and policy framework, which pushes towards the institutionalisation of gender equality in research and innovation.

1. Introduction

Since 2015, institutional change through Gender Equality Plans (GEPs) has been recognised as the key element of national policy frameworks on gender equality in research and innovation (R&I) in Europe, with GEPs as the primary policy instrument (Council of the European Union 2015). This marks a shift from the previous “fixing women” approach to that of “fixing institutions” (Linková and Mergaert 2021, p. 318). Institutional change is a long-term process; it moves in a non-linear fashion, with windows of opportunity opening and closing in a manner that cannot be fully anticipated. It is complex in that every situation is unique, previous success does not guarantee the ability to duplicate, and expertise helps but is not sufficient, as relationships are key (Glouberman and Zimmerman 2002). This complexity also means that patterns become visible only in retrospect (Kurtz and Snowden 2003). Finally, it is an iterative process, where diagnosis leads to problem definition; the identification of objectives, actions, and targets; monitoring and evaluation; and revision of the problems and objectives.

Recently, the European Commission (EC) has decided to make the access of research organisations to its €95.5 billion Horizon Europe funding program (2021–2027) contingent on having a Gender Equality Plan (GEP), built on a comprehensive understanding of gender equality. The GEPs are expected to address organisational culture and work–life balance, gender balance in leadership and decision-making, gender equality in recruitment and career progression, integration of the gender dimension in research and teaching, and measures against gender-based violence. The Commission has also specified four obligatory elements: the GEP must be a public document, resources must be allocated to its implementation, it must be based on sex/gender-disaggregated data collection and monitoring, and training and capacity-building must be provided (European Commission 2021a). This new development can be expected to generate a surge in activity at the institutional level as well as at the state level in the EU countries.

It should be noted that what is now commonly referred to as the Horizon Europe ‘GEP requirement’ does not apply in a homogeneous European landscape for research organisations. As the Standing Working Group on Gender in Research and Innovation, established under the European Research Area and Innovation Committee, has shown (SWG GRI 2021), despite the progress made, there continue to be considerable differences across European countries. For instance, there is huge variability in terms of policy and legislative frameworks. While a minority of nine EU countries have legal or policy provisions for GEPs and/or supporting infrastructures, others do not, and some even openly oppose gender equality measures (SWG GRI 2021, pp. 12–13; GENDERACTION 2021, p. 7). Furthermore, in 2020, more than 50% of higher education institutions (HEIs) in most EU-27 member states and associated countries reported gender equality actions and measures on their websites (European Commission 2021b, p. 169). However, this figure varies from 80% in nine countries (DE, IE, ES, MT, SE, IS, NO, CH, TR) to less than 40% of HEIs in Poland and Slovenia (ibid.). At the national level, therefore, the European landscape is varied in terms of institutional changes and interventions for gender equality. In such a diversified context, it becomes particularly crucial to develop monitoring and evaluation (M&E) approaches that allow context sensitivity while proposing harmonised tools to be consistently applied throughout Europe. Monitoring and evaluating social change, such as structural change towards gender equality, is a complex endeavour, which requires a longer-term perspective than what is usually possible, especially in a project context (Minto et al. 2020). It may thus not come as a surprise that the M&E approaches applied in the structural change projects funded by the European Commission have been quite disparate. Exchanges among the M&E experts of different EU-funded GEP implementing projects have identified issues at stake, which, among others, are the comparability of results across institutions, the time needed before the results of GEPs’ interventions become apparent, and the desirability to strive for coherence of M&E approaches. It is noteworthy that focusing on processes has been recommended as a way forward (Ferguson 2021).

In recent years, two M&E approaches to institutional change for gender equality in research organisations have emerged, which complement each other and which have converged in an attempt to capture the complexity of the task and manage the challenge of fairly and credibly evaluating the efforts deployed in different national contexts, with a focus on processes and allowing comparison across institutions. These approaches are:

-

The model of institutional capacity development (Mergaert et al. 2013; Mergaert and Wuiame 2013), designed to assess the capacity for gender mainstreaming of public administrations, including both the European Commission and in the EU Member States, focusing on the factors driving the institutional change process.

-

The Actor Mobilisation model (Cacace et al. 2016; Kalpazidou Schmidt and Cacace 2018), focusing on the mobilisation of internal actors to sustain institutional change and the internal processes which are set in motion around gender equality efforts.

The first integration attempts of these models have been made in the framework of the Horizon 2020 institutional change projects LIBRA1 and Gender-SMART,2 where they have been used for external monitoring and evaluation. In LIBRA, the Institutional Capacity model was used in combination with the Actor Mobilisation model (Cacace et al. 2019), while in Gender-SMART, the two models were more substantially integrated (Tenglerová and Linková 2020).

In the Horizon 2020 CASPER project,3 funded by the EC to assess the feasibility of introducing an award and/or certification system on gender equality in research and higher education institutions in Europe, this previous work has been taken up and developed further. The result is, we will argue, a powerful Impact Driver (ID) model that allows assessing of the institutional processes and the degree of institutionalisation. The model can be used as a tool for self-assessment, serving as an awareness-raising and capacity-building instrument, but also for assessment by external experts. The tool connected to the model, tested by seven research organisations from different parts of Europe, allows for a comparative analysis through its indicators and graduated stages, providing valuable insights into the factors affecting the change process. In sum, the model has an important potential, not only for generating recommendations for the practice of institutional change and deepening the understanding of its facilitating and hindering factors, but also for feeding further policy debate related to a potential European certification scheme under consideration in the European Commission.

This article is structured as follows: Section 2 presents the origin and evolution of the Impact Driver model, explaining the main focuses and features of the original approaches and the reasons and modalities of their integration. Section 3 presents the Impact Driver model and its attendant tool as they were developed and tested in the framework of the CASPER project. Finally, two sets of conclusions are offered in Section 4: the first is centred around further improvement of the model, while the second discusses its potential role in the evolving European policy framework for gender equality in R&I.

2. History of the Impact Driver Model’s Development

The ID model presented in this paper, consisting of 12 impact drivers progressing in six stages of institutionalisation, builds on previous analyses of institutional change processes toward gender mainstreaming and gender equality, which are summarised in

Table 1. An overview of the development of the Impact Driver model.

Who/Authors	Purpose	Relevance/What?	Impact Drivers
(Brown 2008) (Taylor 2010) (UNIFEM 2010) (European Commission 2007)	Model of organisational development and assessment of institutional capacity	A total of 5 phases of organisational development (Project, Outsider, Growth, Insider, Integrated)
(African Development Bank Group 2011)	Evaluation of gender mainstreaming	Model with 4 Impact Drivers	Leadership actively committed to GM; procedures and processes influence how organisation works; sufficient financial and human resources available for gender activities; organisational incentives and accountability structures support mainstreaming
(Mergaert 2012)	Evaluation of gender mainstreaming implementation	Prerequisites for gender mainstreaming
(Mergaert et al. 2013; Mergaert and Wuiame 2013 )	Evaluation of institutional capacity for gender mainstreaming (in public administrations)	Model with 9 Impact Drivers and 5 stages of institutional development (Project, Isolation, Growth, Integration, Institutionalisation)	Stakeholder involvement; coverage of policy cycle; availability of resources; access to gender expertise; transparency and accountability; structured understanding of gender (in)equality; organisational culture; leadership actively committed to GE/GM; daily routines consider gender
(Cacace et al. 2016; Kalpazidou Schmidt and Cacace 2018)	Evaluation of structural change for gender equality	Actor Mobilisation model focused on the activation of the internal change processes (stress on agency)
(Cacace et al. 2019)	Evaluation of structural change for gender equality	Actor Mobilisation model combined with the Impact Drivers model; isolation stage was renamed Inception	As in (Mergaert et al. 2013; Mergaert and Wuiame 2013)
(Tenglerová and Linková 2020)	Evaluation of structural change for gender equality	Actor mobilisation model and Impact Drivers model integrated: model with 11 Impact Drivers	As above, with 2 new from Actor Mobilisation Model: transformational agent; involvement of internal stakeholders
This paper	Evaluation of institutional capacity for driving structural change for gender equality	A total of 12 Impact Drivers and 6 stages of development (‘starting point’ added as stage 0)	Revision of 11 Impact Drivers + 1 new (data collection and statistical analysis)

below. A detailed commentary on the evolution of the model is provided in Section 2.1 on the Institutional Capacity model and Section 2.2 on the Actor Mobilisation model.

2.1. The Institutional Capacity for Gender Mainstreaming Model

This model was first developed in the context of an unpublished study commissioned by the European Institute for Gender Equality (EIGE), which aimed at assessing the institutional capacity for gender mainstreaming of the European Commission and member states (Mergaert et al. 2013; Mergaert and Wuiame 2013). Institutional capacity for gender mainstreaming refers to the potential of an institution to deliver upon its gender mainstreaming commitments and the ability to identify and solve implementation problems. Capacity deals with a set of functional conditions that allow elaborating and implementing programmes with better performance.

That original model took, as a departing point, an evaluative model for gender mainstreaming used by the African Development Bank Group (2011), based on a review of 26 thematic and country evaluative reports focused on gender mainstreaming, gender equality, and women in development. It is based on a theory of change (see Figure 1 below), which posits that the existence of a number of preconditions, or impact drivers, allow effective change (or impact) to be realised. As a basic principle, if impact drivers are not present, the change process is unlikely to produce impacts. Such an assessment of the presence or not of impact drivers can thus support an assessment of the institutional capacity for gender mainstreaming.

The theory of change model used by the African Development Bank identifies four key impact drivers that need to be present to achieve the mainstreaming of gender equality: (a) effective leadership, (b) adequate financial and human resources, (c) availability of appropriate procedures and processes, and (d) appropriate organisational incentives and accountability structures. For the purposes of the study for EIGE and based on a review of literature on gender mainstreaming (Council of Europe 1998; Mergaert 2012), nine impact drivers were conceptualised by Mergaert and colleagues for the analysis. These impact drivers are coherent with the European Commission’s principles for good governance (openness, participation, accountability, effectiveness, and coherence) (Commission of European Communities 2001).

Furthermore, in addition to identifying the key elements that drive effective change, it was important to integrate a means to assess progress. Therefore, departing from existing conceptual frameworks (UNIFEM 2010; Taylor 2010; Brown 2008; European Commission 2007), the impact driver concept was combined with a model of institutional capacity development in five stages. The underlying rationale is that the attainment of gender equality change is the result of an organisation’s growing capacity for gender mainstreaming along a number of pre-identified intermediate stages. The five stages were labelled: Project, Isolation, Growth, Integration, and Institutionalisation. A visual representation showing the five stages of institutional capacity development for gender mainstreaming is presented in Figure 2.

For determining the institutional capacity, a grid with rubrics has been generated for each indicator and impact driver to identify the stage of institutionalisation. These rubrics are “tools that help to formalise processes of evaluation or assessment by outlining agreed-upon criteria that mark different levels of performance. Rubrics can be tailored to meet context-specific needs, rather than referring to seemingly ‘objective’ outside criteria, i.e., type and degree of change between the different criteria can be chosen on a case-by-case basis.” (UNIFEM 2010) The above-described model thus incorporates the understanding that institutional capacity is a dynamic reality, evolving over time and influenced by various factors (European Commission 2007). By assessing the institutional capacity per impact driver, a refined insight into particular strengths and weaknesses of the institution’s capacity is obtained.

A scoring system was used to synthesise the results, with scores connected to the achievement of the different stages of mainstreaming capacity. The application of the model to the European Commission in the first phase of the study done for EIGE served as a test and confirmed its applicability and relevance, following which an assessment for each member state was performed. The analytical framework thus allowed the European Commission’s and each Member State’s institutional capacity for gender mainstreaming to be comparatively situated (European Commission 2007).

The strength of this model lies in its focus on the process of implementation and what drives it. Through its impact drivers and the five stages of institutionalisation, it is well equipped to take into account different paces and different starting points of institutional development, which is clearly one of the major concerns currently resounding due to the divergence of policy implementation across the EU (SWG GRI 2018, 2021; Wroblewski 2018, 2020).

2.2. The Actor Mobilisation Model

The second model (Cacace et al. 2016; Kalpazidou Schmidt and Cacace 2018) is not primarily focused on the achievement of specific results in the institutionalisation continuum, but rather on the activation of the internal change processes which are expected to lead to such results.

In this model, the change process is broken down into four components, which can be considered separately for analytical purposes, even though they are strongly interconnected, so that the sequence is not always linear, but dependent on contextual factors. These are:

-

The creation of an internal transformational agent;

-

The activation of agency dynamics, with the arousal of supportive or resisting attitudes and behaviours towards the process;

-

The interaction of agency dynamics and structural circumstances, which can reinforce or hinder the change process;4

-

The resulting outcomes in terms of sustainable institutional change.

The model’s strong focus on the agency side of change implies that for a policy like institutional change to be turned into a shared social action, it is extremely important that a transformational agent is present, able to function as a catalyst for change, gaining visibility and building authoritativeness and internal recognition as a legitimate interlocutor on a set of gender-relevant issues. At the start, the agent can be an individual or a small group. In an institutional change process, the transformational agent formally in charge is expected to progressively enlarge, by integrating other groups and individuals committed to gender equality.

Over time, such a recognised and authoritative group is expected to be able to activate broader agency dynamics by starting to reach out to and mobilise previously uninvolved/uninterested internal stakeholders, including within the leadership, gaining their support, achieving more visibility, and spreading awareness around the gender equality agenda. Alliances with groups and individuals concerned with different but compatible issues are key to the process, as well as contacts and partnerships with external stakeholders as possible allies and supporters. Resisting or even overtly conflicting attitudes can also be aroused, which need to be addressed.

Working for institutional change implies, almost by definition, being confronted with the structural features of the organisation, which may, depending on the specific action concerned, facilitate or hinder the work of the transformational agent. Every institutional change process is indeed affected by structural constraints of different kinds, including the cultural and social context, internal functioning, norms and regulations, the availability of different types of resources, etc.

The result of the interaction between the change process and the structural circumstances then defines the range of the sustainable outcomes which will be possible for it to achieve. In the Actor Mobilisation model, outcomes are classified according to four dimensions of institutional change (interpretive, symbolic, institutional, or operational, see

Table 2. The multiple dimensions of institutional change (Kalpazidou Schmidt and Cacace 2018).

Interpretive	The interpretive dimension of change concerns raising awareness of the situation of women in the organisation as to horizontal and vertical segregation patterns; unveiling the functioning of mechanisms reproducing gender inequality; and demonstrating gendered assumptions and consequences of organisational practices and procedures, starting a reflection on the assumed gender neutrality of science.
Symbolic	The symbolic dimension of change addresses the image of science as based on masculine values and symbolism, which conveys stereotyped images of scientists and science itself. Change is sought by increasing the visibility of women scientists and their achievements, particularly in male-dominated fields, as well as making leadership commitment and authoritative support to gender equality visible through various communication means.
Institutional	The institutional dimension of change addresses norms, practices, and procedures in the organisation. Change here concerns the modification of existing rules and structures or the creation of new ones, such as gender equality policy plans and units, up to the introduction of gender mainstreaming mechanisms. It can also imply the creation of new internal institutions (equal opportunity commissions, thematic working groups, etc.) or groups, such as women’s networks or associations.
Operational	The operational dimension targets managerial aspects and entails negotiations to get to the actual and effective implementation of the agreed actions and new arrangements, respecting a specific timeframe, while constantly monitoring and assessing the process of change and its results.

) and according to four strategic areas of institutional change in research, namely Recruitment; Career Development; Work–Life Balance; and Sex and Gender Dimensions of Research.

Figure 3 below presents a streamlined summary of the elements of the dynamic and iterative change process described above, starting from a group endowed with the task of concretely implementing an institutional change process (often via a GEP), through the progressive mobilisation of various supportive stakeholders, confrontation with structural obstacles of various kinds, and the activation of the change process in different relevant change levels and strategic areas. The sustainability of new gender arrangements finally requires the diffusion of transformational attitudes, within and outside the promoting organisation.

2.3. Integrating the Two Models

In the LIBRA project, entailing the implementation and evaluation of GEPs in 10 research institutions, the two models have been used in combination. In particular, the Institutional Capacity model was used (with slight adaptations, e.g., the term “isolation” was replaced by “inception” to keep the evaluation free from negative connotations5) to provide a metric system to assess the progress of the research institutes as concerns the institutionalisation process of gender mainstreaming capacities and arrangements. Furthermore, a scoring system, related to the different stages, was used to synthesise the results. The Actor Mobilisation model was used to provide complementary information about the features and results of the internal process activated by the implementation of the plan. This was meant to not only provide an additional set of results of the GEPs, but also to offer a complementary view of the sustainability of these results, connected to the robustness of the internal mobilisation around gender equality.

In the Horizon 2020 Gender-SMART project, a monitoring and evaluation approach has been designed to capture, in one tool, the institutional capacity for mainstreaming while giving greater recognition to the importance of the mobilisation dynamics, underscored in the Actor Mobilisation model (Tenglerová and Linková 2020). The original nine impact drivers developed by Yellow Window have been expanded to 11, by adding “transformational agent” and “involvement of internal stakeholders” from the Actor Mobilisation model. In line with the Institutional Capacity model, they were assessed in five stages. The five stages indicate that at least some degree of activity is already taking place, with the “project” stage encompassing some level of ad hoc activity without much reach. Following LIBRA, the decision was also made to use the term “inception”. The external monitoring and evaluation approach has been developed in recognition of the great variability among the GEP implementing project partners, the difficulty of achieving changes in the statistical makeup of an organisation within a project life, and especially the importance of factors leading to change that need to be monitored.

3. Description of the Impact Drivers Model

In the CASPER project, the work on the Impact Driver model considered this previous evolution and deployment of the monitoring and evaluation approaches in the two projects, LIBRA and Gender-SMART. CASPER proposed to capitalise on these developments and on hands-on experience in implementing the models for institutional assessments and to reflect on ways to further improve the integrated version used in Gender-SMART. In addition, the model has also been adapted to reflect the evolving research, innovation, and higher education context in the European Research Area. Specifically, these were the Horizon Europe GEP requirement with the four obligatory elements that the ID model now encompasses and the arising need for self-assessment or an external assessment in cases of certification or third-party checks.

Furthermore, the model was upgraded with the latest knowledge and insights about stages of institutionalisation of gender equality and concepts of gender+6 (Lombardo et al. 2017), referring to the combined effects of gender inequality with oppression and discrimination along other axes of inequality. This development work was done by simultaneously considering all the impact drivers, identifying suitable indicators, and formulating rubrics along the different stages of institutionalisation. While doing so, ‘vertical consistency’ has been controlled, meaning that rubrics for each stage towards institutionalisation have been checked for internal coherence across the rubrics. Once a coherent model was ready, it was transposed into an operational tool for self-assessment, in the form of a MS Excel file with accompanying guidelines for its use.

The model and tool were piloted in the CASPER project, with the core teams implementing GEPs at seven research-performing organisations (RPOs) that agreed to fill out the tool and share their feedback and reflections on the use and benefit of the tool. Concretely, the transformational agents of these institutions convened and discussed the status for each impact driver, deciding on the stage of advancement for each indicator based on their knowledge of the organisation. These seven RPOs included five universities: Central European University based in Hungary and Austria, Cyprus University of Technology, the University of Deusto based in Spain, Oxford Brookes University based in the United Kingdom, and the University of Cagliari based in Italy. The two remaining RPOs that participated were research institutes: the French Agricultural Research Centre for International Development and the European Institute of Oncology’s Department of Experimental Oncology based in Italy. All these institutions are or have been partners in one of the so-called structural change projects, through which gender equality plans are designed and implemented, notably SUPERA, GEARING-roles, Gender-SMART, and LIBRA.

In addition to testing the tool, feedback collected from the piloting institutions focused on the following aspects: (1) clarity of the indicators and their rubrics in each ID; (2) clarity of the stage definitions in each indicator; (3) whether the definition of stages is realistic; and (4) whether the definition of stages clearly outlines a gradual progression toward institutionalisation. Based on the feedback received, an additional impact driver has been introduced, that of data collection and statistical analysis, to underscore the importance of this for the institutional change process and link with the European Commission’s obligatory building blocks.

The final ID model, following revision after the test, comprises 12 impact drivers for institutional change towards gender equality and six stages of institutional capacity development. For each of the impact drivers, two to five indicators have been elaborated. The six stages have been labelled: Starting point, Project, Inception, Growth, Integration, and Institutionalisation. The stage of ‘Starting point’, which was not in the original model, represents the point zero, where nothing is in place yet. In order to determine institutional capacity, a grid allows for identifying the stage of institutionalisation for each impact driver, using rubrics. The model makes it possible to synthesise the results of an institutional analysis, as well as to situate the organisation against others doing similar work.

Table 3. Impact drivers and their indicators for gender mainstreaming in research and higher education institutions.

IMPACT DRIVER	Indicators
1. CORE TEAM OF CHANGE AGENTS	A core team of change agents exists, the size and composition of which are commensurate with the size and complexity of the organisation	The core team of change agents comprises motivated people	The core team of change agents has a formal mandate and ownership over the endeavour	The core team of change agents has access to an extended group of change agents
2. CAPACITY/SKILLS OF THE CHANGE AGENTS FOR DRIVING INSTITUTIONAL CHANGE FOR GE	Proficiency in the use of participatory approaches and co-creation techniques	Active understanding of strategic framing and of the political nature of change processes	Capacity to deal with resistances	Institutional learning about GEP implementation	Ability to mobilise and engage stakeholders
3. LEADERSHIP ACTIVELY COMMITTED TO GE/GM	GE is a priority in the institutional agenda/strategic documents	GE is present in the public discourse of leaders	There is an explicit and visible commitment of leaders to GM
4. AVAILABILITY OF RESOURCES	Internal gender knowledge and expertise are available and used	There are funds dedicated to GE	There are capacity-building initiatives on GE issues	Support materials (guidelines, toolkits, directory of resources, etc.) are available
5. DATA COLLECTION AND STATISTICAL ANALYSIS	Institutional gender disaggregated data are collected	Institutional gender disaggregated data and statistics are public and accessible	Intersectional gender disaggregated statistics are collected and published
6. INVOLVEMENT OF INTERNAL STAKEHOLDERS	Leadership engagement with the core team of change agents and GE work	Variety of internal stakeholder groups engaged (coverage)	Numbers of people/size of groups that engage with GE efforts	Degree of adherence to GE goals	Internal stakeholders start initiatives themselves
7. INVOLVEMENT OF EXTERNAL STAKEHOLDERS AND EXPERTS	NGOs/CSOs are involved in the institutional GE work	The gender dimension is addressed in events	External partnerships with relevant institutions have been established for gender equality work	External gender expertise is available and used
8. COVERAGE OF THE DIFFERENT DIMENSIONS/AREAS OF GE INSTITUTIONAL CHANGE	Comprehensiveness of the GEP/GM work in terms of areas addressed	Comprehensiveness and sophistication of the work within addressed areas
9. TRANSPARENCY AND ACCOUNTABILITY	Some GM elements are mandatory (self-imposed)	Incentives and/or sanctions are in place	GE is included in unit reports and assessment for internal monitoring	GE reporting is done and is publicly available
10. INSTITUTIONAL POLICY-MAKING BASED ON A ROBUST UNDERSTANDING OF GE	Structured understanding of gender issues: differences versus inequalities, mechanisms, structures, and systems	Based on data analysis, consistent policies are designed	Contextualised GE and GM goals exist
11. ORGANISATIONAL CULTURE	GE is expressed as an organisational value, both formally and informally	The existence of gender inequality is acknowledged	Obstacles to GE/GM are addressed
12. ORGANISATIONAL GOVERNANCE	Gender analysis is considered in strategic planning	Gender analysis is considered in internal monitoring	Gender analysis is considered in internal audits and institutional assessments	Gender-sensitive routines exist	Gender-specific routines exist

below presents the 12 impact drivers of the model and their corresponding indicators.

Table 4. Example of an impact driver with indicators and rubrics.

Impact driver 1	Indicator: 1. A core team of change agents exists, the size and composition of which are commensurate with the size and complexity of the organisation
	Starting point	Project	Inception	Growth	Integration	Institutionalisation
Core team of change agents	There is no core team	An individual or small group has started working on GE, not yet in a very coordinated way	The core team, as a driver of the institutional change work, takes shape, and there is some internal coordination	There is a coordinated core team, and its composition starts to reflect the features and needs of the organisation	There is a coordinated core team that is not yet fully adequate in view of the organisational structure and size	The core team’s size and composition are commensurate with the size and complexity of the organisation
	Indicator: 2. & 3. The core team of change agents comprises motivated people; the core team of change agents has a formal mandate and ownership over the endeavour
	Starting point	Project	Inception	Growth	Integration	Institutionalisation
	Either nobody has a mandate for GE, even if there are a few individuals interested in GE, or there is no real motivation to take up the issue	With or without a mandate, there is a person that started working on GE, motivated to be a change agent within the organisation	A small group of motivated people is working on GE	A core group of motivated people is steering the GE work and is internally recognised as ‘in charge’	A core group of motivated people has a formal mandate to work on GE, but does not have full ownership over the process	A gender equality unit, with a formal mandate and control over the process, oversees the institutional GE work and has direct links to the leadership
	Indicator: 4. The core team of change agents has access to an extended group of change agents
	Starting point	Project	Inception	Growth	Integration	Institutionalisation
	There is no core team	There is minimal engagement with other people in the organisation regarding GE work	The change agent network consists of the core team and a limited number of supportive individuals	The change agent network grows beyond the core team, to include other allies, ambassadors and supporters who are willing to contribute with skills, expertise, and public support	Beyond the core team and its circle of supporters, there are formally appointed representatives of immediately related units	The formally appointed change agent team works with officially appointed representatives of research and administrative units and leadership

below shows an example of a typical impact driver as it appears in the model. In this case, Impact Driver 1 (core team of change agents) is shown with its four indicators and associated rubrics corresponding to the six stages of institutionalisation.

The assessment tool was constructed in Microsoft Excel, consisting of four sheets. The first sheet is the only sheet intended to be completed by the assessors and contains the ID model with rubrics, as reflected in Figure 4 below. Assessors are invited to indicate the level of institutionalisation for each ID and can determine this level by first identifying the levels of institutionalisation for each of the ID’s separate indicators (making use of the accompanying rubrics). While this intermediate step is not mandatory, it is recommended as it makes the exercise more transparent and allows for a better understanding of the initial assessment when revisiting the tool later for another assessment round. The assessment tool can, in this way, serve to monitor the progress that is made in the organisation.

The assessment tool can be used for self-assessments as well as for an external assessment. In the case of self-assessment, it is recommended that assessors provide a brief justification or explanation of each given rating. When used for external assessment, justifications must be given. Dedicated space for this justification is provided in the first sheet of the tool. The image below gives a visual impression of the tool’s first sheet, with the yellow cells containing the impact drivers, the purple ones the indicators, and the green ones the rubrics across the different stages.

Additional space is reserved for any ‘evidence’ or proof to justify the rating. Completing it will allow assessors to gain better insight into how the institution could sustain the claims made with regard to the capacity for change, in case of an external audit. Lastly, space has been provided for any additional comments or reflections that assessors might have.

The second sheet of the tool provides a list of the 12 impact drivers, each with their own indicators. It is intended to provide clarifications for certain items that might require an explanation, i.e., definitions of certain terms that are used. Assessors should not make any additions or alterations to this sheet.

The third sheet of the tool, visible in Figure 5, is automatically completed based on the assessments made in the first sheet. It presents an overall result, in terms of an institution’s capacity for driving change through GEPs, in the form of a table and two charts (one bar chart and one radar graph). The fourth sheet, not shown here, is a technical one that contains answer options for the drop-down list of answers in the first sheet and is not to be modified by assessors.

Figure 6 presents the anonymised results by way of a bar chart of one of the test organisations in CASPER. This chart shows the degree of institutionalisation for each of the impact drivers. It makes visible the strengths of the organisation’s capacity to drive change forward but also the points that need attention.

4. Discussion and Conclusions

4.1. Considerations on the Institutional Use of the Tool

The ID model and tool have been developed to incorporate the state of the art in relation to change processes for gender equality and have shown potential for being used in multiple ways. Firstly, it can be used for self-assessment of an institution’s capacity for GE change efforts in different types of institutions (research or higher education), in different fields of research, in different countries of Europe with different governance systems in higher education and research, and at different stages of gender equality actions (from starting stage to more advanced institutions), as the pilot test performed within the CASPER project confirmed. It gives institutions an indication of where they stand at a particular point in time and allows for institutional learning/capacity-building, awareness-raising, and self-monitoring by illustrating what dimensions are most in need of focus and investment. Secondly, the model can be used by external evaluators or as an audit tool for assessing an institution (with some cautions, in the latter case, as will be discussed below). Finally, when using the tool at multiple points in time, it allows for the measurement of progress/regress in the institution.

What needs to be considered, however, is that some impact drivers are more straightforward and ‘easier’ to progress in and monitor, while others are more complex and challenging for an institution to (fully) achieve and assess, also depending on the size of the institution. Additionally, some impact drivers (such as the availability of resources and leadership’s commitment) are more vulnerable than others to being dismantled when, for instance, there is a change in the institution’s leadership. Another important point is that filling out the tool requires a considerable level of knowledge of the institution and its internal functioning, which is often tacit. This could inhibit its potential when used by external evaluators who do not have access to this tacit institutional knowledge. For this reason, it is suggested that the justification must be provided when the tool is used for assessment by third parties. The ratings that are given may also vary significantly depending on the strictness and knowledge level of a particular assessor. Current experience shows that institutions that are less advanced in the institutionalisation of their gender equality efforts are more lenient in their self-assessments.

One potential method of making external assessments more reliable is to precede the external assessment with an initial self-assessment by a core team, with justifications and evidence provided (wherever possible) to support the assessment level. After this initial step, an external assessment (possibly by two assessors) will review the self-assessment and the ratings given by the core team, per impact driver or even per indicator. This would require capacity-building of all the assessors involved, for which a community of practice could be established.

Evidently, the Impact Driver model and tool have a potential role to play in the context of a certification scheme, as the introduction of such a system is currently considered by the European Commission, especially since such a scheme would have to allow for the heterogeneity in the institutionalisation of gender equality efforts in the EU while defining clear entry and progression points. This is precisely what the Impact Driver model offers.

The application of the ID model across different contexts could lead to new research to advance our understanding of institutional change processes. Firstly, the model could lend itself easily to gauge progress achieved over time with the concomitant attention to the impact drivers that may be less stable and more vulnerable to backlash (e.g., in relation to a change of leadership). Secondly, with widespread use of the model, analyses could be performed of which IDs are easier to achieve progress in and whether there are any correlations among the IDs. It could also help to understand how legislative and policy frameworks may support institutionalisation in some IDs but less so in others. Finally, having the ID tool available for widespread use could result in the creation of a database where institutions could benchmark themselves against other institutions similar in size or disciplinary orientation.

4.2. Potential of the ID Model in the Evolving European Policy Framework on Gender Equality in Research and Innovation

The issue of monitoring and evaluating institutional change is one of the key concerns for policy makers at the European Commission and member state level, as identified in the Ljubljana Declaration and currently debated as part of developing ERA policy action 5, “Promote gender equality and foster inclusiveness, taking note of the Ljubljana declaration” (European Commission 2021c). This is also reflected in the work to be started by the Horizon Europe project GENDERACTIONplus,7 launched in June 2022, which will focus, among other topics, on the monitoring and evaluation of GEPs, reflecting the differences at the national level and the instruments that national authorities have put in place for monitoring and evaluating them. It is noteworthy that out of the nine EU countries that currently have a GEP requirement at the national level (AT, DE, DK, ES, FI, FR, IE, PT, SE), six have some system in place for monitoring the GEPs (DE, DK, FR, IE, PT, SE) (SWG GRI 2021). Nevertheless, the SWG GRI notes that “the adoption of a policy or strategy does not automatically mean an implementation whereby institutional changes are certain to be achieved. In some countries, the GEP requirement does contain a robust quality assurance feature. To this end, the monitoring and evaluation of GEP implementation shall be a crucial issue to be tackled in the new ERA, including the Horizon Europe GEP requirement, possibly in relation to the potential introduction of a gender equality certification scheme.” (SWG GRI 2021, p. 4). Given this policy concern, the ID model and tool presented in this paper could be an efficient instrument due to the recognition of the iterative, non-linear, and variegated state of gender equality efforts at both the institutional and country level in Europe.

Connected to this broad perspective, additional benefits of the model can be highlighted. First of all, strong concerns have been voiced in the last several years about the risk that external accountability mechanisms, such as the Horizon Europe GEP requirement, may become mere formalities, encouraging institutions to satisfy formal criteria without engendering real change (Højlund 2015; Marx 2019). The ID model and tool, built on extensive experience of institutional change processes at diverse institutions, are precisely meant to give substance and structure to the evaluation process, making tick-the-box exercises more difficult.

In addition, the complexity and context-sensitivity of institutional change processes demand that an encompassing set of qualitative indicators are used to give a reliable picture of the specific profile of strengths and weaknesses of institutions and national systems, avoiding simple headcount indicators. On the other hand, synthetic indicators are also needed to provide comparable results across different institutions and countries. The ID model sets out a complex system of qualitative indicators and identifies, based on state-of-the-art knowledge, frequent configurations of supporting and hindering factors at research institutions, while it is also able to produce synthetic results at different levels (per impact driver or per institution), representing them in simple charts.

Finally, the need for support and capacity building, as well as for self-assessment and self-reflection tools, has also been voiced, particularly in connection to the Horizon Europe requirement, to support GEP implementation.8 In this respect, the ID model represents an excellent opportunity, accompanying and supporting all phases of GEP design and implementation and encouraging a self-reflective and participatory approach, where the involvement of internal stakeholders is strongly encouraged and highly valued.