HOW TO EVALUATE PUBLIC POLICY ADVOCACY ACTIVITY?

The primary means of influencing public policy is direct interaction with decision-mak­ers, allies and other key players. This may include participation in negotiations or meetings, direct com­munications with government ministers, or informal discussions with partners and other contacts. Teams work to influence through persuasion, negotia­tion and lobbying. In more formal spaces this may be conducted through evidence-based dialogue, while in other channels this will require more informal discus­sions and debate. Projects will draw on the relation­ships staff have with various contacts, and will use budget support and other material incentives to influ­ence proceedings directly or, more often, indirectly.

Activity in public policy advocacy is crucial to shape the course of policy. Face-to-face personal inter­action is the strongest factor in facilitating the use of particular policy ideas or evidence.

The evaluation of public policy advocacy activity is based on actors, the relationships between them, and the institutions within which they work. In addition to clear and focused policy goals, the key strategic capacities required are identifying natural allies, developing relationships and credibil­ity with policy actors, and understanding the nature of the policy process and institutional access . The lobbyist’s key working tools are: the monitoring of key players and decision-makers, including their personal his­tory, perspectives and interests; and building coali­tions and alliances (often highly temporary) around particular policy goals .

The spread of influence relies on three types of people:

1. Connectors/Networkers who know who to pass information to and who are respected enough to influence key players;
2. Information Special­ists, who acquire information and educate others;
3. Salespeople powerful, charismatic and persuasive individuals who are trusted, believed and listened to .

 Therefore, keeping systematic track of the various actors, their interests, ideologies, capacities, their alignment with programme goals, and their relation­ships with other players, and how all of these change, is central to managing public policy advocacy, and should be the basis for measuring and understand­ing one’s influence. And understanding the key institutions and spaces, and how they affect decision-making is also crucial – different spaces may shape what kind of policy outcome will occur, based on the structure and rules of dialogue and decision-making.

It is not easy. Public policy advocacy work takes place in highly fluid contexts, based very much on tacit knowledge and experience, and split-second subjective judgements about, and reactions to, people’s attitudes, emotions, positions and perspectives. Expectations about how formalised and standardised monitoring and evaluation can be in such situ­ations must be duly adjusted, and determining attri­bution is simply not feasible in these contexts. Having said this, there are some tools and approaches that can help:

• Recording observations from meetings and nego­tiations is a useful and low-cost activity. This can be done simply by storing emails, meeting minutes or back-to-office reports, or using meeting obser­vation checklists to record how particular issues are covered, or how different actors behaved. For a slightly more in-depth analysis, an ‘after action review’ (a tool designed to help teams come together to reflect on a task, activity or project in an open and honest fashion) can be carried out with the project team to discuss what happened, why, and what can be learned.
• Tracking people and relationships and the project’s interactions with them is another key area. Policy networks show the importance of ‘policy champions’ and ‘opinion leaders’ who can facili­tate the uptake of certain policies . Simple tracking forms can be used to record what actions have been taken with them and when. Tracking the quality of relationships and access to such people provides important informa­tion for managing public policy advocacy work as well as indi­cations about the credibility and influence of the project. A more comprehensive approach can  be to keep spreadsheets or a database on various key actors (including more than just champions), including political intelligence information about their job, their position in decision-making proc­esses, and their perspectives and interests, as well as recording interactions with them.
• Interviewing informants. Building up an ‘informa­tion network’ is seen as essential to effective lob­bying , and is a useful avenue for understanding a project’s influence on policy. Interviewing people with knowledge about the institutions and processes, or particular actors with whom the project is working, can provide invaluable guidance. These could be people with technical expertise on an institution, who have years of experience with a particular individual or organisation, or who are well-placed in terms of their role in decision-making processes. Identifying who may be able to provide information relevant for the project should be done as early as possible, and relationships built up, as the project may need to rely on them to be their ‘eyes and ears’ in many situations where knowledge is quite politicised. Natural allies in lobbying efforts can be used this way, and tools for ‘horizontal evaluation’ may be effective or simply surveys about advocacy efforts.
• In-depth analysis: a variety of tools can provide richer information about the influence of lobbying efforts: the alignment-interests-influence matrix (AIIM) synthesises perspectives and evidence on different actors’ relationship to project goals (and how this changes over time). Social network analysis can function as a way of measuring and understanding actors’ relationships with each other and how they share information or resources. And power analysis or political economy analysis provides tools to look into the workings of decision-making institutions. Three promising tools that have been used in the EU context can provide interesting avenues here:
  1. Process tracing, which attempts to uncover the steps through which meetings and other events led to, and caused, outcomes;
  2. Attributed influence, where observers of key spaces in the policy process are surveyed on their judgement of the influence of a particular actor or action;
  3. Preference attainment, where the influence of actors is judged by the extent to which final policy outcomes reflect their ‘ideal’ positions.