Evaluation Framework

  • Given our project is in early stages, we are trialing activities to determine their impact and the nature of our work is difficult to evaluate, our biannual evaluation process will be iterative.
  • The amount of staff time allocated to each activity indicates how important we think it is relative to the others. We believe the current breakdown will maximise our overall impact. However, during biannual evaluations we will assess the outcomes from each activity and should we conclude that we need to scale up or down some activities we will adjust the amount of staff time allocated to them accordingly.
  • To what extent does working on each of the activities qualify as having “supported” one or more of the core components?
    • We set preliminary targets based on data from the previous six months to establish a baseline.
    • For activities which we did not conduct in the previous six months we will set a target based on what we believe seems feasible, evaluate our success against it and adjust accordingly.
ActivityStaff Time (% of total)Target OutputIndicatorsImpact
Conduct research60%A non-academic research paper per 200 hour investmentResearchers rate the expected value progress at the beginning of each evaluation period and compare it to their rating of the actual value of progress at the end of that evaluation period. Has our research helped us make progress toward identifying large-scale, net-positive interventions?
Number of papers completed
A research paper submitted for peer-review per 480 hour investment. Number of papers completedHas publishing research in peer-reviewed journals led to engagement from our target audience?
Number of submissions accepted to peer-reviewed journals
Number of citations from papers accepted to peer-reviewed journals
A blog post per 20 hour investment. Researcher rates the expected value of progress prior to beginning the blog post and compares it to their rating of the actual value of progress on completion.
Support external research~3%N/ANumber of papers completed out of the number of research projects to which we provided supportHas supporting the work of external researchers helped us make progress toward identifying large-scale, net-positive interventions?
Number of papers we chose to publish on our website out of those to which we provided support
Develop narrow interventionsSome % of research time will be assigned to this activity dependent on findings. Outline of an intervention including a cost-effectiveness analysisReception from expert / experienced WAS advocates Does the intervention plan allow us to assess its cost-effectiveness?
Is the intervention (based on the plan outlined) likely to be accepted and adopted?
Run a research grants competition~5%Organise and publicise the grants competitionNumber of applicationsHas the grants competition allowed us to expand research efforts to identify large-scale, net-positive interventions?
Has the grants competition indicated the level of interest in WAS research from our target audience?
Feedback from applicantsHas the grants competition indicated whether, and if so which, interventions are of most interest to our target audience?
Evaluate applicationsQuality of applicationsHas the grants competition helped us refine our outreach plans and set the groundwork for building an academic WAS field?
Build networks with academia~5%Up to 30 universities contactedNumber of responses / relationships with universitiesHas our approach to building networks with academia allowed us to expand research efforts to identify large-scale, net-positive interventions?
Has our approach indicated whether, and if so which, interventions are likely to be accepted and adopted?
Exploratory study completedResearcher rates the expected value of information prior to beginning research and compares it to their rating of the actual value of information on completion. Has network-building helped us refine our outreach plans and laid the groundwork to build an academic WAS field?
Build networks with research institutes~5%Up to 10 research institutes contactedNumber of responses / relationships with research institutesHas our approach to building networks with research institutes allowed us to expand research efforts to identify large-scale, net-positive interventions?
Has our approach indicated whether, and if so which, interventions are likely to be accepted and adopted?
Draft content for life science media ~6% (time spent on this activity is fungible with research depending on what seems most promising to each researcher)Up to 10 pitches submittedNumber pitches accepted / Number of published articles Has publishing content for life science media led to engagement from our target audience?
Number of article views
Publicise our content online on our website/social medianegligibleAll research papers published on the website.Website analyticsHas publishing our content online led to engagement from our target audience?
Updates shared on social mediaSocial media analytics
Attend and/or present at conferences~3%Applications to speak at all selected conferences submittedNumber of successful applicationsHas presenting at or attending conferences led to engagement from our target audience?
If possible, feedback on the quality of presentationsHas presenting at or attending conferences indicated whether, and if so which, interventions are likely to be accepted and adopted?
Attendance at all selected conferencesFeedback from WASR staff on the value of the experience
Improve accessibility of research~1%Library of WAS content Website analyticsHas improving the accessibility of research led to engagement from our target audience?
Regular online Q&A or FAQQuality of participation with Q&A
Website analytics

Note: the remaining ~12% of time is spent on project management, fundraising, evaluation and other administrative tasks.