Biannual Project Evaluation – T1, 2018

Summary

This is the first evaluation report for the Wild-Animal Suffering Project. This evaluation looks at our progress during the first six months (“T1”) of 2018 and assesses the extent to which our activities have allowed us to make progress towards our strategic goals. It also includes our most significant mistakes and issues as well as our priorities for the second six months (“T2”) of 2018.

Our main accomplishments include:

  • Completing three research papers and six blog posts.
  • Launching Round One of our academic outreach work. We engaged with six academics and two research institutes.
  • Giving six talks and one interview for a podcast.

Our challenges include: (1) failing to develop a strong people management structure, (2) underestimating the necessary time investment in project management and outreach work, (3) errors in our budget estimates and (4) accepting too many speaking engagements.

Our priorities for T2, 2018 include: (1) continuing our research, (2) finalising the first round of our academic outreach project, (3) running a grants competition, and (4) establishing better project management structures.

Activities

We did not determine which activities needed to be completed by T1, 2018. Instead we listed all the activities we want to complete or trial at some point this year. For a summary of these, see our strategic plan. In this section, we evaluate our work in two main categories: research and outreach.

Staff time investment

In T1, 2018:

  • Persis Eskander worked ~880 hours
    • In late T1, Persis took vacation time.
  • Ozy Brennan worked ~140 hours.
    • In early T1, Ozy took parental leave.
  • Georgia Ray worked ~360 hours.
    • For the majority of T1, Georgia worked with the Future of Humanity Institute.

How to read our indicators and impact metrics

In our evaluations, we want to understand:

  1. the value our work has added;
  2. how much progress we’ve made over the evaluation period; and
  3. the quality of our work.

The most important element from our perspective is the value we’ve added. We focus on adding as much value as possible, measured along our impact criteria. These are qualitative and rely largely on our impressions and expectations. As a result, they’re not extremely robust. To support our impact metrics, we also measure our work according to a combination of quantitative and qualitative indicators. These measure our progress and the quality of our work which we believe are useful proxies in understanding value added. Our indicators are supporting rather than primary metrics, and the importance of the information they convey is considered accordingly.

Research

Non-academic research papers

Indicator: Number of papers completed

  • Ozy Brennan completed two research papers that are currently in the feedback process. We expect them to be published by September 2018.
  • Georgia Ray completed a summary of the missing information on invertebrate sentience.
  • Persis Eskander did not complete any research papers in T1, 2018.

Indicator: Expected and actual progress

Invertebrate sentience: Urgent but understudied

  • Author: Georgia Ray
  • Expected value of progress: Not explicitly estimated in advance, I probably would have estimated Medium
  • Actual value of progress: Medium
  • Author’s comments: This piece successfully described one of my takeaways from previous research (while preparing, for instance, the piece Which Invertebrate Species Feel Pain) and captured several reasons that existing literature on invertebrate suffering was insufficient for making large decisions. I’m not aware of any ways in which this information has been utilized so far, but it has certainly contributed to my future work by defining my views on the matter, and hopefully also acts for other people  as a touchpoint for what questions about invertebrate suffering we are unable to answer with our current level of scientific knowledge.

Feeding Wildlife (yet to be published)

  • Author: Ozy Brennan
  • Expected value of progress: Medium
  • Author’s comments: Supplemental feeding is a fairly common intervention and it’d be useful to have an idea of the sign, particularly since a lot of other interventions use food as a delivery mechanism. However, feeding may increase population to the point that it is not a tractable intervention.
  • Actual value of progress: Medium
  • Author’s comments: It’s useful for us to have evidence that supplemental feeding doesn’t work as an intervention and to use this to inform WASR’s opinions on related interventions, such as cleaning bird feeders or using oral bait for contraception.

Wildlife Contraception (yet to be published)

  • Author: Ozy Brennan
  • Expected value of progress: High
  • Author’s comments: From my preliminary reading I assign a ~10% chance that wildlife contraception is a tractable method for improving wild-animal welfare.
  • Actual value of progress: High
  • Author’s comments: At this point I assign a 60% chance that wildlife contraception is a tractable method for improving wild-animal welfare. My uncertainty is caused by the absence of strong methods to assess the effects of wildlife contraception relative to the state of welfare without wildlife contraception.

Impact

Has our research helped us make progress toward identifying large-scale, net-positive interventions?

We believe our research so far has helped us make progress toward identifying large-scale, net-positive interventions. Georgia Ray’s piece on invertebrate sentience helped summarise the gaps in existing research, why they exist, and define her future research on the topic. Given this is a significant open question for WASR, we consider this piece informative, even if it didn’t directly lead to the identification of an intervention. Ozy Brennan’s paper on wildlife contraception reviewed all available literature on the effects of wildlife contraception and a companion intervention report identified it as a very promising intervention for some communities. Ozy Brennan’s second paper on wildlife feeding followed an identical format. The companion intervention report revealed that on existing evidence we shouldn’t prioritise wildlife feeding as a large-scale intervention. Our project is still young and we are still testing how to practically assess the implications of our research. We believe we need more time to determine whether there might be more impactful research paths to which we should shift.

Blog posts

Indicator: Expected and actual progress

To reduce wild-animal suffering we need to find out if the cause area is tractable

  • Author: Persis Eskander
  • Expected value of progress: Low
  • Author’s comments: This blog post wasn’t aimed at discovering new information but at outlining WASR’s current approach and strategic planning.
  • Actual value of progress: Low
  • Author’s comments: I believe the completed post is well written and may offer valuable information to readers, but it was not informative to our team.

Effects of seafood substitutes on wild sea animals

  • Author: Georgia Ray
  • Expected value of progress: High
  • Author’s comments: This topic is eminently practical, ending in solid recommendations. It was spurred by inquiry from someone interested in starting a seafood substitute company attempting to decide what products to develop. A shallow investigation here could have high and near-term rewards.
  • Actual value of progress: Low
  • Author’s comments: The person interested in the results of this ended up not pursuing this project for other reasons, so I don’t think it ended up having much practical value. I’m not sure how useful they would have found it if they had pursued the project. I’m not aware of this research having impacts in other areas (thus far) either. I believe the piece was of good quality, well-sourced and considering many different factors. While it was short and wasn’t a thorough coverage, a more thorough piece of research would have taken substantially longer and it’s not clear this would have been worth the time investment.

Reducing Aquatic Noise to Help Fish

  • Author: Ozy Brennan
  • Expected value of progress: Medium
  • Author’s comments: Reducing aquatic noise is potentially net-positive, but I’m not certain if it is tractable.
  • Actual value of progress: Low
  • Author’s comments: While it is a net-positive intervention, I don’t expect it to be sufficiently high priority for WASR to pursue advocacy on behalf of it or investigate its cost-effectiveness.

Clean Your Bird Feeders/Crucial Considerations for WAS

  • Author: Ozy Brennan
  • Expected value of progress: Low
  • Author’s comments: This post was primarily intended to share information I already had
  • Actual value of progress: Low
  • Author’s comments: This post was primarily intended to share information I already had.

Wild Animal Suffering: A Rights-Based Approach

  • Author: Ozy Brennan
  • Expected value of progress: Low
  • Author’s comments: It could be useful in broadening the appeal of reducing WAS to people who subscribe to a variety of ethical systems or who are morally uncertain.
  • Actual value of progress: Low
  • Author’s comments: I’m not certain how many people subscribe to deontological ethics so it’s not clear how wide the reach of this piece is.

Outreach

Networks with academia

Indicator

Number of responses / relationships with university based academics
  • We reviewed the top 30 universities (around the world) in biological sciences and identified 80 academics across 14 universities and 23 faculties working on research relevant to wild-animal suffering. We then divided the list into three categories: most promising, somewhat promising, and uncertain. Those that fell into the “most promising” list comprised part of “Round One”.
  • In Round One, we contacted 29 academics. Eleven responded, and eight were interested in discussing their work with us. We spoke with five of them and we are in the process of scheduling calls with the remaining three who have so far been unavailable.

Networks with research institutes

Indicator

Number of responses / relationships with institute-based researchers

We reviewed 15 research institutes focusing on either biological sciences or moral philosophy. As above, we divided the list into three categories: most promising, somewhat promising, and uncertain. Those that fell into the “most promising” list comprised the rest of “Round One”. In Round One, we contacted five research institutes, four responded, and two agreed to discuss their work with us.

Impact

Has our approach to building networks with academia allowed us to expand research efforts to identify large-scale, net-positive interventions?

It’s too early in the process to be able to answer this confidently. Our approach to building networks has seen some success. We’ve established relationships, conditional on being able to provide funding, with six academics working on a range of research topics including: wildlife management systems, invertebrate cognition, and population ecology. We believe that if we are able to secure funding for these academics their research may add significant value to our efforts to identify interventions.

Has our approach indicated whether, and if so which, interventions are likely to be accepted and adopted?

Not yet. Our approach has indicated that the research we consider most promising tends to sit in population ecology, ethology, neurobiology, and the philosophy of science (with a specific focus on animal cognition). This is not a significant update as our early research indicated that these would be valuable fields in which to focus. Concretely, and based on the small amount of data we have gathered, it appears to be substantially easier to support research:

  • focused on vertebrates than invertebrates; and
  • that aligns with efforts to reduce the effects of climate change.

It is significantly harder to find existing research projects on alleviating the potentially negative experiences of insects. This is because a high percentage of related work in entomology focuses on integrated pest management techniques to reduce the environmental impact of insecticides.

Talks and Conferences

Indicator

Number of successful applications
  • We submitted a successful application to give a talk and a successful application to organise a whiteboard session at EA Global San Francisco 2018.
  • We submitted an unsuccessful application to give a talk at the National Animal Rights Conference 2018.
  • Persis was invited to speak at a number of EA local groups Norway, Sweden and Germany.
  • Persis was invited to give an interview for the Our Hen House podcast.
Feedback from WASR staff on the value of the experience
  • Ozy Brennan noted that one element they like most about their role is the structured events i.e., giving talks and participating in workshops.
  • Persis Eskander found many early talking engagements e.g., EA Global London 2017 and the Our Hen House podcast, productive as she acquired immediate feedback on her communication skills. However, she believes she quickly hit diminishing returns on her time investment.
  • In future, WASR staff plan to focus on speaking engagements at large, recorded events and offer live Q&A opportunities to smaller groups.

Impact

Has presenting at or attending conferences led to engagement from our target audience?

Yes, however, it’s not clear that this engagement couldn’t have been counterfactually acquired with less effort. Early engagement from EAs (between June 2017 and December 2017) motivated to work on reducing wild-animal suffering did not correlate with WASR staff attendance at speaking engagements. More recently, the pool of EAs interested in volunteering or working on the cause area has increased. Anecdotal reports indicate that this was partly driven by WASR staff participation at conferences. Additionally, a small portion of our donors explicitly pointed to speaking engagements or events as a primary factor in their decision to support WASR.

Engagement

Indicator

Website analytics

Users

  • New users: 92.6%
  • Returning users: 7.4%
  • Average number of sessions per user: 1.23
  • Average time spent on a page: 00:02:09
  • Average number of active users per month: 1200
  • Average number of active users per week: 351

Anomalies

  • In June we saw a significant spike in users following the publication of Clean Your Bird Feeders. The number of users in June was almost 10x the average.

Page performance

Top pages by total and unique pageviews:

Top pages by time spent on a page:

Impact

Has publishing our content online led to engagement from our target audience?

In a sense, yes. However, our rates of engagement are quite low. We do not optimise our website or social media use for widespread engagement. The amount of time we put into publishing content on our website and on social media is negligible. Our goals in publishing content online are: to make information publicly available, demonstrate our progress, and encourage corrections & criticism.

We’d like to see an increase in return users as we believe this would indicate our target audience is engaging with our content regularly. We’d also like to see an increase in the average time spent during a session or during page views as we believe this would indicate our audience is reading our longer content i.e., research papers. We will spend some time in T2, 2018 considering how to improve along these two metrics.

Mistakes and Issues

No concrete people management structure

The biggest mistake we made in T1, 2018 was failing to develop a people management process that met the needs of the WASR staff. The WASR staff mean result for:

  • General job satisfaction = 7/10;
  • Satisfaction with the management structure = 6.5/10; and
  • Satisfaction with their relationship with their manager = 9/10

We’ve identified two reasons this occured:

  1. We scheduled our first team feedback session a full year after launching the project. In hindsight, we should have scheduled a feedback session much earlier as this process would have revealed the need to establish a people management structure. We could then have addressed this issue much earlier.
  2. We focused too heavily on project management and output and failed to anticipate the productivity costs of an ill-defined people management structure.

As a result, we intend to make developing and trialling a people management structure a management priority for T2, 2018.

Inaccurate estimates of staff time investment

At the beginning of this year, we released an Evaluation Plan with estimates of staff time investment in all of our core activities. This has proven to be inaccurate on a number of counts. Most significantly:

  • Project management, evaluation and other administrative tasks were estimated at 12%. In hindsight, it is closer to ~20%.
  • Building networks with academia and research institutes were estimated at 5% each. In hindsight, academic outreach work has so far taken up ~15% of staff time.
  • Whilst time spent on attending or presenting at conferences is roughly in line with our estimate (~3%), we failed to factor in productive time lost because of travel.

These activities were predominantly Persis Eskander’s responsibility. To complete them, she de-prioritized her research tasks which meant she failed to meet her research goals for T1, 2018. As a result, we intend to revise our Evaluation Plan in early T2, 2018 to reflect our actual staff time investments. Persis Eskander also plans to frontload her ongoing research projects in T2, 2018 before beginning new outreach activities.

Inaccurate budget estimates

In our 2017 Fundraiser, we estimated a budget of $187,665 USD which would provide for:

  • Funding to hire a new full-time researcher (or run a RFP);
  • Full-time salaries for Persis Eskander and Georgia Ray;
  • A part-time salary for Ozy Brennan;
  • A small allowance to cover expenses for conferences; and
  • Cover WASR’s share of Operations & IT staff time and expenses provided by our parent organization, the Effective Altruism Foundation (EAF).

When we completed the projected budget for 2018, we estimated WASR’s share of operations and IT services at 5% of the total. There were two mistakes that followed:

  1. We miscalculated the total amount of Operations & IT staff time and expenses, which resulted in underestimating our budget. Our projected budget should have been $205,890 USD.
  2. When EAF completed its financial planning in April 2018, it became clear that WASR’s share of Operations and IT staff time and expenses (e.g., rent, hardware, software etc.,) was actually at ~13% of the total. This raised the project budget total to $240,361 USD.

Whilst we met our 2017 fundraising goal, we had to assign a large percentage of the additional funding we received in T1, 2018 to cover our increased budget. To prevent this occurring in future, EAF has reviewed its financial planning process and is in the process of implementing a more rigorous approach. We also intend to engage closely in future with WASR’s portion of EAF accounting.

Priorities for T2, 2018

Research

Research will continue to be an important focus of WASR through T2, 2018. We intend to do the following:

  • Continue producing high-quality research papers and blog posts as per our research plan.
  • Investigate the possibility of publishing some of our work in peer-reviewed academic journals.

Run a small-scale grants competition

The purpose of this activity is to evaluate the level and nature of interest in WAS research from independent researchers.

  1. Organise and publicise a small-scale RFP grants competition targeted primarily at academic researchers and domain experts (but open to independent researchers as well).
  2. Evaluate applications, distribute grants and manage the progress of grantees.

We launched our grants competition on Monday, 6 August 2018.

Establish better project management structures

As noted above, an important goal for T2, 2018 is to establish better project management structures. We intend to do the following:

  • Develop a people management process that meets the needs of the team;
  • Develop organizational policies; and
  • Begin working on approaches to cost-effectiveness analyses for promising interventions.