Ofwat has published a new paper setting out how it has used collaborative customer research to set Outcome Delivery Incentive (ODI) rates for the 2024 price review (PR24) in which the regulator acknowledges the limitations of the data collection process.

Early on in the research process, some of the water companies commented on the risks of the results of collaborative ODI research coming out late in the business planning process and necessitating late changes.

The paper includes the indicative ODI rates Ofwat now expects the water companies in England and Wales to use in their AMP8 business plans, which are due for submission to Ofwat on in just under 6 weeks' time on  2^nd October 2023.

"Companies must use these indicative rates in their business plans or provide compelling evidence to support any alternatives"

The paper says:

“As part of our quality and ambition assessment, companies must use these indicative rates in their business plans or provide compelling evidence to support any alternatives.

“Working with companies and industry, we will continue to learn and build on this approach to set ODI rates as part of PR24 determinations and at future price reviews…

“By using consistent valuations across the industry, each company has been given the same indicative incentive rate for a unit improvement in performance eg £ per minute of supply interruption to use in their business plan submissions.”

Ofwat opts for "top-down approach" to set indicative ODI rates rather than "bottom-up" approach

For the majority of Performance Commitments (PCs) which underpin the ODIs, Ofwat had originally planned to use a 'bottom-up' approach to set indicative ODI rates primarily based on estimates of marginal benefit from the collaborative customer research.

However, Ofwat has been unable to complete the mapping exercise for half of the Performance Commitments (PCs).

The paper says that without robust marginal benefit estimates from the collaborative customer research and mapping exercise, Ofwat has therefore decided instead to set all indicative ODI rates using a 'top-down' approach based on equity return at risk. The approach is where incentive rates are derived from a company's overall potential payments, for example as a proportion of a company's regulatory capital value (RCV) divided over a selected performance range, and informed by customers' priorities and regulatory judgement.

It is clear from the paper that both the design of the survey questionnaire and the mapping activity which formed part of the research have presented significant challenges.

For PR24 Ofwat set out to design and implement collaborative customer research that would enable it to set indicative ODI rates in a consistent way across the sector. The ODI Rates Research was a major sector-wide customer engagement exercise led by Ofwat and conducted in partnership with CCW. The regulator adopted a collaborative approach to working with companies and stakeholders which enabled Ofwat to take account of their experience in conducting research in the water sector.

The ODIs are intended to align the interests of companies and their investors with the interests of customers and the environment by directly linking performance with expected financial returns.

Previously, at the 2019 price review (PR19), the water companies proposed their own ODI rates as part of their business plan submissions, with supporting evidence. According to Ofwat, the companies' analysis at that time had resulted in customer valuations which varied between companies, with “no credible underlying drivers” that could explain such variation.

Ofwat therefore adopted a new approach for PR24, saying that a greater emphasis on common Performance Commitments provided an opportunity to do things differently, resulting in the same research being commissioned to underpin the setting of indicative ODI rates across all companies.

.

However, the regulator encountered a number of challenges when undertaking the mapping from service incidents customers valued to PC definitions. Ofwat explains:

“Mapping was more complex than anticipated and relied on good quality historic data, for example how likely it is that drought restrictions would be imposed with changes in leakage levels. In some areas this data was readily available, for others we were unable to identify robust data sets in the time available. This meant we were unable to complete the mapping exercise for half of the PCs.

“Where we were able to complete the mapping exercise, the majority of the rates were outside the range of expectations. Some rates were very low, some implausibly so, and would not sufficiently incentivise companies to improve their performance. Other rates were very high, some implausibly so, and were not consistent with the ±1 to ±3% return on regulatory equity(RoRE) each year that we set out in the final methodology.”

Further qualitative research supported Ofwat’s concern that the rates significantly exceeded customers’ underlying willingness to pay.

The paper says Ofwat has continued to use the collaborative research to ensure that the incentive rates reflect the importance customers place on each outcome:

“The use of a top-down approach across all PCs ensures consistency in how we set rates across the package of ODIs. And the collaborative customer research informs the indicative ODI rates we set, with a higher proportion of regulated equity being assigned to the areas prioritised by customers. “

Customer surveys were first used to assess demand for services in the 1999 price review and

have featured in every price review since. The surveys often involve respondents having to make trade-offs based on risks around service delivery and/or putting monetary values on individual or multiple service attributes based on stimulus material within the research.

According to Ofwat, in many other sectors where these methods are used, eg transport, research participants can draw on “well-formed preferences based on experience, but in water the stimulus material is in part educational and preferences have to be formed based on this.”

However, according to Ofwat, the application of these methods “has always been challenging”. The paper cites research commissioned by the Consumer Council for Water (CCW) in 2019 which found that many customers had “low awareness and interest in the sector overall and some felt that related decisions were too technical for them to make.”

“Large-scale customer sample survey … was largely untested which carried risks”

Ofwat’s starting point for the research was a desire for ODI rates to be, as far as possible, driven by customer preferences. The paper says:

“Despite inherent difficulties, a large-scale customer sample survey containing questions relating to some form of stated preference choice methodology is the clearest way in which to give voice to a large

and representative body of both household and non-household customers.

“This method represented a radical step towards a more customer-focused experience of water and wastewater services, but it was largely untested which carried risks. A more meaningful engagement with customers also incurs a trade-off in the sense that the research generates a set of valuations which are a step removed from PC definitions, requiring an additional mapping stage. This was recognised as being analytically and conceptually challenging for some PCs, but we did not have significant doubts about its feasibility at this stage.”

The household survey took place between July 2022 and September 2022, whilst the non-household survey took place between August 2022 and October 2022.

Looking ahead to future Price Reviews, Ofwat says:

Working with companies and industry, we will continue to learn and build on this approach to set ODI rates at future price reviews. Such considerations could include:

further work to assess of the ease of understanding and familiarity of the service issues customers are being asked to rank and value;

investigating the drivers behind high non-household valuations and further improvements that can be made to the sampling and reliability of results;

how challenges of the mapping approach might be overcome, for example developing research questions in tandem with the PC definitions to identify data requirements early on in the process to achieve a robust link through the mapping approach; and

the role of customer valuations in informing ODI rates either directly or indirectly.

Click here to download