The problems with relative ESG ratings and different benchmarking processes

ESG ratings are increasingly being used as a tool to assess ESG (Environmental, Social, Governance) performance and to align investors and companies’ commitments and actions. A few ESG rating providers focus on absolute ESG performance, while most rating providers assess companies based on relative ESG performance. Thus, most ESG ratings are relative assessments compared to the overall ESG performance of the industry sector of the respective company. Some sectors (e.g. IT) tend to have a higher overall ESG performance than other sectors (e.g. defence). Therefore, a company would have a different ESG rating depending on whether it is rated relative to the IT sector or relative to the defence sector. Sustainability risks can be omitted as some companies face more than just the defined set of ESG industry risks by operating in several sectors. The company’s industry allocation influences the final ESG rating. This exemplifies the importance of the rating providers benchmarking processes, defining a company’s peer sector. Research has shown that how data providers define companies’ peer industry groups is crucial in determining ratings (Kotsantonis and Serafeim, 2019). Unfortunately, there is substantial disagreement among rating providers on the question to which sector a certain company belongs to. Facing Finance investigates this issue for the defence sector, analyzing arms companies that are allocated to the defence sector by some rating providers, while being allocated to other sectors by other rating providers.

Some defence companies that are part of the ExitArms database due to their arms exports, make a large part of their sales with armaments, but are not assigned to the Aerospace and Defence sector by all ESG rating providers. ESG ratings which evaluate companies relative to the performance of the respective industrial sector, therefore compare these arms companies not to the defence sector, but to other sectors – thereby influencing the ratings of these companies. Currently, some investors reconsider their perspective on the defence industry and ask themselves whether certain arms companies are investable for ESG investors. Therefore, the need for transparent and accurate ESG data on the defence sector is high. In light of this and the methodological discrepancies within ESG ratings in the defence sector, this article analyses a selection of arms producing companies, their industry allocations and ratings by six different leading ESG rating agencies: Refinitiv, ESG Book, ISS, Sustainalytics, S&P and MSCI.

Therein, we look if there is a common ground between the ESG ratings of the selected arms companies and their sector allocations. First, the current industry allocations of the selected set of companies: The sixteen companies Indra Sistemas, Kongsberg Gruppen, Otokar, Israel Shipyards Ltd, Honeywell International, Fincantieri, VSE Corporation, Kawasaki Heavy Industries, Mitsubishi Heavy Industries, Melrose Industries, Exail Technologies, Inner Mongolia First Machinery Group, Cohort Plc, Leidos Holding, Abu Dhabi Ship Building and Mazagon Dock Shipbuilders are not allocated to the Aerospace and Defence sector by at least some ESG rating providers despite the fact that a large amount of their revenues are dependent on armament.

Key findings for sixteen arms companies and six leading ESG rating providers[1]:

1. 5 out of the 16 arms companies were not allocated to the Aerospace and Defence sector by any of the ESG rating agencies: Indra Sistemas, Otokar, Fincantieri, Melrose Industries and Abu Dhabi Ship Building.
2. There is considerable disagreement among ESG rating providers on which companies belong to the defence sector. We have analyzed sixteen arms exporters of the ExitArms database and their industry allocations of different ESG rating providers. Sustainalytics allocates none of these sixteen arms exporters to the Aerospace and Defence sector. The remaining ESG rating providers allocate some of these arms exporters to the defence sector and others not: Refinitiv (3 companies were allocated to the defence sector: Kongsberg Gruppen, VSE Corporation, Mazagon Dock Shipbuilders), S&P (2: Kongsberg Gruppen, Israel Shipyards), ESG Book (7: Honeywell International, Kawaski Heavy Industries, Mitsubishi Heavy Industries, Exail Technologies, Inner Mongolia First Machinery Group, Cohort Plc, Leidos Holding), ISS (2: Kongsberg Gruppen, Honeywell International), MSCI (1: Kongsberg Gruppen).
3. A number of companies clearly belongs to the arms industry [based on the percentage share of arms sector business in terms of the company’s total turnover] but is nonetheless not allocated to the defence sector by one or more out of the six rating agencies. These companies include:

Indra Sistemas SA – [with an arms sector percentage share of 34%], but not allocated to the defence sector by any of the rating agencies.

Kongsberg Gruppen – [40%], but not allocated to the defence sector by ESG Book and Sustainalytics.

Honeywell International – [39%], but not allocated to the defence sector by Refinitiv, MSCI, S&P and Sustainalytics.

Fincantieri – [71%], but not allocated to the defence sector by any of the rating agencies.

VSE Corporation – [71%], but not allocated to the defence sector by ESG Book, ISS, S&P and Sustainalytics.

Melrose Industries – [100%][2], but not allocated to the defence sector by any of the rating agencies.

Exail Technologies – [50%], but not allocated to the defence sector by ISS and Refinitiv.

Cohort Plc – [91%], but not allocated to the defence sector by Refinitiv.

Leidos Holding – [57%], but not allocated to the defence sector by Refinitiv, ISS, MSCI, S&P and Sustainalytics.

4. The 16 arms companies are instead allocated to different sectors by the six rating agencies. It is vital to consider the discrepancies that exist even within such allocations among rating agencies. Arms companies that are not allocated to the defense sector by various rating agencies, are allocated to a broad range of different industrial sectors. For instance, the arms company Kongsberg Gruppen is allocated to the sector Trucks, Construction & Farm Machinery by ESG Book and to the sector Industrial Conglomerates by Sustainalytics. Other examples include:

Otokar: Motor Vehicles (ESG Book), Heavy Machinery & Vehicles (Refinitiv), Machinery and Electrical Equipment (S&P), Machinery (Sustainalytics)

Fincantieri: Trucks, Construction & Farm Machinery (ESG Book), Machinery and Electrical Equipment (S&P), Shipbuilding (Refinitiv, Sustainalytics)

VSE Corporation: Trucks, Construction & Farm Machinery (ESG Book), Industrial Support Services (ISS), Commercial Services & Supplies (S&P), Commercial Services (Sustainalytics)

Kawasaki Heavy Industries: Consumer Goods Conglomerates (Refinitiv), Industrial Machinery & Equipment (ISS), Machinery and Electrical Equipment (S&P), Industrial Conglomerates (Sustainalytics)

Melrose Industries: Auto Parts (ESG Book), Consumer Goods Conglomerates (Refinitiv), Industrial Conglomerates (ISS, S&P, Sustainalytics)

Leidos Holding: Technology (Refinitiv), IT Consulting & Other Services (ISS), Software & Services (MSCI, Sustainalytics), Professional Services (S&P)

5. For each company and each rating agency, we compared the ESG company score with the average score of the respective sector:

• Considering differences between a company’s ESG performance when allocated to the defence sector by some rating agencies and to other sectors by other agencies, we see that some companies (Kawasaki Heavy Industries, Mitsubishi Heavy Industries, Leidos Holding), receive ESG ratings similar to the average of the defence sector by those rating agencies that allocate the respective company to the defence sector, while receiving ESG ratings better than the average by those rating agencies that allocate the respective company to another sector.
• Overall, for ESG company ratings where the company is allocated to the defense sector, we observe six ratings that are similar to the average of the defense sector, two ratings that are better than average and six ratings that are worse than average. On the other hand, for ESG company ratings where the company is allocated to another sector (not defense), we observe ten ratings that are similar to average, twenty-two ratings that are better than average and thirteen ratings that are worse than average. Thus, ESG ratings where the company is allocated to the defense sector tend to be worse than the average of the defense sector, while ratings where the company is not allocated to the defense sector tend to be better than average. However, the sample size of sixteen arms exporting companies is too small to draw any wider conclusions.

What does this imply and what are its impacts?

Generally, ESG ratings have varying approaches and measures that rating agencies take on. The lack of official definitions and regulation guiding the applied methodologies remains a source of confusion and misunderstandings about ESG ratings (Berg, et al., 2022). There are several initiatives of regulators and market participants to set standards, but so far, they did not bring clarity and alignment on what ratings intend to measure. The ESG ratings market remains largely unregulated and misses comprehensive supervisory or regulatory frameworks for ESG ratings providers (IOSCO, 2021).

The correlation between ESG ratings of different agencies is quite low. Studies find correlations of 0.31 (Chatterji et al., 2014), 0.32 (Capizzi et al., 2021), 0.53 (Mayer and Reizingerné Ducsai, 2022) or 0.54 (Berg, et al., 2022), depending on the samples. The ESG ratings divergence can dampen firms’ motivation to improve their ESG performance, since it is not clear what they should focus on (Hassan, 2022). The low correlation and disagreement between ESG ratings can be partly attributed to a lack of commonality in the definition of environmental, social and governance components (Berg et al. 2019; Billio et al. 2020). This disagreement leads in particular to discrepancies among ESG indices, with very low agreement rates on the constituents of comparable indices; in terms of coverage and sector composition. (Mazzacurati, 2021)

The composite scores provided by rating agencies can confuse investors and mislead their sustainable decision-making process given the inability to differentiate between impact materiality and financial materiality. Adding to this, investors decisions are often misled when they go by a best-in-class investment approach, which is usually based on a relative ESG rating. For example, ESG ratings of Refinitiv, MSCI, Bloomberg or S&P Global that measure financially relevant ESG risks relative to peer industries, tend to provide better scores to companies with a more controversial weapon export track record. First, these ratings give better average ESG scores to companies that export arms to war zones than to defense companies that do not. Second, within the group of companies with controversial arms exports, a higher involvement in arms exports to war zones is associated with better ESG scores. The misunderstandings and widespread unawareness of discrepancies in ESG ratings result in green washing allegations when retail investors invest in defence companies that make deliveries to armies involved in war crimes and human rights violations.

What should the future of ESG ratings look like?

There are several problems about the current state of the ESG ratings market, including:

1. Lack of clarity about ESG ratings objectives: Most ESG rating providers focus on financial materiality. This means that most ESG ratings assess the possible financial consequences of sustainability issues on the business and financial performance of rated companies. Most ESG ratings only measure a company’s financial sustainability risks, i.e. how E, S and G factors can cause financial losses. On the other hand, broader society usually thinks about impact materiality, i.e. the impact that companies have on society and the environment. Retail investors often believe that ESG ratings assess the social and environmental sustainability of a company, its products and business practices. However, most ESG ratings are not at all concerned with whether the rated company is doing business responsibly (Novisto, 2023; Grandjean, 2023). Consequently, there is a large difference between the reality of ESG ratings and the expectations of broader society. This misunderstanding frequently causes confusion and green washing allegations.
2. Combining different factors into one single rating: ESG ratings combine very different environmental, social and governance elements into a single, synthetic score. When providers aggregate largely unrelated E, S and G factors into a single rating, it helps to conceal weaknesses whose later emergence might hurt public trust in ESG ratings.
3. Lack of transparency and conflicts of interest: Some rating providers have become significantly more transparent over the last few years, but overall transparency in the ESG ratings market still must be increased. Especially transparency about critical problems with ESG ratings is still low. For example, ESG rating providers are often under-sourced, having very few analysts at disposal compared to credit ratings. Conflicts of interest are often hidden, instead of being transparently addressed or avoided. Many ESG rating providers provide consulting services to the same companies that they rate in their ESG ratings and some even rate affiliated companies such as their own parent companies.
4. Large disagreement among ESG ratings: The correlation between ESG ratings of different rating providers is very low. The ESG ratings divergence potentially disperses the effect of ESG performance on asset prices, decreases companies’ incentives to improve their ESG performance and makes it difficult to tie borrowing conditions or CEO compensation to ESG performance. Reasons for this high disagreement among ESG rating providers include different approaches to materiality, different methods for weighting and aggregating factors into a single rating, different approaches to relative or absolute ESG performance. Most ESG ratings are relative to the peer industry of the respective company. Therefore, the benchmarking process – how the rating provider defines the peer industry of a company – influences the final ESG rating. For the defence industry, Facing Finance is showing how the different benchmarking processes cause companies to be allocated to the defence sector by certain rating providers, while being allocated to other sectors by other rating providers. These differences also influence the ratings of these companies.

ESG ratings are crucial for sustainable finance and shall help investors to take investment decisions by providing an assessment of investee social, environmental, and governance factors. However, effective regulation is required to ensure that ESG ratings fulfil their task of supporting sustainable investing. Currently, the European Union is discussing an ESG ratings regulation. Following which, civil society has been showcasing the current problems that exist within ESG ratings and how they should be regulated in order to add value to investors while supporting the allocation of capital to sustainable activities.  Civil society has put forward recommendations for EU policymakers in order to improve the reliability, comparability and transparency of ESG ratings (Philipponnat, 2023; Transport and Environment, 2023). In this process, Facing Finance provides an example of the current difficulties with ESG ratings by showing how the different benchmarking processes of ESG rating providers lead to differences in the assessment of defence companies.

As of now the EU Council has agreed on a position related to the upcoming ESG ratings regulation. With this, negotiations about a final legal text between the EU Council and the EU Parliament will start in January and are expected to conclude before the EU election in June 2024. The Council's position would require raters to make it clear if they used financial and/or impact materiality when assessing a company, making it easier to assess the objective of a rating and to avoid confusion. The Parliament's position stresses upon mandatory disaggregation of ESG data - splitting up aggregated scores into individual E, S and G parts to avoid misleading users. (Azizuddin, 2023). The EU regulation opens the chance to fix the current ESG ratings market dysfunctionalities and to make ESG ratings more accurate and real sustainability ratings. In the future, clear, transparent, and well-regulated ESG ratings could be a meaningful basis of sustainable investing and support the transition towards more sustainable economies.