PART I

Thoughts on Relay diversity and neutrality

The Ethereum network is a manifestation of software and infrastructure, as well as people’s commitment to develop, maintain and run it. Doing so comes at a human, energy and dollar cost currently supported by the community that benefits from it: users, companies and protocols (via their DAO or foundation).

One key element of protecting Ethereum as a network is to achieve software diversity at every level and as much decentralization as possible. This is why the Ethereum foundation and other members of the community have been very involved in pushing for more Client diversity especially going into the Merge.

Why is software diversity important?

“Multiple, independently developed and maintained clients exist because client diversity makes the network more resilient to attacks and bugs. However, it is not enough simply to have multiple clients available, they have to be adopted by the community and the total active nodes distributed relatively evenly across them.” - ethereum.org

The same logic needs to be applied at all layers of the Ethereum stack: not only execution clients and consensus clients but also blocks builders and relays.

• Block builders have the power to decide which transactions go in a block and in which order - this is how MEV is produced. They can also implement rules like OFAC filtering to censor transactions.
• Relays have the responsibility to decide which valid blocks (trust assumption) get relayed to proposers based on profitability. They are the gatekeepers of ~85% of the blockspace because block builders need to go through a relay to have an opportunity to include their blocks on chain by being picked by MEV-boost for a proposer.

Currently Relay diversity is bad with one relay - Flashbots - relaying ~80% of all MEV-boost blocks. Since MEV-Boost is used by ~80% of the network, that means Flashbots controls the access to ~64% of the total Ethereum blockspace (MEV and non-MEV).

Currently 93% of relays are running on Flashbots implementation which means that if there is a bug in that implementation at least 386K proposers currently running Flashbots relay would be badly affected - unless they are also running a Dreamboat based relay which is currently the only other open-souce relay implementation.

Currently more than 80% of the Ethereum network is served by MEV-Boost so a bug in flashbots code would have a big impact and potentially cause the ethereum blockchain to produce several empty slots in a row, it could even become a DOS vector. To mitigate this risk, improving software diversity is essential.

Why is Flashbots in a dominant position and why does it matter?

Simply put, Flashbots is a research and development organization that pioneered the MEV sector. They set out to solve MEV related problems and bring more transparency to this market.

They created reference tooling to increase diversity and censorship-resistance in Ethereum by building their own implementation of PBS (proposer-builder separation) with MEV-Boost. They unlocked a new and exciting market for block builders, searchers and validators. They enabled the ecosystem to be less centralized around validators by decoupling the proposer role and the builder role.

To sum up, what Flashbots set out to do is good and beneficial to Ethereum but, as it often is with innovation, it has come with some unforeseen issues that we now need to analyze transparently and collectively deal with.

As pioneers, they were the only actor for a long time and open-sourced the source code of their relay close to the Merge. This was done to encourage other actors to run relays. Most of the open source relays currently are forks of Flashbots except for Dreamboat. This is good news from a decentralization perspective and bad news from a software diversity perspective as 93% of open source relays are running the same source code.

The Flashbots Relay currently handles blocks produced internally - by their own builders - but it also accepts other builders’ blocks to foster builder diversity.

A problem is that most block builders need to take a fee to support their operation - it’s part of their business model. When a builder takes a fee, the proposer gets a smaller share of the profit from the block which makes the block less profitable for the proposer and the block builder less competitive than a builder that doesn’t take a fee.

Since MEV-Boost is set to pick the most profitable block for the proposer and since Flashbots builders don’t take a fee, it means that more often than not Flashbots builders are the most competitive.

Another factor that limits external builders competitiveness is that they are rate limited which means they can only send 60 blocks / minute - 12 blocks per 12 seconds slots - while internal flashbots builders don’t have that limitation. This impacts negatively the inclusion rate of external block builders.

Open sourcing the code was a first step in the direction of diversification, however the current approach of Flashbots seems to be making the market uncompetitive by not taking profit and by not fully succeeding at leveling the playing-field despite efforts.

The problem is, searchers really want to capture MEV and will share their bundles to whoever has the best inclusion rate. Currently with ~80% MEV-Boost dominance, Flashbots builders are the obvious choice. It means that Flashbots will keep on getting the highest volume of MEV opportunities from searchers because their blocks have a better inclusion rate.

But that only makes the problem worse because validators, although they could source blocks from 1, 2 or 8 relays via MEV-Boost, prefer to focus on Flashbots - some might use another relay as backup in case Flashbots have an issue. It is surprising to see such behavior when a simple action to help the situation could be for validators and pools to run MEV-boost with multiple relays instead of just Flashbots. There are no additional costs to run MEV-boost with multiple relays.

In other words, builders engaged in the MEV-boost market are not able to compete fairly against Flashbots builders because:

• Builders who decide to use the Flashbot relay to benefit from a relay with high inclusion rate end up having less profitable blocks comparatively and are rate-limited,
• Builders who decide to use minority relays have less chance at getting included on chain because the relay they are using has a small blockspace share (for example 1% for relayooor versus 82% for flashbots) and they might also get rate limited because of the builder-relay integration but at least they are competing against builders that are taking a profit.

The builder's dilemma

Now it’s important to understand that the role of relays will be embedded in-protocol in the future. However there is no set timeline for that and we can assume that it might not be the case before several years so we have to deal with this situationtoday.

Several years is more than enough time to harm Ethereum if nothing is done to fix the current situation at the relay and builder level.

PART II

Why is achieving relay diversity hard today?

If we want to shift the narrative here, we should work on increasing the blockspace share of the different relays so that one relay doesn’t have the super-majority of blockspace.

Competing successfully in a market where Flashbots is acting as a public good with private funding is hard right now for Relays.

Flashbots currently offers the best level of service there is as a Relay and a Builder while not taking any profit from the proposers. As the best performing relay it can use its dominant position to offer inclusion rates that are better than anyone which further reinforce dominance: it’s the snowball effect.

Why is the Relay market an uncompetitive one?

One simple reaction to the situation would be: minority relays should come up with ways to provide a better service so that people use them instead of Flashbots and accept to pay for some extra features. After all it’s not Flashbots’ fault if they are good at what they do! To some extent, this is a very valid point, but it’s an oversimplification of reality.

Let’s talk about the notion of barriers to entry.

A high barrier to entry means that it is difficult to join a market due to high start-up costs or other obstacles (political, regulatory, environmental). A low barrier to entry means can easily join in and most of Web3 native markets fall in that category. Especially in an open-source world where anyone can fork or get inspiration from someone else’s repo and run their own service.

In order to survive in a low entry barrier market you need a very competitive offer because users can decide to go with a better option at any moment. You also need a strong business model to be profitable while also remaining attractive to users. A key element for users is pricing. Another one can be performance or quality.

If the Relay market has a low barrier to entry and if we already have 2 open source implementations of relay’s specifications available - Flashbots and Dreamboat, why are we not seeing a strong decrease in Flashbots dominance?

Why doesn’t Flashbots’ dominance decrease?

Actually, it has been increasing since the Merge.

Yes the relay market has a low barrier to entry. But it is important to note that it’s not that it is hard to enter the Relay market (all are welcome and encouraged to), but that it is hard to survive in it. Why? Because Relays currently don’t have a business model. The MEV is shared between proposers - who get the lion’s share - and the crumbs go to searchers and builders. The Relay gets zero part of the MEV reward - and users don’t as well, but that’s another story.

So if you want to compete as a Relay you need to find a way to be profitable.That’s why vertical integration is increasing which is diminishing Relay neutrality.

What that means it that most relay operators are also block builders. Simply put, they make money by relaying the blocks they are building themselves. They have an incentive to favor their own builders’ block to generate a profit which means external builders using the relay are not treated equally.

Vertical integration is hurting relay neutrality. It's a business model Relays had to adopt since there is no incentivization for them. The only way to make $ is to have blocks included onchain as a builder. But then why relay blocks of others if you can't make $ from it... https://t.co/JRoqYv56RS

— SajZ 🦇🔊🐼 (@sajidazouarhi) November 3, 2022

The lack of incentives for permissionless Relays creates an artificial barrier to entry as one will not enter a market if they can’t be sustainable after doing so.

Barriers to entry, of any kind, result in the decrease of competition.

Let’s take the ride-sharing sector as an example of a low entry barrier market that turned into a “winner-takes-all” market.

Uber is a private company that has been running their business at a loss since its foundation thanks to the support of their investors - Uber reported 2 billion dollars of loss in 2015 alone. This was done in order to offer unbeatable prices to users. This strategy of “predatory pricing” was key to put them in a leader (close to a monopolistic) position.

Any competitor of Uber that didn’t have enough backing from investors and was trying to run their company with a sustainable business model was failing because they could never decrease their price to match Uber’s. Most competitors ended up quitting that market after losing the price war.

Uber benefits from the “data snowball effect” which means that the large amount of data they collect helps them improve their services, which in turn attracts more users and makes them more dominant.

Different intent here but similar result: Flashbots currently has more than 80% network dominance and is pricing out competition. Flashbots also benefits from the compound effects of having the largest blockspace access which further increase its dominance.

It is important to note that I don’t believe Flashbots’ strategy to be like Uber’s. Without Flashbots we probably wouldn’t be having such interesting problems to solve now. The goal of using this example is to highlight the simple fact that poordynamics can create an uncompetitive market and should be addressed.

Flashbots acts currently as the price maker for the block building sector, and the price they are setting is $0. Meaning all the builder-relays trying to compete for a share of blockspace need to align on this price to be competitive. However, this price doesn’t allow those builder-relays to be sustainable in the upcoming years as they don’t have enough capital to sustain their operations indefinitely. So what can be the outcome?

Well for example the Ethereum foundation, validators pools and other key actors could create grants to support Relay diversity and neutrality. In the meantime we could collaborate in the upcoming years on building a better market or just wait it out until Relays disappears.

Asking Relays to price below their costs - well in this case we are asking Relays to operate for free -  is not a reasonable ask and is in fact harmful for the market because it creates negative effects such as vertical integration. Another end-result of a winner-takes-all market, as can be seen in stock markets and other zero-sum systems, is that it leads to even more inequalities (the rich get richer). There is no second best in block building, either your block gets on chain or not. You either win or you lose. The more Flashbots dominance is important, the more people will want to use Flashbots’ Relay to increase their access to opportunities, which will further assert their dominance.

Until we decide to make some changes here, there is no way this trend can reverse by itself.

As long as Flashbots are taking zero profit and giving away premium blockspace access for free, there is not much that other companies can do to compete. Running a relay costs ~60K per year in infrastructure without accounting for the cost of employees and 24/7 devops monitoring and support.

In a web3 world, an uncompetitive market directly impacts network resiliency and neutrality.

A simple idea:

Flashbots builders could start charging a fee for their service. This will organically improve Builder diversity as Flashbots’s inclusion rate will naturally drop thus increasing other builders’ inclusion rate. Flashbots’ builders would turn a profit and Flashbots’ relay would reduce dominance which in turn would help create healthier market conditions.

I looked at payloads delivered by Flashbots' builders and compared them to the second best price of other builders.

From this, I calculated the possible profit if FB had taken a fixed % profit.

Big assumptions though... (1/n) pic.twitter.com/zbdG3S7jLX

— Bert (@BertKellerman) November 2, 2022

Other ideas to explore:

• Flashbots’ builders could share blocks with other relays to bootstrap adoption
• We could come up with a business model to incentivize Relays

Risk of not looking for a solution:

• Fewer well maintained and secure Relays will emerge and last
• Vertical integration of builder-relay will increase
• Less R&D and engineering investment will be deployed into block building, most Relays will be forks with similar bugs and vulnerabilities
• Lack of software diversity means one team can make a decision that impacts more than 60% of all Ethereum’s blocks - the centralization vector will remain

Interesting reads:

• The risks of vertical integration in MEV-Boost by chayoterabit
• Ideas for Incentivizing Relays by dmarz
• How much can we constrain builders without bringing back heavy burdens to proposers? (the risk of builders centralization) by vbuterin