Research: Bittensor, flawed?

This post on Bittensor was first published on my personal blog, Teng’s Thoughts, in April 2024.

A search for decentralised collective intelligence led me to Bittensor. Would the Bittensor model be able to bootstrap AI networks in specific domains?

My findings were startling and made me pause.

Bittensor has several critical flaws.

A Quick Recap

Bittensor operates as a decentralised incentive network for AI models.

Across its 32 subnets, miners deploy their AI models to compete based on specific criteria. Validators then assess and rank these outputs, and miners earn TAO tokens based on their relative performance. Currently, these subnets are handling tasks such as text generation, image generation, text-to-speech, and model fine-tuning.

TL;dr — it’s horse racing for AI models. The best horse (model) gets the greatest rewards.

The big idea is that monetary incentives will naturally attract the top-performing models and innovators to the most relevant subnets.

I first shared some of my early views on Bittensor in November last year. Since then, Bittensor has grown to become the darling of Crypto Twitter, the flagbearer for the decentralised AI movement. As of today, Bittensor's market capitalization is at $2.8 billion, with a fully diluted valuation of $8.5 billion.

A Rocky Road Ahead

In my opinion, the biggest issues with Bittensor are:

1. Horizontal economics are broken

Incentive-driven competition within subnets to produce the best outputs can make sense — IF implemented properly.

Notice the bold, capitalised “IF”.

Challenges such as the gaming of incentives, relay mining, and assumptions about validators' knowledge of truly correct outputs exist. I believe these problems can be solved with enough research and experimentation. Several subnets have already taken steps or revised their criteria to discourage relay mining and other exploitative practices.

However, the economic model starts to fall apart when competition extends horizontally.

Subnets have to compete with one another to get a larger allocation of inflation rewards. This will become even more pronounced once dynamic TAO is implemented (where subnets will have their own dynamic tokens and token holders can affect the distribution of rewards between subnets)

This begs the question:

How should a TAO holder compare the economic value to the Bittensor network between very different subnets? For example:

(1) a text-to-speech model (subnet 3)

vs

(2) a vision model that screens X-rays for disease? 

Value generation in each case is entirely different. You can spin this question in many ways, but the answer is never clear-cut. Relying on market forces to determine each subnet's value contribution is not optimal for long-term strategic development.

It’s like having every AAPL shareholder vote on the next product Apple should build.

2. Competitive Challenges for Subnets

I’m sceptical that Bittensor subnets can be competitive in bread-and-butter generative AI tasks, such as general image and text generation. Already there are open-source projects, often motivated by reputation, research contributions, and public service, that excel in these areas.

Take, for example, Meta's release of its powerful large language model (LLM), Llama-3, as an open-source tool. Despite the billions of dollars invested in its development, Llama-3 is freely accessible and has garnered significant community support. Since its release last week, it has been downloaded over 500,000+ times on Huggingface.

Comparative benchmarks show that open-source LLMs like Llama-3 are already state-of-the-art, and arguably on par with proprietary models such as OpenAI’s GPT-4.

To date, I have not seen Bittensor subnets achieving outcomes or capabilities that exceed those of existing open-source models. Neither have I seen evidence suggesting they outperform established benchmarks. At some point, you have to ask yourself if they ever will.

3. High costs of running the intelligence network

The network relies heavily on its native token, TAO, to function. Miners join the network primarily when they identify a computing arbitrage opportunity—specifically when the mining rewards exceed the computation costs.

The demand side for TAO is contingent on the following: 

(1) Speculators (a large can of worms we won’t get into here)

(2) Addition of more Bittensor subnets, or increased demand to become miners or validators. However, this growth tends to dilute the incentives for existing participants. A drop in token price leads to a reflexive drop in network activity as the unit economics for miners/validators fall under pressure.

Bittensor has substantial operational costs, with a daily inflation of 7,200 TAO paid to subnet owners + miners + validators. This equates to ~$3.2 million per day or $1.2 billion annually.

This raises a critical question: Where does this funding come from? Predominantly, it's speculators. But is Bittensor producing $1.2B in value in a year? Its valuation has probably gotten too far ahead of itself.

4. Transparency

Although Bittensor offers some level of decentralization, it is important to note that it is NOT open source—a crucial distinction. While participation in a subnet is open to anyone, the internal mechanics are not transparent, leaving users with a limited understanding of the activities within:

• What are the miners doing?

• What algorithms are they running?

• How are the final results generated?

This lack of transparency can restrict the variety of applications that can be developed using subnet outputs, as many developers require a clear understanding of the underlying inference processes.

Some Reflections

I’m not anti-Bittensor. In fact, I find it one of the most intriguing experiments in the emerging Crypto x AI sector.

It has successfully rallied a community of smart people around its vision. It is likely to inspire numerous future projects. Yet, Bittensor must address its current challenges head-on to stand any chance of achieving real-world success — otherwise it will be lost in the graveyard of speculative hype.

A few closing thoughts:

• Greater synergy could be achieved if all subnets within the network were aligned toward specific sectors rather than dispersed across a variety of unrelated tasks.

• For instance, I envision a "Bittensor for genomic research/drug discovery" or a "Bittensor for finance/trading." that can actually do very well. Such vision alignment fosters economic cohesion, strategic direction, and business development efforts.

• Bittensor's economic model — in its current state — may best serve niche areas of AI that do not attract widespread attention but could benefit from targeted incentives. Potential applications include crypto-specific tasks, such as predicting token prices.

Finally, I firmly believe that decentralisation in AI has an extremely important role in the world — acting as a counterbalance to Big Tech's global dominance and fostering innovation, particularly through open-source initiatives.