By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
.png){kind=logo}
The web as we know it today is, in essence, software running on servers hosted by cloud providers. However, due to the abstraction of software from its execution environment, many users remain oblivious to the fact that a significant portion of the services they utilize could actually be controlled by a single entity. In 2023, Amazon commanded 31% of the cloud market, powering the backend for companies like Netflix, Pinterest, and various other services. This relationship, however, is rarely made explicit, leaving users unaware that their entire service suite is subject to the control of a single company.
Enter blockchain, a technology whose raison d’être is decentralization. Despite this, a large portion of what we refer to as “blockchains” continue to operate on the hardware of these very cloud providers. For instance, in 2022, 1000 Solana validators were taken offline when Hetzner, a cloud provider, prohibited all crypto-related activities.
These examples underscore the critical role infrastructure plays in network decentralization. The Internet Computer, operating on a sovereign network, is not subject to the aforementioned limitations.
How? This is achieved through a network of independent Node Providers (NPs). These can be individuals or entities, who invest capital to acquire a certain number of nodes. These nodes are server machines with specific configurations, available from various hardware vendors, and are subsequently hosted in independent data centers across the globe. This strategy minimizes dependence on any single provider, ensuring that even if a NP’’s nodes are compromised, the overall network remains secure.
This report goes behind the nodes to gain deeper insights into the experiences of these NPs. As the Internet Computer marks its third year of operation, it’s an opportune time to reflect on aspects such as onboarding experiences and key challenges encountered, and to look ahead at how these early pioneering NPs envision the growth of the network of NPs.
At the time of writing, there are 125 NPs listed on the public Internet Computer dashboard. While a questionnaire could have been a viable method of data collection, it would not have yielded the same depth and quality of responses that individual interviews with each NP have provided.
The forthcoming content of this report is derived directly from these primary sources. The NPs interviewed were selected without any specific preference. They were simply those who were accessible and expressed interest in participating in an interview.
In total, 13 NPs were interviewed, four of whom are gen 2 NPs, with the remainder being gen 1. To respect their privacy, the names of these NPs will not be disclosed in this report.
The onboarding journey for NPs on the Internet Computer has been a tale of evolution, marked by both challenges and victories. The pioneers, Generation 1 (Gen1) NPs, navigated through a challenging terrain with limited documentation and a high reliance on DFINITY to get up and running. This led to a significant amount of interaction between these NPs and DFINITY, particularly during the setup phase of the IC nodes. Some reported being on a call with a DFINITY engineer whilst setting up the first IC racks.
For Gen1 NPs who adopted a hands-on approach, this was a considerable hurdle. However, for those who engaged remote hands or similar services, it was a matter of connecting their contractors with the main project leads at DFINITY.
The introduction of Generation 2 (Gen2) NPs brought about some significant improvements. The hardware requirements for Gen2 were less vendor-specific, designed to foster the growth of the Internet Computer and allow for community configurations. The IC wiki also saw the addition of more runbooks and processes, enabling multiple NPs to complete their setup without having to depend on DFINITY’s assistance.
While the documentation is comprehensive, it could benefit from more detailed step-by-step processes. However, NPs concurred that if one is committed to taking the time to understand, there is sufficient information available in the current documentation.
Interestingly, the smoothest onboarding experiences were reported by Gen2 NPs who either had direct contact with DFINITY personnel to help clarify any misunderstandings, or those who were already familiar with these processes, due to their experiences running nodes for other networks. For NPs relying solely on the documentation, further clarification was often needed, creating a bottleneck in the process and resulting in delays.
A common theme among both Gen1 and Gen2 NPs was the criticality of choosing the right data center and ISP for their services. This decision can make or break a project, and for NPs new to the space, it can be a big pain point and pitfall. For instance, NPs looking to operate in emerging markets have faced connectivity costs that are significantly higher compared to countries in Europe and the US, making it less economically viable to run nodes there.
To navigate this crucial step, one recommendation was to find the most well connected party in your region's local telecom industry. For some of the NPs interviewed, this was a connection with a managed services provider, who could then connect them to the other required services (colocation, internet, hardware procurement).
Maintaining nodes for the Internet Computer has been smooth for most NPs, with one NP noting that they have operated for three years without any reported issues at their sites. However, there have been occasional bumps along the way, primarily related to hardware.
One surprise is that the substantial investment into the nodes does not prevent them from being dead on arrival. While this might seem alarming, it is considered a “normal” occurrence in such operations.
When the nodes are up and running, the most common hardware issue reported is the failure of the NVMe SSD drives. One NP reported this issue occurring in four different machines. Yet, considering a total of three years in operation, and that they are covered under warranty, this is not overly concerning.
Perhaps the most significant issue reported by these NPs were instances where they had to replace the entire motherboard of the server. This has occurred once for two NPs. However, given that all the nodes are covered under their original warranty, at least for Dell nodes, these NPs have recorded an overall smooth experience with Dell resolving these issues.
The common workflow to address these hardware issues involves an initial assessment of the node to identify the problem. If it cannot be immediately identified, the firmware is updated to see if that resolves the issue (Dell will recommend doing this anyway before proceeding further). If not, logs from the machine are collected and sent to Dell when opening a support ticket, followed by collaboration with Dell to find a solution.
However, when the issue is not hardware-related and support is needed, NPs have found it challenging to get consistent and reliable service. There doesn’t seem to be a single synchronous communication channel with DFINITY to resolve issues in a timely manner. At times, the issues aren’t necessarily difficult to solve, but the time delay makes it a pain to resolve.
For some, mainly Gen1 NPs, remote troubleshooting has not been possible for the majority of their operation, necessitating on-site visits. Sometimes, fixes that could have been implemented remotely through technology like IPMI have to be done on-site. Although DFINITY has released a runbook for this, for some NPs, this arrives near the end of their operating agreement, making them question if it is worth completing this procedure.
For other NPs, namely those already experienced with running validators, they have been successful in setting up remote access and thus, streamlining their troubleshooting processes. However, it’s still unclear from the IC level how to check the health of the node. It’s entirely possible that a node may appear fine at the hardware level, but NPs are still in the dark at the IC level. This may change with the recent release of the Trustworthy Node Metrics.
As this was just recently released, for the moment, NPs cannot fully detach from DFINITY when it comes to seeking support for troubleshooting their node issues.
Specific numbers related to individual NPs cannot be disclosed, but this section will dive into general insights into the financial operations of these NPs.
The initial capital expenditure (CapEx) for NPs, contingent on vendor and purchase location, typically ranged from €11K to €18K per node. Operational expenditure (OpEx) per rack in Western Europe, inclusive of colocation and connectivity costs, was estimated to be between €2500 and €6000 per rack. The most significant portion of the OpEx, which almost all NPs agreed on, was the Internet Service Provider (ISP) cost.
The NPs that were able to cut the most costs from their operations were the ones with established relationships. More concretely, some of the Gen2 NPs that were interviewed have leveraged existing connections to secure preferential pricing across all operational aspects. But these connections aren’t the only way to get a better deal. For another NP, it was strong negotiation skills that helped in obtaining lower prices. This displays the potential for newcomers in the field to negotiate favorable rates with data centers.
That being said, all interviewed NPs agreed that collaboration among NPs to coordinate in the purchase of new hardware, or approaching data centers and ISPs, would result in the most benefit in terms of reducing costs.
Regarding rewards, two primary strategies have been observed. "Larger" NPs, defined here as those with another significant income source, tend to retain most of the rewards in ICP. In contrast, NPs with a more "hobbyist" profile often liquidate a portion of their rewards to cover operational costs, then leaving the remainder in ICP.
At present, most NPs employ their own methods for managing rewards. However, a few have expressed interest in utilizing a digital custody solution, such as a multi-signature, multi-coin wallet, to facilitate easier management of their earnings, especially if such a wallet is hosted on the IC. One common use-case reported by the NPs for such a solution is that it would make it more transparent for their respective accounting firms to report their earnings.
There is consensus among NPs that the network needs to continue to expand. To truly become a global computer, the network needs to match or exceed the size of large cloud providers like Amazon. This doesn’t just mean acquiring more machines, but also expanding into new markets. Currently, there is a high concentration of NPs in Europe and the US.
The overarching objective is to establish nodes owned by NPs in as many countries as possible, with the ideal scenario being representation in every country worldwide. While the configuration of an IC node rack remains relatively consistent, the associated costs, particularly operating expenses, can vary significantly across different markets.
One of the most prominent cost factors is the connectivity requirements of these nodes. In certain markets, particularly emerging ones where node setup would be ideal, these costs can be prohibitively high. For instance, in Kenya, quotes for a 1G flat rate (sufficient to support three nodes) were reported to be no less than $10,000 USD per month, a cost barrier that has deterred many potential setups.
Beyond the financial aspect, technical challenges also exist when setting up nodes in these emerging markets. For example, one NP reported that during an in-depth look of the project’s connectivity requirements, the ISP shared that IPv6 setups are not so common in Thailand.
The network is indeed progressing in a promising direction. A notable advancement is the implementation of the latest remuneration models, which are designed with a granularity based on individual countries. However, it’s important to recognize that rewards are just one facet of supporting the growth of the NP network.
Another crucial element is fostering a sense of community among NPs. Despite the current tendency for NPs to operate independently, they are, in essence, part of a collective team. The sharing of information and experiences could be mutually beneficial and contribute to the overall growth and success of the network.
One potential avenue for cultivating this sense of community is through organized events, such as the ICP Community Conference (in addition to the coming NP ICP Lab of course). The inclusion of a dedicated track for NPs at such events could provide a valuable opportunity for interaction, collaboration, and the exchange of ideas.
Continuing this theme of community is the idea of establishing a working group. The lifecycle challenges of an NP are quite similar. The problems faced by one NP will likely be faced by another. A working group that comes across these issues can work to fix them and provide a well researched recommendation to the NP community at large. For example, if there is an NP that has a great technical setup and monitoring stack, this could be shared with all NPs.
The network at Genesis was characterized by a limited number of large NPs, a fact that remains evident today. This initial structure was necessary to bootstrap the network and ensure a degree of reliability. However, as the Internet Computer (IC) matures, it is becoming increasingly apparent that the network is evolving to favor NPs operating a smaller number of nodes.
Therefore, if the IC experiences higher demand, a significant increase in the number of NPs will be needed to match that. This highlights the importance of collaboration as it would streamline numerous processes and eliminate redundant work currently being undertaken by multiple NPs in silos. This collaboration is the key to realizing the network’s expansion objectives, fostering a more decentralized and robust system.
Despite their crucial role, NPs are largely passive in voting processes. This conclusion comes from a series of similar responses from all the interviewed NPs, revealing a complex web of reasons behind their lack of active participation.
The first hurdle is the lack of an efficient system to track new proposals. Many NPs confessed to being unaware of when a new proposal is up for voting, indicating a communication gap.
Secondly, the technical complexity of some proposals presents a formidable barrier. One NP noted that understanding the impact of certain proposals requires a deep comprehension of the protocol's intricacies - a level of expertise that many NPs do not possess, and shouldn’t have to. Consequently, even when a proposal directly affects them, NPs would still follow the votes of DFINITY.
Lastly, the time investment required to engage in discussions around other proposals is often seen as outweighing the potential benefits. This perception further discourages active participation.
Collectively, these factors have led to a somewhat paradoxical situation where the entities with the most potential voting power - the NPs - are largely passive, often merely echoing the votes of the foundation. While this may not pose an immediate problem, it raises questions about the long-term health of the system. After all, if most are simply following the foundation's lead, can the Network Nervous System (NNS) truly be considered a democratic system as originally envisioned?
While NPs may appear to play a passive role in governance, their involvement in the network extends far beyond simply providing compute capacity for the Internet Computer.
Consider projects like Helix Markets, Estate DAO, and Hot or Not. These are just a few examples of ecosystem projects that are powered by NPs. In some cases, the NP is a developer, building their envisioned application on the Internet Computer. Alternatively, NPs may choose to reinvest their rewards into the ecosystem through these applications. Both strategies create a virtuous cycle, with NPs who are already invested in the Internet Computer doubling down on their investment.
Interestingly, these NPs are not only taking care of the supply side by running their nodes, but they are also helping to drive demand by aiding the development of apps that utilize the very infrastructure they've set up.
If this trend continues, NPs could emerge as potential funding sources for the next wave of decentralized applications (dApps) on the Internet Computer. Alternatively, application developers might consider setting up nodes themselves, providing an additional income stream while their application is still in development.
In essence, NPs, or at least the ones interviewed as a part of this report, are not just passive participants in the network; they are active contributors, shaping the future of the Internet Computer ecosystem.
In the rapidly evolving rate of development of the IC, NPs are seeking clarity on the future. A common request from all NPs is for a clear roadmap outlining the foundation's plans for the network over the next 3-5 years.
For many NPs, their involvement in the network has proven to be a profitable investment, and there is a strong interest in continuing their participation. However, they need insight into the future, particularly regarding remuneration and its conditions. If the status quo were to remain, it's likely that NPs would continue their involvement. But in an industry characterized by constant evolution, this is rarely feasible.
This request for a roadmap is not without its challenges for the foundation, and this was acknowledged by the NPs. Nevertheless, the decisions NPs need to make involve significant planning, and many of these decisions are not easily reversible. Therefore, having advanced knowledge of the roadmap is crucial for NPs to strategize their next steps.
Despite the uncertainties, many NPs express a desire to remain involved in the network and contribute to its growth, whether from an infrastructure or ecosystem perspective.
As the IC celebrates its third anniversary, it's remarkable to witness the network's growth. A crucial part of that has been the dedicated network of NPs. As the technology continues to evolve, so does the interest in becoming an NP. Currently, a diverse range of entities, from businesses, to staking services, and educational institutions are in the process of registering as NPs.
It's crucial to reflect on the experiences of those who have been part of the ecosystem thus far, which is why this report was made. Learning from their experiences can provide valuable insights to improve the process for the next wave of NPs. But is this a task for DFINITY?
As the network of NPs continues to expand, the only sustainable way to support this potential global scale of operations is through collaboration and knowledge sharing among NPs.
Despite operating in different parts of the world, NPs are working with the same technology and likely facing similar challenges. If NPs can collaborate, share, and publish the information they've gained through their years of operation, it could significantly reduce the barriers to entry for new NPs. It could also reduce uncertainty for all NPs who encounter certain issues, as solutions have likely already been worked out.
The result is a more efficient network, where NPs don't waste time solving problems that have already been solved. Instead, they can focus on other aspects that can potentially further promote and boost the IC. This collaborative approach could transform the network of NPs into catalysts for innovation, driving the next phase of growth for the Internet Computer.
.png)
