0:00
/
0:00
Transcript

Infrastructure collapse and full loss of deployed systems

A complete erasure caused by an unresolved hosting bill, wiping out months of coordinated engineering and signalling a severe organisational breakdown

The infrastructure recently completed is now offline because the hosting bill was not settled on time. This failure stopped every service and erased the entire operational environment built to sustain pooled shared resources for the whole network. The financial cost of the server was minimal, yet the accumulated labour invested in its deployment, configuration and security amounted to several hundred hours. That work covered provisioning, identity management integration, secure routing, the full pooled toolchain designed for all members, and the interoperability layers that allowed each component to function as a coherent system. All of it disappeared due to a single missed payment. The issue had been flagged repeatedly, with clear warnings about the risk of a total wipe, but it remained unresolved until the provider removed the instance and destroyed everything running on it.

The impact is severe for the action research network. The platform was engineered to support Moodle, Canvas, expert system development, the production of new one health courses, and the release of open materials. It was a bleeding edge suite offering capabilities far beyond what any single group could sustain independently, yet hosted on a low cost provider to ensure equitable access. The intention was to guarantee that every working group could operate with advanced tools, coordinated processes and a unified structure without bearing individual costs. That foundation enabled not only educational outputs but also research integration, shared data handling and prototype deployment across the entire COST area.

This collapse removed all of it. The system was more than a server. It was an essential backbone for building the next generation of open educational and research infrastructures. Its destruction represents a profound organizational failure and a heavy setback for the collective effort. This failure demands a clear assessment. Poor coordination and weak support collided with proper planning and erased the system that should have anchored the work ahead. The result reflects a lapse in professionalism that must not recur. A network of this scale requires steady organisation, firm responsibility and coherent execution.

The infrastructure was built to embody ethical and open governance. It enabled transparency, shared oversight, real checks and balanced participation. It offered each group the material means to work at the same level, with access to strong tools and a unified structure. These standards must become the norm. Planning, timely action and open dissemination are essential to protect collective progress and ensure continuity. This network exists to strengthen education with committed allies and to model a responsible way of working. The wider world shows how quickly systems collapse when discipline fails and how easily societies turn to harmful paths rather than constructive ones. Our role is to teach a different course and to avoid reproducing the same destructive patterns. To do so, we must uphold the highest expectations at every step and never allow a preventable lapse to undermine the work of many. The network is meant to grow strong, not weakened by disregard for one another’s efforts. Professionalism and responsibility are the minimum required to respect each other’s work and to protect what we build together.

The damage reaches far beyond the technical layer. It disrupts the immediate path toward offering the network a unified workspace designed to guarantee equitable access to advanced digital capabilities at no cost for any member. The intended architecture was built to overcome structural organisational weaknesses by providing a robust system where every group could operate autonomously while remaining aligned. That system required careful engineering and continuous refinement to allow real collaborative work at scale. All of it has now been wiped out.

This outcome is more than a setback. It is a severe signal of dysfunction: the very set of power tools intended to secure coherence, ethical governance and enable proper coordination was destroyed by a basic failure of goodwill and proper coordination. The loss undermines the confidence of participants who dedicated considerable effort to building the platform, and it communicates a message that their work can be erased not by technical error but by inattention to a trivial administrative requirement. It is a tragic case of self sabotage, one that inadvertently devalues the labour contributed by many and casts doubt on the collective ability to protect shared achievements.

The erased system cannot be restored through a simple reactivation of the subscription. Rebuilding demands a full repetition of the previous effort, step by step, component by component. The hundreds of hours of work were not an abstract investment. They represented the transformation of raw infrastructure into a functional foundation for a continent wide initiative. Their disappearance has destroyed ongoing achievements and interrupted the momentum that had finally begun to solidify a common operational space.

There is a narrow but real silver lining. The hosting provider is now offering the possibility to migrate to two dedicated machines built on AMD EPYC 7002 Data Center Series processors. These units deliver far greater computational density, higher parallelisation and stronger I/O throughput than the previous setup. For essentially the same monthly cost once the outstanding fees are settled, the network would gain a backbone capable of doubling the overall capacity. This means smoother video conferencing, stronger real time collaboration, faster file handling and a robust environment for Nextcloud, Moodle, expert systems and any new services required by schools and administrations.

The EPYC 7002 line is engineered for sustained multi core workloads. It brings large cache banks, high memory bandwidth and efficient virtualization, allowing multiple services to run simultaneously without contention. For our use case this translates into stable BigBlueButton calls, low latency for shared editing, reliable data storage operations and room to integrate advanced computational layers such as AI assisted tutoring or simulation modules. It is the type of hardware ordinarily reserved for large institutions, yet accessible to us at a modest cost. Not accepting such an upgrade would be senseless.

Behind this opportunity stands human ingenuity in its best expression. Each component of these systems, from processor architecture to protocol stacks, reflects decades of rigorous engineering, collective research and disciplined cooperation. These technologies did not appear by chance; they were designed by people who worked methodically, respected constraints and maintained standards. The only reason we can deploy a continent spanning educational platform on inexpensive hardware is because those principles were upheld by earlier generations of scientists and engineers.

This is the model to follow. With stable infrastructure, proper responsibility and careful technical work, we can build a sustainable, secure and powerful environment for the whole network. The power tools exist, and we must keep on improving it all for the common good of future generations. The hardware is available and the mission is clear. The challenge is to meet the level of discipline that such systems require.

No further communications or outputs will be issued until the outstanding matter is resolved and a stable foundation is re established. This pause is necessary to prevent additional losses and to ensure that, once work resumes, the infrastructure will be safeguarded against a recurrence of this type of disruption. This text is accompanied by a new video on the human factor as the source of all environmental destruction that is also destroying the preconditions for healthy and prosperous human life.

Discussion about this video

User's avatar

Ready for more?