Heads that don't roll

This is a text automatically translated from Italian. If you appreciate our work and if you like reading it in your language, consider a donation to allow us to continue doing it and improving it.

The articles of Cassandra Crossing I'm under license CC BY-SA 4.0 | Cassandra Crossing is a column created by Marco Calamari with the “nom de plume” of Cassandra, born in 2005.

After disastrous day for Sogei, Cassandra questions the (IT) future of Beautiful country: Is it possible that when it comes to information technology, no one is ever interested in anything and there is never a culprit?

This article was written on April 3, 2022 from Cassandra

Cassandra Crossing 499/ Heads That Don't Roll

A perfect storm has hit PP.AA.'s computer systems; why is everything silent and there are no consequences?

On March 30th a perfect storm hit Italian public IT. A host of vital IT services they suddenly became unavailable, and this resulted in the total stoppage of essential public services.

Throughout Italy, pharmacies could not access electronic prescriptions and dispense drugs, all companies could not invoice, the CERT which monitors and manages the response to IT incidents could not operate, electronic gaming in all its harmful incarnations was blocked, the Revenue Agency services were all at a standstill and all Italian companies could not comply with tax obligations, customs and monopolies had their systems blocked, health cards did not work, green passes could not be issued and verified and the highly publicized IO public apps and Immuni didn't work.

The list goes on and is not complete, but the incident was so far-reaching that even its exact extent is difficult to determine. Certainly essential services such as pharmacies were blocked for a day and a half, but the consequences for those responsible for this situation have not yet manifested themselves.

Why?

Let's rewind the film and start from the beginning. What happened?

According to some rumours, then confirmed in principle by limited official statements, in the Roman datacentre of Sogei, a historic and very large entirely public company, which has always managed a large part of the IT of public administrations, there was a very brief drop in voltage (Acea, the Roman electricity company responsible for supplying the data center, speaks of a duration of one second).

Voltage drops of this magnitude are foreseen at a contractual and service level level, therefore they are "non-events"; nothing abnormal happened.

But the drop in voltage tripped the RCDs that protect some vital network systems, which, doing their job, cut off the power, and this completely isolated the data center's servers, although functioning.

The problem begins now; the technicians attempted to rearm the protections, but were unable to do so. The details are not known, but it took very long, biblical times, even 36 hours, to restore power to the isolated network systems and restart all services.

Now defining such an event as a “perfect storm” is an understatement. Events of this magnitude simply cannot and must not happen. Why?

Because critical infrastructures of this size must absolutely be designed and built with redundancy and decentralization mechanisms designed to absolutely guarantee that all services, perhaps with degraded performance, continue to be available.

And if an overall simple object like the electrical power system is unable to react even to a "non-problem"... somewhere there is a big problem!

What if the power really and completely failed? So what would have happened? Are there appropriate uninterruptible power supplies? Are they tested and verified periodically, as in hospitals and nuclear power plants?

The Italian state (and therefore all of us) pays a lot of money to Sogei so that the services are provided via IT infrastructures designed precisely for these situations.
These are architectures that are much more complex and expensive than normal but which are well known and consolidated, which we know perfectly well how to create, which are, among other things, contractual and legal obligations.

These architectures must and are always periodically and extensively tested to verify that all security and recovery mechanisms work perfectly.
There are regulations and best practices for all these aspects; periodic tests and checks to look for problems or errors in the infrastructure design are the norm.

Now, if a battery of RCDs can really bring public IT to its knees for a day and a half, at least the periodic tests are incorrect, incomplete or omitted, none of this has been guaranteed and the technical and managerial responsibilities are evidently huge.

So why did the news appear like a meteor in the media and disappear just as quickly?

Why aren't we updated daily on the investigations aimed at determining the causes and responsibilities of this "impossible" and inconceivable event?

If the system for verifying the safety and availability of the systems was not adequate or in any case completely failed in its mission, why are there no heads rolling among the responsible managerial and technical roles? Meritocracy may be optional, but “demeritocracy” is essential to the proper functioning of any organization.

It is not Cassandra's job, who does not have adequate information about the situation and events, to explain why; her role as prophetess this time limits her to describing the events that have already occurred and to generically predict a dark future for a country that is unable to guarantee these minimum services to itself and its citizens.

But Cassandra can perfectly identify one of the reasons why we find ourselves in this situation; because the citizens of a country that claims to be among the Greats, despite being affected by serious disservices that simply must never happen, do not metaphorically grab the pitchforks and begin to seek out and prod those responsible.

Even in this case, they, the main victims, become "accomplices" of the veil that covered the whole affair, and once again they resign themselves, hoping that it won't happen again, or at least not to them.

Until the next time…

Marco Calamari

Join communities