Fast track to improved power supply reliability

Executive summary

Businesses, hospitals and other mission-critical applications rely completely on the uninterrupted accessibility of their data center resource – and this, equally, depends crucially on the continuous availability of clean power, under all conditions, to every IT and communications device distributed around the data center site. Meeting this power reliability and resilience challenge calls for a power infrastructure as well-thought out and implemented as the data environment that it protects. While each subsystem, component and connection from the utility supply, Uninterruptible Power Supply (UPS) or generator to the load should be best in class and correctly specified for its role, this alone is not sufficient.  When choosing these key power components, designers must cater for their many aspects of mutual interdependence. For example, selectivity must be set up correctly, to ensure that a local equipment problem only shuts down a minimal part of the power network. Another example involves protection of inter-UPS communications, to prevent loss of multiple UPS synchronization. Additionally, systems must be designed for and resilient to unplanned events such as ageing hardware failures, overloads, short circuits, arcing and mistakes made during maintenance to avoid loss of availability, equipment damage or threat to personnel safety. This White Paper discusses how to solve these issues. It looks at ways to design a UPS and power distribution system sufficiently reliable and resilient to support a mission critical data center load, yet proof against interdependence challenges and unscheduled events. While reviewing recommended solutions, it also shows the importance of this inclusive approach by outlining the consequences of failing to make these provisions. Overviews of relevant norms and standards are also given, providing an insight into current expectations of system performance and safety as well as common reference points for suppliers and users. This Paper does not provide in-depth design solutions. Rather, its intention is to make readers aware of all the issues that must be considered and discussed with every party involved when planning a data center power system. The Paper also offers unique, practical hints and tips gained from Eaton’s extensive experience in the field. When planning moves to a more detailed design stage, Eaton offers full support in the form of more in-depth documentation on every relevant topic, and industry experts who can advise on designing for all product, system and operation-related issues.

Introduction

A data center’s critical load is fed from a power distribution network comprising switchgear, overload and short-circuit protection devices, and cables from up to three sources: The incoming utility mains transformer, a standby generator and a UPS in either normal or battery mode. Not considering the different behavior of these power sources can cause unexpected load losses due to slow-acting protection mechanisms. Problem examples include voltage dips too long for a load to ride through, or voltage dips caused by short circuits. Accordingly, we start by examining the impact of the major power network and UPS components, interactions and relevant norms, and then look at fault conditions, operation and maintenance issues.

3- and 4-pole switching

A choice between 3- and 4-pole switching is necessary for each breaker. 4-pole switching is often necessary to prevent back feed and undesired ground links, yet neutral switching is not always appropriate. The choice has cost and space implications, but selecting the wrong pole number can create undesired ground links, causing conductor overheating, dangerous voltage levels or a floating neutral – which could cause harmful over voltages or over currents that are detected.

Use of 4-pole transfer switches and breakers

IEC60364 principles apply to installations in Europe. In earth referenced systems (TN-C, TN-S), system neutral must always remain earth referenced if multiple sources are used. This ensures proper operation and safe voltage levels in electrical installations, in combination with correct protective device functionality if earth faults occur.

4-pole transfer switches may be commonly accepted in systems without UPSs, though not recommended by IEC 60364.

In most European power distribution systems, the neutral has an earth reference at the power source. The requirement to follow IEC 60364 principles becomes crucial when UPS systems are used, due to the installation rather than the UPS. If the upstream neutral is separated from earth reference the electrical system behaves as a floating IT system without a return path for the earth fault. This results in a non-referenced, floating and unsafe distribution system.

Breaker sizing

The actual critical load may be less than UPS nominal rating. The supplier can advise the maximum input current required to charge the batteries and support the maximum (design) critical load. Installations can also benefit from modern digital UPSs with many adjustable parameters and near-unity power factors. This considerably reduces maximum input current for a given UPS kVA rating and often renders traditional feeder selection criteria as outdated.

This fit for purpose rating provides cost benefits for input switchgear, transformer and back-up generator sizing. Adjusting sizes and fault current levels yields some easy cost savings, yet this attractive approach is currently rarely used.

Impact of a UPS in a power distribution system

Using a UPS in a power distribution system adds two power sources to the transformer and generator power which is usually considered; UPSs supply via the inverter or static bypass during normal operation, and battery operation via the inverter only. Each mode has distinct behavior.

Allowing correctly for these UPS behaviors is essential within mission-critical power systems. Possible consequences of not doing so include load loss during a short circuit, and extended repair time and a long period without power availability. Also, neglecting the UPS in a selectivity study could convert a branch fault into a SPoF (Single Point of Failure).

Selectivity

To achieve proper selectivity in a critical power supply system including a UPS installation, the UPS system must be able to sustain fault current until the downstream load branch circuit protection devices can trip, otherwise a branch fault can kill the power to all the loads. This can be a challenge where scalability is wanted, as the initial configuration’s static switch capacity may be insufficient. The required amount and duration of fault current depends on short circuit capacity, impedances and protection settings of the installation and should be studied as a whole.

Eaton UPSs’ unique capability to mix and match power module (inverter) and static bypass capacity allows tailoring the UPS system to meet both the load rating and fault clearing requirements, with optimization for functionality and cost.

Backfeed protection

Backfeed protection is required by UPS Safety Standard IEC 62040-1 – a legally enforceable European Normative – in UPS installations to ensure service personnel safety. The standard allows this protection to be internal to the UPS, or external, in the UPS supply panel. If relying on an external device, the responsibility for fulfilling the minimum legislative requirements lies with the electrical contractor, or owner of the installation, who may lack adequate knowledge of UPS equipment and installation requirements.

Eaton UPSs include factory-installed and tested internal backfeed protection devices; these ensure that safety requirements are fulfilled without needing action from the electrical contractor.

UPS withstand current rating (IEC 62040-1:2008 Amendment1:2013)

Every switchgear and device used in the installation must be capable of safely managing its fault current levels. The product requirements are given in applicable standards, must be followed, and are typically enforced by legislation.

Since February 2016 it has been mandatory, by law, to follow the Amendment1:2013 for UPS Safety Standard requiring UPS manufacturers to declare either Rated short-time withstand current (ICW) or Rated conditional short-circuit current (ICC) and to state the maximum allowed fault current level at the UPS input terminals. The equipment selected must have equal or higher rating than fault current levels in the installation and these values must not be exceeded under any conditions.

Present and possible future fault currents depend on site conditions, affect where the product can be sited, and are key factors in equipment choice – yet few people are aware of this significance.

When conditional ICC rating is used, the fault current is reduced by overload and short-circuit protection devices to a safe level for UPS internal circuits and components. In any case the standard allows this protection to be internal, or externally sited in the UPS supply panel.

If the UPS supplier relies on external protection, this requirement for UPS system safety lies with the installation and becomes the responsibility of the designer, electrical contractor and installation owner. These may lack the knowledge of product specific standards and requirements essential to ensure the protective devices stated by UPS supplier are used and never replaced with a device giving less protection and having higher let-thru energy.

Eaton UPSs have internal overload and short-circuit protection devices (SCPDs) and have been laboratory-tested at up to 100 kA fault current levels. These greatly exceed the standard’s minimum requirements, suiting them for practically any installation without special requirements for the supply panel feeders used. This means less responsibility for designers and contractors, since Eaton manages product safety, and possible future installation changes are less likely to be in the critical path.

Distributed controls

Centralized controllers are considered as Single Points of Failure, as a problem within them can paralyze the whole system. They can also impact the reliability of, or limit, the concurrent maintenance strategy, impacting overall system availability and resiliency.

Eaton uses distributed control architecture where the parallel system single points of failure have been eliminated and load sharing is inherent, operating without communication or signals between units. With thousands of installations worldwide since 1998, this avoids master-slave arrangements and gives unique resilient paralleling of static inverters.

Fault conditions, operation and maintenance issues

So far, this paper has examined the major power system components’ specifications, and how they interact together. However, it is also important to consider the power network problems that can arise during the use of these components, their possible consequences and the options available to resolve and prevent them.

Arc flashes, which mostly occur during maintenance activities, can inflict considerable damage on both personnel and equipment. However, end-users and maintenance providers are not usually aware of arcs and their consequences, so do not specify arc reduction measures. Personal safety and reliability can be improved if the system builder sacrifices redundancies during maintenance and otherwise minimizes arc flash possibilities.

There are some norms to follow: Switchgear must be built to IEC 61439, and we are proud that all Eaton switchgear is verified by testing – the IEC’s most rigid verification method. However potential danger still exists, because the standard does not impose rules for switchgear open arc behavior. IEC/TR 61641 provides guidelines, but most end customers lack detailed awareness of this standard, and do not specify it.

Eaton’s switchgear uses arc-free design, as we believe that ‘prevention is better than cure’. We support this with further mitigation products and strategies:

Arc Flash Reduction Maintenance System (ARMS), available with Eaton Magnum and NRX air circuit breakers (ACBs), is a tighter protection setting that reduces arc energy through accelerated tripping when people are working close to the switchgear. Arc detection systems operate by ‘seeing’ the arc flash, preferably in combination with detecting a current surge. Current can be switched off within 100 ms of detection. Arc killer Arcon reduces equipment damage by reducing arc duration and energy – reducing arc times from 100-50 ms typically with protection settings only, to below 3 ms with Arcon.

Internal separation in Low Voltage (LV) switchgear LV switchgear has multiple functional units. The form of internal separation, as defined by IEC 61439-2, indicates how these functions are separated from one another.

No internal separation means that an entire switchgear unit must be disconnected before connecting a cable or performing any other modification. Increasing levels or forms of separation can reduce the proportion of the board that must be de-energized for maintenance on a unit, but make the switchgear more complicated and expensive to build.

Operational life and maintenance issues and solutions

LV switchgear is typically expected to operate for 20 years or more, with periods between inspections often extending to several months. If switchgear is neglected for long periods, a small problem like a bad connection can develop into a major failure.

Monitoring Eaton provides continuous predictive monitoring of switchgear and UPSs used in mission critical applications. This identifies and recognizes hotspot issues and problems before they become major failures, while providing insight into the state and availability of the switchgear. It also provides supervisory control and data acquisition systems log data.

Maintenance induced errors Total load loss maintenance errors can arise from incorrectly following predetermined sequences in operating breaker switches. The incidence of such errors can be significantly reduced, and power availability improved, with a well-designed key interlock system and a hardware handshake between UPSs and switchgear.

Norms

It is important to be aware of and understand applicable IEC industry standards, as they define safe products and provide common points of reference. To avoid receiving equipment that is IEC-compliant and attractively priced, yet not fit for purpose, users should ensure that their suppliers not only build to relevant IEC standards, but also work to mutually-agreed interpretations of those standards.

UPS equipment suppliers are often the best source of knowledge about product related standards and norms, as well as upcoming changes or new requirements.

The new IEC 61439 series of international standards defines clear regulations for low-voltage switchgear and control gear assemblies. Annex BB.1 of IEC61439-2 provides a good overview of the agreement to be established between the assembly manufacturer and user for power switchgear and control gear assemblies.

Conclusions

In this paper we have seen how the design, installation, use and maintenance of UPS and switchgear components impact power distribution system safety, performance and availability. The consequences of failing to follow good design guidelines and legislative requirements have been discussed.

The discussion makes it clear that consultation with an experienced, established supplier such as Eaton is essential in overcoming complex challenges and achieving guaranteed safety and reliability. Eaton’s expertise translates into equipment with built-in protection features that address key installation and legislative issues, relieving installers of many design responsibilities – including some that they may not even be aware of.

Visit our website www.eaton.eu/datacenters or contact us to access more detailed documentation on these topics, or to discuss your project requirements with one of our industry experts