SCADA Notes

Ignition platform overview#

Ignition by Inductive Automation is the dominant SCADA platform at US solar plants. The architecture in one breath: a central gateway server runs device connections (drivers for Modbus, DNP3, Allen Bradley, OPC UA and more), which feed tags, which feed the historian and the operator screens (Vision or Perspective). Everything a technician sees on a screen traces back through tag, then device connection, then network, then field device; troubleshoot in that order from the gateway web page (Status section shows device connection health and tag counts).

Site Ignition versions and module sets vary; verify against the site system documentation.

Tags and tag quality#

Every Ignition tag carries a value, a timestamp, and a quality. Good quality means the driver read the point recently. Bad or Bad_NotConnected means the device connection is down. Stale means the subscription stopped updating. Uncertain often means the device answered with a flagged or substituted value. Quality propagates: one dead device connection turns every tag under it bad at the same moment, which is the signature to look for. A single bad tag among good siblings on the same device points at that one register or address, not at the network.

Quality names vary slightly across Ignition versions.

OPC UA connections#

Ignition talks to many PLCs through OPC UA, either its own internal server fronting the native drivers or an external OPC UA server. A faulted OPC UA connection shows every downstream tag bad while the underlying PLC is fine; the gateway Status page distinguishes the OPC connection state from the device state. Certificate expiry is the classic silent killer: OPC UA connections authenticate with certificates that expire years after commissioning, taking the connection down on a seemingly random day.

Check OPC UA certificate validity dates during annual maintenance.

Historian#

The tag historian stores tag values in a database for trends and reports. Gaps in trends mean either the tag went bad quality (comms problem) or the historian pipeline stalled (database disk full, database connection down, store and forward buffering). Store and forward matters in the field: when the database link drops, the gateway buffers history and backfills later, so a trend that fills itself in hours later indicates a database side problem, not a field problem.

Database disk space is the first check for plant wide historian gaps.

Why a tag shows stale or bad quality#

A stale or bad tag is almost never the SCADA server itself; it is the messenger. The chain, from most to least likely: the field device stopped answering (power, fault, address change); the network path broke (fiber, media converter, switch); the polling configuration flags a slow but healthy device as timed out; the device connection in SCADA is misconfigured after a device swap; or the tag itself points at a register that no longer exists in new firmware. Correlation is the diagnostic tool: all tags on one device bad means the device or its path; all devices on one segment bad means network; scattered single tags bad means register map drift.

Check the timestamp of last good value; it usually marks exactly when the field event happened.

Ovation as SCADA and plant controller#

At utility owned plants, Emerson Ovation often runs the whole control room: operator graphics, historian, alarm system, and the power plant controller logic in the same redundant DCS. Point quality and alarm behavior follow DCS conventions rather than Ignition conventions, and configuration lives with the site engineering group. For a field technician the working interface is usually the Ovation operator station: point detail displays show quality, scan status, and the controller that owns each point.

Ovation configurations are site specific; work through the site engineering owner.

Independent confirmation: proving a device is really down#

Read the scope note in Real fault or comms artifact first: this matters for autonomous devices, far less for a device that cannot act without its controller. The flaw in judging a device by its own telemetry is circular. When the channel is down, the thing reporting that the device is fine is the same thing that is broken. THE PRINCIPLE. Verify a device by testing the effect of its function through an independent controller, an independent path and an independent power source. If the equipment is really doing its job, something else in the plant has noticed, and that witness is worth more than anything the silent device could say about itself. Everything below is one instance of this single idea, which is why it is worth learning as a principle rather than as a list: it lets you improvise a check at a site whose equipment you have never seen. The names, if you want the literature. High availability computing solved this decades ago and the vocabulary transfers cleanly. SHARE NOTHING is the rule that confirming evidence must have no component in common with the thing being confirmed, because any shared component is a common mode failure waiting to fool you. FENCING is forcing a suspect node into a known state instead of inferring its state, which is exactly what cutting power is. A WATCHDOG is the timer that declares a node dead when it stops checking in, which is what a heartbeat with a dwell is. Borrowing the names is useful because it points at a body of work where these failure modes are already catalogued. The revenue meter counter test, as the example. The clearest instance, and it needs no site visit: the meter is separately powered, separately wired, and read over a different path, so it shares nothing with the silent device. Watch the kWh totalizer across the outage. If the counter stalls during the gap and then JUMPS when comms restore, the meter kept accumulating the whole time, so the equipment was running and only the telemetry was down. If the counter stalls and then resumes counting flat from where it stopped, production genuinely stopped. The same logic transfers to any independent witness: a transformer that still sees current, a tracker row visibly following the sun, a station service load still drawing. Hardwired alarm and fail contacts. Utility substation practice, and worth copying wherever the equipment supports it: the protective relay's alarm or fail contact is wired to a separate I/O device rather than reported over the same protocol as its measurements. Then the dispatch logic becomes explicit. Loss of comms alarm alone means suspect the channel. Loss of comms alarm PLUS an asserted relay fail contact means the device itself is in trouble, and that combination is what sends a technician. HOW LONG BEFORE YOU CALL IT DEAD. Asking for the right number of seconds is the wrong shape of question, and a fixed plant wide timeout is the usual bad answer: it produces nuisance alarms on slow links and dangerous silence on critical ones. Derive it per device the way a failure mode and effects analysis does. Ask what the actual effects are if THIS device is unreachable for longer than X, and whether those effects cause a problem anyone cares about. A revenue meter quiet for ten minutes costs ten minutes of resolution in a report. A protection relay quiet for ten minutes is a different conversation entirely. The dwell falls out of the consequence, not out of a site convention. A cheap technique where the protocol supports it: put a ROLLING COUNTER inside the heartbeat rather than a static value. A static heartbeat only proves that something answered. A counter that increments proves the far end is still executing its program, rather than a stale cached value or an intermediate device answering on its behalf. The size of the gap in the sequence after recovery also tells you how much you actually missed. WHAT TO ALARM ON FOR A CONTROL ISLAND. When a section runs its own control with its own safety devices, silence by itself is not worth waking anyone for. The conditions that are worth an alarm are that the section is out of control, meaning it is no longer holding its process within limits; that it is not processing its inputs, so work is piling up in front of it; or that it is not producing its outputs, so whatever depends on it downstream is being starved. Each of those is a real consequence, and each can usually be observed from outside the island by the same independent witnesses the principle points at. Alarming on those rather than on loss of comms is the difference between an alarm list people act on and one they learn to ignore. Protocol level counters. Most protocols expose retry, timeout, and frame error counters that turn a subjective impression into a number. As one example, a DNP3 I/O server may flag points as bad value after roughly three retries. There is no standard: the threshold varies between utilities, vendors, and I/O servers. Know your own system's retry count and dwell, because that number is what separates a device that is genuinely gone from one that is merely slow.

Thresholds and dwell times vary by utility, vendor, and I/O server; confirm the values configured on your own system. Contributed by substation practitioners.

Switch side checks: is the path down, or the device?#

Most of the evidence in a device offline call sits on the network equipment, not on the device, and most of it can be read without leaving the control room. Pathing, before anything else. Look at the box the device hangs off and ask whether everything downstream of it went quiet at the same moment. If every device on that switch, media converter or radio dropped together, the device is almost certainly fine and the shared path is not. One device quiet while its neighbours on the same switch keep reporting points the other way. This check costs nothing, takes seconds, and reorders the entire job. Switch port status. Managed switches report port state remotely, which is the cheapest hard evidence available. Link down means the physical path is broken: cable, connector, media converter, transceiver, or the device's own network interface. Link up with the device still unreachable means the physical layer is intact and the problem sits above it, in addressing, the protocol, the master's configuration, or the device application. That single reading splits the problem in half. Ping from the right place. Pinging from wherever the SCADA poll originates only tests the whole chain at once, so a failure tells you nothing about where. Ping from the switch itself, or from an engineering station on the same segment. If the device answers locally but not from the SCADA host, the device is up and the fault is in between: routing, a firewall rule, or the master side. That is not a truck roll. Link transitions and error counters. The switch log timestamps every link up and link down event. A port that has flapped repeatedly points at a marginal cable, connector or transceiver rather than a dead device. Rising CRC or frame error counters on the port mean the physical layer is degraded and passing corrupted frames, which is again a cable, connector or noise problem rather than an application fault. A stable link with clean counters moves suspicion off the physical layer entirely. Once someone is on site: the LEDs. Lit means the device has power and link, so it is up but unreachable and the fault is above the physical layer. Dark means it is down. That is the fastest determination available in the field, and it is worth asking the technician to report it before touching anything else. CUTTING POWER TO FORCE CERTAINTY, and the limit that decides whether it works. Removing power and watching what changes is fencing: it forces a known state instead of inferring one. But the guarantee only holds if whatever cuts the power is itself on an independent network and an independent power path from the device. Command a breaker or a relay over the same network that is already suspect, or through the same power supply, and you have not proved the power went off; you have only sent a request into the same fog. Where you cannot guarantee an independent path, confirm through a side effect instead: current draw stopping, a contactor dropping out, a downstream measurement collapsing. And it stays last regardless, because it destroys the evidence of the original state.

Never remove power from field equipment to test a theory outside the site switching and lockout procedures. Exhaust the read only checks first, and remember that a power cycle erases the very state you were diagnosing.

Switch management access, log detail and counter names vary by vendor and by site policy. On many plants the network equipment is not yours to log into; follow the site cyber security procedures and ask the person who owns it.

Real fault or comms artifact? Scope it first#

SCOPE THIS FIRST. The distinction only earns its keep for AUTONOMOUS devices: equipment that keeps doing its job whether or not anything is talking to it. A central inverter, a tracker controller, a protective relay and a revenue meter all carry on regardless of who is listening, so down versus merely unreachable is a real difference and it changes what you do about it. For a device that cannot act without its controller, the question largely dissolves. A VFD taking speed commands from a PLC is the common example: with the command path gone it is not doing useful work either way, and a correctly configured drive should already have initiated its own reaction to loss of command, whether that is a controlled stop, a hold, or a fail safe state. If it did not, that is the finding. Practitioners have challenged this entire framing on exactly that ground and they are right to, so establish which kind of device you are looking at before spending any effort on the rest. For autonomous devices, the most valuable SCADA skill in solar O&M is telling a real equipment fault from a communications artifact before rolling a truck. Real faults: production actually drops, neighboring measurements corroborate (the meter sees less power, the transformer sees less current), and the device logs a fault locally. Comms artifacts: values freeze or go bad quality while plant output at the meter stays flat, many devices change state at the same timestamp, and the device checks out healthy at its local display. When SCADA and the revenue meter disagree, believe the meter. Better still, do not rely on the telemetry path alone: hardwired fail contacts, switch port status, and independent witnesses to the device's actual function give evidence over a different path. See Independent confirmation and Switch side checks. A caveat from people who do this daily. Experienced technicians push back on the premise itself, and they have a point: roughly nine times in ten it turns out to be a cable, and in plenty of shops somebody is driving out regardless of what the diagnosis says. So be honest about what the distinction actually buys. It does not decide whether you fix it. It decides what you dispatch and how urgently: a network technician with a cable tester and a spare transceiver is a different truck from an equipment technician with a laptop and a spare board, and a comms artifact on a plant that is still producing is not the emergency that a dead inverter at noon is.

Never reset or cycle field equipment purely because SCADA shows it offline. Confirm at the device first.

Cross check against the revenue meter and neighboring device telemetry.