674 lines
19 KiB
Plaintext
674 lines
19 KiB
Plaintext
|
|
|
|
EDAC - Error Detection And Correction
|
|
|
|
Written by Doug Thompson <norsk5@xmission.com>
|
|
7 Dec 2005
|
|
|
|
|
|
EDAC was written by:
|
|
Thayne Harbaugh,
|
|
modified by Dave Peterson, Doug Thompson, et al,
|
|
from the bluesmoke.sourceforge.net project.
|
|
|
|
|
|
============================================================================
|
|
EDAC PURPOSE
|
|
|
|
The 'edac' kernel module goal is to detect and report errors that occur
|
|
within the computer system. In the initial release, memory Correctable Errors
|
|
(CE) and Uncorrectable Errors (UE) are the primary errors being harvested.
|
|
|
|
Detecting CE events, then harvesting those events and reporting them,
|
|
CAN be a predictor of future UE events. With CE events, the system can
|
|
continue to operate, but with less safety. Preventive maintainence and
|
|
proactive part replacement of memory DIMMs exhibiting CEs can reduce
|
|
the likelihood of the dreaded UE events and system 'panics'.
|
|
|
|
|
|
In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
|
|
in order to determine if errors are occurring on data transfers.
|
|
The presence of PCI Parity errors must be examined with a grain of salt.
|
|
There are several addin adapters that do NOT follow the PCI specification
|
|
with regards to Parity generation and reporting. The specification says
|
|
the vendor should tie the parity status bits to 0 if they do not intend
|
|
to generate parity. Some vendors do not do this, and thus the parity bit
|
|
can "float" giving false positives.
|
|
|
|
The PCI Parity EDAC device has the ability to "skip" known flakey
|
|
cards during the parity scan. These are set by the parity "blacklist"
|
|
interface in the sysfs for PCI Parity. (See the PCI section in the sysfs
|
|
section below.) There is also a parity "whitelist" which is used as
|
|
an explicit list of devices to scan, while the blacklist is a list
|
|
of devices to skip.
|
|
|
|
EDAC will have future error detectors that will be added or integrated
|
|
into EDAC in the following list:
|
|
|
|
MCE Machine Check Exception
|
|
MCA Machine Check Architecture
|
|
NMI NMI notification of ECC errors
|
|
MSRs Machine Specific Register error cases
|
|
and other mechanisms.
|
|
|
|
These errors are usually bus errors, ECC errors, thermal throttling
|
|
and the like.
|
|
|
|
|
|
============================================================================
|
|
EDAC VERSIONING
|
|
|
|
EDAC is composed of a "core" module (edac_mc.ko) and several Memory
|
|
Controller (MC) driver modules. On a given system, the CORE
|
|
is loaded and one MC driver will be loaded. Both the CORE and
|
|
the MC driver have individual versions that reflect current release
|
|
level of their respective modules. Thus, to "report" on what version
|
|
a system is running, one must report both the CORE's and the
|
|
MC driver's versions.
|
|
|
|
|
|
LOADING
|
|
|
|
If 'edac' was statically linked with the kernel then no loading is
|
|
necessary. If 'edac' was built as modules then simply modprobe the
|
|
'edac' pieces that you need. You should be able to modprobe
|
|
hardware-specific modules and have the dependencies load the necessary core
|
|
modules.
|
|
|
|
Example:
|
|
|
|
$> modprobe amd76x_edac
|
|
|
|
loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
|
|
core module.
|
|
|
|
|
|
============================================================================
|
|
EDAC sysfs INTERFACE
|
|
|
|
EDAC presents a 'sysfs' interface for control, reporting and attribute
|
|
reporting purposes.
|
|
|
|
EDAC lives in the /sys/devices/system/edac directory. Within this directory
|
|
there currently reside 2 'edac' components:
|
|
|
|
mc memory controller(s) system
|
|
pci PCI status system
|
|
|
|
|
|
============================================================================
|
|
Memory Controller (mc) Model
|
|
|
|
First a background on the memory controller's model abstracted in EDAC.
|
|
Each mc device controls a set of DIMM memory modules. These modules are
|
|
layed out in a Chip-Select Row (csrowX) and Channel table (chX). There can
|
|
be multiple csrows and two channels.
|
|
|
|
Memory controllers allow for several csrows, with 8 csrows being a typical value.
|
|
Yet, the actual number of csrows depends on the electrical "loading"
|
|
of a given motherboard, memory controller and DIMM characteristics.
|
|
|
|
Dual channels allows for 128 bit data transfers to the CPU from memory.
|
|
|
|
|
|
Channel 0 Channel 1
|
|
===================================
|
|
csrow0 | DIMM_A0 | DIMM_B0 |
|
|
csrow1 | DIMM_A0 | DIMM_B0 |
|
|
===================================
|
|
|
|
===================================
|
|
csrow2 | DIMM_A1 | DIMM_B1 |
|
|
csrow3 | DIMM_A1 | DIMM_B1 |
|
|
===================================
|
|
|
|
In the above example table there are 4 physical slots on the motherboard
|
|
for memory DIMMs:
|
|
|
|
DIMM_A0
|
|
DIMM_B0
|
|
DIMM_A1
|
|
DIMM_B1
|
|
|
|
Labels for these slots are usually silk screened on the motherboard. Slots
|
|
labeled 'A' are channel 0 in this example. Slots labled 'B'
|
|
are channel 1. Notice that there are two csrows possible on a
|
|
physical DIMM. These csrows are allocated their csrow assignment
|
|
based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
|
|
is placed in each Channel, the csrows cross both DIMMs.
|
|
|
|
Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
|
|
Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
|
|
will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
|
|
when 2 dual ranked DIMMs are similiaryly placed, then both csrow0 and
|
|
csrow1 will be populated. The pattern repeats itself for csrow2 and
|
|
csrow3.
|
|
|
|
The representation of the above is reflected in the directory tree
|
|
in EDAC's sysfs interface. Starting in directory
|
|
/sys/devices/system/edac/mc each memory controller will be represented
|
|
by its own 'mcX' directory, where 'X" is the index of the MC.
|
|
|
|
|
|
..../edac/mc/
|
|
|
|
|
|->mc0
|
|
|->mc1
|
|
|->mc2
|
|
....
|
|
|
|
Under each 'mcX' directory each 'csrowX' is again represented by a
|
|
'csrowX', where 'X" is the csrow index:
|
|
|
|
|
|
.../mc/mc0/
|
|
|
|
|
|->csrow0
|
|
|->csrow2
|
|
|->csrow3
|
|
....
|
|
|
|
Notice that there is no csrow1, which indicates that csrow0 is
|
|
composed of a single ranked DIMMs. This should also apply in both
|
|
Channels, in order to have dual-channel mode be operational. Since
|
|
both csrow2 and csrow3 are populated, this indicates a dual ranked
|
|
set of DIMMs for channels 0 and 1.
|
|
|
|
|
|
Within each of the 'mc','mcX' and 'csrowX' directories are several
|
|
EDAC control and attribute files.
|
|
|
|
|
|
============================================================================
|
|
DIRECTORY 'mc'
|
|
|
|
In directory 'mc' are EDAC system overall control and attribute files:
|
|
|
|
|
|
Panic on UE control file:
|
|
|
|
'panic_on_ue'
|
|
|
|
An uncorrectable error will cause a machine panic. This is usually
|
|
desirable. It is a bad idea to continue when an uncorrectable error
|
|
occurs - it is indeterminate what was uncorrected and the operating
|
|
system context might be so mangled that continuing will lead to further
|
|
corruption. If the kernel has MCE configured, then EDAC will never
|
|
notice the UE.
|
|
|
|
LOAD TIME: module/kernel parameter: panic_on_ue=[0|1]
|
|
|
|
RUN TIME: echo "1" >/sys/devices/system/edac/mc/panic_on_ue
|
|
|
|
|
|
Log UE control file:
|
|
|
|
'log_ue'
|
|
|
|
Generate kernel messages describing uncorrectable errors. These errors
|
|
are reported through the system message log system. UE statistics
|
|
will be accumulated even when UE logging is disabled.
|
|
|
|
LOAD TIME: module/kernel parameter: log_ue=[0|1]
|
|
|
|
RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ue
|
|
|
|
|
|
Log CE control file:
|
|
|
|
'log_ce'
|
|
|
|
Generate kernel messages describing correctable errors. These
|
|
errors are reported through the system message log system.
|
|
CE statistics will be accumulated even when CE logging is disabled.
|
|
|
|
LOAD TIME: module/kernel parameter: log_ce=[0|1]
|
|
|
|
RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ce
|
|
|
|
|
|
Polling period control file:
|
|
|
|
'poll_msec'
|
|
|
|
The time period, in milliseconds, for polling for error information.
|
|
Too small a value wastes resources. Too large a value might delay
|
|
necessary handling of errors and might loose valuable information for
|
|
locating the error. 1000 milliseconds (once each second) is about
|
|
right for most uses.
|
|
|
|
LOAD TIME: module/kernel parameter: poll_msec=[0|1]
|
|
|
|
RUN TIME: echo "1000" >/sys/devices/system/edac/mc/poll_msec
|
|
|
|
|
|
Module Version read-only attribute file:
|
|
|
|
'mc_version'
|
|
|
|
The EDAC CORE modules's version and compile date are shown here to
|
|
indicate what EDAC is running.
|
|
|
|
|
|
|
|
============================================================================
|
|
'mcX' DIRECTORIES
|
|
|
|
|
|
In 'mcX' directories are EDAC control and attribute files for
|
|
this 'X" instance of the memory controllers:
|
|
|
|
|
|
Counter reset control file:
|
|
|
|
'reset_counters'
|
|
|
|
This write-only control file will zero all the statistical counters
|
|
for UE and CE errors. Zeroing the counters will also reset the timer
|
|
indicating how long since the last counter zero. This is useful
|
|
for computing errors/time. Since the counters are always reset at
|
|
driver initialization time, no module/kernel parameter is available.
|
|
|
|
RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
|
|
|
|
This resets the counters on memory controller 0
|
|
|
|
|
|
Seconds since last counter reset control file:
|
|
|
|
'seconds_since_reset'
|
|
|
|
This attribute file displays how many seconds have elapsed since the
|
|
last counter reset. This can be used with the error counters to
|
|
measure error rates.
|
|
|
|
|
|
|
|
DIMM capability attribute file:
|
|
|
|
'edac_capability'
|
|
|
|
The EDAC (Error Detection and Correction) capabilities/modes of
|
|
the memory controller hardware.
|
|
|
|
|
|
DIMM Current Capability attribute file:
|
|
|
|
'edac_current_capability'
|
|
|
|
The EDAC capabilities available with the hardware
|
|
configuration. This may not be the same as "EDAC capability"
|
|
if the correct memory is not used. If a memory controller is
|
|
capable of EDAC, but DIMMs without check bits are in use, then
|
|
Parity, SECDED, S4ECD4ED capabilities will not be available
|
|
even though the memory controller might be capable of those
|
|
modes with the proper memory loaded.
|
|
|
|
|
|
Memory Type supported on this controller attribute file:
|
|
|
|
'supported_mem_type'
|
|
|
|
This attribute file displays the memory type, usually
|
|
buffered and unbuffered DIMMs.
|
|
|
|
|
|
Memory Controller name attribute file:
|
|
|
|
'mc_name'
|
|
|
|
This attribute file displays the type of memory controller
|
|
that is being utilized.
|
|
|
|
|
|
Memory Controller Module name attribute file:
|
|
|
|
'module_name'
|
|
|
|
This attribute file displays the memory controller module name,
|
|
version and date built. The name of the memory controller
|
|
hardware - some drivers work with multiple controllers and
|
|
this field shows which hardware is present.
|
|
|
|
|
|
Total memory managed by this memory controller attribute file:
|
|
|
|
'size_mb'
|
|
|
|
This attribute file displays, in count of megabytes, of memory
|
|
that this instance of memory controller manages.
|
|
|
|
|
|
Total Uncorrectable Errors count attribute file:
|
|
|
|
'ue_count'
|
|
|
|
This attribute file displays the total count of uncorrectable
|
|
errors that have occurred on this memory controller. If panic_on_ue
|
|
is set this counter will not have a chance to increment,
|
|
since EDAC will panic the system.
|
|
|
|
|
|
Total UE count that had no information attribute fileY:
|
|
|
|
'ue_noinfo_count'
|
|
|
|
This attribute file displays the number of UEs that
|
|
have occurred have occurred with no informations as to which DIMM
|
|
slot is having errors.
|
|
|
|
|
|
Total Correctable Errors count attribute file:
|
|
|
|
'ce_count'
|
|
|
|
This attribute file displays the total count of correctable
|
|
errors that have occurred on this memory controller. This
|
|
count is very important to examine. CEs provide early
|
|
indications that a DIMM is beginning to fail. This count
|
|
field should be monitored for non-zero values and report
|
|
such information to the system administrator.
|
|
|
|
|
|
Total Correctable Errors count attribute file:
|
|
|
|
'ce_noinfo_count'
|
|
|
|
This attribute file displays the number of CEs that
|
|
have occurred wherewith no informations as to which DIMM slot
|
|
is having errors. Memory is handicapped, but operational,
|
|
yet no information is available to indicate which slot
|
|
the failing memory is in. This count field should be also
|
|
be monitored for non-zero values.
|
|
|
|
Device Symlink:
|
|
|
|
'device'
|
|
|
|
Symlink to the memory controller device
|
|
|
|
|
|
|
|
============================================================================
|
|
'csrowX' DIRECTORIES
|
|
|
|
In the 'csrowX' directories are EDAC control and attribute files for
|
|
this 'X" instance of csrow:
|
|
|
|
|
|
Total Uncorrectable Errors count attribute file:
|
|
|
|
'ue_count'
|
|
|
|
This attribute file displays the total count of uncorrectable
|
|
errors that have occurred on this csrow. If panic_on_ue is set
|
|
this counter will not have a chance to increment, since EDAC
|
|
will panic the system.
|
|
|
|
|
|
Total Correctable Errors count attribute file:
|
|
|
|
'ce_count'
|
|
|
|
This attribute file displays the total count of correctable
|
|
errors that have occurred on this csrow. This
|
|
count is very important to examine. CEs provide early
|
|
indications that a DIMM is beginning to fail. This count
|
|
field should be monitored for non-zero values and report
|
|
such information to the system administrator.
|
|
|
|
|
|
Total memory managed by this csrow attribute file:
|
|
|
|
'size_mb'
|
|
|
|
This attribute file displays, in count of megabytes, of memory
|
|
that this csrow contatins.
|
|
|
|
|
|
Memory Type attribute file:
|
|
|
|
'mem_type'
|
|
|
|
This attribute file will display what type of memory is currently
|
|
on this csrow. Normally, either buffered or unbuffered memory.
|
|
|
|
|
|
EDAC Mode of operation attribute file:
|
|
|
|
'edac_mode'
|
|
|
|
This attribute file will display what type of Error detection
|
|
and correction is being utilized.
|
|
|
|
|
|
Device type attribute file:
|
|
|
|
'dev_type'
|
|
|
|
This attribute file will display what type of DIMM device is
|
|
being utilized. Example: x4
|
|
|
|
|
|
Channel 0 CE Count attribute file:
|
|
|
|
'ch0_ce_count'
|
|
|
|
This attribute file will display the count of CEs on this
|
|
DIMM located in channel 0.
|
|
|
|
|
|
Channel 0 UE Count attribute file:
|
|
|
|
'ch0_ue_count'
|
|
|
|
This attribute file will display the count of UEs on this
|
|
DIMM located in channel 0.
|
|
|
|
|
|
Channel 0 DIMM Label control file:
|
|
|
|
'ch0_dimm_label'
|
|
|
|
This control file allows this DIMM to have a label assigned
|
|
to it. With this label in the module, when errors occur
|
|
the output can provide the DIMM label in the system log.
|
|
This becomes vital for panic events to isolate the
|
|
cause of the UE event.
|
|
|
|
DIMM Labels must be assigned after booting, with information
|
|
that correctly identifies the physical slot with its
|
|
silk screen label. This information is currently very
|
|
motherboard specific and determination of this information
|
|
must occur in userland at this time.
|
|
|
|
|
|
Channel 1 CE Count attribute file:
|
|
|
|
'ch1_ce_count'
|
|
|
|
This attribute file will display the count of CEs on this
|
|
DIMM located in channel 1.
|
|
|
|
|
|
Channel 1 UE Count attribute file:
|
|
|
|
'ch1_ue_count'
|
|
|
|
This attribute file will display the count of UEs on this
|
|
DIMM located in channel 0.
|
|
|
|
|
|
Channel 1 DIMM Label control file:
|
|
|
|
'ch1_dimm_label'
|
|
|
|
This control file allows this DIMM to have a label assigned
|
|
to it. With this label in the module, when errors occur
|
|
the output can provide the DIMM label in the system log.
|
|
This becomes vital for panic events to isolate the
|
|
cause of the UE event.
|
|
|
|
DIMM Labels must be assigned after booting, with information
|
|
that correctly identifies the physical slot with its
|
|
silk screen label. This information is currently very
|
|
motherboard specific and determination of this information
|
|
must occur in userland at this time.
|
|
|
|
|
|
============================================================================
|
|
SYSTEM LOGGING
|
|
|
|
If logging for UEs and CEs are enabled then system logs will have
|
|
error notices indicating errors that have been detected:
|
|
|
|
MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
|
|
channel 1 "DIMM_B1": amd76x_edac
|
|
|
|
MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
|
|
channel 1 "DIMM_B1": amd76x_edac
|
|
|
|
|
|
The structure of the message is:
|
|
the memory controller (MC0)
|
|
Error type (CE)
|
|
memory page (0x283)
|
|
offset in the page (0xce0)
|
|
the byte granularity (grain 8)
|
|
or resolution of the error
|
|
the error syndrome (0xb741)
|
|
memory row (row 0)
|
|
memory channel (channel 1)
|
|
DIMM label, if set prior (DIMM B1
|
|
and then an optional, driver-specific message that may
|
|
have additional information.
|
|
|
|
Both UEs and CEs with no info will lack all but memory controller,
|
|
error type, a notice of "no info" and then an optional,
|
|
driver-specific error message.
|
|
|
|
|
|
|
|
============================================================================
|
|
PCI Bus Parity Detection
|
|
|
|
|
|
On Header Type 00 devices the primary status is looked at
|
|
for any parity error regardless of whether Parity is enabled on the
|
|
device. (The spec indicates parity is generated in some cases).
|
|
On Header Type 01 bridges, the secondary status register is also
|
|
looked at to see if parity ocurred on the bus on the other side of
|
|
the bridge.
|
|
|
|
|
|
SYSFS CONFIGURATION
|
|
|
|
Under /sys/devices/system/edac/pci are control and attribute files as follows:
|
|
|
|
|
|
Enable/Disable PCI Parity checking control file:
|
|
|
|
'check_pci_parity'
|
|
|
|
|
|
This control file enables or disables the PCI Bus Parity scanning
|
|
operation. Writing a 1 to this file enables the scanning. Writing
|
|
a 0 to this file disables the scanning.
|
|
|
|
Enable:
|
|
echo "1" >/sys/devices/system/edac/pci/check_pci_parity
|
|
|
|
Disable:
|
|
echo "0" >/sys/devices/system/edac/pci/check_pci_parity
|
|
|
|
|
|
|
|
Panic on PCI PARITY Error:
|
|
|
|
'panic_on_pci_parity'
|
|
|
|
|
|
This control files enables or disables panic'ing when a parity
|
|
error has been detected.
|
|
|
|
|
|
module/kernel parameter: panic_on_pci_parity=[0|1]
|
|
|
|
Enable:
|
|
echo "1" >/sys/devices/system/edac/pci/panic_on_pci_parity
|
|
|
|
Disable:
|
|
echo "0" >/sys/devices/system/edac/pci/panic_on_pci_parity
|
|
|
|
|
|
Parity Count:
|
|
|
|
'pci_parity_count'
|
|
|
|
This attribute file will display the number of parity errors that
|
|
have been detected.
|
|
|
|
|
|
|
|
PCI Device Whitelist:
|
|
|
|
'pci_parity_whitelist'
|
|
|
|
This control file allows for an explicit list of PCI devices to be
|
|
scanned for parity errors. Only devices found on this list will
|
|
be examined. The list is a line of hexadecimel VENDOR and DEVICE
|
|
ID tuples:
|
|
|
|
1022:7450,1434:16a6
|
|
|
|
One or more can be inserted, seperated by a comma.
|
|
|
|
To write the above list doing the following as one command line:
|
|
|
|
echo "1022:7450,1434:16a6"
|
|
> /sys/devices/system/edac/pci/pci_parity_whitelist
|
|
|
|
|
|
|
|
To display what the whitelist is, simply 'cat' the same file.
|
|
|
|
|
|
PCI Device Blacklist:
|
|
|
|
'pci_parity_blacklist'
|
|
|
|
This control file allows for a list of PCI devices to be
|
|
skipped for scanning.
|
|
The list is a line of hexadecimel VENDOR and DEVICE ID tuples:
|
|
|
|
1022:7450,1434:16a6
|
|
|
|
One or more can be inserted, seperated by a comma.
|
|
|
|
To write the above list doing the following as one command line:
|
|
|
|
echo "1022:7450,1434:16a6"
|
|
> /sys/devices/system/edac/pci/pci_parity_blacklist
|
|
|
|
|
|
To display what the whitelist current contatins,
|
|
simply 'cat' the same file.
|
|
|
|
=======================================================================
|
|
|
|
PCI Vendor and Devices IDs can be obtained with the lspci command. Using
|
|
the -n option lspci will display the vendor and device IDs. The system
|
|
adminstrator will have to determine which devices should be scanned or
|
|
skipped.
|
|
|
|
|
|
|
|
The two lists (white and black) are prioritized. blacklist is the lower
|
|
priority and will NOT be utilized when a whitelist has been set.
|
|
Turn OFF a whitelist by an empty echo command:
|
|
|
|
echo > /sys/devices/system/edac/pci/pci_parity_whitelist
|
|
|
|
and any previous blacklist will be utililzed.
|
|
|