9/18/20

 

8900.1 CHG 722

VOLUME 10  SAFETY ASSURANCE SYSTEM POLICY AND PROCEDURES

CHAPTER 1  GENERAL

Indicates new/changed information.

Section 6  Safety Assurance System: Risk Profile Overview

10-1-6-1    GENERAL. This section explains the policy, structure, roles, and responsibilities for the Risk Profile used for Risk-Based Decision Making (RBDM) in the Safety Assurance System (SAS).

A.    Purpose. The Risk Profile in SAS provides a Certificate Holder Index (CHI) and an Assessment Priority Index (API) for use in the planning module to prioritize work in SAS, and for RBDM throughout the Federal Aviation Administration (FAA).

B.    Scope. This section applies to SAS users that have transitioned to the Risk Profile. Unless otherwise specified, instructions and information directed at the principal inspector (PI) also refer to the Training Center Program Manager (TCPM) and/or certification project manager (CPM).

10-1-6-3    RESERVED.

10-1-6-5    BACKGROUND. The Risk Profile supports risk-based decisions concerning priorities for surveillance and certificate management. The PI uses the API to develop a risk-based, data-supported Comprehensive Assessment Plan (CAP). The CHI for each certificate holder in SAS is usable across Title 14 of the Code of Federal Regulations (14 CFR) parts and peer groups to aid RBDM at the office, division, and national levels. The CHI and API do not quantify risk and are not a final measure of a certificate holder’s ability to operate safely, or a direct comparison of certificate holders. The Risk Profile evaluates certificate holder and FAA performance using risk components described in this section and provides numerical values that support RBDM for prioritizing and resourcing work assignments in SAS.

Table 10-1-6A.  Terms and Definitions

Term

Definition

Assessment Priority Index (API)

A numerical value calculated for a system or subsystem that, along with the due date, determines the initial priority/resource order for a System or Subsystem Performance Assessment (SPA), Element Performance Assessment (EPA), Element Design Assessment (EDA), or Custom Data Collection Tool (C DCT) in a principal inspector’s (PI) Comprehensive Assessment Plan (CAP).

Certificate Holder Index (CHI)

A numerical value calculated for each certificate holder used to support Risk‑Based Decision Making (RBDM) for the prioritization of work and the assignment of resources.

Certificate Health

A component of the API with a numerical value equal to one half of the CHI.

CHI Grouping (Quintile)

A numerical value calculated by dividing selected certificate holders in an office, nationally, or by 14 CFR part into five equal groups determined by their CHI.

These groupings are ranked according to their priority for oversight.

PI Risk

A value selected by the PI that reflects their assessment of a risk indicator in the Certificate Holder Assessment Tool (CHAT).

Risk Components

The five principal elements of the Risk Profile. The combined values of these components make up the CHI.

Risk Factors

Specific aspects of the risk components that contain numerical values that can be selected by the PI or are calculated by the automation. The combined values are processed by the automation to make up the risk component values. When a risk component’s value is above zero, it is considered a risk indicator.

Risk Indicators

Risk factors in the CHAT, which have a value above zero.

Risk Indicator Considerations

Summarize the reason(s) a PI risk factor has been identified as a risk indicator.

Risk Profile

Contains information about the certificate holder that pertains to risk and is used to prioritize and resource work.

Risk Profile Assessment Tool (RPAT)

A tool for analyzing the CHI and API in order to assist with RBDM for the prioritization of work in the Safety Assurance System (SAS).

10-1-6-7    THE RISK PROFILE. The Risk Profile has five risk components. These components include several factors (see Figure 10-1-6A, Risk Profile Components and Factors) that have measureable attributes, such as assessment determinations, the number of incidents or accidents, or the extent of growth or downsizing.

A.    Risk Components.

1)    Safety Performance History. This component is comprised of events that occur due to full or partial failure of the certificate holder’s system(s). Examples of these events include fatal and non-fatal accidents, incidents, occurrences, voluntary disclosures, enforcement actions, and Department of Defense (DOD) audit results.
2)    Observed Risk. This component derives its value from Analysis, Assessment, and Action (AAA) results, and the PI Risk value for the risk indicator “Other Observations” in the Certificate Holder Assessment Tool (CHAT).
3)    Organizational and Operational Factors. This component derives its value from factors related to certificate holder operations, management personnel, and organizational structure that can be managed to improve system stability and safety.
4)    Uncertainty. This component describes the risk that occurs when oversight has not taken place. Examples include new entrant status, lack of available resources, or assessments that have been moved outside of the baseline interval.
5)    Flight Exposure. This component helps prioritize certificates with similar CHIs that have differing scales of operations and is determined by the automation using information from the Certificate Holder Operating Profile (CHOP), operations specifications (OpSpecs), and the enhanced Vital Information Database (eVID). The Flight Exposure component uses the factor “Average Seating Capacity” or “Air Carrier Authorization.” See Table 10-1-6B, Automation Risk Factors, for more information about “Flight Exposure.”

B.    Risk Factors. Risk factors contribute to the overall value for each risk component and are weighted by the automation in accordance with its potential impact on safety. For example, the value for the “Fatal Accidents” risk factor has a higher weighting than the value for “Non-Fatal Accidents.”

NOTE:  See the CHI and API Scorecards in the RPAT for more information regarding risk factor scoring.

Figure 10-1-6A.  Risk Profile Components and Factors

Figure 10-1-6A. Risk Profile Components and Factors

C.    Risk Indicator Values. There are two ways the value of a risk indicator are determined:

1)    SAS Automation. Table 10-1-6B contains the risk factors calculated by the automation. These values use information from the CHOP, assessment determinations, or other FAA databases to calculate the values. These factors are not adjustable by the PI.

Table 10-1-6B.  Automation Risk Factors

Risk Component

Risk Factor

Notes

Safety Performance History

Enforcement Actions After Compliance Program (FY16)

Enforcement actions were completed at a higher rate prior to the introduction of the Compliance Program. As a result, the contribution of this risk factor is calculated differently than enforcement actions after the introduction of the Compliance Program.

Safety Performance History

Enforcement Actions Before Compliance Program (FY16)

Enforcement actions were completed at a reduced rate following the introduction of the Compliance Program. As a result, the contribution of this risk factor is calculated differently than enforcement actions prior to the introduction of the Compliance Program.

Safety Performance History

Fatal Accidents

 

Safety Performance History

Incidents

 

Safety Performance History

Non-Fatal Accidents

 

Safety Performance History

Occurrences

 

Safety Performance History

Pilot Deviations

 

Observed Risk

Historical System or Subsystem Performance Assessment (SPA) Results

The previous four SPA Assessment Determinations going back a maximum of five years.

Observed Risk

Most Recent SPAs

The assessment determination for the last completed SPAs.

Observed Risk

Other Assessment Results

The assessment determinations of Element Design Assessments (EDA), Element Performance Assessments (EPA) and Custom Data Collection Tools (C DCT).

Uncertainty

Deferred Surveillance

Assessments that are overdue or scheduled outside of the baseline interval.

Uncertainty

FAA Principal Inspector Turnover

This includes frequent changes in the assignment of PIs.

Uncertainty

New Entrant

A lack of assessment history, safety performance history as well as potential inexperience of the certificate holder.

Uncertainty

Resources Not Available (RNA) for DCTs

 

Flight Exposure
(14 CFR parts 121, 135, 141, 142*, and 147*)

Average Passenger Capacity

The average number of seats per aircraft within the certificate holder’s fleet.

*Defaults to lowest value as passenger-carrying capacity is zero.

Flight Exposure
(14 CFR part 145)

Air Carrier Authorization

Air Carrier Authorization or
No Air Carrier Authorization.

2)    PI Risk. PI risk is managed using the risk indicators found in the CHAT and require the PI to select a PI Risk value when, in their opinion, the presence of the risk indicator could have an adverse impact on the certificate holder’s systems’ ability to achieve the desired results. For a complete description of these risk indicators see Volume 10, Chapter 3, Section 2, Table 10-3-2G, PI Risk Indicators.

NOTE:  The Safety Analysis and Promotion Division will enhance the accuracy of the Risk Profile over time through periodic reviews of the risk model. When this occurs, the CHI, API, and the priority/resource order of assessments on the CAP will change, and a change indicator will appear in the RPAT. The CHI and API Scorecards are available for a detailed review of the calculations for each risk factor.

10-1-6-9    RISK PROFILE INTERFACES. The PI provides input to the Risk Profile using the following components of SAS:

A.    CHOP. The automation uses information from the CHOP to calculate the values for factors under the “Flight Exposure” risk component.

NOTE:  For a newly certificated entity, a “New Entrant” risk factor value is calculated and added to the CHI. The weight of this risk factor value will decrease each year for a period of 5 years.

B.    CHAT. The PI uses the CHAT to update the Risk Profile when changes to the certificate holder’s operations occur, or when information becomes available that could change the certificate holder’s ability to manage risk. The PI risk indicators are used to select the levels of concern for the associated risk factors. The PI will review and adjust the risk indicators quarterly for 14 CFR part 121 certificates and annually for parts 135, 141, 142, 145, and 147, and any time a change occurs to the risk associated with one or more indicators. If a PI identifies the presence of a risk indicator, the PI will select a PI Risk value based on the likelihood that the presence of the risk indicator could cause a certificate holder’s system(s) to fail, and the safety impact of the failure(s). Additionally, the PI will document the consideration(s) that indicate the presence of the risk indicator(s). The PI Risk value selected will adjust the value of the associated risk factor. The selected PI Risk value will display in the “PI Risk” column of the CHAT. The PI may use Volume 10, Chapter 3, Section 2, Table 10-3-2A, PI Risk Decision Matrix, to assist in determining a PI Risk value for PI risk indicators.

NOTE:  There are two CHATs: one for Operations and one for Airworthiness. When the Operations and Airworthiness CHATs have different values for a risk indicator, the CHI is calculated using the CHAT with the higher overall risk indicator value.

C.    AAA. Assessment determination results are used in the CHI and the API.

1)    CHI. Current and previous assessment determination results from System or Subsystem Performance Assessments (SPA), Element Performance Assessments (EPA), Custom Data Collection Tools (C DCT), and Element Design Assessments (EDA) are used to determine the value for the “Observed” risk component of the CHI.
2)    API. The API uses current and previous assessment determination results from SPAs, EPAs, EDAs, and C DCTs to determine the value of the system performance component of the API.

D.    RPAT. The RPAT is located in Module 2, Planning, and is an analysis tool for all information related to the Risk Profile. The RPAT uses a series of visual reports to help the PI and management understand the API and CHI values associated with a certificate holder. These include:

1)    CHI List View. Provides the Risk Profile user with a means to compare multiple CHIs across peer groups and 14 CFR parts.
2)    CHI Details View. A Tableau report that breaks down the CHI by component and factor to assist the PI in understanding how they contribute to the CHI. This report also provides an indicator when the risk model has changed.
3)    CHI Scorecard. Provides a detailed breakdown of the CHI value at the component and factor levels and provides a list of the specific events that contribute to the “Safety Performance History” risk factors.
4)    API List View. Provides the Risk Profile user with a means to compare the APIs of all assessments.
5)    API Details View. Displays a breakdown of the API by component. It also includes historical API values and trend information.
6)    API Scorecard. Provides a detailed breakdown of the factors that contribute to the API for each assessment.

E.    CAP. Assessments are initially prioritized on the CAP by due date, API, and other criteria. The PI can adjust the resource/priority order of the assessments based on RBDM, the need to balance the CAP, or to assist in maximizing efficiency (i.e., bundling). The CAP contains the following views:

1)    Calendar View. Displays the assessments in a graphic form, by quarter, which shows their resourcing status, API (including an API change indicator), resource/priority order, etc.
2)    Assessment List View. Displays the same information as the calendar view but in tabular form.

10-1-6-11    RISK PROFILE OUTPUTS. The two primary outputs of the Risk Profile are the CHI for each certificate holder and the API for individual SPAs.

A.    Certificate Holder Index (CHI). The CHI is used to support RBDM for the prioritization of work and the assignment of resources.

NOTE:  The CHI is not a reflection of FAA oversight. The CHI is a tool to assist the PI in achieving the best possible risk-based decisions.

B.    Assessment Priority Index (API). Each SPA, EPA, EDA or C DCT on the CAP will receive the API for the associated system or subsystem. The API is calculated by the automation to assist the PI in determining the priority order of assessments.

1)    API Components and Factors. (See Figure 10-1-6B.) The API for a system or subsystem is calculated using the following factors:
a)    Certificate health,
b)    System or subsystem performance,
c)    Deferred surveillance,
d)    Master List of Functions (MLF) associated risk indicators, and
e)    Criticality of the system or subsystem.
2)    PI C DCTs. Use the highest API of the associated system or subsystem from which the questions are derived.
3)    National/Divisional C DCTs. Appear on the bottom of the CAP and will be placed into the appropriate resource/priority order by the PI.

Figure 10-1-6B.  API Components and Factors

Figure 10-1-6B. API Components and Factors

C.    CHI Grouping. The Risk Profile will divide all certificate holders and assessments into five groups of equal size based on their CHI and/or API.

NOTE:  See Volume 10, Chapter 3, Section 2 for Risk Profile procedures.

10-1-6-13 through 10-1-6-29 RESERVED.