5/1/12

 

8900.1 CHG 210

Volume 10  Air Transportation Oversight System

Chapter 2  Procedures for Design and Performance Assessment

Section 10  Performance Assessment Analysis and Assessment

Figure 10-54.    Module 7: Performance Analysis and Assessment

Figure 10-54.    Module 7: Performance Analysis and Assessment

10-298    INTRODUCTION. The analysis and assessment process module for Performance Assessment (PA) determines if the air carrier’s or applicant’s system performs as intended by regulations in such a way that it controls environmental hazards. The process uses data collected by Certificate Management Team (CMT) or Certification Project Team (CPT) members. The principal inspector (PI) or certification project manager (CPM) may use data from other sources to make this bottom-line PA.

10-299    ANALYZE ELEMENT PERFORMANCE INSPECTION (EPI) DATA BY ELEMENT. After the collected EPI data undergoes a data quality review, the PI or CPM may request assistance from the Operations Research Analyst (ORA). The PI or CPM determines if the air carrier is following its system and meeting the established performance measures. This analysis involves looking at responses to EPI questions for that element, including “No” responses and explanations, “Yes” responses and comments, responses by question category and drop‑down menu subjects, questions responded to as “Not Applicable” or “Not Observed,” and text entered in the “Inspector Action Taken” box. Before analyzing or assessing element performance, it is essential to understand the meaning of the various possible responses to each element’s EPI questions. (See flowchart process step 7.1.)

A.    “Yes” Response. A “Yes” response means that the inspector observed conditions at the air carrier or applicant, or in its documentation, that meets the criteria specified by the Data Collection Tool (DCT) question.

B.    “No” Response. A “No” response means the inspector observed a condition at the air carrier or applicant, or in its documentation, that does not meet the criteria specified by the DCT question.

NOTE:  Drop-down menus are available when an inspector responds “No” to DCT questions. Select one or more options to narrow the suspected cause for the identified deficiency.

C.    “Not Applicable” Response. A “Not Applicable” answer means that the DCT question does not apply to the air carrier under evaluation because of the type of operation, type of aircraft, area of operation, etc.

D.    “Not Observed” Response. A “Not Observed” answer means that while conducting the entire EPI, the inspector was not able to observe conditions required to answer the DCT question and planned no further data collection activities.

10-300    ASSESS AIR TRANSPORTATION OVERSIGHT SYSTEM (ATOS) ANALYSIS DATA BY ELEMENT (CONDUCT RISK ASSESSMENT IF REQUIRED). Once the EPI data for the current PA have been analyzed by element, an analysis of these data in relation to other historical EPI and other data available for that element is conducted. (See flowchart process step 7.2.)

A.    Consolidate and Summarize Relevant Data. This detailed analysis requires the use of the ATOS data analysis package that consolidates and summarizes all other relevant data needed to properly analyze and assess that element. By methodically assessing the data, the PI or CPM can make a bottom-line assessment of system performance.

1)    When conducting the analysis and assessment of the information contained within the ATOS data analysis package, it is beneficial to compare these findings to any pertinent information gathered while performing the EPI analysis.
2)    For existing air carriers who desire to add or change a program, historical EPI data also helps provide a useful source of information for consideration before assessing system design for the added or changed program.
3)    For new applicants, no historical EPI data may exist to aid the CPT in assessing system performance.

B.    Additional Reports. There are various ways to construct other reports that ORAs or Safety Performance Analysis System (SPAS) provide to include the data necessary to complete the analysis and assessment process more effectively. The PI or CPM may request that the ORA generate additional analyses and reports to allow for adequate assessment of the element.

C.    PA Meeting. The PI or CPM may convene a performance risk assessment meeting with other CMT or CPT members, as required, to review the performance data and to aid in making the bottom-line assessment of the system performance. This may be held in conjunction with the design risk assessment meeting. The PI or CPM can use the results of the performance risk assessment meeting in decisionmaking, action planning, completion of the next Air Carrier Assessment Tool (ACAT), and evaluating air carrier or applicant actions.

10-301    AFFIRMATION OF ELEMENT PERFORMANCE. After assessing the ATOS data analysis package with input from other CMT or CPT members, the PI or CPM determines whether the air carrier system performance for that element is affirmed. The PI or CPM considers the data collected that are specific to the element as well as other applicable information from the ATOS data analysis package and design or performance risk assessment meeting results (See flowchart process step 7.3).

Indicates new/changed information.

10-302    DOCUMENT AFFIRMED PA. Once the PI or CPM completes the bottom-line assessment, he or she documents the decision to affirm performance. (See Table 10-4, Bottom-line PA Categories.) The PI or CPM includes the rationale for bottom-line assessments categories 2 through 4 (green and yellow). No rationale is required for bottom-line assessment category 1 (green) for “No issues observed.” (See flowchart process step 7.4.)

10-303    DOCUMENT PERFORMANCE NOT AFFIRMED. In some cases, the performance data indicates that the air carrier’s system is not performing as intended. If the PI or CPM determines that the applicant’s or air carrier’s system performance is not affirmed, the PI or CPM documents the decision. The PI or CPM includes the rationale for the decision and notes any issues or concerns. (See flowchart process step 7.5.)

Table 10-4.    Bottom-line PA Categories

1

Performance Affirmed

No issues observed

No action required

2

Performance Affirmed

Minor issues observed

No action required

3

Performance Affirmed

Minor issues observed

Action Required

4

Performance Affirmed

Issues of concern observed

Action Required

5

Performance Not Affirmed

Safety and/or regulatory issues observed

Action Required

6

Performance Not Affirmed

Systemic safety and/or regulatory issues observed

System reconfiguration by air carrier or applicant is required

Legend

Green: 1–2

Yellow: 3–5

Red: 6

RESERVED. Paragraphs 10-304 through 10-318.