11/21/14

 

8900.1 CHG 368

Volume 10  AIR TRANSPORTATION OVERSIGHT SYSTEM

CHAPTER 2  PROCEDURES FOR DESIGN AND PERFORMANCE ASSESSMENT

Section 10  Performance Assessment and Analysis

Title: Change Bar - Description: Indicates new/changed information. Title: Change Bar - Description: Indicates new/changed information. Indicates new/changed information.

Figure 10-54.  Module 7: Performance Assessment

Figure 10-54. Module 7: Performance Assessment

10-298    INTRODUCTION. The analysis and assessment process module for Performance Assessment (PA) determines if the air carrier’s or applicant’s system performs as intended by regulations in such a way that it controls environmental hazards. The process uses data collected by Certificate Management Team (CMT) or Certification Project Team (CPT) members. The principal inspector (PI) or certification project manager (CPM) may use data from other sources to make this bottom-line PA.

10-299    ANALYZE ELEMENT PERFORMANCE INSPECTION (EPI) DATA BY ELEMENT (see flowchart process step 7.1). After the collected EPI data undergoes a data quality review, the PI or CPM may request assistance from the Operations Research Analyst (ORA). The PI or CPM determines if the air carrier is following its system and meeting the established performance measures. This analysis involves looking at responses to EPI questions for that element, including “No” responses and explanations, “Yes” responses and comments, responses by question category and drop-down menu subjects, questions responded to as “Not Applicable” or “Not Observable,” and text entered in the “Inspector Action Taken” box. Before analyzing or assessing element performance, it is essential to understand the meaning of the various possible responses to each element’s EPI questions.

Title: Change Bar - Description: Indicates new/changed information.

A.    “Yes” Response. A “Yes” response means that the aviation safety inspector (ASI) observed conditions at the air carrier or applicant, or in its documentation, that meets the criteria specified by the Data Collection Tool (DCT) question.

Title: Change Bar - Description: Indicates new/changed information.

B.    “No” Response. A “No” response means the ASI observed a condition at the air carrier or applicant, or in its documentation, that does not meet the criteria specified by the DCT question.

Title: Change Bar - Description: Indicates new/changed information.

NOTE:  Drop-down menus are available when an ASI responds “No” to DCT questions. Select one or more options to narrow the suspected cause for the identified deficiency.

C.    “Not Applicable” Response. A “Not Applicable” answer means that the DCT question does not apply to the air carrier under evaluation because of the type of operation, type of aircraft, area of operation, etc.

Title: Change Bar - Description: Indicates new/changed information.

D.     “Not Observable” Response. A “Not Observable” answer means that while conducting the entire EPI, the ASI was not able to observe conditions required to answer the DCT question and planned no further data collection activities.

Title: Change Bar - Description: Indicates new/changed information.

10-300    ASSESS ASSESSMENT DATA PACKAGE BY ELEMENT (CONDUCT RISK ASSESSMENT MEETING, IF REQUIRED) (see flowchart process step 7.2). The assessment data package provides both current and historical data for the element.

A.    Consolidate and Summarize Relevant Data. This detailed analysis requires the use of the Air Transportation Oversight System (ATOS) data analysis package that consolidates and summarizes all other relevant data needed to properly analyze and assess that element. By methodically assessing the data, the PI or CPM can make a bottom-line assessment of system performance.

1)    When conducting the analysis and assessment of the information contained within the ATOS data analysis package, it is beneficial to compare these findings to any pertinent information gathered while performing the EPI analysis.
2)    For existing air carriers who desire to add or change a program, historical EPI data also helps provide a useful source of information for consideration before assessing system design for the added or changed program.
3)    For new applicants, no historical EPI data may exist to aid the CPT in assessing system performance.

B.    Additional Reports. There are various ways to construct other reports that ORAs or Safety Performance Analysis System (SPAS) provide to include the data necessary to complete the analysis and assessment process more effectively. The PI or CPM may request that the ORA generate additional analyses and reports to allow for adequate assessment of the element.

Title: Change Bar - Description: Indicates new/changed information.

C.    Conduct Risk Assessment Meeting. The PI or CPM may convene a meeting with other CMT or CPT members, as required, to review the performance data and to assist the PI or CPM in making the bottom-line assessment of the system performance. This may be held in conjunction with the design risk assessment meeting. The PI or CPM can use the results of the performance risk assessment meeting in decisionmaking, action planning, completion of the next Air Carrier Assessment Tool (ACAT), and evaluating air carrier or applicant actions.

10-301    10-301 AFFIRMATION OF ELEMENT PERFORMANCE? (see flowchart process step 7.3). After assessing the ATOS data analysis package with input from other CMT or CPT members, the PI or CPM determines whether the air carrier system performance for that element is affirmed. The PI or CPM considers the data collected that are specific to the element as well as other applicable information from the ATOS data analysis package and design or performance risk assessment meeting results.

10-302    DOCUMENT AFFIRMED PERFORMANCE ASSESSMENT (see flowchart process step 7.4). Once the PI or CPM completes the bottom-line assessment, he or she documents the decision to affirm performance. (See Table 10-4, Bottom-line Performance Assessment Categories.) The PI or CPM includes the rationale for bottom‑line assessments categories 2 through 4 (green and yellow). No rationale is required for bottom-line assessment category 1 (green) for “No issues observed.”

10-303    10-303 DOCUMENT PERFORMANCE NOT AFFIRMED (see flowchart process step 7.5). In some cases, the performance data indicates that the air carrier’s system is not performing as intended. If the PI or CPM determines that the applicant’s or air carrier’s system performance is not affirmed, the PI or CPM documents the decision. The PI or CPM includes the rationale for the decision and notes any issues or concerns.

Title: Change Bar - Description: Indicates new/changed information.

Table 10-4.  Bottom-line Performance Assessment Categories

1

Performance Affirmed

No issues observed

No action required

2

Performance Affirmed

Minor issues observed

No action required

3

Performance Affirmed

Minor issues observed

Action Required

4

Performance Affirmed

Issues of concern observed

Action Required

5

Performance Not Affirmed

Safety and/or regulatory issues observed

Action Required

6

Performance Not Affirmed

Systemic safety and/or regulatory issues observed

System reconfiguration by air carrier or applicant is required

Legend

Green: 1–2

Yellow: 3–5

Red: 6

RESERVED.    Paragraphs 10-304 through 10-318.