3/2/21

 

8900.1 CHG 742

VOLUME 2  AIR OPERATOR AND AIR AGENCY CERTIFICATION AND APPLICATION PROCESS

CHAPTER 3  THE CERTIFICATION PROCESS—TITLE 14 CFR PART 121

Section 4  Safety Assurance System: Phase 4—Performance Assessment

Indicates new/changed information.

2-322    GENERAL. This section details the process for Title 14 of the Code of Federal Regulations (14 CFR) part 121 air carrier certification, Phase 4—Performance Assessment (PA). The Safety Assurance System (SAS) contains “Certification Projects” on the menu, which is the entry point for creating and accessing data and documents related to the certification process. Refer to the appropriate initial certification checklist in the SAS Resource Guide (SRG) for detailed instructions on using the SAS automation during the certification process.

2-323    PHASE 4: PERFORMANCE ASSESSMENT (PA).

A.    Confirm Operational Readiness.

Indicates new/changed information.
1)    Certification Project Manager (CPM). Ensure that the applicant conducts an internal safety assessment of operating systems using company audit procedures and documentation. The applicant submits the results to the CPM.
Indicates new/changed information.
2)    CPM. Ensure that the applicant used their Safety Assurance (SA) process to properly address all concerns identified.

NOTE:  During this phase, the Certification Project Team (CPT) reviews the applicant’s Safety Management System (SMS) for acceptance using SMS Design Demonstration Custom Data Collection Tools (C DCT).

B.    Planning (Module 2).

Indicates new/changed information.
1)    CPM. Update the Comprehensive Assessment Plan (CAP) to include the appropriate Element Performance Data Collection Tools (EP DCT) and SMS Design Demonstration C DCT.
a)    The CPT may use C DCTs instead of EP DCTs to prevent a large number of “not observable” answers.
Indicates new/changed information. Indicates new/changed information.
b)    The CPM should review the SMS C DCTs available in the Flight Standards Information Management System (FSIMS) to determine which ones will be added to the CAP. It is important to collect enough data to be able to support a determination regarding the viability and acceptability of the applicant’s SMS. (See Figure 2-101, Phase 4 Safety Management System Demonstration Guide.) There is no requirement to complete all of the SMS Design Demonstration C DCTs. However, the CAP should include the following:

    Safety Communications Design Demonstration,

    Safety Risk Management (SRM) (Process/Department Owner) Design Demonstration,

    SRM (Organizational) Design Demonstration, and

Indicates new/changed information.

    Continual Improvement Process (CIP) Design Demonstration.

c)    As scenarios develop and the applicant applies their SMS, the SMS components will come into play and provide an opportunity for the CPT to observe the applicant’s actions and complete the appropriate C DCT.
2)    CPM. Enter principal inspector (PI) instructions for each EP DCT and C DCT.

Figure 2-101.  Phase 4 Safety Management System Demonstration Guide

Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information.

PHASE 4—SMS DESIGN DEMONSTRATION GUIDE

1.   Planning SMS Design Demonstration C DCTs.

To ensure that an applicant has an acceptable SMS, the CPM needs to ensure the applicant adequately performs certain design demonstrations. Below is a description of key SAS SMS Design Demonstration DCTs.

2.   Key SAS SMS Design Demonstration C DCTs.

a.   Safety Policy.

1)   SMS Safety Communications Design Demonstration.

This is a supplemental assessment to any technical process inspection. During these evaluations, the CPT will be able to observe how the applicant communicates safety information and determine if the employees are aware of the company’s safety policies and hazard reporting procedures. By examining the organizational chart, the CPM should ensure that the CPT samples each area. This C DCT must be thought of in a “group perspective.” That means, did the organization do a good job of communicating the safety policy, and are the majority of the employees able to submit hazard reports per the company approved process?

b.   Safety Risk Management (SRM).

1)   SRM (Organizational) Design Demonstration.

2)   SRM (Process/Department Owner) Design Demonstration.

Ensure that all the departments on the organizational chart are included in these assessments. Identify all process owners (those who are developing or revising guidance documents) for each department and make sure that they have demonstrated their ability to assess risk according to the company’s SRM process. Much of this work is accomplished during the natural course of the Design Assessment (DA) phase. However, as design failures are identified in Phase 4, a process owner will be demonstrating their ability to evaluate risk using the company’s SRM process. The CPT should document the process owner’s ability to assess risk using the SAS SMS Design Demonstration C DCT. The CPM will need to identify those departments that might have made it through the DA phase without having to perform a manual revision, and only test process owners from those departments. The CPT should be highly confident in the applicant’s ability to use the company’s SRM process to analyze risk whenever design changes occur. Design change includes any documents employees use to perform their technical process work (not limited to approved/accepted manuals). All employee guidance documents in use must have some type of version control for the applicant to do SRM appropriately.

c.   Safety Assurance (SA).

1)   CIP Design Demonstration.

This is the corrective action process C DCT. Departments will demonstrate this capability if they failed any of the proving test scenarios or tabletops. Normally, in the course of the certification process, discrepancies will be observed and documented. The CPM should ensure that those areas missed during the normal course of the certification process are tested either through scenarios or tabletops.

2)   Accountable Executive Review Design Demonstration.

The Accountable Executive review is built into the SMS process toward the end of the certification process after proving tests. This demonstrates the applicant’s ability to analyze and assess data, and to take action on that information. It demonstrates the Accountable Executive’s oversight of safety. During the certification, the company should have set objectives and gathered information on their company’s operations. This is the point where the company assesses the operations and builds a plan of action. The CPT should be assured that the applicant will follow their Accountable Executive review process and that the appropriate action will be taken to correct deficiencies. This action will set a baseline date for future assessments, as required by 14 CFR part 5, § 5.73.

C.    Resource Management (Module 3).

Indicates new/changed information.
1)    Certificate Management Office (CMO) Manager. Establish a Certificate Management Team (CMT), including the PIs, to participate in this phase of the certification process as necessary.
2)    CPM. Recommend team members for each PA in the SAS automation.
3)    Certification and Evaluation Program Office (CEPO) Assistant Manager and CMO Frontline Manager (FLM). Review the Schedule of Events (SOE), ensure the availability of resources (i.e., travel funds, passports, and country clearance requests), and assign team members to support the CAP.
Indicates new/changed information.
4)    CEPO Assistant Manager. Review the CAP and provide verbal concurrence.

D.    Aircraft Configuration Evaluation.

1)    CPM. Ensure that the applicant provides notification of aircraft availability at least 10 business‑days before the proposed aircraft evaluation and provides the completed Aircraft Configuration Control Document (ACCD) (or applicant’s equivalent) and the Request List (see Volume 10, Chapter 9, Section 1, Figure 10-9-1A).
Indicates new/changed information.

NOTE:  The CPM must ensure that the applicant provides an aircraft lease or proof of ownership, if not previously submitted in the formal application package.

2)    CPM/CPT. Complete a review of the applicant’s aircraft documents to confirm all required records are present and of sufficient quality to continue the evaluation.
3)    CPT. Evaluate the applicant’s aircraft configuration process using the ACCD and documentation provided by the applicant.
Indicates new/changed information.
4)    CPM. Notify the applicant in writing of discrepancies identified during the aircraft configuration evaluation.
5)    CPM. Ensure that the applicant submits a letter containing the required corrections and updates the SOE, if required.
Indicates new/changed information.
6)    CPM/CPT. Confirm that the applicant corrected the discrepancies prior to beginning demonstrations.
Indicates new/changed information.
7)    CPM. Upload copies of the correspondence to the SAS Document Management.

E.    Demonstration Plans.

Indicates new/changed information.
1)    CPM. Document the completion of each step in the proving test process. (See Volume 3, Chapter 29, Section 6.)
2)    CPM. Ensure that the applicant submits the following plans as appropriate for the proposed operation:

    An emergency evacuation plan,

    A ditching plan, and

    A proving test plan.

3)    CPT. Review and accept the submitted plans. (See Volume 3, Chapters 29 and 30; and Volume 4, Chapter 1, Section 2.)
Indicates new/changed information.
4)    CPM. If applicable, respond to the applicant’s request for a reduction of proving test hours. (See Volume 3, Chapter 29, Section 7.)
Indicates new/changed information.
5)    CPM. Ensure that the applicant submits a request for a Letter of Authorization (LOA) in accordance with 14 CFR part 119, § 119.33(c) at least 10 business-days prior to any aircraft flight demonstrations. (See Volume 3, Chapter 29, Section 4.)
Indicates new/changed information.
6)    CPM. Prior to conducting proving tests, notify the Transportation Security Administration (TSA), Aviation Programs Office at Charters-AirCargo-S@tsa.dhs.gov.
7)    PIs. Confer with the applicant and ensure that the draft operations specifications (OpSpecs) are completed and reflect the proposed operation.
Indicates new/changed information.
8)    CPM. Upload copies of the correspondence and other documentation to the SAS Document Management.

F.    Tabletop Exercises.

1)    CPT. Develop scenarios to assess the applicant’s performance during tabletop exercises. (See Volume 3, Chapter 29, Section 5.)
2)    CPT. Conduct tabletop exercises and discuss the results of each scenario with the applicant.
Indicates new/changed information.
3)    CPM/CPT. If the result of any scenario is unsatisfactory, confirm that the applicant has applied SRM and made corrections to its system design; or has corrected any performance issues before proceeding to proving tests.

G.    Evacuation/Ditching Demonstrations (as required).

Indicates new/changed information.
1)    CPM. Ensure that briefings are conducted in accordance with Volume 3, Chapter 30, Section 3.
2)    Aviation Safety Inspector—Cabin Safety (ASI-CS). Discuss the demonstration plan(s) with the CPT.
Indicates new/changed information.
3)    CPT. Observe the applicant conduct evacuation/ditching demonstrations.
4)    CPT. Document the results of each demonstration on Federal Aviation Administration (FAA) Form 8430-1, Emergency Evacuation Demonstration Report, and complete the assigned EP DCT(s) and/or C DCTs. Use Dynamic Observation Reports (DOR) if it is necessary to document concerns discovered that do not pertain to the assigned element. (See Volume 3, Chapter 30, Section 8.)
Indicates new/changed information.
5)    CPM. Forward copies of each completed FAA Form 8430-1 to the Office of Air Carrier Safety Assurance (ACSA) and Air Transportation Division and save a copy to include in the certification report.

H.    Assess the Applicant’s Facilities.

Indicates new/changed information.
1)    CPT. Assess the applicant’s facilities, as applicable, using appropriate EP DCTs and/or C DCTs. The CPT may accomplish these assessments prior to or during proving tests by observing the applicant conduct their own internal safety assessment.

I.    Prepare for Proving Tests.

1)    CPM. Confirm the Department of Transportation (DOT) issued the Show Cause Order, or in the case of a 14 CFR part 135 applicant applying for part 121 authority, the Final Order.
Indicates new/changed information.
2)    CPM. Verify that the applicant has met all other Phase 4 Gate requirements. (See Figure 2-102, Phases and Gates for Initial Certification.)
Indicates new/changed information.
3)    CMO Manager. Issue a LOA. (See Volume 3, Chapter 29, Section 4 and Section 7; and refer to § 119.33(c).)
4)    CPM. Conduct a predemonstration test meeting with the CPT. (See Volume 3, Chapter 29, Section 4.)
Indicates new/changed information.
5)    CPM. Schedule the CPT for proving tests.
6)    CPT. Develop proving test scenarios that measure the applicant’s operating systems at the element level to confirm that the applicant is following its procedures and producing the intended result. (See Volume 3, Chapter 29, Section 5.)

J.    Proving Test Protocols.

1)    CPT. Meet with the applicant to discuss the proving test protocols and the proposed flight schedule. (See Volume 3, Chapter 29.)
Indicates new/changed information.
2)    CPT. Prior to the proving tests each day, conduct an FAA meeting to review the day’s plan.
Indicates new/changed information.
3)    CPT. Prior to the proving tests each day, conduct a briefing with the applicant to review the day’s plan.
Indicates new/changed information.
4)    CPM. Prior to the planned diversion or other scenario involving air traffic control (ATC), notify the controlling ATC facility(s), per Volume 3, Chapter 29, Section 4.

K.    Conduct Proving Tests.

1)    CPT. Determine if the CPT has collected adequate data and discuss the applicant’s performance.
Indicates new/changed information.
2)    CPT. Conduct a briefing/meeting with the applicant after the proving tests each day.
Indicates new/changed information.
3)    CPT. Document results of each scenario on the appropriate worksheet. (See Volume 3, Chapter 29, Section 5.)
4)    ASI—Aircraft Dispatch (ASI-AD). Administer an initial competency check to the Air Transportation Supervisor (ATS)/Check Dispatcher during the proving tests. (See Volume 3, Chapter 20, Section 1.)
5)    CPM/CPT. When the applicant has met all test objectives and repeatedly demonstrated their ability to conduct line operations in compliance with regulations and safe operating practices, recommend that the proving tests conclude. (See Volume 3, Chapter 29.)
Indicates new/changed information.
6)    CPM/CPT. If the applicant has not completed all test objectives or not demonstrated their ability to conduct line operations in compliance with regulations and/or safe operating practices, evaluate the need for an extension or termination of the proving tests. (See Volume 3, Chapter 29.)
Indicates new/changed information.
7)    CEPO Assistant Manager. Notify ACSA of the normal completion of proving tests or of proving tests terminated due to unsatisfactory performance.
Indicates new/changed information.

NOTE:  Early completion of proving tests should be coordinated and approved according to Volume 3, Chapter 29, Section 7, Table 3-109, Coordination Requirements and Approval Authority for Proving Run Flight Hour Reductions.

8)    CPM. Complete the proving test report. (See Volume 3, Chapter 29, Section 6.)

L.    Data Reporting (Module 4).

Indicates new/changed information.
1)    CPT. Enter EP DCT, C DCT and/or DOR(s) data into SAS in accordance with data quality guidelines (DQG) and submit for data quality review.

M.    Data Review (Module 4).

Indicates new/changed information.
1)    CPM/Data Quality Reviewer (DQR). Ensure that data meets the DQGs and accept for Analysis and Assessment.

N.    Analysis, Assessment, and Action (AAA) (Module 5).

Indicates new/changed information.
1)    CPM/CPT. Conduct a data analysis meeting to analyze the data collected for each element.
2)    CPM. Make an assessment determination and document it in AAA.
3)    CPM. Follow the guidance in Volume 10, Chapter 6 to determine the appropriate course of action for each element.

O.    Phase 4 Review.

Indicates new/changed information.
1)    CPM. Send an email to notify the Safety Analysis and Promotion Division’s Field Support Program Office (FSPO) manager that the certification project is close to completion.
Indicates new/changed information.
2)    CPM. Send an email to notify the TSA Aviation Programs Office at Charters-AirCargo-S@tsa.dhs.gov that the certification project is close to completion. The email should include the CMO manager’s contact information.
3)    CPM. Ensure that all correspondence and supporting documentation are uploaded to the SAS Document Management.
Indicates new/changed information.

NOTE:  Section 119.39(b)(1) states, “An application for a certificate may be denied if ‘[t]he Administrator finds that the applicant is not properly or adequately equipped or is not able to conduct safe operations under this subchapter.’” In addition, the applicant must maintain an active project. The CEPO and ACSA must evaluate an inactive period that exceeds 90 calendar‑days. Inactivity of greater than 90 calendar-days may be cause to terminate the certification process. The FAA may also terminate the process when it is clear that continuing will not result in approval or acceptance (i.e., multiple failures of the applicant’s submissions or performance). If there is a change to the aircraft make/model during the certification process, the project must start over at the beginning of Phase 2 or reviewed for possible termination.

Figure 2-102.  Phases and Gates for Initial Certification

Indicates new/changed information.

Figure 2-102. Phases and Gates for Initial Certification

RESERVED. Paragraphs 2-324 through 2-329.