VOLUME 2 AIR OPERATOR AND AIR AGENCY CERTIFICATION AND APPLICATION PROCESS
CHAPTER 3 THE CERTIFICATION PROCESSTITLE 14 CFR PART
Section 4 Safety Assurance System: Phase 4Performance Assessment
2-322 GENERAL. This section details the process for Title 14 of the Code of Federal Regulations (14 CFR) part
carrier certification, Phase 4Performance Assessment (PA).
2-323 PHASE 4: PERFORMANCE ASSESSMENT (PA).
A. Confirm Operational Readiness.
1) Certification Project Manager (CPM)/Certification Team Leader (CTL). Ensure the applicant conducts an internal safety assessment of operating
systems using company audit procedures and documentation. The applicant submits the results to the CPM.
2) CPM/CTL. Ensure that the applicant has properly addressed all concerns identified during their internal safety assessment.
3) CPM/CTL. During this phase, the applicant’s Safety Management System (SMS) will be reviewed for acceptance using SMS Design
Demonstration Data Collection Tools (DCT). The Safety Risk Management (SRM) design that was accepted in Phase 3 must be use during this phase.
B. Planning (Module 2).
1) CPM/CTL. Update the Comprehensive Assessment Plan (CAP) to include the appropriate Element Performance (EP) DCTs and SMS Design Demonstration
Custom DCTs (C DCT).
a) The Certification Project Team (CPT) may use C DCTs instead of EP DCTs to prevent a large number of “not observable” answers.
b) The CPM and CTL should review the SMS C DCTs available in the Flight Standards Information Management System (FSIMS) to determine which ones are
appropriate, and determine if additional C DCTs are necessary. It is important to collect enough data to be able to support a determination regarding the viability and
acceptability of the applicant’s SMS. (See Figure 2-101, Phase 4 Safety Management System Demonstration Guide.) There is no requirement to complete all of the Design
Demonstration C DCTs. However, the CAP should include the following:
• Safety Communications Design Demonstration;
• SRM (Process/Department Owner) Design Demonstration;
• SRM (Organizational) Design Demonstration; and
• Continuous Improvement Process Design Demonstration.
NOTE: As scenarios develop and the applicant applies their SRM, these parts of the SMS will come into play and provide an opportunity for the CPT to
observe the applicant’s actions and complete the appropriate C DCT.
NOTE: There should also be an opportunity for the Accountable Executive to conduct a Safety Performance Assessment (PA) (SMS C DCT Accountable
Executive Review Design Demonstration) at or near the completion of Phase 4 as required by 14 CFR part
This action will set a baseline date for future assessments.
NOTE: When the SMS is accepted, the certificate-holding district office (CHDO) manager may elect to notify the applicant by letter. The letter is not
required, since the actual acceptance of the SMS is indicated by the issuance of an air carrier certificate.
2) CPM/CTL. Ensure that the instructions for the assigned DCTs contain relevant information and include the name of the team member recommended.
Figure 2-101. Phase 4 Safety Management System Demonstration Guide
1. Planning SMS Design Demonstration Custom DCTs.
Understand the organization
and the applicant’s SMS processes to assure that the applicant’s SMS processes include the entire aviation enterprise. In a new certification, the organization
typically would be simple and flat. Most small companies might have a core process for their SRM and Safety Assurance (SA). To ensure a company has an acceptable SMS,
the CPM needs to ensure certain design demonstrations are adequately demonstrated. The CPT can observe SAS Design Demonstration DCTs along with technical inspections such
as the company demonstrating the audit processes in Continuing Analysis and Surveillance System (CASS). Below is a discussion of key SAS SMS Design Demonstration DCTs.
2. Key SAS SMS
Design Demonstration C DCTs.
a. Safety Policy.
Communications Design Demonstration.
This is a supplemental assessment
to any technical process inspection. During these evaluations the inspector will be able to observe how safety information is communicated and determine if the employees
are aware of the company’s safety policies and hazard reporting procedures. By examining the organizational chart (Org Chart) the CPM should ensure each area
is sampled. This C DCT must be thought of in a “group perspective.” That means, did the organization do a good job of communicating the Safety Policy, and are
the majority of the employees able to submit hazard reports per the company approved process?
1) SRM (Organizational)
(Process/Department Owner) Design Demonstration.
Ensure that all the departments
on the Org Chart are included in these assessments. Find all process owners (those who are developing or revising guidance documents) for each department on the Org Chart
and make sure they have demonstrated their ability to assess risk according to the company SRM process. However, much of this work is already done during the natural
course of the certification process. As design failures are identified in the design demonstration phase, a process owner will be demonstrating their ability to evaluate
risk using the company’s SRM process. The CPT should document the process owner’s ability to assess risk using the SAS SMS Design Demonstration C DCT. The CPM
will need to identify those departments that might have made it through the design demonstration phase without having to perform a manual redesign and test just those process
owners from those departments. The CPT should be highly confident in the applicant’s ability use the company’s SRM process to analyze risk whenever design changes
occur. Design change includes any documents being used by employees to perform their technical process work (not limited to approved/accepted manuals). All employee guidance
documents being used in the system must have some type of versioning control for the applicant to do SRM appropriately.
c. Safety Assurance.
Improvement Process Design Demonstration.
This would be the corrective
action process C DCT. Departments will demonstrate this capability if they failed any of the proving run scenarios or tabletops. Perform additional corrective action C DCTs
on any departmental process owners that may not have demonstrated this capability during the proving runs/tabletops. Normally, in the course of the certification process
discrepancies will be observed and documented. The CPM should ensure that those areas missed during normal businesses are tested either through scenarios or tabletops.
Executive Review Design Demonstration.
Accountable Executive review is built into the process a proving run/tabletop toward the
end of the certification process. This demonstrates the company’s ability to analyze, assess data, and to take action on the information. It demonstrates the business
manager’s oversight of safety. During the certification the company should have set objectives and gathered information on their company’s operations. This is
the point where the company assesses the operations and builds a plan of action. The CPT should be assured that the company will follow their Accountable Executive review
process and that the appropriate action will be taken to correct deficiencies.
C. Resource Management (Module 3).
1) CPM and CTL. Provide input to Frontline Managers (FLM)/assistant managers to help them identify team resources to complete the CAP.
2) CHDO FLMs and Safety Analysis and Promotion Division (AFS-900) Assistant Manager. Review the Schedule of Events (SOE), ensure the availability
of resources (e.g., travel funds, passports, and country clearance requests), and assign team members to support the CAP.
3) CHDO Manager. Review the CAP and provide verbal concurrence.
D. Aircraft Conformity Evaluation.
1) CPM. Ensure the applicant notifies the CHDO of aircraft availability at least 10 business-days before the proposed aircraft evaluation, and
provide the completed Aircraft Configuration Control Document (ACCD) (or applicant’s equivalent) and the Request List (see
Volume 10, Chapter 9, Section 1,
2) CPM/CTL. Complete a review of the applicant’s conformity submissions to confirm all required records are present and of sufficient
quality to continue.
3) CPT. Evaluate the applicant’s aircraft conformity process using the ACCD and documentation provided by the applicant.
4) CPM/CTL. Notify the applicant in writing of discrepancies identified during the aircraft conformity evaluation.
5) CPM. Ensure the applicant submits a letter containing the required corrections and updates the SOE, if required.
6) CPM/CTL. Confirm that the applicant corrected the discrepancies prior to beginning demonstrations.
7) CPM. Post the correspondence in the SAS document management system.
E. Demonstration Plans.
1) CPM/CTL. Ensure the applicant submits the following plans as appropriate for the proposed operation:
• Emergency evacuation plan;
• Ditching plan; and
• Proving test plan.
2) CPT. Review and accept the submitted plans. (See Volume 3, Chapter 29; Volume 3, Chapter 30; and
Volume 4, Chapter 1, Section 2.)
3) CPM/CTL. If applicable, respond to the applicant’s request for a reduction of proving test hours. (See
Volume 3, Chapter 29, Section 7.)
4) CPM/CTL. Ensure the applicant submits a request for a letter of authorization (LOA) in accordance with 14 CFR part
least 10 business-days prior to any aircraft flight demonstrations. (See
Volume 3, Chapter 29, Section 4.)
5) CPM. Prior to conducting proving tests, notify the Transportation Security Administration (TSA) Primary Security Inspector (PSI) Office. All
TSA PSIs share a common mailbox:
It is important to make this notification so the PSI can coordinate the issuance of security sensitive documents for the air carrier’s mandatory security programs.
6) CPM/CTL. Confer with applicant and ensure the draft operation specifications (OpSpecs) are completed and reflect the proposed operation.
8) CPM. Update the SAS document management system.
F. Tabletop Exercises.
1) CPT. Develop scenarios to assess the applicant’s performance during tabletop exercises.
(See Volume 3, Chapter 29, Section 5.)
2) CPT. Conduct tabletop exercises and discuss the results of each scenario with the applicant.
3) CPM/CTL. If the result of any scenario is unsatisfactory, confirm that the applicant has applied SRM and made corrections to its system
design; or has corrected any performance issues before proceeding to proving tests.
G. Evacuation/Ditching Demonstrations (as required).
1) Aviation Safety Inspector-Cabin Safety (ASI-CS). Discuss the demonstration plan(s) with the CPT.
2) CPM/CTL/ASI-CS. Brief the applicant on the plan and expectations.
3) CPT. Observe the applicant conduct evacuation/ditching demonstrations.
4) CPT. Document the results of each demonstration on FAA Form 8430-1, Emergency Evacuation Demonstration Report, and complete the assigned
EP DCT(s), and/or C DCTs. Use Dynamic Observation Reports (DOR) if it is necessary to document concerns discovered that do not pertain to the assigned element. (See
Volume 3, Chapter 30, Section 8.)
5) CPM. Forward each completed Form 8430-1 to the regional Flight Standards division (RFSD) and save a copy to include in the certification report.
H. Assess the Applicant’s Facilities. CPT: Assess the applicant’s facilities, as applicable, using appropriate EP DCTs and/or C DCTs.
These assessments may be accomplished prior to or during proving tests by observing the applicant conduct their own internal safety assessment. (See subparagraph 2-323A.)
I. Prepare for Proving Tests.
1) CPM. Confirm the Department of Transportation (DOT) issued the Show Cause Order, or in the case of a 14 CFR part
applying for part
the Final Order.
2) CPM/CTL. Verify that the applicant has met all other Phase 4 Gate requirements. (See Figure 2-102, Phases and Gates for Initial Certification.)
3) CPM/CTL. Conduct a pre-demonstration test meeting with the CPT. (See
Volume 3, Chapter 29, Section 4.)
4) CPM/CTL. Schedule the CPT for proving tests.
5) CPT. Develop proving test scenarios that measure the applicant’s operating systems at the element level to confirm that the applicant
is following its procedures and producing the intended result.
(See Volume 3, Chapter 29, Section 5.)
J. Proving Test Protocols.
1) CPT. Meet with the applicant to discuss the proving test protocols and the proposed flight schedule. (See Volume 3, Chapter 29.)
2) CPT. Prior to each day’s proving tests, conduct an FAA meeting to review the day’s plan.
3) CPT. Prior to each day’s proving tests, conduct a briefing with the applicant to review the day’s plan.
K. Conduct Proving Tests.
1) CPT. Determine if the CPT has collected adequate data and discuss the applicant’s performance.
2) CPT. Conduct a briefing/meeting with the applicant after each day’s proving tests.
3) CPT. Document results of each scenario on the appropriate worksheet. (See
Volume 3, Chapter 29, Section 4.)
4) CPM/CTL. When the applicant has met all test objectives and repeatedly demonstrated their ability to conduct line operations in compliance
with regulations and safe operating practices, recommend that the proving tests conclude. (See Volume 3, Chapter 29.)
5) CPM/CTL. If the applicant has not completed all test objectives or not demonstrated their ability to conduct line operations in compliance
with regulations and/or safe operating practices, evaluate the need for an extension or termination of the proving tests. (See Volume 3, Chapter 29.)
6) CHDO Manager. Notify the RFSD of the normal completion of proving tests or of proving tests terminated due to unsatisfactory performance.
NOTE: Early completion of proving tests requires RFSD approval. (See Volume 3, Chapter 29.)
7) CTL. Upload completed scenarios worksheets to the SAS document management system.
L. Data Reporting (Module 4). CPT: Enter EP DCT, C DCT and/or DOR(s) data into SAS in accordance with Data Quality Guidelines (DQG) and submit.
M. Data Review (Module 4). CPM/Data Reviewer (DR): Ensure data meets the DQGs and accept for Analysis and Assessment.
N. Analysis, Assessment, and Action (AAA) (Module 5).
1) CPM/CTL. Conduct an Element Performance Assessment (EPA) meeting to analyze data collected by element, with applicable CPT members.
2) CPM/CTL. Make an assessment determination and document it in the AAA.
3) CPM/CTL. Follow the guidance in Module 5 to determine the appropriate course of action for each element.
O. Phase 4 Review.
1) CTL. Send an email to notify the AFS-900 Field Support Program Office (FSPO) manager that the certification project is close to completion.
2) CPM. Make entries into the SAS document management system.
3) CTL. Solicit applicant and CPT feedback by interviewing the CHDO and applicant.
“An application for a certificate may be denied if the Administrator finds that the applicant is not properly or adequately equipped or is not able to conduct safe
operations under this subchapter.” In addition, the applicant must maintain an active project. The CHDO must evaluate an inactive period that exceeds 90 calendar-days.
Inactivity of greater than 90 calendar-days may be cause to terminate the certification process, or the process may be terminated when it is clear that continuing the process
will not result in approval or acceptance (i.e., multiple failures of the applicant’s submissions). If there is a change to the aircraft make/model during the certification
process, the project must be restarted at the beginning of Phase 2 or reviewed for possible termination.
Figure 2-102. Phases and Gates for Initial Certification
RESERVED. Paragraphs 2-324 through 2-329.