1/12/18

 

8900.1 CHG 571

VOLUME 2  AIR OPERATOR AND AIR AGENCY CERTIFICATION AND APPLICATION PROCESS

CHAPTER 3  THE CERTIFICATION PROCESS—TITLE 14 CFR PART 121

Section 4  Safety Assurance System: Phase 4—Performance Assessment

Indicates new/changed information.

2-322    GENERAL. This section details the process for Title 14 of the Code of Federal Regulations (14 CFR) part 121 air carrier certification, Phase 4—Performance Assessment (PA).

2-323    PHASE 4: PERFORMANCE ASSESSMENT (PA).

A.    Confirm Operational Readiness.

1)    Certification Project Manager (CPM)/Certification Team Leader (CTL). Ensure the applicant conducts an internal safety assessment of operating systems using company audit procedures and documentation. The applicant submits the results to the CPM.
Indicates new/changed information.
2)    CPM/CTL. Ensure that the applicant has properly addressed all concerns identified during their internal safety assessment.
3)    CPM/CTL. During this phase, the applicant’s Safety Management System (SMS) will be reviewed for acceptance using SMS Design Demonstration Data Collection Tools (DCT). The Safety Risk Management (SRM) design that was accepted in Phase 3 must be use during this phase.

B.    Planning (Module 2).

1)    CPM/CTL. Update the Comprehensive Assessment Plan (CAP) to include the appropriate Element Performance (EP) DCTs and SMS Design Demonstration Custom DCTs (C DCT).
a)    The Certification Project Team (CPT) may use C DCTs instead of EP DCTs to prevent a large number of “not observable” answers.
Indicates new/changed information.
b)    The CPM and CTL should review the SMS C DCTs available in the Flight Standards Information Management System (FSIMS) to determine which ones are appropriate, and determine if additional C DCTs are necessary. It is important to collect enough data to be able to support a determination regarding the viability and acceptability of the applicant’s SMS. (See Figure 2-101, Phase 4 Safety Management System Demonstration Guide.) There is no requirement to complete all of the Design Demonstration C DCTs. However, the CAP should include the following:

    Safety Communications Design Demonstration;

    SRM (Process/Department Owner) Design Demonstration;

    SRM (Organizational) Design Demonstration; and

    Continuous Improvement Process Design Demonstration.

NOTE:  As scenarios develop and the applicant applies their SRM, these parts of the SMS will come into play and provide an opportunity for the CPT to observe the applicant’s actions and complete the appropriate C DCT.

NOTE:  There should also be an opportunity for the Accountable Executive to conduct a Safety Performance Assessment (PA) (SMS C DCT Accountable Executive Review Design Demonstration) at or near the completion of Phase 4 as required by 14 CFR part 5, § 5.73. This action will set a baseline date for future assessments.

NOTE:  When the SMS is accepted, the certificate-holding district office (CHDO) manager may elect to notify the applicant by letter. The letter is not required, since the actual acceptance of the SMS is indicated by the issuance of an air carrier certificate.

2)    CPM/CTL. Ensure that the instructions for the assigned DCTs contain relevant information and include the name of the team member recommended.

Figure 2-101.  Phase 4 Safety Management System Demonstration Guide

PHASE 4—SMS DESIGN DEMONSTRATION GUIDE

1.  Planning SMS Design Demonstration Custom DCTs.

Understand the organization and the applicant’s SMS processes to assure that the applicant’s SMS processes include the entire aviation enterprise. In a new certification, the organization typically would be simple and flat. Most small companies might have a core process for their SRM and Safety Assurance (SA). To ensure a company has an acceptable SMS, the CPM needs to ensure certain design demonstrations are adequately demonstrated. The CPT can observe SAS Design Demonstration DCTs along with technical inspections such as the company demonstrating the audit processes in Continuing Analysis and Surveillance System (CASS). Below is a discussion of key SAS SMS Design Demonstration DCTs.

2.  Key SAS SMS Design Demonstration C DCTs.

a.  Safety Policy.

1)  SMS-Safety Communications Design Demonstration.

This is a supplemental assessment to any technical process inspection. During these evaluations the inspector will be able to observe how safety information is communicated and determine if the employees are aware of the company’s safety policies and hazard reporting procedures. By examining the organizational chart (Org Chart) the CPM should ensure each area is sampled. This C DCT must be thought of in a “group perspective.” That means, did the organization do a good job of communicating the Safety Policy, and are the majority of the employees able to submit hazard reports per the company approved process?

 

b.  Safety Risk Management.

1)  SRM (Organizational) Design Demonstration.

2)  SRM (Process/Department Owner) Design Demonstration.

Ensure that all the departments on the Org Chart are included in these assessments. Find all process owners (those who are developing or revising guidance documents) for each department on the Org Chart and make sure they have demonstrated their ability to assess risk according to the company SRM process. However, much of this work is already done during the natural course of the certification process. As design failures are identified in the design demonstration phase, a process owner will be demonstrating their ability to evaluate risk using the company’s SRM process. The CPT should document the process owner’s ability to assess risk using the SAS SMS Design Demonstration C DCT. The CPM will need to identify those departments that might have made it through the design demonstration phase without having to perform a manual redesign and test just those process owners from those departments. The CPT should be highly confident in the applicant’s ability use the company’s SRM process to analyze risk whenever design changes occur. Design change includes any documents being used by employees to perform their technical process work (not limited to approved/accepted manuals). All employee guidance documents being used in the system must have some type of versioning control for the applicant to do SRM appropriately.

c.  Safety Assurance.

1)  Continuous Improvement Process Design Demonstration.

This would be the corrective action process C DCT. Departments will demonstrate this capability if they failed any of the proving run scenarios or tabletops. Perform additional corrective action C DCTs on any departmental process owners that may not have demonstrated this capability during the proving runs/tabletops. Normally, in the course of the certification process discrepancies will be observed and documented. The CPM should ensure that those areas missed during normal businesses are tested either through scenarios or tabletops.

2)  Accountable Executive Review Design Demonstration.

Accountable Executive review is built into the process a proving run/tabletop toward the end of the certification process. This demonstrates the company’s ability to analyze, assess data, and to take action on the information. It demonstrates the business manager’s oversight of safety. During the certification the company should have set objectives and gathered information on their company’s operations. This is the point where the company assesses the operations and builds a plan of action. The CPT should be assured that the company will follow their Accountable Executive review process and that the appropriate action will be taken to correct deficiencies.

C.    Resource Management (Module 3).

Indicates new/changed information.
1)    CPM and CTL. Provide input to Frontline Managers (FLM)/assistant managers to help them identify team resources to complete the CAP.
Indicates new/changed information.
2)    CHDO FLMs and Safety Analysis and Promotion Division (AFS-900) Assistant Manager. Review the Schedule of Events (SOE), ensure the availability of resources (e.g., travel funds, passports, and country clearance requests), and assign team members to support the CAP.
3)    CHDO Manager. Review the CAP and provide verbal concurrence.

D.    Aircraft Conformity Evaluation.

Indicates new/changed information.
1)    CPM. Ensure the applicant notifies the CHDO of aircraft availability at least 10 business-days before the proposed aircraft evaluation, and provide the completed Aircraft Configuration Control Document (ACCD) (or applicant’s equivalent) and the Request List (see Volume 10, Chapter 9, Section 1, Figure 10‑9‑1A).
2)    CPM/CTL. Complete a review of the applicant’s conformity submissions to confirm all required records are present and of sufficient quality to continue.
3)    CPT. Evaluate the applicant’s aircraft conformity process using the ACCD and documentation provided by the applicant.
4)    CPM/CTL. Notify the applicant in writing of discrepancies identified during the aircraft conformity evaluation.
Indicates new/changed information.
5)    CPM. Ensure the applicant submits a letter containing the required corrections and updates the SOE, if required.
6)    CPM/CTL. Confirm that the applicant corrected the discrepancies prior to beginning demonstrations.
Indicates new/changed information.
7)    CPM. Post the correspondence in the SAS document management system.

E.    Demonstration Plans.

Indicates new/changed information.
1)    CPM/CTL. Ensure the applicant submits the following plans as appropriate for the proposed operation:

    Emergency evacuation plan;

    Ditching plan; and

    Proving test plan.

2)    CPT. Review and accept the submitted plans. (See Volume 3, Chapter 29; Volume 3, Chapter 30; and Volume 4, Chapter 1, Section 2.)
3)    CPM/CTL. If applicable, respond to the applicant’s request for a reduction of proving test hours. (See Volume 3, Chapter 29, Section 7.)
Indicates new/changed information.
4)    CPM/CTL. Ensure the applicant submits a request for a letter of authorization (LOA) in accordance with 14 CFR part 119, § 119.33(c) at least 10 business-days prior to any aircraft flight demonstrations. (See Volume 3, Chapter 29, Section 4.)
Indicates new/changed information.
5)    CPM. Prior to conducting proving tests, notify the Transportation Security Administration (TSA) Primary Security Inspector (PSI) Office. All TSA PSIs share a common mailbox:
Charters-AirCargo-S@tsa.dhs.gov. It is important to make this notification so the PSI can coordinate the issuance of security sensitive documents for the air carrier’s mandatory security programs.
6)    CPM/CTL. Confer with applicant and ensure the draft operation specifications (OpSpecs) are completed and reflect the proposed operation.
Indicates new/changed information.
7)    CPM/CTL. Issue an LOA. (See Volume 3, Chapter 29, Section 4 and Section 7; and § 119.33(c).)
Indicates new/changed information.
8)    CPM. Update the SAS document management system.

F.    Tabletop Exercises.

Indicates new/changed information.
1)    CPT. Develop scenarios to assess the applicant’s performance during tabletop exercises. (See Volume 3, Chapter 29, Section 5.)
Indicates new/changed information.
2)    CPT. Conduct tabletop exercises and discuss the results of each scenario with the applicant.
3)    CPM/CTL. If the result of any scenario is unsatisfactory, confirm that the applicant has applied SRM and made corrections to its system design; or has corrected any performance issues before proceeding to proving tests.

G.    Evacuation/Ditching Demonstrations (as required).

1)    Aviation Safety Inspector-Cabin Safety (ASI-CS). Discuss the demonstration plan(s) with the CPT.
2)    CPM/CTL/ASI-CS. Brief the applicant on the plan and expectations.
Indicates new/changed information.
3)    CPT. Observe the applicant conduct evacuation/ditching demonstrations.
Indicates new/changed information.
4)    CPT. Document the results of each demonstration on FAA Form 8430-1, Emergency Evacuation Demonstration Report, and complete the assigned EP DCT(s), and/or C DCTs. Use Dynamic Observation Reports (DOR) if it is necessary to document concerns discovered that do not pertain to the assigned element. (See Volume 3, Chapter 30, Section 8.)
Indicates new/changed information.
5)    CPM. Forward each completed Form 8430-1 to the regional Flight Standards division (RFSD) and save a copy to include in the certification report.

H.    Assess the Applicant’s Facilities. CPT: Assess the applicant’s facilities, as applicable, using appropriate EP DCTs and/or C DCTs. These assessments may be accomplished prior to or during proving tests by observing the applicant conduct their own internal safety assessment. (See subparagraph 2-323A.)

I.    Prepare for Proving Tests.

Indicates new/changed information.
1)    CPM. Confirm the Department of Transportation (DOT) issued the Show Cause Order, or in the case of a 14 CFR part 135 applicant applying for part 121 authority, the Final Order.
2)    CPM/CTL. Verify that the applicant has met all other Phase 4 Gate requirements. (See Figure 2-102, Phases and Gates for Initial Certification.)
3)    CPM/CTL. Conduct a pre-demonstration test meeting with the CPT. (See Volume 3, Chapter 29, Section 4.)
4)    CPM/CTL. Schedule the CPT for proving tests.
5)    CPT. Develop proving test scenarios that measure the applicant’s operating systems at the element level to confirm that the applicant is following its procedures and producing the intended result. (See Volume 3, Chapter 29, Section 5.)
Indicates new/changed information.

J.    Proving Test Protocols.

1)    CPT. Meet with the applicant to discuss the proving test protocols and the proposed flight schedule. (See Volume 3, Chapter 29.)
2)    CPT. Prior to each day’s proving tests, conduct an FAA meeting to review the day’s plan.
Indicates new/changed information.
3)    CPT. Prior to each day’s proving tests, conduct a briefing with the applicant to review the day’s plan.
Indicates new/changed information.

K.    Conduct Proving Tests.

1)    CPT. Determine if the CPT has collected adequate data and discuss the applicant’s performance.
2)    CPT. Conduct a briefing/meeting with the applicant after each day’s proving tests.
3)    CPT. Document results of each scenario on the appropriate worksheet. (See Volume 3, Chapter 29, Section 4.)
4)    CPM/CTL. When the applicant has met all test objectives and repeatedly demonstrated their ability to conduct line operations in compliance with regulations and safe operating practices, recommend that the proving tests conclude. (See Volume 3, Chapter 29.)
5)    CPM/CTL. If the applicant has not completed all test objectives or not demonstrated their ability to conduct line operations in compliance with regulations and/or safe operating practices, evaluate the need for an extension or termination of the proving tests. (See Volume 3, Chapter 29.)
6)    CHDO Manager. Notify the RFSD of the normal completion of proving tests or of proving tests terminated due to unsatisfactory performance.
Indicates new/changed information.

NOTE:  Early completion of proving tests requires RFSD approval. (See Volume 3, Chapter 29.)

7)    CTL. Upload completed scenarios worksheets to the SAS document management system.

L.    Data Reporting (Module 4). CPT: Enter EP DCT, C DCT and/or DOR(s) data into SAS in accordance with Data Quality Guidelines (DQG) and submit.

Indicates new/changed information.

M.    Data Review (Module 4). CPM/Data Reviewer (DR): Ensure data meets the DQGs and accept for Analysis and Assessment.

N.    Analysis, Assessment, and Action (AAA) (Module 5).

1)    CPM/CTL. Conduct an Element Performance Assessment (EPA) meeting to analyze data collected by element, with applicable CPT members.
2)    CPM/CTL. Make an assessment determination and document it in the AAA.
3)    CPM/CTL. Follow the guidance in Module 5 to determine the appropriate course of action for each element.

O.    Phase 4 Review.

1)    CTL. Send an email to notify the AFS-900 Field Support Program Office (FSPO) manager that the certification project is close to completion.
Indicates new/changed information.
2)    CPM. Make entries into the SAS document management system.
3)    CTL. Solicit applicant and CPT feedback by interviewing the CHDO and applicant.

NOTE:  Section 119.39(b) and (b)(1) states, “An application for a certificate may be denied if the Administrator finds that the applicant is not properly or adequately equipped or is not able to conduct safe operations under this subchapter.” In addition, the applicant must maintain an active project. The CHDO must evaluate an inactive period that exceeds 90 calendar-days. Inactivity of greater than 90 calendar-days may be cause to terminate the certification process, or the process may be terminated when it is clear that continuing the process will not result in approval or acceptance (i.e., multiple failures of the applicant’s submissions). If there is a change to the aircraft make/model during the certification process, the project must be restarted at the beginning of Phase 2 or reviewed for possible termination.

Figure 2-102.  Phases and Gates for Initial Certification

Figure 2-102. Phases and Gates for Initial Certification

RESERVED. Paragraphs 2-324 through 2-329.