9/13/16

 

8900.1 CHG 479

VOLUME 10  SAFETY ASSURANCE SYSTEM POLICY AND PROCEDURES

CHAPTER 8  CERTIFICATE HOLDER EVALUATION PROCESS

Section 1  Safety Assurance System: Certificate Holder Evaluation Process

10-8-1-1    GENERAL.

Indicates new/changed information.

A.    Purpose. The Certificate Holder Evaluation Process (CHEP) provides the Flight Standards Service (AFS) and the Office of Hazardous Materials Safety (ADG) with standard policies and procedures to evaluate Title 14 of the Code of Federal Regulations (14 CFR) parts 121, 135, and 145 certificate holders. If you are the principal inspector (PI); Frontline Manager (FLM); CHEP aviation safety inspector (ASI); or national, regional, or local AFS manager, this section applies to you. When information is applicable to someone other than the positions listed, the procedures are specific. Evaluations are an extension of Element Performance Assessments (EPA), Element Design Assessments (EDA), and System/Subsystem Performance Assessments (SPA).

B.    Scope. The CHEP is a process used at the national, regional, and office levels.

10-8-1-3    RESERVED.

10-8-1-5    BACKGROUND.

A.    Objectives. The CHEP allows for an indepth look at the certificate holder’s system and has three primary goals:

·    Verify the certificate holder complies with applicable regulations;

·    Evaluate whether the certificate holder is operating at the highest possible degree of safety in the public interest in accordance with Title 49 of the United States Code (49 U.S.C.) § 44702; and

·    Identify hazards and mitigate associated risks.

B.    Certificate Holder’s Participation.

IF

THEN

The evaluation is conducted at the national level by the Flight Standards National Field Office (FSNFO) (AFS-900)

Certificate holder personnel will not be participating members of the National CHEP.

The evaluation is not conducted at the national level by AFS-900

Certificate holder personnel may be participating members at the discretion of the manager responsible for the CHEP.

The certificate holder participates in the evaluation (joint evaluation)

·    The CHEP Team Leader (TL) ensures the certificate holder’s key management officials (as defined in 14 CFR part 119, § 119.65 or part 145, § 145.151) receive a briefing on the CHEP and the provisions of the current edition of Advisory Circular (AC) 00-58, Voluntary Disclosure Reporting Program, before beginning the evaluation.

·    AC 00-58 applies, as written, during the evaluation period. However, in the case of a joint evaluation, the certificate holder may apply AC 00-58 when the discovery of a possible violation is made by either party.

·    Certificate holder management must understand the requirement to use the appropriate Element Design Data Collection Tool (ED DCT) to assess and document the comprehensive fix of any element deficiencies discovered by the evaluation team that involves an apparent violation of FAA regulations.

The certificate holder does not participate in the evaluation, or the FAA decides not to include them (evaluations not conducted jointly)

The provisions and protections contained in AC 00-58 do not apply to apparent violations of FAA regulations discovered by the evaluation team during the specified evaluation period.

10-8-1-7    CHEP.

Figure 10-8-1A.  Certificate Holder Evaluation Process Flowchart

Figure 10-8-1A. Certificate Holder Evaluation Process Flowchart

10-8-1-9    PROCEDURES.

Indicates new/changed information.

A.    Determine the Need for an Evaluation (see flowchart process step 10-8-1-9A). If you are the national, regional, or local AFS manager or ADG manager, then steps 10-8-1-9A through 10-8-1-9F apply to you. The following conditions may indicate the need to conduct an evaluation:

·    Substantial change in certificate holder management;

·    Substantial turnover in personnel or reduction in force (RIF);

·    Labor dispute;

·    Rapid expansion or growth;

·    Merger, takeover, or change in ownership;

·    Enforcement actions;

·    Noncompliant attitude;

·    Accidents/incidents/occurrences;

·    Department of Defense (DOD) reviews;

·    Department of Transportation (DOT)/Office of the Secretary of Transportation (OST) economic authority/insurance requirements;

·    Change in fleet type;

·    Substantial change in outsourcing;

·    Financial distress;

·    Substantial customer or employee complaints;

·    Hotline complaints; and

·    Certificate transfer (refer to § 119.63 regarding the status of operation/dormant certificates).

B.    Identify and Select Certificate Holders for Evaluation (see flowchart process step 10-8-1-9B). There are several ways to select a certificate holder for evaluation.

IF

THEN

At the national level

·    AFS-900 or the Office of the Director (AFS-1) will review a variety of data sources to select a certificate holder for evaluation, including requests from AFS regional and certificate-holding district office (CHDO) managers.

·    Once a certificate holder is selected, the national manager responsible for the evaluation (or designee) will notify the regional division manager and request the region identify the regional coordinator and a CHDO point of contact (POC) for the evaluation.

·    The regional coordinator and CHDO POC are advised of the dates of the in-brief and out-brief and may attend either or both briefings.

At the regional level

·    The AFS regional division manager responsible for the certificate holder’s oversight may identify and select a certificate holder for evaluation.

·    The AFS regional division manager notifies the appropriate CHDO manager of the certificate holder selected for an evaluation.

·    AFS managers within a region may request a region-level evaluation.

Indicates new/changed information.

At the office level

The CHDO/ADG manager responsible for the certificate holder oversight may identify and select the certificate holder for an office-level evaluation.

C.    Determine the Type of Evaluation (see flowchart process step 10-8-1-9C). The CHEP includes two types of evaluations.

IF

THEN

Indicates new/changed information.

Conducting a focused evaluation

·    Select elements/programs using System/Subsystem Performance Data Collection Tools (SP DCT), ED DCTs, Element Performance Data Collection Tools (EP DCT), or other tools as necessary.

·    The following Custom Data Collection Tools (C DCT) are for ADG use only: HM System Design, HM System Performance, ADG‑Training, ADG-Acceptance/Rejection, ADG-Handling/Storage/Loading, and ADG‑COMAT.

·    There is a specified period for the evaluation. However, should the CHEP team determine that the evaluation should continue based on findings, the evaluation period may be extended as necessary.

Conducting a comprehensive evaluation

·    Select the elements necessary to reissue operations specifications (OpSpecs) using SP DCTs, ED DCTs, EP DCTs, and other tools.

·    There is no specified period for the evaluation.

·    Refer to § 119.63 for dormant certificates.

NOTE:  An Operations Research Analyst (ORA) can generate reports upon request of the manager.

Indicates new/changed information.

D.    Decide on Evaluation Team Composition (see flowchart process step 10-8-1-9D). The level and type of evaluation, along with the complexity of the certificate holder, will help determine the composition of the evaluation team. Evaluation teams may consist of national, regional, or local FAA personnel. In some cases, personnel from the certificate holder and other entities may be asked to participate.

TEAM COMPOSITION

ACTIONS

CHEP TL

NOTE: AFS-900, the AFS regional division manager, or the CHDO/ADG manager, depending on the level of evaluation, designates the TL.

·    Responsible for conducting the evaluation in accordance with the policies and procedures.

·    Guides the evaluation team members on a daily basis during the evaluation period.

·    Ensures team members receive a briefing on certificate-holder-specific information applicable to the scope of the evaluation.

·    Decides how the team communicates.

·    Ensures team members are briefed on the CHEP and the contents of the evaluation notification document.

·    Organizes team meeting(s) (face-to-face or by other means) as necessary to review the TL instructions in the Comprehensive Assessment Plan (CAP) and any other relevant information.

CHEP ASIs

No CHEP ASI may be assigned to a CHEP if that person had responsibility for inspecting the operations/ratings of that certificate holder in the 5-year period preceding the date of the evaluation.

·    May consist of AFS-900 personnel, local CHDO/ADG personnel, personnel from another CHDO or ADG office, and/or regional or national specialists, depending upon the level and scope of the evaluation.

·    May include certificate holder personnel as participants (if not a national AFS-900 CHEP) at the discretion of the manager responsible for the CHEP.

PI

·    Not assigned to perform data collection as part of the evaluation. (This prohibition includes other ASIs assigned to the certificate holder being evaluated.)

·    Must be available to provide clarification pertaining to such items as program approvals, authorizations, and exemptions that apply to assigned certificate holder.

Certificate holder personnel

 

Joint evaluations (FAA and certificate holder) include certificate holder personnel as active participants.

·    When invited by the FAA to participate during joint evaluations, have an active role in determining and resolving element evaluation issues.

·    Certificate holder personnel not participating as members of the CHEP team may serve as POCs for the certificate holder on particular elements or areas of expertise.

Indicates new/changed information.

E.    Complete Evaluation Notification Document (see flowchart process step 10-8-1-9E). AFS-900, the regional AFS division manager, or the CHDO/ADG manager, depending on the level of evaluation, must develop and send an evaluation notification document to the CHDO/ADG manager. This document will include the following:

·    Level and type of evaluation;

·    Elements to be evaluated;

Indicates new/changed information.

·    Performance Assessment (PA) or Design Assessment (DA);

·    For ADG, the National/Regional C DCTs;

·    Evaluation period, including projected start and completion dates;

·    Name and contact information of designated TL and assistant manager; and

·    Certificate holder participation (if any).

Indicates new/changed information.

F.    Notify Certificate Holder of Evaluation (see flowchart process step 10-8-1-9F). The CHDO/ADG manager informs the certificate holder of the evaluation and provides a briefing on the process to certificate holder personnel. The certificate holder is briefed by the AFS-900 team for a National CHEP. The notification can be verbal; the CHDO/ADG manager follows up in writing. The CHDO/ADG manager sends the written notification at least 2 weeks before the evaluation begins. The notification contains information such as that found in the evaluation notification document.

Indicates new/changed information.

G.    Revise Comprehensive Assessment Plan (see flowchart process step 10-8-1-9G). The CHEP TL will add the assessments that have been selected for the evaluation to the Airworthiness and Operations CAP. ADG CHEP TLs will populate the CAP with the ADG C DCTs and recommend ADG CHEP ASIs for data collection. This information will be coordinated with the specific principal operations inspector (POI) in advance since the POI will be able to view the ADG assessments on his or her CAP. The CHEP assessments will appear as a triangle at the bottom of the appropriate CAP and do not affect the Certificate Management Team (CMT) CAP. The CMT will adjust their CAP accordingly, taking into account the assessments that are completed by the CHEP Team.

H.    Assign Resources to Complete Evaluation (see flowchart process step 10-8-1-9H).

1)    Prior to the start date of the evaluation period, the AFS-900 FLM (for National CHEPs) or the appropriate Safety Assurance System (SAS) Administrator will add the evaluation team members to the CMT in the SAS automation.
2)    The CHEP TL distributes tasks by system/subsystem, element, individual questions, or some combination to one or more team members, to allow the timely collection of accurate data.
Indicates new/changed information.
3)    The FLM assigns team members to data collection activities to support the evaluation after the team members receive a briefing from the CMT on the certificate holder operations appropriate to the scope of the evaluation. ADG CHEP FLMs assigns the ADG C DCT to the ADG CHEP ASI.

NOTE:  There is no concurrence for assessments supporting a CHEP.

I.    Collect Evaluation Data (see flowchart process step 10-8-1-9I). Evaluation team members perform their assigned activities to collect data in accordance with the CAP and TL instructions.

Indicates new/changed information.

J.    Report Evaluation Data (see flowchart process step 10-8-1-9J). Team members enter their responses into the SAS automation in accordance with the data quality guidelines (DQG) after collecting evaluation data. Critical or time-sensitive information must be reported to the CHEP TL and the PI immediately. Any unsafe condition that would result in an accident, incident, or violation discovered during the evaluation should be handled based on the applicable guidance. Objective documents which detail and support inspection findings may be attached for the PI’s review with the final report. Documents that support findings can include flight logs and maintenance records. Enter data collected by certificate holder personnel if they are part of the evaluation, and indicate the source in the SAS automation. You may also document any identified program strength(s) in the comment box of the applicable SP DCT, ED DCT, EP DCT, or C DCT question, provided it is not proprietary information.

Indicates new/changed information.

NOTE:  ADG personnel are limited to reporting data only in the NR C DCTs and the Dynamic Observation Report (DOR). ADG personnel are not allowed to schedule or use any other DCT such as En Route, random inspection (RI), SP DCT, or EP DCT.

Indicates new/changed information.

K.    Review Evaluation Data by Team Leader (see flowchart process step 10-8-1-9K). The evaluation TL or ADG FLM is typically the Data Quality Reviewer (DQR). In some instances, it may be necessary for the DQR to proxy the data review function to a team member other than the evaluation TL. DQR rights are granted to specific CHEP team members if necessary. If the DQR finds any critical or time-sensitive information during the review, he or she must notify the TL and PI immediately, unless the record indicates notification has already taken place.

L.    Analyze and Assess Evaluation Data by Team Leader (see flowchart process step 10-8-1-9L). After all data are collected, reported, and reviewed, the evaluation team completes the evaluation using the SAS Analysis, Assessment, and Action (AAA) procedures and tools (Module 5). The TL is responsible for completing the Assessment Determination, Module 5 of the AAA, considering input from the team. The PI completes the Action Determination section of the AAA and follows through with any actions required.

Indicates new/changed information.

NOTE:  The ADG FLM and POI collaborate to determine if any action is required in the AAA Module. If action is required, then the POI will select “other” and document the type of action recommended by ADG personnel. ADG will coordinate and complete all actions.

M.    Team Leader Makes Finalized Evaluation Report (see flowchart process step 10-8-1-9M). Evaluation TLs generate an evaluation report when requested by AFS-1, the regional AFS division manager, or the CHDO manager. Upon the request of the FLM, the Regional Manager, or the CHDO Manager, the ORA assigned to the certificate holder should generate a report on the evaluation data collected.

N.    Determine and Implement Followup Actions (see flowchart process step 10-8-1-9N). The PI determines and implements followup actions in response to PAs and DAs completed during the evaluation.

O.    CMT and PIs Brief Certificate Holder and Other Stakeholders (see flowchart process step 10‑8‑1‑9O). The manager responsible for initiating the evaluation determines who briefs the certificate holder or other stakeholders on the results of the evaluation.

NOTE:  If the CHEP identifies an issue with a DCT, policy, or automation, then use the feedback process to report that issue. When a CHEP identifies a possible improvement to a CHDO process, these suggestions can be tracked in the QMS. The CHDO should use their QMS process to improve its processes. SAS will store historical data on SAS evaluations performed, including information on systems, subsystems, and program elements assessed, for each certificate holder.

10-8-1-11 through 10-8-1-29 RESERVED.