3/13/18

 

8900.1 CHG 581

VOLUME 10  SAFETY ASSURANCE SYSTEM POLICY AND PROCEDURES

CHAPTER 8   CERTIFICATE HOLDER EVALUATION PROCESS

Section 1 Safety Assurance System: Certificate Holder Evaluation Process

10‑8‑1‑1    GENERAL.

Indicates new/changed information.

A.    Purpose. The Certificate Holder Evaluation Process (CHEP) provides the Flight Standards Service with standard policies and procedures to evaluate Title 14 of the Code of Federal Regulations (14 CFR) parts 121, 135, and 145 certificate holders. If you are the principal inspector (PI); Frontline Manager (FLM); CHEP; aviation safety inspector (ASI); or national, regional, or local Flight Standards manager, this section applies to you. When information is applicable to someone other than the positions listed, the procedures are specific. Evaluations are an extension of Element Performance Assessments (EPA), Element Design Assessments (EDA), and System/Subsystem Performance Assessments (SPA).

B.    Scope. The CHEP is a process used at the national, regional, and office levels.

10‑8‑1‑3    RESERVED.

10‑8‑1‑5    BACKGROUND.

A.    Objectives. The CHEP allows for an indepth look at the certificate holder’s system and has three primary goals:

1)    Verify the certificate holder complies with applicable regulations;
2)    Evaluate whether the certificate holder is operating at the highest possible degree of safety in the public interest in accordance with Title 49 of the United States Code (49 U.S.C.) § 44702; and
3)    Identify hazards and mitigate associated risks.
Indicates new/changed information.

B.    Certificate Holder’s Participation.

IF:

THEN:

The evaluation is conducted at the national level by the Safety Analysis and Promotion Division (AFS‑900),

Certificate holder personnel will not be participating members of the National CHEP.

The evaluation is not conducted at the national level by AFS‑900,

Certificate holder personnel may be participating members at the discretion of the manager responsible for the CHEP.

The certificate holder participates in the evaluation (joint evaluation),

    The CHEP Team Leader (TL) ensures the certificate holder’s key management officials (as defined in 14 CFR part 119, § 119.65 or part 145, § 145.151) receive a briefing on the CHEP and the provisions of Advisory Circular (AC) 00‑58, Voluntary Disclosure Reporting Program, before beginning the evaluation.

    AC 00‑58, as written, applies during the evaluation period. However, in the case of a joint evaluation, the certificate holder may apply AC 00‑58 when the discovery of a possible violation is made by either party.

    Certificate holder management must understand the requirement to use the appropriate Element Design Data Collection Tool (ED DCT) to assess and document the comprehensive fix of any element deficiencies discovered by the evaluation team that involves an apparent violation of Federal Aviation Administration (FAA) regulations.

The certificate holder does not participate in the evaluation, or the FAA decides not to include them (evaluations not conducted jointly),

The provisions and protections contained in AC 00‑58 do not apply to apparent violations of FAA regulations discovered by the evaluation team during the specified evaluation period.

10‑8‑1‑7    CHEP.

Figure 10‑8‑1A.  Certificate Holder Evaluation Process Flowchart

Figure 10-8-1A. Certificate Holder Evaluation Process Flowchart

10‑8‑1‑9    PROCEDURES.

Indicates new/changed information.

A.    Determine the Need for an Evaluation (see flowchart process step 10‑8‑1‑9A). If you are the national, regional, or local Flight Standards manager, then steps 10‑8‑1‑9A through 10‑8‑1‑9F apply to you. The following conditions may indicate the need to conduct an evaluation:

    Substantial change in certificate holder management;

    Substantial turnover in personnel or reduction in force (RIF);

    Labor dispute;

    Rapid expansion or growth;

    Merger, takeover, or change in ownership;

    Enforcement actions;

    Noncompliant attitude;

    Accidents/incidents/occurrences;

    Department of Defense (DOD) reviews;

    Department of Transportation (DOT)/Office of the Secretary of Transportation (OST) economic authority/insurance requirements;

    Change in fleet type;

    Substantial change in outsourcing;

    Financial distress;

    Substantial customer or employee complaints;

    Hotline complaints; and

    Certificate transfer (refer to § 119.63 regarding the status of operation and/or dormant certificates).

Indicates new/changed information.

B.    Identify and Select Certificate Holders for Evaluation (see flowchart process step 10‑8‑1‑9B). There are several ways to select a certificate holder for evaluation.

IF:

THEN:

At the national level,

  AFS‑900 or the Office of the Executive Director, Flight Standards Service (AFX‑1) will review a variety of data sources to select a certificate holder for evaluation, including requests from Flight Standards regional and certificate-holding district office (CHDO) managers.

  Once a certificate holder is selected, the national manager responsible for the evaluation (or designee) will notify the regional division manager and request the region identify the Regional Coordinator (RC) and a CHDO point of contact (POC) for the evaluation.

   The RC and CHDO POC are advised of the dates of the in‑brief and out‑brief and may attend either or both briefings.

At the regional level,

   The Flight Standards regional division manager responsible for the certificate holder’s oversight may identify and select a certificate holder for evaluation.

   The Flight Standards regional division manager notifies the appropriate CHDO manager of the certificate holder selected for an evaluation.

   Flight Standards managers within a region may request a regional level evaluation.

At the office level,

The CHDO manager responsible for the certificate holder oversight may identify and select the certificate holder for an office-level evaluation.

Indicates new/changed information. Indicates new/changed information.

C.    Determine the Type of Evaluation (see flowchart process step 10‑8‑1‑9C). The CHEP includes two types of evaluations.

IF:

THEN:

Conducting a focused evaluation,

   Select elements/programs using System/Subsystem Performance Data Collection Tools (SP DCT), ED DCTs, Element Performance Data Collection Tools (EP DCT), or other tools as necessary.

  There is a specified period for the evaluation. However, should the CHEP team determine that the evaluation should continue based on findings, the evaluation period may be extended as necessary.

Conducting a comprehensive evaluation,

  Select the elements necessary to reissue operations specifications (OpSpecs) using SP DCTs, ED DCTs, EP  DCTs, and other tools.

   There is no specified period for the evaluation.

   Refer to § 119.63 for dormant certificates.

NOTE:  An Operations Research Analyst (ORA) can generate reports upon request of the manager.

D.    Decide on Evaluation Team Composition (see flowchart process step 10‑8‑1‑9D). The level and type of evaluation, along with the complexity of the certificate holder, will help determine the composition of the evaluation team. Evaluation teams may consist of national, regional, or local FAA personnel. In some cases, personnel from the certificate holder and other entities may be asked to participate in an evaluation.

TEAM COMPOSITION:

ACTIONS:

Indicates new/changed information.

CHEP TL

NOTE:  AFS‑900, the Flight Standards regional division manager, or the CHDO manager, depending on the level of evaluation, designates the TL.

   Responsible for conducting the evaluation in accordance with the policies and procedures.

   Guides the evaluation team members on a daily basis during the evaluation period.

   Ensures team members receive a briefing on certificate‑holder‑specific information applicable to the scope of the evaluation.

   Decides how the team communicates.

   Ensures team members are briefed on the CHEP and the contents of the evaluation notification document.

   Organizes team meeting(s) (face‑to‑face or by other means) as necessary to review the TL instructions in the Comprehensive Assessment Plan (CAP) and any other relevant information.

Indicates new/changed information.

CHEP ASIs

NOTE:  No CHEP ASI may be assigned to a CHEP if that person had responsibility for inspecting the operations and/or ratings of that certificate holder in the 5‑year period preceding the date of the evaluation.

   May consist of AFS‑900 personnel, local CHDO personnel, personnel from another CHDO, and/or regional or national specialists, depending upon the level and scope of the evaluation.

   May include certificate holder personnel as participants (if not a national AFS‑900 CHEP) at the discretion of the manager responsible for the CHEP.

PI

   Not assigned to perform Data Collection as part of the evaluation. (This prohibition includes other ASIs assigned to the certificate holder being evaluated.)

   Must be available to provide clarification pertaining to such items as program approvals, authorizations, and exemptions that apply to the assigned certificate holder.

Certificate holder personnel

NOTE:  Joint evaluations (FAA and certificate holder) include certificate holder personnel as active participants.

   When invited by the FAA to participate during joint evaluations, have an active role in determining and resolving element evaluation issues.

   Certificate holder personnel not participating as members of the CHEP team may serve as POCs for the certificate holder on particular elements or areas of expertise.

Indicates new/changed information. Indicates new/changed information.

E.    Complete Evaluation Notification Document (see flowchart process step 10‑8‑1‑9E). AFS‑900, the regional Flight Standards division manager, or the CHDO manager, depending on the level of evaluation, must develop and send an evaluation notification document to the CHDO manager. This document will include the following:

    Level and type of evaluation;

    Elements to be evaluated;

    Performance Assessment (PA) or Design Assessment (DA);

    Evaluation period, including projected start and completion dates;

    Name and contact information of designated TL and assistant manager; and

    Certificate holder participation (if any).

Indicates new/changed information.

F.    Notify Certificate Holder of Evaluation (see flowchart process step 10‑8‑1‑9F). The CHDO manager informs the certificate holder of the evaluation and provides a briefing on the process to certificate holder personnel. The certificate holder is briefed by the AFS‑900 team for a National CHEP. The notification can be verbal; the CHDO manager follows up in writing. The CHDO manager sends the written notification at least 2 weeks before the evaluation begins. The notification contains information such as that found in the evaluation notification document.

Indicates new/changed information.

G.    Revise Comprehensive Assessment Plan (see flowchart process step 10‑8‑1‑9G). The CHEP TL will add the assessments that have been selected for the evaluation to the Airworthiness and Operations CAP. The CHEP assessments will appear as a triangle at the bottom of the appropriate CAP and do not affect the Certificate Management Team (CMT) CAP. The CMT will adjust their CAP accordingly, taking into account the assessments that are completed by the CHEP Team.

Indicates new/changed information.

H.    Assign Resources to Complete Evaluation (see flowchart process step 10‑8‑1‑9H). Use the following information as a guide for assigning resources to complete evaluation:

1)    Prior to the start date of the evaluation period, the AFS‑900 FLM (for National CHEPs) or the appropriate Safety Assurance System (SAS) Administrator will add the evaluation team members to the CMT in the SAS automation.
2)    The CHEP TL distributes tasks by system or subsystem, element, individual questions, or some combination of these to one or more team members, to allow the timely collection of accurate data.
Indicates new/changed information.
3)    The FLM assigns team members to data collection activities to support the evaluation after the team members receive a briefing from the CMT on the certificate holder operations appropriate to the scope of the evaluation.

NOTE:  There is no concurrence for assessments supporting a CHEP.

I.    Collect Evaluation Data (see flowchart process step 10‑8‑1‑9I). Evaluation team members perform their assigned activities to collect data in accordance with the CAP and TL instructions.

Indicates new/changed information.

NOTE:  When conducting an SP DCT, ASIs can view the related EP DCT using the “Related Element Questions” link. If the ASI determines that additional information is needed to answer or validate an answer for an SP DCT question, the ASI selects the “Add EP DCT” option and that EP DCT is added to the ASI’s Individual Work Plan (IWP). Not all the EP DCT questions have to be answered: only the ones the ASI determines are necessary.

J.    Report Evaluation Data (see flowchart process step 10‑8‑1‑9J). Team members enter their responses into the SAS automation in accordance with the data quality guidelines (DQG) after collecting evaluation data. Critical or time-sensitive information must be reported to the CHEP TL and the PI immediately. Any unsafe condition that would result in an accident, incident, or violation discovered during the evaluation should be handled based on the applicable guidance. Objective documents which detail and support inspection findings may be attached for the PI’s review with the final report. Documents that support findings can include flight logs and maintenance records. Enter data collected by certificate holder personnel if they are part of the evaluation, and indicate the source in the SAS automation. Team members may also document any identified program strength(s) in the comment box of the applicable SP DCT, ED DCT, EP DCT, or Custom Data Collection Tool (C DCT) question, provided it is not proprietary information.

Indicates new/changed information. Indicates new/changed information.

K.    Review Evaluation Data by Team Leader (see flowchart process step 10‑8‑1‑9K). The evaluation TL is typically the Data Quality Reviewer (DQR). In some instances, it may be necessary for the DQR to proxy the data review function to a team member other than the evaluation TL. DQR rights are granted to specific CHEP team members if necessary. If the DQR finds any critical or time‑sensitive information during the review, he or she must notify the TL and PI immediately, unless the record indicates notification has already taken place.

Indicates new/changed information.

L.    Analyze and Assess Evaluation Data by Team Leader (see flowchart process step 10‑8‑1‑9L). After all data are collected, reported, and reviewed, the evaluation team completes the evaluation using the SAS Analysis, Assessment, and Action (AAA) procedures and tools (Module 5). The TL is responsible for completing the Assessment Determination, Module 5 of the AAA, considering input from the team. The PI completes the Action Determination section of the AAA and follows through with any actions required.

Indicates new/changed information.

M.    Team Leader Makes Finalized Evaluation Report (see flowchart process step 10‑8‑1‑9M). Evaluation TLs generate an evaluation report when requested by AFX‑1, the regional Flight Standards division manager, or the CHDO manager. Upon the request of the FLM, the regional manager, or the CHDO manager, the ORA assigned to the certificate holder should generate a report on the evaluation data collected.

N.    Determine and Implement Followup Actions (see flowchart process step 10‑8‑1‑9N). The PI determines and implements followup actions in response to PAs and DAs completed during the evaluation.

O.    CMT and PIs Brief Certificate Holder and Other Stakeholders (see flowchart process step 10‑8‑1‑9O). The manager responsible for initiating the evaluation determines who briefs the certificate holder or other stakeholders on the results of the evaluation.

NOTE:  If the CHEP identifies an issue with a DCT, policy, or automation, then use the feedback process to report that issue. When a CHEP identifies a possible improvement to a CHDO process, these suggestions can be tracked in the Quality Monitoring System (QMS). The CHDO should use their QMS process to improve its processes. SAS will store historical data on SAS evaluations performed, including information on systems, subsystems, and program elements assessed, for each certificate holder.

10‑8‑1‑11 through 10‑8‑1‑29 RESERVED.