9/18/20

 

8900.1 CHG 722

VOLUME 10  SAFETY ASSURANCE SYSTEM POLICY AND PROCEDURES

CHAPTER 5  DATA COLLECTION, DATA REPORTING, AND DATA REVIEW

Section 1  Safety Assurance System: Module 4 Data Collection

Indicates new/changed information.

10-5-1-1    GENERAL. The aviation safety inspector (ASI) collects data in accordance with the Comprehensive Assessment Plan (CAP). Other employees create ad hoc activities and complete planned activities, National Flight Standards Work Program Guidelines (NPG) activities, and tasks to document specific work items.

Indicates new/changed information.

A.    Purpose. The ASI uses Data Collection Tools (DCT) to document assessments of the certificate holder’s or applicant’s design of systems, surveillance of certificate holder performance, identification of safety concerns or statutory/regulatory noncompliance, and any other relevant information. The DCT information helps the principal inspector (PI), Training Center Program Manager (TCPM), and certification project manager (CPM) assess the certificate holder’s or applicant’s system performance and design.

Indicates new/changed information.

B.    Scope. This section applies to the ASI and other employees. The aviation safety technician (AST), when trained and qualified, can provide technical support to the ASI during data collection activities when assigned by their Frontline Manager (FLM). The AST can also provide technical support with research and documentation. In Module 4, Data Collection, the ASI and other employees will:

    Prepare for data collection;

Indicates new/changed information.

    Add ad hoc activities;

    Collect data;

    Complete planned activities, NPG activities, and tasks;

    Determine if safety issues or regulatory issues were identified;

    Act on safety and regulatory issues;

Indicates new/changed information.

    Create a Dynamic Observation Report (DOR), when needed, to document an unfavorable observation of performance outside the scope of the assigned DCT; and

    Inform the PI/TCPM/CPM of significant safety issues or regulatory noncompliance.

10-5-1-3    RESERVED.

Indicates new/changed information.

10-5-1-5    BACKGROUND. Module 4 contains the processes for data collection, data reporting, and data review.

A.    DCTs. There are three levels of DCTs: System, Subsystem, or Element. Each DCT has a purpose and objective statement and associated Safety Attributes: Responsibility, Authority, Procedures, Controls, Process Measurement, Interfaces, and Safety Ownership. DCTs are not checklists of questions to ask the certificate holder’s or applicant’s personnel; instead, they are a means to assess a system, subsystem, or element.

1)    Purpose. The purpose statement defines the desired outcome for which the certificate holder is responsible.
Indicates new/changed information.
2)    Objective. The objective statement establishes the Federal Aviation Administration’s (FAA) oversight objective. Before the PI accepts or approves the certificate holder’s process design, they use the data collected by the ASI to determine if the certificate holder is able to:
a)    Comply with the regulations, and
b)    Achieve the desired outcome in the purpose statement.
Indicates new/changed information.
3)    Safety Attributes. The Safety Assurance System (SAS) consists of seven Safety Attributes: Responsibility, Authority, Procedures, Controls, Process Measurement, Interfaces, and Safety Ownership. These attributes apply to all certificate holders and applicants regardless of size and scope, and should be present in well-designed systems and processes. All DCT questions are based on Safety Attributes and assist the PI in the assessment of the certificate holder/applicant’s programs and their associated risks. The PI considers the characteristics of each Safety Attribute in both the design and performance of a program. Safety Attributes may not be required by regulation; however, when incorporated into the certificate holder/applicant’s programs, they help to identify and mitigate risk. Procedures Attribute questions assess technical processes and contain regulatory and guidance references, while the remaining Safety Attribute questions assess Safety Management.
a)    Responsibility. A clearly identified individual who is accountable for ensuring financial and human resources to ensure the safety and quality performance of the certificate holder.
Indicates new/changed information.
b)    Authority. A clearly identifiable, qualified, and knowledgeable individual who effectively plans, directs, and controls resources; changes procedures; and makes key determinations, including safety risk acceptance decisions.
c)    Procedures. Methods or practices that are written or unwritten, regulatory or nonregulatory, designed into a process that a certificate holder/applicant uses to accomplish a desired result.
d)    Controls. The checks and restraints that exist within a process that ensure the potential effects of risks are reduced to an acceptable level.
e)    Process Measurement. A method to monitor and measure the outputs and performance of a process, and identify problems or potential problems, in order to take corrective action.
f)    Interfaces. Interactions between processes that must be managed in order to ensure desired outcomes.
g)    Safety Ownership. An individual’s understanding of how his or her role contributes to the overall safety of the organization.
Indicates new/changed information.

NOTE:  Safety Attribute questions should be directly observed or, if an observation is not possible, may be addressed verbally through a discussion. The ASI must always assess Safety Attributes throughout the certificate holder’s or applicant’s programs. If there is evidence that the certificate holder or applicant is addressing a Safety Attribute, then a favorable response is appropriate. If not, an unfavorable response should be selected.

B.    DCT-to-Go. Provides the ASI with the scoped DCT questions that can be printed or viewed with space for the ASI to write in responses.

C.    Note. A note contains information to clarify and help the ASI to better understand the question.

D.    Specific Regulatory Requirements (SRR). Some questions validate compliance with SRRs (Title 14 of the Code of Federal Regulations (14 CFR)). Questions that have SRRs appended to them indicate the question is based on a specific regulation (14 CFR). Questions that do not have an SRR appended to them are not based on a regulatory requirement, but are based on system safety principles and/or other FAA guidance. An unfavorable response to a question that does not have an SRR, while not a violation, may indicate a hazard with an increased level of risk (see Table 10-5-1B, Types of Responses).

NOTE:  DCTs may contain answer sets that include Yes/No, Descriptive Scale (What did you see?), and Sampling (e.g., records or parts). Table 10-5-1A, Data Collection Type and Description, describes the different types of DCTs.

Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information. Indicates new/changed information.

Table 10-5-1A.  Data Collection Type and Description

Data Collection Type

Description

System or Subsystem Performance Data Collection Tool (SP DCT)

What Does it Contain? SP DCTs contain questions related to the performance of each element within the system or subsystem. The SP DCT will include questions at either the system or subsystem level, depending on the peer group of the certificate holder/applicant. All SP DCT questions require a response.

•    For Peer Groups A, B, and E, the SP DCT will include seven questions relating to each element within the subsystem, to include one question for each Safety Attribute.

•    For Peer Groups C, D, F, G, and H, the SP DCT will include one Procedures Attribute question for each element within the system and six additional questions addressing each of the remaining Safety Attributes. These six attributes will be addressed at the system level.

•    The SP DCT Procedures Attribute questions are high-level questions that ask about a specific process or program within the system or subsystem and are based on specific regulatory requirements (SRR) and guidance. SP DCTs are directly related to and supported by the Element Performance Data Collection Tool (EP DCT).

•    If while conducting an SP DCT, the ASI determines additional surveillance is needed at the element level, then consider adding an EP DCT to document any additional surveillance. Do not document unfavorable findings from an SP DCT in the EP DCT unless the unfavorable finding is observed again during subsequent surveillance. SP DCTs include Safety Attribute questions at the element level. For example, SP DCT 1.1 asks the questions for 1.1.2, 1.1.3, etc.

•    If the unfavorable SP DCT response is related to a technical process, record the response in the “Procedures” attribute question. If it is related to Safety Management, record the unfavorable response in the applicable Safety Attribute question. If the unfavorable response is outside the scope of the assigned DCT, then submit a Dynamic Observation Report (DOR).

•    Reviewing and/or answering the questions in the EP DCT will assist the aviation safety inspector (ASI) in answering the high-level SP DCT Procedures question.

•    The ASI can select “Related Element Questions” to view the associated EP DCT procedures questions. ASIs can also print/save the associated EP DCT(s) from the “Prepare DCT” tab under “Related DCTs.” Another option is that the ASI can add an EP DCT to their Individual Work Plan (IWP) from the SP DCT. Not all the EP DCT questions require a response, only the questions the ASI determines need a response to assist with answering the associated SP DCT question.

Output:  Process observation is the expected output for the SP DCT, which is focused on the observation of certificate holder personnel performing a procedure or process.

What Are the Response Choices?  The response choices vary by question type and may include sampling, descriptive scale, “Yes,” “No,” or “Not Observable” (N/O), as appropriate.

Element Performance Data Collection Tool (EP DCT)

What Does it Contain?  EP DCTs contain detailed performance questions about each element that mirror the Element Design Data Collection Tool (ED DCT) content. EP DCTs are accomplished at the element level and are linked to the related SP DCTs. Some elements by design will not have an associated ED DCT.

When is it Accomplished?  At the discretion of the principal inspector (PI) and the certification project manager (CPM).

Output:  Process observation is the expected output for the EP DCT, which is focused on the observation of certificate holder personnel performing a procedure or process.

What Are the Response Choices?   The response choices vary by question type and may include sampling, descriptive scale, “Yes,” “No,” “Not Applicable” (N/A), or “Not Observable” (N/O), as appropriate.

Element Design Data Collection Tool (ED DCT)

What Does it Contain?  ED DCTs contain detailed design questions. Some elements by design will not have an associated ED DCT. ED DCTs are accomplished at the element level. For initial certification, the ED DCT is used to validate initial certification programs.

Output:  An assurance that a certificate holder’s programs are designed to comply with the intent of the regulations and incorporate system safety.

What Are the Response Choices?  “Yes” or “No.”

Custom Data Collection Tool (C DCT) (National/Divisional or PI C DCT)

What Does it Contain?  A C DCT is used to assess data outside the planned Performance Assessment (PA) or Design Assessment (DA). Both a PI C DCT and a National/Divisional C DCT can be used for focused inspections and special emphasis oversight, and to collect data on specific areas of concern outside of the normal planning schedule. If data needs to be collected on functions not covered by EP DCTs and ED DCTs, then a National/Divisional C DCT will need to be created.

A PI can create a C DCT to include:

•    Both scoped and unscoped questions,

•    Design questions, or

•    Performance questions.

A PI cannot create a C DCT to:

•    Combine design and performance questions, or

•    Combine airworthiness and operations questions.

C DCTs are planned and can be used at the office, divisional, or national level. If the PI determines that Safety Management System (SMS) Subsystem 1.5 requires a DA, they may add an SMS C DCT design validation, also known as a National/Divisional C DCT.

Maintenance Provider Custom Data Collection Tool (MP C DCT)

What Does it Contain?  MP C DCTs contain scoped performance-based questions that may be used by the Airworthiness PI for any ASI to perform surveillance of a maintenance provider.

The MP C DCT is:

•    Created from Master List of Functions (MLF) System 4.0, Technical Operations.

•    Includes element-level questions.

•    Identified or labeled by subsystem.

•    May be used for Coordinated Surveillance or Non-Coordinated Surveillance.

•    Seven separate MP C DCT templates.

MP C DCT characteristics include the following:

•    Scoped for the certificate holder (14 CFR part 121, 135, or 121/135 with operations specification (OpSpec) D072).

•    Allows an ASI to capture a discrepancy contained within any MP C DCT not initially resourced. This is accomplished by adding the MP C DCT to the resourced MP C DCT. Any added MP C DCT does not require all questions to be answered.

•    Allows the ASI to capture any discrepancy not associated with the assigned MP C DCT by drilling across System 4.0 or other MLFs, or adding element questions from other MP C DCTs.

•    If the ASI brings in a question from another MP C DCT, then the added question(s) will be assessed in the originating subsystem.

•    The PI will assess each discrepancy within the originating subsystem.

•    The MP C DCT may be assigned to an ASI or a team. A shared ASI may not be part of the team.

•    MP C DCTs are assessed on their own in the current quarter.

Random Inspection (RI) Custom Data Collection Tool (C DCT)

What Does it Contain?  RIs are planned or unplanned ramp inspections created from a National Template. The PI instructions may contain risk-based special emphasis items specific to a certificate holder. It could also contain National/Divisional instructions.

What Specialty Do I Select?  To record an unfavorable finding in the automation, on the “Create DCT” tab, the ASI must select the AW or OP specialty button to which the discrepancy/finding applies and not the specialty of the ASI observing it; this ensures the appropriate PI will see the unfavorable finding.

What is Required?  The ASI should document all items that were observed during the RI, including any risk-based special emphasis items. A minimum of one question must be documented.

What Are the Responses?  “Yes,” “No,” or “Not Observable” (N/O). The N/O selection may be used if risk-based special emphasis items requested in the PI or National/Divisional instructions cannot be observed.

En Route Inspection C DCT

What Does it Contain?  En Route inspections are planned or unplanned flight deck or cabin inspections created from a National Template. The PI instructions may contain risk‑based special emphasis items specific to a certificate holder. It could also contain National/Divisional instructions.

What Specialty Do I Select?  The ASI must select their specialty: Airworthiness or Operations.

What is Required?  The ASI should document all items that were observed during the En Route inspection, including any risk-based special emphasis items. A minimum of one question must be documented.

What Are the Responses?  “Yes,” “No,” or “Not Observable” (N/O). The N/O selection should only be used if risk-based specialty emphasis items requested in the PI or National/Divisional instructions cannot be observed.

Dynamic Observation Report (DOR)

What Does it Contain?  The DOR is used to record an unfavorable observation of performance outside the scope of the assigned DCT. A DOR has two options: Question‑Based or “No Applicable Questions.” The Question-Based DOR is used when there are existing SP DCT or EP DCT questions related to the observation. If no SP DCT questions relate to the observation, select “No Applicable Questions.” The Flight Standards Service (FS) can use a DOR to notify the Office of Hazardous Materials Safety (AXH) when documenting a hazardous materials (HAZMAT) safety issue for 14 CFR parts 121, 129, and 135 by selecting the “AXH” organization in the automation. AXH has the ability to notify FS of any safety issue by submitting a DOR for parts 121 and 135.

Use the DOR in the following situations:

•    Safety issues unrelated to the DCT under assessment;

•    Safety issues for which there is not an applicable DCT question;

•    Safety issues for a certificate holder or applicant to which you are not assigned; or

•    Office, divisional, or national inspection events as directed.

Indicates new/changed information. Indicates new/changed information.

Table 10-5-1B.  Types of Responses

Favorable

A “Favorable” response means that the observed conditions met the criteria specified by that DCT. Only the first response listed in the DCT answer set is favorable. A “No” response can be considered a favorable response.

Unfavorable

An “Unfavorable” response means that the observed conditions did not meet the criteria specified by the DCT. Any response not listed first in the DCT answer set is considered unfavorable. A “Yes” response can be considered an unfavorable response.

Not Observable (N/O)

A “Not Observable” (N/O) response means that the ASI was not able to observe the situation described by a particular question at the certificate holder’s or applicant’s place of operation.

Not Applicable (N/A)

A “Not Applicable” (N/A) response is used when a question does not apply to that certificate holder or applicant. Choosing this option is appropriate only for questions that are not applicable due to the type of operation authorized for the certificate holder or applicant.

10-5-1-7    DATA COLLECTION.

Figure 10-5-1A.  Module 4 Performance Assessment Data Collection Process Flowchart

Figure 10-5-1A. Module 4 Performance Assessment Data Collection Process Flowchart

10-5-1-9    PROCEDURES.

Indicates new/changed information.

A.    Prepare for Data Collection (see flowchart process step 10-5-1-9A). The first step in the data collection process is for the ASI to prepare for data collection. The following items should be reviewed prior to starting data collection:

1)    En Route Inspection and RI Instruction Reports. Assist the ASI in preparation for an En Route inspection or RI, the following reports include the risk-based areas identified by the PI and national/divisional offices:
a)    En Route/Random Instructions Report. Located on the SAS Standard Reports page, provides access to all certificate holder instructions.
b)    En Route/Random Instructions. Available when creating En Route inspection and RI C DCTs, provides PI instructions for the selected certificate holder.
c)    DCT-to-Go. Available during data collection and reporting for the En Route inspection and RI C DCTs.
2)    Pre-Inspection Checklist. When preparing for SP, EP, and ED data collection, the ASI must complete and acknowledge the Pre-Inspection Checklist in order to move to Step 10-5-1-9B, Collect Data. For Team DCTs, the team coordinator (TC) will ensure team members have completed all pre-inspection items and then acknowledge the Pre-Inspection Checklist.
a)    Review and ensure you understand the PI or CPM instructions.
b)    Review and ensure you understand the DCT questions.
c)    Review and ensure you understand the DCT purpose and objective statements.
d)    Review the SRRs.
e)    Review FAA guidance.
f)    Review the certificate holder’s or applicant’s operations specifications (OpSpecs), training programs, manuals, policies, procedures, and instructions.
g)    Review the results of previous DAs and PAs (when available).

NOTE:  The ASI should review applicable Safety Alerts for Operators (SAFO) in accordance with FAA Order 8000.87, Safety Alerts for Operators, because they are not included in the DCT‑To‑Go guidance. (Refer to National Transportation Safety Board (NTSB) Safety Recommendation A-10-031.)

3)    Related DCTs. Related DCTs are other DCTs associated with the SP DCTs or element-level DCTs assigned. The ASI may select the “Review and Print Related DCTs” link under “Related DCTs” to view or print DCTs that interface with the assigned DCT.

B.    Collect Data (see flowchart process step 10-5-1-9B).

Indicates new/changed information.
1)    Data Collection. The ASI collects data to verify that the certificate holder’s or applicant’s system complies with the intent of the regulations and safety standards. Responses to DCT questions should be directly observed. If during surveillance the ASI is unable to directly observe a process (e.g., the situation did not occur), then the ASI may address the response verbally through a discussion with the certificate holder’s or applicant’s personnel. The “DCT‑to‑Go” option provides the ASI with the scoped DCT questions that can be printed or viewed in a PDF or Word format. The ASI should complete the DCTs within the timeframes specified by the PI/TCPM/CPM. For SP DCTs, EP DCTs, and ED DCTs only:

    If a newer version of a DCT becomes available, the automation will notify you if the status of the DCT is “In Progress.”

    If you have not answered any questions, choose the newest version of the DCT for data collection.

    If you have answered questions, continue with the previous version of the DCT.

    If you are completing an En Route inspection or RI C DCT, the PI/National/Divisional instructions and questions are locked when the “Continue” button is selected under the “Prepare DCT” tab.

Indicates new/changed information.
2)    ED DCTs Conducted in Partnership with the Certificate Holder. When certificate holder personnel participate in the conduct of an ED DCT, they are considered members of the ED DCT team.

NOTE:  If noncompliance is discovered during data collection, and the holder is proactive and willing to correct the issue, then the ASI must document the Compliance Action in accordance with Volume 14.

C.    Were Safety or Regulatory Noncompliance Issues Observed? (see flowchart process step 10‑5‑1‑9C). If the ASI observes safety issues or regulatory noncompliance, then see Step 10-5-1-9D, Act on Safety Issues or Regulatory Noncompliance. If the ASI observes conditions other than safety issues or regulatory noncompliance, then see Step 10-5-1-9E, Document Issues Other Than Safety or Regulatory Noncompliance.

D.    Act on Safety Issues or Regulatory Noncompliance (see flowchart process step 10-5-1-9D). If the ASI observes a condition that could result in an accident, incident, or regulatory noncompliance:

1)    Stop data collection, intervene by notifying the appropriate certificate holder’s or applicant’s personnel, and take action to resolve the issue.
2)    Inform the PI/CPM.
3)    Document the finding and any action by the ASI in the “Inspector Action Taken” field in accordance with Volume 10, Chapter 5, Section 2.
Indicates new/changed information.
4)    If the ASI observes issues not related to the assigned DCT, then document the unfavorable finding using a DOR.

E.    Document Issues Other Than Safety or Regulatory Noncompliance (see flowchart process step 10-5-1-9E). For unfavorable responses not associated with a safety issue or regulatory noncompliance, the ASI should document the finding in the DCT and any action by the ASI in the “Inspector Action Taken” field in accordance with Volume 10, Chapter 5, Section 2.

10-5-1-11 through 10-5-1-29 RESERVED.