Volume 10 AIR TRANSPORTATION OVERSIGHT SYSTEM
CHAPTER 2 PROCEDURES FOR DESIGN AND PERFORMANCE ASSESSMENT
Section 4 Design Assessment Data Collection
Figure 10-48. Module 4: Data Collection
10-171 INTRODUCTION. The objective of this process module is to collect design data in accordance with the Comprehensive Assessment Plan (CAP) and principal
inspector (PI) or certification project manager (CPM) instructions. Data collected on the Safety Attribute Inspection (SAI) Data Collection Tool (DCT) and/or the
Constructed Dynamic Observation Report (ConDOR) is used to assess air carrier or applicant system design.
10-172 COLLECT REQUIRED DATA. Trained and qualified Federal Aviation Administration (FAA) Operations, Airworthiness, Cabin Safety, and/or Aircraft
Dispatcher safety inspectors assigned to an Air Transportation Oversight System (ATOS) Certificate Management Team (CMT) or Certification Project Team (CPT)
use SAI DCTs to collect Design Assessment (DA) data. A team of inspectors or a single inspector may complete the SAI. PIs should consider the nature and complexity of the element under scrutiny,
and consider whether the single-inspector method is appropriate in each case. Each SAI receives a team coordinator (TC). (See flowchart process step 4.1.)
A. DAs Conducted in Partnership With the Air Carrier. The air carrier may collaborate with the FAA to complete the DA. When collaborating
on the assessment, air carrier personnel are active participants and working members of the SAI team, and determine and resolve element evaluation issues with the PI.
1) When an air carrier actively participates as a working member of the evaluation team, the provisions of the current edition of Advisory Circular (AC)
Disclosure Reporting Program, govern apparent violations of FAA regulations discovered during the assessment and subsequent enforcement action.
2) When an air carrier elects to not actively participate in the assessment as a working member of the SAI team, the provisions and protections contained in AC
00-58 do not apply to apparent violations of FAA regulations discovered
during the specified evaluation period.
B. Coordinate SAI Team and Establish Communication Methods. The SAI TC decides how the team communicates. Coordination and communication are
especially important if members are spread among different locations. After reviewing the PI instructions, the TC organizes a team meeting. This meeting
can be in person, over the phone, or by other means.
C. Distribute and Schedule Tasks. Either a team of inspectors or a single inspector collects the data. Distribute tasks by element, safety
attribute, individual question, or some combination to allow the timely collection of accurate data. The TC also ensures that activities, such as the operator’s
personnel interviews, are not redundant and that all activities are complete and accurately answers the questions on the SAI. The TC distributes tasks among
the SAI team and develops a timeline to complete the assigned data collection activities. The TC, in conjunction with the remaining SAI team members, divides
and distributes the SAI activities, but is not the supervisor. If the TC encounters difficulties with a team member during an assessment, the situation becomes
elevated through his or her Front Line Manager (FLM) for resolution.
D. Prepare to Perform Assigned Data Collection Activities. Inspectors prepare for DA data collection by reviewing, at a minimum, the following:
1) PI or CPM instructions.
2) Current SAI DCTs, available online, for the applicable element.
3) Specific regulatory requirements (SRR) related to the element.
4) Relevant FAA guidance, such as orders and ACs.
5) Air carrier or applicant policies and procedures (e.g., manuals, operations specifications (OpSpecs), training programs) for the applicable element.
6) The results of previous DAs and Performance Assessments (PA).
E. Perform Data Collection Using the SAI. Each SAI team member submits their responses into ATOS automation after they complete their data
collection activities. Communication between team members is essential, but sharing answers is not necessary or desirable because of possible duplication. Inspectors should complete SAIs within
the timeframes the PI or CPM establishes so that data are available for timely completion of the DA. Inspectors follow the general instructions for using SAI DCTs.
NOTE: Some elements are interrelated with associated inspector specialties (Operations, Airworthiness—e.g., 2.1.1, 5.1.9). In instances like this, both
specialties should work those SAIs concurrently.
F. SAI DCT Sections. SAIs have seven sections: Element Summary Information, Supplemental Information, Procedures Attribute, Controls Attribute,
Process Measurement Attribute, Interfaces Attribute, and Management Responsibility and Authority Attribute.
NOTE: Any aviation safety inspector (ASI) who is conducting an SAI or Element Performance Inspection (EPI) must first access the ATOS Temporary Revisions
to determine if a Temporary Revision is applicable to the DCT. Temporary Revisions are found in Flight Standards Information Management System (FSIMS) Publications
and can be accessed via a link under the ATOS “News and Documentation” page.
1) Element Summary Information: Scope of Element. Scope of element
is a consideration of the air carrier’s responsibility and the FAA’s responsibility.
a) Purpose. All elements represent processes the air carrier performs. The purpose statement defines the intent of the element and the scope of the certificate
holder’s responsibility. Policies and procedures describe a certificate holder’s process. The safety attributes in each SAI help organize the content of the
SAI and aid the PI in determining the acceptability or approvability of the air carrier’s process.
b) Objective. The objective tells the ASI the scope of element in general terms and identifies the FAA’s responsibility.
c) Specific Instructions. Some DCTs may contain specific instructions for additional training, experience, or qualifications that may be helpful in determining
inspector assignments. Specific instructions may also include additional references, background information, or manuals to review, as well as suggestions for specific
types of activities and/or reporting instructions.
2) Supplemental Information.
a) Regulatory Requirements. Each SAI includes regulatory requirements as
references for the inspector. The regulatory requirements are displayed by the Code of Federal Regulations (CFR) part and section only.
b) Related CFRs and FAA Policy/Guidance.
1. Related CFRs. This section is intentionally left blank.
2. FAA Policy and Guidance. FAA policy and guidance are included for
background information that is necessary to accomplish the inspection. This information may not be mandatory and may not be regulatory. It provides guidance
and outlines a method of compliance.
NOTE: Regulatory and FAA policy/guidance references appear at the question level.
3) SAI Attribute Sections. Each SAI attribute section includes
an objective. The following paragraphs describe some of the content in each section of the DCT.
a) Procedures Attribute. The questions in this section of the SAI will help verify that the air carrier documented procedures identify who, what, when, where,
and how those procedures are accomplished. These procedures must allow all personnel to perform their duties and responsibilities with a high degree of safety. Written
policies, procedures, or instructions and information that are interrelated and located in different manuals must be consistent and ensure the effective
coordination of work activities from one person, workgroup, or organization to another to ensure the desired outcome for the process. Many of these questions
have SRRs for this process, although the air carrier may have some latitude in implementing others. For this reason, a response of “No” to one of these
questions does not necessarily mean that the company is not complying with a regulation, or that any action is necessary.
b) Controls Attribute. The questions in this section of the SAI will help determine if the air carrier designed controls (e.g., checks and restraints)
into the processes associated with this element to ensure that it follows policies and procedures to achieve desired results. While most controls are not regulatory,
they are an important safety attribute with desirable features that help to reduce unacceptable levels of risk. Each SAI lists a series of controls. Some
common types of controls are flags, data system backups, authorized signatures, separation of duties, or a final review. It is important to note that certificate
holders must be able to show the effectiveness of their controls. Few of these controls have their basis in SRRs. For this reason, a response of “No” to one
of these questions does not necessarily mean that the company is not complying with a regulation or that any action is necessary.
c) Process Measures Attribute. Process measures ensure that the operator uses an
internal evaluation function to detect, identify, and eliminate or control hazards and the associated risk. For airworthiness elements, this is a required function
of the operator’s Continuing Analysis and Surveillance System (CASS) required by Title 14 of the Code of Federal Regulations (14 CFR) part
Director of Safety (DOS) and the quality assurance (QA) department often work together to accomplish this function for the operator. Negative findings could require amendments to the Safety/Internal
Evaluation Program (IEP) or CASS audit forms or checklists. In most cases, process measures are nonregulatory. For this reason, a response of “No” to one of these questions, while not a violation,
may indicate a hazard with an increased level of risk, which may require additional CMT or CPT action.
d) Interfaces Attribute. Data collected in this section helps the PI determine if the air carrier identifies, documents and manages change between this process
and other related processes within the air carrier organization. It is important for the air carrier to identify and document where interactions between processes
exists, and to have a method of managing change between these processes. Written policies, procedures, or instructions and information that are interrelated
and located in different manuals within the air carrier’s manual system must be consistent to allow personnel to perform their duties and responsibilities with a high degree of safety.
e) Management Responsibility and Authority Attribute. Data from questions in this section will help determine if there is an identifiable, qualified (when
required by a CFR), and knowledgeable person who is responsible for the process, answerable for the quality of the process, and has the authority to establish
and modify the process. Often, many organizations disperse authority and responsibility. A person can be an individual, a department, a committee, or a position (such
as vice president of flight operations). The intent is to identify the highest level person (at the appropriate level within the organization) who is responsible
or has the authority for that particular element of the certificate holder’s system.
4) Tasks. Each attribute section of the DCT contains the statement, “The inspector must accomplish the following task(s).” Various activities comprise
a single task. The following are typical tasks that an SAI includes:
a) Review the information listed in the “Supplemental Information” section of this DCT. The “Supplemental Information” section of the SAI provides a list
of the regulatory requirements and FAA policy and guidance documents that are pertinent to the questions of the DCT for a given element. Regulatory and FAA
policy and guidance references also appear at the question level.
b) Review the duties and responsibilities for management and other personnel who accomplish the process associated with this element.
c) Review the documentation of the processes associated with the element. Review the policies, procedures, instructions, and information to understand
the controls associated with this element. This usually involves reviewing sections of the appropriate OpSpecs training programs or other documents, as well as
the manuals related to the process.
d) Review the interfaces associated with the processes for the element. Some questions in the “Procedures” section contain references to interfaces
in the “Related Design” Job Task Item (JTI). The inspector reviews those references to identify the interfaces in the certificate holder’s manual.
e) Identify the person who has overall responsibility for the element. The inspector must sufficiently understand the certificate holder’s system to know
who is responsible for the quality of each process.
f) Identify the person(s) who has overall authority to revise the procedures associated with the element. The inspector must sufficiently understand the
certificate holder’s system to know who has the authority to establish or modify each process.
g) Review the duties and responsibilities of the person(s) documented in the certificate holder’s manual. The inspector must sufficiently understand
the certificate holder’s system to know the duties and responsibilities of individuals assigned the responsibility for each process or authority to change each process.
h) Review the appropriate organization chart. The inspector must sufficiently understand the certificate holder’s organization to identify who has the authority
and responsibility for certain processes. Often, many organizations disperse authority and responsibility. A person can be an individual, a department, a committee, or a position.
5) Questions. Each SAI lists a series of questions for the SAI team to answer based on their review and analysis during various activities.
Answer questions on each activity report in response to the reviews and analysis on that single activity. The DCTs are not checklists of questions to ask directly of the certificate holder’s personnel.
a) Questions based on regulatory requirements have an SRR appended to them. Therefore, answering “No” to such a question may require an enforcement investigation
for a certificated air carrier, or may lead to rejecting a program or authorization proposed by an applicant.
b) Questions that do not have an SRR appended to them are not tied to a literal regulatory requirement, but are based on system safety principles. A
“No” answer to this type of question, while not a violation, may indicate a hazard with an increased level of risk that may require additional CMT or CPT
action, including a decision to withhold approval or acceptance of a system or program.
c) A “D” or “Domestic,” “F” or “Flag,” “S” or “Supplemental,” or any combination of these identifies each question. These identify the scope of operation and
help the inspector determine if the question applies. For instance, if the air carrier is a supplemental air carrier, and there is a “D” or “Domestic” next
to the question, this question would not be applicable to a supplemental operation.
NOTE: DCT users are responsible to ensure that they reference the current edition of the guidance.
6) JTIs. JTIs are for inspector reference only. JTIs aid the inspector in determining if a certificate holder’s written policies, procedures,
instructions, and information are adequate. The FAA does not expect the inspector to respond to each JTI individually. The JTIs listed below each question are
there to aid inspectors in answering the question. If a question appears too nonspecific, review the associated JTI to identify the specific requirements.
Many JTIs contain FAA policy and guidance referenced within the JTI. Due to automation issues, the policy and guidance reference may only appear in FSIMS copies of the JTIs.
7) Notes. Notes supplement SAI and EPI questions. Notes contain clarifying information to help the ASI better understand the question.
G. Perform Data Collection Using ConDORs. The PI or the CPM may request inspectors to collect DA data using a ConDOR to address focused or unique
situations. The ASI performs the appropriate tasks listed on the ConDOR for each inspection to answer accurately all the questions the PI or CPM require.
10-173 IDENTIFICATION OF REGULATORY NONCOMPLIANCE ISSUES. During data collection activities for DA, the inspector may identify issues of regulatory
noncompliance that require an immediate response. (See flowchart process step 4.2.)
A. Interpreting “No” Responses. Not all “No” responses to SAI DCT questions indicate regulatory noncompliance. A “No” answer can also mean
inadequate inclusion of the safety attributes in the area under evaluation, or that the certificate holder’s approved or accepted procedures are inadequate.
B. Regulatory Basis of a “No” Response. A “No” answer does not necessarily equate to an unsafe condition or a regulatory violation, unless
that particular “No” has a regulatory basis (including regulatory intent) and the inspector observed a possible violation or unsafe condition.
10-174 NOTIFY THE PI/CPM OF REGULATORY NONCOMPLIANCE. After identifying an issue of regulatory noncompliance that requires an immediate response, the
inspector notifies the PI or CPM. The PI follows the guidance outlined in the current edition of FAA Order 2150.3, FAA Compliance and Enforcement Program,
to address the issue. (See flowchart process step 4.3.)
10-175 OBSERVED UNRELATED SAFETY ISSUES. While collecting the required data for the DA, the inspector may observe unrelated safety issues, which are
outside the scope of the SAI DCT questions. The inspector takes appropriate action, including communicating the safety issue to air carrier or applicant
personnel and the PI or CPM. ATOS does not change an inspector’s responsibility to investigate and act on safety or regulatory concerns. (See flowchart process step 4.4.).
A. Address Significant Issues. The inspector should promptly convey to the appropriate PI or CPM significant issues or items of immediate concern.
B. Address Significant Issues Involving Hazardous Materials (Hazmat).
The inspector must promptly convey to the appropriate PI or CPM significant issues or items of immediate concern regarding all aspects of the air transportation
of hazmats. The PI or CPM will coordinate with the regional hazmat branch manager.
C. Address Significant Issues Involving Drugs or Alcohol. Significant issues or items of immediate concern regarding all aspects of the air carrier’s
or applicant’s drug testing program and alcohol misuse prevention program must be promptly conveyed to the appropriate PI or CPM who then coordinates with the Drug Abatement Division (AAM-800).
10-176 DOCUMENT UNRELATED SAFETY ISSUES VIA A DYNAMIC OBSERVATION REPORT (DOR). Use the DOR to document observations associated with unrelated safety
issues found during data collection activities. These observations may involve any part
carrier at any location, at any time. DORs are not a substitute for the planned inspections. Managers and inspectors use the DOR in the following
situations (see flowchart process step 4.5.):
· Safety issues unrelated to the ATOS element under assessment;
· Safety issues for which there is not an applicable ATOS element or DCT question;
· Safety issues for an air carrier to which the inspector is not assigned;
· Specific inspection events as directed by a handbook or other national directives;
· Safety issues observed by CMT members (assigned to that air carrier)
during incidental travel from one location to another to perform official business; and
· Safety issues observed while conducting a random inspection if
there is not a question to address the safety issue.
RESERVED. Paragraphs 10-177 through 10-190.