Volume 10 AIR TRANSPORTATION OVERSIGHT SYSTEM
chapter 2 PROCEDURES FOR DESIGN AND PERFORMANCE ASSESSMENT
Section 4 Design Assessment Data Collection
Figure 10-48. Module 4: Data Collection
10-171 INTRODUCTION. The objective of this process module is to collect
design data in accordance with the Comprehensive Assessment Plan (CAP) and principal
inspector (PI) or certification project manager (CPM) instructions. Data collected
on the Safety Attribute Inspection (SAI) Data Collection Tool (DCT) and/or the
Constructed Dynamic Observation Report (ConDOR) is used to assess air carrier
or applicant system design.
10-172 COLLECT REQUIRED DATA (see flowchart process step 4.1). Trained and qualified
Federal Aviation Administration (FAA) Operations, Airworthiness, Cabin Safety,
and Aircraft Dispatcher aviation safety inspectors (ASI) assigned to an Air
Transportation Oversight System (ATOS) Certificate Management Team (CMT) or
Certification Project Team (CPT) use SAI DCTs to collect Design Assessment (DA)
data. A team of ASIs or a single ASI may complete the SAI. PIs should consider
the nature and complexity of the element under scrutiny and consider whether
the single-ASI method is appropriate in each case. Each SAI receives a team coordinator (TC).
A. DAs Conducted in Partnership with the Air Carrier. The air
carrier may collaborate with the FAA to complete the DA. When collaborating
on the assessment, air carrier personnel are active participants and working
members of the SAI team and determine and resolve element evaluation issues with the PI.
1) When an air carrier actively participates as a working member
of the evaluation team, the provisions of the current edition of Advisory Circular (AC)
Disclosure Reporting Program, govern apparent violations of FAA regulations
discovered during the assessment and subsequent enforcement action.
2) When an air carrier elects to not actively participate in
the assessment as a working member of the SAI team, the provisions and protections contained in AC
not apply to apparent violations of FAA regulations discovered
during the specified evaluation period.
B. Coordinate SAI Team and Establish Communication Methods. The
SAI TC decides how the team communicates. Coordination and communication are
especially important if members are spread among different locations. After
reviewing the PI instructions, the TC organizes a team meeting. This meeting
can be in person, over the phone, or by other means.
C. Distribute and Schedule Tasks. Either a team of ASIs or a single ASI
collects the data and distributes tasks by element, safety attribute, individual
question, or some combination to allow the timely collection of accurate data.
The TC also ensures that activities, such as the air carrier’s personnel interviews,
are not redundant and that all activities are complete and accurately answer
the questions on the SAI. The TC distributes tasks among the SAI team and develops
a timeline to complete the assigned data collection activities. The TC, in conjunction
with the remaining SAI team members, divides and distributes the SAI activities
but is not the supervisor. If the TC encounters difficulties with a team member
during an assessment, the situation becomes elevated through his or her Front
Line Manager (FLM) for resolution.
D. Prepare to Perform Assigned Data Collection Activities. ASIs prepare
for DA data collection by reviewing, at a minimum, the following:
1) PI or CPM instructions.
2) Current SAI DCTs, available online, for the applicable element.
3) Specific regulatory requirements (SRR) related to the element.
4) Relevant FAA guidance, such as orders and ACs.
5) Air carrier or applicant policies and procedures (e.g., manuals,
operations specifications (OpSpecs), training programs) for the applicable element.
6) The results of previous DAs and Performance Assessments (PA).
E. Perform Data Collection Using the SAI. Each SAI team member submits their
responses into ATOS automation after they complete their data collection activities.
Communication between team members is essential, but sharing answers is not
necessary or desirable because of possible duplication. ASIs should complete
SAIs within the timeframes the PI or CPM establishes so that data are available
for timely completion of the DA. ASIs follow the general instructions for using SAI DCTs.
F. SAI DCT Sections. SAIs have seven parts: Element Summary Information,
Supplemental Information, and five sections for the Procedures Attribute, Controls
Attribute, Process Measurement Attribute, Interfaces Attribute, and Management
Responsibility and Authority Attribute. These attributes are not standards that
must be complied with, but rather serve as a framework we use to assess the
capabilities of the process.
NOTE: Prior to conducting an SAI or EPI, ASIs must first review the ATOS Broadcast
Message(s) (BCM) as applicable to the DCT. This ensures the ASI considers the
most current question information during the assessment. BCMs appear as a pop-up
on the ATOS home page until acknowledged. BCMs may also be found on the Flight
Standards Information Management System (FSIMS) Web site via a link under the
“Publications” tab. The previous quarter’s BCMs remain available via a link
under the ATOS “New and Documentation” page.
1) Element Summary Information: Scope of Element. Scope of element
is a consideration of the air carrier’s responsibility and the FAA’s responsibility.
a) Purpose. All elements are air carrier safety-critical processes. The purpose
statement defines the desired outcome that is the air carrier’s responsibility.
b) Objective. The objective statement establishes the FAA’s oversight
objective. Before accepting or approving the air carrier’s process design, ASIs
collect data that the PI uses to determine if the air carrier will be able to:
1. Comply with the regulations.
2. Achieve the desired outcome in the purpose statement.
c) Specific Instructions. Some DCTs may contain specific instructions for additional
training, experience, or qualifications that may be helpful in determining ASI
assignments. Specific instructions may also include additional references, background
information, or manuals to review, as well as suggestions for specific types
of activities and/or reporting instructions.
2) Supplemental Information.
a) Regulatory Requirements. Each SAI includes regulatory requirements
as references for the ASI. The regulatory requirements are displayed by the Code of Federal
Regulations (CFR) part and section only.
b) Related CFRs and FAA Policy/Guidance.
1. Related CFRs. This section is intentionally left blank.
2. FAA Policy and Guidance. FAA policy and guidance are included
for background information that is necessary to accomplish the inspection. Air carriers are
not required to comply with FAA policy or guidance. ASIs are required to use
the applicable policy and guidance (orders, notices, and ACs) during the evaluation
of an air carrier’s program or system to determine if the air carrier met the
regulations, including the intent of the regulations. Policy and guidance material
may include language that refers to regulations or identifies safety requirements.
Policy and guidance material may also describe an acceptable means, but not
necessarily the only means, for demonstrating that procedures comply with the
applicable regulations and meet the intent and level of safety.
3) SAI Safety Attribute Sections. The key to safety lies in managing
the quality of safety-critical processes. This is a primary responsibility of
an air carrier in meeting its regulatory obligations. Applicants and air carriers
are not expected to use the safety attributes as an outline for their process
manuals development. For instance, some mistakenly use the safety attributes
as paragraph headings. This practice only complicates the procedures and makes
it difficult for the end user, the employee, to gain meaningful information
to accomplish work. This is especially serious when used to present safety-critical
procedures and tasks. A properly constructed process manual should provide the
stated policy or goal related to the procedure or task, followed by clearly
written procedures and steps to accomplish the task or activity in a logical
manner. ATOS employs six safety attributes to evaluate the design of an air
carrier or applicant’s safety critical processes. Each SAI attribute section
includes an objective. The following paragraphs describe some of the content
in each attribute of the SAI DCT.
a) Procedures Attribute. The questions in this section of the SAI
will help verify that the air carrier documented procedures that identify who,
what, when, where, and how those procedures are accomplished. These procedures
must allow all personnel to perform their duties and responsibilities with a
high degree of safety. Written policies, procedures, or instructions and information
that are interrelated and located in different manuals must be consistent and
ensure the effective coordination of work activities from one person, workgroup,
or organization to another to ensure the desired outcome for the process. Many
of these questions have SRRs for this process, although the air carrier may
have some latitude in implementing others. For this reason, a response of “No”
to one of these questions does not necessarily mean that the company is not
complying with a regulation, or that any action is necessary.
b) Controls Attribute. The questions in this section of the SAI will
help determine if the air-carrier-designed controls (e.g., checks and restraints) into the
processes associated with this element to ensure that it follows policies and
procedures to achieve desired results. While most controls are not regulatory,
they are an important safety attribute with desirable features that help to
reduce unacceptable levels of risk. Each SAI lists a series of controls. Some
common types of controls are flags, data system backups, authorized signatures,
separation of duties, or a final review. It is important to note that air carriers
must be able to show the effectiveness of their controls. Few of these controls
have their basis in SRRs. For this reason, a response of “No” to one of these
questions does not necessarily mean that the company is not complying with a
regulation or that any action is necessary.
c) Process Measures Attribute. Process measures ensure that the air
carrier uses an internal evaluation function to detect, identify, and eliminate or control
hazards and the associated risk. For airworthiness elements, this is a required
function of the air carrier’s Continuing Analysis and Surveillance System (CASS)
required by Title 14 of the Code of Federal Regulations (14 CFR) part
Director of Safety (DOS) and the quality assurance (QA) department
often work together to accomplish this function for the air carrier. Negative
findings could require amendments to the Safety/Internal Evaluation Program
(IEP) or CASS audit forms or checklists. In most cases, process measures are
nonregulatory. For this reason, a response of “No” to one of these questions,
while not a violation, may indicate a hazard with an increased level of risk,
which may require additional CMT or CPT action.
d) Interfaces Attribute. Data collected in this section helps the
PI determine if the air carrier identifies, documents, and manages change between
this process and other related processes within the air carrier organization.
It is important for the air carrier to identify and document where interactions
between processes exists and to have a method of managing change between these
processes. Written policies, procedures, or instructions and information that
are interrelated and located in different manuals within the air carrier’s manual
system must be consistent to allow personnel to perform their duties and responsibilities
with a high degree of safety.
e) Management Responsibility and Authority Attribute. Data from questions in this
section will help determine if there is an identifiable, qualified, and knowledgeable
individual who is responsible for the quality of the entire process. While other
employees may be assigned partial responsibility for various parts of the process,
the responsible individual is always answerable for the quality of the entire
process. Data from this section also helps determine if there is a clearly identifiable,
qualified, and knowledgeable individual who has the authority to establish and
modify the process. When the authority individual is absent, the authority to
establish and modify the process should be delegated to another individual.
The intent is to identify the highest level individual (at the appropriate level
within the organization) who is responsible for the quality of the entire process
and the individual who has the authority to establish and modify the process
for that particular element of the air carrier’s system.
4) Tasks. Each attribute section of the DCT contains the statement,
“The ASI must accomplish the following task(s).” Various activities comprise
a single task. The following are typical tasks that an SAI includes:
a) A review of the information listed in the “Element Summary” and
the “Supplemental Information” section of this DCT. All questions in the DCT
should collect data that supports the FAA Objective to help the PI determine
if the air carrier will be able to comply with the regulations and achieve the
desired outcome as stated in the purpose statement. The “Supplemental Information”
section of the SAI provides a list of the regulatory requirements and FAA policy
and guidance documents that are pertinent to the questions of the DCT for a given element.
Regulatory and FAA policy and guidance references also appear at the question level.
b) Review the duties and responsibilities for management and other
personnel who accomplish the process associated with this element.
c) Review the documentation of the processes associated with the
element. Review the policies, procedures, instructions, and information to understand
the controls associated with this element. This usually involves reviewing sections
of the appropriate OpSpecs training programs or other documents, as well as
the manuals related to the process.
d) Review the interfaces associated with the processes for the element.
e) Identify the individual who has overall responsibility for the quality of the
process for the element. The ASI must sufficiently understand the air carrier’s
system to know who is responsible for the quality of each process.
f) Identify the individual(s) who has overall authority to establish
and revise the procedures associated with the element. The ASI must sufficiently
understand the air carrier’s system to know who has the authority to establish
or modify each process.
g) Review the duties and responsibilities of the individual(s) documented
in the air carrier’s manual. The ASI must sufficiently understand the air carrier’s
system to know the duties and responsibilities of individuals assigned the responsibility
for each process or authority to change each process.
h) Review the appropriate organization chart. The ASI must sufficiently
understand the air carrier’s organization to identify who has the authority
and responsibility for certain processes. Often, many organizations disperse
authority and responsibility.
5) Questions. Each SAI lists a series of questions for the SAI team to answer
based on their review and analysis during various activities. Answer questions
on each activity report in response to the reviews and analysis on that single
activity. The DCTs are not checklists of questions to ask directly of the air carrier’s personnel.
a) Questions based on regulatory requirements have an SRR appended
to them. Therefore, answering “No” to such a question may require an enforcement
investigation for a certificated air carrier or may lead to rejecting a program
or authorization proposed by an applicant.
b) Questions that do not have an SRR appended to them are not tied
to a literal regulatory requirement but are based on system safety principles.
A “No” answer to this type of question, while not a violation, may indicate
a hazard with an increased level of risk that may require additional CMT or
CPT action, including a decision to withhold approval or acceptance of a system or program.
c) A “D” or “Domestic,” “F” or “Flag,” “S” or “Supplemental,” or any combination
of these identifies the scope of the question and can help you determine if
the question applies. For instance, if the air carrier is a supplemental air
carrier, and there is a “D” or “Domestic” next to the question, this question
would not be applicable to a supplemental operation.
6) Job Task Items (JTI). JTIs are for ASI reference only. JTIs
aid the ASI in determining if an air carrier’s written policies, procedures,
instructions, and information are adequate. The FAA does not expect the ASI
to respond to each JTI individually. The JTIs listed below each question are
there to aid ASIs in answering the question. If a question appears too nonspecific,
review the associated JTI to identify the specific requirements. Many JTIs contain
FAA policy and guidance referenced within the JTI. Due to automation issues,
the policy and guidance reference may only appear in FSIMS copies of the JTIs.
7) Notes. Notes supplement SAI and EPI questions. Notes contain clarifying
information to help the ASI better understand the question.
G. Perform Data Collection Using ConDORs. The PI or the CPM may
request ASIs to collect DA data using a ConDOR to address focused or unique
situations. The ASI performs the appropriate tasks listed on the ConDOR for
each inspection to answer accurately all the questions the PI or CPM require.
10-173 WERE ISSUES OF REGULATORY NONCOMPLIANCE IDENTIFIED? (see flowchart process
step 4.2). During data collection activities for DA, the ASI may identify
issues of regulatory noncompliance that require an immediate response.
A. Interpreting “No” Responses. Not all “No” responses to SAI DCT questions
indicate regulatory noncompliance. A “No” answer can also mean inadequate inclusion
of the safety attributes in the area under evaluation, or that the air carrier’s
approved or accepted procedures are inadequate.
B. Regulatory Basis of a “No” Response. A “No” answer does not necessarily
equate to an unsafe condition or a regulatory violation, unless that particular
“No” has a regulatory basis (including regulatory intent) and the ASI observed
a possible violation or unsafe condition.
10-174 NOTIFY PI/CPM OF REGULATORY NONCOMPLIANCE (see flowchart process step 4.3).
After identifying an issue of regulatory noncompliance that requires an immediate
response, the ASI notifies the PI or CPM. The PI follows the guidance outlined
in the current edition of FAA Order
Compliance and Enforcement Program, to address the issue.
10-175 WERE UNRELATED SAFETY ISSUES OBSERVED? (see flowchart process step 4.4).
While collecting the required data for the DA, the ASI may observe unrelated
safety issues, which are outside the scope of the SAI DCT questions. The ASI
takes appropriate action, including communicating the safety issue to air carrier
or applicant personnel and the PI or CPM. ATOS does not change an ASI’s responsibility
to investigate and act on safety or regulatory concerns.
A. Address Significant Issues. The ASI should promptly convey to the appropriate
PI or CPM significant issues or items of immediate concern.
B. Address Significant Issues Involving Hazardous Materials (Hazmat).
The ASI must promptly convey to the appropriate PI or CPM significant issues
or items of immediate concern regarding all aspects of the air transportation
of hazmats. The PI or CPM will coordinate with the regional hazmat branch manager.
C. Address Significant Issues Involving Drugs or Alcohol. Significant
issues or items of immediate concern regarding all aspects of the air carrier’s
or applicant’s drug testing program and alcohol misuse prevention program must
be promptly conveyed to the appropriate PI or CPM, who then coordinates with
the Drug Abatement Division (AAM-800).
10-176 DOCUMENT UNRELATED SAFETY ISSUES VIA DOR (see flowchart process step 4.5).
Use the Dynamic Observation Report (DOR) to document observations associated with safety issues unrelated to your
assigned data collection activities. These observations may involve any part
carrier at any location, at any time. DORs are not a substitute for the planned
inspections. Managers and ASIs use the DOR in the following situations:
· Safety issues unrelated to the ATOS element under assessment;
· Safety issues for which there is not an applicable ATOS element or DCT question;
· Safety issues for an air carrier to which the ASI is not assigned;
· Specific inspection events as directed by a handbook or other national directives;
· Safety issues observed by CMT members (assigned to that air carrier)
during incidental travel from one location to another to perform official business; and
· Safety issues observed while conducting a random inspection if
there is not a question to address the safety issue.
RESERVED. Paragraphs 10-177 through 10-190.