11/21/14

 

8900.1 CHG 368

Volume 10  AIR TRANSPORTATION OVERSIGHT SYSTEM

chapter 2  PROCEDURES FOR DESIGN AND PERFORMANCE ASSESSMENT

Section 4  Design Assessment Data Collection

Figure 10-48.  Module 4: Data Collection

Figure 10-48. Module 4: Data Collection

10-171    INTRODUCTION. The objective of this process module is to collect design data in accordance with the Comprehensive Assessment Plan (CAP) and principal inspector (PI) or certification project manager (CPM) instructions. Data collected on the Safety Attribute Inspection (SAI) Data Collection Tool (DCT) and/or the Constructed Dynamic Observation Report (ConDOR) is used to assess air carrier or applicant system design.

Indicates new/changed information.

10-172    COLLECT REQUIRED DATA (see flowchart process step 4.1). Trained and qualified Federal Aviation Administration (FAA) Operations, Airworthiness, Cabin Safety, and Aircraft Dispatcher aviation safety inspectors (ASI) assigned to an Air Transportation Oversight System (ATOS) Certificate Management Team (CMT) or Certification Project Team (CPT) use SAI DCTs to collect Design Assessment (DA) data. A team of ASIs or a single ASI may complete the SAI. PIs should consider the nature and complexity of the element under scrutiny and consider whether the single-ASI method is appropriate in each case. Each SAI receives a team coordinator (TC).

A.    DAs Conducted in Partnership with the Air Carrier. The air carrier may collaborate with the FAA to complete the DA. When collaborating on the assessment, air carrier personnel are active participants and working members of the SAI team and determine and resolve element evaluation issues with the PI.

1)    When an air carrier actively participates as a working member of the evaluation team, the provisions of the current edition of Advisory Circular (AC) 00-58, Voluntary Disclosure Reporting Program, govern apparent violations of FAA regulations discovered during the assessment and subsequent enforcement action.
2)    When an air carrier elects to not actively participate in the assessment as a working member of the SAI team, the provisions and protections contained in AC 00-58 do not apply to apparent violations of FAA regulations discovered during the specified evaluation period.

B.    Coordinate SAI Team and Establish Communication Methods. The SAI TC decides how the team communicates. Coordination and communication are especially important if members are spread among different locations. After reviewing the PI instructions, the TC organizes a team meeting. This meeting can be in person, over the phone, or by other means.

Indicates new/changed information.

C.    Distribute and Schedule Tasks. Either a team of ASIs or a single ASI collects the data and distributes tasks by element, safety attribute, individual question, or some combination to allow the timely collection of accurate data. The TC also ensures that activities, such as the air carrier’s personnel interviews, are not redundant and that all activities are complete and accurately answer the questions on the SAI. The TC distributes tasks among the SAI team and develops a timeline to complete the assigned data collection activities. The TC, in conjunction with the remaining SAI team members, divides and distributes the SAI activities but is not the supervisor. If the TC encounters difficulties with a team member during an assessment, the situation becomes elevated through his or her Front Line Manager (FLM) for resolution.

Indicates new/changed information.

D.    Prepare to Perform Assigned Data Collection Activities. ASIs prepare for DA data collection by reviewing, at a minimum, the following:

1)    PI or CPM instructions.
2)    Current SAI DCTs, available online, for the applicable element.
3)    Specific regulatory requirements (SRR) related to the element.
4)    Relevant FAA guidance, such as orders and ACs.
5)    Air carrier or applicant policies and procedures (e.g., manuals, operations specifications (OpSpecs), training programs) for the applicable element.
6)    The results of previous DAs and Performance Assessments (PA).

E.    Perform Data Collection Using the SAI. Each SAI team member submits their responses into ATOS automation after they complete their data collection activities. Communication between team members is essential, but sharing answers is not necessary or desirable because of possible duplication. ASIs should complete SAIs within the timeframes the PI or CPM establishes so that data are available for timely completion of the DA. ASIs follow the general instructions for using SAI DCTs.

F.    SAI DCT Sections. SAIs have seven parts: Element Summary Information, Supplemental Information, and five sections for the Procedures Attribute, Controls Attribute, Process Measurement Attribute, Interfaces Attribute, and Management Responsibility and Authority Attribute. These attributes are not standards that must be complied with, but rather serve as a framework we use to assess the capabilities of the process.

Indicates new/changed information.

NOTE:  Prior to conducting an SAI or EPI, ASIs must first review the ATOS Broadcast Message(s) (BCM) as applicable to the DCT. This ensures the ASI considers the most current question information during the assessment. BCMs appear as a pop-up on the ATOS home page until acknowledged. BCMs may also be found on the Flight Standards Information Management System (FSIMS) Web site via a link under the “Publications” tab. The previous quarter’s BCMs remain available via a link under the ATOS “New and Documentation” page.

1)    Element Summary Information: Scope of Element. Scope of element is a consideration of the air carrier’s responsibility and the FAA’s responsibility.
Indicates new/changed information.
a)    Purpose. All elements are air carrier safety-critical processes. The purpose statement defines the desired outcome that is the air carrier’s responsibility.
b)    Objective. The objective statement establishes the FAA’s oversight objective. Before accepting or approving the air carrier’s process design, ASIs collect data that the PI uses to determine if the air carrier will be able to:

1.    Comply with the regulations.

2.    Achieve the desired outcome in the purpose statement.

Indicates new/changed information.
c)    Specific Instructions. Some DCTs may contain specific instructions for additional training, experience, or qualifications that may be helpful in determining ASI assignments. Specific instructions may also include additional references, background information, or manuals to review, as well as suggestions for specific types of activities and/or reporting instructions.
2)    Supplemental Information.
Indicates new/changed information.
a)    Regulatory Requirements. Each SAI includes regulatory requirements as references for the ASI. The regulatory requirements are displayed by the Code of Federal Regulations (CFR) part and section only.
b)    Related CFRs and FAA Policy/Guidance.

1.    Related CFRs. This section is intentionally left blank.

2.    FAA Policy and Guidance. FAA policy and guidance are included for background information that is necessary to accomplish the inspection. Air carriers are not required to comply with FAA policy or guidance. ASIs are required to use the applicable policy and guidance (orders, notices, and ACs) during the evaluation of an air carrier’s program or system to determine if the air carrier met the regulations, including the intent of the regulations. Policy and guidance material may include language that refers to regulations or identifies safety requirements. Policy and guidance material may also describe an acceptable means, but not necessarily the only means, for demonstrating that procedures comply with the applicable regulations and meet the intent and level of safety.

Indicates new/changed information.
3)    SAI Safety Attribute Sections. The key to safety lies in managing the quality of safety-critical processes. This is a primary responsibility of an air carrier in meeting its regulatory obligations. Applicants and air carriers are not expected to use the safety attributes as an outline for their process manuals development. For instance, some mistakenly use the safety attributes as paragraph headings. This practice only complicates the procedures and makes it difficult for the end user, the employee, to gain meaningful information to accomplish work. This is especially serious when used to present safety-critical procedures and tasks. A properly constructed process manual should provide the stated policy or goal related to the procedure or task, followed by clearly written procedures and steps to accomplish the task or activity in a logical manner. ATOS employs six safety attributes to evaluate the design of an air carrier or applicant’s safety critical processes. Each SAI attribute section includes an objective. The following paragraphs describe some of the content in each attribute of the SAI DCT.
a)    Procedures Attribute. The questions in this section of the SAI will help verify that the air carrier documented procedures that identify who, what, when, where, and how those procedures are accomplished. These procedures must allow all personnel to perform their duties and responsibilities with a high degree of safety. Written policies, procedures, or instructions and information that are interrelated and located in different manuals must be consistent and ensure the effective coordination of work activities from one person, workgroup, or organization to another to ensure the desired outcome for the process. Many of these questions have SRRs for this process, although the air carrier may have some latitude in implementing others. For this reason, a response of “No” to one of these questions does not necessarily mean that the company is not complying with a regulation, or that any action is necessary.
Indicates new/changed information.
b)    Controls Attribute. The questions in this section of the SAI will help determine if the air-carrier-designed controls (e.g., checks and restraints) into the processes associated with this element to ensure that it follows policies and procedures to achieve desired results. While most controls are not regulatory, they are an important safety attribute with desirable features that help to reduce unacceptable levels of risk. Each SAI lists a series of controls. Some common types of controls are flags, data system backups, authorized signatures, separation of duties, or a final review. It is important to note that air carriers must be able to show the effectiveness of their controls. Few of these controls have their basis in SRRs. For this reason, a response of “No” to one of these questions does not necessarily mean that the company is not complying with a regulation or that any action is necessary.
Indicates new/changed information. Indicates new/changed information.
c)    Process Measures Attribute. Process measures ensure that the air carrier uses an internal evaluation function to detect, identify, and eliminate or control hazards and the associated risk. For airworthiness elements, this is a required function of the air carrier’s Continuing Analysis and Surveillance System (CASS) required by Title 14 of the Code of Federal Regulations (14 CFR) part 121, § 121.373. The Director of Safety (DOS) and the quality assurance (QA) department often work together to accomplish this function for the air carrier. Negative findings could require amendments to the Safety/Internal Evaluation Program (IEP) or CASS audit forms or checklists. In most cases, process measures are nonregulatory. For this reason, a response of “No” to one of these questions, while not a violation, may indicate a hazard with an increased level of risk, which may require additional CMT or CPT action.
d)    Interfaces Attribute. Data collected in this section helps the PI determine if the air carrier identifies, documents, and manages change between this process and other related processes within the air carrier organization. It is important for the air carrier to identify and document where interactions between processes exists and to have a method of managing change between these processes. Written policies, procedures, or instructions and information that are interrelated and located in different manuals within the air carrier’s manual system must be consistent to allow personnel to perform their duties and responsibilities with a high degree of safety.
Indicates new/changed information.
e)    Management Responsibility and Authority Attribute. Data from questions in this section will help determine if there is an identifiable, qualified, and knowledgeable individual who is responsible for the quality of the entire process. While other employees may be assigned partial responsibility for various parts of the process, the responsible individual is always answerable for the quality of the entire process. Data from this section also helps determine if there is a clearly identifiable, qualified, and knowledgeable individual who has the authority to establish and modify the process. When the authority individual is absent, the authority to establish and modify the process should be delegated to another individual. The intent is to identify the highest level individual (at the appropriate level within the organization) who is responsible for the quality of the entire process and the individual who has the authority to establish and modify the process for that particular element of the air carrier’s system.
4)    Tasks. Each attribute section of the DCT contains the statement, “The ASI must accomplish the following task(s).” Various activities comprise a single task. The following are typical tasks that an SAI includes:
a)    A review of the information listed in the “Element Summary” and the “Supplemental Information” section of this DCT. All questions in the DCT should collect data that supports the FAA Objective to help the PI determine if the air carrier will be able to comply with the regulations and achieve the desired outcome as stated in the purpose statement. The “Supplemental Information” section of the SAI provides a list of the regulatory requirements and FAA policy and guidance documents that are pertinent to the questions of the DCT for a given element. Regulatory and FAA policy and guidance references also appear at the question level.
b)    Review the duties and responsibilities for management and other personnel who accomplish the process associated with this element.
c)    Review the documentation of the processes associated with the element. Review the policies, procedures, instructions, and information to understand the controls associated with this element. This usually involves reviewing sections of the appropriate OpSpecs training programs or other documents, as well as the manuals related to the process.
d)    Review the interfaces associated with the processes for the element.
Indicates new/changed information.
e)    Identify the individual who has overall responsibility for the quality of the process for the element. The ASI must sufficiently understand the air carrier’s system to know who is responsible for the quality of each process.
f)    Identify the individual(s) who has overall authority to establish and revise the procedures associated with the element. The ASI must sufficiently understand the air carrier’s system to know who has the authority to establish or modify each process.
g)    Review the duties and responsibilities of the individual(s) documented in the air carrier’s manual. The ASI must sufficiently understand the air carrier’s system to know the duties and responsibilities of individuals assigned the responsibility for each process or authority to change each process.
h)    Review the appropriate organization chart. The ASI must sufficiently understand the air carrier’s organization to identify who has the authority and responsibility for certain processes. Often, many organizations disperse authority and responsibility.
Indicates new/changed information.
5)    Questions. Each SAI lists a series of questions for the SAI team to answer based on their review and analysis during various activities. Answer questions on each activity report in response to the reviews and analysis on that single activity. The DCTs are not checklists of questions to ask directly of the air carrier’s personnel.
a)    Questions based on regulatory requirements have an SRR appended to them. Therefore, answering “No” to such a question may require an enforcement investigation for a certificated air carrier or may lead to rejecting a program or authorization proposed by an applicant.
b)    Questions that do not have an SRR appended to them are not tied to a literal regulatory requirement but are based on system safety principles. A “No” answer to this type of question, while not a violation, may indicate a hazard with an increased level of risk that may require additional CMT or CPT action, including a decision to withhold approval or acceptance of a system or program.
Indicates new/changed information.
c)    A “D” or “Domestic,” “F” or “Flag,” “S” or “Supplemental,” or any combination of these identifies the scope of the question and can help you determine if the question applies. For instance, if the air carrier is a supplemental air carrier, and there is a “D” or “Domestic” next to the question, this question would not be applicable to a supplemental operation.
6)    Job Task Items (JTI). JTIs are for ASI reference only. JTIs aid the ASI in determining if an air carrier’s written policies, procedures, instructions, and information are adequate. The FAA does not expect the ASI to respond to each JTI individually. The JTIs listed below each question are there to aid ASIs in answering the question. If a question appears too nonspecific, review the associated JTI to identify the specific requirements. Many JTIs contain FAA policy and guidance referenced within the JTI. Due to automation issues, the policy and guidance reference may only appear in FSIMS copies of the JTIs.
Indicates new/changed information.
7)    Notes. Notes supplement SAI and EPI questions. Notes contain clarifying information to help the ASI better understand the question.

G.    Perform Data Collection Using ConDORs. The PI or the CPM may request ASIs to collect DA data using a ConDOR to address focused or unique situations. The ASI performs the appropriate tasks listed on the ConDOR for each inspection to answer accurately all the questions the PI or CPM require.

Indicates new/changed information.

10-173    WERE ISSUES OF REGULATORY NONCOMPLIANCE IDENTIFIED? (see flowchart process step 4.2). During data collection activities for DA, the ASI may identify issues of regulatory noncompliance that require an immediate response.

Indicates new/changed information.

A.    Interpreting “No” Responses. Not all “No” responses to SAI DCT questions indicate regulatory noncompliance. A “No” answer can also mean inadequate inclusion of the safety attributes in the area under evaluation, or that the air carrier’s approved or accepted procedures are inadequate.

Indicates new/changed information.

B.    Regulatory Basis of a “No” Response. A “No” answer does not necessarily equate to an unsafe condition or a regulatory violation, unless that particular “No” has a regulatory basis (including regulatory intent) and the ASI observed a possible violation or unsafe condition.

Indicates new/changed information.

10-174    NOTIFY PI/CPM OF REGULATORY NONCOMPLIANCE (see flowchart process step 4.3). After identifying an issue of regulatory noncompliance that requires an immediate response, the ASI notifies the PI or CPM. The PI follows the guidance outlined in the current edition of FAA Order 2150.3, FAA Compliance and Enforcement Program, to address the issue.

Indicates new/changed information.

10-175    WERE UNRELATED SAFETY ISSUES OBSERVED? (see flowchart process step 4.4). While collecting the required data for the DA, the ASI may observe unrelated safety issues, which are outside the scope of the SAI DCT questions. The ASI takes appropriate action, including communicating the safety issue to air carrier or applicant personnel and the PI or CPM. ATOS does not change an ASI’s responsibility to investigate and act on safety or regulatory concerns.

Indicates new/changed information. Indicates new/changed information.

A.    Address Significant Issues. The ASI should promptly convey to the appropriate PI or CPM significant issues or items of immediate concern.

B.    Address Significant Issues Involving Hazardous Materials (Hazmat). The ASI must promptly convey to the appropriate PI or CPM significant issues or items of immediate concern regarding all aspects of the air transportation of hazmats. The PI or CPM will coordinate with the regional hazmat branch manager.

C.    Address Significant Issues Involving Drugs or Alcohol. Significant issues or items of immediate concern regarding all aspects of the air carrier’s or applicant’s drug testing program and alcohol misuse prevention program must be promptly conveyed to the appropriate PI or CPM, who then coordinates with the Drug Abatement Division (AAM-800).

Indicates new/changed information.

10-176    DOCUMENT UNRELATED SAFETY ISSUES VIA DOR (see flowchart process step 4.5). Use the Dynamic Observation Report (DOR) to document observations associated with safety issues unrelated to your assigned data collection activities. These observations may involve any part 121 air carrier at any location, at any time. DORs are not a substitute for the planned inspections. Managers and ASIs use the DOR in the following situations:

Indicates new/changed information.

·    Safety issues unrelated to the ATOS element under assessment;

·    Safety issues for which there is not an applicable ATOS element or DCT question;

·    Safety issues for an air carrier to which the ASI is not assigned;

·    Specific inspection events as directed by a handbook or other national directives;

·    Safety issues observed by CMT members (assigned to that air carrier) during incidental travel from one location to another to perform official business; and

·    Safety issues observed while conducting a random inspection if there is not a question to address the safety issue.

RESERVED.    Paragraphs 10-177 through 10-190.