Conducting even a small clinical audit and carrying out a plan of improvement needs resources and the commitment of all participants (management, clinical teams and audit department) to complete the process. Audits that are restricted to a few patients in a single clinical unit and using minimal resources may be valuable in improving practice. Large, multi centre studies require substantial planning and resources. Such large studies may be needed to identify, on a multi centre or national scale, current practices or areas needing improvement.

Research creates new knowledge about best practice that should be used to improve guidelines. Clinical audit examines actual practice, compares it with guidelines, and tests compliance with them (example downloadable below).

Even the best guidelines or SOP’s are only useful if they are followed. Audit is the way to test compliance. Clinical Audit should be part of a continuous improvement process or quality improvement cycle consisting of the following steps:

Choose the target

Plan to audit a topic that is clinically important, with evidence of room for improvement such as errors, adverse events or reactions, large variations in practice or patient complaints (downloadable below)

Define the aim

There should be a clear ‘audit question’ (or questions), just as any research proposal should start with a concise statement of the research question or hypothesis to be tested (downloadable below).

Select the criteria

Local criteria (developed by the hospital’s clinicians) should be used as the basis for audit. Such locally developed guidelines should be based on current national recommendations. These take account of the best available evidence.

Define the methods

Decide what is to be observed or measured, how the data will be collected, quality controlled, analysed, and presented.

Implement an action plan for improvement

Decide what will be done to improve practice if the audit shows that improvement is needed, plan and implement (downloadable below).

Repeat the audit

Test for evidence that practice has improved.

Use an existing design

It may be possible to save time and work by using a pre-existing set of audit questions and tools, modified if necessary. This also makes it easier to compare results between institutions. Even if an existing design is used, a small pilot study should still be performed.

Design a new audit

Build the frame of reference, which should be based on the most recent, available, relevant documentation, including:

• Regulations: European directives, national laws and decrees
• Professional documentation: Clinical guidelines, consensus conferences, scientific literature, expert opinion.
• International (ISO, EN), national or professional standards

Select criteria

A critical step is to reach agreement on the criteria against which practice is to be audited. Although guidelines for many aspects of transfusion process exist, they are frequently based on inadequate evidence. There may be local barriers – such as individual clinicians’ opinions – to general acceptance of a guideline. The process of negotiating an agreed and measurable objective standard of care that can be endorsed by all stakeholders and used in the audit process can be an extremely valuable means of encouraging clinicians to review their practice (R29 document attached below).

Prospective or retrospective?

Prospective audit is based on the collection of information about patients during their process of care. It permits more reliable and complete clinical data collection since the data required is pre-defined and can be validated and errors corrected while the data collection is in progress. A possible disadvantage is that practice could be altered if staff are aware that they are being observed and data is being collected.

Retrospective audit is generally based on review of records of discharged patients. This may provide information that is more representative of day-to-day practice, but it is more difficult to obtain complete data on every subject in the sample. Retrospective audit may make use of computer databases provide the data they contain is of adequate quality.

Develop the audit criteria

A criterion is a principle or standard by which something may be judged. Audit criteria should describe the aspect of care that is being measured. Explicit means SMART:

• Specific: Unambiguous. Relates to a specific area of care and states specific boundaries
• Measurable: There must be objectively measurable aspects to allow comparison
• Achievable: Must be achievable either within available resources, availability of cases, etc.
• Research based: Wherever possible, there should be sound research evidence that shows the best available treatment or method for the aspect of care being audited
• Timely: Criteria should reflect current practice

Each criterion included in the frame of reference should ideally be derived from the reference guidelines or regulations. It should be designed to identify clearly whether practice conforms or does not conform to the guideline or regulation.

Write the protocol

The protocol should include:

Objectives
• Sources of information used to build the frame of reference
• Definition of the audited hospitals, teams or individuals
• Criteria for inclusion and exclusion (when patients’ files are concerned)
• Type of study (prospective, retrospective)
• Type of data collection (observation by external auditor, self-evaluation, interviews, data collection from patients’ files, from hospital blood bank and from blood transfusion establishment.)
• Description of the role of each person involved.
• Any requirements related to consent, confidentiality or ethical issues

Develop or adapt tools for data collection

Data collection must be simple to perform, valid and reliable.
Data collection forms (often called Case Report Forms or CRF) must have a unique identifier, clearly identify the person who completes the document, and show the date of completion. Questions should require a simple unambiguous response (e.g. YES / NO/ Information not available). Responses such as ”not applicable” should not be permitted. The units in which quantitative results are recorded must be stated e.g. “Hb concentration to be recorded as g/l”. Clinical terms must be defined explicitly (e.g. “bleeding”, “cardiovascular disease”). Questions that require free text response should be avoided or strictly limited as they are difficult to analyse (R24 document attached below).

User manual

For a simple audit the user manual should be short and simple, explain exactly how data collectors should obtain the data items for the audit, how they should enter the information in the case report form or equivalent, and how completed CRFs should be submitted to the audit team.

Statistical support

It is strongly advised to engage the help of a statistician right from the planning stage and throughout the audit to provide expert advice on design of the study including aspects such as sample size, power, sampling methods, development of the analysis plan, conduct of the analysis and presentation of findings. If the results of the audit are to change practice, the results and analysis must be robust and also simple enough for everyone in the care process to understand.

The statistician should generally be a co-author in any report prepared for publication (R23 document attached below).

Sample size

Where a research study will need large numbers of subjects to show which intervention is best, clinical audit only needs to determine the extent to which practice complies with standards or criteria. Smaller sample sizes can often provide the information. The information collected from the audit sample should be representative, i.e. should allow 95% confidence that the results will be within 5% of the results that would be obtained from the relevant population. Sample size calculators are useful when determining an appropriate sample size (R31 document attached below).

Sampling

There are several methods of choosing which cases to include in an audit, including:

Random Sampling: Assumes your audit population will remain the same throughout the audit period and that each subject will have an equal chance of being chosen, either by drawing names out of a hat or choosing every nth subject from a list (e.g. choosing every 3rd or 5th patient).

Interval Sampling: Assumes your audit population will change over the period of the audit. In these circumstances, the audit sample is often determined by a period of time e.g. all patients transfused during May and June.

Stratified Sampling: Is a method used to ensure that the proportions of different groups in the population are reflected in the sample. For example, if investigating donor deferrals, and if male blood donors make up 40% of the donor population, you would ensure that 40% of your sample are male.

Rapid Cycle Sampling: Small data sets are audited to improve and monitor care. This approach can make the change cycle quicker and is useful if a problem is suspected and results are needed quickly. Auditing a small sample can show the nature of the problem. After implementing the action plan for improvement, a repeat the audit on another small sample can quickly show if improvement has been achieved.

Because a poorly chosen sample can skew results and give inaccurate information, the advice is repeated to seek advice from statistician or audit department (R31 document attached below).

Pilot testing

The data collection process should always be pilot tested before the full implementation of the audit. This will often lead to improvements in the data collection forms and improve the final result.

The audit team is responsible for:

• Informing all personnel involved in the audit
• Ensuring there is a clear agreement about roles, responsibilities, and authorship of the final report and any publications that result from the audit
• Training data collectors
• Providing the tools for data collection, e.g. paper case report forms with subsequent input of data to computer, PDA or other portable device
• Quality control of the collected data and of the data input process
• Ensuring anonymity of patients, audited staff or institutions when required by protocol
• Being available to respond quickly to questions and problems that arise during the audit
• Ensuring deadlines are met

(R24 downloadable below)

The type of analysis depends on the type of information collected. Quantitative data is concerned with numerical or specific data. E.g. Yes/No, Age, Gender, Blood Pressure, Blood Groups. The analysis of this type of data is performed using simple mathematical techniques. Qualitative data is usually descriptive rather than numerical. E.g. comments on questionnaires, or donor complaints. This data needs to be analyzed differently using specialized techniques (R23 downloadable below)

.

Each team audited must have the opportunity to participate in the analysis and to study and comment on the results, express their opinions on the audit, identify causes for non-compliance and propose improvement actions. The team leader should generously acknowledge the contribution of all participants. At this stage, the team leader should be ready to provide a final validated report at the institution level. If a report is to be submitted for journal publication the active contributors should be properly recognised (R28 downloadable below).

Tables and graphs to present the results should be as simple as possible. Presentation should focus on the quality and completeness of participation and compliance with audit instructions (rate of non evaluable answers, etc.), identification of the major positive points and of the major points of nonconformity that will require improvement. It is important, at this stage, to propose a preliminary root cause analysis, in order to stimulate discussion among the participants; for each nonconforming item:

• Identify the nature of the problem
• Identify the possible causes for non-compliance
• Propose a classification to help build an action plan
• Propose an improvement action plan for consultation and agreement

(R28 downloadable below)

Search for improvement

Analysis of the audit results should define the improvements that can be proposed to the audited teams and to management. The action plan must define the objectives and the approaches to be used.

Examples

The plan may aim to improve deficiencies in the design of process or the resources that are revealed by the audit. This might involve developing or updating an SOP that is (Table 9.1) missing or out of date or correcting deficiencies in resources or training.

Table 9.1 Responding to audit findings. Examples: deficiencies in processes or resources

Responding to audit findings – deficiencies in processes or resources
Criterion	National and local guideline requires that hospitals have a validated procedure for provision of blood to patients in an emergency
Audit finding	Hospital does not have a major haemorrhage procedure
Corrective action	Take steps to ensure that staff who provide this service are supported by written procedures, effective training, and appropriate practises (fire drills) to test the procedures periodically
Criterion	Staff must receive appropriate training for their task(s)
Audit finding	Audits may uncover deficiencies in education and training in any area of practice
Corrective action	Develop and implement training programme
Criterion	National and local guideline requires perioperative monitoring of patients’ haemoglobin levels
Audit finding	Equipment for “near patient” measurement of haemoglobin concentration is not available
Corrective action	Operating departments must be supplied with suitable equipment

 

Alternatively the audit may show that there are non-compliances even though all the appropriate procedures, personnel, training equipment, etc. are in place. (Table 9.2)

Table 9.2 Responding to audit findings. Examples: non-compliance

Responding to audit findings – non-compliance
Criterion	Guidelines require that patient records contain a record of the clinician’s reason for prescribing each red cell transfusion.
Audit finding	A doctor’s record of the reason for transfusing is found in only 20% of patients’ files.
Corrective action	Obtain the agreement of clinical staff to achieve a target of 90% documentation of the reason to transfuse and to participate in education on the importance of clinical accountability for transfusion and to a repeat audit.
Criterion	Guidelines require that all patients undergoing transfusion have observations of pulse, blood pressure, respiration and temperature recorded before and at specified time intervals during the transfusion.
Audit finding	These “routine observations” are performed incompletely or not at all in a substantial proportion of transfusion episodes.
Corrective action	Obtain the agreement of clinical staff to achieve a target of 90% documentation of patient observations according to the guidelines and to a repeat audit. To overcome the problem that nursing staff believe that they do not have time to perform the task, consider action such as: Review the priorities among nursing duties to make more nursing resource available; or A planned change to the Guideline. The clinical responsibility for the decision should be clearly defined, and the Hospital Transfusion Committee should decide if notification to other authority(ies) is required
Criterion	EU Directive requires that the final fate of all blood components issued for recipients is recorded by the hospital blood bank.
Audit finding	Hospital blood bank does not have data on the final fate of all components
Corrective action	Obtain the agreement of clinical staff to achieve an initial target of 98%. Inform staff that monthly reports will be provided to senior nursing managers, identifying the clinical areas not meeting the agreed target. These mangers will be required to identify how non-conformance will be addressed.

Practical proposals for corrective measures must be made to the hospital management. These must be endorsed by all participants in the audit. The presentation should identify both positive and negative findings of the audit, since sharing examples of good practice may contribute as much to improving quality as does the identification of inadequate practice. The improvements expected from carrying out the action plan should be described as accurately as possible. The improvement action plan should be finalised and approved by the project team, the audited team and the hospital management during or shortly after this presentation.

The Audit report

The final report should present the overall project and should include the following sections:

• Objectives
• Participants (Project team, Audited teams)
• Organisation and methodology
• Frame of reference and timeline from planning to report
• Both positive findings and issues requiring improvement
• Agreed action plan for improvement
• Annexes (documents used, e.g. audit protocol, user manual, reference sources)

Publication in the professional literature may add greatly to the value of the audit, for the participants, the hospital and the wider professional community.

Practical examples of audit tools and reports can be found in the resources listing and in the file list below.