Some thoughts on leading measures - with education as an example

Leading measures are things that tell us how we are performing, right now, against things that matter. We are not very good at measuring them. Instead we measure things like percent compliance against a protocol, or satisfaction rates, or exam pass rates. These are all lagging measures - they happen after the event and do not really tell us very much about what we can do to improve. They also don't usually tell us whether what we are doing is actually very helpful/desirable; and as a consequence of this tend to lead to 'gaming'; in which individuals or organisations tend to alter their behaviour in order to hit the target. "Hit the target, miss the point" as my mentor Andy Brogan would say.

The trouble is that the lagging measures are so engrained in all that we do in healthcare that it's very difficult to start thinking differently. So I find it helpful to think about things in which I have only a rudimentary understanding of the process, but I have quite strong views (as a citizen) of what matters. Education is one example, and so here are some thoughts on what matters, and how I might measure these as a school governor. With thanks to @Primary_Ed for the structure around growth mindsets.

1. It matters that my child enjoys school
Ask a child when they turn up for school in the morning if they are looking forward to it.

2. It matters that my child has a 'growth mindset'. 
Note I, personally, am not interested in whether my child has learned any facts - this is what Wikipedia is for. But it does matter that they know how to learn, and they know how to access facts, make sense of them, and use them to solve real problems and be interested in the world around them.

I think I would want every child to show me a balanced example of these things about their work, either in books, or in the classroom:

a. This work is OK - but is it my best work?
I know what I am going to do next to make this work better
I understand what I am doing at the moment and am now practising making sure I can do it well.

b. I have made a mistake - and this is good because I can learn from it
I have made a mistake and I know what I need to do next to learn from it
I find this work hard but I am working hard to understand it

c. This work is awesome - I'm on the right track to being the best that I can.
This is work I didn't think I could do before, and I have worked hard to get here.
I am good at what I am doing now and I am enjoying using my new skills

I really think it is very important we ask children about their attitudes to mistakes : 
I am happy when I make a mistake
I won't be told off if I make a mistake. 
My teacher helps me know what to do next if I make a mistake
I like to help my friends if they make a mistake and I know how to do it.

There are other things in the growth mindset that look at how children approach problems ("This is too hard"; "I can't do French"; "I'll never be as good as her"; "I can't get any better at this"; "I give up") that might be measurable. I am hoping they are, to some extent, captured in the measures above (so for instance, the measure of 'awesomess' is a personal one, and reflects, to me, the extent to which the teacher knows the child and what constitutes challenge and success for them)

3. It matters that my child has enjoys a rich variety of experiences
I'm not sure how I would measure this. How about something like "Number of things my child does that are led by a specialist who is not their usual teacher."

Sex registers and pathology testing

I'm off to London to talk about pathology. On the way I have been listening to the latest Freakeconomics podcast on the economic cost of sex offending. It's big, and lifelong. Society exacts a high price, in a way that is unique amongst all offences. One of the biggest costs is associated with sex registers, which effectively proscribe offenders from even attempting to live a normal life. But this may be the price society exacts for a crime that is seen as most heinous.

What I find interesting, though, is the effect of registers on offending rates. Zero. I don't think this is very surprising. And perhaps it's also not surprising that registers have a wider effect on society. So house prices fall by 4% if you live within half a mile of a registered sex offender in the US.  But then there is the fear they engender. It suddenly becomes a lot easier to imagine your child becoming a victim. And of course it is fear of crime, rather than probability of being a victim of crime, that is the biggest problem, especially for those of us with enough time and resource to be able to search registers.

So the link with pathology? We have created an industry in monitoring of chronic disease which is supposed to make people feel safe, and cared for, but which has actually done the opposite. It has made people live in fear of deviation from the median, and encouraged treatments and health behaviours that lack evidence of actual benefit (as would be defined by the citizen); chasing arbitrary targets that are, at best, a step removed from the true purpose of what we're trying to achieve.

My response to this social disease of iatrogenic harm? I see it as my responsibility, as a pathologist with some degree of influence, to ensure that we act as stewards of pastoral care - only delivering testing when it makes people better. Not using testing to create layers of anxiety. The link here with crime? As a citizen, I see the main role of law enforcement agencies is to make me feel safe. This may involve tackling the causes of crime, but this can only be a part of it. Sex registers most definitely take you away from this purpose - even if they worked (they don't) they make you feel unsafe. The role of health care is similarly not to deliver a long life - it's to make me, as a citizen, feel cared for. We need to start remembering that.

Cognitive dissonance, and turning evidence into behaviour change in antibiotic stewardship

I was walking down a corridor at the hospital today and bumped into one of the urology consultants. He thanked me for some advice I'd given him last week for a patient with a prosthetic valve undergoing a urological procedure. He had thought that prophylaxis wasn't indicated (we had talked about this on a previous occasion, and he knew I'd said quite firmly that it was or wasn't indicated - he just couldn't remember which!). However, his colleague was adamant the patient should be given an antibiotic. So I had reminded him about the NICE guidance which essentially says that there is no evidence for it, and does not recommend it. The trouble with this advice is that it appears rather sterile (if you forgive the pun) in relation to the thought that you might give the patient in front of you a potentially fatal disease (endocarditis). It is especially hard to stop doing something that you have successfully being doing all your working life ie. not giving patients endocarditis, which you have correlated with giving them a pre-procedure antibiotic. So this sets up some cognitive dissonance - your brain is trying to hold two competing views, one in which antibiotics are good (and which chimes with everything you have previously said and done) and one in which antibiotics are bad (which implies that you have been wrong for the last 20 years). It is then a natural reaction to try to dismiss the thing that says you were wrong. We can do this in many ways, from attacking the basis of the evidence, to just forgetting about it. This is why rational cases for behaviour change rarely work. So we need to find ways to address this.

And this is where the conversation became more interesting. He went on to say that the thing that made him remember the 'right' answer, and feel confident that it was OK to withhold antibiotics, was me saying that everyone is bacteraemic whenever they brush their teeth. So if you are going to give antibiotics for this procedure then you should probably consider giving them everytime you brush your teeth. This information allowed him to create an alternative narrative that had equal mental validity with his prior experience. My advice could now sit comfortable in his mental model of the world.

I had a similar experience today when reviewing the latest NICE guidance on the relative risks of different antibiotics for causing C difficile. Not unexpectedly, 3rd generation cephalosporins are high risk. I was pleased to see carbapenems as quite high risk (this is what I wanted to hear). I was also pleased that first generation cephalosporins are low risk, while co-amoxiclav is high risk (we somewhat controversially recommend cefalexin as first line oral option for pyelonephritis in the community). So that was all good. But then trimethoprim came in at high risk. Not what I was expecting, and not what I really wanted to see as it starts to challenge the order of antibiotics we recommend for many infections. My response? I have ignored it. Well, not quite, but I know that I don't believe it. The only way I would believe it is to be taken to the cases of patients who had a clear diagnosis of C diff, and then to be shown the offending trimethoprim prescription. And then for this process to continue until basically my mental model collapsed and I had to rebuild a new truth.

And here we see two powerful ways of tackling dissonance. One is to provide a story which has sufficient power to immediately override prior assumptions. This can be a new piece of evidence, or it could be a single highly emotional patient history. The other way is to take people to the data, and show them it until they accept its validity and their defences collapse. But they have to do this for themselves - meta-analyses will never work.

So in that spirit, I have started adding some statements to some revised UTI guidelines we are writing. This is also bearing in mind the comments of Alison Holmes at the recent ECCMID conference, that we need to talk much more about general principles of stewardship, and worry much less about specifics. The idea is to try to find things that may have the power to immediately shift people's view of the world. This is normative change. Here are some starters - some are general things, some are specific to our local guidelines. I would love to hear some feedback on these, and please feel free to contribute more.

Single doses of antibiotics are often effective

A single dose of any antibiotic is an effective treatment for UTI in most patients. Therapeutic concentrations will be reached for at least 12-24 hours. This is sufficient to achieve cure in over two thirds of patients.

This has been shown for acute cystitis using the following agents:

Amoxicillin     Harbord RB et al (1981) BMJ 283, 1301-2

Septrin             Gossius G (1984) Scand J Infect Dis 16, 373

Quinolones      Saginur R et al. (1992) Arch Intern Med152 1233-7
Tetracycline    Rosenstock J et al (1985) Antimicrob Agents Chemother. 27 652-4

Nitrofurantoin  Gossius G (1984) Curr Ther Res 35, 925-3

Cefaclor          Greenberg RN (1981) Am J Med 71, 841-5

 A single dose of an antibiotic, initiated by the patient when first symptomatic, is effective in recurrent UTI

                                   Wong ES (1985) Ann Intern Med 102, 302-7

UTI symptoms may resolve spontaneously

Up to 50% of women with UTI symptoms recover spontaneously within a week

                                    Mody L (2014) JAMA 311, 844-54

What do others do when trying to diagnose infection in elderly patients with vague symptoms?

A negative dipstick (no nitrites or pyuria) effectively rules out infection in elderly women

                                    Mody L (2014) JAMA 311, 844-54

It is reasonable to observe elderly patients with non-specific symptoms, while correcting hydration and reviewing medication. If, after 24 hours, an infection remains possible, then a dipstick may help in deciding whether to send an MSU, and this result may then guide treatment.

                                    Mody L (2014) JAMA 311, 844-54

 

What is wrong with using dipsticks in catheterised patients?

Dipsticks are completely pointless in catheterised patients.  Even with the best catheter care, 1 in 20 catheters will become colonised with bacteria every day. Long term catheters are almost all colonised with bacteria. Culture is only useful in detecting resistant bacteria, not in making the diagnosis.

Breitenbucher RB (1984) Arch Intern Med144 1585-8

What are the best antibiotics to treat UTI?

 

It is not necessary to cover enterococci in empirical UTI treatment protocols. Cephalosporins and quinolones do not treat enterococci, but are well researched options to treat pyelonephritis as single agents. Gentamicin has been shown to be a good single agent option to treat pyelonephritis

Wie SH (2014) Clin Microb Infect 20, 1211-8

Cefalexin is an effective treatment for UTI in North Devon and is relatively low risk for causing C. difficile. Sensitivity data shows approximately 9 out of 10 UTIs in North Devon. It is not associated with C difficile (unlike 2^nd and 3rd generation cephalosporins, co-amoxiclav or ciprofloxacin)
Clostridium difficile infection: risk with broad-spectrum antibiotics. NICE 2015

What are the downsides of prescribing abx?

Antibiotic resistance is a predictable and very common adverse drug reaction.

Resistant pathogenic bacteria can be detected in one third of patients after antibiotic treatment in hospital.

Gorska et al (2015) ECCMID O001 oral session

Your normal flora might be worth looking after. They may prevent more serious infection :Mice challenged with S. aureus a week before being given influenza are MORE like to survive

Guery et al (2015) ECCMID O001 oral session

Your normal flora might be worth looking after, particularly in children. They may promote a more normal inflammatory response to allergens in the lung, perhaps reducing the risk of asthma. Guery et al (2015) ECCMID O001 oral session

Your normal flora might be worth looking after. Abnormal bacteria in the gut have been associated with many diseases, from obesity to Crohns disease.  Antibiotic treatment induces profound effects in metabolism. Perez-Cobas (2013) Gut 62, 1591

Every dose of antibiotic is a role of the dice - there is a chance that beneficial bacteria may be lost; and harmful bacteria may be gained. The effects of ciprofloxacin are unpredictable, and may sometimes be profound and long lasting, and of unknown physiological relevance Dethlefeson (2011) PNAS108, 4554

How not to harm patients (and control antibiotic use while you're about it)

So I am often asked, as a microbiologist, what I would do if it were me, or my kids, with a sore throat/sinusitis/cough and so on. People seem surprised when I say I would have some paracetamol (maybe with a little whisky and lemon obviously). As though I know that antibiotics are really what's needed, for an individual patient, but that I put public health concerns above this when advising others. Taking this approach, as a doctor I would have to weigh up the benefits of giving an antibiotic (24 hours less symptoms perhaps) but then try to factor in some professional responsibility to 'do the right thing', even if that is in direct conflict with what the patient might want.

But of course a patient wants more than just to get better. They also don't want to be exposed to potential harm. And antibiotic use is not benign. Data presented at ECCMID shows that there are high rates of colonisation with resistant organisms after antibiotic treatment. Clinically relevant? Who knows... But as a patient I might like to know that it might happen. People know that MRSA is not a good thing to have on your skin. And of course there are myriad other risks. From drug reactions to irreversibly altered gut flora (with who knows what consequences).

So looked at this way, over prescribing is not patient centred medicine triumphing over the paternalism of population health. Rather it is a complete failure on behalf of the medical profession to properly consent patients for the things we do to them. So let's start sharing our uncertainties with patients in more honest and open ways.

I was thinking of an analogy. You are driving from A to B. You could do so very fast, breaking the law, increasing the chance of death to you and others; or you could do so more sedately, arriving a bit later, but safely. We might normally do the second, but in some situations, say if time is tight, we might do the first. To make that choice requires 'consent' by considering all consequences, both short and long term, and weighing them up. The answer we get to may not always be rational to others, but perhaps by focusing on rationality we lose the argument? So perhaps the answer to problems of antibiotic stewardship is purely designing ways of sharing risks and benefits with patients and then letting them make the call with us as their impartial advisors.

Designing pathology services through studying demand

So I'm here at ECCMID2015 in lovely Copenhagen. It is clear that the predicted shift from culture to molecular is gaining pace. This will have a major impact on how we organise our laboratories and support clinical medicine. Many people see this as the death knell for local services. I will argue the contrary. For the first time, we have the tools in microbiology (beyond simple microscopy) to deliver results in a time frame that impacts on empirical decisions. In order to work, these tools need to be as near to the decision point as possible, as inevitably transport and other logistical issues will come to dominate the time taken to get a result. We will move from batch processing, to random access or point of care testing. These are things that local services can deliver, but that centralised reference laboratories cannot provide. The irony may be that it is reference labs that will remain the bastion of culture based methods (and perhaps non time critical molecular tests, such as viral load).

So that is all good. But in order to do this we need to change our thinking. We cannot replicate what we do with culture and just do it faster. We need to reconsider the questions we are asking and think about how the results we produce inform action. Just because you can do a test does not mean that you should. We heard today about how highly accurate molecular tests for tuberculosis actually perform poorly in low probability settings. So without considering exactly what test results mean, when applied to a specific setting, we run a real risk of doing more harm than good. And the mystique of molecular makes the risk of blind acceptance of results even greater. I will suggest that we need to go back to studying demand, and by that I mean asking what questions a patient would want to ask, and thinking about how we can answer them. So the question is not 'how can molecular do what I now do faster/better?' Rather it is, 'what clinical decisions can I now support using molecular?'.

And even more importantly, we need to be careful that we don't substitute good clinical assessment for the false security of a test result. Test results can only ever be interpreted in the context of the prior probability that flows from the extraction of clinically relevant information. We need to find ways that help us to resist the siren call of false reassurance, and back our clinical judgement when it is appropriate to do so.

The other concern I have is that small laboratories lack the skills with which to appraise these new technologies. It is easy to be bamboozled by reps promising sensitivities and specificities in excess of 99%, with business cases ready to pull from the shelf. What we need are experts who have a deep understanding of the emerging platforms, with attendant risks and benefits, and perhaps more importantly how they can be fitted together into a coherent package. We need advice on optimum staffing to run these services.

But these are certainly exciting times to be an infection specialist.

Pathology in a system : Wound swabs as an example of how pathology can spot system failures

Pathology in a system: The identification of system failures by Darunee Whiting

Systems theories have been used to describe and help predict how systems behave. It has been used for space projects, financial systems and can also be applied to the healthcare system. Key system concepts are that actors within systems are interdependent but may be disparate. There may be a time lag between actions and effects within systems, so that the effect of an action is not immediately felt and thus the root cause of a problem often goes unrecognised. This can be especially so when actors, actions and effects are in different parts of the system. Systems have feedback processes that can be accelerate growth/decline or rebalance. We see pathology tests as key decision points in a networked system of decision making with the aim of supporting patients, and those helping patients, to make good and timely decisions about their care.

Key decision points trigger actions within the system which can help patients on a path to better health, or in some cases lead them off this path! Tests are placed at key decision points. In primary care, within a clinical consultation there are four key decision points/ actions: these are whether or not to

1. Support the patient
2. Test
3. Prescribe or
4. Refer.

Systems can have patterns of behaviour. To help identify root causes of problems- these patterns of behaviour- system archetypes- have been identified. Our work on wound swabs in lower leg wound care demonstrates one of these archetypes and shows how primary care pathology can identify the root causes of health system failures, as pathology tests/ actions are placed at key decision points in care. Pathology can deliver value here and better health for patients by reducing system failures, through testing decision support and the optimisation of testing.

Lower leg wounds are common. They can become chronic in patients with underlying diseases such as venous or arterial insufficiency, diabetes and anaemia. The majority, approximately 80%, are due to venous insufficiency and require compression dressings to heal in optimal time. The remainder are made up of arterial and mixed arterial/ venous ulcers. It is important to perform a doppler ABPI (ankle brachial pressure index) test on all leg wounds to help decide the likely aetiology of an ulcer and subsequent dressing/ treatment. This is a key decision point. A venous ulcer treated with compression dressings should heal within 3 months. An arterial ulcer would deteriorate with compression dressings which constrict the circulation further. A doppler test can help differentiate between venous and arterial ulcers- and places a patient on the right path to quick healing.

What we did:

1) Identification of high volume/ low value test requesting= Lower leg wound swabs from primary care to the lab. Why was this? A clinical conversation between microbiology and tissue viability suggested that wound care in primary care was not optimal- as suggested by inappropriate tissue viability referrals for patients who hadn't received good primary treatment.

2) Go See: What was happening in primary care: The lab led a collaborative multidisciplinary meeting with microbiology/ vascular surgeon/ tissue viability/ practice nurse/ podiatrist/ GP. It was important to look jointly at the problems with the views of all healthcare professionals involved in the system.

3) Audit of patients with wound swabs/ lower leg wounds in primary care

4) Mapping of key decision points

5) Audit of patients vs mapped decision points

Findings:

Doppler tests not being done routinely or in a timely fashion- due to the lengthy time taken to do the test (30-40 minutes, not enough time was given in nurse timetables for this) This led to patients with venous ulcers not being put into compression dressings for a long time and consequently very very long heal times, and patients with arterial ulcers being referred very late to vascular surgeons for healing interventions This led to more nurse appointments for lower leg wounds that were not healing....which in turn made it more difficult to offer extended nurse appointments for doppler tests... (the system here was accelerating decline) When wounds were not healing, clinicians would often take a wound swab to look for infection. They would find bacteria colonising the wound, not causing infection, and often give unnecessary antibiotics. This demonstrates the 'Quick fix' systems archetype and often led to 'repeated quick fixes- fixes that fail to treat the underlying cause' It is much easier to take a swab, which takes seconds, and believe that this is the right decision, than to take the time to do a doppler (which the system also makes difficult to do in practice)

Here is a systems diagram showing this 'quick fix'

So, the root cause of the problem is in primary care: Wrong action: Doppler tests not being done Consequences of this action are felt throughout the system. Consequences for primary care: more nurse appointments. Consequences for pathology lab: More wounds swabs of low value Consequences for secondary care: Late vascular referrals- how much of this is contributing to high SARS rate for amputations? Consequences for patients: Unnecessary antibiotics, very long heal times/ morbidity Consequences for the health economy: cost

Pathology identified the route cause: a test which was a quick fix (wound swab) used instead of another test (doppler) at a key decision point- the root cause that would fix the whole system failures. The next step was to support primary care decision making and the optimisation of testing. Less wounds swabs- more doppler tests. We showed the key decision points and audit data. It was important to tell the story of how the system was not working and how this impacted on patients. The story for GP practices was that if they prioritised time for a lengthly doppler test, they would actually free up more time later through reduced nurse appointments and quicker heal rates. Addressing the root cause here would also reduce whole system failures- which is the patients story.... which was also important to tell. And the story for pathology? How many other times are tests used as quick fixes that lead to systems failures? Lets Go See!

What matters to patients about diagnostics?

What matters to patients about pathology results? Two talks over recent days have brought this question into sharp focus. First Joan Saddler talked from the perspective of a patient representative. Patients need to know

• why is a test being done (or not done)
• when will the result be available?
• what does the result mean?

So for me we have to think about the consent process (do patients /their carers understand the point of a test? Have they thought about what they will do with a result? Were other actions at this requesting decision point considered/correct?)

In terms of result availability, we need to think about whether necessary information available at time of key decisions. Was this information error free? Was there a predictable flow?

In terms of results there is a lot to do! We need to think about how we convey information in a way that informs decisions. (Was an appropriate action taken as a consequence of receiving the result?) This will include much more thought as to how we define and then convey inherent uncertainties in test results - if these are important when assessing options for action. If there is a 10% variability in creatinine measurement, how do we relay this? How do we deal with variation between laboratories? We cannot transfer uncertainty without considering how patients want this to be done. It will almost certainly be different for different patients.

And today listened to Bertie Squire talk about management of TB in Tanzania. A diagnosis of TB can account for 90% of a person's annual income - and this is with free access to medicine. It is important to get this right. Diagnostics may be expensive - but they are a tiny party of the system cost. This is why this work on delivering value is important. It is actually a matter of life and death.