When are numbers bad?

Two years ago, I wrote a post about SMART goals, wondering whether society had developed a predilection to dismiss any goals or efforts that are not measurable.  I defended such immeasurable goals as being perfectly valid.  Then last week, a participant in our chapter-by-chapter exploration of The Value Crisis, asked if we should actually avoid SMART goals because they were number-based.  This is a common musing among readers: Is the book saying that numbers are bad?

The question highlights a distinction that cannot be over-emphasized:

Numbers are great.  It is our reliance on Number-Based Values that I question.

To answer the query that was raised at the meeting, there is nothing wrong with SMART goals.  If you can define a goal numerically, then it makes total sense to measure (and celebrate) your progress towards that goal.  If you choose to save $5,000 for a two-month vacation next year, I don't find any fault with that particular example of value-based decision-making.  Such a goal is not a demonstration of number-based values.  Why not?  Because we have not defined a situation in which more is always worth more.

On the contrary, the ultimate goal is quite specific: taking a two-month vacation.  The money is simply a means to an end.  Furthermore, the very nature of a properly formulated SMART goal is that it should incorporate a specific target, which can (and should) be interpreted as a definition of sufficiency.

Contrast this with a SMART goal that sets an objective of saving $10,000 more every year.  Now we are beginning to cross over the line.  A goal phrased in this way has milestones but no specific endpoint.  The money is no longer the means to an end - it has become the end in itself.  And we have made the tacit assumption that more money is always worth more.  That's true in a monetary (number-based) value system, but is it true when it comes to our quality of life?  Well, that's the $64,000 question, isn't it?

This is not an easy distinction to make.  As the discussion progressed, another participant asked if it was therefore more acceptable to set a specific dollar goal for money to be saved for your retirement (say $1,000,000) as opposed to setting a goal of a specific level of annual growth for your retirement investments (like 4%, for example).  Yes, there are probably subtle differences between the two goals, but I prefer to look at the bigger picture.  Is your overall goal really to have a certain amount of money when you retire, or should you be trying to define a certain quality of life that you hope to be enjoying when that day comes?

Better yet, perhaps you should be looking at the overall concepts of work and retirement.  That inquiry might involve examining your quality of life before your 65th birthday.  How many of us devote decades of our lives struggling at jobs that we consider onerous, with the major aim of better enjoying life when (and if) we reach a retirement age?  How many youth make life-altering career decisions solely on the basis of how much money they might make for the next four or five decades?

In The Value Crisis, I tell the story of how I observed folks in the previous generation 'retiring' but still working, and I asked them to define what retirement really meant for them.  It meant that they didn't have to work the same hours, but could choose to.  They stopped doing work that didn't interest them, and created understandings where they could take time off to enjoy a new project if an opportunity came up.  They avoided long-term work commitments and gave greater respect to their lives away from work  Why would anyone wait until they were in their 60's to take that approach to life?

So, working from that definition, I declared myself 'retired' before I was 40.  It doesn't mean I don't have to earn money any more - of course I do.  But it gave me a whole new outlook on how much I need (or don't), and what I am willing to do to get it.  My standard of living has gone down since then, but my quality of life has gone up.  A smart goal?  Well, for the most part, it's worked for me.

SMART Goals and Anecdotal Evidence

A reader who works as a teacher wrote to me the other day:

I thought of your book once again this week, in an all day meeting of our school leadership team.  We spent the morning confirming that there is much more to educating young people than just generating marks, and that just focusing on numeric measures was the wrong way to go.  And then we spent the afternoon talking about nothing but numbers.  Stupid numbers.  Like the TDSB Director's vision to decrease the number of students needing Special Ed support by 50%.  (Maybe they just need extra vitamins.)  And stupid comparisons of unrelated data, like the achievement of one group of 14-year-olds vs the achievement of an entirely different group of 14-year-olds a year later.  Actually understanding statistics has become a curse!

The Toronto District School Board (TDSB) comment got me curious, so I looked on the web and found their Years of Action: 2013-2017 report.  This document promises 36 separate actions, resulting in 102 different outcomes - most of which have numeric measurements attached.  Wow.  102 targets that the TDSB can be directly scored on.

Such targets are often referred to as SMART goals - goals that are (S)pecific, (M)easurable, (A)cheivable, (R)elevant, and (T)imelined.  Since The Value Crisis talks about how that which cannot be measured is dismissed in this age of scientific thinking, I started to wonder whether that also means that unmeasurable goals therefore 'lack validity' in modern society.

I'm guessing there are three reasons for making a goal measurable.  The first is so that you can track progress and see if you are doing the right things.  The second is so that you will know when you have achieved your goal and can celebrate accordingly.  The third (more cynical reason) is for promises made in the public sphere, so that you can prove to others that, despite appearances, your target has been reached, according to objective measure.  (This might be where classic manipulations of statistics come into play.)

But what if your goal can't be measured so easily?  What if your aim is to be a better parent or to be happier?  The experts would say that you need some way of measuring this.  I say that's nonsense.  If you didn't need a measurement to identify the need for the goal in the first place, you don't need a measurement to know if you are experiencing success.

(I tried to come up with a new M-word for qualitative SMART goals.  So far, the best that I've come up with is "Motivational":  If you are motivated to continue what you're doing, you are likely moving towards the goal that you desired in the first place.  Please make better suggestions!)

So if you're not measuring your progress by some quantifying means, what can you use for feedback on your goal?  Well, you might go with anecdotal evidence.  <Insert dramatic organ chord.>  Anecdotal evidence!  The mere use of the term immediately raises the hackles on statisticians' necks.  Well, I'm here as the Devil's Advocate to suggest that anecdotal evidence still has plenty of value when drawing general conclusions.  On the other hand, the scientific method, when applied inappropriately, can actually the lower value of information.

I'll illustrate this with an old story (an anecdote!) of three professors traveling on a train through a foreign land.  The historian, looking out the window and seeing a black sheep grazing alone on the hillside, says: "How interesting!  The sheep of this land are black."  The statistician looks out and says: "Well, to be more correct, what we know is that in this part of the country there is one sheep that is black."  The philosopher, without looking up from his book, immediately corrects him: "Actually all we can say for certain is that in this specific area there is a sheep which is black on one side."

Had I been in the next seat, I might have added my own quip: "Excuse me, gentlemen, but I believe the only conclusion you can draw is that, looking out the window of this train, an object was observed that looked like what you know of as a sheep, whose wool (at least what you could see) appeared to be very dark in colour."  The historian's statement could clearly be misleading.  The statistician tried to make a more accurate (and thus, presumably, more valuable) statement.  However, further refinements reduced what began as a simple observation to a meaningless statement, of no practical value to anyone.  So where does anecdotal evidence cross the line from being a scientific menace to being meaningful and useful information?

As a scientist, I am fully aware of the abuse heaped upon anecdotal evidence - and rightly so, when it is used to 'prove' things that can only be proved by more rigorous analysis.  On the flip side, one might argue that almost everything we know, as individuals, comes from anecdotal evidence.  We know that last year's vacation was enjoyable or that last night's movie was boring because that was how we experienced them.  We know that flying insects with yellow-and-black-striped bodies can inflict pain, either through personal history or from the anecdotes of others.  That might not be 100% true every time, but it is certainly very useful information.  Indeed, at one time, the entire store of mankind's knowledge was anecdotal and communicated through story-telling.  You may say that it was naive and imperfect, but if I were stranded in the wilderness, I would take the knowledge of our Stone Age ancestors over that of your average physicist any day of the week.

The very things that make anecdotal evidence suspect - such as our innate propensity to focus on exceptional events (confirmation bias) can have their own value.  For example, one adult, telling a child that they are clever enough to do well in school, can alter that child's self-esteem, willingness to work hard, and ultimately improve their academic performance, even if many other adults are providing only discouragement.  (I heard about that happening once, so it must be true. ;-)  Even medical researchers have to concede the validity of the placebo effect, in which measurable outcomes can be altered by our own beliefs in anecdotal evidence.

So where am I going with all this?

Our brains are wired to make use of anecdotal evidence, and while we sometimes get it wrong, I'll wager that it is (more often than not) useful and valuable information.  A single anecdote, such as being out of breath after climbing some stairs, can suggest to us that perhaps we are out of shape and need to exercise more.  If a few more pieces of breathless anecdotal evidence confirm this, we might set a goal to improve our physical fitness.  If we start being more active because of this and find that we can now take the same set of stairs two steps at a time with ease, then we there is no need for science to measure whether or not we have improved our muscle tone or cardiac output.  It is sufficient and rewarding enough for us to feel we have succeeded in what we set out to do and be happier for the effort.

Similarly, in life, if it is our goal to be happier (and who wouldn't want that?), then we don't need measurable 'SMART' goals to tell us if we are doing the right things.  As for my teacher friend, while you might want quantifiable studies to alter policies for the entire board, that does not change the fact that in a specific classroom, for a specific child, anecdotal evidence is the best proof available that what you are doing is working, or not.  So why does number-based policy so often trump anecdotal evidence for individual circumstances?

P.S.  If classroom marks were the truest indicator of value and potential, I wouldn't be here.