You are here:
-A A +A

Evaluating Treatments: How Do You Know When a Treatment Really Works?

Date Published: 
August 27, 2007

Deciding on treatment choices for a child with autism can be overwhelming. Many different interventions and techniques are offered, from medications to diets, from vitamins to occupational therapy, from cognitive-behavioral techniques to biofeedback training. It can be difficult to decide what to try, and even harder to tell what is effective. How do you know when a treatment is working?

In this article, we offer some suggestions about what to consider as you evaluate treatments, and relate these back to research principles you can apply at home.

Suggestion #1: Do Your Homework

Learn all you can about the proposed treatment from the most trusted sources possible. Consult with professionals, and talk to other parents. What claims are being made about a certain treatment or intervention, and who is making them? What specific symptoms is the treatment meant to target? What are its possible side-effects? How much time and effort does it require, and are these worth the benefit?

Be sure to ask people how they know what they know. You might ask a physician or therapist, "What clinical trials or other research-based evidence support the use of this treatment?" You might ask a parent who is enthusiastic about a certain intervention, "What changes in your child did you observe after you started this treatment? How long did it take for you to notice the change? Were there any negatives associated with this treatment?" Try not to put too much stock in other parents' enthusiasm if they've only just started something new. It's common for people to be enthusiastic at the beginning of any treatment. What is more important is how they feel about it after some time has elapsed.

As in all areas of modern life, beware of hype. Of course, you must pay something for most treatments, but if claims are being made by people who are trying to sell you something, take that into account.  As a consumer, you have been constantly exposed to people who are working very hard to get you to buy everything from cars to weight loss programs. Use the same instincts you have developed as a consumer in general to make judgments about autism treatments. Know that your desperation may be viewed by some as a business opportunity, proceed with caution, and temper hope with skepticism.

Suggestion #2: Know Your Baseline

Sometimes you've tried so many things in such rapid succession, you're beginning to forget what your child looked like when there were no interventions at all. Make sure you know where you started before you evaluate a new treatment.

For example, to know if a child is really having fewer tantrums, you have to have a good idea of how many tantrums there were before, not to mention how severe they were and how long they lasted. Everyone hopes for an intervention that eliminates a difficult behavior, but it may be helpful to know that an intervention has cut the difficult behavior in half. The same goes for behaviors you are trying to encourage, whether that is instances of joint attention or gains in language. To see if vocabulary is increasing, you have to know how many words your child knew before an intervention began.

Like a researcher, you can track where you started by taking notes and keeping good records. Taking plentiful videos of your child in his or her natural environment --at home, at the park, or at school-- can also help you document what your child's behavior was at a certain point in time, and if it changed during or after an intervention. (See The Importance of Record Keeping.)

Suggestion #3: Start One New Treatment at a Time

Imagine that, anxious to help your child, you start a new diet, a new medication, and a new therapy all during the same week. Your child is suddenly doing better!

That's the good news. The bad news is you don't know which of the interventions is responsible for the change. If anything goes downhill, you won't know which intervention is responsible for that negative effect, either.

It's best to start new treatments one at a time so that you can be clear on both the positive and negative impacts of that particular treatment.

You must also take into account how long any one treatment is supposed to take in order to have an effect. Does a medication have to build up in the system for three weeks before it achieves its desired effect? Then you will need to wait that long to make judgments about whether or not the medication is making a difference. It's best not to start other new interventions in the meantime if that can be avoided.

Suggestion #4: Take Natural Child Development Into Account

Human beings develop through a natural progression: from crawling to walking, from babbling to talking. Even individuals with developmental delays continue to make progress along this innate developmental trajectory.

When an intervention is used to address a developmental concern -- say, to improve speech and language -- it is important to recognize that a child would have continued to make gains, on whatever delayed time table, even without any intervention. The difficult part is determining if they gained more thanks to the intervention than they would have if left on their own.

Because autism is a developmental disorder, it is particularly important to keep this factor in mind when evaluating an autism treatment. Small gains made over a long period of time may be due less to a certain treatment and more to the natural unfolding of human development.

Suggestion #5: Be Aware of "Good Weeks and Bad Weeks"

Children with autism may have good days and bad days, good weeks and bad weeks, no matter what treatments they are on. Cycling through good and bad periods occurs in most diseases, and especially in developmental disorders like autism.

These natural ups-and-downs can make it difficult to discern the true effect of a treatment.  If parents start their child on a new treatment when they are most desperate (that is, when their child's behavior is at its worst), there will be a tendency to see the child swing back towards improvement regardless of the treatment being tried because the child was about to cycle back "up" anyway.

The trick is to separate out how much improvement is due to this natural cycling towards a "good week" or a "good month" and how much improvement, if any, is actually due to the intervention.

If a parent looks back and realizes she has tried a series of different treatments that "worked for awhile and then wore off," it may be that she was experiencing the effects of this up-and-down cycle. A child started on an intervention at the worst point of a difficult period may have cycled back towards a better period of behavior, and so the parent believed that the intervention was working. As the child continued through the cycle to head back down into a period of increasingly difficult behavior, the treatment seemed to stop working, and the disappointed parent, again at the worst point in the cycle, decided to try something else, which also seemed to work, but only for awhile.

Over time, parents do develop a sense of their child's pattern of  "good weeks and bad weeks."  It's important to keep that pattern of ups and downs in mind when you assess any new intervention.

Suggestion #6: Keep Other "Time Effects" in Mind

Time of day, month, or year can impact how a person is doing, and so can warp measures of treatment effectiveness. The dark days of winter can lead some people to feel much more depressed or unable to cope, as can the misery of springtime allergies. Irritability associated with a woman's menstrual cycle may impact the effectiveness of a treatment given at a certain time. Many children with behavior problems or issues of mood and attention may find the beginning of school, or a return to school after winter holidays, stressful.

A new treatment started during a child's bleak winter mood and right after the holiday break might not appear as effective as the same treatment started during the summer. Timing of an intervention can interfere with our assessment of true treatment effects.

Suggestion #7: Be Observant About Other Factors that Impact Treatment Outcome

There are countless other factors that interfere with the measurement of the true effectiveness of an intervention. For instance:

  • A child may like or dislike the teacher, doctor, or therapist providing an intervention. Similarly, their parents may like or dislike a care provider, and their attitude may impact the child's cooperation and outcome. Imagine a program that seems to help when Suzie, a sweet, empathetic OT, is the therapist, but doesn't work at all when Liz, a gruff, irritable OT, performs the same therapeutic steps.
  • Starting a new treatment may increase parents'  hope and decrease anxiety, such that a mother and father get along better and deal with their child more consistently. The child may start having fewer tantrums, but this may be due to the parents' change in behavior, not to the treatment itself.
  • Who gives information on a survey or checklist often provides a very different picture of a child’s status. Perspectives of mothers, fathers, and teachers may differ substantially from one another.

All these factors have the potential to mask the true effectiveness of a treatment or intervention.

So... How Do You Know What Works?

First of all, your awareness of all the factors listed above can be helpful. If your child tends to have two good weeks followed by two bad weeks, for example, you know not to celebrate the success of a new treatment until he's had four "better than the worst" weeks in a row. You may know that a certain time of year is always a low point, and so may wait to judge a new treatment's full effectiveness until that period is over. You may realize that increasing maturity may be a result of developmental gains, and not necessarily due to a current treatment. You may recognize that sometimes the person providing the treatment is just as important as the treatment itself, and look for a good match in teachers and therapists.

In addition, to some extent, you can do what researchers do to determine the true effectiveness of a treatment. You can try implementing some "research methods" at home.

Research at Home

The dilemmas a parent faces when trying to evaluate a treatment's effectiveness are actually very similar to those researchers face. Researchers, too, want to make sure that a treatment they are testing is actually doing something, that it doesn't just seem to do something because of other, unrelated factors. Working with your physician, you can apply principles that guide researchers when you are evaluating a new treatment.

  • Researchers often design a study so that there are at least two groups --one receiving the treatment and one receiving either no treatment; a placebo; or a different treatment-- for comparison. They try to ensure that assignment to these groups is random to avoid bias.

    Parents can't set up different groups getting different interventions, but they can observe their child on and off a certain treatment.
  • Researchers may keep a study double-blind so that neither patients nor researchers know which of two groups is receiving a certain treatment or a placebo. This eliminates bias on the part of researcher and patient alike.

    Parents can ask a person who doesn't know when a child is on or off a treatment being tested to evaluate how the child is doing. A teacher, for example, who doesn't know whether a child is on or off a new medication, vitamin regimen, or special diet might fill out behavior scales at regular intervals. In effect, the teacher is "blinded" to the test. If she rates the child as much better during a period the therapy is "on," parents can have some confidence that the therapy is having a helpful effect. (What the interval is will depend on the treatment, how long it takes to have an effect, and how long that effect is expected to linger once treatment stops.)
  • Researchers may have a control group which receives a placebo instead of the intervention being tested so that they can separate out the placebo effect --the effect of hope, faith, and positive expectations alone on a person's well being.

    Sometimes, parents can use a placebo, too. If you are trying to evaluate a new medication, for example, you could work with your physician to put your child on the med some weeks, and on a placebo (such as a sugar pill), other weeks. If your child is old enough to have his own hopes and expectations for the medication's effects, you will be separating out the placebo effect from his own experience of the medication.

    In addition, by using a placebo, you might even keep one parent "blind" to when the child was on or off the real medication, and ask him or her to rate how the child is doing.

Knowing What Works: Becoming a Better Advocate

Families know their children better than anybody else. They know their strengths and their challenges. They know their "up-and-down" cycles. They know how they behave differently with different people or at different times of year. This parental insight, combined with an awareness of the pitfalls of treatment evaluation outlined here, will make you the best judge of how a new treatment is impacting your child. Such awareness will also better equip you to evaluate treatment outcome research and the claims of those marketing autism treatments.

Sharing What You Learn: Participating in IAN

The Interactive Autism Network (IAN) is collecting treatment data over the internet from parents of children with Autism Spectrum Disorders (ASDs). By participating in IAN, you can share your hard won experience regarding any treatment you have tried with researchers all over the world. As thousands of families contribute such data, researchers will be able to track what treatments are being used, and which ones families are finding helpful. There is so little useful outcome research on treatments in autism. By participating in IAN, you can change this.

To participate in IAN Research, register today at www.IANresearch.org.

If you have any questions, please do not hesitate to contact the IAN team at researchteam@ianproject.org.

Please rate the helpfulness of this article: 
Average: 5 (3 votes)