How To Evaluate A Medical Study

I still maintain that the best college course I ever took was psychological statistics. During the course we learned how to create and read psychological studies, including things like what a population is and how big a population should be before the results can be extrapolated to the affected population at large.

We also learned why double-blind  studies were important. You can read more about that type of study and why they are relevant study types here.

Let’s start with what a “population” is. In the layman’s world, a population refers to how many people live in a town, country or even the planet. In a study, the population refers to the number of people participating in the study. Say you have a study testing the effectiveness of a new anxiety medication and you have 100 people in the control group, 100 people in the placebo group and 100 people receiving the actual medication. In this case the study population is 300.

As study populations go, that’s still a relatively small size if we’re going to extrapolate the data to nearly 30 million people in the US alone that suffer from some level of anxiety that requires medication. A decent study size would be at least 1000. Even then, one study is NEVER sufficient. When evaluating research, you need to consider the ENTIRE body of research, which should include 4 or 5 good studies with a large population that has been conducted with controls in place.

Let’s take a look at someone practicing quack medicine. Dr. Steven Novella of Skeptic’s Guide to the Universe and NeuroLogica fame recently posted an article that included information about the Burzynski Clinic and the doctor who is touting a “unique cancer cure” that has no evidence to back up its effectiveness beyond anecdotes from patients.

Dr. Novella points out that Dr. Burzynski is still doing “preliminary research” after 30 years. This is a red flag. After 30 years of testing, he should have been out of preliminaries a long time ago IF his treatment had any positive effect.

Next, we’ll take a look at the study of Natural Frequency Technology on Philip Stein watches. The Philip Stein website makes the claim that their Natural Frequency technology, “is a combination of natural frequencies generated by Earth. The technology, embedded in all Philip Stein watches is delivered via a metal disk that has been infused with key frequencies in a proprietary process. When worn on the wrist, the watch exposes frequencies and information to the biofield of the person wearing it. The biofield is the master energy field that regulates the body’s functions and informs the body to relax.”

As if that wasn’t enough to send you running for the hills, they also claim that the Journal of Sleep Diagnosis and Therapy has released a study claiming that “study participants who wore Philip Stein watches fell asleep faster, stayed asleep longer and felt more refreshed in the morning.”.

The website goes on to say:

Subjects in the study were both males and females who were poor sleepers and reported un-refreshing sleep for at least the past three months. None of the subjects reported having a serious medical or psychological condition. Each of the participants was given a device, half which contained the Natural Frequency Technology found in Philip Stein products, half which did not. In all, 96 percent of subjects tested responded positively to at least one of the Natural Frequency Technology variables:

  • 64 % reported waking up more refreshed
  • 61 % reported having more pleasant dreams
  • 43 % reported falling asleep faster
  • 43 % reported sleeping longer
  • 18 % reported lying awake less at night

It sounds impressive, doesn’t it? Well let’s just take a look at the study, itself, shall we? Notice, if you will, under “Methods”, the population of the study is a grand total of 28 people.

So let’s break down these percentages and see what the numbers tell us, shall we?

96% of the study group reported positive results. So out of their total study population, the percentages apply to 26.88 people. Let’s be generous and round it up to 27.

64% of study participants reported waking up feeling refreshed. Of the study population, that’s a TOTAL of 17 (leveled from 17.28) people reporting this result.

61% reported having more pleasant dreams- a total of 16 (16.47) people

43% fell asleep faster – that’s 12 (rounded up from 11.61) out of 27.

43% reported staying asleep longer- Is that the SAME 12 people?

18% reported lying awake less at night- a grand total of 5 (rounded up from 4.86) people.

Conclusion: This study is unreliable at best. The population is far to small to draw any reasonable conclusion about the effect this product claims. There is also only this one study which, as we know, is not nearly enough data to come to the conclusion that is asserted on the Philip Stein website.

If you read the study conclusion itself, the researcher even asserts, “…the current overall results are not statistically significant…”. On the website, however, “A just-published study in Sleep Diagnosis and Therapy Journal (2010; V5 N5, 27-29 Breus and Rubik) provides the first clinical evidence that Philip Stein watches featuring Natural Frequency Technology provide improved sleep.

Uhm, no. A study this small is not evidence. Evidence is testable and repeatable. They have not repeated positive results on any statistically significant level whatsoever. Without evidence, they are just blowing hot air and trying to sell a product that doesn’t do what it claims on that level.

In short, it’s bunk.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: