DOE&ProcessControl

Many respected scholars have suggested that process control is necessary (e.g., Deming, Ryan, and Wheeler) whereas other equally respected scholars have stated just the opposite (e.g., Fisher, Box, Bisgaard, Vining, and Montgomery). Bisgaard's article in Quality Engineering [Bisgaard, 2008] entitled "Must a Process be in Statistical Control before Conducting Designed Experiments?" argues that R.A. Fisher settled the issue in 1925 when he published a paper showing that DOE is able to extract meaningful inferences from systems that were not in a state of statistical control.

So if the issue was settled more than seventy-five years ago why is it still an area of confusion today? Did we forget everything we were taught? Well, it seems that other experts claimed that control is necessary. In fact, Deming states:

Students are not warned in classes nor books that for analytic purposes (such as to improve a process), distributions and calculations of mean, mode, standard deviation, chi-square, t-test, etc. serve no useful purpose for the improvement of the process unless the data were produced in a state of statistical control. (p. 312)

So which position is correct? Could they both be correct? Let me suggest that both can be correct to varying degrees depending on the nature of the instability. However, my first observation is that this entire discussion seems have people falling for the classical semantic logic "Black or White" fallacy. This binary logical thinking precludes the possibility that there is a middle ground and prevents the analyst from seeing it.

Dr. Vining’s analysis sheds a good deal of light on the issue because he shows that randomization transmits the special cause effects to the error term. Now this is true, if the special cause generates a stable sub-process (i.e., like a step function, or a plot in an agricultural experiment that is stable), or a dynamic but modelable sub-process (i.e., a trend or cycle). However, if the special cause generates complete chaos, then there is no model possible, and DOE will fail on such a process. Can this happen? Consider the process in which x_i ~ N(m_i, s_i), m_i ~ N(m₁, s₁), and s_i ~ N(m₂, s₂). An individuals control chart for x will be unstable because each x_i comes from a distribution having a different mean and a different standard deviation.

The closer that the real world process comes to being in a state of chaos, the more problematic the use of DOE will become in extracting valid inferences from the data. As Dr. Vining comments:

The net consequences from the presence of special causes are increased variability and decreased power for any test. Would it be desirable not to have special causes present? Of course, we would have more power and better predictions. However, it is completely inaccurate to say that an experiment is invalid in the presents of special causes.

On this last point, I think it would be more accurate to say that if there is sufficient data in each modelable process subset generated by the special causes, then blocking will make it possible to extract valid inferences from the data. However, if the amount of data in each modelable process subset is small, then inferences drawn will have a high degree of uncertainty (i.e., large confidence intervals). Perhaps so much so that the practitioner should have very little confidence in inferences thus derived. Hence, depending on the nature and level of chaos experienced by the process, and the amount of data in the modelable subsets, some amount of stability is required in order to generate meaningful levels of confidence in the inferences from designed experiments.

Commentaries

I asked two very knowable friends to comment on this article and they provided the following insightful observations:

From Ed Russell

I think it might be easier to explain another way. In a nutshell, having a process in control permits a MUCH simpler DOE. We teach MUCH simpler DOEs in industry. But if you want to hire a statistician and are willing to fund a more complex DOE, we can design one for you, which will work.

Why? When a process is in control, special causes are essentially eliminated (or accounted). Each special cause is a factor in a more complex DOE. Some of these special causes interact, some require blocking, and some require nesting, and sometimes a combination of nesting and blocking and interactions are required to isolate the signal that you wish to study from those, which you don't. Each special cause that is eliminated simplifies the "remaining" DOE, which is needed to achieve the desired result.

BTW: Only main effect special causes are cleanly "transmitted" to the error structure. If the special cause blocks, causes nesting, or interacts with other factors, you have a real mess..... Ugly issues, like Simpson's paradox, can arise if subpopulations due to that special cause if they are not handled correctly.

From Lester Wollman

Ed is right. Given enough resources, we can design and run a DOE for any type of situation, no matter how messy. So, theoretically, a DOE can be performed even if the process is completely out of control. But my experience is that no one likes complicated experiments, especially when the end result is a model that includes variables that you are trying to eliminate in the first place. A complicated DOE also holds up production and produces many units that are not shippable; hence they are considered to be a waste. It is more economical to put the initial effort into getting the process into some kind of control (which is what you have to do anyway), then performing a simpler DOE, rather than spending more resources in a complicated experiment right up front. The process does not have to be completely in control, as the article says, but it is good to strive for more control.

Reference:

Bisgaard, S. (2008). Must a Process be in Statistical Control before Conducting Designed Experiments? Quality Engineering, Taylor & Francis, Vol. 20, No. 2.
Deming, W.E. (1986). Out of the Crisis. MIT Center for Advanced Engineering Study. Cambridge, MA.
Vining, G. G. (2008). Discussion. Quality Engineering, Taylor & Francis, Vol. 20, No. 2.

John J. Flaig, Ph.D.
Fellow of the American Society for Quality
Managing Director
Applied Technology
Tel: 408-266-5174
E-mail: JohnFlaig@yahoo.com
Web: www.e-AT-USA.com