Testing

An experiment allows you to test and refine solution(s) and question underlying assumptions. At different stages of prototyping, different types of testing are performed.

Testing the Concept

The idea or concept behind the design is generally tested with early prototypes. Testing the concept involves gathering feedback from users about different aspects of the design.

Testing the Functionality

It is often useful to test individual components or sub-systems of a device to determine if the sub-system will meet your design criteria. To determine if different components of a device function as intended, you must design an experiment (test) to evaluate the functionality of the part.  Critically, the test will determine a range of functionality rather than just pass or fail.  The goal of your experiment is to gather data — identify a question, make a prediction, and make measurements.

Designing your own Experiment

Up to now most of you have run experiments that have been designed by someone else. For instance, in your Chemistry or Physics lab the question the experiment is designed to answer, the materials required, the safety issues involved, and the identification of critical variables are all completed in advance. You are only responsible for the final three steps: running the experiment, collecting the data, and drawing conclusions from the data. The goal of this section it to introduce you to designing the experiment.

While the setup of these steps will be presented linearly.  Often by working through the process, you will need to go back to earlier steps to refine your earlier plans.

1. Identify a question that you want to answer.  For your semester long design project, each member of the team will be responsible for identifying a question to answer about your project. Formulating a meaningful question is often the most difficult part of the process. Give yourself sometime to think through this when you need to do this for your project. In addition, it is a good idea to support your question with some explanation. Why is the question meaningful?  How did you reach the conclusion to focus on this question? If you are using prior information to formulation your question, make a note of your source and your evaluation of the source’s reliability.

2. Prediction. Predict the answer to your question.  This helps make sure that you have the right equipment and correct resolution to complete the experiment. It also makes clear what you are expecting to happen.  Predicting a result does not imply that the results are pre-determined, just that you have some expectations. Sometimes experiments do not follow expectations, but to predict a result you will have to consider possible sources of error which will be helpful when analyzing the collected data.

3. Identification of variables. In general, you will have one dependent variable and one independent variable but many controlled variables.
   • Dependent variable – The variable that you will measure.
   • Independent variable – The variable that you will change.
   • Controlled variables – The variables that you will keep the same.

4. Risk Assessment. Consider the possible hazards involved with measuring or changing those variables and think about ways to minimize the risk. While it is not possible to eliminate all risks associated with a laboratory, there are steps we can do to reduce risk to an acceptable level. This means that safety works at different levels. To ensure that users are safe in the laboratory, we consider safety at three levels: community standards, general laboratory safety, and hazards posed by specific tools. Community standards ask you to consider how you interact with the other patrons of the lab. The lab is small, and it is easy to get into each other’s way – these rules establish a minimum level of expectations. General laboratory safety addresses the broad rules applicable in the lab that apply to just entering the space. Together community standards and general laboratory safety govern how you will interact with the laboratory space and people within it.  You do not need to address these general rules, you should instead focus on the hazards posed by specific tools and what users should do to mitigate those hazards.

4. Materials. List the materials and equipment required for safety, such as safety googles.  List all other materials and equipment required to run the experiment.

6. Method. Outline a list of steps that you will follow to complete the experiment.  A detailed method ensures that you can repeat the experiment following the same steps for each iteration.  It also makes it possible for others to understand the experiment that you performed and repeat the experiment to confirm your results.

7. Data collection. To make the process of collecting data and keeping notes easier it is helpful to create a draft table of the data you will collect.  In general, the independent variable goes in the first column.  The remaining columns are used to show repeated trials of the multiple dependent variables measured. An example with data filled in is shown below.

Table 1: Impact of salt concentration on time it takes water to boil

Testing the Experience

Testing the experience only occurs with high fidelity prototypes which allow users to try out the new device similar to how the final product would be used. This type of test requires the entire prototype to be built and functional so that interaction data can be collected about the device.  The main difference between testing the concept and testing the experience is that now, the goal is to validate the design rather than learn more from the user.  As a result, the process of gathering feedback is more structured and uses a set of pre-defined survey questions that can be asked of each user.  There is also space for open ended questions that could be used to refine or update the design, but the focus is on quantifying if the design as a whole is fit for purpose.

Analysis

The goal of analysis to evaluate a scientific argument in light of the gathered evidence.  Using the experimental question, prediction, and collected data to determine if collected data supports or refutes the prediction.  The data should then be analyzed by considering any existing supporting theories, the accuracy or inaccuracy of the measurement (sources of error), and evaluate your confidence in the results. Notice that data analysis will require you to consider multiple competing ideas about data, pick the conclusion you feel is most accurate, and support that conclusion with evidence.