Approximate time needed to complete the tutorial: 20 min.
This tutorial will give you a quick and non-mathematical introduction to statistical tests. We hope to give you a more “empirical” understanding of statistics than what you may have gained from following courses in statistics. The tutorial also serves to refresh your understanding of statistics and its limitations, because next week you will perform a lot of analyses.
As a secondary aim, you will get a flavor of programming in Matlab.
Several questions are asked. Discuss the answers with the student sitting next to you and ask the tutors if you run into problems.
A statistical test is a method which you use to evaluate the risk that the results you find just happened by random coincidence. For example, let us say you measure the amplitude in the alpha frequency band (8-13 Hz) using EEG in two groups A and B, and you find that the average in group A is 1 μV lower than in group B. Is this difference significant, i.e., can you be sure that if you measured more people from the two groups you would also find a difference? A statistical test can be used to tell if a difference is statistically significant result meaning that the result will only happen by chance with a certain probability (also called the significance level, or the p-value). It is common to set the significance level at 5%.
Make two sets of data; A and B. Type in the matlab command window A = randn(20,1); B = randn(20,1);. This will make two sets of data A with 20 values, and an average of 0; and B also with 20 values, and an average of 0. Imagine these two datasets are the measurements (e.g., EEG amplitudes) on two groups (group A and group B) of students (e.g., males and females). In the next step we will test if there is a difference in amplitude between the groups.
A and B are by definition equal in their average value (which is 0). Type [h, p] = ttest2(A,B) this is the student's t-test. p is the p-value. The probability that the average difference between A and B happened by chance. If you run the steps 1-3 several times you will not get the same p-value.
Run the code 1000 times (type the code below in the matlab command window).
A = randn(20,1); B = randn(20,1);
[h, p(i)] = ttest2(A,B);
What is the percentage of p-values less than 0.05? Write length(find(p<0.05))/10 in the matlab command window to compute this percentage. You should get around 5%. Do you understand why?
Power: Is a difference always significant?
We will now simulate two data sets with a difference to see if the t-test will find a significant difference. Do it 1000 times (use the code below)
A = randn(20,1); B = 1 + randn(20,1);
[h, p(i)] = ttest2(A,B);
A and B by definition have a difference in average B-A = 1. We should find that A and B are significantly different. What is the percentage of p-values less than 0.05? Write length(find(p<0.05))/10 in the matlab command window. You will find a percentage around 86%. It means that 86% of all cases the t-test found a significant difference between A and B; this is called the statistical Power of your test.
We see that a statistical test will not always find a significant difference even if two data sets are different. The reason is that the overall distribution of A (red in Figure below) and B (blue in Figure below) are over-lapping. Since we are only sampling 20 values from these two distributions it is very easy to get two data sets that are so close that the t-test is not significant. Can you think of a way to get a higher power?
One way of increasing the power in a statistical test is to increase the number of samples (or, i.e., subjects). A major concern in the design of an experimental study is, therefore, to ensure a sufficient power. But how many subjects are enough? We do not want too many subjects, because each subject we include in a study costs both money and time. To get an estimate you can do a power analysis.
Power analysis: How many subjects do I need?
In a power analysis we want to map the relationship between power and number of subjects. We, therefore, generate data sets with different numbers of subjects, and calculate the power. Type the following code in the Matlab command window.
A = randn(n,1); B = 1 + randn(n,1);
[h, p(i)] = ttest2(A,B);
ttestpower(n) = length(find(p<0.05))/10;
When the code is finished, type plot(ttestpower). You will get a figure with number of subjects on the x-axis, and statistical power on the y-axis. You will see that you need a certain number of subjects to get a good power, but also that it is not necessary to use a huge number of subjects. The power depends on the number of subjects, but also on the difference in mean between the two datasets, in our case the difference between the mean of A and the mean of B. The power depends also on the standard deviations of A and B. You can use the function nbt_PowerAnalysis to try different means and standard deviations of the data sets. If you want to analyse the power of a test between two datasets, one with mean 0 and standard deviation 1.5, and the other with mean 2 and standard deviation 1, you should write:
No. You can not always trust a statistical test. Statistical tests have certain assumptions, e.g., a t-test assumes that the data are normally distributed, and if these assumptions are not met then the test is inaccurate or not valid at all.
Usually, however, a t-test is a good first choice for testing simple differences between two groups.
In your analysis you will test if different biomarkers are correlated. A and B are said to be correlated if there is a linear relation between them. For example, you might want to know if there is a relation between the amplitude of the EEG data and the rating of the ARSQ item: I felt sleepy (on a scale from 1 to 5). In this case you have paired observations: for every subject there is the EEG amplitude and the rating of the ARSQ statement. The i-th place in the vector A contains the amplitude of subject i, and the i-th place in B the ARSQ score of subject i. If you plot A on the x-axis, and B on the y-axis, you have a first impression of the correlation between the two variables. In other words, you visualize how brain activity relates to cognition. The more the data points are concentrated on a line, the stronger the variables are correlated. The following code gives an example of two uncorrelated variables. The correlation coefficient is computed with the function corrcoef, which also generates a P-value to test whether there is correlation: a low P-value corresponds to a significant correlation. The output of corrcoef is a matrix; when running the following code, only the relevant information is displayed.
A = randn(100,1);
B = randn(100,1);
Note, also uncorrelated variables might result in a high correlation, by chance. Run the code several times to see that the variables are uncorrelated. Now we create two correlated variables by setting A equal to B plus some noise:
The correlation coefficient is a measure which tells you how strong the correlation is. In the figure below you see how the correlation coefficient changes for different distribution of data points. In the lowest row you see that the correlation coefficient might not reveal more complex relations between A and B.