Effects on indicators of critical thinking

We can look at how different indicators of critical thinking are affected by the technology used for the discussions, by grouping together all the statement counts for relevance, importance, novelty, outside material, linking ideas, justification, criticism, resolving ambiguity, widening the discussion and practical grounding. Figure 3 shows the overall pattern, comparing all the analysed computer conference and seminar transcripts. No ratios were plotted for W (widening) and P (practical grounding) for the computer conferences because the sample was too small to make the ratio reliable.

Average indicator depth over all groups for CC and Seminars

Figure 3. Patterns in depth of critical thinking by indicator for all CC and f2f seminars.

On average, the students taking part in the computer conferences brought in relevant outside material more often and were better at linking together ideas and solutions, while in the face-to-face seminars students came up slightly more often with new rather than old ideas. The students seem to have adopted a more serious, worthier, style when taking part in the computer conferences, as if it were writing an essay, as shown by the higher ratio for important statements.

How consistent are these differences? One test is to calculate the differences between depth of CT ratios for matched samples: i.e. group A CC - group A seminar, and so on. These are plotted in Figure 4. Only for the O (bringing in outside knowledge and material) indicator is the difference consistent for all three groups, significant at 0.3% (t = 18.8 > critical value of 4.3). None of the other indicator differences are significant (at p = 5%), nor is a repeated measures multivariate analysis of variance (Table 3). Given the small sample size, this is not surprising, but there are notable differences for I (importance), N (novelty) and L (linking ideas) in some groups.

Differences between CC and Seminars for each indicator

Figure 4. The CC-Seminar differences in depth of CT ratio for different indicators.

Table 3. Repeated neasures ANOVA of CC-Sem. differences by indicator.
VariableHypoth. MSError MSFSig. of FETA SquarePower
R0.020.011.80.3120.470.13
I0.240.064.00.1840.670.22
N0.000.090.00.8640.020.05
O0.090.00353.90.0030.991.00
L0.000.800.00.9780.000.05
J0.170.043.80.1910.650.21
C0.080.017.10.1170.780.33

Figure 6 shows the pattern for the computer conferences run by different groups. The patterns are all similar except for the ratios for L (linking ideas). Figure 5 shows a similar variability in L for face-to-face seminars, and a similar variability in I (important statements). Looking more closely at Figure 5, we notice that these differences are consistent with the subject discussed. Although the group makeup and size are different, the group seems to have less effect on the depth of CT ratios than the subject.

Seminar depth of CT ratios for different indicators

Figure 5. Seminar depth of CT for different indicators.

CC depth of CT ratios for different indicators

Figure 6. CC depth of CT for different indicators.

The I and L depth of CT ratios are negative for both privacy discussions, and positive for the others. Whereas the N depth of CT ratios are higher for the privacy discussions. It appears that the students generated a lot of new ideas in their discussions of IT and privacy, and kept the discussion wide, but were less able to link these ideas together, resolve ambiguities, bring in relevant outside material, or to keep the discussion centred on important, non-trivial, issues. It is with such subjects that computer conferencing could be of most benefit, since it shows signs of supporting those aspects of critical thinking that were most lacking in this face-to-face discussion.

As a check on the significance of the subject discussed upon the depth of critical thinking ratios, we ran a 2-way analysis of variance on subject by indicator, for the depth of CT ratios found in the two office automation and the two privacy seminars.

Table 4. Subjects discussed by CT indicators ANOVA.
Source of VariationSSdfMSFP-valueF crit
Subjects0.531.000.5310.60.50%4.5
Indicators3.567.000.5110.10.01%2.7
Interaction3.507.000.5010.00.01%2.7
Within0.8016.000.05
Total8.3931.00

This showed a strong, highly significant interaction effect. So the subject discussed affected the depth of processing adopted in seminar discussions differently for different indicators.