Chapter Three: Methodologies
Experimental Research Design and Analysis offers a rational approach to the quantitative methods of educational experiments. In its innovative presentation of the most commonly used experimental designs, this advanced text/reference discusses the logical reasons for selecting a particular design and shows how experimental results can be analyzed and interpreted. Real-world examples from different areas of educational experimentation are featured throughout the paper to illustrate how practical issues of design and analysis are handled.
For many true experimental designs, pretest-posttest designs are the preferred method to compare participant groups and measure the degree of change occurring as a result of treatments or interventions. Pretest-posttest designs grew from the simpler posttest only designs, and address some of the issues arising with assignment bias and the allocation of participants to groups. One example is education, where researchers want to monitor the effect of a new teaching method upon groups of children. Other areas include evaluating the effects of counseling, testing medical treatments, and measuring psychological constructs. The only stipulation is that the subjects must be randomly assigned to groups, in a true experimental design, to properly isolate and nullify any nuisance or confounding variables.
A scientific control group is an essential part of most research designs, allowing researchers to eliminate and isolate confounding variables and bias. Normal biological variation, researcher bias and environmental variation are all factors that can skew data, so scientific control groups provide a baseline. As well as eliminating other variables, scientific control groups help the researcher to show that the experimental design is capable of generating results. A researcher must only measure one variable at a time, and using a scientific control group gives reliable baseline data to compare their results with. For example, a medical study will use two groups, giving one set of patients the real medicine and the other a placebo, in order to rule out the placebo effect. In this particular type of research, the experiment is double blind. Neither the doctors nor the patients are aware of which pill they are receiving, curbing potential research bias. In the social sciences, control groups are the most important part of the experiment, because it is practically impossible to eliminate all of the confounding variables and bias. For example, the placebo effect for medication is well documented, and the Hawthorne Effect is another influence where, if people know that they are the subjects of an experiment, they automatically change their behavior. There are two main types of control, positive and negative, both providing researchers with ways of increasing the statistical validity of their data.
In any of these fields, ethical considerations and the wellbeing of the participants are the single most important consideration. The researcher must ensure that he causes no harm to the group, and it is generally accepted that honesty is the first parameter; the researcher must be open about purpose and intent. The ethical considerations concerning permissions, consent and possible suffering are very similar to guidelines governing psychology researchers. These are the main points, but an in-depth analysis is available here. Wherever possible, the observer should strive to understand the particular community. This may be a knowledge of the language, or some experience with the culture.
One example would be studying sexuality – whilst the observer need not be gay or lesbian to understand those groups, it does help, giving them a unique insight into the unique difficulties faced by gay communities.
There must be no chance of causing psychological or physical suffering to the participants, and they should be treated as partners in research. A researcher using human research subjects must avoid the aloof approach required by quantitative methods.
It is vital that the social science subjects are willing participants in the research, and are not coerced or induced into participating through false promises or benefits.
The social science subjects should be fully informed of the research and the possible implications should be transmitted through a pre-experimental briefing. Verbal and written information, in a language that they understand, should always be sought.
The participants should be fully informed of how their information will be used, how anonymous the information will be, and for how long it will be stored.
The participant should be able to withdraw at any stage during the research, and may also ask that all of their information, including film, photographs and testimonials be removed.
On occasion, the exact nature of the research cannot be revealed to the social science subjects, in case it influences the findings. In this case, the work must be constantly overseen by an independent ethical review panel and peers. In addition, the right to withdraw consent must be maintained.
These ethics are extremely important for maintaining the integrity of participation. It is very easy for researchers using social science subjects to cross the line and cause lasting damage to a group or community.
Historically, the use of ethics have been sloppy in some social science experiments, such as the use of deception in the milgram study, the stanford prison experiment, the bobo doll experiment or theasch experiment. These studies would probably have been disallowed today. This is especially important with the number of documentaries following groups or tribes, because it is very easy to stray into attempting to edit unfavorably and sensationalizing footage for ratings.
Pretest-posttest designs are an expansion of the posttest only design with nonequivalent groups, one of the simplest methods of testing the effectiveness of an intervention.
In this design, which uses two groups, one group is given the treatment and the results are gathered at the end. The control group receives no treatment, over the same period of time, but undergoes exactly the same tests.
Statistical analysis can then determine if the intervention had a significant effect. One common example of this is in medicine; one group is given a medicine, whereas the control group is given none, and this allows the researchers to determine if the drug really works. This type of design, whilst commonly using two groups, can be slightly more complex. For example, if different dosages of a medicine are tested, the design can be based around multiple groups.
Whilst this posttest only design does find many uses, it is limited in scope and contains many threats to validity. It is very poor at guarding against assignment bias, because the researcher knows nothing about the individual differences within the control group and how they may have affected the outcome. Even with randomization of the initial groups, this failure to address assignment bias means that the statistical power is weak.
The results of such a study will always be limited in scope and, resources permitting; most researchers use a more robust design, of which pretest-posttest designs are one. The posttest only design with non-equivalent groups is usually reserved for experiments performed after the fact, such as a medical researcher wishing to observe the effect of a medicine that has already been administered.
Experimental Research
True experimental design is regarded as the most accurate form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical analysis.
For some of the physical sciences, such as physics, chemistry and geology, they are standard and commonly used. For social sciences, psychology and biology, they can be a little more difficult to set up.
The independent variable, also known as the manipulated variable, lies at the heart of any quantitative experimental design.
This is the factor manipulated by the researcher, and it produces one or more results, known as dependent variables. There are often not more than one or two independent variables tested in an experiment, otherwise it is difficult to determine the influence of each upon the final results.
There may be more than several dependent variables, because manipulating the independent can influence many different things. For example, an experiment to test the effects of a certain fertilizer, upon plant growth, could measure height, number of fruits and the average weight of the fruit produced. All of these are valid analyzable factors, arising from the manipulation of one independent variable, the amount of fertilizer. The term independent variable is often a source of confusion; many people assume that the name means that the variable is independent of any manipulation. The name arises because the variable is isolated from any other factor, allowing experimental manipulation to establish analyzable results. Some research papers appear to give results manipulating more than one experimental variable, but this is usually a false impression. Each manipulated variable is likely to be an experiment in itself, one area where the words ‘experiment’ and ‘research’ differ. It is simply more convenient for the researcher to bundle them into one paper, and discuss the overall results.
The botanical researcher above might also study the effects of temperature, or the amount of water on growth, but these must be performed as discrete experiments, with only the conclusion and discussion amalgamated at the end.
Selection of Sample
At initial it was a plan to select 120 students for experimental and 120 students for control group. Some students finalized to withdraw from the research study. For maintaining the uniformity of the groups total 200 students were considered for the study and were randomely distributed in 4 groups 2 experimental and 2 control groups comprising 50 students each.
Table 3.1 : Sample
| | Boys | Girls | Total |
| Experimental Group | 50 | 50 | 100 |
| Control Group | 50 | 50 | 100 |
| Total | 100 | 100 | 200 |
Note: (All students were selected from the batch of seventh standard of Sushil Himmatsingka Vidyalaya of Wardha)
Statistical Analysis Tools:
For examining the effectiveness of the Computer Aided Learning Packages upon the enhancement of Critical Competencies application of Statistical Analysis tools were considered vital. T Test and Chi Square Test Tools were considered suitable ones in this regard.
Chi Square Test
The chi-square is one of the most popular statistics because it is easy to calculate and interpret. There are two kinds of chi-square tests. The first is called a one-way analysis, and the second is called a two-way analysis. The purpose of both is to determine whether the observed frequencies (counts) markedly differ from the frequencies that we would expect by chance.
The observed cell frequencies are organized in rows and columns like a spreadsheet. This table of observed cell frequencies is called a contingency table, and the chi-square test if part of a contingency table analysis.
The chi-square statistic is the sum of the contributions from each of the individual cells. Every cell in a table contributes something to the overall chi-square statistic. If a given cell differs markedly from the expected frequency, then the contribution of that cell to the overall chi-square is large. If a cell is close to the expected frequency for that cell, then the contribution of that cell to the overall chi-square is low. A large chi-square statistic indicates that somewhere in the table, the observed frequencies differ markedly from the expected frequencies. It does not tell which cell (or cells) are causing the high chi-square...only that they are there. When a chi-square is high, you must visually examine the table to determine which cell(s) are responsible.
When there are exactly two rows and two columns, the chi-square statistic becomes inaccurate, and Yate's correction for continuity is usually applied. Statistics Calculator will automatically use Yate's correction for two-by-two tables when the expected frequency of any cell is less than 5 or the total N is less than 50.
If there is only one column or one row (a one-way chi-square test), the degrees of freedom is the number of cells minus one. For a two way chi-square, the degrees of freedom is the number or rows minus one times the number of columns minus one.
Using the chi-square statistic and its associated degrees of freedom, the software reports the probability that the differences between the observed and expected frequencies occurred by chance. Generally, a probability of .05 or less is considered to be a significant difference.
A standard spreadsheet interface is used to enter the counts for each cell. After you've finished entering the data, the program will print the chi-square, degrees of freedom and probability of chance.
In a 2X2 table (four Cells ) there is a simple formula that eliminates the need to calculate the theoretical frequencies for each cell-
Table 3.2: Achievement – Non-achievement Data
| Competency Achievements | Experimental Group | Control Group | Total No of Students |
| Achievers | A | B | A+B |
| Non-achievers | C | D | C+D |
| Total | A+C | B+D | N |
| Degree of Freedom = (Rows – 1)(Column – 1) = 1 | |||
| X 2 = | N[|AD- BC|] 2 |
| (A+B)(C+D)(A+C)(B+D) |
For each level of significance there exists a critical value of chi-square. For rejection of the Null Hypothesis, the calculated value of chi-square must equal or exceed the critical value depicted in the table of Critical Values (Table 3)
Table 3.3: Critical Value of Chi-Square
| Particulars | Critical Values at different Levels of Significance and 1 degree of freedom | |
| Levels of Significance | 0.05 | 0.01 |
| Chi Square Value | 3.84 | 6.64 |
In statistics, the number of degrees of freedom is the number of values in the final calculation of a statistic that are free to vary.
Estimates of statistical parameters can be based upon different amounts of information or data. The number of independent pieces of information that go into the estimate of a parameter is called the degrees of freedom (df). In general, the degrees of freedom of an estimate is equal to the number of independent scores that go into the estimate minus the number of parameters estimated as intermediate steps in the estimation of the parameter itself.
The number of degrees of freedom is the number of independent observations in a sample of data that are available to estimate a parameter of the population from which that sample is drawn. For example, if we have two observations, when calculating the mean we have two independent observations; however, when calculating the variance, we have only one independent observation, since the two observations are equally distant from the mean.
Computer as a tool of Research
Application library as framed for the purpose of Research kept equipped with different tools and mechanisms of learning and instructional designs. Important aspects of those learning and teaching tools made the research activity more dynamic and result oriented.
CALL software is representative of an acquisition-oriented approach because it:
1. Promotes a communicative interaction between the learner and the computer.
2. Provides comprehensible input at a level.
3. Promotes a positive self-image in the learner.
4. Provides a challenge but does not produce frustration or anxiety.
5. Does not include overt error correction.
6. Allows the learner the opportunity to produce comprehensible output.
7. Promotes effectively, acting as a catalyst, the leamer -leamer interaction in the target language.
Traditional CALL software lent itself effectively for developing mainly reading skills, through vocabulary and grammar exercises and secondly writing skills. The use of computer to practise grammar will comfort especially an ESP/T teacher who usually has to teach students, with insufficient linguistic competence. Exercises can range from simple ones as filling with the right article to practicing reported speech which is a fairly difficult area of grammar. This is the case in the Experimental Group Within the group students were taught mainly reading skills and secondly listening and writing in the class, and then they participated in Self Access Centre for further practice with the assistance of the computer aided learning programs.
In some cases a technological attribute distinguishes one medium from others in terms of the learning experiences it affords. The technology of computer based instruction allows the kind of individualisation and interaction not permitted by other media (e.g. video). This situation facilitates learning by presenting the learner with a stimulus and evoking a response. Such a situation has been overlooked in the case of control group.
Curricular Suggestions for CALL course for teachers include familiarity with the history of microcomputers, with technology, and with educational concerns touching the use of CALL. Consideration is given to learner's needs and professional objectives in the use of the new medium. An additional area of consideration in judging the probable effectiveness of foreign language software is the degree to which the materials may directly or indirectly promote the use of particular strategies in the learner. There are several types of strategies that seem particularly well-suited to, being introduced and practiced on the computer. In reading, for example, psycholinguistic research pointed to the importance of skimming exercises. In writing, there are production strategies such as writing dialogues, brain storming, list making and flexible outlining that many second-language learners are either unaware of or ignore.
Simulations vial CAI/CALL is limited only by the scope and imagination of teachers and students. Students can interact with computer programs on a background of illustrations with the simulation itself or they can build screen displays to show their growing control of the competence.
Multimedia programs can display text, high quality sound, animation and video. There are CD-ROM programs like "Longman English Works, for example, which help the learner practice his/her skills in listening and speaking and develop his/her pronunciation. The learner has the ability to listen to the dialogues and passages with or without the written text. He/She can record his voice on the hard disk and incorporate photos through the use of a scanner. So, the emphasis is extended from reading or writing skills to spoken language and listening skills. Using the numerous activity options, the learner can decide at any time how easy or how challenging he/she wants the exercise to be. Multimedia offer great opportunities for differentiation, especially in mixed-ability classes.
Some of the Computer Aided Learning Packages from the evergrowing library are discussed below:
1. Longman Interactive English Dictionary (LIED). It is an exciting learning tool which combines a computer database with sound, video and pictures. The user has the access to many different kinds of information contained on the database( about grammar, the meanings of words, pronunciation, famous people and places, etc.) and can see, hear and read through the use of the compact disk and video. The drawings and photographs it contains, help the user to understand the meanings of the words, and there are short films which show how English is used in real -life situations. It is ideal for practicing pronunciation and listening comprehension. Learners can get help with the pronunciation of individual words by calling up the entries from the Longman pronunciation Dictionary and listening to them. In addition the video mini-dramas provide examples of natural dialogue. The user can have the text of the video dialogues in a separate window on screen while the video is playing.
2. Business Challenges Interactive. It is for learners of business English at false beginners and elementary level who want the freedom and flexibility of interactive learning. It provides over 60 hours of materials including video footage, audio clips, photographs, graphics and exercises to practice English for work and social situations. Tests allow learners to check progress at any time. Records monitor learner's scores in exercises and tests, and show how much of the program has been completed.
3. The Grammar ROM. The program is designed to help the student revise and practice his English grammar in a new and exciting way. It includes exercises and tasks on all the main grammar points at intermediate level. Student can use The Grammar ROM by him/herself as part of a course of study , or to supplement any intermediate level English language study.
4. The Electronic Business Letter Writer (BLW). It is a software package for people who need help in writing business letters. It is also a learning tool for intermediate to advanced students of business English. BLW contains over 200 model texts. These example documents are classified according to Typesetter, fax, memorandum etc.), subject area (banking, Insurance, Credit etc.), purpose( Advice, complaints Enquiries etc.) There is also an Info Bank which contains on-line help in two main areas:
1. How to perform any task in BLW
2. More general topics related to business letter writing.
This section contains:
§ Structure and Layout
§ Content and Style
§ Type of Document
§ Subject Area
§ Purpose of writing
Contents: Information about specific topics, such as Documentary Credit Advice on how to produce effective business letters, faxes etc.
5. BBC English Expressions. It is an English language course on CD ROM. BBC English Expressions is designed to teach you the spoken language required to deal with some of the most common situations you are likely to meet if you visit a foreign country such as: Eating out, Travelling by train, Asking for information etc. The user can select activities for each dialogue from the activity bank. The dialogue starts by clicking at Story. When you need individual words and phrases for the topic the student may click at the Useful words. Speaking Test is a role play exercise: the student speaks in place of one of the characters.
6. Useful Words + provides student with extra words and phrases for the topic he/she has chosen.
In Speaking Test + the student will be presented with unexpected responses and tasks. When clicking at Listening Test the student is given a comprehension test and finally by clicking at Text Exercises the student is given exercises which test both the ability to understand the language he/she has learned and his/her ability to assemble correct text sentences.
7. Welcome to English for Business. These CDs are for people who wish to improve their English language skills as used within the English speaking business environment. The series aim to enable the user to understand real business English. They use between 25-30 minutes video, which forms the centre of the learning activities. All the scenes contain authentic business people talking about their own situations. By looking at the subtitles while watching the video the student gains a better understanding of the relationship between the spoken and written forms. The main learning features of these CDs are:
ease of use
real authentic business language
develops listening comprehension
uses modem methodology
gives instant feedback on learning tasks
contains over 250 screens of learning tasks
develops business grammar, business language functions and business vocabulary.
1. Computer as a communication tool
Within the scope of this research activity plan has been made to utilize computer as per the scope of application pursuable in making the process more and more effective. It has been also planned to frame computer as a research tool, as a powerful interface and also as a dynamic learning and teaching companion in the experimental group.
As per its nature and configuration Computers can foster communication among local and extended community members. And teachers can give feedback to their students through computer communication tools such as electronic bulletin boards and instant messages. Teacher can also add information on the electronic bulletin boards to guide students through the lessons.
In a study by Blumenfeld et al (1996), they used web-based facilities to support and keep track of synchronous dialogue among students that then serve as a public archive of conversations: “…conversations can be stored, reflected on and reacted to, creating a common knowledge base that is open to review and comment and manipulation” (Blumenfeld, Marx, Soloway, & Krajcik, 1996, p.39)
In a social constructivists’ perspective, the success of these web-based communications depends on the opportunities afforded to students for critiquing the ideas of others as well as soliciting alternative ideas, sorting out conflicting information and responding to other learners. (Linn, 1998) If all students post their ideas on a central database accessible through a network, they can establish a discourse community comparing and reflecting on the multiple perspectives of others. (Kearney, 2004)
2. Computer as a scaffold
Internet websites provide student centered learning environments. The control over pacing of computer-based learning gives students the flexibility and time to thoroughly build their understandings.
Besides, computers help and guide learning by reducing complexity, highlighting concepts and fostering metacognition. For example, the use of computer program such as e-chem helps students create more scientifically acceptable representations of molecules. Software support complex processes that students are not capable of completing without assistance. (Singer, Marx, and Krajcik, 2000, p173) Therefore, extensive use of learning technologies helps students develop deep understanding of scientific concepts and processes by themselves.
3. Computer as a backdrop of real world
Learning technologies expand the range of topics that can be taught in the classroom. Especially, computers and its Internet access extend student-learning experiences beyond the classroom by introducing real-world issues with movies, simulations and animations. They promote contextualized understanding of scientific phenomena in real world. In his research, Kearney (2004) used computer-mediated video clips to show difficult, expensive, time consuming or dangerous demonstrations of real projectile motions. The real-life physical settings depicted in the video clips provided interesting and relevant contexts for the students. Salomon, Perkins, and Globerson (1991) argue that the effect of the technology is more lasting effects as a consequence of students’ mindful engagement with the tool.
We expect students to interact effectively in front of computer. For ensuring this task minimum standard of learning of Computer operations must be ensured for enabling them in handling different teaching learning tools entangled with computer. It is expected that the Categorical list of standards should be used for standardizing the level of understanding of learners. (Appendix J: Computer/Technology Standards of Learning).
Tendency toward delivering online instructional materials and evaluation schemes increased substantially these days. Consideration of the quality perspective and standard of the content used for preparing subject materials remain questionable affair. Users and facilitators of Computer Aided Learning are still not completely aware of the importance of such package in bringing vibrant change in the Curriculum transaction process and dynamic learning mechanism.
____________