Table of Contents
In the article Basics of research methodology I for medical students, we addressed all important issues ranging from forming hypotheses to research criteria. In this article, we will now approach the topics from study design to evaluation of the results.
|Forming hypotheses||What is the research question? What is the hypothesis?|
|Operationalization||Describes how the theoretical construct can be made ‘measurable’|
|Research criteria||Quality criteria of a psychometric test: objectivity, reliability, and validity|
|Study design||Type of investigation and its process need to be carefully planned|
|Methods of data collection||Psychological tests, interviews, systematic observations, and registering psychophysiological processes|
|Data analysis||Analysis by means of statistical tests|
|Evaluation of the results||Repeatability and generalizability are required.|
What is meant by study design? In order to conduct a scientific study, a study design is created or followed which contains all the information about research planning.
In an experiment, the change of a situation due to systematic manipulation is assessed. The objective is to uncover cause-effect relationships. Usually, studies involve variables which can be divided into independent and dependent variables:
- Independent variables (IV): the influencing conditions that can be manipulated
- Dependent variables (DV): the subject of investigation and of research interest
Types of study designs: cross-sectional, longitudinal or case-control study?
|Cross-sectional study||Examination of a sample population at a specific (single) point in time. Exposure and outcome are determined simultaneously.|
|Case-control study||Comparing the group of patients under investigation with a group of patients who do not have the condition. The study begins with a group of people processing the outcomes and they are examined for the presence or absence of possible causative factors|
|Evaluation study||A measure is evaluated (e.g., the medical education in your university).|
|Randomized control trials||Subjects are randomly assigned to the experimental conditions|
|Randomized study||Subjects are randomly assigned to the experimental conditions.|
|Ex post-facto study||Data is already collected and the investigation is carried out afterwards (usually via surveys)|
|Single case study||Individual cases are analyzed (very low scientific validity, generalization not possible!)|
|Case report series||Involves a report on a series of patients with the outcome of interest. There is no control group involved.|
Sample is defined as the subset of a population that is selected following specific criteria.
A sampling error describes the deviation of the values measured in the sample from the entire population since a sample hardly ever represents the whole population it is taken from. The sampling error can be reduced by a sampling size as large as possible and a small variance of the sample distribution.
Which subtypes of samples are there?
- Random sample: Individual is randomly selected from the population. When the population is previously divided into sub-populations, this is called a stratified random sample.
- Quota sample: ‘Miniature sample’ of the population according to specific characteristics (e.g., percentage, age groups, sex, etc.)
- Cluster sample: Groups are pooled into clusters (e.g., streets, districts, regions, etc.)
- Extreme group: Subjects with personality traits larger than 2 standard deviations
- Exposed group: Subjects under certain conditions (e.g., unemployment)
Methods of Data Collection
Basically, a distinction is made between 4 different types of data collection:
- Behavioral observations
- Psychological tests
- Assessment of psychophysiological processes
The following synoptic table summarizes which types of data can be collected:
|TYPES OF DATA|
|Individual data||Specific collected individually|
|Aggregated data||Aggregation of individual data|
|Primary data||Raw data, directly assessed data that has not been subjected to processing.|
|Secondary data||Modeled, processed primary data (originally assessed for other purposes)|
|Self-assessment||Examinees themselves report personal data.|
|External assessment||Examinees are evaluated by other persons.|
|Types of observation: the combination is possible and preferable!|
|immediate||retrospective||with participation||without participation|
The great advantage of systematic observation is that it is mostly unfettered by the observer and its interpretation. The systematization is given through well-defined criteria (place, time, recording sheet, etc.).
For participant observation, the observer is integrated into the event of observation. Here, a frequent problem is to simultaneously participate and record. The observation without participation requires “mere” observing and recording, which can be assessed by respective media (e.g., video camera).
Interviews for behavioral observations
Primarily, the interview should inquire about information in a goal-oriented manner about e.g., symptoms of a clinical picture. The survey is conducted personally, in written form or by telephone.
Quantitative interviews are highly standardized. There are different levels of standardization:
- Structured: Content, order, and the exact wording of the questions are clearly defined. It is a type of directive interview since the interviewer fully leads the survey.
- Unstructured: Contrary to the structured interview, nothing is predetermined, except for the topic of conversation. This procedure is called a non-directive.
- Semi-structured: This type of interview is a mixture of both. Subject areas of the questions are predetermined. However, the interviewer can decide, to a certain extent, which topics he or she wishes to elaborate upon.
Qualitative interviews are part of the hermeneutic methods. The individual viewpoint of the respondent is the center of main interest.
- Ethnographical: assessment of culture-specific characteristics
- Narrative: the respondent has to talk about the topic of interest
- In-depth interview: technique in psychoanalysis
Types of questions: open, closed, and leading
In open questions, the respondent has a wide range of answering at his disposal. The anamnesis usually starts with open questions and, later on, leads to closed, more specific questions.
- What led you here?
- How are you feeling today?
Here, the inquirer limits the possible responses. Closed questions include dichotomous questions (2 possible responses) and multiple-choice questions (more than 2 possible responses).
- Where exactly are you feeling the pain?
- What was your job before your retirement?
- Is your pain in the area of the knee or the calf?
- Did you sleep better last night than the night before?
- Is your headache pulling, pulsating or stabbing?
- Do you have to frequently go to the bathroom for passing water in the morning, afternoon or evening?
Leading questions guide the respondent in a specific direction and may distort the response. The respondent might feel pressured into answering ‘appropriately’.
- Certainly, you have reduced your alcohol consumption by now, since your liver function values were increased last time?
- Are you sure you want to decide against this surgery against medical advice?
Research criteria and quality criteria were already discussed in the article on the basics of research methodology I. Here, you can find a classification into achievement and personality tests with examples and possible sources of errors.
Achievement tests are divided into speed tests (constant task difficulty with limited time) and power tests (increasing task difficulty with constant time).
- IQ tests (e.g., Wechsler adult intelligence scale WAIS, Intelligence structure test IST)
- Academic achievement tests
- Aptitude tests
- Concentration tests (e.g., Test of Attention d2)
Objective personality tests
- Sometimes I feel rather blue for no reason (neuroticism): yes/no
- I have frequent headaches (somatic complaints): yes/no
16 PF, 16-Personality factor questionnaire: 16 personality factors are measured, with 12 items each. In each case, 3 possible responses are given; 30–45 minutes of work time.
- I have frequent mood swings (emotional stability): true/cannot say/false
- I don’t let others discourage me (anxiety): true/cannot say/false
MMPI, minnesota multiphasic personality inventory: Psychopathological symptoms are assessed via 556 items. The scales are comprised of, for instance, depression, hypochondria, and schizophrenia; 30-40 minutes of work time.
- There is something wrong with my mind (schizophrenia): yes/no
- I wish I could be as happy as others seem to be (depression): yes/no
Neo-FFI, neo five-factor inventory: 5 personality traits (Big 5) – neuroticism, extraversion, and openness to experience, agreeableness, and conscientiousness are assessed by 12 items each; 10 minutes of work time.
- I try to be friendly to everybody I meet. 1 (strongly disagree) – 2 – 3 – 4 – 5 (strongly agree)
- I keep my belongings neat and clean. 1 (strongly disagree) – 2 – 3 – 4 – 5 (strongly agree)
Sources of errors in personality tests
The most common source of errors in personality tests is that subjects answer in a way they think is ‘socially desirable’. If possible answers are scaled (as in the case of the NEO-FFI). There is a strong tendency towards the middle instead of the extreme options. Simulation and dissimulation can perhaps be revealed by questions like ”I never lie”.
For this type of test, the defense mechanism projection is utilized. Projective tests don’t assess on the basis of the subject’s statement, but rather the ‘true’, probably covert desires are read into the test material. The point of criticism for this type of test is the missing evaluation objectivity.
- Rorschach test (inkblot test): Subjects make associations with various inkblot pictures which are then interpreted.
- Thematic apperception test: Subjects write a story based on pictures, followed by an analysis of the contents.
- Baum test (tree-drawing test): The subject is requested to draw a tree. The interpretation follows specific criteria (what do the roots/branches/trunk/… look like?).
- SF-36, Short-Form-36-Health Survey: Assesses the disease-spanned, health-related quality of life.
- GBB, Giessener Beschwerdebogen: Assesses somatic complaints.
- BDI, Beck Depression Inventory: Assesses the symptoms of depression.
Data Analysis and Data Interpretation
Qualitative data are comprised of non-numerical data, e.g., obtained from interviews. Quantitative data are obtained from scales or category systems.
Qualitative analysis methods
Qualitative analysis methods are sparsely generalizable. The types of analyses focus on the content assessment of individual questions.
- Content analysis: Analysis of communication material (videos, audiotapes).
- Document analysis: Type of content analysis
- Sociometrics: Statements of people’s attitudes towards each other
- In–depth interviews
- Group discussions
Quantitative analysis methods
Quantitative analysis methods are divided into univariate, bivariate, and multivariate analyses.
The univariate analysis (analysis of 1 feature)
|Absolute frequencies||Relative frequencies||Cumulative frequencies|
|How many people suffer from periodontosis?||The proportion of women and men with depression||Successively summarized category frequencies, e.g., percentage of high school graduations graded as ‘excellent’, ‘good’, ‘satisfactory’, etc.?|
Measures of average
|Sum of all measured values divided by their number||Linear split into 2 even halves: 50% above and 50% below||Most frequent value of a distribution (peak)|
Measures of dispersion
|Variance s2||Standard deviation s|
|Ratio of the sum of squared deviations of all measured values and the number of all measured values||The square root of the variance (the standard deviation allows statements about heterogeneity and homogeneity)|
The normal distribution is characterized by 5 criteria:
- The curve of the distribution has the shape of a bell (therefore often called the bell curve; the standard normal distribution has the shape of the ‘Gaussian bell’).
- The distribution is symmetric.
- Mode, median, and arithmetic mean are identical.
- The distribution asymptotically approaches the X-axis.
- Two-thirds of the total area is located between the X-values of the inflection points.
Bivariate analysis (analysis of 2 correlating features)
The correlation describes a statistical technique that tests for relations. The strength and direction of this statistical relation are called correlation coefficient. The correlation coefficient doesn’t make statements about causal relations.
|r = 0||r = 1||r = -1|
|no correlation||linear correlation between the features||inverse linear correlation|
Multivariate analysis (analysis of several correlating features)
Memorize the following methods:
- Multiple regression and path analysis
- Discriminant analysis
- Factor analysis
- Multidimensional scaling
- Cluster analysis
Evaluation of Results: Repeatability and Generalizability
Now, the scientific study is over, but how meaningful are the results de facto? Repeatability and generalizability are the criteria that have to be met. Results are regarded as repeatable if the same effects are detected repeatedly and general rules can be derived from it.
If these general rules occur and are not restricted to specific groups of subjects, this indicates generalization. Whether a research project is ethically justifiable, is examined in advance by the ethical commission.
The cross-validation is a statistical technique for assessing the validity of study outcomes. For this purpose, the procedure is applied to a second sample.
Evidence-based medicine (EBM): intensive evaluation of results
You will be faced with this term throughout your entire studies. The goal of EBM is to complement the practical experience of clinicians with relevant research. Hereby, medical care shall be further optimized and only actually effective interventions and therapies shall be identified, applied or stopped. Guidelines should be especially be established based on EBM.