How to best verify the accuracy of self-reported data?

Data

What is Data Collection?

Data collection is the procedure of collecting, measuring, and analyzing accurate insights for research using standard validated techniques.

Put simply, data collection is the process of gathering information for a specific purpose. It can be used to answer research questions, make informed business decisions, or improve products and services.

To collect data, we must first identify what information we need and how we will collect it. We can also evaluate a hypothesis based on collected data. In most cases, data collection is the primary and most important step for research. The approach to data collection is different for different fields of study, depending on the required information.

A significant number of scientific investigations denote a lack of rigor,  and this is largely due to the non-validation of the instruments used. This is much more evident in the behavioral sciences, where the most frequent methodology is qualitative,  a type of research where an indiscriminate use of instruments is observed, which are not typical of this methodology. This responds to an interest in the search for contextualization and homogeneity.

In an analysis carried out on 102 doctoral theses developed in the last 10 years, it was detected that the most used instrument is the survey; that each investigation designed its own instrument; and that, in the best of cases, responded to the objectives set (Conference in postdoctoral course: Analysis of the use of instruments in doctoral research, presented in 2014 by Tomás Crespo Borges, at the Pedagogical University of Villa Clara).

Due to its importance and complexity of application, instrument validation is considered a type of study within intervention studies, that is, at the same level as experimental, quasi-experimental, among others.

The questionnaire is an instrument for collecting information, designed to quantify and universalize it. For this reason, the moment of validation is of great importance, since the results obtained from its application can falsify the research, and thus, lead to fatal consequences in robust studies, in the social, constructive, and life of a patient. , among others.

In this work, dividing sections will be used; in practice it is a process that is presented as a system, where all its elements have an important function.

A first conception that has two phases is described below:

Phase 1: Generalities of validation

An instrument must meet two fundamental elements: validity and reliability, to match the gold standard instrument. If it does not exist, then it must meet a series of requirements to be reliable enough to accept the results in scientific research.

First: Validation involves two fundamental concepts, what has been applied up to this point? Is it good, surely? Second: How accurate is the new instrument to compare it with the one accepted by the scientific community, as correct in its measurements?

Phase 2: Internal validity

Validity is the degree to which an instrument measures what it is intended to measure. To obtain it, the instrument to be used must be compared with the ideal, gold standard or  Gold Standard .

Reaffirmed as a process, five sources of evidence have been postulated for it: according to the content, the internal structure, in relation to other variables, in the consequences of the instrument and in the response processes.

Reliability is the degree of congruence with which an instrument measures the variable. It is obtained by evaluating reproducibility, which is when there is a good correlation in the measurements at different times; and on the other hand, reliability, which is the accuracy of the measurements at different times. The application of both concepts is revealed in a recent article, where an instrument is validated with the purpose of being used in a study on tourist destinations in the province of El Oro, Ecuador.

When exploring the state of the art, the first thing to do is verify the existence of instruments applied in previous research, used for the same purpose, that have been validated at the time, as part of the investigative process. The most used tests, depending on the measurements of the variables, can be Student’s t  or Anova, if the data follow a normal distribution; otherwise, their non-parametric counterparts; Wilcoxon or Kruskal Wallis, in the case of two or three measurements, respectively, in both situations.

When there is no instrument that fits the objectives of the research, then it must be formed and contrasted with the ideal or gold standard.

In the second option, validity is very difficult to prove, since it has been decided to use an instrument different from those existing in the literature consulted.

Next, reliability is verified. For this, reproducibility is measured .  The instrument is applied several times (two or more) in samples that belong to the same universe or population where the research is carried out. To obtain a correlation considered good in the results (according to the Pearson, Spearman coefficients or the CCC coefficient of agreement) between the measurements, a value greater than 0.7 is accepted, although the ideal is 0.9.

For reliability, it is proven that in the different measurements, taken in the same universe or population, the responses of the subjects do not differ significantly, that is, there is accuracy in the instrument measurements at different times. The most used statistical tests are Aiken’s V and Dahlberg’s error. Therefore, validity is measured with another instrument, and reliability with the same one.

Other authors include the term optimization. It is associated with minimizing the error when providing a criterion, at the time of decision-making, based on the results obtained from the instrument.

In general sense, in the studies discussed it can be seen that there are several ways to carry out the validation of measurement instruments. The one that the researcher considers most appropriate can be used, but keeping in mind that the one selected meets all the necessary scientific rigor.

Below, a methodology will be shown to validate a measurement instrument, which is a hybrid between the conception of two different groups of authors, who are essentially similar.

Qualitative, which coincides with content analysis, is part of internal validity. To this are added the reliability and the construct, which belong to the quantitative, as well as the criterion, stability and performance. These last three correspond to external validity.

A second conception, which has six phases, in correspondence with  Supo ‘s idea , is described below:

Phase 1 : qualitative or content validation. It is part of internal validity. It is the creation of the instrument. It is divided into three moments, which do not have to follow an order, but are mandatory. It coincides with a type of diagnostic investigation.

• Approach to the population: its purpose is to investigate the problem being addressed, approach the units of analysis or variables that should be used in the research. To do this, interviews, population survey studies and others can be carried out to provide this information.
• Expert judgment: the selected experts are responsible for assessing whether the items in the instrument are clear, precise, relevant, coherent and exhaustive.
• Rational validity (knowledge): they must be concepts that have been searched in the literature. It is assumed that the researcher is knowledgeable about the topic being studied.

Phase 2 : quantitative or reliability. It is within the internal validity of the instrument.

This phase was detailed previously. According to  Aiken : “…strictly speaking, rather than being a characteristic of a test, reliability is a property of the scores obtained when the test is administered to a particular group of people, on a particular occasion, and under specific conditions.”