Evidence Standards (wip)

Note: This operations doc will be developed in tandem with the Presenting Evidence and Fact Checking Guide. All three docs are currently in WIP (Work in Progress) status.

Introduction

At vBriefings.org, part of our mission is to give readers trustworthy briefings on complex issues, some of which may be emotionally charged. To do that well, we must be honest about our own capabilities. As a team, we are not trained to evaluate scientific research at a professional level. We would not pass a research‑evaluation test, and we cannot reliably judge whether a study is well‑designed, biased, or making claims it cannot support.

Research evaluation is a specialized skill that requires formal training in research and causal inference. Most people, including most writers and editors, do not have that training.

In a world saturated with misinformation and disinformation, not acknowledging this limitation would be irresponsible. This document is an attempt to acknowledge our limitations while still accurately reporting on topics that need to be covered due to the nature of our platform.

Going Forward

Our current briefings do not fully meet the standards outlined here. We have always been deeply concerned about factual accuracy, and that concern is reflected in the fact that we have paid independent credentialed fact‑checkers to review our briefings. This already places us above the standards used by many sites — and in some cases, far above them. But it does not go far enough.

Going forward, we will apply these new evidence standards to all new briefings we produce. As we continue to grow, we will also revise existing briefings to bring them into alignment with these standards. We want our readers to know that when they come to vBriefings.org, they are receiving information grounded in expert consensus, ethical clarity, and methodological humility.

Vegan Bias

We openly promote ethical veganism. It is part of our mission, our values, and our public identity. We believe that avoiding participation in the exploitation of animals, as far as reasonably possible, is an ethical position worth advancing. But our commitment to ethical veganism does not give us permission to distort evidence, overstate claims, or selectively interpret research. If we did that, we would undermine our own credibility and contribute to the same misinformation ecosystem we are trying to counteract.

Because we are not trained to evaluate scientific research, we must be especially careful not to let our ethical commitments turn into bias. We cannot dismiss evidence simply because it complicates our narrative. We will let the chips fall where they may. If expert consensus supports a claim that aligns with ethical veganism, we will report it. If expert consensus contradicts a claim commonly made within vegan advocacy, we will report that too.

Ethical veganism is our value system, not our filter for evidence.

General Guidelines

To uphold this standard, we will rely on testimony from credentialed scientists, consensus‑based literature reviews, and expert summaries rather than interpreting individual studies ourselves. This protects us from accidentally amplifying weak or flawed research simply because it appears to support veganism. It also ensures that our briefings remain trustworthy to readers who may not share our ethical commitments.

We will be transparent when evidence is mixed, uncertain, or evolving. We will avoid overstating health claims, environmental claims, or scientific conclusions about veganism. We will distinguish clearly between ethical arguments and empirical claims, and we will make clear when we are speaking from values rather than scientific evidence.

In short, we will advocate for ethical veganism without compromising our evidence standards. We will not allow our mission to become a source of bias. We will let the chips fall where they may, and we will trust expert consensus to guide the empirical side of our briefings.

Evidence Standards Rules

We will not interpret individual scientific studies ourselves. Single studies are fragile and easy to misread. Without methodological training, we cannot determine whether a study’s design supports its conclusions, whether its statistics are valid, or whether its results are meaningful.

We will rely on testimony from credentialed, respected scientists. Experts can evaluate research in ways we cannot. When we quote them, we are reporting the conclusions of people who actually know how to evaluate scientific evidence.

Note: This does not represent an “appeal to authority” bias; it is a recognition that scientific expertise is a skill set built on training, methodology, and experience. In fields where we lack that training, deferring to qualified experts is not a logical fallacy — it is the responsible way to avoid misinterpretation and prevent misinformation. We are not asking readers to trust experts blindly; we are grounding our briefings in the consensus of people who are professionally equipped to assess the quality, validity, and limitations of scientific research.

We will selectively use online research assessment tools to evaluate the truthfulness of factual statements and the quality of evidence. Tools like consensus.app synthesize multiple studies, assess methodological strength, and present conclusions grounded in expert review rather than isolated findings. By relying on consensus‑based platforms, we reduce the risk of cherry‑picking, avoid the pitfalls of interpreting single studies, and ensure that our briefings reflect the broader scientific landscape rather than outlier results. These tools help us anchor our work in vetted, stable evidence and protect us from unintentionally amplifying flawed or misleading research.

We will treat research as something to report, not something to interpret. Our role is to translate expert consensus into clear language, not to act as researchers ourselves.

We will be transparent about uncertainty. When experts disagree, when evidence is mixed, or when research is still emerging, we will say so. We will not present uncertainty as certainty.

We will prioritize accuracy over speed. If we cannot verify a claim through expert consensus or credible review, we will not include it. vBriefings.org is a place for careful, responsible communication — not rapid commentary.

We will maintain strict separation between ethical commitments and empirical claims. Ethical veganism guides our values, not our interpretation of evidence. Empirical claims will be grounded in expert consensus, even when they challenge assumptions within vegan advocacy.

Supplemental Information

Research EvaluationCompetency Test

This test will help you determine if you are qualified to evaluate research results.

Questions

1. A study measures exposure and outcome at the same single time point and claims the exposure caused the outcome. What fundamental requirement for causal inference is violated?
2. A study uses a longitudinal design but measures the predictor only once and the outcome multiple times. What key question must be asked about the stability of the predictor?
3. A study finds that people who take supplement X are healthier, but supplement users are wealthier and exercise more. What is the core validity threat?
4. A study uses self‑reported dietary intake collected once at baseline to predict disease 20 years later. What is the biggest methodological concern?
5. A study compares two treatments, but 30% of participants dropped out of one group. What issue must be evaluated before trusting the results?
6. A study claims causation but uses a cross‑sectional design. What specific limitation of cross‑sectional data prevents establishing causal direction?
7. A study finds a statistically significant effect, but the confidence interval is extremely wide. What does this imply about the estimate?
8. A study reports dozens of outcomes, but only the significant ones are discussed. What evaluation question should you ask?
9. A study uses a convenience sample of college students to make claims about all adults. What validity issue is present?
10. A study shows a correlation between screen time and anxiety, but anxiety was measured after screen time. What key question should be asked about temporal ordering?
11. A study uses different measurement instruments for different groups. What threat does this pose to group comparisons?
12. A study’s control group receives no attention from researchers, while the treatment group receives weekly check‑ins. What bias might this introduce?
13. A study claims an intervention works, but the control group was aware they were not receiving the treatment. What evaluation issue arises?
14. A study uses a pre‑post design without a control group. What is the biggest threat to interpreting improvement?
15. A study finds a significant effect, but the sample was not randomized. What question must be asked about group differences?
16. A study uses a survey with leading questions. What type of bias should be evaluated?
17. A study finds that people who meditate have lower stress, but meditation was self‑selected. What evaluation concern is present?
18. A study uses a very short follow‑up period to evaluate long‑term outcomes. What limitation should be considered?
19. A study’s results differ dramatically between subgroups, but subgroup analyses were not pre‑registered. What issue should be flagged?
20. A study reports a p‑value of 0.049 and claims strong evidence. What should be questioned about the strength of the finding?

Answer Key

1. A study measures exposure and outcome at the same single time point and claims the exposure caused the outcome. What fundamental requirement for causal inference is violated?
   
   Correct answer: Temporal precedence
   
   Explanation: To infer causation, the cause must be shown to occur before the effect. Cross‑sectional data cannot establish which variable came first, so causal direction cannot be determined.
2. A study uses a longitudinal design but measures the predictor only once and the outcome multiple times. What key question must be asked about the stability of the predictor?
   
   Correct answer: Whether the predictor remains stable over time
   
   Explanation: If the predictor changes after the single measurement, later outcomes may not be related to the original value, undermining the validity of the longitudinal inference.
3. A study finds that people who take supplement X are healthier, but supplement users are wealthier and exercise more. What is the core validity threat?
   
   Correct answer: Confounding
   
   Explanation: Wealth and exercise influence both supplement use and health outcomes, creating a spurious association that cannot be interpreted as causal.
4. A study uses self‑reported dietary intake collected once at baseline to predict disease 20 years later. What is the biggest methodological concern?
   
   Correct answer: Measurement error and instability over time
   
   Explanation: Diet changes substantially across decades, and self‑report is unreliable. A single baseline measure cannot validly represent long‑term exposure.
5. A study compares two treatments, but 30% of participants dropped out of one group. What issue must be evaluated before trusting the results?
   
   Correct answer: Differential attrition
   
   Explanation: If dropout rates differ between groups, the remaining participants may no longer be comparable, biasing the treatment comparison.
6. A study claims causation but uses a cross‑sectional design. What specific limitation of cross‑sectional data prevents establishing causal direction?
   
   Correct answer: Lack of temporal ordering
   
   Explanation: Cross‑sectional studies measure all variables at once, making it impossible to determine which variable preceded the other, a requirement for causal inference.
7. A study finds a statistically significant effect, but the confidence interval is extremely wide. What does this imply about the estimate?
   
   Correct answer: The estimate is imprecise
   
   Explanation: Wide confidence intervals indicate high uncertainty, meaning the true effect could be much smaller or larger than the point estimate suggests.
8. A study reports dozens of outcomes, but only the significant ones are discussed. What evaluation question should you ask?
   
   Correct answer: Whether multiple comparisons were corrected
   
   Explanation: Testing many outcomes inflates the chance of false positives. Without correction, significant results may be meaningless.
9. A study uses a convenience sample of college students to make claims about all adults. What validity issue is present?
   
   Correct answer: Limited external validity
   
   Explanation: College students differ systematically from the general population, so results cannot be generalized broadly.
10. A study shows a correlation between screen time and anxiety, but anxiety was measured after screen time. What key question should be asked about temporal ordering?
    
    Correct answer: Whether screen time preceded anxiety
    
    Explanation: To infer directionality, the cause must occur before the effect. If timing is unclear, causal interpretation is invalid.
11. A study uses different measurement instruments for different groups. What threat does this pose to group comparisons?
    
    Correct answer: Measurement non‑equivalence
    
    Explanation: If instruments function differently across groups, observed differences may reflect measurement artifacts rather than true differences.
12. A study’s control group receives no attention from researchers, while the treatment group receives weekly check‑ins. What bias might this introduce?
    
    Correct answer: Attention bias
    
    Explanation: Extra contact can influence outcomes independently of the treatment, contaminating the comparison.
13. A study claims an intervention works, but the control group was aware they were not receiving the treatment. What evaluation issue arises?
    
    Correct answer: Expectancy effects
    
    Explanation: Knowing one is in the control group can alter behavior or outcomes, biasing the results.
14. A study uses a pre‑post design without a control group. What is the biggest threat to interpreting improvement
    
    Correct answer: History and maturation effects
    
    Explanation: Changes over time may be due to external events or natural development rather than the intervention.
15. A study finds a significant effect, but the sample was not randomized. What question must be asked about group differences?
    
    Correct answer: Whether groups differed at baseline
    
    Explanation: Without randomization, pre‑existing differences may explain the observed effect rather than the treatment.
16. A study uses a survey with leading questions. What type of bias should be evaluated?
    
    Correct answer: Response bias
    
    Explanation: Leading questions influence how participants answer, distorting the data.
17. A study finds that people who meditate have lower stress, but meditation was self‑selected. What evaluation concern is present?
    
    Correct answer: Selection bias
    
    Explanation: People who choose to meditate may differ systematically from those who do not, making causal inference impossible.
18. A study uses a very short follow‑up period to evaluate long‑term outcomes. What limitation should be considered?
    
    Correct answer: Insufficient follow‑up
    
    Explanation: Short follow‑up cannot capture long‑term effects, making conclusions about durability invalid.
19. A study’s results differ dramatically between subgroups, but subgroup analyses were not pre‑registered.
    
    What issue should be flagged?
    
    Correct answer: Data dredging
    
    Explanation: Unplanned subgroup analyses increase false positives and often produce unreliable findings.
20. A study reports a p‑value of 0.049 and claims strong evidence. What should be questioned about the strength of the finding?
    
    Correct answer: The fragility of the result
    
    Explanation: A p‑value barely below 0.05 indicates weak evidence; small changes in data or assumptions could eliminate significance.