Common Pitfalls In The Research Process

Issues of Concern

There are 5 phases of research: planning, data collection and analysis, writing, journal submission, and the rejection/revisions/acceptance phases.

Phase I Pitfalls: Planning a Study

The highest yield of preempting pitfalls in the research process occurs in the planning phase. This is when a researcher can set the stage for an optimal research process. Below are pitfalls that can occur during the planning phase.

Pitfall: Underestimating what committing to a research project requires

Conducting a research study and achieving publication sounds fulfilling, right?

Consider the many steps: conducting a literature search, writing an IRB proposal, planning and having research meetings, long and cumbersome data collection processes, working with statisticians or analyzing complex data, having unexpected research setbacks (eg, subjects drop out or newly published papers on same topic), the possibility that after data collection you have no statistically (or clinically) significant findings, conducting an updated literature search, writing introduction, methods, results, and discussion sections of a paper, going through the many journal options to determine best fit while aiming for high impact factors, adhering to journal guidelines/fixing drafts, writing cover letters stating importance of the topic to respective journals, creating journal portal accounts, possibly being rejected numerous times, waiting months for journal decisions, working on numerous revisions and being informed by numerous individuals about all of the flaws in your writing and research.

Does it sound, maybe less fulfilling?

Conducting a research project from inception to publication can be a rewarding experience. Research requires significant time. Setbacks are normal. To produce an important and sought-after research product, an individual must understand the magnitude of commitment required.

Pitfall: Choosing the wrong research pursuit/topic lacks precision

Consider an investigator interested in substance use research. The first challenge is the immense amount of research already published on this topic. Fortunately, there is still a massive amount of uncharted territory in substance use research.

It is important to understand what has been done and what remains to be discovered in your area of research. Do not simply study a topic because you find it interesting; passion is advantageous, but you should ensure that your study contributes to a field/specialty or research in a significant way.

How does your research differ from what has been done?

How will it impact practice in a way that no previous study has?

Consider these questions when selecting a research topic. Otherwise, you may struggle to get the work published. It can be demoralizing if you have already written your paper and realize that your paper is not going to get accepted by a reputable journal due to the presence of other papers already describing the same concepts you have.

As always, the first step is a thorough literature search.

Pitfall: Not considering research bias

A common theme in the literature is that bias can, unfortunately, lead to failures to reproduce results, raising concerns about the integrity of science.[4] Bias can be considered various (inadvertent) poor strategies related to data design, analysis, and results reporting that produce spurious results and papers that perhaps should not be published.[5]

While one cannot completely eliminate bias from the research process, researchers should take steps to understand sources of bias in their studies and to minimize bias during the study's planning phase. 

Pitfall: Not focusing on which variables to collect

Researchers often seek to collect as much data as possible, but should not compile a list of variables that includes every detail about subjects if the variables collected are unlikely to yield insight into the research topic. The longer the data collection instrument, the higher the likelihood of (human) errors (if manual data entry) and the longer duration of the data collection phase. Instead of building a database with many variables, consider removing irrelevant variables and using the time saved to increase the sample size. Determine, based on your own clinical knowledge and published empirical works, which variables are most crucial. 

Pitfall: Worrying about the statistics after the data has been collected

A vital part of the research process is ensuring you have a rigorous statistical approach. Involve your statistician very early in the project, preferably in the planning stages. They have insight into the types of variables to collect and help shape the research methods. Statistical power is an important consideration prior to data collection to avoid false-negative results (Zlowodzki et al., 2006). Furthermore, other concepts, such as covariates, should be included in the planning phase. Do not wait until after the data collection phase to give data to the statistician, who cannot transform the data you have into the outputs you want.

Pitfall: Not setting defined author roles

It is important to define the designated authors at the outset of the research process to avoid conflicts. Do most people want to be an author? Sure. Does everybody do the work worthy of authorship? No. While placing general comments in the margin of a shared document may slightly improve the paper, it probably should not qualify for authorship. Review authorship criteria to determine what constitutes authorship. Clear expectations can ensure that everyone is aligned and that the process is perceived as fair, especially for individuals who plan to invest significant time in the project. Clear expectations for each author should occur before any writing begins, including deadlines and specific contributions.[6][7][6]

Pitfall: Not considering the limitations of the work before the paper is written

Avoid this pitfall by reviewing recent manuscripts and reading the limitations sections of these papers. Many of these limitations sections make notions about generalizability to other populations. Some discuss low power. Even the best papers in the top journals have many limitations. The best way to avoid or mitigate limitations in your work is to consider them during the planning phase.

How can you set up your project to limit your limitations section?

What (types of) samples should you include in your study?

Were you originally considering a retrospective design, or would it be prospective?

What steps can you utilize to control baseline characteristics between groups?

Consider all limitations and determine how to control them before data collection.

Phase II Pitfalls: Data Collection and Analysis

After planning, typically following institutional review board (IRB) approval, the data collection and analysis phase can proceed. The entire team should typically stay involved throughout these phases. Below are pitfalls to avoid.

Pitfall: Not being involved in the data collection phase

It is important to be involved with the data collection phase, even if you do not personally collect data. Train data collectors to ensure alignment and provide periodic oversight to ensure the accuracy and quality of the data over time.[8] Do not assume the data collection phase is going smoothly – you may find yourself with a huge dataset riddled with inconsistencies or errors. Schedule periodic meetings to review data.

Pitfall: Not being involved with the statistical analysis phase

If you are not conducting the statistical analysis, do not assume that the person who is analyzing the data is 100% on the same page. Have meetings about the data, how to interpret the data, and the limitations of the data. Ask what other ways the data could be analyzed and how reviewers might negatively critique the data itself or the statistical methods.

The person conducting the analysis may not have the same level of familiarity with the topic, and you may not be as familiar with the outputs. By understanding one another, you can produce clearer, more robust methods and results in sections of the paper; limit critiques of the statistical approach and data outcomes; better understand the research for presentations, discussion, or future work; and foster a positive collaboration for future work.

Phase III Pitfalls: The Writing Phase

The next phase is the writing phase. While this section covers pitfalls during the writing phase, for recommendations on conducting a literature search, writing, and publishing research, see StatPearls Evidence-Base Medicine Chapter: How to Write and Publish a Scientific Manuscript.[9] Below are pitfalls that can occur during the writing phase. 

Pitfall: Poor or outdated references

When writing your paper, perform multiple literature searches to ensure all recent, salient references are covered—claims about recent similar work or research that frames your study, if the references are outdated. Journals may even ask reviewers to comment on the presence or absence of up-to-date/suitable references. Conduct a literature search prior to data collection and stay on top of references throughout the research process as new papers become available.

Pitfall: No clearly defined purpose of the paper

Many aspects of manuscripts can get overlooked. Lack of a clear purpose statement can doom a paper to futility. Remind the readers of the goal of the project. You do not want consumers of your research to read the results section and forget what the goals/main outcomes are. The purpose statement should be located at the end of the introduction section. 

Pitfall: Unclear methods make research hard to reproduce

A common concern in science is the lack of transparency in methods, which undermines reproducibility. The methods section should allow a reader to understand exactly what was done and conduct the study. Consider examining the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) checklist for the methods (as well as other paper sections) to ensure best reporting practices for reproducibility.[10]

Pitfall: The tables and narratives are the same

Reviewers prefer you not to state findings in narratives that are in tables. Tables focus readers on the most important results and do not duplicate the written content. Make call-outs to the table in the paper's narrative sections, but do not state information found in tables. 

Pitfall: Not reporting all data/outcomes

Some authors state the main outcome of interest or have a statement such as “there were no other statistically significant findings between other groups.” Authors must report all outcomes and statistical analyses for the reproducibility of the research. While this may be difficult with a broad approach, use tables and appendices to report all outcomes, thereby enhancing transparency and reducing researcher bias.

Pitfall: Repeating results in discussion

Do not simply restate in the discussion what you already have in the results section. Utilize this section of the paper to link other references to your work and reflect on other empirical investigations' similarities or differences. Explain why your research provides an impactful contribution to the topic.

Pitfall: Making conclusions that do not align with your work

Authors sometimes note in their conclusions how the work impacts a topic due to X reason, when X may be too broad a claim, and the work doesn’t really support or prove that notion. Researchers should align their conclusions with their own results and highlight the significance of their findings.

Pitfall: Thinking the title is not a big deal

A strong title helps with the impact/readership of your paper. Consider keeping a short title that provides the main takeaway. Papers with more concise titles and that present the study's conclusion receive greater impact and receive more citations.[11]

Pitfall: Completing the abstract at the last minute

Similar to the title, do not underestimate an abstract. Journal and conference reviewers (and the general audience) may only read your abstract. The abstract must present the study's key results and contributions and be well written.

Phase IV Pitfalls: Submitting to a Journal

After the paper has been written, it is time to choose the journal. This phase also has numerous pitfalls. Below are pitfalls that can occur during this phase.

Pitfall: Choosing the wrong journal

Choosing a journal for your work can be overwhelming, given the number of options. Always look at the aims and scope of prospective journals. Look through the author guidelines to ensure that your manuscript adheres. This saves time. Review your reference list for any journals that appear more than once; if so, consider submitting to that journal. You do not want to submit your paper, wait 2 weeks, and then get a desk rejection because the editors state the paper is not aligned with the journal's aims and scope.

Additionally, researchers may aim too high and spend months (and numerous hours on journal submission portals) attempting to publish a manuscript in a journal with a very high impact factor. Although admirable, if the research design and results lack “gold standard” reporting, authors should consider submitting to a journal more likely to accept. Find a balance between the quality of your paper and the quality of the journal. Seek feedback from the other authors and/or senior colleagues who can provide honest feedback.

Pitfall: Poor cover letter on journal submission

Do not submit work with a flawed cover letter (errors or lack of clarity in how your work contributes to the body of literature). Spend time writing a detailed cover letter once, have it edited by a peer, and use it for all future projects. Highlight the differences (eg, the purpose of this work and our results) for each project. Use the cover letter to highlight the study's significance while adhering to disclosure guidelines (eg, conflicts of interest, authors' contributions, or data releases). This helps the editorial board determine not only the paper's suitability for the journal but also streamline the review process.[12]

Pitfall: Assuming that after the paper has been submitted to a journal, the work is done

The paper has been submitted! You think you are finished…but, unfortunately, the publishing game may still be far from over. Researchers often do not recognize the time required for the submission/rejection/revisions phases. Revisions can sometimes be total overhauls, more work than writing a whole new paper. Be prepared to continue working.

Phase V Pitfalls: The Rejections, Revisions, and Acceptance Phase

Finally, perhaps the most unpredictable phase, the rejections, revisions, and acceptance phase, has unique pitfalls and other obstacles.

Pitfall: Mourning rejections too long/“sitting on” a rejected paper

Did you get a desk to reject (ie, the manuscript was not even sent for blind review)? That is unfortunate but common. You do not have time to sulk. Get that paper submitted somewhere else. The older the data, the less desirable your paper becomes. If the paper went in for a full review and was rejected, that may be even tougher than a desk reject because more time has elapsed. The good news is that (hopefully) you received feedback to incorporate in a revision. Do not spend too much time grieving rejections.

Pitfall: Not laying to rest rejected papers when it is indeed their time to go

Did you write a paper a couple of years ago, and have you submitted it to 20 different journals? The data is getting old. The topic wasn’t focused on. The sample size was small. Perhaps the project is no longer worth pursuing. Do not give in to the sunk cost fallacy. If, however, you are proud of the work and stand by the paper, do not give up. If you believe that, after the numerous rejections, the topic/project is flawed, you can use this failure as a personal learning/growth opportunity. Do not repeat controllable mistakes on future projects.

Pitfall: Not addressing all of the reviewer feedback

Did you get a revise and resubmit? Great news! The reviewers and editors may ask you to respond to each comment when you resubmit. Address all of the reviewer feedback. Take your time reading through the feedback, digest it, and re-read it. Carefully respond and decide how to revise your manuscript based on the feedback. Share the reviews and the duties of revision with coauthors. In your response to reviewers, stay professional and address each statement, even if you disagree with what is stated. If you do not respond to each statement, the reviewers often highlight the concern(s) again.

Pitfall: Thinking you know what the reviewers are going to say

Research reviewers are like a box of chocolates. You never know what you are going to get. You may be worried about a section of your paper/research approach, and the reviewers do not mention it at all in their review; instead, they criticize a section of your manuscript that you are most proud of.

In some reviews, you may get feedback like the following:

Reviewer #1

Please change lines 104-108, as I believe they are irrelevant to your study.

Reviewer #2

Please build on lines 104-108, as I believe they are the foundation of your study.

Sometimes, after multiple revisions, reviewers raise new concerns. This can be disheartening. Should some regulations restrict reviewers from bringing up new ideas/concerns during revision #7? Perhaps. Does any current rule prevent them from doing this? No.

During the review process, we must have faith that the reviewers are knowledgeable and provide fair, insightful, and constructive feedback. While the review process can be arbitrary or frustrating in some cases, peer review remains the gold standard in a scientific publication. Stay positive and persistent. Stay professional in responses to the reviewers. Remember that the review process can be very beneficial as it often leads to feedback that truly elevates your work and makes the product (and you) look better.[13]

Pitfall: Not rewarding yourself for a published paper

You did it! Celebrate your accomplishment. Reflect on the merit of your effort before you move on to other work or re-enter the cycle of IRBs, data coding, and journal submissions. Remember and appreciate how remarkable it is that you just contributed knowledge to the world.

Clinical Significance

Many pitfalls can occur throughout the research process. Researchers should understand these pitfalls and employ strategies to avoid them, thereby producing high-quality, sought-after research results that are useful for basic science and clinical practice.