Introduction

Reasonable health care is possible due to the availability of knowledge in the field of symptoms, pathogenesis, diagnosis, prognosis, and treatment of diseases. Conducting research, especially clinical trials, provides valuable evidence about treatments and interventions (1). The clinical trial is one of the types of studies that are important for developing science (1-2). These studies are the most reliable methods of examining the effects of treatment and care interventions on human subjects and are widely used in clinical research. (3). Although the number of these types of studies is increasing and their results are as gold standard data as the basis for many healthcare professionals’ decision-making (4-6), evidence has shown that many clinical trial studies are poorly designed (7). The quality of the designing randomized clinical trials significantly impacts the reliability of their results. Poorly designed clinical trial reports can exaggerate treatment outcomes by as much as 30%, which may result in treatment failures for patients in the future (8-9). The ongoing enhancement of study methods is vital for advancing science and research, and it poses a significant challenge in translating scientific findings into clinical practice (10-13). Critical appraisal of evidence-based reports is one of the basic principles of evidence-based medicine, which helps in clinical evaluation to select the best evidence provided (14-15). Therefore, assessing the quality of studies, especially clinical trial studies, is interesting to researchers (16).

Throughout the world, several checklists and criteria have been designed and introduced to evaluate the quality of the design and report of randomized clinical trials (17-18). The CONSORT checklist is one of the most reliable tools for critical appraisal of the quality of clinical trials (9, 18). In addition to identifying weaknesses in clinical trials, this protocol is a guide to clarify how to report study results (19). The items in this checklist focus on different parts of a study. These items provide necessary standards for designing, analyzing, and interpreting trials (20). Evidence has shown that many published clinical trials are not of good quality. In this regard, Schulz et al. (1994) reported that clinical trials conducted in obstetrics and gynecology are of poor quality (21). Meanwhile, the quality of clinical trials in herbal medicine has been reported to be less than optimal (22). Herbal medicine as a branch of complementary medicine is accepted by 80% of people (23). But due to the poor design and report of studies related to herbal medicine, their effectiveness is controversial (24). In gynecology and obstetrics, one of the common complications that can cause problems for the mother and the infant is nipple fissure. Various topical chemical products such as creams, lotions, and ointments are used to prevent and treat nipple fissures, none of which have a scientific basis. Most of these methods are unsuccessful in treating and preventing nipple fissures. Therefore, the tendency to use complementary and herbal medicine for nipple fissures treatment has increased (25). But the research conducted in this area has not been evaluated in terms of quality, and there is not enough information about the robustness of the design and quality of these studies. Therefore, for the widespread use of these treatments, it is necessary for the stakeholders, including patients, healthcare providers, and policymakers, to have access to accurate and reliable evidence in this field (26, 27). As a result, considering the importance of the health of mothers and infants and emphasizing exclusive breastfeeding due to its benefits, the treatment of nipple fissures is necessary. Also, considering the importance of clinical trials in complementary and herbal medicine and its probable effect on wound healing and pain relief caused by nipple fissures, it is necessary to check the quality of the report and methodology of these studies. Therefore, this study was conducted to evaluate the quality of reports of clinical trials published regarding the effect of herbal products on the healing of nipple fissures in lactating women.

Materials and Methods

This review and critical assessment aimed at the critical appraisal of the quality of clinical trials on the effect of herbal products on the improvement and prevention of nipple fissures in lactating women, which was conducted in 2025. English databases including PubMed, Scopus, Web of Science, Cochrane Central, ProQuest, as well as Persian databases of Magiran, and SID were searched to retrieve articles. To search these databases, the keywords matching Mesh including؛ medicinal plant, complementary therapies, herbal therapy, herbal medicine, breast fissure, Sore nipple, Nipple fissure, breast wound, and all their possible combinations using AND and OR boolean operators and their Persian equivalents were used. Databases were searched by two authors, separately until February 2025 without any time limitation. In the current study, the review was reported according to the protocol recommended in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. A manual search was also conducted in the sources of the extracted articles. After extracting 783 articles from the databases, duplicate cases were removed using ENDNOTE (X8 Toronto, Canada) suitable for Windows. The remaining 417 articles were reviewed separately by two authors. In the first stage, the title and abstract, and in the second stage, the full text of the articles was reviewed.

 The inclusion criteria of this study included clinical trial designs that investigated herbal approaches in the treatment and prevention of nipple fissures and pain. Also, studies that provided reliable data for extraction. The exclusion criteria in this study included other types of documents such as dissertations, presentations at conferences, study protocols, letters to the editor, reviews of non-herbal compounds and studies published in the languages other than English and Persian. After removing irrelevant studies, 68 potentially relevant articles were reviewed. Forty-six articles were excluded due to non-compliance with the inclusion criteria and lack of access to the original article. Finally, 21 articles entered the critical appraisal process. The flow chart of selecting articles is shown in Figure 1.

CONSORT 2010 checklist was used for critical appraisal. Moberg-Mogren et al. (2012) examined the validity and reliability of the CONSORT 2010 checklist in occupational therapy clinical trials. They evaluated two independent evaluators in reviewing the articles. In their study, the percentage of agreement between raters was reported between 63% and 100%, and Kappa coefficients between -0.032 and 1.00. They also found that 77% of the Consort checklist items showed a high ICC (ICC>0.7) (28), which indicates the validity and reliability of the CONSORTChecklist in evaluating the quality of clinical trials.

CONSORT checklist (2010) includes 25 items to evaluate six sections, title and abstract, introduction, method, findings, discussion, and other information. Each item has a sub-set of questions, a total of 37 questions in the checklist used to evaluate all parts of the article. The selected articles were given a score (1 or 0) for each item based on a two-level scale. In this way, if the desired item is mentioned in the text, a score of one and if not, zero were scored (20). Therefore, the maximum and minimum scores were 37 and zero. After selecting the articles, their full text was carefully read by each author and scored based on the CONSORT checklist. In the case of disagreement on scoring between the two authors, a consensus was reached upon the definitive score for those items. Data analysis was conducted using SPSS software (version 25). Descriptive and analytical tests were used to report the results. The difference in scores of different sections of articles was examined based on the number of authors and the time range of publication of the articles.Because the data were not normal, the nonparametric Kruskal-Wallis test was used to examine the difference.

Results

In this critical assessment, 21 selected articles on the effect of herbal products on the improvement and prevention of nipple fissures were critically appraised. The publication period of the articles was from 1997 to 2024. Two (9.52%) articles were published before 2010, 5 (23.8%) between 2010-2015, and 14 (66.68%) between 2016-2024. The distribution of articles in terms of publication language included fifteen (76.20%) articles in English and 5 (23.80%) articles in Persian. Among the included articles, 5 (23.8%) articles were published in the first quartile (Q1) journals, 4 (19.05%) in the second quartile (Q2) journals, 4 (19.05%) in the third quartile (Q3) journals, 3 (14.3%) in the fourth quartile (Q4) journals, and 5 (23/8%) articles were published in journals without quartile ranking. In preparing the report of findings of included studies, 1-3 authors participated in 11 (52/38%) articles, and 4-6 authors participated in 10 (47/62%) articles. The results of the critical appraisal of the quality of the articles based on all items of the CONSORT 2010 checklist are presented in Table 1.

 The overall compliance of the quality of different parts of the included articles with the CONSORT checklist criteria is shown in Table 2 and Diagram 1.

The highest compliance was in the introduction (100%), and the lowest was related to the results section (57.61%) and the other information (49.20%). In 9(42.9%) articles, the term clinical trial was not mentioned in the title. Among the included studies, 19 (95.2%) articles had structured abstracts. All included articles reported specific objectives and hypotheses in the introduction. In the methods section, 8 (38.1%) of the studies did not mention randomization, and 12(57.1%) articles did not mention the type of randomization and randomization sequence mechanism. Also, 14 (66.7%) articles did not report blinding, and 15 (71.4%) studies did not report the method of allocation of participants to interventions. In the results section, 16(76.2%) of the included studies did not report the estimated effect size and precision. Also, 14(70%) articles did not mention subgroup analysis, and 14 (66.7%) did not report important side effects. In the discussion section, 16(76.2%) articles did not report the external validity and applicability of the results. In relation to the important information section, 9(42.9%) articles did not mention the registration number and name of the trial registry. Also, 11(52.4%) articles did not report financial sources supporting the study. The scores in different sections of articles based on the CONSORT checklist are reported in Tables 1, 2 and Figure2.

To investigate the effect of the year of publication and the number of authors participating in the design of the clinical trial and preparation of the reports of the included study on the quality of articles, the normality of the data was checked with the Kolmogorov-Smirnov test (the score of each part of the study from the CONSORT checklist). The data distribution was non-normal (P<0.05). The Kruskal-Wallis test showed no statistically significant difference between the scores of the different parts of the articles and the number of authors (P<0.05). Also, the Kruskal-Wallis test showed no statistically significant difference between the scores of different parts of the articles and the year of publication (P<0.05). A summary of the included trials is shown in Table 3.

Discussion

In the present study, the clinical trials conducted on the effects of herbal products on the improvement and prevention of nipple fissures in lactating women were critically evaluated. The average overall score of the articles based on Consort 2010 criteria was 23.5 ± 4.1(score range: 13-29), which shows that the quality of the design and report of the studies is not appropriate. Researchers' inadequate knowledge about the standard guidelines for reporting clinical trials can lead to poor reporting. Sharifi F KM, Latifnejad Roudsari R. (2021) also reported the quality of clinical trials on complementary medicine in infertility as moderate. In their study, the average overall score of clinical trials based on CONSORT 2010 criteria was 22.68 ± 6.17 (score range: 9-33) (29). Also, Alirezaei S and Latifnejad Roudsari R. (2022) reported the overall average score of clinical trials on the use of herbal products in reducing pregnancy striae as 20.86 ± 7.18 (score range: 9-33), which is similar to the results of the present study. On the other hand, Khojazadeh et al. (2013) reported the overall average score of clinical trials in gynecology and obstetrics based on CONSORT 2010, 31.35 ± 3.18 (score range: 19-37) (31), which is different from the results of this study. The reason for the difference in the results can be attributed to the type of articles selected for review. Khojazadeh et al. evaluated the first quartile journals (Q1), and in the present study, the Persian and English journals indexed in all four quartiles (Q) were examined. In the present study, only 25% of the evaluated articles were published in Q1 journals. In the current research, the overall compliance of the articles with CONSORT criteria was 63.1%. This is consistent with the results of the study by Sharifi F KM, Latifnejad Roudsari R. (2021) (29) and Irani et al. (2017) (32). In the study of Alirezaei and Latifnejad Roudsari, (2022) the overall compliance rate was 46% (30). This difference can be related to the difference in the version of the CONSORT checklist used in the two studies. Alirezaei and Latifnejad Roudsari. used the 2018 version of the CONSORT checklist. This version has an additional section titled “stakeholder investments” with three items. Most published research did not report these items. In the present study, the compliance percentage of two parts, the important information, and the results, was minimal. The major weakness of the important information section was related to sthe tudy registration and financial sources. These results are similar to the study of Irani et al. (2017) (32) and Alirezaei and Latifnejad Roudsari (2022) (30). The reason could be related to the policies of journals that publish clinical trials and their decision to use the CONSORT checklist in evaluating the quality of articles before publication.

In the current study, one of the weak points in the methods section was related to the randomization. In clinical trials, randomization is one of the basic principles. In a clinical trial, it is necessary to mention whether or not to generate randomization. In addition to the mechanisms used to generate the randomization sequence, the point that who has created the randomization sequence should also be reported (33). In the present study, most of the articles did not mention items such as random allocation (38.1%), randomization sequence (57.1%), and the method of assigning participants to the intervention and control groups (71.4%). These results are similar to the studies of Irani et al. (2017) (32), Khojazadeh et al. (2013) (31), and Sharifi F KM, Latifnejad Roudsari R. (2021) (29). Another principle of clinical trials is the random allocation of participants to study groups, which can reduce the risk of bias in the study. For this issue, there are different methods for randomization (33). Sometimes, non-standard methods such as using even and odd numbers or using days of the week as a randomization method were reported in the studies (34-36). To solve this problem, standard protocols in clinical trial design, including CONSORT 's checklist, can be used. Also, using the advisory opinion of statisticians is helpful.

 Blinding can prevent bias in studies that test the effect of different theraputic and care approaches, especially during the application of those methods and the time of assessing their impacts (37). In the present study, 35% of the studies reported blinding. In study by Alirezaei and Latifnejad Roudsari (2022), 50% of articles (30) in the study of Sharifi F KM, Latifnejad Roudsari R. (2021), 40% of articles (29), and in study conducted by Irani et al (2017), 15% of studies (32) mentioned this issue. In study by Khojazadeh, 61% of articles reported blinding (31). The possible reason for this difference can be the selection of articles from journals with different indexes. In general, the low percentage of blinding in the studies suggests the inadequate knowledge of authors and lack of attention to the importance of implementing and reporting these parts. In the current study, the included articles, the statistical methods used, and the comparison of the groups in terms of primary and secondary outcomes were favorably reported, which is similar to the study of Sharifi F KM, Latifnejad Roudsari R. (2021). (29) and Sarailo et al. (38). In the results section, most of the items are reported as optimal. Complications of interventions were reported in only 35% of articles. This result is similar to the study of Sharifi F KM, Latifnejad Roudsari R.(2021) (29). For the clinical application of the results of clinical trials that investigate the effect of medicinal plants, side effects are one of the main items. Therefore, the non-reporting of the side effects of these medications shows a considerable weakness in reporting. In the discussion section of the studies, the report on the generalizability of the results was inappropriate. Also, the report of clinical trial registration number and financial resources were not favorable. These results are similar to the study by Sharifi F KM, Latifnejad Roudsari R. (2021) (29) and Sarayloo K and Latifnejad Roudsari R (2018) (38). In general, the results of this study showed that the quality of clinical trials of herbal products in the improvement and prevention of nipple fissures is moderate, and they are not of good quality, which is due to inadequate knowledge and lack of attention to the importance of standard guidelines of trials. Because the evaluations of other designs of studies also indicate their quality is less than optimal (39-40), researchers must follow and be familiar with standard checklists for research design and reporting.

One of the limitations of the present study was the lack of access to some databases, including Embas, for searching. Therefore, there may be other studies that we did not have access to appraise. One of the strengths of our study was following the principles of the systematic review protocol, which allows the replication of the study.

Conclusion

According to the findings of the present study, the quality of published articles in relation to the effect of herbal products on the healing of nipple fissures in in lactating women based on the CONSORT checklist is not optimal. Paying attention to the health of women and newborns and encouraging and promoting exclusive breastfeeding is very important. Because the results of clinical trial studies are the basis of evidence-based medicine. On the other hand, the tendency to use complementary medicine, especially medicinal plants, is high, so it is necessary to design and report research in this field based on standards of the correct method. This is possible through training researchers and journal editors to use standardized and coordinated guidelines and checklists for research design, implementation, and reports.

Declarations

Acknowledgements

The authors would like to appreciate the researchers whose articles were used in this critical appraisal review.

Conflicts of interest

Authors declared no conflicts of interest.

Ethical considerations and ethical approval

Not applicable.

 Funding

Not applicable.

 Authors’ contribution

FG and FH designed the study. FG and FH performed the literature search. FH and FG performed the qualitative assessment of the studies based on the CONSORT checklist. FG and FH were responsible for extracting data. FG wrote the manuscript. All authors read and approved the final manuscript and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved’.