Can and Should ART Centers Be Compared? NO!

There are a large number of reasons this question can only be answered in the negative. These reasons address issues regarding both "can" and "should." The first question is whether ART centers should be compared.

The Fertility Clinics Success Rate and Certification Act of 1992 (Wyden Law) was proposed by Congressman Wyden because of concerns that the success rates being reported in the 1980's were not accurate and that not all centers were reporting data. This law has now been passed and implemented. It requires clinics to report annually their clinic-specific success rates and to have a report published annually that lists these results. It also requires a listing of clinics that do not report. It was not intended initially or subsequently to provide direct comparative data among clinics.

Furthermore, the Centers for Disease Control, in recognizing the nature and purpose of the report, have agreed with SART to print the following statement on every page of clinic-specific results:

A comparison of clinic success rates may not be meaningful because patient medical characteristics and treatment approaches may vary from clinic to clinic.

Therefore, the CDC, which is responsible for the publication of the data as required by law, does not think ART Centers should be compared.

ASRM and SART, having reviewed this issue in detail over many years, have developed a policy titled "Guidelines for Advertising for ART programs." As this policy states, "because comparison of success rates between practices is invalid, using SART Clinic Specific Data for advertising/marketing that ranks or compares clinics or practices is unacceptable and is not permitted." SART believes this is important, as clinics that advertise inappropriately can lose their memberships.

RESOLVE, the national infertility association representing consumers, participates with SART/ASRM and the CDC in publication of the annual report, and also adheres to the same contents and format. Recognizing that the report is not intended for this purpose, they also state in information given to patients that the comparison of clinics is problematic.

Why would the CDC and ASRM/SART and RESOLVE make such strong statements? The reason is that the report should not and cannot provide the information necessary to make comparisons. Reasons for this are multiple.

First, the report is restricted to one page of information on each of 360 clinics in the United States. This limits the amount of information that can be provided about any individual clinic, especially because the data need to be categorized by age. There is simply insufficient information to compare clinics.

The report currently lists total live birth rate per cycle start as the success rate. Because of the risk to babies and mothers of high-order multiples, many people now feel that the report should list live births of singletons as a measure of success and list triplets and higher-number births as complications. The issue of whether twins are a success or complication is heatedly debated, and with no consensus being reached, again because of the notable (though not dramatic) increase in fetal morbidity and mortality. Therefore, the current report can be misinterpreted as to what "success" really is. This problem with the report is currently being addressed as the format of future reports is developed.

In addition, there are serious problems with data collection methodology. These include variability in who collects the data, follow-up on outcomes and abnormalities in babies. Abnormalities are serious adverse events that cannot be precisely determined from the current report. Factors that make this so difficult include:

• variability in definitions

• self-reporting

• under-reporting

• ascertainment bias

• different ages of male and female patients,

• choice to abort or not to abort

• effect of racial heterogeneity

• socioeconomic factors

• major versus minor malformations

• multiple malformations in one child

• inclusion or exclusion of abortus and stillbirth abnormalities

• differences in coding data

• sampling errors

• variation in reporting.

Another data collection problem is the time it takes from the year the ART cycle occurs until the time it is reported. The reasons for the delay are several, including the need to wait ten months until all the pregnancies from the year are delivered, two months for the clinics to finish collecting and submitting the data, four months to verify the data (including validation visits to 10% of the clinics) and several months for the CDC to prepare and approve the report for publication. Thus, it is at least two years behind the actual time of treatment. In that period of time, much can change in a clinic, including physicians, embryologists, other staff, patient population being seen, new technologies, protocols, nature of services provided and others, any one of which could have a significant impact on pregnancy rates in that program.

There are also problems with statistical limitations of the data. This occurs largely because over half the programs perform fewer than 100 cycles per year. This is too small a number to be able to determine a precise pregnancy rate. Naturally occurring random variation in pregnancy rates results in extremely wide standard deviations or confidence intervals, making almost all comparisons statistically meaningless.

This problem is further compounded when the data are categorized into smaller groups by age and the type of ART procedure performed. Only approximately 5% of clinics have significantly higher or lower pregnancy rates than other clinics based on this problem alone, meaning that over 90% of clinics cannot be meaningfully compared based only on statistical limitations. This problem is made worse because the report lists cycles rather than patients, reducing independence of the variables.

Another very important reason that clinics cannot be compared is that patient populations vary dramatically from one clinic to the next. Among other differences, this occurs because of:

• geographic differences

• program selection criteria

• available program services

• the availability of alternatives, such as surgery or adoption

• socioeconomic factors

• insurance coverage issues

Thus, this problem cannot be adequately addressed in a short report on each clinic.

Finally, each clinic has unique attributes that cannot always be reduced simply to the pregnancy rate. These include the protocols they use, such as their aversion to risks like hyperstimulation and the criteria for canceling cycles, the number of embryos replaced which increases pregnancy rates but also multiple birth rates, the availability of donor oocytes, cryopreservation and use of blastocyst culture. The values of the clinic personnel, laws of the state and social norms affect the decisions of caregivers and choices of patients. These are immeasurable, but they do impact pregnancy rates.

For all the above reasons, ART centers cannot and should not be compared using the current annual report. This does not mean the report is not without value. Indeed, the report has probably increased the quality of infertility services dramatically. This has occurred because of the attention SART/ASRM, the CDC and RESOLVE are paying to all of these issues and to marked improvement in standardization of reporting, to identifying important issues and to communicating these to the clinics. The clinics are responding by improving the quality of technology, services and patient care. Patients are also much better informed consumers, and as a result they are making better choices for themselves.

Patients need to make decisions about their care and eventually choose a clinic. This choice should be made based on their unique needs. Many factors are important in meeting their needs. These include their rapport with their physician and embryologists, financial issues, location and service, convenience, complication rates, values and many other factors. Obviously, the quality of the physician and laboratory are critically important. However, the current reporting system, while being very valuable and having improved the overall quality of ART care, cannot and should not be used to compare clinics.