REPLY TO PEER REVIEW COMMENTS FOR FSIS Risk Assessment for Guiding Public Health Risk Based Poultry Slaughter Inspection

Transcript

1 R EPLY TO EER R EVIEW C OMMENTS FOR P OULTRY R ISK A SSESSMENT FOR G UIDING P UBLIC H EALTH R IS K FSIS B ASED P - S I NSPECTION LAUGHTER FSIS Risk Assessment for Guiding Public From January through February 2006, the 2005 Health - Based Poultry Slaughter Inspection was independently peer reviewed under Risk a contract with the Research Triangle Institute in accordance with the Office of 1 Management . A list of peer reviewers is found in and Budget peer review guidelines in Appendix II. Appendix I; and the charge to the reviewers is found Based on this peer only on Salmonella review, the 2005 risk assessment was substantially revised to focus contamination data, include data from PR/HACCP sampling programs in lieu of the original twenty poultry slaughter plants, and an appro ach from the scientific literature was used to model the public health impact. Therefore, many comments below are not germane to the current version (January 2008) of the risk assessment. Based on technical review and comments on the 2008 risk assessment r eceived from stakeholders, such as the National Advisory Committee on Meat and Poultry Inspection, the risk assessment . will be further revised 1 Office of Management and Budget‟s “Final Information Quality Bulletin for Peer Review” (December, bulletin establishes 2004): - 03.pdf . This http://www.whitehouse.gov/omb/memoranda/fy2005/m05 government - wide guidance aimed at enhancing the practice of peer review of government science documents.

2 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment FSIS Below are itemized replies for each of the peer review comments received for the ding Public Health Risk Assessment Risk - Based Poultry Slaughter Inspection. for Gui light editing was done for corrections in spelling and grammar Though s reviewer , comments are otherwise reproduced in this document verbatim. Itemized Replies to Reviewer #1 The data Comment: sources are modest, largely derived from internal studies sponsored by various arms of USDA (FSIS and ARS), and mostly unpublished . It would, however, be unlikely that published work could be used to populate the risk assessment model, and the fact that 90 % of the young chicken production is represented for the Salmonella prevalence data means that the data set is fairly comprehensive. However, the small sample size (20 plants) and the short time period (1 year) make this a limited data set with which to wo rk . Furthermore, all data on prevalence and enumeration are reported and used as mean log values, without consideration of variability. For further discussion 10 of ramifications, see b and f below . Admittedly, the small sample size and sampling pe riod limit ed the conclusions Reply: could be drawn from the original 2006 draft risk assessment model, as was that In response to these and related comments, the model was acknowledged in the report. f refitted in 2006 ata for the prevalence o 2007 to incorporate d Salmonella in poultry - carcasses representing 154 young chicken slaughter establishments. This data c ame from - the USDA/FSIS HACCP sampling c ollection p rogram for 2003 Salmonella 2005 . This data will be supplanted by the completed 2008 young chicken base line study in late fall 2008. Comment: The overall approach used for modeling risk based inspection versus non - risk - based inspection, as described in the Report, has several shortcomings that undermines the suitability of the algorithm and insights from the analysis for use in risk management - making. In this review, we have attempted to identify and expand upon or policy decision the critical problems with the approach, and have also suggested alternative gestions be tested and their methodologies. However, we recommend that such sug suitability verified using available data before any substantial conclusions be reached . The first shortcoming is regarding the methodology used to quantify the relationship with “dependent” variables. between selected so called “independent” variables Specifically, only one “independent” variable is considered at a time and its effect on the selected dependent variable is quantified . Consequently, possible interaction effects between selected independent variables are left ou t of the analysis, meaning that this simplified regression model has limitations with respect to fully explaining the relationship between variables. Reply: The original 2006 model did not consider complex multivariable relationships. as enhanced in 2006 - 2007 to multivariate version using some 34 However, the model w 2

3 Reply to Peer Review Comments June 2008 - Public Health Risk based Poultry Slaughter Inspection Risk Assessment explanatory variables simultaneously. These enhancements are described in detail in the current documentation of the model. Although the Report mentions that limiting the analysis t o one variable at a Comment: time is a weakness of the approach (Page 43), there is no discussion given to the magnitude of the impact of this limitation on the results. It would be expected that the regression model would behave differently when simultaneous varia tion of all independent variables is taken into account. Such an analysis can be done using a multivariate regression approach (Neter et al. , 1996; Sen and Srivastava, 1990). However, this approach cannot be applied easily to the Risk Based Poultry Slaught er - Inspection model, as selected variables for analysis violate the key assumption of regression analysis with respect to independency between inputs. This point and possible remedy solutions are further discussed in response to the charge question “ f ” and are not repeated here. By way of summary, selected variables have possible dependency structures that will introduce multicollinearity . It would be beneficial to consider the interaction effect of different variables in the model . tive For example, how enumera values change may not only depend on the number of type 01 and 03 unscheduled procedures completed, but also to how these two sanitary procedures interact. Reply: We agree . This comment becomes mute due to enhancements in current version. se Please e reply to above comment. Comment: The authors seem to ignore the fact that regression models are only as good 2 values as their coefficient of determinations, i.e., R . The coefficient of determination explains how much of the output variability is explain ed (captured) by terms included in the regression model. The authors did not provide any indication in the Report regarding 2 the magnitude of R values for their simple one variable regression models. Although they did indicate in the Report that they would - of - fit of the like to improve the goodness regression models in the future (Page 43), there was no discussion about the reliability of 2 values may be results using the current model. The reviewers, however, suspected that R relatively low. In response to this and our own concerns, we modified the given code in 2 order to estimate the R values for each regression model in each bootstrap replication . A 2 values based on summary of our results is provided in Table 2 (see attached) as mean R 5,000 bootstrap repl ications for selected dependent and independent variables. Results 2 show that R values are quite low. For example, on average only 3% of the change in - Salmonella prevalence between post chill steps can be attributed to the change and pre - 2 - line inspectors. Typically, R values are between as low as 0.03 and on in the number of as high as only 0.16 for the relationships between different independent and dependent 2 values indicate that there is practically no variables. Such drastically low R association betwe en, for example, prevalence or enumerative data in selected poultry slaughter plants and the number of on - line or off - line inspectors or the number of unscheduled sanitary processes completed in the plant. The reviewers believe that regression coefficients are not statistically significant either, and hence, there is not enough proof that they are even different from zero. We conclude that regression lines should not be used as the basis of e Report. We suspect further scenario analyses as performed in the current version of th 2 values may be indicative of either: (a) poorly chosen that such substantially low R 3

4 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment independent variables, which actually do not have any relationship to the dependent due to limited sample size variables; and/or (b) a small degree of variation in the dataset . In conclusion, the current modeling approach shows no (i.e., 20 poultry slaughter plants) relationship between the independent variables chosen and the dependent variables of pathogen prevalence and/or load. 2 Reply: The reason for the small R values may be due to explanation “a,” in which case the results from the model suggest that reallocating inspectors in the plant will not lead to an increase in pathogen prevalence and/or levels on young poultry carcasses. We also agree th at explanation “b” offered by the reviewers is likely. As a result, we are collecting additional data that will be incorporated into the model with the 2008 young . If incorporation of these additional data shows no improvement chicken baseline study in exp lanatory ability, then we can conclude that the impact of changes to FSIS inspection resources would be inconsequential. If, however, explanatory ability increases, then recommendations that are more specific may be forthcoming. Please see replies above r egarding enhancements to the current version of the model. It may be that pathogen prevalence or enumeration values for broilers are not Comment: influenced by the number of inspectors or the number of sanitary procedures completed within the plants Beca use incoming product frequently is contaminated at the pre - . harvest level (particularly relevant for Campylobacter ), the impact of number of inspectors is unclear, since simple inspection will not necessarily lead to identification of pathogen contamination . Even recognizing that there is substantial cross - contamination occurring during processing, controlling this would rely on the efficacy of the control steps implemented, not necessarily on the number of control steps . However, there may be other independ ent variables not yet considered which would be relevant for inclusion in the model . Likewise, alternative dependent variables might be considered as well. Careful design of the pilot study which FSIS proposes to do in the near future may be an ideal o identify alternative variables . way t We agree that the pilot study will help identify variables. The study design has Reply: been extensively reviewed at FSIS. Comments from peer reviewers and stakeholders en baseline study will be incorporated have been incorporated. Results of the young chick into this analysis as released later in 2008. Comment: The analysis is based on a very small sample size (i.e., 20 poultry slaughter plants). Consequently, there is not much variability in the dataset, and hence, lin ear effects of selected independent variables do not substantially contribute to that variability. Reply: The sample size has been expanded. Comment: We believe that by refining the scale of the analysis, further variability could dataset, and hence, selected independent variables are more likely be introduced into the to show statistically significant relationships with dependent variables. 4

5 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment Reply: We agree that the scale of the analysis limited the variability, and we have increase d the scale of the an alysis . Comment: The FSIS risk assessment team used an averaging technique to estimate the annual prevalence and enumerative values for each of the selected poultry slaughter plants (Page 14 of the Report). This approach reduces inherent variability in the dataset with respect to both within and between plant variability. We suggest using a smaller time scale for the averaging process. For example, one can look at prevalence or n enumeration data averaged weekly for the whole selected calendar year and the investigate if there is any relationship between number of inspectors (either on - line or - line) or total number of unscheduled procedures completed within each week with off ence or selected dependent variables. Although the authors may argue that data for preval enumeration values are not available on a weekly basis, this should not be of great concern because they can establish an unbalanced experiment for which the number of samples can be different for multiple inputs considered in the analysis. Experim ents with unbalanced design are discussed elsewhere (e.g., Montgomery, 1997). It would also be possible to use a nested - plot design whereby poultry slaughter plants can be classified into, for example, 4 groups consisting of 5 slaughter plants with similar production plot) designs are discussed elsewhere (Neter et volumes. Nested - plot (also known as split - al., 1996). Taken together, by refining the scale of the analysis and using nesting approaches, the risk assessors may have enough data to populate differ ent treatments of the factors‟ combinations, which they can use to test hypotheses such as the possible effects of slaughter plant volume, inspector type, or unscheduled procedure type on try slaughter plants prevalence and enumeration data. Variability within and between poul will also be quantified. Reply: We agree that much of the variability was hidden in the averaging process used for calculating yearly values for prevale nce enumeration. In the current version of the s model, a single data point consist 1 - month period within each plant. of a Comment: The Executive Summary states that the risk assessment evaluates changes in the prevalence and/or level of microbial contamination ( Salmonella or Campylobacter ) on young chickens as a result of changes in a ssignment and activity of poultry inspection . However, the risk assessment outcomes are expressed as either probability of personnel change (increase or decrease) in prevalence or enumerative data, or probability of change a function of assignment/activity changes . This may seem in attributable illness, both as like a minor point, but the Report never actually specifies the degree to which changes in prevalence and counts might be impacted by changes in poultry inspection . Based on the four risk management questions summarized in the Report and above, the risk assessment modeling approach, while it does address the relative change in prevalence and enumerative values, does not provide clear estimates of a measurable impact of those changes on prevalence or c ounts . We must conclude that, in its current form, it is impossible to determine if the overall approach has utility for addressing the proposed risk management questions. Please refer to the response to the charge question “ b ” to identify problems that sh ould be addressed before being able to evaluate the utility of the approach. 5

6 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment The report has been updated to specify the degree to which changes in prevalence Reply: Salmonella and counts of The 2008 are impacted by changes in poultry inspection. icken baseline study will allow us to reintroduce young ch enumeration campylobacter changes as well. Comment: We believe that the model is not complex enough to adequately address the hey proposed risk management questions. The simplifying assumptions are such that t adversely affect the credibility of the results and the modeling approach. The limitations of the model with respect to general methodology and also sensitivity and scenario analyses are discussed in the responses to the charge questions “ ” and “ f ” an d are not b repeated here . Simply, we believe that inherent variability in the dataset is not properly quantified. As explained in our response to charge question “ ”, authors averaged b prevalence and enumeration data within each poultry slaughter plant durin g the selected calendar year. This approach substantially reduces the variability in the dataset, and hence, reduces the chance of quantifying any statistically significant effect on independent variables . Because the main objective of the work was to quan tify such a relationship, the authors should refrain from using any methodology that reduces the data variability. As we suggested above, a solution is to refine the time scale of the analysis and focus on weekly variation of data rather than just annual a veraging. Reply: The data and complexity of analysis i n the revised risk assessment model have been expanded considerably. We agree that much of the variability was hidden in the averaging process used for calculating yearly values for prevalence enumera tion. In the updated version of the model, a single data point will consist of a - month period within 1 each plant. Comment: We believe that the methodology used for quantification of uncertainty, i.e., bootstrap simulation, is sound and sufficient. ly: Rep N/A. There are key limitations with respect to the modeling techniques that are Comment: fully discussed in our response to charge questions “ b ” and “ f ”. These limitations are not repeated here. Reply: N/A. Comment: The reviewers‟ main concern is th e lack of transparency of the source code. Very few informative comments are given within the visual basic code or inside Microsoft Excel worksheets. Thus, it was a tedious task to understand the modeling flow and connection between different cells in each worksheet. It was not possible to understand some sections of the model. For example, the purpose of defining a dummy variable for current HIMP as given in cell number “Q2” in the worksheet named ture is in the form of “RawData” was not clear. Because most of the modeling struc embedded equations inside different cells, it was not practical or even possible to verify 6

7 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment that the model had been accurately coded. However, we were able to execute the code and generate similar results as those given in the Report. For example, with a sample size of 10,000 we were able to reproduce a similar graph to the one given on Page 27 (i.e., Figure 1) for the change in Salmonella prevalence versus reduction in on - line inspectors, line inspector ratio. increase in off - line inspectors, and reduction in on - line to off - Reply: The source code has been updated to increase its transparency. All modeling of equations.,etc are coded in Microsoft excel. Visual Basic coding is used only as a means for simulation. Comment: Reviewers b elieve limited and inadequate sensitivity analysis was performed. The methodology used for sensitivity analysis is based on the comparison of cumulative probability distributions of the model outputs when alternative scenarios are performed. For example, a n increase in unscheduled sanitation procedures (type 01) is found to be most effective in lowering Campylobacter and E. coli counts, while an increase in unscheduled HACCP procedures (type 03) is found to be most effective in lowering coliform counts (Pag e 43). However, we believe that this methodology has critical shortcomings. As indicated in the Report (Page 43), the analysis is limited to single variable regression analysis. This suggests that simple comparison of the results in terms rences in the shape of the model output distribution can provide of possible diffe misleading insight regarding model sensitivity. Figure 1 is an example of a misleading insight given in the Report. Figure 1 shows estimated change in prevalence Salmonella due to a change in the number of inspectors. Based on the results, there was an approximate 80% probability that Salmonella prevalence would decrease when the number of on - line inspectors was reduced. Similarly, there was a 70% probability that Salmonella prevalence would d ecrease when the number of off - line inspectors increased. However, when these two events happened simultaneously (i.e., the ratio of on - line to off - line inspectors decreases), we do not see any significant change with respect to st of the simulations. Salmonella prevalence in mo A methodology that incorporates simultaneous variation of inputs should be used instead. One method is to use multivariate regression analysis (Cohen, 1983; Neter et al., 1996; Devore, 1999). However, due to dependency introduced i - n the inputs (e.g., ratio of on - line inspectors is a function of the number of on line to off line and off - line inspectors - that are also used as independent input variables in the model), typical least square multivariate regression analysis techniques can not be used. In multivariate regression analysis one should be concerned with the nature and significance of the relationship between the independent variables and the dependent variable. Typically, we want to find answers to questions such as: What is th e relative importance of the effects of the different independent variables?; What is the magnitude of the effect of a given independent variable on the dependent variable?; Can any independent variable be dropped from the model because it has little or no effect on the dependent variable?; Or should any independent variables not yet included in the model be considered for possible inclusion? These questions typically represent the objectives of performing sensitivity analysis. For the case of the Risk - Base d Poultry Slaughter Inspection model, because the independent variables are correlated among themselves, multicollinearity 7

8 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Multicollinearity among inputs exists (Mueller, 1996; Wang, 1996; Grapentine, 1997). tions. can cause ambiguity in answers to the above ques Multicollinearity affects the stability of the parameter estimates calculated in multivariate regression and discriminate analysis models. According to Mueller (1996), the problem racterized by a high of multicollinearity in its simplest form has been traditionally cha . correlation between two or more independent variables in a regression equation Erroneous interpretations of the results, mainly due to a lack of stability of coefficients across samples, can follow. Multicollinearity can also cause la rge forecasting errors and make it difficult to assess the importance of each independent variable in the model. There are several methods that researchers can use to handle multicollinearity in regression and discriminate analysis (see Mueller, 1996; Wa ng, 1996 and Grapentine, 1997). We suggest using the principle component analysis approach to reduce the effects of multicollinearity. The objective of the principle component analysis is to identify a f the data with respect to each of new set of orthogonal axes such that the coordinates o the axes give the values for the new variables, called principle component scores. The first new variable accounts for the maximum variance in the data and is a linear new variables are uncorrelated among combination of the original variables, such that the themselves (Sharma, 1996). Further detail is available elsewhere (Mueller, 1996; Sharma, 1996; Wang, 1996 and Grapentine, 1997). Moreover, the nested design experimental - question “ b ” should also be taken into approach discussed in the response to the charge account in this regard. Mokhtari and Frey (2005) suggested a methodology to quantify uncertainty in the form of sampling distribution of F values when using Analysis of Variance (ANOVA). This type of sensitivity anal ysis approach uses bootstrap simulation to quantify the sampling distribution of sensitivity indices (i.e., F values). This methodology can be also adapted Based Poultry Slaughter Inspection model. In this case, for application to the Risk - because the samp le size is very small (20 poultry slaughter plants), there is substantial uncertainty associated with the regression coefficients (as indicated in the Report). Similar to the methodology used in the current version of the model, random samples can be taken from k - dimensional inputs and output space. A multivariate regression model can be fitted to the resampled data taking into account the effect of dependency between inputs and the use of principle component analysis. At each bootstrap simulation, “statist ically significant” inputs are ranked based on the relative magnitude of the partial sum of squares associated with each input as a sensitivity index (Gardner and Trabalka, et al. , 1991). This process is repeated for alternative bootstrap replic ations. 1985; Rose To the extent that the sensitivity analyses yield similar results about the rank ordering of inputs regardless of uncertainty, an analyst or decision maker will have greater confidence that the results of the analysis are robust to uncertainty. If the ranking of key inputs changes substantially from one bootstrap replication to another, the identification of key inputs would be uncertain. Some statistics such as mean rank or 95% confidence interval of ranks can be provided based on the results of th e analysis for overall comparison of importance of inputs. 8

9 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment We recognize the problem of using single variable regression analysis and agree Reply: that multivariable regression would be a better approach. As the reviewer aptly points out, correlation betwe The en the independent variables presents a challenge. enhancement of the model from a univariate analysis to a multivariate analysis in the current version make much of what is said in the above comment mute. We benefitted ailed comments, however, and expect to do more greatly from the reviewer‟s det sensitivity analysis as we get updated data. The authors to some extent failed to present the structure of the model in Comment: sufficient detail. Only the deterministic values of the prevalence and enumera tive data were given based on the available data. However, information regarding the independent variables (e.g., number of inspectors or number of various measurements of completed/uncompleted PBIS procedures) is not tabulated in the report, and the reade r was forced to look up this information in the Excel worksheets provided as a part of documentation. We summarized those values in Table 1 based on what we obtained from the provided Excel file. The authors should tabularize the information regarding all model parameters with sufficient detail. Reply: Admittedly, most of the input data from the plants is not included in the documentation of the analysis. Given the bootstrapping procedure that was used to generate the stochastic model simulations, this did not seem relevant: independent draws from the pool of data were used in individual model iterations to generate parameter values. Comment: It was difficult to understand the structure of the model from the information ers were forced to refer to Excel worksheets and the provided in the Report. Thus, review Model Description visual basic code for this purpose. Some explanation is given in Section on Pages 23 - 26. However, the text in this section is poorly written which brings del structure and the analytical approach. A better some ambiguity regarding the mo approach would be to make the structure of the model clear in the documentation with - by - step execution of the further illustrative examples given with respect to the step model. For instance, we as review ers had some difficulty in understanding how the two selected scenarios were implemented, which required continuous reference to the code and Excel sheet, a burdensome task . We suggest that the risk assessment team offer one illustrative example for a sele ct pathogen (e.g., Campylobacter ) at a specific section of . the poultry slaughter plant (e.g., pre - chill) They could then present some of the bootstrap replications, providing a clear illustration of how the two selected scenarios were applied in the model. and executed Reply: T he model is documented more clearly in the update d report . Comment: As discussed above, the authors chose Microsoft Excel using visual basic macro programming, which results in a black box model that cannot be easily check for pro gramming errors. The huge number of parameters and equations included in the analysis are embedded within cells in different Excel worksheets. Thus, it is difficult to and the connection between different cells inside the flow of the model understand 9

10 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment alter native worksheets. The modeling should be more transparent, specifically to prevent users inadvertently making changes that result from the inability to see every detail of the programming. Furthermore, it would be beneficial to provide sufficient comments within worksheets and also the code to facilitate understanding of the modeling flow One . suggestion is to use a programming environment rather than using embedded equations in Microsoft Excel. The choice of programming environments depends on the skill o f the modeler, the use of add - ins, and the scope of the analysis. For models that are extensive and that will be used for multiple analyses, a programming language environment and good software engineering practices are recommended. The choice of modeling environment should account for the trade - off, if any, between the skills of the analyst, resources, anticipated needs for future model refinements, and desired flexibility with regard to sensitivity analysis. Microsoft Excel is a widely used and ea sily understood tool for this type of Reply: analysis; thus, we thought it appropriate here. The use of Visual Basic macro programming in this model is extremely limited and does not include any of the equations within the model. Visual Basic is simply used to si mulate simultaneously all scenarios. Comment: Finally, other aspects of the proposed rule, i.e., establishing new standards of identity for product, new chilling regulations, and new guidelines for on - line reprocessing and should probably not be included in the Report. are not addressed by the risk model Reply: Information about new standards of identity, chilling regulations, etc. was added - based initiative. We felt they enhanced the report as such. to give context to the risk Comment: At this time it is impossible to determine if the selected scenarios are adequate to capture all the significant differences that might be expected to occur when risk - based inspection is implemented. Please refer to the response to the charge questions “ b ” and “ f ” to iden tify problems that should be addressed before being able to evaluate the adequacy of the scenarios. N/A. Reply: Comment: In the statement of work, the authors stated that their intention was to al reallocation of USDA examine the public health impact associated with the potenti . While this is laudable, the effort inspection personnel in poultry (broiler) slaughter plants given to characterizing the public health burden associated with the consumption of e quite crude . The general approach contaminated broilers is minimal and the estimates ar was to use FoodNet data for the incidence of human salmonellosis and campylobacteriosis, and extrapolate these to the entire population using U.S. census estimates and under reporting multipliers . These numbers are then modified using - attribution factors (for foodborne, poultry, and broiler fractions), which allowed the risk assessment team to estimate the total foodborne illnesses attributable to the consumption of young chickens . However, the attribution estimates, whic h are derived from several sources (Mead et al., 1999; USDA ERS; and the FSRC), are expressed as single point . estimates, and even the authors of these estimates admit that they are crude at best There 10

11 Reply to Peer Review Comments June 2008 - Public Health Risk based Poultry Slaughter Inspection Risk Assessment - response relation ship; one cannot assume that more is also no consideration of the dose or less linear reduction in human disease will occur as a function of reduced pathogen load, as this relationship is much more complex Furthermore, it is not clear from the . ness numbers were used in the analysis Report narrative exactly how these foodborne ill (see Figures 9 12) . Because the human disease estimates are so uncertain, and the - analysis really focuses on the impact of inspection activities on pathogen prevalence, we . ysis would suggest foregoing this part of the anal Reply: We have enhanced the linkage to attributable human illnesses considerably in the current version of this analysis. Please refer to pages 14 18 of the current version of the - risk analysis report for a description of how we are now modeling uncertainty about estimates of attributable human Salmonella illnesses. Then on pages 28 - 29 we discuss our method for modeling Salmonella illnesses avoided due to changes in establishment procedures. http://www.fsis.usda.gov/PDF/Poultry_Slaughter_Risk_Assess_Jan2008. pdf 11

12 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment Itemized Replies to Reviewer #2 C omment: Generally, the approach described in the document “Risk Assessment of - Based Poultry Slaughter Inspection” (December 2005) address the four risk Risk management questions that were presented. The first question asks, “Is there a measurable difference (relationship between pathogen prevalence/indicator counts and inspection resources and assigned tasks) between risk based inspection systems for - - risk poultry and non based inspection systems for poultry plants using current inspection - methods? ” The report does not appear to discuss a relationship between pathogen prevalence and indicator counts, which may not be needed or appropriate . Note that this question (from page 2) is worded differently on page 7, where “indicator counts” has been removed. R The document has been updated so that the questions are worded consistently. eply: omment: C nterpretation (p. 3) that reassigning inspectors This reviewer agrees with the i to off - line duties may not lower the incidence of campylobacteriosis cases, since off - line inspection tasks focus on control of Salmonella rather than Campylobacter . Since the line inspectors are number of illness es is predicted to increase when off - Campylobacter increased, but not when the ratio of on - line/off - line inspectors decreases, you may want illnesses based on prevalence, in addition to Campylobacter to consider predicting enumeration. – The cu rrent data being used in the analysis does not include Campylobacter R eply: only Salmonella . The new data available in fall 2008 will include Campylobacter . The analysis will be updated at that time. omment: Some of the microbiological data was from samples collected between C October 2004 and September 2005, while the data on inspection activities was for calendar year 2004 . Some people may expect the inspection and microbiology sample data to overlap the same time period. eply: R prevalence Salmonella Data for inspection activit ies have now been paired with data for the same establishments and timeframes. C omment: The criteria for selecting unscheduled procedures completed, as independent . Why don‟t you include scheduled procedures variables, needs further explanation and 08 05, ? completed for ISP code activities 01, 03, And, how many of 13,339 ISP codes were unscheduled Type 01, 03, 05 or 08 procedures? R eply: The rationale for including unscheduled procedure completed as independent variables was that th ey are useful as “decision” variables. That is, they are those that the risk manager may make changes. Experts in the field were asked to choose those procedure codes that they thought would be most relevant to the policy questions at hand. 12

13 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment the scenario where inspectors are reduced in each plant to one per shift, omment: C For please clarify if this is the same as one per line per shift. This point is clarified in the revised report. R eply: omment - written and formatted. C : The report is well R N/A. eply: omment : The proposed inspection system has many similarities to the current HIMP C . “experiment” Somewhere the report should explain the specific differences between HIMP and the proposed system, and if the new system would replace the one used in HIMP pl Perhaps the 4 HIMP plants data should not be included, since these plants ants . have much different inspection procedures currently. eply: The report has been updated to include a brief discussion describing differences R between HIMP and the prop o and similarities sed new system. omment: and coliform tests appears to be minimally used in the The data for E. coli C . Perhaps this information should be removed, or further incorporated into risk assessment the assessment. Salmonella R eply: . The current version of the anal ysis focuses exclusively on omment: The Conclusions section (page 43) remarks on predicted lowering of E. coli C . Perhaps this should be removed . I cannot give a strong and coliform counts recommendation on how to include information or predic tions of E. coli / coliform data . While the enumeration tests for these organisms may be required at this time, you may . Other reviewers or constituents may not want to predict how enumeration may change use your report to justify the use or removal of E. co li / coliform testing to indicate Salmonella presence or process control . This effort may distract from your goal to improve public health or reassign inspection personnel duties. R Please see above comment. eply: . C You may want to clarify “plant v olume” on page 26 and elsewhere omment: Does this refer to number of carcasses, or liveweight pounds? R eply: This referred to the number of birds slaughtered per plant. The text has been revised to make this clear. C omment : The two scenarios are good choices to study what could happen if risk - based inspection is implemented . I would be interested in seeing what may occur if unscheduled procedures are increased by 25%. Can you estimate the number of inspectors (per out 25% more unscheduled procedures? line/shift/plant) that would be needed to carry 13

14 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment available is R no information Because there for the capacity of workers to perform eply: unscheduled procedure checks, the risk assessment cannot estimate the number of inspectors needed to carry out 25% more unschedule We agree, however, d procedures. information for policy makers. that this would be beneficial omment: The interpretation that current inspection procedures better control C prevalence, rather than Campylobacter enumeration is appropriate to he lp Salmonella Campylobacter explain why illness are predicted to increase when the number of off - line inspectors are increased. N/A. R eply: C omment: The approach to estimate illness is generally appropriate based on data that While improvements in c onsumer cooking and handling of raw poultry are available. could have a more significant impact on reducing the number of illnesses, this risk assessment was strictly focused on inspection and slaughter procedures The model could . Camp I assume that data is prevalence . additionally consider using data on ylobacter available, but not included in the risk assessment document. Even though there are good Campylobacter through a quantitative arguments for monitoring or controlling performance standard, a qualitative (presence o r absence) determination with this counts in post . chill organism is important too - The large reduction in Campylobacter carcasses may not correspond to a large reduction in prevalence of contamination . eply: We agree that the approach to estimate illnes R ses was appropriate in this instance, as explained in our reply to the final comment by Reviewer #1 above. Please note that he current version of the analysis focuses exclusively on Salmonella . t C The data presented in Table 8 shows that the Campy lobacter omment: populations were reduced, on average, by a factor of at least 1,000X . The potential reduction in illness (Table 12) is not nearly as significant . Is it possible that a reduction in Campylobacter prevalence is more appropriate factor to study? R ep ly: Campylobacter is not included in the revised risk assessment. It is not clear whether Campylobacter is a more appropriate factor to study. C omment: The change in number of illnesses in Table 12 is not significant compared to the total number of illnes s estimated in Table 9 . I did not notice a similar conclusion in the report. eply: The suggested conclusion has been added. R C omment: P. 3 (top): Statement #2 is unclear (“The public health impact in the log enumeration...”) t has been clarified. The text of the repor R eply: 14

15 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment C omment: P. 3, bullet #4: The last sentence of this statement implies that Campylobacter . Is that what you is a more significant contributor to foodborne illness want to say here? eply: No. The text has been reworded accordingly. R C P. 27: The phrase “of Salmonella prevalence” should be inserted twice into omment: the last sentence, as follows: “Individually, results vary from 90% no increase of Salmonella prevalence for sanitation (type 01) procedures to about 60% no increase of monella prevalence for unscheduled sampling (type 05) procedures.” Sal R eply: The suggested change has been made. axis of Figures 1 - 8 should be C omment: Figures: The scale or units used on the x - identified The scale could be increased for some figures (2, 4, 6, 8, and 12 ) to make them . easier to interpret. R eply: These changes have been made. C omment: Page 46, Tables 14 & 15: The variables “NC##” are not defined. R eply: All variables are now defined in footnote s . omment: Page 37: In line 9 of the para graph under “Reduction in on - line inspectors” C change “ Salmonella ” to “ Campylobacter ” in the sentence: “The results show confidence that modeled changes will not increase Salmonella - related illness approaching a 70% likelihood of no increase in illness.” This change has been made. R eply: 15

16 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Itemized Replies to Reviewer #3 The risk assessment evaluates different scenarios (regarding inspectors and Comment: inspection procedures) and their predicted effects on pathogen prevalence and numbers and ultimately on . The model assumes a “cause and effect” relationship human illnesses between pathogens (prevalence and numbers) and human illness. As the writers indicate in the limitations section, “a formal analysis between these changes and the level in the final produ ct and the relationship between dose and illness has not been evaluated”. Model predictions are based on univariable regressions and the assessment of multiple variables is not considered, but a suggestion is made on pages 43 that this will be done at a la ter time . Uncertainty in regression model predictions of numbers of human illnesses is captured through bootstrapping methods . The model assumes that there will be no changes in patterns of consumption of young poultry when the risk - mented and that the sensitivity of detection of based system is imple contamination problems with these pathogens will not be affected with a reallocation of inspectorial tasks . There is no obvious accounting for variability in predicted illnesses allowing for differences in ag e susceptibility or dose - dependent responses to pathogen load in humans . These model simplifications seem reasonable to me, given the underlying questions that the risk assessment is attempting to address. The main strength of the model is its simplicity including its availability in Excel. However, this is also a weakness since it presents a very simple depiction of a complex - For example, issues of dose response relationship in human illnesses do biologic process . the fact that many other factors (including not seem to have been considered nor has cross - contamination) subsequent to chill will impact the prevalence and load of these pathogens on poultry - products ingested by humans. Two scenarios were used to evaluate - : observations on re hang and post the change in incidence of human illness - chill Salmonella prevalence, and log enumeration of re - hang and post - chill Campylobacter . sampling . Log enumeration data for generic E. coli and coliforms were not used the primary reason for including the lat ter data in the report was Presumably, to provide additional confidence about the change in microbial load. Reply: See replies below . Comment: I unable to comment on issues related to key studies and data that might be missing. In my opinion, the risk assessment wo uld benefit from increased transparency of data sources and a critical assessment of their quality and utility for their proposed purpose . In addition, a section of the report specifically dedicated to model assumptions would be helpful. on of data quality and utility has been added, as has a description of A discussi Reply: model assumptions. 16

17 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Comment: There are a number of issues that warrant more careful consideration and discussion in the report. Salmonella prevalence data based on dif ferent tests and Comparability of sampling strategies, namely culture vs. PCR, and rinses vs. swabs vs. ground chicken (page 10). Ideally, the goal should be to use true prevalence rather than test - based (apparent) prevalence where the data are based on different testing m ethods, especially if methods have changed over time . To effectively make this adjustment, sensitivity and specificity estimates are needed for each test . To simplify calculations, it might be reasonable to assume perfect specificity of all culture and PCR methods . My assumption is that the authors have made the inherent assumption that all test methods have equivalent sensitivity and specificity . Reply: We did assume that the various test methods yielded results with equal sensitivity and specificity. A discussion has been added to the text to explore this issue further. Comment: Expert elicitation of poultry attributable fractions . The methods described to . obtain these estimates from each expert should be given What question were they asked – namely, were experts asked for their best guess of the proportion and a value that they were 95% sure that the proportion was above or below How ? many experts were included and what was the variability in their estimates? The individual expert values and how the final values used in the predictions (0.3351 for Salmonella and 0.6936 for Campylobacter) were obtained should be described since these values have a major effect on the numbers of predicted illnesses. he reference list . Websites with source documents should be provided in t References 18 and 19 provide minimal guidance about the scientific basis of the expert opinion. Reply: Although we appreciate the comment, we only used published work of others within our model. We had no control over their expert elici tation studies. Critical evaluation of FoodNet data (page 17) and Performance based Inspection - System (PBIS) data (pages 22/23). The summary data for 2003 for FoodNet – April 30, 2004) show 14.5 Salmonella (MMWR cases per 100,000 rather than 14.4 . This ra ises the general issue of quality of these data for the proposed risk assessment. Were data checks done to check for internal consistency, duplications, and omissions or were summary values from CDC used ? For the PBIS data, there is adequate description of how the data were tabulated by ISP codes but no summary table by establishment, nor indication of what data checking procedures were used . It is unclear to me exactly how these data were basis. used in the model, although I am assuming it was on a plant - specific 17

18 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Only a smal l fraction of the PBIS data was used in the model. Reply: Comment: The model is based on a simple linear relationship between . There is little motivation/justification for this prevalence/numbers and human illnesses relationship. Why not some other functional form, e.g. curvilinear ? If choice of a linear anything, the model is under - parameterized since there are many intermediate steps (e.g. transportation, handling at the retail level and by consumers) that might affect the final alence and level of contamination, and each independent variable has only been prev considered by itself . Reply: Multiparameter equations have been fitted in the revis ed risk assessment . The human illness linkage has been greatly enhanced as well. There is m inimal capturing of the uncertainty associated with the predictions. Expert opinion is modeled as a point estimate rather than as a distribution . It is also likely that “slaughter plant” will be an important source of variability because of differences in chain speed, lighting, and skill and dedication of inspectors . This variability might even be time dependent as inspectors are rotated among plants. This will be difficult to report. numerical quantify but should at least warrant some qualitative discussion in the We agree that “slaughter plant” is likely an important source of variability due to Reply: the points mentioned above. Values for things such as chain speed, lighting, etc. were imate of the considered beyond our control and were therefore captured in the est intercept term for each replication of the simulation. The distribution of uncertainty bootstrap replication . surrounding these estimates was captured through Comment: Some of the modeling issues have been discussed in section c) . An alternat ive approach might be to develop a Bayesian model using Markov - chain Monte Carlo 269) for food attribution in simulation as used by Hald et al. (Risk Analysis 2004; 24:255 - Presumably, a Bayesian regression approach was considered as an alternative Denmark . but this would be more complex to implement. I am unable to comment on the Visual basic code because of lack of familiarity. The following mathematical and statistical issues warrant consideration: The correlation between independent variables (page 25 , nos. 1 to 8) should be shown somewhere in the report (even in an appendix) since this will have . important ramifications if a multivariable model is fit It is unclear to me whether the total number of inspectors was fixed (within a plant) or allowed to v ary . I would assume that the ability to quickly reallocate inspectors to other plants is limited. correlation between Reply: In the revised analysis, multivariable regression is used; and, independent variables is documented. 18

19 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment ighlighted in yellow) in Table 9 are incorrect Some of the numbers (cells h according to my calculations Step Campylobacter Salmonella 12.6 14.4 Cases 1 Denominator 1 100000 100000 Population 2 290788976 290788976 Reports 36639 41874 Underreporting multiplier 3 38 38 Tota l illnesses 1x2x3 = 7 1392298 1591197 Foodborne fraction 4 0.8 0.95 4x7 = 8 1113838 1511637 Total foodborne illnesses 0.3351 Poultry attributable fraction 5 0.6936 Young chicken fraction 0.838 0.838 6 Illnesses (poultry) 9 772558 506550 Illnesses (yo ung chickens ) 10 647404 424489 The values for Campylobacter are not included in the revised risk assessment. Reply: Those for Salmonella have been updated accordingly. Definition of an uncertainty model. Reference is made to use of an uncertainty model to estimate risk of human illness on page 8 . The only source of uncertainty that appears to have been captured in the model is the uncertainty in the regression line parameters . Uncertainty in food attribution does not seem to have been considered. Moreov er, it would be important to know whether there is relatively more or less uncertainty in the Salmonella estimates than in the Campylobacter estimates. I found the description of the notation used and the model on pages 24 to 25 . It wo uld have made it easier to follow if it had been made difficult to follow explicit that “i” related to plants, and a brief description of “j”and “k” had been given directly after the first equation in which they were used . More detailed explanation could follow in later parag raphs . Some of my difficulty in understanding may have arisen because of lack of clarity in the superscripts in my printout . For non - scientists, perhaps a simple numeric calculation would aid in the understanding of the overall basis of the calculations. Reply: The language has been clarified in the revised text . Comment: I was unable to find any section of the report that explicitly described the sensitivity analyses (if any) that were done . The use of a linear model means that the key determinant of ch anges in illnesses is the proportion of human Salmonella and Campylobacter illnesses that are attributed to young poultry . Hence as a minimum, I For Campylobacter, values such as 60% suggest that a range of plausible values be used . 19

20 Reply to Peer Review Comments June 2008 Public Health Risk based Poultry - Slaughter Inspection Risk Assessment and 80% would seem reas onable to use, as would 25% and 45% for Salmonella. Note: May wish to endogenize uncertainty for these estimates in the model. Reply: The revised model includes revised distributions in place of point estimates. Comment: areas where improvements could be made in In my opinion, there are several layout, and general presentation of the report. Suggested improvements are the structure, . In the final section of the report, I have identified made in the following sections re rewording would help clarity. typographical errors and sentences whe Executive Summary. This clearly is in a very preliminary form but it would have been helpful to this reader have a clearer description of the strengths and weaknesses of the modeling approach, an assessment of variability and uncertainty, and results of sensitivity analysis, if done . The section on “Limitations of the analysis” requires more detailed discussion of the strengths and weaknesses of the model. It is unclear whether there is a plan to validate model prediction s with data from establishments that participate in the . new scheme The sections on Salmonella and Campylobacter epidemiology in humans require expansion with more than just reporting of trends in human cases . At least some reviewed publications tha - t deal with food attribution (as used by the Food peer Safety Research Consortium) should be presented and there should be at least some discussion of the uncertainty associated with the estimate of the poultry attributable fraction (step 5 in Table 9). Reply: The suggested revisions have been made. The approach presented in this model seems fundamentally sound, although Comment: the implicit assumptions on which the model is based require better documentation – e.g. fixed number of inspectors, equal or better sensitivity of detection of contamination/problems despite change to no maximum line speed, etc . Are there any intangible benefits/downsides that should be considered as part of the resource reallocation e.g. improved (decreased) job quality, ability to r ecruit and maintain inspectors? Reply: Estimating intangible (and largely subjective) benefits/downsides resultant from resource allocation (such as job quality and the like) was outside the scope of our analysis. Though this is an interesting idea, the p rimary drawback to doing this is that data are not available to arrive at objective conclusions on these points. The key issues that warrant reconsideration are the simple proportional Comment: the use of predictions based relationship between prevalence/counts and human illnesses, on poorly documented attribution proportion for both pathogens, the lack of consideration 20

21 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment of a lag structure in the data that would better relate pathogen levels and human illnesses . Ideally, longitudinal data that w ould indicate that a reduction in pathogen temporally prevalence and counts at the plant was transposed into lower prevalences/counts at a - retail level would provide at least more indirect assurance of the benefits of the risk based approach. equate pathogen load in young chickens to human illness was The decision Reply: to borne of two considerations: First, experience has taught us that “anchoring” risk assessments to surveillance data (including those from FoodNet) is often necessary to ensure stakeholder acceptan ce. Second, given the time constraints in conducting the risk assessment, we did not feel it appropriate to develop a full - blown dose - response model. As for uncertainty in the estimates of illness, we acknowledge that such uncertainty exists. However, ther - e is as much if not more uncertainty is the currently available dose Salmonella Campylobacter . response relationships for We agree that data indicating and a reduction in pathogens at the plant was transposed to reduction in pathogens at retail would have been valuable. Unfortunately, however, to the best of our knowledge, such data do not exist. page Comment: – Tables should be listed before Figures as page v; some Page iv numbers are incorrect – appendix and references start on pages 45 and 55, respectiv ely. These formatting changes have been made. Reply: Comment: – I am not sure that “likelihood” is the best term here to describe the Page 3 predictions Reply: “Likelihood” has been replaced with “probability.” Comment: Page 3 – Fifth bullet – “may be more effective” than what? Reply: This has been updated. Targeted allocation of poultry slaughter resources... Comment: Page 5 – This has been updated. Reply: Comment: Page 6, line 2 and 3 – should “young chicken slaughter establishments” be defined for completeness? Reply: Yes. The term is now defined. – Page 10 – I am not sure that “respiration” is the correct term Comment: suggest “conditions” The suggested revision has been made. Reply: 21

22 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Page 12, line 4 – presence in food and water “sugge sts” Comment: Reply: This has been updated. Comment: 10 should be “# of sample(s) per year Page 14, equation on line Reply: The suggested revision has been made. Comment: - it is not obvious to me why there should be should be such Page 14, Table 6 rence between “annual average prevalence” and “weighted prevalence within a large diffe plant” for some plants e.g. 10 and 11. Perhaps a numeric example would help with understanding. Reply: The suggested revision has been made. Comment: – Do the plant numbers in the 3 tables have any meaning ? For Pages 14 to 17 example, is plant 3 the same for all tables? Reply: Yes. This has been clarified. “for” rather than “for4” Comment: Page 15, footnote – Reply: The suggested revision has been made. Comment: Page in the Campylobacter section, I am unable to work out where the 16, number 40 comes from . There are 20 samples per plant collected 4 times per year. Are there 2 counts per sample? Reply: Yes. This has been clarified. Comment: Page 21, line 23 – perhaps use “documen t” rather than “chapter” Reply: The suggested revision has been made. Comment: Page 24, need to indicate here that “i” refers to plants Reply: The suggested revision has been made. Comment: – the word data is plural so it should be “data Page 24, second last paragraph were” The suggested revision has been made. Reply: Comment: Page 27, figure 1 caption – suggest making it explicit by adding “on chicken” after Salmonella prevalence so that the figure can stand - alone. Same comment applies to o 8 Figures 2 t 22

23 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment The suggested revision has been made. Reply: Page 31 Comment: Table seems out of place here . Suggest locate it later in the – document Reply: The suggested revision has been made. Comment: Page 32 (34 and 36), line 2 – “Error! Reference source not found Reply: This error was due to a mistake in cross - referencing. It has been corrected. Comment: Page 35 – for completeness, the inspection codes might be included in the figure caption or as a footnote The suggested revision has been made. Reply: Co mment: Page 38 – standard errors or standard deviations? This portion of the text is not in the revised report. Reply: – Table 12 should be introduced earlier, perhaps on line 8 after “783 Page 38 Comment: Salmonella related illnesses” The sugges ted revision has been made. Reply: Comment: Page 38, last paragraph – might be safer to indicated that illnesses are predicted to decline rather than use the word “decline” alone Reply: The suggested revision has been made. suggest making it explicit by adding “in human” Comment: Page 39, figure 9 caption – after illnesses so that the figure can stand Same comment applies to Figures 10 to alone. - 12. Reply: The suggested revision has been made. iased towards... Comment: Page 43, conclusions section, paragraph 2, line 3 – are b This has been updated. Reply: Comment: Page 45 – For completeness, the 9 prevalences (P1 to P9) should be listed one under the other immediately after the second sentence The suggested revision has been made. Reply: 23

24 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment the reference format could be made more uniform. For Comment: Pages 55 to 57 – example, sometimes “et al.” is used when more than 4 authors, “and” is used to link authors for some references and not others; the journal title is not abbreviated for the last reference (page 57) Reply: The references have been made uniform. Comment: Tables 1 and 2 - The term “confidence interval” is used but this is not really a confidence interval in the classical statistical sense. id not appear in tables 1 or 2. Reply: As best we can tell, the term “confidence interval” d Comment: – Should be “prevalence” rather than “ prevalence ”; “slaughter” Table 2 rather than “slaughter” . Reply: The suggested revision s have been made. Comment: – Graphs x - axis should be “prevalence” rather than “prevale nce” We are confused by this comment. Graphs have been checked, however, for Reply: accuracy and revisions made if necessary. Comment: Raw data table has multiple typographic errors The typographic errors have been corrected. Reply: 24

25 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Reviewer #4 Itemized Replies to Comment: The risk assessment has a strong utility for answering this question [Is there a measurable difference (relationship between pathogen prevalence and inspection resources and assigned tasks) between risk - based poultry inspection syste - ms and non risk - based poultry inspection systems currently in place in young chicken slaughter plants?] . The risk assessment team has established a good database, consisting of information collected over one year from twenty different plants, all with diff erent - . - line inspection and different levels of inspection procedures patterns of on line vs. off Since this is not a designed experiment, but rather uses real world data, there are some limitations of the data used . For example, any conclusions regarding t he effect of the ratio of on/offline inspectors needs to be tempered with the knowledge that in the dataset used in the RA, the number of online inspectors always exceeds the number of offline . If the propos ed rule moves forward, those inspectors so the ratio is always greater than 1 plants that switch to the risk - based inspection system should expand the database of observations and improve the data used in any subsequent risk assessments . y in the revised risk The database of observations has been expanded considerabl Reply: assessment. As additional data become available, e.g. those for Salmonella enumeration from young poultry, the risk assessment will be updated accordingly. Comment: How The modeling approach has some utility for answering this question [ will a reallocation of inspection resources away from on - line procedures, either out of the plant or to other HACCP verification procedures and/or sanitation verification procedures, affect prevalence, as well as other process control indicators?], as the data used are concerned with on line vs. off - line inspection and various sorts of HACCP and - sanitation verification procedures. Since the data use here don‟t actually represent true “reallocation” as such (i.e. allocation from plant 1 is compared to a llocation from plant 2, rather than allocation within plant 1 compared to allocation within plant 1 at a different time), I am somewhat concerned that - the strength of the conclusions may be over stated. used to study such reallocation, and we That being said, there are no such data that can be must do the best we can with the data we have . As noted above, should the proposed rule move forward, we would be provided with an excellent test bed to study “true o collect such data, if it is at all reallocation” and I strongly encourage the agency t possible. Reply: The reviewer‟s comments are appreciated. Comment: I question the value of including the 08 unscheduled data in the analysis at all . The dataset includes no scheduled 08‟s (hence no unperformed 08‟s) and only 8 plants where unscheduled 08‟s occurred, and only 3 plants where significant (i.e. double This very sparse dataset is a likely cause for the very digit) unscheduled 08‟s occurred . slight effect on pathogen prevalence or level. 25

26 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Since only 01, 03, 05 and 08 data are presented in the final analysis, I‟m not sure what the value is for including the 04 and 06 data in the “RawData” part of the spreadsheet, or even for including them in the beginning of the report. e 08 unscheduled data are those for biosecurity. Th Reply: We agree with this comment . We included the 08 category in the model so that as more data become available, the model may be updated to reflect unscheduled biosecurity procedure checks. Comment: As noted elsewhere in these comments, i t is clear that (while debatable) a 1% change in Salmonella prevalence could be expected to produce a 1% change in illness, the exact logic the risk assessors are using to relate a change in Campylobacter enumeration to human illness is not clear . A simple example, using actual numbers, rather than equations would go a long way towards improving the readability of the document. A worked example is now included. Reply: Comment: I have not noted any missing studies or data. Reply: N/A Comment: The bootstr apping approach used here is a clever one, and the risk assessment team is to be praised for their ingenuity and willingness to push the envelope in risk assessment methodology. I am concerned, however, that the casual reader will simply skim the seemingl y complex math and statistics and not realize that what essentially drives the results of the risk assessment is the correlation (or lack of correlation) between the microbial outputs inspectors, (prevalence or enumeration) and the inputs (on - line inspectors, off - line unscheduled inspections of various types). The risk assessment hides these simple correlations using many equations and the tables of uni - and bivariate correlations and transformations in the appendix. It is possible to “fish out” these rel ationships using the information from the “RawData” portion of the spreadsheet, and let me state unequivocally that the risk assessment team is to be praised for including this raw data in the information provided to the most highly reviewers. To illustra te my point I am including a few simple correlation plots from the data . Constructing these plots helped me to make some sense out of what were some rather . nonsensical findings of the risk assessment 26

27 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment - line inspectors caused ducing on The chief source of confusion was the finding that re to fall (which makes sense and which agrees with the Salmonella Campylobacter - findings), but that increasing off line inspectors causes levels to rise. Campylobacter To try to understand this, I decided to go back to the r aw data and plot the relationship This yields the plot . between Campylobacter log reduction and the number of inspectors below: 6 y = 0.005x + 4.1324 2 R = 0.0103 5 4 3 y = 0.0275x + 3.9378 On line 2 2 = 0.0543 R Campy log reduction Of f line Linear (On line) 1 Linear (Of f line) 0 50 60 0 10 20 30 40 Num ber of inspectors The first point that is apparent is that the correlation with any of the data is quite weak, . ion Given that the agency needs to move forward with the but that there is some correlat modernization of the inspection system, and this is the best (only?) data available, the team should not be faulted for using it. - The second (and more important point) is that increasing on line inspectors line an d off - are A similar plot of the . both correlated with increasing Campylobacter log reductions correlations shows that the on and off line slopes are inversely related: Salmonella 27

28 Reply to Peer Review Comments June 2008 - Public Health Risk based Poultry Slaughter Inspection Risk Assessment 1 0.9 0.8 y = 0.0013x + 0.4652 0.7 2 = 0.0076 R 0.6 0.5 0.4 On line 0.3 Of f line Salmonella prev reduction 0.2 Linear (On line) y = -0.0041x + 0.5296 0.1 2 Linear (Of f line) = 0.0141 R 0 40 50 60 0 10 20 30 Num ber of inspectors This relationship leads to the more sensible findings for Salmonella . A third (and perhaps most important point) is that two plants have very high numbers (39 - and 56) of on Now my analysis is an admittedly simplistic one, and I line inspectors . have not bothered to search exhaustively for the correct transforma tions for either the prevalence, enumeration or number of inspectors, but the analysis does lead to some interesting findings as follows: 2 can be quite sensitive to situations where one or two points lie It is well known that R inuum of points. If we exclude these points from both the well to one end of the cont Campylobacter and Salmonella analyses, some interesting findings emerge: The 2 on - line correlation essentially drops to zero (R 05), while the = 7E - Campylobacter 2 Salmonella correlation improves (R = 0.0272). In short, I believe the puzzling finding of the unexpected relationship between increased off - line inspectors and an increase in Campylobacter enumeration is due to the great I encourage the ris power that those two plants have in the analysis . k assessment team to re - run the analysis omitting the data for those two plants. Reply: helpful insight s. The analysis has been rerun and We thank the reviewer for these appendices have been included to assist the more technical reader in interpreting the quantitative model. A dditional data have also been added to the model thus reducing the power of individual outliers. Comment: The risk assessment uses the difference between re - hang and post chill as an justified . An argument could be made indicator of the risk reduction . This needs to be 28

29 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment post - chill values alone could be most representative of the risk posed to the public that . Now this may be a moot point since the final counts and reductions are correlated, but ded. Discuss as team. some discussion and justification are nee Reply: Discussion has been added to provide justification. In the current version of the is used. We still defend, however the use of difference data analysis only post - chill data – thogen levels on the carcasses. . as this helps account for variance in incoming pa The model is of an appropriate level of complexity, given the data that are Comment: . available While the model itself is fairly simple, as noted elsewhere in this review, the presentation of the model gives the impression of great complexity . A simpler presentation of the model may improve understanding and acceptance. Reply: We have simplif ied the model presentation. Comment: Parameterization is appropriate, except as noted elsewhere, I suggest removing 08 data from the analysis since these data are very limited, and their impact on the results are minimal . It would be sufficient to simply note that these data are limited, were modeled, but were omitted from the final because of little effect on prevalence or . enumeration Reply: The 08 unscheduled data are those for biosecurity. We included the 08 category in the model so that as more data become available, the model may be updated to reflect unscheduled biosecurity procedure checks. See above regarding the 08 data and need to simplify the presentation. Comment: Reply: See above reply. Comment: I believe the bootstrapping approach is adequate to characterize uncertainty in the relationships and variability in the data . Reply: N/A. Comment: See above regarding comme nts on two plants with high numbers of on - line inspectors. Reply: These two plants have less influence in the current analysis. Comment: The mathematics and equations appear to be adequate . As noted elsewhere in this review, the clarity of the explanatio n of the equations could be improved. We have sought to clarify explanation of the equations. Reply: 29

30 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment Comment: . As The methods used for estimating the parameters appear to be adequate noted elsewhere in this review, the clarity of explanation of the boots trapping approach, and the relationship between the input and output data could be improved. Reply: We have sought to clarify this point. Comment: The calculations in the spreadsheet appear to be accurate . The nature of models developed in Excel is that while they are easy to build, they are very difficult to check for errors. Reply: N/A. Comment: . A search of the report There is no formal “sensitivity analysis” as such . he documentation for the word “sensitivity” does not find any instances of this word T - nature of Figures 1 12 are such that it is possible to compare the slope of the cumulative probability plots to determine which inputs the model outputs are most sensitive to . Tables 11 and 12 likewise essentially constitute a sensitivity analysis, alth ough they are not described as such . The model outputs are most sensitive to inputs where the th difference between the mean and standard deviation (or median and 95 percentile) is greater . The authors could add a table, or better yet a figure that compares the magnitude of these differences for all the independent variables, and their relationship to zero (or no change in illness). Reply: We agree with the reviewer‟s comment. We have added additional appendices that address this issue. The repor t is generally understandable, but detailed page - by - page comments Comment: review, which have been provided elsewhere in this may help to improve clarity and completeness. Reply: N/A Comment: The risk assessment team is to be applauded for attacking such a comple x Despite the numerous issues and concerns detailed in this review, I find the problem . . Indeed, I can imagine no other approach that overall approach to be fundamentally sound would use real world data to try and address this complex issue . Fundamentally, I believe - that there is strong scientific support for a more risk - based approach to meat and poultry inspection, the key question is how to get there ! This study takes the best available data based way, the and lays out, in a risk - probability, or likeliho od that certain changes will, or will not increase or decrease risk Studies such as this one can only give indications as . to what the results of changing the inspection system might be . If such studies indicate that such a change represents little increas e in risk (and may in fact decrease risk) then the way forward is clear. N/A Reply: 30

31 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment The scenarios presented here are generally adequate . It might be useful to be Comment: - able to model more complex scenarios such as decrease in on d line inspections couple with an increase in various levels of type 01, 03 and 05 procedures. Reply: More complex scenarios have been developed that combine decreases in online inspectors with increases in unscheduled type 01, 03, and 05 procedures in separate scenarios. The approach taken to estimate illnesses appears to be reasonable, however, Com ment: as noted elsewhere in review, it is not exactly clear how the changes in Campylobacter . This should be clarified in the repor t concentrations are related to changes in illness The . underlying data used to provide baseline estimate are sound, and are the appropriate data to use. Reply: Please note that the current version of the analyses does not include C ampylobacter results – but this will be included in the next version, us ing the fall 2008 young chicken baseline sampling data. Page 2: When “young chickens” are first mentioned, it should be noted that Comment: these used to be called “broilers”. Reply: The suggested revision has been made. Comment: Page 3: “If the number - line FSIS inspectors is decreased...” by how of on much? Reply: The section in question is not included in the revised risk assessment report. We have edited the risk assessment to provide information about percentage of reductions/increases in inspectors w hen making statements similar to that highlighted above. Comment: Page 3: It is not clear what 05, 01 and 03 mean in the context of these sentences: “Increasing unscheduled sampling procedures (05) conducted by FSIS ing the level of Campylobacter contamination inspectors is most effective at decreas compared to other unscheduled sampling procedures (01, 03). It is clear from reading ” the rest of the document, but the executive summary should stand on own. its Reply: We agree. The text has been revised so t hat the Executive Summary can be - understood as a stand alone document. Page 7: “Other aspects of the proposed rule including the establishment of Comment: standards of identity for products coming off the line, new generic E. coli testing procedures, pote ntial changes to current chilling regulations and new on - line reprocessing guidelines, are not addressed in the quantitative analysis” . Why are these other aspects Is it simply because the data are not available to address them? not addressed ? 31

32 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment The risk management questions did not include the aspects discussed above. In Reply: future updates of the risk assessment, however, we may be able to address these. The risk assessment will be presented publicly and we will work with stakeholders to refine and risk assessment analyses. questions Page 7: Hazard identification usually also identifies the population of Comment: concern... in this case the target population is general public. Reply: The text has been clarified on this point. Comment: or: “This risk assessment has estimated ... and HAS Page 8: Grammar err considered...” The suggested revision has been made. Reply: Comment: Page 11: Abbreviation “GBS” not needed, since term is never used again. Reply: The suggested revision has been made. Comment: Page 12: “By this is meant...” is awkward phrasing. Reply: The sentence has been revised. , Tables 4 and 5: Why are percentages expressed only to the nearest Comment: Page12 whole percent ? E.g. 1292/1297 = 99.6% not 100 as shown. Reply: The revised report includes values to two decimal points. Comment: Page 13 - 14: It is not possible to check the calculations for “Weighted prevalence within a plant” since monthly data is not provided. N/A. Reply: Comment: Page 15: Typo, number in middle of phase: “methodologies fo r 4 both sets”. Reply: The error has been corrected. Comment: ? Is Page 17: Why does prevalence and concentration go down after chilling ? this real reduction or just injury Perhaps the concentration goes down because the organisms are not recoverable from cold chicken skin. Reply: Prevalence and concentration go down after chilling because bacterial cells are washed away and/or destroyed. There has been much research conducted to address the of decreases in bacteria issue poultry following chilling (see on http://www.fsis.usda.gov/PDF/Slides_022306_JNorthcutt.pdf for a summary). Though 32

33 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment - non - - but viable culturable bacteria may account for some of the decrease, the majority is believed due to destruction of cells. Error! Reference source not found. Comment: Page 18: Extra CR Typo: “( ).” This error has been fixed. Reply: Pages 22 - 23: The discussion of ISP codes is quite complex and the reader Comment: ters codes and would benefit from a table or tables explaining the different prefix, let suffixes . Also, since only 01, 03, 05 and 08 are used in the model, it is not clear what the point of discussing 04 and 06 codes might be. ISP codes 04 (Economic/Wholesomeness) and 06 (Other Inspection Reply: Requirements) are used in the model. We considered the suggested table; however, the narrative description seemed more appropriate. The table did not seem to add information. Page 23: Possible missing word or punctuation . Should the text read “as Comment: nce...”? Adding a comma after expected would also be would be expected AS prevale acceptable. The suggested revision has been made. Reply: Comment: Page 24: What does “½ log” mean in this context ? Also, the reason for ost - chill alone be most including both re - hang and post - chill data is not clear . Wouldn‟t p representative of the risk ? Later on this page the document mentions ½ natural log . Terminology should be consistent (i.e. always use natural log) and the transformations it is used. reason for transformation should be referenced the first time Reply: Discussion of “ ½ log ” has been removed . Post - chill alone would indeed be most representative of exposure, and thus risk. The reason for examining both pre chill and - post - chill, however, is to examine the effect of various mitigations ( such as chilling) in reducing contamination of Salmonella on poultry. Comment: It is not exactly clear how illness reduction is calculated for the , it is logical that a 25% decline in Campylobacter data . In the case of Salmonella prevalence will lead to a 25% decline in illness . How are these calculations made for ? Campylobacter when the data are in concentration or log concentration Does a log reduction in correspond to a log reduction in human illness? Campylobacter Reply: Information for Campylobacter is no longer included in the report. Comment: Page 25: It is not immediately clear why variables 3 and 8 are needed, since 7 respectively. they can be derived from variables 1 and 2 and 4 - 33

34 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Parameter estimates were made from combinations of 1 - 2 and 4 - Reply: 7. These were derived from simulation and are not otherwise attainable. Comment: Page 25: Why “350 thousand” and not 350,000? Reply: The suggested revision has been made. tercept Comment: Page 25: In the sentence “Once parameter estimates for slope ( ) and in ( )...” why are the symbols superscripted? Reply: This section is no longer included in the report. Page 25: Why the capitalization: “First, ON - line inspectors”? Comment: Reply: This has been corrected. Comment: Page 26: Explain how the 5,000 ite rations mentioned here related to the 350,000 iterations mentioned on page 25. not Reply: The “5,000” value was a n error . The current analysis utilized 20,000 iterations 350,000. Error! Reference source not Page 27: Extra CR Typo: “of scenarios ( Comment: found. ),” The suggested revision has been made. Reply: Comment: Page 27, figure 1: Yellow is quite hard to read, I suggest another color . There is something wrong with the x - axis labels . The lines are evenly sp aced, but the tick labels I are not, i.e. 0.003, 0.005, 0.008, and 0.01 . This may be rounding error . In any event, suggest that x - axis be labeled in percent, e.g. 2, 4, 6, rather than 0.002, 0.004, 0.006 etc. New colors have been used and the x - axi s label updated as suggested above. Reply: Comment: Figure 1 also does not stand on its own: What were the reductions or increases associated with each line? Reply: Information has been added to the text immediately above the figure. Comment: The caption for F igure 1 is not exactly correct; it should read “number OR RATIO of inspectors.” Reply: The model examined number of inspectors for particular inspection activities, not necessarily the ratio in all cases. Therefore, we believe the caption is appropriate a s written. 34

35 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment Figure 2, page 28: This figure is also hard to read . Red and green lines are Comment: . The figure does not stand on its obscured by the blue line own, as the reader has no way to know what 01, 03, 05 or 08 mean. Reply: The figure has been upda ted for clarity. In addition, a discussion has been added to explain the contents of the figure. Comment: Page 29: If this page is intentionally blank, indicate this. Reply: The page was inadvertently blank. The report has been revised. - 31, Tables 11 - 12: The column header “iterations > 0” is not terribly Page 30 Comment: . This column represents the fraction of the time the risk assessment predicts that helpful the described change in the independent variable results in an increase in pathogen prevalence or concentration . Perhaps a header like “fraction of time risk increases” would be more helpful . On the other hand, the results may be viewed more favorably if the header were “fraction of the time risk ” and 1 minus the percentage shown decreases where us ed instead. Reply: The tables in question are not included in the revised risk assessment. numbers in this table appear to be Comment: Page 30, Table 11: The Salmonella 4 3 incorrect . or 10 . These should prevalence changes should be small numbers, not 10 ? This is mentioned on hy is there no separate line for the change in 08 type inspections W . See my suggestion below on removing type 08 from page 38, but should be noted here the report altogether. The table in question is not included in the revised risk assessment. Reply: Comment: - line to off - line Page 31, Table 12: Why is median decrease in ratio of on - 31)” and not “( - )31” ? As above, why is there no separate line for inspectors listed as “( This is mentioned on page 38, ? but should be noted the change in 08 type inspections here . See my suggestion below on removing type 08 from the report altogether. Reply: The value “( - )31” is given in the revised report. As stated in the report: “Because calendar year 2004 data for the original analysis, unsche duled biosecurity (type we used 08) procedures are not frequently recorded in the data that and therefore results are not reported in Table 11 and Table 12.” Page 32: “Figure 3 and Figure 4Error! Reference source not found.” Comment: Reply: This error has been corrected. Comment: Page 32, Figure 3: Yellow is hard to read, Figure doesn‟t stand alone, what were the reductions or increases associated with each line ? Caption should read “number doesn‟t seem suited axis is for change in level, but the scale - OR RATIO of inspectors. ” X 35

36 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment to this . The scale shows values of 0.01 or 0.02, but change in levels from Table 11 shows 3 5 or 10 . numbers like 10 Reply: The specific figure in question is not in the revised report. More generally, t he ted automatically by the graphing program. We have colors of the lines were genera worked to increase resolution and feel all colors are discernible. The model examined number of inspectors for particular inspection activities, not necessarily the ratio in all e the caption is appropriate as written. We have also checked cases. Therefore, we believ to ensure that values in tables and figures are consistent throughout. Comment: Page 32: “Removing on - line inspectors seems to lead to no change or a reduction in enumeration”... are on - line insp ectors REDUCED or REMOVED? Reply: “Removing” has been changed to “reducing.” Comment: - line ranks, we cannot Page 32: “When those inspectors are added to off predict a reduction in Campylobacter enumeration with confidence.” This seems totally non - intuiti ve . This finding should be discussed . Note that it is discussed later (page 38) but warrants some discussion here as well . This is likely to be a key weakness exploited . by anyone wishing to dismiss the model as invalid Reply: Campylobacter are no longer included in the model. The reason for Data for Salmonella . If additional doing so was that we were able to obtain much more data for data are generated, we may then include them in updated versions of the model. Comment: Page 33, Figure 4: Can‟t see diff erences between lines, especially 05 and 08 . Symbols for 05 and 08 obscure other lines . Use colors or different symbols . The scale 3 shows values of 0.01 or 0.02, but change in levels from Table 11 shows numbers like 10 5 or 10 . Reply: The particular figure in question is not included in the revised report. As a general matter, we have gone through the report to examine each of the figures, increase resolution, and in many instances size of the figures. We believe each figure is clearly discernible. : Page 34: “Figure 5 and Figure 6Error! Reference source not found.” Comment Reply: This error has been fixed. Comment: Page 34, Figure 5: Caption should read “number OR RATIO of inspectors. ” axis is for change in level, but the scale doesn‟t seem suited to t X - his . The scale shows 5 3 values of 0.01 or 0.02, but change in levels from Table 11 shows numbers like 10 or 10 . Reply: The specific figure in question is not in the revised report. More generally, t he graphing program. We have colors of the lines were generated automatically by the worked to increase resolution and feel all colors are discernible. The model examined 36

37 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment number of inspectors for particular inspection activities, not necessarily the ratio in all iate as written. We have also checked cases. Therefore, we believe the caption is appropr to ensure that values in tables and figures are consistent throughout. Comment: Page 35, Figure 6: Can‟t see differences between lines, symbols obscure other lines. We have gone through the report to examine e Reply: ach of the figures, increase resolution, and in many instances size of the figures. We believe each figure is clearly discernible. Comment: Page 36: “Figure 7 and Figure 8.Error! Reference source not found.” Reply: This error has been corrected. : Page 36, Figure 7: Yellow is hard to read. Comment Reply: Colors in the figures were generated automatically by the software used to generate them. We have gone through the report and worked to increase clarity and clearly discernible. resolution of all figures. We believe each is Comment: Page 38: “a 77% likelihood of no increase in illness” ... actually it‟s stronger than this - a 77% likelihood of A DECREASE OR no increase in illness. The statement in question is not in the revised risk assessment. Reply: Co Page 38: “The results show confidence that modeled changes will not mment: - increase Salmonella related illness approaching a 70% likelihood of no increase in illness” ... I think you are actually talking about Campylobacter (Fig 10) here, not Salmonella . Re ply: The statement in question is not in the revised risk assessment. Comment: Page 38: “Similar decreases in illness occur on average for both Salmonella 343) scenarios” ( - 346) and Campylobacter ( - State explicitly what these numbers mean: . “Similar decre ases in illness occur on average for both Salmonella (346 FEWER CASES PREDICTED) and Campylobacter (343 FEWER CASES PREDICTED) scenarios. Reply: The statement in question is not in the revised risk assessment. Comment: Page 38: “However, when HACCP (type 03) or sampling (type 05) unscheduled procedures are increased similarly ... while illnesses Campylobacter increase on average” . As above, this is totally non - intuitive and really needs a careful critique. The statement in question is not in the rev ised risk assessment. Reply: 37

38 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment Comment: Page 38 - 39: “Because we used calendar year 2004 data for this analysis, unscheduled biosecurity (type 08) procedures are not frequently recorded in the data that ” and therefore results are not reported in Table 11 and Table 1 2. If this is really the case, then the whole report can be simplified and all type 08 procedures removed from the analysis and report entirely. Reply: The 08 unscheduled data are those for biosecurity. We included the 08 category more data become available, the model may be updated to reflect in the model so that as unscheduled biosecurity procedure checks. th Comment: line here: lightly dotted and Page 40, Figure 10: There appears to be a 4 following the green line. The figure in question is not Reply: included in the revised report. Page 41, Figure 11: Figure legend cropped by box. Comment: This has been corrected. Reply: Page 42, Figure 12: Why is the x 20,000 - Comment: axis asymmetric, extending out to - cases ? ion? Why the long tail on type 08 distribut Reply: The figure in question is not included in the revised report. All other figures have been checked for axis symmetry and other formatting issues. Comment: Page 42: “It is limited by assumptions made early on in an already compressed analysis p eriod, ” this sounds a little whiney: like risk assessors complaining they didn‟t have enough time. Reply: It appears we expressed ourselves clearly. The statement in question, however, has been removed. Comment: , but perhaps not for the next, in Page 43: “We were limited in this analysis ” Do you mean that in the next iteration of that only 1 calendar year‟s data was available. this risk assessment that more years‟ worth of data will be available ? If so, just say this directly. Reply: Yes. Additional data will be available. Importantly, those for enumeration of Salmonella on young poultry will be available in the next year following completion of a baseline study. Comment: Page 43: “Additionally, the estimated change in illnesses is assumed That is, a formal analysis . proportional t o the changes in prevalence and enumeration between these changes and the level in final product and the relationship between dose 38

39 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment and illness has not been evaluated.” See comments above (comments on page 24) ow human illness reduction is calculated. regarding exactly h This comment pertains to calculations for Campylobacter Reply: is no longer included in the report. data are no longer included in the report. Campylobacter Page 43: “functional form of the regression eq uations” is a bit jargony . It is Comment: not clear what “variety of lag structures” means. Reply: The language has been clarified. When we refer to lag structures, we are referring to lags in time between observations of the independent variable vs. the dependent variable. Comment: Page 43: “Our scenario results indicate consistently an approximate 80% likelihood that Salmonella illnesses attributable to young chicken will either decrease or remain the same. ” If what ? What is missing here is the idea that this is the likelihood of illness IF the inspection system is modified as proposed. Reply: The language of the report has been modified accordingly. Comment: Page 43: “There is a 70% likelihood that Campylobacter illnesses attributable to young chicken plants w ill either decrease or remain the same” This statement sweeps line inspector (Figure 3) under the rug. - the increase in illness from increase in off Reply: Data for Campylobacter are not included in the revised report. Comment: Page 43: “Blanket reassignm ents of on - line inspectors to currently aligned off - line procedures may not be useful in lowering Campylobacter and other indicator organism counts.” In fact, your data indicate that these changes will likely INCREASE campy counts! Reply: Information abou t Campylobacter is not included in the revised risk assessment. Comment: Page 43: Typo: “are biasED towards” Reply: This particular instance has been removed in the updated risk assessment. In o fix errors of this type. addition, we have done multiple proofreadings in an effort t Comment: Page 43: “inspectors to completions of unscheduled health procedures” awkward. Reply: This language is not included in the revised risk assessment. Comment: Page 43: “a 40% likelihood that Campylobacter counts will be lowered or remain the same” again you misrepresent your findings . The analysis shows a 60% counts will increase. chance Campylobacter 39

40 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment Information Reply: Campylobacter is no longer included in the risk assessment. about Comment: Page 43: “increases in un scheduled sanitation procedures (type 01) are most . effective” ARE most effective, or APPEARS to be most effective, based on the risk assessment assumptions. Reply: The language of the report has been modified to focus on associations between inspector pr ofiles and Salmonella contamination. Page 43: “increases in unscheduled sanitation procedures (type 01) are most Comment: effective in lowering and generic E. coli counts” This finding suggests Campylobacter - line /off - line Campy rise with increased type 01 that it might be possible to offset the on . Can the current model evaluate this scenario? inspections Reply: Data for Campylobacter and generic E. coli are not included in the revised risk assessment. Comment: Page 45: “Ultimately, only the four differe nces between P2 and P3; P4 and P5; P6 and P7; and P8 and P9 were modeled.” These variable names are introduced here, Definition in a table would be appropriate or define but defined later in the paragraph . each term as it is introduced. Reply: We have rev ised the report and attempted to define each term at first mention. Comment: Page 45: “Prevalence was transformed to correspond to a normal distribution using the logit transform.” This is called a logistic transformation in the body of the document . Term inology should be consistent. Reply: The terminology has been revised for consistency . In those instances where specific “logistic models” are described, the terminology has unchanged. remained Comment: Page 45: “This enumeration data was transformed by taking one - half the The text says “natural log” common logarithm” . These are NOT the same. . Reply: This was a typo in the original analysis. This transformation is not being used in the current analysis since enumeration is not being modeled. Pa ge 45: It appears that from the dependent variables paragraph that what is Comment: being modeled is the reduction from re hang to post - chill . This was not at all clear from - the text . The text on page 24 - 25 should be clarified. The Model Description section of the report, including discussion of dependent Reply: variables, has been expanded to provide more detail and clarity. Again, please note that the current version of the analysis is not using the reduction – just post - chill prevalence - t variable. estimates as the dependen 40

41 Reply to Peer Review Comments June 2008 Public Health Risk based Poultry - Slaughter Inspection Risk Assessment Comment: Page 45: “Number Cruncher Statistical Systems 2004 version software” The reference #23 lists the year as 2000 not 2004. The reference has been updated. Reply: Comment: Page 45 - 46: This reviewer would have found it helpful and inter esting to review the normal probability plots generated for each dependent and independent variable . I strongly suggest that at least some representative normal probability plots appear in the revised version of the risk assessment. Representative normal probability plots now appear in the revised version of the Reply: risk assessment. Comment: Page 46: “(P1 - P9) are given” should this be “WERE given”? Reply: This language is not included in the revised report. Comment: Page 46: What is variable NC04? This is never defined. - compliances for wholesomenss procedures. It is Reply: Variable NC - 4 describes non defined in Table 7 of the revised report. Comment: Page 48 entries, which should - 49: Tables 13 and 14 contain some duplicates 4: P1 P3, ON, OFF, ON/OFF, etc . If there is some reason why be removed from Table 1 - these need to be listed in Table 14, it‟s not clear from the text, and the text should be revised. Reply: All tables have been revised to avoid duplicate entries. Comment: re are entries in Table 15 that duplicate Table 13 Page 51: Likewise, the entries and these should be removed, or the reason for inclusion clarified. Reply: All tables have been revised to avoid duplicate entries Comment: Worksheet “Start”, Overview text box: The “i” in E. co li is mistakenly not italicized. Reply: This has been fixed. Comment: Worksheet “RawData”, Cells J2:L2: Numerous typos. Reply: These have been fixed. Comment: Worksheet “RawData:, Cell C5, Tools, Formula Auditing, Trace dependents, in list entry Double click on arro w head, select first 41

42 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment ([RBPSI_simulation39.xls]Scenario02!$C$6), This cell is in a column headed “Campy . Post Chill difference enumeration data 04” - Pre It is not clear why Salmonella prevalence data is being used to calculate Campy enumeration data. is not included in the revised risk assessment. All Reply: Information for Campylobacter worksheets have been updated accordingly. 42

43 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment Itemized Replies to Reviewer #5 ublic Comment: Indication for a quantitative microbial risk assessment to evaluate the p - health benefit of moving from the current poultry slaughter inspection system to a “risk based” inspection system: Adequate indication and rational are presented for the described “ risk - based” inspection system”. development of the Reply: N/A. Com Please see comments [in italics] under column 4 of Table 1 ment: . Summary Table of differences between the current inspection system (CFR § 1 381.67) and the proposed new system for young chicken slaughter establishments. 1 2 3 4 Proposed New Current Inspection System Data needed to assess System the difference Establis FSIS determines condemnation hments are Ascertainment of the Carcass required to sort of carcasses; establishments do adequacy of the Sorting carcasses and not sort carcasses. establishment’s ensure carcasses are performance of this task is not adulterated missing. Differences can before entering be measured by chilling tanks. comparing prevalence of the target organisms (Salmonella and Campylobacter) in carcass samples randomly picked from the lines that are currently under the FSIS supervision and after the establishment taken over. Design verification activities to focus on those aspects of proces s where loss of control is more likely to occur or where a loss of control would have serious public health consequences and to intensify inspection if there is evidence that the plant is losing, or has lost, control after the implementation of the new tem. sys Use consumer complaint and other data from 43

44 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment outside plant to guide in - plant verification activities after the implementation of the new system. Establishments will continue to Establishments must Use of performance Performance address CFR § 381.65(e). meet the food safety standards to measure Standards performance control after establishment standards for poultry personnel taken over on slaughter defects activities. on line - (zero fecal, zero Acceptable measurements septicemia /toxemia) should be listed as well as animal http://www.fsis.usda.gov/ ( disease performance OPHS/baseline/contents.h standards. ) tm Establishments will adhere to No maximum line Line Speed regulatory limits (CFR § 381.67). speeds. Rather, limits Use of performance Line on line speed will be speeds are dependent on standards to measure slaughter class. based on control after establishment’s ishment’s personnel establ ability to maintain taken over on on line - process control and activities. meet performance standards. Efforts by FSIS personnel should be spent to detect any drawbacks that may be associated with “no maximum” speed of the line policy. If no drawbacks, final approval can be provided for this change in p olicy. E. coli New process control Current CFR § 381.94(a) will Generic Define the new process performance apply. Process control performance standards will be Control standards adopted. Standard of Identity New proposed Standards of No significant differences Standards of Identity regulation s will provide a regulations for Identity f standard of quality o standard of quality for whole whole chickens. chickens. All establishments will be required to maintain a process control plan to ensure that whole chickens meet the proposed standard of identity. Current poultry Establishments will adhere to Describe the amendment Time and chilling requirements CFR § 381.66. briefly. Temperature in CFR § 381.66 amended to provide more flexibility to establishments. Establishments will adhere to On - line reprocessing Define the On line - On - line of pre - chill poultry CFR § 381.91. reprocessing of pre - chill Reprocessing carcasses poultry carcasses. accidentally 44

45 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment contaminated with digestive tract contents at slaughter. Repl y: Though we appreciate the reviewer‟s comments, we have not added the column to the table. The table was provided solely for background. We believe that many of the comments suggested go outside the scope of the risk assessment. Issues of data required to address differences between systems are discussed with risk managers. In the report, attempts have been made to address those data needs deemed most critical. We also wish to avoid making statements about what FSIS inspectors efforts should be spent doing (for instance, in detecting “drawbacks associated with „no maximum‟ speed of the line policy.” These are policy decisions best left to policy makers. Instead, in the report, we attempted to lay out objective evidence, from which policy makers can make inf ormed decisions. Comment: This report indicated that reallocation of inspection resources will be to off - line PR/HACCP. question # 2 of the reviewer‟s charge does include the impact However, of such reallocation to out of the plant. Additionally, the repo rt states that analysis between these changes and the level in final product and the relationship between dose and illness has not been evaluated . My suggestions for both scenarios are as follows: procedures out of the plant: - a. Reallocation of inspection resources away from on line This will represent “professional working force reduction”. Impact should be assessed by conducting a pilot experiment in which the performance standards (% positive for of representative , , and E. coli Salmonella Campylobacter randomly collected from on - line poultry carcass samples from large, and small establishments) should be compared using samples drawn scientifically (randomly) from the on - line poultry carcasses before the reallocation of inspection resources from on - line t o out of the plant and after the reduction. Test for proportions ( X square) of samples positive for the target bacteria before and after the force reduction will reveal the impact. Acceptable difference (tolerance of small increase in the proportions of po ultry carcasses that are positive for the target organisms should be defined). b. Reallocation of inspection resources away from on line procedures to off - line - PR/HACCP verification procedures and/or sanitation verification procedures: Similar design to the first scenario (a) should be conducted. However, it is hoped that the final comparison between the bacterial contamination (prevalence data) will be performed on samples of the poultry carcasses processed before and after the increase of resources at t - line PR/HACCP verification procedures and/or sanitation he off verification procedures to reveal if implementation of such reallocation will improve the quality of the poultry product from the establishments in terms of the reduction in the prevalence of t he listed pathogens in the final products. Data on prevalence and enumeration data for campylobacter and salmonella on carcasses before and after various processing steps such as scalding, defeathering, ted. evisceration, washing and chilling, should be genera 45

46 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment - line Time that should be given after the filling of the additional positions at the off PR/HACCP verification procedures and/or sanitation verification procedures should be considered before a meaningful analyses could be performed. One month after th e manning of those positions may be a reasonable time to reveal the positive impact , Campylobacter , and the non - (reduction in prevalence of E. coli ) Salmonella specific in samples of the poultry products leaving the establishment. c . Model indication for changes in Salmonella prevalence: Results can be viewed in Figures 1 and 2. These results are consistent across the - various measurement scenarios. Approximately 70 90% of the time the model predicts that prevalence will not increase because of considered changes Salmonella to inspector assignments. In the second series of scenarios (Figure 2), the model simulates 25% increases in the number of unscheduled procedure completions for all - related procedures (procedure codes 01, 03, 05, and 08). Almost 9 0% of the health the model predicts that Salmonella prevalence will not increase when all four - time, health procedures are increased. Individually, results vary from 90% no increase for sanitation (type 01) procedures to about 60% no increase for unscheduled samp ling (type 05) procedures. Reply: We thank the reviewer for suggestions regarding future data gathering efforts. Comment: , and Salmonella, Campylobacter of Baseline data for the prevalence/counts the nonspecific E. coli reallocation is implemented. To should be established before such the baselines data, the following data can be compared to reveal the impact of higher completion rates for inspection system procedure (ISP) assignments effect on prevalence/counts, as well as other process control indicator scheduled - s such as the un up on necessary inspection for sanitation and other procedures: tasks to follow - Salmonella, Campylobacter , and the i. Prevalence and enumeration data for E. coli on carcasses before and after various processing steps such nonspecific as scalding, defeathering, evisceration, washing and chilling. ii. Prevalence and enumeration data for Salmonella, Campylobacter , and the nonspecific E. coli on carcasses comparing various methods of chilling (e.g. air chilling, water chilling, wate r chilling with chlorine). iii. Prevalence and enumeration data for Salmonella, Campylobacter , and the nonspecific E. coli on carcasses comparing different scalding temperatures or alternate scalding configurations (e.g. multi - tank scalding systems). Data describing the actual cross - contamination between positive and iv. negative flocks and within flocks during the different slaughter processes. Reply: We thank the reviewer for these suggestions. A baseline program is Salmonella ongoing that will yield enumeration data for on chicken. The resultant data will indeed improve the model. 46

47 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment : Impact of the reallocation would eventually be evaluated as an enhanced Comment ability to evaluate and perform a better HACCP system for the reduction of Salmonella Ca and in poultry carcasses produced by large and small establishment. mpylobacter Since this model is rather simplistic, the following data should be planned for the construction of the next models: o Outbreak and epidemiological data, specifically indicating: S almonella and Campylobacter cell number in the implicated poultry amount consumed, accurate estimates of the size of ill and exposed populations, accurate characterization of the population including age profiles, medical status, sex and other potential sceptibility factors. su Characterization and quantification of the impact of the food matrix effects, host - o pathogen interactions and virulence factors and their effect on the probability of infection and/or illness due to Salmonella and Campylobacter. New dose - response models that improve the ability to estimate the o probability of illness due to Salmonella and Campylobacter. the reviewer for specifying these particular data needs. Reply: We thank Comment In this risk assessment project, the risk assessme nt were based on only two : scenarios; 1) - hang and post - chill the public health impact between observations on re Salmonella prevalence sampling and 2) the public health impact in the log enumeration of re - hang and post - chill Campylobacter sampling using sur rogate data from ARS prevalence and enumeration data for pre - and post - chill in broilers (PBIS database for calendar year 2004; FSIS volume data for plants for 2004); and CDC data on estimates of human salmonellosis and campylobacteriosis . unrealistic, using this scenario analysis, to have accurate or valid estimates It is therefore of the changes in number of illnesses attributable to inspection resource reallocation. model‟s estimate can be use as a useful tool to guide the management of However, the ins pection force to ensure a more efficient utilization of the professional manpower that is effective in the enhancement of the mitigation of the risk associated with the contamination of the poultry products with salmonella and Campylobacter pathogens. Rep ly: Data for Campylobacter are not included in the revised model, while those for Salmonella have been greatly expanded. The revised assessment is designed to focus on associations between contamination on young poultry and specific inspection Salmonella t asks. We agree that the model is useful for guiding efficient and effective inspection. Comment: [Have all key studies and data been identified?] Yes, with the few defined assumptions and small numbers of the variables the model was based on, key studies that were used. are pertinent to the establishment of this model 47

48 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment N/A. Reply: Comment: [Have the data been correctly interpreted, analyzed, and used in the risk assessment?] Yes. Reply: N/A. Comment: The report used the Nationwide Young Chicken Microbio logical Baseline . November 1999 – October 2000. Data Collection Program http://www.fsis.usda.gov/Science/Baseline_Data/index.asp . Nov 1999 - Oct 2000 ) for ( Salmonella prevalence estimated from PR/HACCP samples . and Salmonella the prevalence estimated from the FSIS chicken rinse study. These data are the most obiological contamination of poultry comprehensive data regarding the prevalence of micr Therefore, the input data are valid and appropriate in the risk on the national bases. assessment report. Reply: N/A. Comment: The risk assessment model is based on the estimates of changes in human Salmonella as a function solely of predicted changes in illness that were considered prevalence or Campylobacter counts on chicken and that these estimates assessment assume that changes in microbial contamination on chicken are proportional to predicted changes in the num ber of related human illnesses. These assumptions are limited because they ignore several factors such as: 1) - response modeling that improves the ability to estimate the probability dose of illness, 2) food matrix effects, 3) host - pathogen interactions, 4) virulence f actors and their effect on the probability of infection and/or illness. Therefore, one can state that the model may not be complex enough to adequately address all risk management questions. Reply: Campylobacter are not included in the revised r isk assessment, while Data for Salmonella those for have been greatly expanded. We agree that extrapolating from prevalence of Salmonella on poultry to human illness is difficult. As enumeration data for Salmonella on poultry become available, the model will be str engthened by their inclusion. The model looks simplistic and reasonably parameterized. Comment: Reply: N/A. Comment: The team adequately described the limitation of the model. Therefore, the simplifications will not significantly detract from the model utility. N/A. Reply: 48

49 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment was Comment: The model adequately characterized the uncertainty and the variability addressed sufficiently. Reply: N/A. Comment : Model techniques, as described by the authors, including the mathematics and equations, seem to be app ropriate. N/A. Reply: Data analysis and source code are accurate. Comment: N/A. Reply: Comment: Most important variables in the model been were identified and adequate sensitivity analysis has been provided. Reply: N/A. Comment: model is well documented. The report was clearly This risk assessment written and sounds complete in addressing the areas it meant to address. Reply: N/A. Comment: With the assumptions and limitations that were addressed in the report, I find le overall approach for modeling the risk - the report described an acceptab based inspection (in terms of results of the reallocation of the inspection resources of - line to PR/ HACCP activities within the same establishment) versus the current arrangement of the FSIS inspection resources w ithin poultry slaughtering establishments . I would like to reiterate the aspects that should be considered in future efforts in this area The risk assessment model is based on the estimates of changes in human illness that were considered prevalence or as a function s olely of predicted changes in Salmonella counts on chicken and that these estimates assessment assume that Campylobacter changes in microbial contamination on chicken are proportional to predicted changes in the number of related human illnes ses. These assumptions are limited because they ignore several factors such as dose - response modeling that improves the ability to esti mate the probability of illness; food matrix effects; host - pathogen interactions; [and] virulence factors and their effec t on the probability of infection and/or illness. from prevalence of We agree that extrapolating Reply: Salmonella on poultry to human illness is difficult. As enumeration data for Salmonella on poultry become available, the their inclusion. model will be strengthened by 49

50 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment Comment: A part from further explanation that may be needed for the predicted increase in campylobacter prevalence expected post implementation of the risk - based inspection, I n the report, adequate in find the selected scenarios, as described and rationalized i - capturing the significant differences expected to occur when risk based inspection is implemented. Reply: Information for Campylobacter is not included in the revised risk assessment. and Campylobacter of illness from Comment: The report used data from Incidence Salmonella based on U.S. population estimate for 2003. It accounted for underreporting, estimating proportion of infections that are foodborne, estimating proportion of foodborne infections from poultry. The estimates on the proportion of foodborne infections from young chickens were based on data from the Economic Research Service (ERS) were used to estimate the proportion of poultry - related Campylobacter and Salmonella chicken, which infections that are due to young comp rises approximately, 84% of poultry production in the U.S. in 2004. The final estimates for annual illnesses from foodborne illnesses from Campylobacter and Salmonella on young chickens, therefore, are valid and appropriate. N/A. Reply: 50

51 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment APPENDIX I: PEER REVIEWER BIOGRAPHIES Dr. Lee Ann Jaykus and Dr. Amirhossein Mokhtari. Dr. Jaykus is an associate - professor in the department of food science at North Carolina State University. She earned her PhD in Environmental Sciences and Engineering from the Univers ity of North Carolina at Chapel Hill in 1993. Dr. Jaykus‟s research activities focus on application of molecular biological methods for the detection of pathogenic microorganisms in foods. Current research projects involve the development of nucleic acid a mplification technology for the detection of human enteric viruses (human enteroviruses, hepatitis A virus, Norwalk virus) in - to - eat food commodities. Additional research shellfish, fresh produce, and ready includes developing similar methods for the detec tion of Listeria monocytogenes and from dairy food products, with specific focuses on bacterial Salmonella concentration and refining molecular methods to facilitate the real - time detection of foodborne pathogens. She is also actively involved in the appli cation of quantitative risk assessment methods for the evaluation of public health risks of Dr. Mokhtari is a postdoctoral fellow training under the foodborne pathogens. direction of Dr. Jaykus. He has a Ph.D. in Environmental Engineering from North Caroli na State University (NCSU) and specializes in uncertainty, variability, and . sensitivity analyses and quantitative exposure and risk assessment Dr. Joseph Eifert – Dr. Eifert is currently an Associate Professor and Extension Specialist in the Department of Food Science and Technology of Virginia Tech. His research program focuses on the prevention and reduction of microbial pathogens in processed foods, and surface microbiological sampling procedures. His Extension program emphasizes microbiological safet y and quality issues for poultry processors and food safety education for a variety of audiences. Additionally, he teaches the graduate course "Food Regulatory Affairs". Dr. Eifert received his graduate degrees in food science and technology from Virginia Tech, and his B.S. degree in biology from Loyola Marymount University. Previously, he worked as a laboratory manager for the Nestlé USA Quality Assurance Laboratory in Dublin, Ohio, and as an analytical chemist for the U.S. Food and Drug Administration in Los Angeles, California. in the School of – Dr. Gardner is a Professor of Epidemiology Dr. Ian Gardner Veterinary Medicine at the University of California, Davis. His main expert ise is in analytic epidemiology and his research interests include diagnostic test evaluation, risk analysis for livestock diseases and food safety, development of methods for certification of pathogen freedom in animal populations, and the epidemiology and transmission of Johne‟s disease in cattle . He is an author of more than 190 peer - reviewed publications and has served on many national and international committees, panels, and review teams. He is the leader of the Epidemiology and Biostatistical Core for the Food Safety Research and Response ). www.fsrrn.org Network ( 51

52 June 2008 Reply to Peer Review Comments based Poultry - Public Health Risk Slaughter Inspection Risk Assessment Dr. Donald Schaffner – Dr. Schaffner is an Extension Specialist in Food Science and Professor at Rutgers, The State University of New Jersey. His research interests include quantitative microbial risk assessment and predictive food reviewed - microbiology. Dr. Schaffner has aut hored more than 100 peer publications, book chapters, and abstracts. He has been the recipient almost $3 million in grants and contracts, most of which has been in the form of competitive national grants. He has educated thousands of Food Industry professionals through numerous short courses and workshops in the United States and more than a dozen countries around the world. Dr. Schaffner has also served on expert committees for US National Academy of Sciences, the World Health Organization and Foo d and Agriculture Organization of the United Nations, and has chaired two expert workshops on microbial risk for WHO/FAO. He was most recently a member of Institute of Food Technologists Expert Panel that developed a quantitative risk - ranking framework for the Food and Drug Administration. Dr. Schaffner is currently serving a 5 - year term as Editor for the journal Applied and Environmental Microbiology. In May 2005, he was also appointed to serve on the National Advisory Committee on Microbial Criteria for F oods (NACMCF). Dr. Schaffner is active in several scientific associations including the International Association for Food Protection, the Institute of Food Technologists, the Society B.S. in for Risk Analysis, and the American Society for Microbiology. He holds a Food Science from Cornell University and a M.S. and Ph.D. in Food Science and Technology from the University of Georgia. Dr. Mahdi Saeed DVM, MPH, PhD, ACVPM, is a full professor of Dr. Saeed, – Food Safety Epidemiology and Public Health with joi nt appointments in the National Food Safety Center, College of veterinary medicine, and the Department of Epidemiology at the College of Human Medicine of Michigan State University. His main area of research is the epidemiology and risk assessment of food borne diseases, and the development of prevention plans for of food borne illnesses. Currently he is collaborating with the Michigan Department of Community Health and other professionals in a study of comparative epidemiology of an. He is using data from the last 10 years to describe Salmonella cases in Michig Salmonella the nature of outbreaks and sporadic infections. He focuses on identifying the foods related to contamination and evaluating risk factors in order to plan for effective control and preventi on measures. Dr. Saeed has developed and taught on - line courses on risk assessment and the public health impact of food borne illnesses. The course is a part of a newly initiated Web - based master's - degree program. The on line option has attracted many prof essionals seeking further education who do not have time for the traditional classes offered at Michigan State University. He is the editor - in - chief of Salmonella Enterica Serovar Enteritidis in Humans and Animals: Epidemiology, Pathogenesis and Control, a book cited in medical journals and chosen by reviewers as one of the best 200 out of 2,800 medical books. Saeed wrote four chapters of the book and recruited contributors from around the world who were experts in their field. Dr. 52

53 Reply to Peer Review Comments June 2008 - based Poultry Public Health Risk Slaughter Inspection Risk Assessment gree at Washington State University in 1983. He then Saeed received his PhD de took a position as assistant professor of infectious diseases at St. Louis University School of Medicine. He worked there for four years before joining a training program at the University of Washington School of Public Medical Health and Community Medicine in the areas of epidemiology and infectious diseases. Dr. Saeed earned an MPH degree in Epidemiology and Public Health. During his research, Dr. Saeed discovered the important role of food, and exposur e to animals as risk factors in an illness called Campylobacter gastroenteritis. 53

54 Reply to Peer Review Comments June 2008 based Poultry Public Health Risk - Slaughter Inspection Risk Assessment TO PEER REVIEWERS APPENDIX II: CHARGE Peer reviewers were charged with evaluating the risk assessment and responding to the following questions: a. Evaluate whether the risk assessment modeling approach has utility for addressing 2 . specific risk management questions b. Review the available data and derived variables in conjunction with the underlying assumptions used in this risk assessment. 1) Have all key studies and data must been identified? If not, the reviewer provide additional data sources and citations (where appropriate). 2) Have the data been correctly interpreted, analyzed, and used in the risk assessment? If not, the reviewer provide alternate interpretations, must analysis, or suggested utilization of the data. 3) Please address the validity and appropriateness of all input data in the model. c. Review the complexity of the model. Is the model too complex or not complex enough to adequately address the risk management q uestions? Is the model over or under parameterized? Are there simplifications that will not detract from the model‟s utility? State whether the model adequately characterizes the uncertainty present and whether variability has been addressed sufficiently. In areas where the must reviewer identifies limitations, weakness, or inadequacies, the review provide alternate data, data analysis, and/or modeling approaches. d. Evaluate the risk assessment model source code and mathematics. The model is a bootstrap regr ession model in Microsoft Excel with Visual Basic for applications 2 This risk assessment was developed to inform the specific FSIS risk management questions provided below: urable difference (relationship between pathogen prevalence and inspection resources and (1) Is there a meas - based poultry inspection systems and non assigned tasks) between risk based poultry inspection systems - - risk currently in place in young chicken slaughter plants? (2) How will a reallocation of inspection resources away from on - line procedures, either out of the plant or to other HACCP verification procedures and/or sanitation verification procedures, affect prevalence, as well as other process control indicators? (3 ) How will higher completion rates for ISP procedure assignments affect prevalence, as well as other process control indicators (this includes un - scheduled tasks to follow - up on necessary inspection for sanitation and other procedures)? io analysis, what will be the change in number of illnesses attributable and/or $ cost due to (4) Using scenar inspection resource reallocation? 54

55 Reply to Peer Review Comments June 2008 based Poultry - Public Health Risk Slaughter Inspection Risk Assessment used to collect and summarize the results. There is a total of about 350 lines of code. Are the modeling techniques (model mathematics and equations) 1) appropriate? If not, the reviewer must provide alternate modeling techniques. 2) Are the methodologies used in the risk assessment for estimating parameters from the data appropriate (i.e., follow scientifically accepted provide an alternate approach. must methodologies)? If not, the reviewer The reviewer should examine and verify that the data analysis and source 3) code are accurate. e. Evaluate whether adequate sensitivity analysis has been provided. Have the most important variables in the model been identified? Has an important variable been provide an alternate approach or application for left out? If so, the reviewer must sensitivity analysis and/or identify those parameters that should have been included. f. Comment on the adequacy of the risk assessment model documentation. Is the report clearly written? Is it complete? Does it follow a logical structure and layout? If not, the reviewer must provide an alternate outline and/or approach for adequately and clearly documenting this risk assessment. based Is the overall approach for modeling risk - g. spection versus non - risk - based in inspection, as described, fundamentally sound? If not, what problems exist and how should they be addressed? Are the selected scenarios adequate to capture all the significant differences that h. may be expected to occur when r isk - based inspection is implemented? If not, what additional scenarios should be included? i. Evaluate the approach taken to estimate illnesses due to Salmonella and Campylobacter . Is the approach a reasonable approach given that the model‟s main focus is to estimate changes in prevalence and level? If not, what additional approach should be taken? Evaluate the utility of underlying data used to estimate baseline estimates of 2004 Salmonella and Campylobacter illnesses attributable to young chickens in the US . Should other data be considered, if so, what additional data should be included? 55

Related documents