Session: 16-01-01: Government Agency Student Poster Competition

Paper Number: 150547

150547 - Repeatedly Solving Similar Simulation-Optimization Problems: Insights From Data Farming 

We study a setting in which a decision maker solves a series of similar simulation-optimization (SO) problems, where parameters of the problem vary over time to reflect the most up-to-date conditions. For example, in a generalized newsvendor problem, the newsvendor must choose how much of each raw material to order each day in response to the latest prices for procuring and salvaging raw materials. At the beginning of each day, a simulation-optimization solver is used to recommend a solution for that day’s problem.

An SO solver usually possesses some hyperparameters whose settings can significantly impact its finite-time performance, namely, the solver's ability to consistently find good solutions under limited computational budget. Recommended default values for the hyperparameters are often based on large-scale experimentation or informed by theory. However, attaining optimal performance of an SO solver on a given problem instance requires tuning its hyperparameters. This hyperparameter tuning adjusts the solver's operational logic based on properties of the problem and comes at added computational cost.

The values of the problem specifications in each problem instance are known before the solver is run and can affect the properties of that problem including the geometry of its objective function, and consequently its optimal solution. As a result, it may be possible for the optimal solver hyperparameters to also vary for each problem instance. However, when using a solver on a series of similar problems, performing hyperparameter tuning on a repeated basis can become a computational hurdle. We instead ask whether a systematic approach of conducting a designed experiment offline and analyzing the resulting data can offer the decision maker guidance about how best to customize and run their SO solver.

We take a different approach to hyperparameter tuning by harnessing the power of data farming, an inference framework that involves building metamodels using a dataset “grown'' from an experimental design. This application of data farming for studying SO solvers enables users to learn more about the interaction effects between problem and solver factors, which can aid in making decisions about whether and how to tune a solver's hyperparameters. We implemented this large-scale experimentation on the repeated newsvendor problem with 3 raw materials that are used in different proportions for 4 products. By analyzing the performance of multiple variations of the solver and problem, it was revealed that there are interactions between solver factors that affect performance. One factor that was particularly influential was the sample size used within the solver to estimate the objective function. Our solver uses adaptive sampling, which means choosing the sample size on the fly, but it starts from an initial sample size. We observed that different factors led to better solver performance depending on whether the initial sample size was large or small.  It was also seen that some factors settings may be more robust to a larger range of simulation budget (and hence computation time) choices.

Selecting solver hyperparameters based on problem factors never resulted in a lower expected daily profit: the benefit estimates range from $3.75 to $12.17 with an average of $7.25. This corresponds to a very small but statistically significant (p-value < 0.0001) benefit in terms of the percent improvement in profit, ranging from 0.7% to 2.5%.  Nonetheless, this comes at essentially no additional cost---with such a small number of potential cost configurations. We anticipate a more significant impact in larger and more realistic problem settings.

Presenting Author: Nicole Felice North Carolina State University

Presenting Author Biography: Nicole Felice is an incoming Ph.D. student of the Operations Research Graduate Program at North Carolina State
University. Her research interests are modeling and optimization of stochastic systems and combining those with advanced computational practices.

Authors:

Nicole Felice North Carolina State University
David Eckman Texas A&M University
Susan Sanchez Naval Postgraduate School
Sara Shashaani North Carolina State University