Abstract

Gaussian process (GP) regression or kriging has been extensively applied in the engineering literature for the purposes of building a cheap-to-evaluate surrogate, within the contexts of multi-fidelity modeling, model calibration, and design optimization. With the ongoing automation of manufacturing and industrial practices as a part of Industry 4.0, there has been a greater need for advancing GP regression techniques to handle challenges such as high input dimensionality, data paucity or big data problems, these consist primarily of proposing efficient design of experiments, optimal data acquisition strategies, sparsifying covariance kernels, and other mathematical tricks. In this work, our attention is focused on the challenges of efficiently training a GP model, which, to the authors opinion, has attracted very little attention and is to-date poorly addressed. The performance of widely used training approaches such as maximum likelihood estimation and Markov Chain Monte Carlo (MCMC) sampling can deteriorate significantly in high-dimensional and big data problems and can lead to cost deficient implementations of critical importance to many industrial applications. Here, we compare an Adaptive Sequential Monte Carlo (ASMC) sampling algorithm to classic MCMC sampling strategies and we demonstrate the effectiveness of our implementation on several mathematical problems and challenging industry applications of varying complexity. The computational time savings of the ASMC approach manifest in large-scale problems helping us to push the boundaries of applicability and scalability of GPs for model calibration in various domains of the industry, including but not limited to design automation, design engineering, smart manufacturing, predictive maintenance, and supply chain manufacturing.

References

1.
Williams
,
C. K.
, and
Rasmussen
,
C. E.
,
2006
,
Gaussian Processes for Machine Learning
,
MIT Press
,
Cambridge, MA
.
2.
Bae
,
S.
,
Park
,
C.
, and
Kim
,
N. H.
,
2020
, “
Estimating Effect of Additional Sample on Uncertainty Reduction in Reliability Analysis Using Gaussian Process
,”
ASME J. Mech. Des.
,
142
(
11
), p.
111706
. https://doi.org/10.1115/1.4047002
3.
Li
,
M.
, and
Wang
,
Z.
,
2018
, “
Confidence-Driven Design Optimization Using Gaussian Process Metamodeling With Insufficient Data
,”
ASME J. Mech. Des.
,
140
(
12
). 10.1115/1.4040985
4.
Bostanabad
,
R.
,
Chan
,
Y.-C.
,
Wang
,
L.
,
Zhu
,
P.
, and
Chen
,
W.
,
2019
, “
Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111402
. 10.1115/1.4044257
5.
Ghoreishi
,
S. F.
,
Friedman
,
S.
, and
Allaire
,
D. L.
,
2019
, “
Adaptive Dimensionality Reduction for Fast Sequential Optimization With Gaussian Processes
,”
ASME J. Mech. Des.
,
141
(
7
), p.
071404
. 10.1115/1.4043202
6.
Pan
,
J.-X.
, and
Fang
,
K.-T.
,
2002
, “
Maximum Likelihood Estimation
,”
Growth Curve Models and Statistical Diagnostics
.
Springer
, pp.
77
158
.
7.
Kennedy
,
M. C.
, and
O’Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models
,”
J. R. Stat. Soc.: Ser. B (Statistical Methodology)
,
63
(
3
), pp.
425
464
. 10.1111/1467-9868.00294
8.
Robert
,
C.
, and
Casella
,
G.
,
2013
,
Monte Carlo Statistical Methods
,
Springer Science & Business Media
,
New York
.
9.
Gelman
,
A.
,
Carlin
,
J. B.
,
Stern
,
H. S.
, and
Rubin
,
D. B.
,
1995
,
Bayesian Data Analysis
,
Chapman and Hall/CRC
,
Boca Raton, FL
.
10.
Cowles
,
M. K.
, and
Carlin
,
B. P.
,
1996
, “
Markov Chain Monte Carlo Convergence Diagnostics: A Comparative Review
,”
J. Am. Stat. Assoc.
,
91
(
434
), pp.
883
904
. 10.1080/01621459.1996.10476956
11.
Gilks
,
W. R.
,
Richardson
,
S.
, and
Spiegelhalter
,
D.
,
1995
,
Markov Chain Monte Carlo in Practice
,
CRC press
,
Boca Raton, FL
.
12.
Green
,
P. J.
,
1995
, “
Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination
,”
Biometrika
,
82
(
4
), pp.
711
732
. 10.1093/biomet/82.4.711
13.
Ghahramani
,
Z.
, and
Rasmussen
,
C. E.
,
2003
, “
Bayesian Monte Carlo
,”
Advances in Neural Information Processing Systems
,
Vancouver, British Columbia, CA
.
14.
Peng
,
H.
,
Zhe
,
S.
,
Zhang
,
X.
, and
Qi
,
Y.
,
2017
, “
Asynchronous Distributed Variational Gaussian Process for Regression
,”
Proceedings of the 34th International Conference on Machine Learning-Vol. 70
, JMLR.org, pp.
2788
2797
.
15.
Tsilifis
,
P.
,
Pandita
,
P.
,
Ghosh
,
S.
,
Andreoli
,
V.
,
Vandeputte
,
T.
, and
Wang
,
L.
,
2020
,
Bayesian Learning of Orthogonal Embeddings for Multi-Fidelity Gaussian Processes. arXiv, 2008.02386
.
16.
Hensman
,
J.
,
Fusi
,
N.
, and
Lawrence
,
N. D.
,
2013
, “
Gaussian Processes for Big Data
,”
Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence
,
Arlington, VA
.
17.
Ghosh
,
S.
,
Asher
,
I.
,
Kristensen
,
J.
,
Ling
,
Y.
,
Ryan
,
K.
, and
Wang
,
L.
,
2018
, “
Bayesian Multi-Source Modeling With Legacy Data
,”
AIAA Non-Deterministic Approaches Conference
,
Kissimmee, FL
.
18.
Kristensen
,
J.
,
Ling
,
Y.
,
Asher
,
I.
, and
Wang
,
L.
,
2016
, “
Expected-Improvement-Based Methods for Adaptive Sampling in Multi-Objective Optimization Problems
,”
ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
.
19.
Lv
,
L.
,
Zong
,
C.
,
Zhang
,
C.
,
Song
,
X.
, and
Sun
,
W.
,
2021
, “
Multi-Fidelity Surrogate Model Based on Canonical Correlation Analysis and Least Square
,”
ASME J. Mech. Des.
,
143
(
2
), p.
021705
. 10.1115/1.4047686
20.
Liu
,
D.
, and
Wang
,
Y.
,
2019
, “
Multi-Fidelity Physics-Constrained Neural Network and Its Application in Materials Modeling
,”
ASME J. Mech. Des.
,
141
(
12
), p.
121403
. 10.1115/1.4044400
21.
Sarkar
,
S.
,
Mondal
,
S.
,
Joly
,
M.
,
Lynch
,
M. E.
,
Bopardikar
,
S. D.
,
Acharya
,
R.
, and
Perdikaris
,
P.
,
2019
, “
Multifidelity and Multiscale Bayesian Framework for High-Dimensional Engineering Design and Calibration
,”
ASME J. Mech. Des.
,
141
(
12
), p.
121001
. 10.1115/1.4044598
22.
Kennedy
,
M.
, and
O’Hagan
,
A.
,
2001
, “
Bayesian Calibration of Computer Models (With Discussion)
,”
J. R. Stat. Soc. (Series B)
,
68
, pp.
425
464
.
23.
Arendt
,
P. D.
,
Apley
,
D. W.
,
Chen
,
W.
,
Lamb
,
D.
, and
Gorsich
,
D.
,
2012
, “
Improving Identifiability in Model Calibration Using Multiple Responses
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100909
. 10.1115/1.4007573
24.
Arendt
,
P. D.
,
Apley
,
D. W.
, and
Chen
,
W.
,
2012
, “
Quantification of Model Uncertainty: Calibration, Model Discrepancy, and Identifiability
,”
ASME J. Mech. Des.
,
134
(
10
), p.
100908
. 10.1115/1.4007390
25.
O’Hagan
,
A.
,
1987
, “
Monte Carlo is Fundamentally Unsound
,”
The Statistician
,
36
(
2/3
), pp.
247
249
. 10.2307/2348519
26.
Haario
,
H.
,
Laine
,
M.
,
Lehtinen
,
M.
,
Saksman
,
E.
, and
Tamminen
,
J.
,
2004
, “
Markov Chain Monte Carlo Methods for High Dimensional Inversion in Remote Sensing
,”
J. R. Stat. Soc.: Ser. B (Statistical Methodology)
,
66
(
3
), pp.
591
607
. 10.1111/j.1467-9868.2004.02053.x
27.
Doucet
,
A.
,
De Freitas
,
N.
, and
Gordon
,
N.
,
2001
, “An Introduction to Sequential Monte Carlo Methods,”
Sequential Monte Carlo Methods in Practice
,
Springer
, pp.
3
14
.
28.
Doucet
,
Arnaud
,
de Freitas
,
Nando
, and
Gordon
,
Neil
,
2001
,
Sequential Monte Carlo Methods in Practice
,
Springer
,
New York, NY
.
29.
Andrieu
,
C.
,
Doucet
,
A.
, and
Punskaya
,
E.
,
2001
, “
Sequential Monte Carlo Methods for Optimal Filtering
,” Sequential Monte Carlo Methods in Practice,
Springer
, pp.
79
95
.
30.
Andrieu
,
C.
,
Doucet
,
A.
, and
Holenstein
,
R.
,
2010
, “
Particle Markov Chain Monte Carlo Methods
,”
J. R. Stat. Soc.: Ser. B (Statistical Methodology)
,
72
(
3
), pp.
269
342
. 10.1111/j.1467-9868.2009.00736.x
31.
Bilionis
,
I.
, and
Koutsourelakis
,
P.-S.
,
2012
, “
Free Energy Computations by Minimization of Kullback–leibler Divergence: An Efficient Adaptive Biasing Potential Method for Sparse Representations
,”
J. Comput. Phys.
,
231
(
9
), pp.
3849
3870
. 10.1016/j.jcp.2012.01.033
32.
Bilionis
,
I.
,
Drewniak
,
B. A.
, and
Constantinescu
,
E. M.
,
2015
, “
Crop Physiology Calibration in the CLM
,”
Geosci. Model Dev.
,
8
(
4
), pp.
1071
1083
. 10.5194/gmd-8-1071-2015
33.
Kristensen
,
J.
,
Bilionis
,
I.
, and
Zabaras
,
N.
,
2013
, “
Relative Entropy As Model Selection Tool in Cluster Expansions
,”
Phys. Rev. B
,
87
(
17
), p.
174112
. 10.1103/PhysRevB.87.174112
34.
Pandita
,
P.
,
Kristensen
,
J.
, and
Wang
,
L.
,
2019
, “
Towards Scalable Gaussian Process Modeling
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
.
35.
Kristensen
,
J.
,
Bilionis
,
I.
, and
Zabaras
,
N.
,
2017
, “
Adaptive Simulation Selection for the Discovery of the Ground State Line of Binary Alloys with a Limited Computational Budget
,”
Recent Progress and Modern Challenges in Applied Mathematics, Modeling and Computational Science
,
Springer
, pp.
185
211
.
36.
Kennedy
,
M.
, and
O’Hagan
,
A.
,
2000
, “
Predicting the Output From a Complex Computer Code When Fast Approximations Are Available
,”
Biometrika
,
87
(
1
), pp.
1
13
. 10.1093/biomet/87.1.1
37.
Higdon
,
D.
,
Nakhleh
,
C.
, and
Williams
,
B.
,
2008
, “
A Bayesian Calibration Approach to the Thermal Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2431
2441
. 10.1016/j.cma.2007.05.031
38.
Gattiker
,
J.
,
Myers
,
K.
,
Williams
,
B.
,
Higdon
,
D.
,
Carzolio
,
M.
, and
Hoegh
,
A.
,
2017
,
Handbook of Uncertainty Quantification
,
Springer: Cham, Switzerland
, pp.
1
41
.
39.
Wang
,
L.
,
Fang
,
X.
,
Subramaniyan
,
A.
,
Jothiprasad
,
G.
,
Gardner
,
M.
,
Kale
,
A.
,
Akkaram
,
S.
,
Beeson
,
D.
,
Wiggs
,
G.
, and
Nelson
,
J.
,
2011
, “
Challenges in Uncertainty, Calibration, Validation and Predictability of Engineering Analysis Models
,”
Proceedings of ASME Turbo Expo 2011
,
Vancouver, Canada
,
June
.
40.
Ghosh
,
S.
,
Pandita
,
P.
,
Atkinson
,
S.
,
Subber
,
W.
,
Zhang
,
Y.
,
Kumar
,
N. C.
,
Chakrabarti
,
S.
, and
Wang
,
L.
,
2020
, “
Advances in Bayesian Probabilistic Modeling for Industrial Applications
,”
ASCE-ASME J. Risk Uncert in Engrg Sys. Part B: Mech. Engrg.
,
6
(
3
), p.
1
. 10.1115/1.4046747
41.
Gelman
,
A.
,
Roberts
,
G. O.
, and
Gilks
,
W. R.
,
1996
, “
Efficient Metropolis Jumping Rules
,”
Bayesian Stat.
,
5
(
599–608
), p.
42
.
42.
Gordon
,
N. J.
,
Salmond
,
D. J.
, and
Smith
,
A. F.
,
1993
, “
Novel Approach to Nonlinear/Non-Gaussian Bayesian State Estimation
,”
IEE Proceedings F (Radar and Signal Processing)
,
140
, pp.
107
113
.
43.
Brent
,
R. P.
,
1980
, “
An Improved Monte Carlo Factorization Algorithm
,”
BIT Numer. Math.
,
20
(
2
), pp.
176
184
. 10.1007/BF01933190
44.
Poudou
,
O.
, and
Pierre
,
C.
,
2003
, “
Hybrid Frequency-Time Domain Methods for the Analysis of Complex Structural Systems with Dry Friction Damping
,”
44th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference
,
Norfolk, VA
.
You do not currently have access to this content.