Abstract

Recently, it was demonstrated that the design synthesis of truss structures can be modeled as a Markov decision process (MDP) and solved using a tabular reinforcement learning method. In this setting, each state corresponds to a specific design configuration represented as a finite graph. However, when the structural design domain is relatively large, and depending on the constraints, the dimensionality of the state space becomes quite large rendering tabular reinforcement learning algorithms inefficient. Hence, in this study, the design synthesis MDP framework is significantly extended to solve structural design problems with large state spaces, by integrating deep reinforcement learning (DRL) into the general MDP framework. This is beneficial because with DRL, a deep neural network can be used to approximate the state-action value function, such that the network has much fewer parameters than the cardinality of the state space. This parameterization relies upon a problem relevant set of features and reward function. Thus, for this extended DRL design synthesis (DRLDS) framework, a compact set of features and a reward function are devised that are suitable for structural design problems where structural configurations are represented as finite graphs. Through the application of seven different structural design synthesis examples, the DRLDS framework is demonstrated to be capable of adeptly learning optimal policies that synthesize high, if not the highest, performing design solutions more frequently. The DRLDS framework does this with fewer finite element model evaluations than other considered alternative methods, further demonstrating the effectiveness of the developed set of features and reward function.

References

1.
Cagan
,
J.
,
Campbell
,
M. I.
,
Finger
,
S.
, and
Tomiyama
,
T.
,
2005
, “
A Framework for Computational Design Synthesis: Model and Applications
,”
ASME J. Comput. Inf. Sci. Eng.
,
5
(
3
), pp.
171
181
.
2.
Helms
,
B.
,
Shea
,
K.
, and
Hoisl
,
F.
,
2009
, “
A Framework for Computational Design Synthesis Based on Graph-Grammars and Function-Behavior-Structure
,”
International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 49057
,
San Diego, CA
,
Aug. 30–Sept. 2
, pp.
841
851
.
3.
Antonsson
,
E. K.
, and
Cagan
,
J.
,
2005
,
Formal Engineering Design Synthesis
,
Springer-Verlag London
,
London
.
4.
Chakrabarti
,
A.
,
2013
,
Engineering Design Synthesis: Understanding, Approaches and Tools
,
Springer Science & Business Media
,
London
.
5.
Hooshmand
,
A.
, and
Campbell
,
M. I.
,
2016
, “
Truss Layout Design and Optimization Using a Generative Synthesis Approach
,”
Comput. Struct.
,
163
, pp.
1
28
.
6.
Vale
,
C. A.
, and
Shea
,
K.
,
2003
, “
A Machine Learning-Based Approach to Accelerating Computational Design Synthesis
,”
In DS 31: Proceedings of ICED 03, the 14th International Conference on Engineering Design
,
Stockholm, Sweden
,
Aug. 19–21
, pp.
183
184
.
7.
Campbell
,
M. I.
, and
Shea
,
K.
,
2014
, “
Computational Design Synthesis
,”
AI EDAM
,
28
(
3
), pp.
207
208
.
8.
Königseder
,
C.
, and
Shea
,
K.
,
2016
, “
Visualizing Relations Between Grammar Rules, Objectives, and Search Space Exploration in Grammar-Based Computational Design Synthesis
,”
ASME J. Mech. Des.
,
138
(
10
), p.
101101
.
9.
Königseder
,
C.
, and
Shea
,
K.
,
2016
, “
Comparing Strategies for Topologic and Parametric Rule Application in Automated Computational Design Synthesis
,”
ASME J. Mech. Des.
,
138
(
1
), p.
011102
.
10.
Ororbia
,
M. E.
, and
Warn
,
G. P.
,
2020
, “
Structural Design Synthesis Through a Sequential Decision Process
,”
In ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Virtual Online
,
Aug. 17–19
,
American Society of Mechanical Engineers
.
11.
Ororbia
,
M. E.
, and
Warn
,
G. P.
,
2021
, “
Design Synthesis Through a Markov Decision Process and Reinforcement Learning Framework
,”
ASME J. Comput. Inf. Sci. Eng.
,
22
(
2
), p.
021002
.
12.
Watkins
,
C. J.
, and
Dayan
,
P.
,
1992
, “
Q-Learning
,”
Mach. Learn.
,
8
(
3–4
), pp.
279
292
.
13.
Burnap
,
A.
,
Liu
,
Y.
,
Pan
,
Y.
,
Lee
,
H.
,
Gonzalez
,
R.
, and
Papalambros
,
P. Y.
,
2016
, “
Estimating and Exploring the Product Form Design Space Using Deep Generative Models
,”
In ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Charlotte, NC
,
Aug. 21–24
,
American Society of Mechanical Engineers Digital Collection
.
14.
Dering
,
M. L.
, and
Tucker
,
C. S.
,
2017
, “
Generative Adversarial Networks for Increasing the Veracity of Big Data
,”
In 2017 IEEE International Conference on Big Data
,
Boston, MA
,
Dec. 11–14
,
IEEE
, pp.
2595
2602
.
15.
Dering
,
M. L.
, and
Tucker
,
C. S.
,
2017
, “
Implications of Generative Models in Government
,”
In 2017 AAAI Fall Symposium Series
,
Washington, DC
,
Nov. 9–11
.
16.
Shu
,
D.
,
Cunningham
,
J.
,
Stump
,
G.
,
Miller
,
S. W.
,
Yukish
,
M. A.
,
Simpson
,
T. W.
, and
Tucker
,
C. S.
,
2020
, “
3D Design Using Generative Adversarial Networks and Physics-Based Validation
,”
ASME J. Mech. Des.
,
142
(
7
), p.
071701
.
17.
Heyrani Nobari
,
A.
,
Chen
,
W. W.
, and
Ahmed
,
F.
,
2022
, “
Range-GAN: Design Synthesis Under Constraints Using Conditional Generative Adversarial Networks
,”
ASME J. Mech. Des.
,
144
(
2
) p.
021708
.
18.
Regenwetter
,
L.
,
Nobari
,
A. H.
, and
Ahmed
,
F.
,
2022
, “
Deep Generative Models in Engineering Design: A Review
,”
ASME J. Mech. Des.
,
144
(
7
), p.
071704
.
19.
Yu
,
Y.
,
Hur
,
T.
,
Jung
,
J.
, and
Jang
,
I. G.
,
2019
, “
Deep Learning for Determining a Near-Optimal Topological Design Without Any Iteration
,”
Struct. Multidiscipl. Optim.
,
59
(
3
), pp.
787
799
.
20.
Jang
,
S.
,
Yoo
,
S.
, and
Kang
,
N.
,
2022
, “
Generative design by reinforcement learning: enhancing the diversity of topology optimization designs
,”
Computer-Aided Design
,
146
, p.
103225
.
21.
Sun
,
H.
, and
Ma
,
L.
,
2020
, “
Generative Design by Using Exploration Approaches of Reinforcement Learning in Density-Based Structural Topology Optimization
,”
Designs
,
4
(
2
), p.
10
.
22.
Chen
,
W.
, and
Ahmed
,
F.
,
2021
, “
PaDGAN: Learning to Generate High-Quality Novel Designs
,”
ASME J. Mech. Des.
,
143
(
3
), p.
031703
.
23.
Zhang
,
W.
,
Yang
,
Z.
,
Jiang
,
H.
,
Nigam
,
S.
,
Yamakawa
,
S.
,
Furuhata
,
T.
,
Shimada
,
K.
, and
Kara
,
L. B.
,
2019
, “
3D Shape Synthesis for Conceptual Design and Optimization Using Variational Autoencoders
,” In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol.
59186
,
American Society of Mechanical Engineers
, Paper No. V02AT03A017.
24.
Vermeer
,
K.
,
Kuppens
,
R.
, and
Herder
,
J.
,
2018
, “
Kinematic Synthesis Using Reinforcement Learning
,”
In ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Quebec City, Quebec, Canada
,
Aug. 26–29
,
American Society of Mechanical Engineers
.
25.
Raina
,
A.
,
McComb
,
C.
, and
Cagan
,
J.
,
2019
, “
Learning to Design From Humans: Imitating Human Designers Through Deep Learning
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111102
.
26.
Puentes
,
L.
,
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2020
, “
Modeling a Strategic Human Engineering Design Process: Human-Inspired Heuristic Guidance Through Learned Visual Design Agents
,”
In Proceedings of the Design Society: DESIGN Conference, Vol. 1
,
Virtual, Online
,
Oct. 26–29
,
Cambridge University Press
, pp.
355
364
.
27.
Raina
,
A.
,
Puentes
,
L.
,
Cagan
,
J.
, and
McComb
,
C.
,
2021
, “
Goal-Directed Design Agents: Integrating Visual Imitation With One-Step Lookahead Optimization for Generative Design
,”
ASME J. Mech. Des.
,
143
(
12
), p.
124501
.
28.
Hayashi
,
K.
, and
Ohsaki
,
M.
,
2020
, “
Reinforcement Learning and Graph Embedding for Binary Truss Topology Optimization Under Stress and Displacement Constraints
,”
Front. Built Environ.
,
6
, p.
59
.
29.
Dorn
,
W. S.
,
Gomory
,
R. E.
, and
Greenberg
,
H. J.
,
1964
, “
Automatic Design of Optimal Structures
,”
J. de Mecanique
,
3
(
1
), pp.
25
52
.
30.
Zhu
,
S.
,
Ohsaki
,
M.
,
Hayashi
,
K.
, and
Guo
,
X.
,
2021
, “
Machine-Specified Ground Structures for Topology Optimization of Binary Trusses Using Graph Embedding Policy Network
,”
Adv. Eng. Softw.
,
159
, p.
103032
.
31.
Sahachaisaree
,
S.
,
Sae-Ma
,
P.
, and
Nanakorn
,
P.
,
2020
, “
Two-Dimensional Truss Topology Design by Reinforcement Learning
,”
In ICSCEA 2019
,
Ho Chi Minh City, Vietnam
,
Oct. 24–26
,
Springer
, pp.
1237
1245
.
32.
Seshu
,
S.
, and
Reed
,
M. B.
,
1961
, Linear Graphs and Electrical Networks.
33.
Schmidt
,
L. C.
,
Shetty
,
H.
, and
Chase
,
S. C.
,
1998
, “
A Graph Grammar Approach for Structure Synthesis of Mechanisms
,” In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol.
80333
,
American Society of Mechanical Engineers
, Paper No. V003T03A034.
34.
Lin
,
Y.-s.
,
Shea
,
K.
,
Johnson
,
A.
,
Coultate
,
J.
, and
Pears
,
J.
,
2009
, “
A Method and Software Tool for Automated Gearbox Synthesis
,”
In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 49026
,
San Diego, CA
,
Aug. 30–Sept. 2
, pp.
111
121
.
35.
Jagadeesh
,
G. R.
,
Srikanthan
,
T.
, and
Quek
,
K.
,
2002
, “
Heuristic Techniques for Accelerating Hierarchical Routing on Road Networks
,”
IEEE Trans. Intell. Transp. Syst.
,
3
(
4
), pp.
301
309
.
36.
Kaveh
,
A.
,
1986
, “
Graph Theoretical Methods for Efficient Flexibility Analysis of Planar Trusses
,”
Comput. Struct.
,
23
(
4
), pp.
559
564
.
37.
Kaveh
,
A.
,
1991
, “
Graphs and Structures
,”
Comput. Struct.
,
40
(
4
), pp.
893
901
.
38.
Königseder
,
C.
,
Shea
,
K.
, and
Campbell
,
M. I.
,
2013
, “
Comparing a Graph-Grammar Approach to Genetic Algorithms for Computational Synthesis of PV Arrays
,”
In CIRP Design 2012
,
London, UK
,
Nov. 20
,
Springer
, pp.
105
114
.
39.
Lipson
,
H.
,
2008
, “
Evolutionary Synthesis of Kinematic Mechanisms
,”
AI EDAM
,
22
(
3
), pp.
195
205
.
40.
Bathe
,
K.-J.
,
2006
,
Finite Element Procedures
,
Klaus-Jurgen Bathe
,
Cambridge, MA
.
41.
Whalen
,
E.
, and
Mueller
,
C.
,
2022
, “
Toward Reusable Surrogate Models: Graph-Based Transfer Learning on Trusses
,”
ASME J. Mech. Des.
,
144
(
2
), p.
021704
.
42.
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2022
, “
Design Strategy Network: A Deep Hierarchical Framework to Represent Generative Design Strategies in Complex Action Spaces
,”
ASME J. Mech. Des.
,
144
(
2
), p.
021404
.
43.
Hayashi
,
K.
, and
Ohsaki
,
M.
,
2022
, “
Graph-Based Reinforcement Learning for Discrete Cross-Section Optimization of Planar Steel Frames
,”
Adv. Eng. Inform.
,
51
, p.
101512
.
44.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Graves
,
A.
,
Antonoglou
,
I.
,
Wierstra
,
D.
, and
Riedmiller
,
M.
,
2013
, “Playing Atari With Deep Reinforcement Learning”. arXiv preprint arXiv:1312.5602.
45.
Rumelhart
,
D. E.
,
Hinton
,
G. E.
, and
Williams
,
R. J.
,
1986
, “
Learning Representations by Back-Propagating Errors
,”
Nature
,
323
(
6088
), pp.
533
536
.
46.
Kingma
,
D. P.
, and
Ba
,
J.
,
2014
, “Adam: A Method for Stochastic Optimization”. arXiv preprint arXiv:1412.6980.
47.
Achtziger
,
W.
, and
Stolpe
,
M.
,
2008
, “
Global Optimization of Truss Topology With Discrete Bar Areas-Part I: Theory of Relaxed Problems
,”
Comput. Optim. Appl.
,
40
(
2
), pp.
247
280
.
48.
Achtziger
,
W.
, and
Stolpe
,
M.
,
2009
, “
Global Optimization of Truss Topology With Discrete Bar Areas-Part II: Implementation and Numerical Results
,”
Comput. Optim. Appl.
,
44
(
2
), p.
315
.
49.
Stolpe
,
M.
,
2015
, “
Truss Topology Optimization With Discrete Design Variables by Outer Approximation
,”
J. Global Optim.
,
61
(
1
), pp.
139
163
.
50.
Kaveh
,
A.
, and
Talatahari
,
S.
,
2009
, “
Particle Swarm Optimizer, Ant Colony Strategy and Harmony Search Scheme Hybridized for Optimization of Truss Structures
,”
Comput. Struct.
,
87
(
5–6
), pp.
267
283
.
51.
Li
,
L.
,
Huang
,
Z.
, and
Liu
,
F.
,
2009
, “
A Heuristic Particle Swarm Optimization Method for Truss Structures With Discrete Variables
,”
Comput. Struct.
,
87
(
7–8
), pp.
435
443
.
52.
Kripka
,
M.
,
2004
, “
Discrete Optimization of Trusses by Simulated Annealing
,”
J. Braz. Soc. Mech. Sci. Eng.
,
26
(
2
), pp.
170
173
.
53.
Kaveh
,
A.
, and
Kalatjari
,
V.
,
2003
, “
Topology Optimization of Trusses Using Genetic Algorithm, Force Method and Graph Theory
,”
Int. J. Numer. Methods Eng.
,
58
(
5
), pp.
771
791
.
54.
Silver
,
D.
,
Hubert
,
T.
,
Schrittwieser
,
J.
,
Antonoglou
,
I.
,
Lai
,
M.
,
Guez
,
A.
,
Lanctot
,
M.
,
Sifre
,
L.
,
Kumaran
,
D.
,
Graepel
,
T.
, and
Lillicrap
,
T.
,
2018
, “
A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play
,”
Science
,
362
(
6419
), pp.
1140
1144
.
You do not currently have access to this content.