Automated synthesis of steady-state continuous processes using reinforcement learning

doi:10.1007/s11705-021-2055-9

Front. Chem. Sci. Eng.

2022, Vol. 16

Issue (2) : 288-302 https://doi.org/10.1007/s11705-021-2055-9

RESEARCH ARTICLE

Automated synthesis of steady-state continuous processes using reinforcement learning

Quirin Göttl¹(

), Dominik G. Grimm^2,^3,⁴, Jakob Burger¹

¹. Technical University of Munich, Campus Straubing for Biotechnology and Sustainability, Laboratory of Chemical Process Engineering, 94315 Straubing, Germany
². Technical University of Munich, Campus Straubing for Biotechnology and Sustainability, 94315 Straubing, Germany
³. Weihenstephan-Triesdorf University of Applied Sciences, 94315 Straubing, Germany
⁴. Technical University of Munich, Department of Informatics, 85748 Garching, Germany

Download: PDF(833 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Automated flowsheet synthesis is an important field in computer-aided process engineering. The present work demonstrates how reinforcement learning can be used for automated flowsheet synthesis without any heuristics or prior knowledge of conceptual design. The environment consists of a steady-state flowsheet simulator that contains all physical knowledge. An agent is trained to take discrete actions and sequentially build up flowsheets that solve a given process problem. A novel method named SynGameZero is developed to ensure good exploration schemes in the complex problem. Therein, flowsheet synthesis is modelled as a game of two competing players. The agent plays this game against itself during training and consists of an artificial neural network and a tree search for forward planning. The method is applied successfully to a reaction-distillation process in a quaternary system.

Keywords automated process synthesis flowsheet synthesis artificial intelligence machine learning reinforcement learning

Corresponding Author(s): Quirin Göttl

Just Accepted Date: 29 March 2021 Online First Date: 18 May 2021 Issue Date: 10 January 2022

Cite this article:

Quirin Göttl,Dominik G. Grimm,Jakob Burger. Automated synthesis of steady-state continuous processes using reinforcement learning[J]. Front. Chem. Sci. Eng., 2022, 16(2): 288-302.

URL:

https://academic.hep.com.cn/fcse/EN/10.1007/s11705-021-2055-9
https://academic.hep.com.cn/fcse/EN/Y2022/V16/I2/288

Fig.1 Scheme of the RL framework for flowsheet synthesis using only discrete decisions without prior knowledge.

Case	$I D$ /k€	$I R$ /k€	$I M$ /k€	$p A$ /(k€·kmol^–1)	$p B$ /(k€·kmol^–1)	$p C$ /(k€·kmol^–1)	$p D$ /(k€·kmol^–1)
Case 1	10000	10000	1000	1	1	1	1
Case 2	10000	10000	1000	–0.125	–0.125	2	2

Tab.1 Investment costs

I u

for distillation D, reactor R and mixer M, and prices

p i

of compounds A, B, C, D used in the determination of the net present value in the present work

Case	$N steps$	$N matrix$	$N memory$	$N batch$	$N layer$	$N node$	$K$	a	β
Case 1	5000	10	256	32	2	32	20	–0.9	0.0001
Case 2	20000	10	256	32	2	64	40	–0.9	0.0001

Tab.2 Numerical tuning parameters used in the examples

Fig.2 Construction of flowsheet matrix

F

along an example flowsheet (

F

contains the information of the stream table combined with structural information on the flowsheet. See text for an explanation of the nomenclature).

Fig.3 Structure of the agent’s ANN in the SynGameZero method (The ANN has an actor-critic architecture. It calculates from the state input

s

both a policy vector π and a scalar value

v

. To obtain

p

, infeasible actions are filtered out of the vector π).

Fig.4 Example tree search at the beginning of the game (flowsheets of both players empty) with three possible actions {T, D1, R} (Unexplored leaf nodes are shown with dotted frames. Terminated flowsheets and terminal nodes are marked with bold frames. The order of the two flowsheets is switched after every action. The current player is the one who takes the next action. His/her flowsheet is shown in the left half of the nodes).

Fig.5 Illustrative example for the evolution of the agent during the training process in case 1 (Flowsheets proposed by the agent to separate an equimolar quaternary mixture are shown).

Fig.6 Example for the flowsheets proposed by the trained agent after the training process (The panels (1), (2), (3) and (4) refer to feed situations 1, 2, 3 and 4, respectively).

Fig.7 Example for the evolution of the agent during the training process for situation 1 (The 3D plots show the value of three highlighted actions of the ANN’s output vector

p

over a subset of the feed space (

n ˙ A

n ˙ B

n ˙ C

n ˙ D

) for the first action of the agent. Action 1 is mixing both feed streams. Action 2 is placing a distillation column of type D3 at the CD feed stream. Action 3 refers to placing a reactor R at the AB feed stream).

Metric	Situation 1	Situation 2	Situation 3	Situation 4
$R 1$	0.84	0.99	1	1
$R 2$	1.08	1.13	1	1
$R 3$	1.12	1.13	1	1
$R 4$	0.88	0.99	−	−

Tab.3 Average performance metrics as defined in Eqs. (14)–(17) for 5 training processes in case 2^a)

1	A W Westerberg. A retrospective on design and process synthesis. Computers & Chemical Engineering, 2004, 28(4): 447–458 https://doi.org/10.1016/j.compchemeng.2003.09.029
2	G Stephanopoulos, G V Reklaitis. Process systems engineering: from solvay to modern bio- and nanotechnology. A history of development, successes and prospects for the future. Chemical Engineering Science, 2011, 66(19): 4272–4306 https://doi.org/10.1016/j.ces.2011.05.049
3	J J Siirola. Strategic process synthesis: advances in the hierarchical approach. Computers & Chemical Engineering, 1996, 20(S2): S1637–S1643 https://doi.org/10.1016/0098-1354(96)85982-5
4	Q Chen, I E Grossmann. Recent developments and challenges in optimization-based process synthesis. Annual Review of Chemical and Biomolecular Engineering, 2017, 8(1): 249–283 https://doi.org/10.1146/annurev-chembioeng-080615-033546
5	H Yeomans, I E Grossmann. A systematic modeling framework of superstructure optimization in process synthesis. Computers & Chemical Engineering, 1999, 23(6): 709–731 https://doi.org/10.1016/S0098-1354(99)00003-4
6	G Stephanopoulos, A W Westerberg. Studies in process synthesis II, evolutionary synthesis of optimal process flowsheets. Chemical Engineering Science, 1976, 31(3): 195–204 https://doi.org/10.1016/0009-2509(76)85057-9
7	T Zhang, N V Sahinidis, J J Siirola. Pattern recognition in chemical process flowsheets. AIChE Journal. American Institute of Chemical Engineers, 2019, 65(2): 592–603 https://doi.org/10.1002/aic.16443
8	R Gani, J P O’Connell. A knowledge based system for the selection of thermodynamic models. Computers & Chemical Engineering, 1989, 13(4-5): 397–404 https://doi.org/10.1016/0098-1354(89)85019-7
9	R L Kirkwood, M H Locke, J M Douglas. A prototype expert system for synthesizing chemical process flowsheets. Computers & Chemical Engineering, 1988, 12(4): 329–343 https://doi.org/10.1016/0098-1354(88)85045-2
10	A K Tula, M R Eden, R Gani. Process synthesis, design and analysis using a process-group contribution method. Computers & Chemical Engineering, 2015, 81: 245–259 https://doi.org/10.1016/j.compchemeng.2015.04.019
11	M M Daichendt, I E Grossmann. Integration of hierarchical decomposition and mathematical programming for the synthesis of process flowsheets. Computers & Chemical Engineering, 1997, 22(1-2): 147–175 https://doi.org/10.1016/S0098-1354(97)88451-7
12	M Martin, T A Adams II. Challenges and future directions for process and product synthesis and design. Computers & Chemical Engineering, 2019, 128: 421–436 https://doi.org/10.1016/j.compchemeng.2019.06.022
13	I E Grossmann, I Harjunkoski. Process systems engineering: academic and industrial perspectives. Computers & Chemical Engineering, 2019, 126: 474–484 https://doi.org/10.1016/j.compchemeng.2019.04.028
14	G Stephanopoulos. Artificial intelligence in process engineering—current state and future trends. Computers & Chemical Engineering, 1990, 14(11): 1259–1270 https://doi.org/10.1016/0098-1354(90)80006-W
15	G Stephanopoulos, C Han. Intelligent systems in process engineering: a review. Computers & Chemical Engineering, 1996, 20(6-7): 143–191 https://doi.org/10.1016/0098-1354(95)00194-8
16	D M Dimiduk, E A Holm, S R Niezgoda. Perspectives on the impact of machine learning, deep learning, and artificial intelligence on materials, processes, and structures engineering. Integrating Materials and Manufacturing Innovation, 2018, 7(3): 157–172 https://doi.org/10.1007/s40192-018-0117-8
17	V Venkatasubramanian. The promise of artificial intelligence in chemical engineering: is it here, finally? AIChE Journal. American Institute of Chemical Engineers, 2019, 65(2): 466–478 https://doi.org/10.1002/aic.16489
18	J H Lee, J Shin, M J Realff. Machine learning: overview of the recent progresses and implications for the process systems engineering field. Computers & Chemical Engineering, 2018, 114: 111–121 https://doi.org/10.1016/j.compchemeng.2017.10.008
19	J Eason, S Cremaschi. Adaptive sequential sampling for surrogate model generation with artificial neural networks. Computers & Chemical Engineering, 2014, 68: 220–232 https://doi.org/10.1016/j.compchemeng.2014.05.021
20	I Fahmi, S Cremaschi. Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Computers & Chemical Engineering, 2012, 46: 105–123 https://doi.org/10.1016/j.compchemeng.2012.06.006
21	F A N Fernandes. Optimization of fischer‐tropsch synthesis using neural networks. Chemical Engineering & Technology, 2006, 29(4): 449–453 https://doi.org/10.1002/ceat.200500310
22	R S Sutton, A G Barto. Reinforcement Learning: An Introduction. 2nd ed. Cambridge, MA: The MIT Press, 2018
23	M Lapan. Deep Reinforcement Learning Hands-On. 1st ed. Birmingham, E.K.: Packt Publishing Ltd., 2018
24	J Shin, T A Badgwell, K H Liu, J H Lee. Reinforcement learning—overview of recent progress and implications for process control. Computers & Chemical Engineering, 2019, 127: 282–294 https://doi.org/10.1016/j.compchemeng.2019.05.029
25	Z Zhou, X Li, R N Zare. Optimizing chemical reactions with deep reinforcement learning. ACS Central Science, 2017, 3(12): 1337–1344 https://doi.org/10.1021/acscentsci.7b00492
26	A Khan, A Lapkin. Searching for optimal process routes: a reinforcement learning approach. Computers & Chemical Engineering, 2020, 141: 107027 https://doi.org/10.1016/j.compchemeng.2020.107027
27	D Silver, J Schrittwieser, K Simonyan, I Antonoglou, A Huang, A Guez, T Hubert, L Baker, M Lai, A Bolton, Y Chen, T Lillicrap, F Hui, L Sifre, G van den Driessche, T Graepel, D Hassabis. Mastering the game of Go without human knowledge. Nature, 2017, 550(7676): 354–359 https://doi.org/10.1038/nature24270
28	D Silver, T Hubert, J Schrittwieser, I Antonoglou, M Lai, A Guez, M Lanctot, L Sifre, D Kumaran, T Graepel, T Lillicrap, K Simonyan, D Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 2018, 362(6419): 1140–1144 https://doi.org/10.1126/science.aar6404
29	Y Wang, Y Li, Y Song, X Rong. The influence of the activation function in a convolution neural network model of facial expression recognition. Applied Sciences (Basel, Switzerland), 2020, 10(5): 1897 https://doi.org/10.3390/app10051897
30	M Abadi, A Agarwal, P Barham, E Brevdo, Z Chen, C Citro, G S Corrado, A Davis, J Dean, M Devin, et al.. TensorFlow: large-scale machine learning on heterogeneous systems. 2015, arXiv: 1603.04467
31	E Alpaydin. Introduction to Machine Learning. 2nd ed. Cambridge, MA: The MIT Press, 2010
32	A Zinser, L Rihko-Struckmann, K Sundmacher. Computationally efficient steady-state process simulation by applying a simultaneous dynamic method. Computer-Aided Chemical Engineering, 2016, 38: 517–522 https://doi.org/10.1016/B978-0-444-63428-3.50091-6
33	A Hoffmann, M Bortz, J Burger, H Hasse, K H Küfer. A new scheme for process simulation by optimization: distillation as an example. Computer-Aided Chemical Engineering, 2016, 38: 205–210 https://doi.org/10.1016/B978-0-444-63428-3.50039-4
34	M Hausknecht, P Stone. Deep reinforcement learning in parameterized action space. 2015, arXiv: 1511.04143
35	J Xiong, Q Wang, Z Yang, P Sun, L Han, Y Zheng, H Fu, T Zhang, J Liu, H Liu. Parametrized deep q-networks learning: reinforcement learning with discrete-continuous hybrid action space. 2018, arXiv: 1810.06394
36	M Neunert, A Abdolmaleki, M Wulfmeier, T Lampe, J T Springenberg, R Hafner, F Romano, J Buchli, N Heess, M Riedmiller. Continuous-discrete reinforcement learning for hybrid control in robotics. 2020, arXiv: 2001.00449v1

[1]	Haoqin Fang, Jianzhao Zhou, Zhenyu Wang, Ziqi Qiu, Yihua Sun, Yue Lin, Ke Chen, Xiantai Zhou, Ming Pan. Hybrid method integrating machine learning and particle swarm optimization for smart chemical process operations[J]. Front. Chem. Sci. Eng., 2022, 16(2): 274-287.
[2]	Ewan Chee, Wee Chin Wong, Xiaonan Wang. An integrated approach for machine-learning-based system identification of dynamical systems under control: application towards the model predictive control of a highly nonlinear reactor system[J]. Front. Chem. Sci. Eng., 2022, 16(2): 237-250.
[3]	Patrick Otto Ludl, Raoul Heese, Johannes Höller, Norbert Asprion, Michael Bortz. Using machine learning models to explore the solution space of large nonlinear systems underlying flowsheet simulations with constraints[J]. Front. Chem. Sci. Eng., 2022, 16(2): 183-197.
[4]	J. Sargolzaei, A. Hedayati Moghaddam. Predicting the yield of pomegranate oil from supercritical extraction using artificial neural networks and an adaptive-network-based fuzzy inference system[J]. Front Chem Sci Eng, 2013, 7(3): 357-365.

Viewed

Full text

Abstract

Cited

Shared

Discussed