An Overview of Performance Measurement for Demand Forecasting Based on Artificial Neural Networks
DOI:
https://doi.org/10.33568/rbs.4434Keywords:
decision support systems, artificial neural networks, performance measurements, supply chain managementAbstract
Demand forecasting is an essential task to match supply and demand. From a supplier’s view, demand forecasting is important to optimize supply chains and thus maximize profits. The ever-increasing availability of data that can be used as input factors for predictive models allows more and more sophistication for diverse forecasting tasks in the context of demand forecasting. On the one hand, increasingly complex models have been used for demand forecasting over the last years, from simple exponential smoothing methods and ARIMA models up to complex, hybrid (deep) artificial neural networks. On the other hand, little attention is paid to the methods that evaluate the forecasting performance of these models, which are essential for the selection from among potential forecasting models. In this article, we aim to answer the question of what are the most favourable measurements in recent literature on applied neural network demand forecasting for supply chain management. To this end, we analyzed 193 relevant publications in which demand forecasting was applied using artificial neural networks. We found that in artificial neural network demand forecasting used to evaluate forecasting performance, Mean Absolute Percentage Error, Root Mean Squared Error, Mean Squared Error and Mean Absolute Error are by far the most popular methods. Furthermore, we found that when forecasting performance measurements are combined, the most common combination is the combination of Mean Absolute Error, the Root Mean Squared Error and the Mean Absolute Error.
References
Aggarwal, (2018). Machine Learning with Shallow Neural Networks. In C. C. Aggarwal (Ed.) Neural Networks and Deep Learning (pp. 53–104). Springer International Publishing. https://doi.org/10.1007/978-3-319-94463-0_2
Andrawis, R. R., Atiya, A. F., & El-Shishiny, H. (2011). Combination of long term and short term forecasts, with application to tourism demand forecasting. International Journal of Forecasting, 27(3), 870–886. https://doi.org/10.1016/j.ijforecast.2010.05.019
Anisa, M. P., Irawan, H., & Widiyanesti, S. (2021). Forecasting demand factors of tourist arrivals in Indonesia’s tourism industry using recurrent neural network. IOP Conference Series: Materials Science and Engineering, 1077(1), 012035. https://doi.org/10.1088/1757-899x/1077/1/012035
Armstrong, J. S. (Ed.). (2001). Principles of forecasting: a handbook for researchers and practitioners. International Series in Operations Research & Management Science (Vol. 30). Kluwer Academic. https://doi.org/10.1007/978-0-306-47630-3
Armstrong, J. S., & Collopy, F. (1992). Error measures for generalizing about forecasting methods: Empirical comparisons. International journal of forecasting, 8(1), 69-80. https://doi.org/10.1016/0169-2070(92)90008-W
Babai, M. Z., Syntetos, A., & Teunter, R. (2014). Intermittent demand forecasting: An empirical study on accuracy and the risk of obsolescence. International Journal of Production Economics, 157, 212-219. https://doi.org/10.1016/j.ijpe.2014.08.019
Bowerman, B. L., O'Connell, R. T., Murphree, E., Huchendorf, S. C., Porter, D. C., & Schur, P. (2003). Business statistics in practice (pp. 728-730). McGraw-Hill.
Burger, C. J. S. C., Dohnal, M., Kathrada, M., & Law, R. (2001). A practitioners guide to time- series methods for tourism demand forecasting – a case study of Durban, South Africa. Tourism management, 22(4), 403-409. https://doi.org/10.1016/S0261-5177(00)00068-6
Byrne, R. F. (2012). Beyond Traditional Time-Series: Using Demand Sensing to Improve Forecasts in Volatile Times. Journal of Business Forecasting, 31(2).
Carbone, R., & Armstrong, J. S. (1982). Note. Evaluation of extrapolative forecasting methods: results of a survey of academicians and practitioners. Journal of Forecasting, 1(2), 215-217. https://doi.org/10.1002/for.3980010207
Carbonneau, R., Laframboise, K., & Vahidov, R. (2008). Application of machine learning techniques for supply chain demand forecasting. European journal of operational research, 184(3), 1140-1154. https://doi.org/10.1016/j.ejor.2006.12.004
Chawla, A., Singh, A., Lamba, A., Gangwani, N., & Soni, U. (2019). Demand Forecasting Using Artificial Neural Networks – A Case Study of American Retail Corporation. In Malik, H., Srivastava, S., Sood, Y., Ahmad, A. (Eds.) Applications of Artificial Intelligence Techniques in Engineering. (pp. 79-89). Springer, Singapore. https://doi.org/10.1007/978- 981-13-1822-1_8
Chai, T., & Draxler, R. R. (2014). Root mean square error (RMSE) or mean absolute error (MAE). Geoscientific model development discussions, 7(1), 1525-1534. https://doi.org/10.5194/gmdd-7-1525-2014
Chen, Z., & Yang, Y. (2004). Assessing forecast accuracy measures. Iowa State University.
Clements, M. P., Franses, P. H., & Swanson, N. R. (2004). Forecasting economic and financial time-series with non-linear models. International journal of forecasting, 20(2), 169-183. https://doi.org/10.1016/j.ijforecast.2003.10.004
Clements, M. P., & Hendry, D. F. (1993). On the limitations of comparing mean square forecast errors. Journal of Forecasting, 12(8), 617-637. https://doi.org/10.1002/for.3980120802
Constantino, H. A., Fernandes, P. O., & Teixeira, J. P. (2016). Tourism demand modelling and forecasting with artificial neural network models: The Mozambique case study. Tékhne, 14(2), 113-124. https://doi.org/10.1016/j.tekhne.2016.04.006
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018). Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1), 53-65. https://doi.org/10.1109/MSP.2017.2765202
De Gooijer, J. G., & Hyndman, R. J. (2006). 25 years of time series forecasting. International journal of forecasting, 22(3), 443-473. https://doi.org/10.1016/j.ijforecast.2006.01.001
Diebold, F. X., & Lopez, J. A. (1996). 8 Forecast evaluation and combination. Handbook of statistics, 14, 241-268. https://doi.org/10.1016/S0169-7161(96)14010-4
Diebold, F. X., & Mariano, R. S. (2002). Comparing predictive accuracy. Journal of Business & economic statistics, 20(1), 134-144. https://doi.org/10.1198/073500102753410444
Fildes, R., Ma, S., & Kolassa, S. (2022). Retail forecasting: Research and practice. International Journal of Forecasting, 38(4), 1283-1318. https://doi.org/10.1016/j.ijforecast.2019.06.004
Ghalehkhondabi, I., Ardjmand, E., Weckman, G. R., & Young, W. A. (2017). An overview of energy demand forecasting methods published in 2005–2015. Energy Systems, 8, 411-447. https://doi.org/10.1007/s12667-016-0203-y
Goodfellow, I. & Bengio, Y & Courville, A. (2018). Deep Learning – Das umfassende Handbuch. 1. Auflage. Verlags GmbH & Co. KG.
Granger, C. W. J., & Pesaran, M. H. (2000a). Economic and statistical measures of forecast accuracy. Journal of Forecasting, 19(7), 537-560. https://doi.org/10.1002/1099-131X(200012)19:7%3C537::AID-FOR769%3E3.0.CO;2-G
Granger, C. W. J., & Pesaran, M. H. (2000b). A decision theoretic approach to forecast evaluation. Statistics and finance: An interface (pp. 261-278). https://doi.org/10.1142/9781848160156_0015
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., & Chen, T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354–377. https://doi.org/10.1016/j.patcog.2017.10.013
Haberleitner, H., Meyr, H., & Taudes, A. (2010). Implementation of a demand planning system using advance order information. International Journal of Production Economics, 128(2), 518-526. https://doi.org/10.1016/j.ijpe.2010.07.003
Hanke,John E., Arthur G. (1995). Business forecasting. (5th ed.). Prentice Hall.
Herrera-Granda, I. D., Chicaiza-Ipiales, J. A., Herrera-Granda, E. P., Lorente-Leyva, L. L., Caraguay-Procel, J. A., García-Santillán, I. D., & Peluffo-Ordóñez, D. H. (2019). In: Rojas, I., Joya, G., Catala, A. (Eds.) Advances in Computational Intelligence. (pp. 362-373). Springer Cham. https://doi.org/10.1007/978-3-030-20518-8_31
Hodson, T. O. (2022). Root-mean-square error (RMSE) or mean absolute error (MAE): When to use them or not. Geoscientific Model Development, 15(14), 5481-5487. https://doi.org/10.5194/gmd-15-5481-2022
Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International journal of forecasting, 22(4), 679-688. https://doi.org/10.1016/j.ijforecast.2006.03.001
Jnr, E. O. N., Ziggah, Y. Y., & Relvas, S. (2021). Hybrid ensemble intelligent model based on wavelet transform, swarm intelligence and artificial neural network for electricity demand forecasting. Sustainable Cities and Society, 66, 102679. https://doi.org/10.1016/j.scs.2020.102679
Ke, J., Zheng, H., Yang, H., & Chen, X. M. (2017). Short-term forecasting of passenger demand under on-demand ride services: A spatio-temporal deep learning approach. Transportation research part C: Emerging technologies, 85, 591-608. https://doi.org/10.1016/j.trc.2017.10.016
Kilimci, Z. H., Akyuz, A. O., Uysal, M., Akyokus, S., Uysal, M. O., Atak Bulbul, B., & Ekmis, M. A. (2019). An improved demand forecasting model using deep learning approach and proposed decision integration strategy for supply chain. Complexity, 2019. https://doi.org/10.1155/2019/9067367
Kim, S., & Kim, H. (2016). A new metric of absolute percentage error for intermittent demand forecasts. International Journal of Forecasting, 32(3), 669-679. https://doi.org/10.1016/j.ijforecast.2015.12.003
Koutsandreas, D., Spiliotis, E., Petropoulos, F., & Assimakopoulos, V. (2022). On the selection of forecasting accuracy measures. Journal of the Operational Research Society, 73(5), 937-954. https://doi.org/10.1080/01605682.2021.1892464
Liang, Y. H. (2022). Forecasting International Tourism Demand Using the Recurrent Neural Network Model with Genetic Algorithms and ARIMAX Model in Tourism Supply Chains. International Journal of Machine Learning and Computing, 12(5). https://doi.org/10.1155/2022/3376296
Lewis, C. D. (1997). Demand forecasting and inventory control: A computer aided learning approach. Routledge.
Makridakis, S. (1993). Accuracy measures: theoretical and practical concerns. International Journal of Forecasting, 9(4), 527-529. https://doi.org/10.1016/0169-2070(93)90079-3
Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E. & Winkler, R. (1982). The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of forecasting, 1(2), 111-153. https://doi.org/10.1002/for.3980010202
Maragkos, N. (2020). Retail Demand Forecasting with Artificial Neural Networks [Doctoral dissertation, Aristotle University of Thessaloniki].
McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5, 115-133. https://doi.org/10.1007/BF02478259
Panapakidis, I. P., & Dagoumas, A. S. (2017). Day-ahead natural gas demand forecasting based on the combination of wavelet transform and ANFIS/genetic algorithm/neural network model. Energy, 118, 231-245. https://doi.org/10.1016/j.energy.2016.12.033
Petrucci, A., Barone, G., Buonomano, A., & Athienitis, A. (2022). Modelling of a multi-stage energy management control routine for energy demand forecasting, flexibility, and optimization of smart communities using a Recurrent Neural Network. Energy Conversion and Management, 268, 115995. https://doi.org/10.1016/j.enconman.2022.115995
Porteiro, R., Hernández-Callejo, L., & Nesmachnow, S. (2022). Electricity demand forecasting in industrial and residential facilities using ensemble machine learning. Revista Facultad de Ingeniería Universidad de Antioquia, (102), 9-25. https://doi.org/10.17533/udea.redin.20200584
Qi, J., Du, J., Siniscalchi, S. M., Ma, X., & Lee, C. H. (2020). On mean absolute error for deep neural network based vector-to-vector regression. IEEE Signal Processing Letters, 27, 1485-1489. https://doi.org/10.1109/LSP.2020.3016837
Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386-408. https://doi.org/10.1037/h0042519
Ryu, S., Noh, J., & Kim, H. (2016). Deep neural network based demand side short term load forecasting. Energies, 10(1), 3. https://doi.org/10.3390/en10010003
Salehinejad, H., Sankar, S., Barfett, J., Colak, E., & Valaee, S. (2017). Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078. https://doi.org/10.48550/arXiv.1801.01078
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117. https://doi.org/10.1016/j.neunet.2014.09.003
Schneider, P., & Xhafa, F. (2022). Anomaly Detection and Complex Event Processing Over IoT Data Streams: With Application to EHealth and Patient Data Monitoring. Academic Press.
Slimani, I., El Farissi, I., & Achchab, S. (2015). Artificial neural networks for demand forecasting: Application using Moroccan supermarket data. 2015 15th International Conference on Intelligent Systems Design and Applications (ISDA) (pp. 266-271). IEEE. https://doi.org/10.1109/ISDA.2015.7489236
Suganthi, L., & Samuel, A. A. (2012). Energy models for demand forecasting – A review. Renewable and sustainable energy reviews, 16(2), 1223-1240. https://doi.org/10.1016/j.rser.2011.08.014
Swanson, N.R., Ghysels, E. & Callan, M. (1999). A Multivariate Time Series Analysisof the Data Revision Process for Industrial Production and the Composite Leading Indicator. In R.F. Engle & H. White (Eds.), Cointegration, Causality and Forecasting: A Festschrift in Honor of Clive W.J. Granger. (pp.45-75). Oxford University Press.
Thonemann, U. (2010). Operations Management, 2., aktualisierte Auflage, Pearson.
Wanchoo, K. (2019, March). Retail demand forecasting: a comparison between deep neural network and gradient boosting method for univariate time series. 2019 IEEE 5th International Conference for Convergence in Technology (I2CT) (pp. 1-5). IEEE. https://doi.org/10.1109/I2CT45611.2019.9033651
Wang, B., Liu, K., & Zhao, J. (2016, August). Inner attention based recurrent neural networks for answer selection. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1288-1297). https://doi.org/10.18653/v1/p16- 1122
Wang, Z., & Bovik, A. C. (2009). Mean squared error: Love it or leave it? A new look at signal fidelity measures. IEEE signal processing magazine, 26(1), 98-117. https://doi.org/10.1109/MSP.2008.930649
Willmott, C. J., & Matsuura, K. (2005). Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance. Climate research, 30, 79-82. https://doi.org/10.3354/cr030079
Willmott, C. J., Matsuura, K., & Robeson, S. M. (2009). Ambiguities inherent in sums-of- squares-based error statistics. Atmospheric Environment, 43(3), 749-752. https://doi.org/10.1016/j.atmosenv.2008.10.005
Yu, D., Li, Z., Zhong, Q., Ai, Y., & Chen, W. (2020). Demand management of station-based car sharing system based on deep learning forecasting. Journal of Advanced Transportation, 2020, 1-15. https://doi.org/10.1155/2020/8935857
Zhang, B., Li, N., Shi, F., & Law, R. (2020). A deep learning approach for daily tourist flow forecasting with consumer search data. Asia Pacific Journal of Tourism Research, 25(3), 323- 339. https://doi.org/10.1080/10941665.2019.1709876
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Steffen Robus, Virág Walter, Zsolt Kőmüves
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
