Abstract

Grey-box and black-box dynamic models are routinely used to model the behavior of real-world dynamic systems. When creating such models, the identification of an accurate model structure (often referred to as covariate selection, feature selection, or sparsity identification) is a critical step required to achieve suitable predictive performance by minimizing the effects of overfitting. Recently, causation entropy has been shown to be quite useful in data-driven covariate selection as it provides a mechanism to measure the causal relationships between the set of covariates and the state dynamics. This work extends previous results by applying the causation entropy covariate selection technique to data from an experimental nonlinear system consisting of a ball rolling on an actuated inclined ramp. Data collected from the system is processed by the causation entropy-based algorithm and covariate selection is performed on a black-box dynamic model. The resulting optimized model is shown to provide better predictive performance than an optimized black-box model which includes extraneous covariates. This study represents the first application of causation entropy-based covariate selection to real-world experimental data, illustrating its use as a practical system identification method.

References

1.
Juang
,
J.-N.
,
Phan
,
M.
,
Horta
,
L. G.
, and
Longman
,
R. W.
, March-April
1993
, “
Identification of Observer/Kalman Filter Markov Parameters: Theory and Experiments
,”
J. Guid., Controls, Dyn.
,
16
(
2
), pp.
320
329
.10.2514/3.21006
2.
Ljung
,
L.
,
1998
, “
System Identification
,”
Signal Analysis and Prediction. Applied and Numerical Harmonic Analysis
,
A.
Prochazka
,
J.
Uhlir
,
P.
Rayner
, and
N.
Kingsbury
, eds.,
Birkhauser
,
Boston, MA
, pp.
163
173
.
3.
Keesman
,
K. J.
,
2011
,
System Identification
,
Springer-Verlag
,
London
.
4.
Juang
,
J.-N.
,
1994
,
Applied System Identification
,
Prentice Hall
,
Upper Saddle River, NJ
.
5.
Clarke
,
T.
, and
Sun
,
X.-D.
,
1998
, “
Minimal State-Space Model Realisation of a Nonlinear Helicopter
,”
IEE Proc.-Control Theory Appl.
,
145
(
4
), pp.
415
422
.10.1049/ip-cta:19982111
6.
Juang
,
J.-N.
, and
Pappa
,
R. S.
,
1985
, “
An Eigensystem Realization Algorithm for Modal Parameter Identification and Model Reduction
,”
J. Guid., Control, Dyn.
,
8
(
5
), pp.
620
627
.10.2514/3.20031
7.
Johnson
,
E. N.
, and
Kannan
,
S. K.
,
2005
, “
Adaptive Trajectory Control for Autonomous Helicopters
,”
J. Guid., Control, Dyn.
,
28
(
3
), pp.
524
538
.10.2514/1.6271
8.
Kabaila
,
P.
,
1983
, “
On Output-Error Methods for System Identification
,”
IEEE Trans. Autom. Control
,
28
(
1
), pp.
12
23
.10.1109/TAC.1983.1103141
9.
Iliff
,
K.
,
1989
, “
Parameter Estimation for Flight Vehicles
,”
J. Guid., Control, Dyn.
,
12
(
5
), pp.
609
622
.10.2514/3.20454
10.
Taylor
,
B.
, and
Rogers
,
J.
,
2014
, “
Experimental Investigation of Real-Time Helicopter Weight Estimation
,”
J. Aircr.
,
51
(
3
), pp.
1047
1051
.10.2514/1.C032449
11.
Nelles
,
O.
,
2013
,
Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models
,
Springer Science and Business Media
, Heidelberg, Germany.
12.
Babyak
,
M. A.
, June
2004
, “
What You See May Not Be What You Get: A Brief, Nontechnical Introduction to Overfitting in Regression Type Models
,”
Psychosomatic Med.
, (
3
), pp.
411
421
.10.1097/01.psy.0000127692.23278.a9
13.
Zou
,
H.
,
Hastie
,
T.
, and
Tibshirani
,
R.
,
2007
, “
On the Degrees of Freedom of the Lasso
,”
Ann. Stat.
,
35
(
5
), pp.
2173
2192
.10.1214/009053607000000127
14.
Tibshirani
,
R.
,
1996
, “
Regression Shrinkage and Selection Via the Lasso
,”
J. R. Stat. Soc. Ser. B
,
58
(
1
), pp.
267
288
.10.1111/j.2517-6161.1996.tb02080.x
15.
Zou
,
H.
, and
Hastie
,
T.
,
2005
, “
Regularization and Variable Selection Via the Elastic Net
,”
J. R. Stat. Soc. Stat. Methodol. Ser. B
,
67
(
2
), pp.
301
320
.10.1111/j.1467-9868.2005.00503.x
16.
Maldonado
,
S.
, and
Weber
,
R.
,
2009
, “
A Wrapper Method for Feature Selection Using Support Vector Machines
,”
Inf. Sci.
,
179
(
13
), pp.
2208
2217
.10.1016/j.ins.2009.02.014
17.
Bertsimas
,
D.
,
King
,
A.
, and
Mazumder
,
R.
,
2016
, “
Best Subset Selection Via a Modern Optimization Lens
,”
Annals Stat.
,
44
(
2
), pp.
813
852
.10.1214/15-AOS1388
18.
Kim
,
P.
,
Rogers
,
J.
,
Sun
,
J.
, and
Bollt
,
E.
,
2017
, “
Causation Entropy Identifies Sparsity Structure for Parameter Estimation of Dynamic Systems
,”
ASME J. Comput. Nonlinear Dyn.
,
12
(
1
), p. 011008.10.1115/1.4034126
19.
Sun
,
J.
, and
Bollt
,
E.
,
2014
, “
Causation Entropy Identifies Indirect Influences, Dominance of Neighbors and Anticipatory Couplings
,”
Phys. D
,
267
, pp.
49
57
.10.1016/j.physd.2013.07.001
20.
AlMomani
,
A. A. R.
,
Sun
,
J.
, and
Bollt
,
E.
,
2020
, “
How Entropic Regression Beats the Outliers Problem in Nonlinear System Identification
,”
Chaos
,
30
(
1
), p.
013107
.10.1063/1.5133386
21.
Elinger
,
J.
, and
Rogers
,
J.
,
2018
, “
Information Theoretic Causality Measures for System Identification of Mechanical Systems
,”
ASME J. Comput. Nonlinear Dyn.
,
13
(
7
), p.
071005
.10.1115/1.4040253
22.
Elinger
,
J.
, and
Rogers
,
J.
,
2021
, “
Practical Considerations for Use of Causation Entropy in Sparsity Identification
,”
ASME J. Comput. Nonlinear Dyn.
,
16
(
5
), p.
051006
.10.1115/1.4050615
23.
Elinger
,
J.
, and
Rogers
,
J.
,
2019
, “
Influence of Noise on Information Theoretic Causality Measures for System Identification
,”
American Control Conference
, Philadelphia, PA, July 10–12.10.23919/ACC.2019.8814656
24.
Cover
,
T.
, and
Thomas
,
J.
,
1991
,
Elements of Information Theory
,
Wiley
,
New York
.
25.
Elinger
,
J.
, and
Rogers
,
J.
,
2017
, “
Information Theoretic Tools for Parameter Estimation and Model Order Reduction for Mechanical Systems
,”
ASME
Paper No. IMECE2017-70744.10.1115/IMECE2017-70744
26.
Moon
,
Y.-I.
,
Rajagopalan
,
B.
, and
Lall
,
U.
,
1995
, “
Estimation of Mutual Information Using Kernel Density Estimators
,”
Phys. Rev. E
,
52
(
3
), pp.
2318
2321
.10.1103/PhysRevE.52.2318
27.
Beirlant
,
J.
,
Dudewicz
,
E.
,
Gyor
,
L.
, and
Meulen
,
E.
,
1997
, “
Nonparametric Entropy Estimation: An Overview
,”
Int. J. Math. Stat. Sci.
,
6
(
1
), pp.
17
39
.
28.
Archer
,
E.
,
Park
,
I. M.
, and
Pillow
,
J. W.
,
2014
, “
Bayesian Entropy Estimation for Countable Discrete Distributions
,”
J. Mach. Learn. Res.
,
15
(
1
), pp.
2833
2868
.
29.
Sun
,
J.
,
Cafaro
,
C.
, and
Bollt
,
E.
, June
2014
, “
Identifying the Coupling Structure in Complex Systems Through the Optimal Causation Entropy Principle
,”
Entropy
,
16
(
6
), pp.
3416
3433
.10.3390/e16063416
30.
Smith
,
S.
,
1997
,
The Scientist and Engineer's Guide to Digital Signal Processing
,
California Technical Publishing
, San Diego, CA.
31.
Press
,
W. H.
,
Vetterling
,
W. T.
,
Flannery
,
P. P.
, and
Teukolsky
,
S.
,
1992
,
Numerical Recipes in C: The Art of Scientific Computing
,
Cambridge University Press
, Cambridge, UK.
32.
LeVeque
,
R. J.
,
2007
,
Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems
,
SIAM
, Philadelphia, PA.
33.
More
,
J.
,
1977
,
The Levenberg-Marquardt Algorithm: Implementation And Theory,”
Conference on Numerical Analysis, Dundee, Scotland, June 28–July 1, Paper No. 770636-1
.
34.
Martí
,
R.
,
2003
, “
Multi-Start Methods
,”
Handbook of Metaheuristics
,
Springer
, Berlin, pp.
355
368
.
35.
Ugray
,
Z.
,
Lasdon
,
L.
,
Plummer
,
J.
,
Glover
,
F.
,
Kelly
,
J.
, and
Martí
,
R.
,
2007
, “
Scatter Search and Local Nlp Solvers: A Multistart Framework for Global Optimization
,”
Informs J. Comput.
,
19
(
3
), pp.
328
340
.10.1287/ijoc.1060.0175
You do not currently have access to this content.