ABSTRACT
Title of dissertation: A THEORY OF CRAM?ER-RAOBOUNDS FOR CONSTRAINED
PARAMETRIC MODELS
Terrence Joseph Moore, Jr.,
Doctor of Philosophy, 2010
Dissertation directed by: Professor Benjamin Kedem
Department of Mathematics
A simple expression for the Cram?er-Rao bound (CRB) is presented for the
scenario of estimating parameters ? that are required to satisfy a differentiable
constraint function f(?). A proof of this constrained CRB (CCRB) is provided
using the implicit function theorem, and the encompassing theory of the CCRB is
proven in a similar manner. This theory includes connecting the CCRB to notions of
identifiability of constrained parameters; the linear model under a linear constraint;
the constrained maximum likelihood problem, it?s asymptotic properties and the
method of scoring with constraints; and hypothesis testing. The value of the tools
developed in this theory are then presented in the communications context for the
convolutive mixture model and the calibrated array model.
A THEORY OF CRAM?ER-RAO BOUNDS FOR CONSTRAINED
PARAMETRIC MODELS
by
Terrence Joseph Moore, Jr.
Dissertation submitted to the Faculty of the Graduate School of the
University of Maryland, College Park in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
2010
Advisory Committee:
Professor Benjamin Kedem, Chair/Advisor
Professor Radu Balan
Professor Paul Smith
Professor Doron Levy
Professor Prakash Narayan
c?Copyright by
Terrence Joseph Moore, Jr.
2010
Acknowledgments
There are perhaps too many people to thank here without the regret of omit-
ting someone important to me. So many have helped me become a mathematician.
To the very talented high school math teachers at West Hills High School in Santee,
CA, who encouraged the best students to strive further, to the many remarkable
professors at American University who delighted in interacting with their students
and no matter how difficult the material ensured the experience was always enjoy-
able, and to the excellent class of professors at the University of Maryland who
challenged and pushed their students to excel. To all these people, I thank you.
I would like to acknowledge my undergraduate and Master?s thesis advisor,
Dr. Stephen Casey at American University, for his direction in the beginning of my
college education. I do need to acknowledge the contribution of my first advisor,
Dr. Dennis Healy, who encouraged the construction of this current work from the
beginning. He allowed me the freedom to develop the theory using an approach I
developed and generated a positive exuberance for whatever results I discovered. I
only wish he could have survived to see its completion. I also need to acknowledge
my second advisor, Dr. BenjaminKedem, for hiswillingness to allow me to complete
this document as it was originally conceived as well as for guiding me through the
defense process and the selection of the committee.
I also need to acknowledge the financial support I have received from my
employer, the U.S. Army Research Lab. Additionally, the professional support and
mentoring I received from Dr. Brian Sadler and the advice from team members at
ii
the lab has been invaluable in assisting me succeed through the process. I also need
to thank Dr. John Gowens, Mr. Glenn Racine, Dr. Barbara Broome, Dr. Alex
Kott, and Dr. Brian Rivera for allowing me the time at and away from work to
complete this degree.
iii
Table of Contents
List of Figures vii
List of Symbols and Abbreviations viii
1 INTRODUCTION 1
1.1 A note on the notation . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 THE CRAM?ER-RAO BOUND 5
2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Identifiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.1 Local identifiability . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.2 Strong Identifiability . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Best Linear Unbiased Estimators . . . . . . . . . . . . . . . . 12
2.3.2 Gaussian noise . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4 Maximum likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4.1 Efficient estimation . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.2 Asymptotic Normality . . . . . . . . . . . . . . . . . . . . . . 14
2.4.3 Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5.1 The Rao statistic . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5.2 The Wald statistic . . . . . . . . . . . . . . . . . . . . . . . . 17
2.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 THE CONSTRAINED CRAM?ER-RAO BOUND 19
3.1 The Constrained CRB . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.1.1 A proof of the CCRB . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.2 Alternative formulas . . . . . . . . . . . . . . . . . . . . . . . 31
3.1.2.1 Gorman-Hero-Aitchison-Silvey CCRB . . . . . . . . 32
3.1.2.2 Aitchison-Silvey-Crowder CCRB . . . . . . . . . . . 35
3.1.2.3 Xavier?s & Barroso?s Intrinsic Variance Lower Bound 36
3.1.3 Simple Extensions to the CCRB . . . . . . . . . . . . . . . . . 38
3.1.4 Properties of the CCRB . . . . . . . . . . . . . . . . . . . . . 41
3.1.5 Derivation of g(?) . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.1.5.1 A Taylor series derivation . . . . . . . . . . . . . . . 50
3.1.5.2 A fixed point derivation . . . . . . . . . . . . . . . . 51
3.2 Identifiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2.1 Local identifiability . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2.1.1 Local identifiability in the Aitchison-Silvey-Crowder
CCRB formula . . . . . . . . . . . . . . . . . . . . . 57
3.2.2 Strong Identifiability . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 Linear Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
iv
3.3.1 Best Linear Unbiased Estimation . . . . . . . . . . . . . . . . 60
3.3.2 Uniform Minimum Variance Estimation under Gaussian noise 63
3.4 Constrained Maximum Likelihood Estimation . . . . . . . . . . . . . 66
3.4.1 Efficient estimation . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4.2 Asymptotic Normality . . . . . . . . . . . . . . . . . . . . . . 69
3.4.3 The Method of Scoring Under Parametric Constraints . . . . . 71
3.4.3.1 Convergence Properties . . . . . . . . . . . . . . . . 75
3.4.3.2 Linear constraints . . . . . . . . . . . . . . . . . . . 78
3.5 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.5.1 The Rao statistic . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.5.2 The Wald statistic . . . . . . . . . . . . . . . . . . . . . . . . 83
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4 Applications of the CCRB in Communications Models 86
4.1 Convolutive Mixture Model . . . . . . . . . . . . . . . . . . . . . . . 88
4.1.1 Equivalent Convolutive Mixture Models . . . . . . . . . . . . 90
4.1.1.1 The Vector-Matrix Model . . . . . . . . . . . . . . . 90
4.1.1.2 The Z transform model . . . . . . . . . . . . . . . . 91
4.1.2 Strict Identifiability . . . . . . . . . . . . . . . . . . . . . . . . 95
4.1.3 The Fisher information of the convolutive mixture model . . . 98
4.1.3.1 Complex-valued Fisher information . . . . . . . . . . 98
4.1.3.2 CFIM for the convolutive mixture model . . . . . . . 99
4.1.3.3 Properties of the CFIM . . . . . . . . . . . . . . . . 100
4.1.4 Constraints for the convolutive mixture model . . . . . . . . . 108
4.1.4.1 Pathways to regularity . . . . . . . . . . . . . . . . . 109
4.1.4.2 Norm channel + real-valued source constraint . . . . 110
4.1.4.3 Semiblind constraints: s(k)(t) =p(t) for t?T . . . . 112
4.1.4.4 Unit Modulus constraint + Semiblind constraint . . 113
4.2 Calibrated Array Model . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.2.1 The Fisher information of the calibrated array model . . . . . 118
4.2.1.1 Indirect derivation of the FIM . . . . . . . . . . . . . 118
4.2.1.2 Direct derivation of the FIM . . . . . . . . . . . . . . 120
4.2.1.3 Properties of the FIM . . . . . . . . . . . . . . . . . 121
4.2.2 Constraints for the calibrated array model . . . . . . . . . . . 122
4.2.2.1 Constraints onthe complex-valuedchannelgain: ? =
IK?K . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.2.2.2 Semiblind constraints: s(t) =p(t) for t?T . . . . . 124
4.2.2.3 Finite alphabet constraint: s(k)(n)?S . . . . . . . . 125
4.2.2.4 Unit modulus constraints: |s(n)|= 1 for all n . . . . 127
4.2.2.5 Unit modulus constraint; real-valued channel gain:
Im(?k) = 0 for all k . . . . . . . . . . . . . . . . . . . 129
4.2.2.6 Semi-blind and unit modulus constraint . . . . . . . 129
4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
v
A Appendices 136
A.1 A proof of the CCRB using the Chapman-Robbins version of the
Barankin bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
A.2 A proof of the CCRB using the method of implicit differentiation . . 138
A.3 Alternative proof of asymptotic normality . . . . . . . . . . . . . . . 140
B Proofs of Convergence Properties of Constrained Scoring 142
C Proofs of Theorems in Chapter 4 147
Bibliography 161
vi
List of Figures
3.1 Reparameterization of f(?) = 0 to ? =g(?). . . . . . . . . . . . . . . 26
3.2 Projection of the observations x onto the linear space H? and the
linear constraint space H?f. . . . . . . . . . . . . . . . . . . . . . . 61
3.3 Path created by iterates from the method of scoring with constraints. 74
4.1 Convolutive Mixture: Finite Impulse Response (tapped delay line)
model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2 Convolutive Mixture: Norm-constrained channel estimation perfor-
mance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.3 Convolutive Mixture: Example of multipath channel. . . . . . . . . . 115
4.4 Convolutive Mixture: Source estimation with varying ??. . . . . . . 116
4.5 Calibrated array model geometries. . . . . . . . . . . . . . . . . . . . 118
4.6 Calibrated Array: CCRBs on AOA for blind, constant modulus, and
known signal models. . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.7 Calibrated Array: CCRBs on AOA for semiblind, constant modulus
+ semiblind, and known signal models. . . . . . . . . . . . . . . . . . 132
4.8 Calibrated Array: CCRBs on signal phase for blind, constant modu-
lus, semiblind, and constant modulus + semiblind. . . . . . . . . . . . 134
vii
List of Symbols and Abbreviations
Symbols
b(?) the bias of an estimator of ?
C the complex numbers
C? Cuniontext{?}
CN(a,B) complex normal with mean a and (co)variance matrix B
?2r chi-square distribution with r degrees of freedom
d? convergence in distribution
E?(?) the expectation of (?) with respect to the density modeling ?
f(?) = 0 the constraint function
F(?) the Jacobian (or gradient) of the constraint function at ?
I(?) the Fisher information for the parameter ?
Im?m or Im the identity matrix of size m
Im(?) the imaginary part of (?)
s(x;?) the Fisher score
N(a,B) normal with mean a and variance (covariance) matrix B
p(x;?) the probability density function (pdf)
R the real numbers
Re(?) the real part of (?)
t(x) an estimator of some parameter (vector)
? the parameter
? the parameter space
?f the constraint manifold (parameter space)
??ML(x) the ML estimator of ? depending on the observation x
x the observation (sample)
(?)+ (?) if the value is nonnegative, otherwise zero
Abbreviations
BER bit-error rate
BLUE best linear unbiased estimate (or estimator)
CCRB constrained Cram?er-Rao bound
CFIM complex FIM
CLSE constrained least squares estimate (or estimator)
CML constrained maximum likelihood
CMLE constrained maximum likelihood estimate (or estimator)
CRB Cram?er-Rao bound
FIM Fisher information matrix
FIR finite impulse response
iid independent and identically distributed
LSE least squares estimate (or estimator)
MIMO multiple-input, multiple-output
ML maximum likelihood
MLE maximum likelihood estimate (or estimator)
MSE mean-square error
MVUE minimum variance unbiased estimate (or estimator)
viii
NLB nullity lower bound of the FIM (or CFIM)
SER symbol-error rate
SIMO single-input, multiple-output
SISO single-input, single-output
ix
Chapter 1
INTRODUCTION
The Cram?er-Rao bound (CRB), the inverse of the Fisher information, is a
limit on the performance of the estimation of parameters under certain conditions.
Hence, the variance of any unbiased estimator cannot be lower than this bound,
and the CRB may also be interpreted in some sense as a measure of performance
potential. For a large number of scenarios, it is often of interest to gauge the
performance of estimation under parametric constraints. The traditional approach
in deriving CRBs for these cases is to find a reparameterization that represents
the constraint. However, such an approach is not always feasible for a large class
of constraints and numerical approximations are often restricted to the particular
model. An alternative and equivalent option to reparameterization, the constrained
Cram?er-Rao bound, is presented herein, which is also computationally simple to
program.
In communications design research, estimation performance metrics are often
interpreted to represent performance potential to study the feasibility of a model to
meet a certain measure of desired reliability. This approach is often practical since
it avoids the necessity of searching for the best performer over a class of estimators
for a particular trial model and since the CRB is analytically or numerically simple
to compute. The downside to this approach is that for each model, the CRB needs
1
to be derived each time. This effectively prohibits the practitioner from studying
an overly large class of models or, in the context of this work, a large class of
constraints on some base model. This restriction is to a great extent eliminated
using the constrained Cram?er-Rao bound, as the Fisher information, which involves
an integration over many variables, for the base model only needs to be evaluated
once.
Chapter 2 offers a quick review of several connections with the Cram?er-Rao
bound (CRB) within mathematical statistics and serves as a reference point for the
study of parameters under parametric equality constraints, discussed in Chapter 3.
With the possible exception of the identifiability relationship in section 2.2, much
of this section is familiar and well represented in standard mathematical statistics
texts.
In Chapter 3, a general theory of the constrained Cram?er-Rao bound (CCRB)
is presented. In section 3.1, the CCRB is defined and proven, alternative formu-
las are presented, and several interesting properties of the bound are detailed. In
section 3.2, a connection is made between the CCRB and two different notions of
identifiability under certain conditions. In section 3.3, the linear model with linear
constraints is examined in the context of the CCRB. In section 3.4, connections
between the CCRB and constrained maximum likelihood estimation are detailed,
includingan asymptotic normality result and an adaptation of the method of scoring
to the constrained parameter scenario. This chapter concludes with the consider-
ation of hypothesis testing under constraints in section 3.5. Chapters 2 and 3 are
designed so that the section numbers correspond directly, i.e., section 3.x relates
2
a concept for constrained parameters that section 2.x reviews for unconstrained
parameters.
In Chapter 4, the analytic tools developed in Chapter 3 are applied in the
communications context of the convolutive mixture model (section 4.1) and the
calibrated array model (section 4.2). These models are defined and their Fisher
information matrices developed in section 4.1.3 and section 4.2.1, respectively. A
variety of constraints for these models are considered in sections 4.1.4 and 4.2.2.
1.1 A note on the notation
All elements will be denoted in lowercase math font: a. All vectors will be
column vectors and be denoted in a lowercase bold math font: a. Hence, the ith
element of the column vectora will be denoted as ai. (This should not be confused
with ai, which is often used as a subvector of the vector a and will be defined
in context.) All matrices will be denoted in an uppercase bold math font: A.
All scalars, vectors, and matrices are assumed to have elements with real-valued
numbers unless otherwise noted as complex-valued (where the complex number i=
j =??1 should be clear from context).
For vectors and matrices, (?)T will denote the transpose operator (do not with
confuse (?)prime, whichisoccasionally usedhere asa dummyvariable), (?)? willdenote the
conjugate operator (do not confuse with (?)star, which is occasionally used as a variant
of another vector or matrix), and (?)H will denote the Hermitian (or conjugate
transpose) operator. When avector dependson another vectorvalue as ina(?), then
3
the Jacobian will be a matrix denoted as A(?) where the ith row is the transposed
of the vector ???ai(?) and ai(?) is the ith row element of a(?). For square matrices,
(?)?1 will denote the inverse of the matrix and (?)? will denote the pseudoinverse of
the matrix. Of course, (?)2 will denote the square of the matrix.
For symmetricmatrices(or Hermitianmatricesinthe complex-valuedcase, the
expressionA>B will denote that the matrixA?B is positive definite. Similarly,
A?B will denote that the matrix A?B is positive semidefinite.
Sets will be denoted in an uppercase blackboard math font: A or in an up-
percase Greek letter math font: ?. Also, all sets will be assumed open sets unless
otherwise noted.
For convenience to allow the reader to find referenced items, numbered theo-
rems, corollaries, and examples share numbering. Thus, the theorem immediately
following Example x.4 in Chapter x is numbered Theorem x.5 even though the
previous theorem is Theorem x.1.
4
Chapter 2
THE CRAM?ER-RAO BOUND
The Cram?er-Rao bound (CRB) is a lower bound on the error covariance of
any unbiased estimator under certain regularity conditions. As such, it is a measure
of the optimal performance of an unbiased estimator under a given model. There
are other mean-square error lower bounds, such as the Ziv-Zakai bound [78], the
Hammersley-Chapman Robbins bound [26, 16], the Barankin bound [8], the Bhat-
tacharyya bound [11], etc., and indeed, depending on the model, there are numerous
other possible performance measures, such as classification bounds like bit-error rate
(BER) or symbol-error rate (SER), but the CRB remains a very popular benchmark
due to it?s computational simplicity and its underlying well-developed theory.
This theory has led to numerous connections in areas of mathematical statis-
tics, e.g., identifiability, linear models, maximum likelihood, including asymptotic
normality and the method of scoring, and hypothesis testing. This short chapter is
a quick review of just a few of these connections. A more complete discussion of the
utility and application of the topics discussed in this chapter, as well as proofs of the
definitions, theorems and statements herein, may be found in many standard sta-
tistical inference texts, such as Shao [62] or Casella and Berger [14] for statisticians,
Kay [36] for signal processors, or Van Trees [72] for engineers.
5
2.1 Definition
Suppose we have an observation x in X ? Rn from a probability density
function (pdf) p(x;?) where? is in an open set ??Rm is a vector of deterministic
parameters. The Fisher information matrix (FIM) of this model is given by
I(?) ?= E?braceleftbigs(x;?)sT(x;?)bracerightbig
where s(x;?) is the Fisher score defined by
s(x;?) ?= ?logp(x;?
prime)
??prime
vextendsinglevextendsingle
vextendsinglevextendsingle
?prime=?
and the expectation is evaluated at ?, i.e., E?(?) =
integraldisplay
X
(?)p(x;?)dx. And suppose as
regularity conditions, the pdf is differentiable with respect to ? and satisfies
?
??TE?(h(x)) =E?(h(x)s
T(x;?)) (2.1)
for h(x)?1 and h(x)?t(x) where t(x) is an unbiased estimator of ? [62]. These
conditions are assured under a number of scenarios, for example, when the Jacobian
and Hessian of the density function p(x;?) is absolutely integrable with respect to
bothxand? [72], and essentially permit switching between the order of integration
and differentiation. Under these assumptions, we have the following information
inequality theorem [14, 62, 36, 72], independently developed by Cram?er [17] and
Rao [56].
Theorem 2.1. The Cram?er-Rao bound is the inverse of the FIM,
CRB(?) ?=I?1(?), (2.2)
6
if it exists, and the variance of any unbiased estimator t(x) satisfies the inequality
Var(t(x))?CRB(?)
with equality if and only if t(x)?? = CRB(?)s(x;?) in the mean-square sense.
Example 2.2. Let x ? CN(?,?2) with unknown complex-valued mean ? and
known variance ?2. In terms of real-valued parameters, the equivalent model is
bracketleftbiggRe(x)
Im(x)
bracketrightbigg
?N(
bracketleftbiggRe(?)
Im(?)
bracketrightbigg
,?
2
2 I2?2).
From a well-known result on normal distributions [36, equation (3.31)],
I(
bracketleftbiggRe(?)
Im(?)
bracketrightbigg
) = 2?2I2?2
and the CRB is ?22 I2?2.
2.1.1 Extensions
The performance of a function of parameters, e.g. the transformation ? =
k(?), is often of more interest than the performance of the parameters. If the
Jacobian of the transformation function is K(?) = ?k(?
prime)
??primeT
vextendsinglevextendsingle
vextendsingle
?prime=?
, then the CRB on
the performance of an unbiased estimator of ? is [62, 36]
CRB(?) =K(?)I?1(?)KT(?),
i.e., if S(x) is an unbiased estimator of ?, then Var(S(x))?K(?)CRB(?)KT(?).
Implicit in this inequality for the transformation is that ? is differentiable with
respect to ? and (2.1) must also be satisfied for h(x) ? S(x). Consequently, if
7
an estimator t(x) is biased with bias b(?) = E?t(x)??, then the transformation
formula above can be used to attain a bound for ?= ?+b(?). Then Var(t(x))?
CRB(?+b(?)) where
CRB(?+b(?)) = (Im +B(?))CRB(?)parenleftbigIm +BT(?)parenrightbig
with B(?) = ?b(?
prime)
??primeT
vextendsinglevextendsingle
vextendsingle
?prime=?
.
Theorem 2.1 requires a nonsingular Fisher information, however, there are a
number of interesting cases where this requirement can not be met yet the model is
still of interest. For this scenario, the pseudoinverse of the FIM is occasionally used
as a bound in place of the CRB, i.e.,
Var(t(x))?I?(?)
for an unbiased estimator t(x). This bound inequality is trivial for nonidentifi-
able functions of the parameters [66], i.e., the variance is finite only if H(?) =
H(?)I(?)I?(?) where H(?) = K(?) + B(?) and t(x) is a biased estimator of
?=k(?) with bias b(?).
2.2 Identifiability
The ability to identify parameters determines the validity and utility of cer-
tain structural models. Criteria on the identifiability of parameters has numerous
connections to parametric statistical measures, such as Kullback-Leibler distance
[13] and the Fisher information matrix [58, 29, 69]. In this section, two of these con-
nections are developed to establish conditions under which a particular parametric
model is identifiable.
8
2.2.1 Local identifiability
To proceed in examiningthe identifiabilitycriterionfrom the CRB, a definition
of identifiability is required. A parameter ????Rm is identifiable in the model
p(x;?) if there is no other other ?prime ?? such that p(x;?) = p(x;?prime) for all x?Rn.
A parameter is locally identifiable if there exists an open neighborhood of ? such
that ? is identifiable in that neighborhood.
A parameter is (locally) identifiable in the additive noiseless case if the param-
eter is solvable (locally). Estimable parameters, i.e., expected values of functions
of the observations [57], are also identifiable.1 Hence, a non-identifiable parameter
is not estimable regardless of the scheme or the number of observations. These
scenarios exist when some inherent ambiguity exists in the model.
Example 2.3. The parameter vector ? = [?1,?2]T in the model
x= ?1 +?2 +w,
where w represents the observation noise, is not identifiable since it is indistinguish-
able from the parameter vector?prime = [?1 +a,?2?a]T for any real number a.
The Fisher information matrix is called regular if I(?) is full rank, and ? is
said to be a regular point of I(?). If the FIM is singular, ? is a singular point.
Example 2.4. The FIM for the model
x= ?2 +w,
1The converse is not true. While it is possible to develop estimation schemes for any identifiable
parameters, there is no guarantee that those estimators will be unbiased.
9
where w?N(0,1), is I(?) = 4?2. For any ?negationslash= 0, ? is locally identifiable but not
identifiable and at the same time a regular point of the FIM. For ? = 0, ? is a
singular point, yet is identifiable.
Fisher information-regularity implies local identifiability, but as the example
demonstrates the converse is not true. Rothenberg [58] found a connection between
local identifiability and the FIM under certain conditions.
Theorem 2.5 (Rothenberg). Assume the FIMI(?) has constant rank locally about
?. Then ? is locally identifiable if and only if I(?) is regular.
2.2.2 Strong Identifiability
Suppose that p(x;?) is a normal pdf with mean ?(?) ? Rp and variance
?(?), whose elements may be explicitly defined by a map ? : ? ? Rq where
q?p+p(p+ 1)/2 and it is assumed m?q. Then by the given definitions, (local)
identifiability holds when ? is injective (locally) and since by a transformation on
the FIM [12, p. 157]
I(?) = ??
T(?)
?? I(?(?))
??(?)
??T
regularity holds when the Jacobian ??(?)??T has full rank m.
Suppose there exists a set of indicesi1,...,im?{1,...,q}that make??(?) =
[?i1(?),...,?im(?)]T injective on ?. Then each ??? is strongly identifiable and
?? is a representative mapping. By this definition, if I(?) is regular at ? then ? is
in a strongly identifiable open neighborhood, and if ? is strongly identifiable on ?
then it is also identifiable on ?. The converses are not generally true. The following
10
theorem establishes conditions under which the converses are true [29].
Theorem 2.6 (Hochwald and Nehorai). Let?: ??Cq be a holomorphic mapping
of z??????A??, where ????Cm and ?? is open in Cm for each ?. Then
(a) ifI(z) is regular, there exists a strongly identifiable open neighborhood about
z, and
(b) if there exists a representative mapping ??? : ?? ? Cq for each ?, I(z) is
regular for everyz??.
Therefore, the existence of a proper holomorphic function(s) equates Fisher in-
formation regularity with strong identifiability for normal distributions. And locally
constant rank in the FIM equates regularity with local identifiability for arbitrary
distributions.
2.3 Linear Model
Suppose we have observations x from a linear model
x=H?+w, (2.3)
on ?, where H is an observation matrix consisting of known elements and w is the
noise from the observations with mean zero and variance C.
11
2.3.1 Best Linear Unbiased Estimators
The Gauss-Markov theorem [14] states that the best linear unbiased estimator
(BLUE) is given by the (weighted) least squares solution
??LS(x) =parenleftbigHTC?1Hparenrightbig?1HTC?1x, (2.4)
so calledfor minimizingthe (weighted)least squares (x?H?)T C?1 (x?H?). For
any other LUE of ?, i.e. Ax, then Var(Ax) ? Var(??LS(x)) with equality if and
only if A = parenleftbigHTC?1Hparenrightbig?1HTC?1. This assumes a full column rank observation
matrix H. Otherwise, for any estimable function of the parameters dT?, where d
is in the column space of the transposed observation matrix HT, its least squares
solution is dT ??LS(x) where
??LS(x) =parenleftbigHTC?1Hparenrightbig?HTC?1x, (2.5)
and (?)? is the generalized pseudoinverse of (?) [57, theorems 11.2B,11.3A-D]. This
solution is also the BLUE with variance dT parenleftbigHTC?1Hparenrightbig?d.
2.3.2 Gaussian noise
Additionally, if the noise has a Gaussian distribution, i.e.,w?N(0,C), then
the least squares solution is also the maximum likelihood estimator (MLE) and the
minimum variance unbiased estimate (MVUE).
Theorem 2.7. If the observations obey the linear model in (2.3), where H is a
known full column rank matrix, ? is an unknown parameter vector, and w is a
12
zero-mean normal random vector with variance C, then the MVUE is
??LS(x) =parenleftbigHTC?1Hparenrightbig?1HTC?1x (2.6)
with estimator covariance equaling the CRB I?1(?) =parenleftbigHTC?1Hparenrightbig?1.
Similarly, if H is not full rank, and dT? is an estimable function, then the
MLE is dT ??MLE(x) where ??MLE(x) = ??LS(x) from (2.5). This MLE is also the
MVUE [57, theorems 11.3F-G].
2.4 Maximum likelihood
Given observations x from a likelihood (or pdf) p(x;?) depending on an un-
known parameter ?, a popular method of estimating the parameter is the method
of maximum likelihood. This approach chooses as an estimator ??ML(x) that, if true,
would have the highest probability (the maximum likelihood) of resulting in the
given observations x, i.e., the optimization problem:
??ML(x) = argmax
?
logp(x;?)
where for convenience the log-likelihood is equivalently maximized since log(?) is
monotone. An analytic solution of the MLE can be found from the first-order
conditions on the log-likelihood by considering solutions ??(x) of
s(x;?prime) = 0. (2.7)
This is the method of maximum likelihood. Provided ? is an open set, ??ML(x) will
satisfy (2.7).
13
2.4.1 Efficient estimation
If an efficient estimator exists, it is well-known that the method of maximum-
likelihood finds the estimator [36, exercise 7.12], i.e., such an estimator must be a
stationary point of the maximum-likelihood optimization problem. More formerly,
we have the following theorem.
Theorem 2.8. Ift(x) is an estimator of?, which is also efficient with respect to the
CRB, then the estimator is a stationary point of the following optimization problem:
max
?
logp(x;?).
2.4.2 Asymptotic Normality
Let the samples x1,x2,...,xn be iid as x from the pdf p(x;?). Denote yn =
(x1,x2,...,xn) to be the collection of these samples, so that the likelihood will
be p(yn;?) =
nproductdisplay
i=1
p(xi;?), with the maximum likelihood of these samples denoted
??(yn).
Theorem 2.9. Assuming the regularity conditions stated earlier on the pdf p(x;?),
the MLE of the parameter ? is asymptotically distributed according to
?nparenleftBig??(y
n)??
parenrightBig d
?Nparenleftbig0,I?1(?)parenrightbig
where I(?) is derived from the pdf p(x;?), i.e., it is the Fisher information of a
single observation or sample.
14
2.4.3 Scoring
There exists a number of approaches to finding the maximum likelihood, in
some cases requiring iterative techniques. One such technique is Fisher?s method
of scoring. Given an observation x from a likelihood or pdf p(x;?) depending on
an unknown parameter ? and given some initial estimate ??(1) of ?, then iteratively
using the update
??(k+1) = ??(k) +I?1( ??(k))s(x; ??(k)) (2.8)
will find the MLE under certain conditions, e.g., provided the initial estimate is
sufficiently close in a locally convex region.
2.5 Hypothesis testing
Given the inclusion of the CRB quantity in the asymptotic normality results
in section 2.4.2, it is not surprising that there would also exist connections to some
asymptotic hypothesis tests. Assumeh: Rm ?Rr is a consistent and nonredundant
differentiable function, which defines the null hypothesis
H0 :h(?) = 0
in the likelihood(or model)p(yn;?) versusthe alternative hypothesisH1 :h(?)negationslash= 0.
2.5.1 The Rao statistic
The Rao (or score) test statistic is given by
?(yn) = 1nsT(yn; ??h(yn))I?1(??h(yn))s(yn; ??h(yn)) (2.9)
15
where the Fisher score at yn is s(yn, ??h(yn)) =
nsummationdisplay
i=1
s(xi, ??h(yn)) and ??h(yn) is the
MLE of ? under the null hypothesis h(?) = 0. A variant of this statistic, the
Lagrange multiplier test [3, 63]
??TnH(??h(yn))I?1(??h(yn))HT(??h(yn))??n,
was developed by Silvey [63]. The equivalence between the Rao and Lagrange mul-
tiplier test comes from the first order condition to satisfy the constraint [45], i.e.,
s(yn; ??h(yn)) +HT(??h(yn))??n = 0,
h(??h(yn)) = 0
where ??n?Rr is a vector of Lagrange multiplier estimates. First order Taylor-series
expansions of both equations about the true parameter ? produces
s(yn;?)?In(?)(??h(yn)??) +HT(??h(yn))??n = o(n?1/2)
h(?) +H(?)(??h(yn)??) +o(n?1/2) = h(??h(yn))
where In(?) = nI(?) (n times the Fisher information based on a single sample x).
Hence under the null hypothesis, the latter implies H(?)(??h(yn)??) = o(n?1/2),
and premultiplying the former by H(?)I?1n (?), then
H(?)I?1n (?)s(yn;?)+H(?)I?1n (?)HT(??h(yn))??n = o(n?1/2).
Since s(yn;?)?N(0,In(?)), then applying Slutsky?s theorem and the continuity
of the Fisher information and the hypothesis function,
H(??h(yn))I?1n (??h(yn))HT(??h(yn))??n d?Nparenleftbig0,H(?)I?1n (?)HT(?)parenrightbig.
16
Therefore,
parenleftBig
H(??h(yn))I?1n (??h(yn))HT(??h(yn))
parenrightBig?1/2 ?
?n isar-dimensional standard
normal variable in distribution, and
?(yn) d??2r.
Hypothesis H0 is rejected if ?(yn) > ?2r,?, where ?2r,? is the (1??)-quantile of the
chi-square distribution with r degrees of freedom.
2.5.2 The Wald statistic
The Wald test statistic is given by
?(yn) = nhT(??(yn))
parenleftBig
H(??(yn))I?1(??(yn))HT(??(yn))
parenrightBig?1
h(??(yn)) (2.10)
where H(?) = ?h(?
prime)
??primeT
vextendsinglevextendsingle
vextendsinglevextendsingle
?prime=?
and ??(yn) is an MLE of ?. (The nonredundancy of h
implies that H(?) is full row rank.) From section 2.1.1, the CRB of h(?) in the
model p(x;?) is H(?)I?1(?)HT(?) and therefore, using theorem 2.9,
?nparenleftBigh(??(y
n))?h(?)
parenrightBig d
?Nparenleftbig0,H(?)I?1(?)HT(?)parenrightbig.
Hence, under H0, using Slutsky?s theorem, the convergence in probability of the
MLE, and continuity of the FIM and Jacobian of the test function,
?(yn) d??2r.
Therefore, the hypothesis H0 is rejected if ?(yn) >?2r,?.
2.6 Discussion
In this section, the Cram?er-Rao bound (CRB) was defined and in theorem
2.1 it was stated to be a bound on mean-square error performance of an unbiased
17
estimator. The theory of the CRB?s connection to a variety of useful theorems
and equations in mathematical statistics was demonstrated. The CRB is included
in conditions for local identifiability (theorem 2.5) as well as conditions for strict
identifiability (theorem 2.6). The CRB is the projection matrix of the BLUE under
a Gaussian model in equation (2.6). There is a connection between the existence of
efficiency with respect to the CRB and the method of ML (theorem 2.8). The CRB
is also the asymptotic variance of the ML estimator (theorem 2.9) and appears in
the update formula in (2.8) for the method of scoring. The CRB also appears in the
formulas for the Rao test statistic in (2.9) and for the Wald test statistic (2.10).
These connections are not exhaustive, e.g., the CRB formula can also be useful
in defining confidence regions or in useful as a cost function, but these are perhaps
the most prevalent general topics in mathematical statistics theory and for that
reason serveas auseful comparison for the constrained Cram?er-Rao bound inchapter
3 and its connections in the theory.
18
Chapter 3
THE CONSTRAINED CRAM?ER-RAO BOUND
While the Cram?er-Rao bound (CRB) is a useful measure of parametric esti-
mation, it does not inherently measure the performance of estimators of parameters
that satisfy side information in the form of a functional equality constraint
f(?) = 0. (3.1)
The statistical literature is surprisingly somewhat limitedin addressing performance
measures under this general scenario. The traditional practice is to find some equiv-
alent reparameterization of the particular model and then find the CRB on the
parameters of interest using the reparameterized transformation. This approach,
however, does not lend itself to theoretical meaning beyond the particular reparam-
eterized model. Typically, works that do examine (3.1) in a general manner are
focused on developing methods for decisions (hypothesis testing) instead of mea-
suring estimation performance. Nevertheless, these results have connections to a
CRB incorporating the side information in (3.1), or a constrained CRB. A number
of papers, including Aitchison and Silvey [3] and Crowder [18], using the method of
Lagrangian multipliers, examine the asymptotic normality of the constrained maxi-
mum likelihood estimator (CMLE) and as a consequence unintentionally develop a
CRB under equality constraints. Under certain conditions, the asymptotic variance
of the MLE equaling the CRB lends credence to the claim that the asymptotic vari-
19
ance of the CMLE should equal to the CRB under equality constraints, although the
authors did not always appear cognizant of this fact. Others, including Silvey [63] to
develop his Lagrange multiplier test, Osborne [55] with linear constraints to develop
a scoring algorithm, and Waldorp, Huizenga, and Grasman [73] to develop a Wald-
type test, also use the Lagrange multiplier approach in developing a constrained
bound. Again, since these authors were primarily focused on asymptotic properties
or hypothesis testing, the nature and perhaps utility of this mathematical quantity
in their work is not explicitly stated as a CRB or bound on performance estima-
tion of parameters under constraints. The creation of a constrained bound strictly
for the use in performance analysis wasn?t achieved until Gorman and Hero [23].
Gorman and Hero derived such a measure by taking the limit of the Hammersley-
Chapman-Robbins bound with test points restricted to exist only in the constraint
space. This constrained Cram?er-Rao bound (CCRB)
I?1(?)?I?1(?)FT(?)parenleftbigF(?)I?1(?)FT(?)parenrightbig?1F(?)I?1(?) (3.2)
utilizes the Jacobian F(?) of the functional constraint f(?) and the inverse of
the Fisher Information matrix (FIM) I(?) (based on the unconstrained model),
which must be nonsingular. As with the CRB, there exist a number of alternative
derivations. The works of Crowder, Waldorp, et al, Gorman and Hero, and Aitchison
[2] include the formula in (3.2) in some manner for which the CRB might be used for
the unconstrained scenario in their works, a fact that implicitlyproves the validity of
the CCRB.Withthe explicitproof by Gorman and Heroasa guideline,Marzetta[47]
providesan elementaryproof of thisCCRB,whichavoidstheuse of the applicationof
20
the Cauchy-Schwarz inequality, and avoids the use of pseudo-inverses, by examining
the inequality created from the positive-semidefiniteness property for the variance
of a properly defined random variable. A similar construction was used by Stoica
and Ng [68] to formulate a more general CCRB
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?) (3.3)
that incorporates the constraint information without the assumption of a nonsin-
gular FIM. This CCRB utilizes the unconstrained FIM and an orthonormal com-
plement matrix U(?) whose vectors span the null space of the constraint Jacobian
matrix. Furthermore, when the FIM is nonsingular, the Stoica-Ng CCRB in (3.3) is
equivalent to the Gorman-Hero version of (3.2). Hence, while much of the previous
work used the formula in (3.2), the more general formula (3.3) is also applicable.
Osborne, independently from much of these other works, developed a method of
scoring with constraints that utilizes the Stoica-Ng CCRB formula in (3.3) as the
projection matrix in place of the CRB. There are, of course, numerous instances of
matrix structures of the same form as (3.3), for example as part of the projection
matrix of the generalized least squares estimate of the mean of a linear model.
In this section, we develop a very simple derivation of the CCRB in (3.3).
Rather than assuming the parameters satisfy functional constraints, we approach
the problem theoretically from an alternative, yet equivalent, perspective and as-
sume the parameters locally fit a reduced parametric model. This approach permits
the extension of the existing classical theory underlying the CRB to the case of a
model under parametric constraints. While it is true that several of these exten-
21
sions already exist in the literature, there does not exist a cohesive treatment of
these results. However, this chapter should not be viewed as simply a collection of
historical results, but a unified and comprehensive development of the theory of the
constrained Cram?er-Rao bound.
3.1 The Constrained CRB
Suppose we observe x?X?Rn from a probability density function p(x;?)
where????Rm isa vectorof unknown deterministicparameters and, inaddition,
suppose these parameters are required to satisfy k consistent and nonredundant
continuously differentiable parametric equality constraints, i.e., f(?) = 0 for some
consistent and nonredundant f : ??Rk. We shall denote
?f =
braceleftBig
?prime ?? :f(?prime) = 0,f consistent, nonredundant
bracerightBig
(3.4)
to be the feasible set satisfying the constraints. Hence, the constraint can also be
stated???f. The constraints being consistent means that the set ?f is nonempty.
The constraints beingnonredundant means that the JacobianF(?prime) = f(?
prime)
??primeT has rank
k whenever f(?prime) = 0.
As before, the Fisher information matrix (FIM) of this model (ignoring the
constraint) is given by
I(?) = E?braceleftbigs(x;?)sT(x;?)bracerightbig
where s(x;?) is the Fisher score defined by
s(x;?) = ?logp(x;?
prime)
??prime
vextendsinglevextendsingle
vextendsinglevextendsingle
?prime=?
22
and the expectation is evaluated at ?, i.e., E?(?) =
integraldisplay
X
(?)p(x;?)dx.
Incorporating this additional side information into the information from the
observations x directly would require an alteration of the pdf?s dependence on the
unknown parameter. Such an approach can often be analytically impractical or
numerically complex. Hence, it is desirable to have a formulaic or prescriptive ap-
proach to include the side information, or constraints, indirectly. To meet this need,
Stoica and Ng developed a method to incorporate parametric equality constraints
into the CRB [68, theorem 1].
Theorem 3.1 (Stoica & Ng). The constrained Cram?er-Rao bound on ? ? ?f is
given by
CCRB(?) =U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?) (3.5)
where U(?) is a matrix whose column vectors form an orthonormal basis for the
null space of the Jacobian F(?), i.e.,
F(?)U(?) = 0 , UT(?)U(?) =I(m?k)?(m?k). (3.6)
Thus, ift(x) is an unbiased estimator of?, which satisfies the constraint (3.4), then
Var(t(x))?CCRB(?)
with equality if and only if t(x)?? = CCRB(?)s(x;?) in the mean-square sense.1
1The originaltheorem requires the estimator to satisfy the constraint. In general, the parameter
and its unbiased estimator will not simultaneously satisfy the constraint since the implication,
mainly that f(E?t(x)) = E?f(t(x)), is true only under particular conditions. However, the
CCRB is the same if either assumption is made exclusively. In this treatise, I assume that the
actual parameter ? satisfies the constraint and the unbiased estimator t(x) does not. (In section
3.4, the constrained maximum likelihood estimator (CMLE) is assumed to satisfy the constraint,
but unbiasedness is not assumed.)
23
The Jacobian F(?) having full row rank is not necessary since the Jacobian
does not explicitly appear in the CCRB formula in (3.5). Indeed, the requirement
that the column vectors ofU(?) form an orthonormal basis is also unnecessary, only
that they be linearly independent and that they span the basis of the null space of
the row vectors of F(?), i.e., only that the column space of U(?) is an orthogonal
complement of the row space of F(?), is required since it is clear from the structure
of (3.5) that the CCRB is invariant to automorphisms inRm?rank(F(?)) onU(?). Re-
gardless, for convenience,and except where otherwise noted, we will assume that the
constraints are nonredundant and the columns of U(?) are orthonormal to ensure
that rank(U(?)) = m?k and UT(?)U(?) = Im?k, respectively. The existence of
the bound only requires thatUT(?)I(?)U(?) rather than the FIM be nonsingular.2
The original proof of this theorem given by Stoica and Ng considers the variance
inequality generated by the random variable t(x)???WU(?)UT(?)s(x;?) and
maximizesW to attain the tightest bound for Var(t(x)??).
Example 3.2 (Unit Modulus Constraint). Suppose ? in example 2.2 is constrained
to be unit modulus, i.e., f(?) =|?|2?1. Then its gradient in terms of the param-
eter vector ? =
bracketleftbiggRe(?)
Im(?)
bracketrightbigg
is F(?) = bracketleftbig2Re(?), 2Im(?)bracketrightbig, which has an orthonormal
complementU(?) =
bracketleftbigg?Im(?)
Re(?)
bracketrightbigg
. The CCRB for this constraint is then
CCRB(?) = ?
2
2
bracketleftbigg Im(?)2 ?Im(?)Re(?)
?Re(?)Im(?) Re(?)2
bracketrightbigg
.
2A corresponding regularity condition to that mentioned in section 2.1 will be discussed in
section 3.1.4.
24
3.1.1 A proof of the CCRB
While the proof given by Stoica and Ng is sufficient to establish the validity
of the CCRB, these proofs ignore the existing classical theory encompassing the
CRB and FIM, which is already sufficient to prove the CCRB. However, prior to
developing the foundation for the CCRB from the existing theory, we require a foray
into multivariable calculus and, specifically, the use of the implicit function theorem.
The reward for this approach will be a seamless presentation of statistical inference
involving the constrained Cram?er-Rao bound.
From the perspective of multivariable calculus, the constraint f(?) = 0 effec-
tively restricts? to a manifold ?f of the original vector space ?, with the manifold
having dimension m?k since k degrees of freedom are lost when rank(F(?)) = k
for all ? ? ?f. More formally, the following theorem [65, theorems 5-1 and 5-2]
applies.
Theorem 3.3 (Implicit Function Theorem). Let U ? Rm be an open set and
assume f : U?Rk is a differentiable function such that F(?) has rank k whenever
f(?) = 0. Then ?f ?U is an (m?k)-dimensional manifold in Rm, and for every
? ? ?f ?U there is an open set V owner ?, an open set W ? Rm?k, and a 1-1
differentiable function g? : W?Rm such that
(a) g?(W) = ?f ?U?V, and
(b) the Jacobian of g?(?prime) has rank m?k for each ?prime ?W.
Therefore, there exists a function g? : Rm?k ? Rm, and sets O owner? and P
open in ?f and Rm?k, respectively, such that g? : P?O is a diffeomorphism on P,
25
i.e., a continuously differentiable bijection with a continuously differentiable inverse.
A geometric example is shown in Figure 3.1. Note this diffeomorphism depends on
the parameter ? as the reparameterization is only guaranteed to exist in a local
neighborhood of ?; however, for convenience, we will omit this notation in this
subsection so that g=g?. Thus, we may proceed under the assumption that every
?prime ?O ? ?f is the image of a unique reduced parameter vector ?prime ? P?Rm?k
under g, or simply
?prime =g(?prime). (3.7)
Necessarily, there exists some unique ??P for which ? =g(?). We will denote the
Jacobian of g to be G(?prime) = ?g(?
prime)
??primeT , which also implicitly depends on ?.
4
6
?0.8
?0.6
?0.4
?0.2
0
0.2
0.4
0.6
?o
?o
2D manifold in 3D space Reparameterized Space
f(?)=0
? g(?)
g?1(?) ?
Figure 3.1: Reparameterization of f(?) = 0 to ? =g(?).
Example 3.4 (Unit Modulus Constraint). As an example of this principle, con-
sider a complex parameter ? with a modulus constraint (as in example 3.2). The
parameter vector in this case may be ? = [?1,?2]T = [Re(?),Im(?)]T ? R2 with
the constraint being f(?) = ?21 + ?22 ?1 = 0. By the implicit function theorem,
26
constraining ? to be unit modulus is tantamount to assuming the existence of a
??R such that ?= e?j?. Hence, ? is a function of ?, i.e.,
? =
bracketleftbiggRe(?)
Im(?)
bracketrightbigg
=
bracketleftbigg cos(?)
?sin(?)
bracketrightbigg
=g(?). (3.8)
Also, g(R) ={??R2 : f(?) = 0} and G(?) = bracketleftbig?sin(?) , ?cos(?)bracketrightbigT has rank 1
for every ?. For the model in example 3.2, then
CCRB(?) = ?
2
2
bracketleftbigg sin2(?) ?sin(?)cos(?)
?cos(?)sin(?) cos2(?)
bracketrightbigg
,
which is exactly as before.
It must be noted thatgwillnot be unique. For the previousexample,?= ej? is
another possible reparameterization. Nor is anyg satisfying the theorem necessarily
a 1-1 correspondence between Rm?k and ?f; again, for the previous example, g is
periodic. Thus, the bijection is only guaranteed locally. Finding a particularg for a
givenf and? may not be obvious. Methods for approximating an implicit function
will be discussed in section 3.1.5. Regardless, as shall be shown in the context of
the CCRB, knowledge of any particular g is unnecessary; only its existence, given
by the implicit function theorem, is necessary. Why? Using the implicit function
theorem to assume a locally equivalent reparameterization for the constraint limits
the information from the observations to the density function?s local dependence
on the unknown parameter. But as the CRB (and hence CCRB) is a local bound
that only characterizes the local noise ambiguities in the model, i.e., the average
local curvature of the density at the parameter value of interest, this limitation is
invariant to the local curvature restricted to a space determined by the constraints.
27
Theorem 3.5 ([50]). The CRB on??O under the assumption of (3.7) is given by
G(?)parenleftbigGT(?)I(g(?))G(?)parenrightbig?1GT(?) (3.9)
and if g in (3.7) is an implicit function of f(?) = 0 then this bound is equivalent to
the constrained Cram?er-Rao bound in (3.5).
Proof. This CRB can be developedfrom a transformation of parameters on the FIM
and the CRB. From the CRB transformation of parameters on the CRB (see section
2.1.1), if t(x) is an unbiased estimator3 of ? =g(?) then
Var(t(x))?CRB(g(?)) =G(?)?I?1(?)GT(?) (3.10)
with equality if and only if t(x)?g(?) = G(?)?I?1(?)?s(x;?) in the mean-square
sense [36, Appendix 3B], where ?I(?) = E??s(x;?)?sT(x;?) is the FIM on ? and
?s(x;?) = ? logq(x;?
prime)
??prime
vextendsinglevextendsingle
vextendsingle
?prime=?
is the Fisher score of the pdf with respect to?, this pdf be-
ing q(x;?) = p(x;g(?)). By application of the derivative chain rule the Fisher score
of x with respect to ? is ?s(x;?) = GT(?)s(x;?). Hence, from the transformation
of parameters on the FIM [12, p. 157], the FIM on ? is4
?I(?) = Ebraceleftbig?s(x;?)?sT(x;?)bracerightbig
= EbraceleftbigGT(?)s(x;?)sT(x;?)G(?)bracerightbig
= GT(?)I(?)G(?)
= GT(?)I(g(?))G(?). (3.11)
3Again, there is no actual use of the assumption here that t(x)??f although if t(x)?g(P)
then t(x) does indeed satisfy the constraint. The theorem result for unbiased estimators holds
regardless and this will depend on the regularity condition detailed in section 3.1.4.
4Implicitly, it is assumed that the regularity conditions of section 2.1 hold with respect to ?.
For how this applies to ?, see section 3.1.4.
28
Substituting (3.11) into (3.10) establishes the CRB under the assumption? =g(?).
To establish the equivalence with the CCRB, note sincef?g(?) = 0, then by taking
the Jacobian with respect to ? we have
0 = ???T 0 = ???f(g(?)) =F(g(?))G(?) =F(?)G(g?1(?)).
Hence, the columns of G(?) reside in the null space of the row vectors of F(?).
And since g has an inverse locally at ? = g?1(?), then G(?) has full column rank
m?k locally about ?. (This is true on the whole set P ? Rm?k.) Therefore,
span(G(g?1(?))) = span(U(?)) and there exists an m?k?m?k full rank trans-
formation matrix S(?) such that G(g?1(?))S(?) = U(?). (This is true on the
whole set O??f.) This matrixS(?) is merely an orthonormalizing change of basis
on the columns of G(g?1(?)). Therefore,
G(?)parenleftbigGT(?)I(g(?))G(?)parenrightbig?1GT(?)
= G(g?1(?))parenleftbigS?T(?)UT(?)I(?)U(?)S?1(?)parenrightbig?1GT(g?1(?))
= G(g?1(?)S(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1ST(?)G(g?1(?))
= U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?).
Also, Var(t(?))?CCRB(?) with equality if and only if
t(x)?g(?) = G(?)?I?1(?)?s(x;?)
t(x)?? = G(?)?I?1(?)GT(?)s(x;?)
= CCRB(?)s(x;?).
29
This proof of the CCRB uses the implicit function theorem, the CRB transfor-
mation formula, the FIM transformation formula, as well as well-known properties
of rank and the derivative chain rule. An advantage of establishing the CCRB from
these classical results will become clear as we establish the connection throughout
statistical information theory. An example of this is immediately evident in an
alternative proof of a proposition of Stoica and Ng [68, proposition 1].
Corollary 3.6 (Stoica & Ng). Given the regularity conditions on ?, a necessary
and sufficient condition for the existence of a finite CCRB of ? is
vextendsinglevextendsingleUT(?)I(?)U(?)vextendsinglevextendsinglenegationslash= 0,
i.e.,UT(?)I(?)U(?) is nonsingular.
Proof. From the prior theorem, it is clear thatUT(?)I(?)U(?) is nonsingular if and
only if GT(?)I(g(?))G(?) is nonsingular if and only if ?I(?) is nonsingular. Since a
necessary and sufficient condition for the existence of a finite CRB of g(?) is that
?I(?) is nonsingular, the corollary is proven.
Thus, with the usual regularity conditions (see section 2.1) being maintained
for ?, the conditions for the existence of the CCRB with respect to? are equivalent
to the conditions for the existence of the CRB in the reduced parameter space of ?.
Moreover, the matrix UT(?)I(?)U(?) is nonsingular if and only if U(?) has
fullcolumnrankm?kandno linearcombinationof the columnsofU(?)reside inthe
null space of I(?). The first condition is satisfied always by definition. The second
condition implies,for example, that ifL(?)LT(?) is the Cholesky decomposition [20,
30
p.194] of the FIM, whereL(?)?Rm?rank(I(?)) is a full column rank lower triangular
matrix with strictly positive values on the diagonal, then U(?) = L(?)A(?) +
K(?)B(?) where K(?) is an orthogonal complement of L(?) and for some full
column rank A(?)?Rrank(I(?))?m?k.
Further properties of the CCRB will be discussed in section 3.1.4.
3.1.2 Alternative formulas
The formula Stoica and Ng used to express the constrained Cram?er-Rao bound
is a generalization of an expression developed earlier first by Gorman and Hero [23,
theorem 1] and later by Marzetta [47, theorem 2]. This is the CCRB formula in
(3.12). Although Gorman and Hero?sformula requires a nonsingular Fisher informa-
tion, the version developed by Stoica and Ng appears to be inspired by a result [23,
(19) in lemma 2] in Gorman and Hero?s paper that unnecessarily assumes a positive
definite FIM. However, this result was, in essence, not unknown in the literature.
Gorman and Hero were aware of the prior work of Aitchison and Silvey [3, theorem 2
and P on p.823], which is concerned with the asymptotic variance of the maximum
likelihood estimator subject to restraints. But they were perhaps unaware (by lack
of citation) of a later paper on hypothesis tests associated with the maximum like-
lihood, in which Aitchison and Silvey suggest a solution to the problem of singular
information matrices [4, section 6]. This is the CCRB formula in (3.14). This ver-
sion of the CCRB was also proven by Crowder [18, theorem 3]. Concurrent to the
Gorman and Hero effort, Hendriks [27] and later Oller and Corcuera [53] developed
31
an extension of the Cram?er-Rao bound intrinsic to the manifold using Riemannian
geometry. More recently, Xavier and Barroso [74, 75] specified the lower bound on
the geodesic of estimators to the true parameter. The original and latest versions
of their bound are expressed in (3.16) and (3.17), respectively.
While these CCRB expressions are not the focus of the current treatise, they
are still important for possible insights into the constrained Cram?er-Rao bound. In
this section, aspects of these insights will be briefly discussed as well as conditions
for equality with the CCRB expression in (3.5).
3.1.2.1 Gorman-Hero-Aitchison-Silvey CCRB
Aitchison and Silvey used the method of Lagrange multipliers to show that
the weighted asymptotic variance of the constrained maximum likelihood estimator,
which should be the CCRB (implicitly), tends to
CCRB2(?) =I?1(?)?I?1(?)FT(?)parenleftbigF(?)I?1(?)FT(?)parenrightbig?1F(?)I?1(?) (3.12)
when the Fisher information is nonsingular. Alternatively, Gorman and Hero de-
veloped this same CCRB by restricting test points in the Chapman-Robbins bound
(a Barankin-type bound) to be in the constraint space ?f and then finding the
derivatives as the limit of the finite difference expressions in the Chapman-Robbins
bound.5 A simpler proof was provided by Marzetta by considering the positive
semidefiniteness of a properly chosen random variable.
A particular advantage to this presentation of the CCRB is the explicit quan-
5A definition of the Chapman-Robbins bound as well as a variant of the Gorman and Hero
proof, which allows for a singular FIM, can be found in Appendix A.1.
32
tification of the gain in performance potential. Imposing a constraint f(?) = 0
on a set of parameters improves (lowers) the unconstrained bound from I?1(?) by
exactly I?1(?)FT(?)parenleftbigF(?)I?1(?)FT(?)parenrightbig?1F(?)I?1(?). Since the CRB of f(?)
is F(?)I?1(?)FT(?) then its inverse is the Fisher information of f(?) or 0. And
FT(?)parenleftbigF(?)I?1(?)FT(?)parenrightbig?1F(?) is the Fisher information of ? generated from
the constraint f(?) = 0. A disadvantage is the requirement of a nonsingular FIM.
There exist numerous scenarios that require constraints for the original model to be
identifiable (see Chapter 4). Additionally, this CCRB formula requires nonredun-
dant constraints, i.e., the Jacobian F(?) must be full row rank.
Similarities include the computational complexity of both formulas, which
is O(m3). And when the Fisher information is nonsingular (and the constraints
nonredundant), both formulas are equivalent.
Theorem 3.7. When the Fisher information is nonsingular and the constraints
nonredundant, then an equivalent formula for the CCRB in (3.5) is CCRB2(?).
Proof. This is a different proof than the one provided in [68, corollary 1].6 The
existence of the Gorman-Hero-Marzetta formula assumes that the FIM I(?) and
the CRB of the constraint F(?)I?1(?)FT(?) are regular (non-singular) [23, 47].
Correspondingly, the existence of the Stoica and Ng CCRB formula assumes that
6In addition to this alternate proof, they reference an algebraic identity from [37] that is useful
in establishing the result.
Lemma 3.8 (Khatri). Suppose A is p?q and B is p?p?q have ranks q and p?q respectively
such that BTA= 0. Then for any symmetric positive definite matrix S,
S?1?S?1A(ATS?1A)?1ATS?1 =B(BTSB)?1BT.
Substituting I(?) for S, F(?) for AT, and U(?) for B shows the equivalence between the two
CCRBs for when the FIM is nonsingular.
33
UT(?)I(?)U(?) is non-singular. Now, right-multiplying both formulas by FT(?)
returns the results
CCRB(?)FT(?) = U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?)FT(?) = 0,
CCRB2(?)FT(?) = I?1(?)FT(?)?I?1(?)FT(?) = 0
since F(?)U(?) = 0 in the first equation and the elimination of the inverse in
CRB(f(?)) in the second. Alternatively, right-multiplying both formulas by the
matrix I(?)U(?) returns the results
CCRB(?)I(?)U(?) = U(?),
CCRB2(?)I(?)U(?) = U(?)
again by eliminating the inverse in the first equation and since F(?)U(?) = 0 in
the second. Hence we have the equality
CCRB(?)bracketleftbigFT(?) I(?)U(?)bracketrightbig= CCRB2(?)bracketleftbigFT(?) I(?)U(?)bracketrightbig
and if it can be shown that the matrixbracketleftbigFT(?), I(?)U(?)bracketrightbigis regular, then the two
CCRB formulas are shown to be equivalent. Suppose there exists vectors ??Rk
and ??Rm?k such that
FT(?)?+I(?)U(?)?= 0. (3.13)
Pre-multiplying (3.13) by UT(?) implies that
UT(?)I(?)U(?)?= 0.
34
Since UT(?)I(?)U(?) is regular then ? = 0. Likewise, premultiplying (3.13) by
F(?)I?1(?) implies that
F(?)I?1(?)FT(?)?= 0.
Since F(?)I?1(?)FT(?) is regular then ? = 0. Hence FT(?)?+I(?)U(?)? = 0
implies?= 0 and ? = 0, which proves that bracketleftbigFT(?) I(?)U(?)bracketrightbig is full rank.
3.1.2.2 Aitchison-Silvey-Crowder CCRB
The solution for resolvinginvertibilityof singular FIMsin the variance and test
results of Aitchison and Silvey [4] was to load the Fisher information with a matrix
of the formFT(?)F(?). This was made more rigorous by Crowder [18] by replacing
the Fisher information I(?) with a loaded FIM D(?) = I(?) + FT(?)KF(?),
where K is any positive semidefinite matrix such that D(?) is regular. Hence, a
generalization of (3.12) is
CCRB3(?) =D?1(?)?D?1(?)FT(?)parenleftbigF(?)D?1(?)FT(?)parenrightbig?1F(?)D?1(?).
(3.14)
This extension permits a singular FIM. This formulation is also independent of the
choice of K.
Theorem 3.9. An equivalent formula for the CCRB in (3.5) is CCRB3(?).
Proof. Replace I(?) with D(?) in the proof of theorem 3.7.7
7Inappendix A.3, thisCrowder formulaforthe asymptoticvarianceof the constrained maximum
likelihood estimate is shown to be equivalent to the CCRB using lemma 3.8 (also see section 3.4.2).
35
As the computational complexity of the two formulas are O(m3), there is no
direct advantage of one CCRB version over the other. Certainly, in the nonsingular
FIM case, CCRB3(?) is the same as the CCRB2(?) (with K = 0). Unfortunately,
this CCRB formula does not appear to have simple connections to other areas of
statistical inference in an inherent manner. Nevertheless, this possible inefficacy in
theoretical applications does not effect its practical use.
3.1.2.3 Xavier?s & Barroso?s Intrinsic Variance Lower Bound
The prior CCRB metrics were in Euclidean Rm space, i.e., the lower bound is
on the measurementof the distance (insome directionor dimension) of the estimator
to the true value of the parameter measured by ?cutting through? the manifold. In
some scenarios, it may be of interest to know what the bound is on the measurement
of the distance ?over the surface? of the manifold. Since dimensional directions can
be somewhat ambiguous depending on the manifold, of particular interest is the
geodesic, or shortest distance.
For this scenario, Xavier and Barroso [74, 75] formulated an inequality, the
intrinsicvariance lowerbound (IVLB),for the variance of the geodesic to anunbiased
estimator intrinsic to the manifold. Their results are derived from those of Hendriks
[27] and Oller and Corcuera [53]. Ignoring elementsof Riemannian geometric theory
that are beyond the scope of this presentation, their IVLB result essentially relies
on the inequality
?
C
radicalbig
var(?)cot(
?
C
radicalbig
var(?))?
radicalbigvar(?)
??
?
, (3.15)
36
where C is an upper bound on the sectional curvature of the manifold, ?? is a
bound on variance of the Euclidean estimator error, and var(?) is the variance of
the geodesic. Precisely, C = max???f K? where
K? = max
v1,v2 orthonormal
F(?)vi=0
< ?(v1,v1),?(v2,v2)>?< ?(v1,v2),?(v1,v2) >
and ?(?,?) is the second fundamental form [34] of ?f defined by
?(a,b) =?FT(?)parenleftbigF(?)FT(?)parenrightbig?1
bracketleftBig
aT ?Fi(?)?? b
bracketrightBig
k?1
onU?U whereU = span{U(?)}.
Xavier and Barroso use a polynomial bound on the cotangent to solve the
lower bound of a quadratic in terms of var(?). In an earlier variant of the IVLB [74],
Xavier and Barroso chose ??1? = max
v?U,||v||=1
vTI(?)v and bounded tcot(t)?1?23t2,
for 0?t?T ?1.35, in (3.15) where t =?Cradicalbigvar(?), to bound on the variance of
the estimator?s geodesic to the mean by
var(?)? 4C + 3???
radicalbig?
?(9?? + 24C)
8
3C
2 . (3.16)
Unfortunately, this bound was optimistic in the limit for the simple Euclidean case
(C = 0). In a more recent paper [75], the authors improved the bound by choosing
?? = tr(UT(?)I(?)U(?))?1 and an alternative lower bound for tcot(t) to obtain
var(?)???C + 1?
?2?
?C + 1
1
2C
2?? . (3.17)
Although the authors omitted a proof8, the alternative lower bound for tcot(t)
appears to be tcot(t) ? 1? 12t2 for 0 ?t?T. An immediate benefit from this
8Xavier and Barroso stated in [75] that the proof would ?be found in the companion paper
[14]?, but this companion paper appears to have never been published.
37
improved bound is the agreement with the simple (C = 0) Euclidean case, i.e.,
var(?)??? = tr(UT(?)I(?)U(?))?1. This consistency is entirely due to the choice
of ??. Even so, greater improvement of the bound, not found in the literature, is
possible in the curved scenarios (C > 0) at least in the bound of tcot(t). Note, for
0?t<pi
tcot(t) = 1+
?summationdisplay
s=1
(?1)s22sB2st2s
(2s)! = 1?
?summationdisplay
s=1
2
parenleftbiggt
pi
parenrightbigg2s
?(2s)
? 1?
?summationdisplay
s=1
2
parenleftbiggt
pi
parenrightbigg2s pi2
6 = 1?
pi2
3
parenleftbiggt
pi
parenrightbigg2
1?
parenleftbiggt
pi
parenrightbigg2
? 1?b2t2
where B2s are Bernoulli numbers, ?(?) is the Riemann zeta function, and b =
2/3
1?
?T
pi
?2 ?0.8177. Then
var(?)???Cb+ 1?
?2?
?Cb+ 1
1
2C
2b2?? . (3.18)
3.1.3 Simple Extensions to the CCRB
As with unconstrained parameters, the performance of a function of parame-
ters often may be of greater interest. Consider a continuously differentiable func-
tion k : ?f ? Rq. Denote the Jacobian of this transfer function to be K(?) =
?k(?prime)
??primeT
vextendsinglevextendsingle
vextendsingle
?prime=?
. We have a simple extension of the classical transformation of parame-
ters in section 2.1.1.
Corollary 3.10. If f(?) = 0, then the variance of any unbiased estimator S(x) of
38
?=k(?) satisfies the inequality
Var(S(x))?CCRB(?) ?=K(?)CCRB(?)KT(?).
Proof. Let g? be the implicit function defined by f(?) = 0. Then ? = k(g?(?))
has a Jacobian K(?)G?(?). From the classical transformation of parameters, the
inequality Var(S(x))?CRB(?) holds where
CRB(?) = K(?)G?(?)?I?1(?)GT?(?)KT(?)
= K(?)U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?)KT(?).
This transformation property is useful in extending the constrained Cram?er-
Rao bound to biased estimation.
Example 3.11 (Biased Estimation). Assume t(x) is a biased estimator of a con-
strained parameter ? with bias b(?) = E?t(x)?? and constraint f(?) = 0. De-
fine k(?) = ? +b(?). Then t(x) is an unbiased estimator of ? = k(?). Since
K(?) =Im?m +B(?) where B(?) = ?b(?
prime)
??primeT
vextendsinglevextendsingle
vextendsingle
?prime=?
, then we have the inequality
Var(t(x))?parenleftbigIm?m +B(?)parenrightbigCCRB(?)parenleftbigIm?m +BT(?)parenrightbig ?= CCRB(?+b(?)).
Often, when the Fisher information matrix is singular, its pseudoinverse is
used as a bound on the variance of an estimator. As mentioned in section 2.1.1, the
bound is trivially true for some component of the estimator except under certain
conditions.
39
Corollary 3.12. If t(x) is an estimator of k(?) having bias b(?), where ? satisfies
the constraint f(?) = 0, then the inequality
Var(t(x))?H(?)U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?)HT(?),
where H(?) = K(?) +B(?), is nontrivially satisfied, i.e., all components of t(x)
have finite variance, if and only if
H(?)U(?) =H(?)U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?)I(?)U(?).
Proof. Suppose t(x) is an estimator of ? = k(?) with bias b(?) and under the
constraint f(?) = 0. Define H(?) = ???primeT parenleftbigk(?) +b(?)parenrightbig
vextendsinglevextendsingle
vextendsingle
?prime=?
= K(?) +B(?). If
g? is the implicit function defined by f, then we have the inequality
Var(t(x))? ?H(?)?I?(?) ?HT(?)
where ?H(?) = ???primeT parenleftbigk(g?(?prime)) +b(g?(?prime))parenrightbig
vextendsinglevextendsingle
vextendsingle
?prime=?
= H(?)G?(?). (This can also
be inferred from [23, lemma 2], although no assumption is made here about the
singularity of the Fisher information.) Hence, all the components of t(x) can have
finite variance if and only if [66]
?H(?) = ?H(?)?I(?)?I?(?)
H(?)G?(?) = H(?)G?(?)?I?(?)?I(?)
= H(?)G?(?)?I?(?)GT?(?)I(?)G?(?)
= H(?)U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?)I(?)G?(?)
H(?)U(?) = H(?)U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?)I(?)U(?).
40
By definition U(?) will always be full column rank, so singularity of the ma-
trix UT(?)I(?)U(?) implies singularity in the FIM I(?) and insufficiency in the
constraints to resolve the inherent ambiguities in the model. This scenario will be
discussed in greater detail in the next section.
3.1.4 Properties of the CCRB
The constrained Cram?er-Rao bound for the constraint f(?) = 0 is equiva-
lent to the CRB for any reparameterization of the parameters satisfying the con-
straint. Implicit in that equivalence and in the proof of the CCRB presented in
section 3.1.1 are the regularity conditions for the CRB on the implicit parameter
?, i.e., ???TE?(h(x)) = E?(h(x)?sT(x;?)) for h(x)?1 and h(x)?t(x), where t(x)
in this case is an unbiased estimator of ? = g(?). This condition translates to
?
??TE?(h(x))G(?) = E?(h(x)s
T(x;?)G(?)) or, strictly in terms of ?,
?
??TE?(h(x))U(?) =E?(h(x)s
T(x;?)U(?)) (3.19)
for h(x)?1 and h(x)?t(x). From this condition, it can be shown that
E?(t(x)??)sT(x;?)U(?)UT(?) =U(?)UT(?), (3.20)
which is a variant of the regularity condition required and proven by Marzetta [47]
and the same regularity condition simply stated (but not proven) by Stoica and Ng
[68] in their proofs of the CCRB. Thus, as with the CCRB, the regularity condition
for the constraint is equivalent to the regularity condition for the reparameterization
of parameters satisfying the constraint. This general fact, a restatement of theorem
41
3.5, is the quintessential property of the CCRB formula. More explicitly,the CCRB
is a generalization of both the degenerate and the determinate constraint cases, as
examples 3.13 and 3.14 demonstrate.
Example 3.13 (Degenerate Case). The scenario with no constraint is equivalent
to the statement that the function which describes the constraint is null. That is,
f : ??R0 and f(?) = [ ]. Then F : ??R0?m is also a null gradient row vector
having rank 0. Any nonsingular m?m matrix U(?) satisfies (3.6), and thus
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?) =I?1(?).
Therefore the CCRB formula also incorporates the unconstrained scenario.
Example 3.14 (Determinate Case). Suppose the constraint f completely deter-
mines the parameter. Then necessarily, since we have m unknowns in the param-
eter vector ?, there must be at least k ? m equations in the constraint equa-
tion and the Jacobian F(?) must have rank m. Since f is assumed to have
nonredundant constraints, F(?) is actually a nonsingular square matrix. Only
the null vector U : ? ? Rm?0 satisfies (3.6) (UT(?)U(?) = I0?0), therefore
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?) is a null element, and the CCRB does not exist
for a completely determined parameter set.
Thus far, only parametric equality constraints have been considered. Gorman
and Hero show that only active constraints results in a reduction in the bound [23,
lemma 4], i.e., inactive (or strict) inequality constraints do not contribute informa-
tion to the model in the CCRB sense.9 As the test points approach the parameter
9Because the CCRB (CRB) is a local bound that only accounts for local fluctuations of the
42
in the Chapman-Robbins bound, they become interior points of the constraint set
and the inequalities have no impact on the information metric. This can also be
shown without resorting to the Chapman-Robbins approach.
Example 3.15 (Inequality Constraints Are Non-Informative). Assume, in addition
to the constraint set ?f, the parameters are required to satisfy the strict inequality
h(?)< 0 where h : ??R is a continuously differentiable function. To incorporate
this constraint, we introduce a dummy parameter ? and add a new equality con-
straint?2 =?h(?), which is equivalent to the strict inequality constraint (whenever
?negationslash= 0). In addition we create an extended parameterization ?prime =
bracketleftbigg?prime
?prime
bracketrightbigg
?Rm+1 and
constraint functionfstar(?prime) =
bracketleftbigg f(?prime)
?prime2 +h(?prime)
bracketrightbigg
. This will generate a Fisher information
and a Jacobian matrix defined by
Istar(?) =
bracketleftbiggI(?) 0
0 0
bracketrightbigg
, Fstar(?) =
bracketleftbiggF(?) 0
HT(?) 2?
bracketrightbigg
,
respectively, where H(?) = ?h(?
prime)
??prime
vextendsinglevextendsingle
vextendsingle
?prime=?
. Note that Fstar(?) will be full row rank as
long as ?negationslash= 0. If U(?) is defined as in (3.6), then Ustar(?) =
bracketleftbigg U(?)
vT(?,?)
bracketrightbigg
will also
satisfy (3.6) with respect to Fstar(?), where v(?,?) = 12?HT(?)U(?). By theorem
true parameter, inactive constraints have no impact on performance potential. As such the CCRB
(CRB) only provides information on the pdf for the mainlobe of the density function, which typi-
cally corresponds to the true parameter. For a number of scenarios, e.g., when there is sufficiently
large variance in the pdf, sidelobes of the distribution impact the performance, thereby making
the CCRB (CRB) overly optimistic. This occurs frequently in communications when the signal-
to-noise ratio (SNR) or data transmission size decreases.
43
3.1, then
Ustar(?)parenleftbigUstarT(?)Istar(?)Ustar(?)parenrightbig?1UstarT(?)
= Ustar(?)parenleftbigU(?)I(?)U(?)parenrightbig?1UstarT(?)
=
bracketleftBigg
U(?)parenleftbigU(?)I(?)U(?)parenrightbig?1U(?) U(?)parenleftbigU(?)I(?)U(?)parenrightbig?1v(?,?)
vT(?,?)parenleftbigU(?)I(?)U(?)parenrightbig?1UT(?) vT(?,?)parenleftbigU(?)I(?)U(?)parenrightbig?1v(?,?)
bracketrightBigg
.
Hence, the CCRB on the ? components of ?, the upper-left submatrix, remains
unchanged.
Equality constraints, however, do add side information to the model. Thus,
it is intuitive to expect that the constrained model should result in a lower bound
compared to the bound for the model without constraints. This statement was made
in Gorman and Hero [23, p.1292], but not in Marzetta [47] nor Stoica and Ng [68].
In the latter case, the statement is only true under certain conditions. Prior to
establishing when the bound is lowered, a powerful lemma will be proven.
Lemma 3.16. For an arbitrary full column rank matrix A, and an arbitrary sym-
metric positive semidefinite matrix B, the inequality
AparenleftbigATBAparenrightbig?AT ?B? (3.21)
holds over the projection subspace ofB?Bwith equality if and only if rank(ATBA)
= rank(B).
Proof. Let LLT be the Cholesky decomposition of B [20, p. 194]. Then L ?
Rm?rank(B) is a full column rank lower triangular matrix with strictly positive values
on the diagonal. To show the inequality, consider linear unbiased estimates of the
44
mean in the model y = D? +epsilon1 where epsilon1?N(0,(LTL)?2), and ? is treated as
the unknown parameter. In particular,y is such an estimate with variance equaling
(LTL)?2 where as the best linear unbiased estimate (see section 2.3.1)
D??=DparenleftbigDT(LTL)2Dparenrightbig?DT(LTL)2y,
with variance equal to DparenleftbigDT(LTL)2Dparenrightbig?DT. By the Gauss-Markov theorem we
have the inequality with the BLUE?s variance
DparenleftbigDT(LTL)2Dparenrightbig?DT ?(LTL)?2
with equality if and only if (LTL)DparenleftbigDT(LTL)2Dparenrightbig?DT(LTL) = Irank(L)?rank(L).
Substituting in D = parenleftbigLTLparenrightbig?1LTA in 3.1.4 and pre- and post-multiplying both
sides by L and LT, respectively, we have the inequality
B?BAparenleftbigATLLTAparenrightbig?ATBB? ?B?.
Considering quadratic forms in B?Bv proves the result since for any vector v ?
Rm, vTB?BB?BB?v = vTB?v. Moreover, from the definition of D, we have
(LTL)DparenleftbigDT(LTL)2Dparenrightbig?DT(LTL) =Irank(L)?rank(L) if and only if it can be shown
that LTAparenleftbigATLLTAparenrightbig?ATL = Irank(L)?rank(L), which is so if and only if A and L
satisfy rank(ATL) = rank(B).
This lemmaproves the following results, including that in a linear subspace the
CCRB(?) is lesser than or equal to the CRB(?) in the matrix sense and that con-
straints strictly on the ambiguous information have no impact on the performance
of the unambiguous information.
45
Theorem 3.17. Let U(?) be defined as in (3.6) from a constraintf(?) = 0. Then
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?)?I?(?)
over the linear projection subspace defined by I?(?)I(?).
Proof. From the lemma, defining A = U(?) and B = I(?) gives the inequality.
The conditions under which U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?UT(?) and I?(?) are
nontrivial bounds are detailed in corollary 3.12 and section 2.1.1, respectively. The
projection space of I?(?)I(?) corresponds to the identifiable components of ? with-
out the constraints (e.g., see [18, section 4]). The theorem establishes the result
that the constraints can only lower the bound and thereby increase performance po-
tential for (functions of) parameters that are already identifiable. This is regardless
of whether the Fisher information is singular or whether the parameters are iden-
tifiable under constraints (whether UT(?)I(?)U(?) is singular, see Theorem 3.24).
Theorem 3.17 is more general than the result shown by Ash10, which is given by the
following corollary.
Corollary 3.18 (Ash). Provided no linear combination of U(?) lies in the null
space of the Fisher information matrix, then
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?)?I?(?)
over the linear projection subspace defined by I?(?)I(?).
10Although not stated in this manner, this result appears in Ash?s thesis [5, equation (3.63)] as
well as in a publication of his third chapter in Ash and Moses [6, equation (63)]. The results from
Lemma 3.16 are more general than those in [5] or [6].
46
Proof. Since U(?) is full column rank and the range space of its columns does not
exist in the null space of the FIM, then parenleftbigUT(?)I(?)U(?)parenrightbig is nonsingular.
Furthermore, if the Fisher information matrix is nonsingular, then I?(?)I(?)
is the identity matrix, so the inequality holds as a quadratic form over Rm and
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?)?I?1(?).
This was also cleverly shown by Gorman and Hero [23, (44) in remark 4], again,
with the unnecessary assumption that I(?) is regular (non-singular). Next, we use
the lemma to observe the existence of non-informative constraints.
Corollary 3.19. Assume the row vectors of F(?)?Rk?m form a linearly indepen-
dent basis for the null space of I(?). Then F1(?)?Rkprime?m is a linear combination
of a submatrix of F(?) if and only if
U1(?)parenleftbigUT1 (?)I(?)U1(?)parenrightbig?UT1 (?) =I?(?),
where U1(?) is defined as in (3.6) relative to F1(?).
Proof. Without loss of generality, partition F as
bracketleftbiggF
1(?)
F2(?)
bracketrightbigg
. Then if U(?) is defined
as in (3.6), define
U1(?) =
bracketleftBig
U(?),
parenleftBig
Im?m?FT1 parenleftbigF1FT1 parenrightbig?1F1
parenrightBig
FT2 (?)D(?)
bracketrightBig
,
where D(?) represents a Gram-Schmidt processing matrix which orthonormalizes
the column vectors of
parenleftBig
Im?m?FT1 parenleftbigF1FT1 parenrightbig?1F1
parenrightBig
FT2 (?) FT2 (?) (these are already
orthogonal to the column vectors ofU(?)). This satisfies (3.6). Since span(I(?))?
47
span(U1(?)) then rank(UT(?)I(?)) = rank(I(?)), and thus, by lemma 3.16,
U1(?)parenleftbigUT1 (?)I(?)U1(?)parenrightbig?UT1 (?) =I?(?).
This corollary details the necessary and sufficient characteristics for a con-
straint to be non-informative. Implicit from the lemma and above corollary is the
CCRB?s invariance to linear combinations of columns of U(?) that exist in the null
space of I(?). The specific case when span(I(?)) = span(U(?)) is also shown by
Ash in [5, equation (3.73)] and [6, equation (73)].11
The CCRB can be interpreted geometrically, as in Figure 3.1 as a contraction
of the information ambiguity to find the bound and then an expansion. The column
vectors of U(?), being in the null space of the Jacobian of the constraints, restrict
the information in ? into the constraint space ?f. The bound can be found from
(the inverse of) the information in ?f, and is then projected back into the origi-
nal space of the parameters of interest. This down-and-up projection is easiest to
observe for linear constraints, where any local reparameterization is also a global
reparameterization.
Example 3.20. Assume the parameters satisfy the linear constraint
f(?) =F?+v = 0.
In this case, solutions of ? are of the form? =?FT parenleftbigFFTparenrightbig?1v+U?=g(?) where
11In Ash?s thesis and paper, this scenario was referred to as the minimally constrained case,
where only all the unknown information ambiguities are constrained. However, in this current
work, the minimal number of constraints is zero, the degenerate constraint case of example 3.13,
thus it makes more sense to refer to this scenario as a non-informative constraint.
48
U is defined as in (3.6). That this is a one-to-one correspondence locally (or in
this case globally) is made clear by the existence of the inverseg?1(?) =UT? =?.
Then, since G(?) =U, we have from (3.9)
CCRB(?) =UparenleftbigUTI(?)Uparenrightbig?1UT
which is exactly the CCRB provided in (3.5).
Alternatively, another interpretation [9] is that the CCRB is less than (in a
matrix sense) the CRB because the performance bound is over an expanded class of
estimators. The (non-biased) CRB is a bound on the mean-square error of unbiased
estimators in ?, whereas the CCRB is a bound for estimators that only need to be
unbiased on ?f and not on the whole set ?.
3.1.5 Derivation of g(?)
In the proof of the CCRB as well as its application, the need for an explicit
reparameterizationg? proved unnecessary. However, in examples 3.4 and 3.20 of the
previous section, we presented scenarios where, given the constraint function f, we
were able to define a locally equivalent continuously differentiable g?. There may
exist other scenarios where it is desirable to obtain a g? explicitly.
Inthissection, wepresenttwoproceduresto doso. First, we detailanapproach
using the Taylor expansion of g? given explicit knowledge of U(?prime). Then, we
demonstrate an approach based on fixed point methods.
49
3.1.5.1 A Taylor series derivation
The Taylor series expansion of ?prime =g?(?prime) about ? is given by
g?(?prime) =?+G?(?)(?prime??)+???
where ? =g?(?). For any differentiable function h : Rm?R we have
?
??primeT h(?
prime) = ?
??primeT h(?
prime)?G
?(?
prime).
Since the implicit function from Rm?k into ?f is not unique, we can choose a repa-
rameterization which uses the transformation matrix S(?) = Im?k, i.e., we choose
a reparameterization with Jacobian G?(?prime) = U(g?(?prime)) for any null space matrix
U(?) that satisfies (3.6) and choose ? = 0. With this selection of the reparam-
eterization, the rth order derivatives of g? are the (r?1)st order derivatives of
the elements of U(?prime) with respect to ?prime, which are to be evaluated at ? for the
coefficients in the Taylor series.
Example 3.21. Reviewing example 3.4, note
U(?prime) =
bracketleftbigg ?prime
2
??prime1
bracketrightbigg
=
bracketleftbigg 0 1
?1 0
bracketrightbiggbracketleftbigg?prime
1
?prime2
bracketrightbigg
and hence
?U(?prime)
??primeT =
bracketleftbigg 0 1
?1 0
bracketrightbigg
(3.22)
is independent of ?prime, so using (3.22) as a reference, we have
?rU(?prime)
??primer =
?r?1
??primer?1
parenleftbigg
?U(?prime)
??primeT ?U(?
prime)
parenrightbigg
=???=
parenleftbigg
?U(?prime)
??primeT
parenrightbiggr
U(?prime).
50
With a natural selection of ? = g?(0) = (1,0)T as an initial value for the Taylor
series, the reparameterized function is found to be
?prime =g(?prime) =
?summationdisplay
r=0
parenleftbigg
?U(?prime)
??primeT
parenrightbiggr
??
primer
r!
=
?summationdisplay
r=0
bracketleftbigg 0 1
?1 0
bracketrightbiggrbracketleftbigg1
0
bracketrightbigg?primer
r!
=
?
??
??
?
?summationdisplay
r=0
(?1)r ?
prime2r
(2r)!
?summationdisplay
r=0
(?1)r+1 ?
prime2r+1
(2r+ 1)!
?
??
??
?
=
bracketleftbigg cos(?prime)
?sin(?prime)
bracketrightbigg
.
This particular choice of ? produces a reparameterizationg? in agreement with the
one chosen in (3.8). Any alternative reparameterization can be found utilizing an
alternative transformation matrix S(?prime) or an alternative initialization.
Such an approach will only derive a local bijective map and convergence of the
Taylor series may not result in a known functional form for any given constraint f.
When U(?prime) is not known as a function of ?prime, numerical techniques are available to
find the derivatives of U(?prime) with respect to ?prime using the equationF(?prime)U(?prime) = 0.
3.1.5.2 A fixed point derivation
More commonly, a fixed point approach is takento the derivation of an implicit
function. As in appendix A.2, the parameter vector is partitioned ? =
bracketleftbigg?
1
?2
bracketrightbigg
and
the constraint function rewritten as f : Rm?k ?Rk ?Rk defined as fstar(?prime1,?prime2) =
f(
bracketleftbigg?prime
1
?prime2
bracketrightbigg
). If fstar?prime
2
(?1,?2) = ???primeT
2
fstar(?1,?prime2)
vextendsinglevextendsingle
vextendsingle
?prime2=?2
is nonsingular, then there exist a
51
unique continuous function ?2 = ?2(?1) about ?1 such that fstar(?1,?2(?1)) = 0.
This is merely a particular variation of the implicit function theorem. One proof of
this version proves the existence of a fixed point in the contraction map
d(?2(?prime1)) =?2(?prime1)?D?1(?1,?2)f(?prime1,?2(?prime1))
where D(?1,?2) =fstar?prime
2
(?1,?2). This motivates the use of the iteration
?(i+1)2 (?prime1) =?(i)2 (?prime1)?D?1(?1,?2)f(?prime1,?(i)2 (?prime1))
to generate the (fixed point) implicit function. The iteration is essentially an appli-
cation of Newton?s method [64].
Example 3.22. Reviewing example 3.4 again, we have the constraint fstar(?1,?2) =
?21 +?22?1 = 0 and wish to find a local reparameterization at (?1,?2) = (1,0). First
note,
fstar?1(?1,?2) = 2?1 = 2 , fstar?2(?1,?2) = 2?2 = 0,
and we only have a nonsingularity with respect to ?1. Hence, we cannot find a find
a function for ?2 in terms of ?1 using this approach at (1,0), but we can find ?prime1(?prime2)
there. Defining D(?1,?2) = fstar?1(?1,?2) = 2 and initializing with ?prime(1)1 (?prime2) = 1, the
fixed point method dictates the next iterate to be
?prime(2)1 (?prime2) = ?prime(1)1 ?D?1(?1,?2)
parenleftBig
(?prime(1)1 (?prime2))2 +?prime22 ?1
parenrightBig
= 1?12parenleftbig1+?prime22 ?1parenrightbig= 1?12?prime22 .
Continuing, the third iterate is
?prime(3)1 (?prime2) = 1?12?prime22 ?12
parenleftBigparenleftbig
1? 12?prime22 parenrightbig2 +?prime22 ?1
parenrightBig
= 1?12?prime22 ?18?prime42 ,
52
and the fourth iterate is
?prime(4)1 (?prime2) = 1?12?prime22 ?18?prime42 ?12
parenleftBigparenleftbig
1?12?prime22 ? 18?prime42 parenrightbig2 +?prime22 ?1
parenrightBig
= 1?12?prime22 ?18?prime42 ? 116?prime62 ? 1128?prime82 .
As expected, as r??then ?prime(r)1 (?prime2) approaches a limit function
?prime1(?prime2) =
radicalBig
1??prime22 = 1?12?prime22 ?18?prime42 ? 116?prime62 ? 5128?prime82 +O(?prime92 )
near ?2 = 0.
3.2 Identifiability
Identifiability conditions based on the CRB (or FIM) were detailed in Section
2.2, and the definitions of local and strong identifiability are given therein. In this
section, the identifiability of parameters under functional equality constraints is
considered.
3.2.1 Local identifiability
To establish a new identifiability criterion from the CCRB, we will first ex-
amine an existing criterion. Rothenberg [58, theorem 6] developed conditions for
identifiability of a parameter vector ? under the constraints f(?) = 0, which was
later partially re-derived by Crowder [18, lemma 1 is the ?only if? portion of the
theorem statement].
Theorem 3.23 (Rothenberg-Crowder). Assume both F(?prime) and
M(?prime) =
bracketleftbiggI(?prime)
F(?prime)
bracketrightbigg
53
have constant rank in a local neighborhood about ?. Then ? is locally identifiable
if and only if M(?) has full column rank m.
The proof of this theorem is based on the unconstrained proof of Theorem
2.5. An immediate implication is that if the FIM I(?) is regular, then not only is
the unconstrained model locally identifiable, but the constrained model is as well.
Otherwise, if the FIM is singular, then the constraints must be such that the row
vectors of its Jacobian F(?) eliminate the null space of the FIM. In doing so, the
constraints eliminate whatever inherent ambiguity was in the model that led to local
unidentifiability and a singular FIM (Theorem 2.5). If row vectors of the Jacobian
did not eliminate the null space of the Fisher information then there would exist a
linear combination of column vectors ofU(?) such thatUT(?)I(?)U(?) is singular,
as shall be shown shortly.
Additionally, regardless of the information, in the trivial case where the con-
straints are such that the Jacobian is full column rank, then the theorem says the
model is locally identifiable. Indeed, when rank(F(?)) = m the constraints com-
pletely determine the parameter (e.g., see example 3.14).
Rothenberg?s theorem for unconstrained identifiability is useful in establishing
an alternative criterion for identifiability in relation to a component of the CCRB.
Theorem 3.24. Let ???f and assume UT(?prime)I(?prime)U(?prime) has constant rank in a
neighborhood of ?. Then ? is locally identifiable if and only if UT(?)I(?)U(?) is
regular.
Proof. Let g? satisfy Theorem 3.3. If UT(?prime)I(?prime)U(?prime) has constant rank in a
54
local neighborhood about ? = g?(?). Then ?I(?prime) has constant rank in a local
neighborhood about ?. And since g? is injective, ? is locally identifiable in ?f if
and only if? is locally identifiable in Rm?k. By Theorem 2.5,? is locally identifiable
if and only if ?I(?) is regular. And ?I(?) is regular if and only if UT(?)I(?)U(?) is
regular.
This is the corresponding theorem to Rothenberg?s Theorem 2.5 and agrees
with his Theorem 3.23, although the proof does not rely on this latter theorem?s re-
sult because the implicit functiong? simplifies the approach. It is, however, possible
to prove a more general result connecting the rank of the M(?) matrix of Theorem
3.23 to the implicit Fisher information UT(?)I(?)U(?).
Theorem 3.25. Assume UT(?prime)I(?prime)U(?prime) has constant rank in a neighborhood
of ?. Then
nullity(M(?prime)) = nullity(UT(?prime)I(?prime)U(?prime)).
Proof. First, for some fixed ?prime, assume M(?prime) is not full column rank and the
vectors v1,...,vr are linearly independent and span the null space of M(?prime), i.e.,
for l = 1,...,r,
I(?prime)vl = 0
F(?prime)vl = 0.
LetU(?prime) be a matrix definedas in(3.6). SinceF(?prime)vl = 0 theneachvl =U(?prime)wl
for some w1,...,wr ?Rm?k. Now if
rsummationdisplay
l=1
?lwl = 0, then
rsummationdisplay
l=1
?lvl =
rsummationdisplay
l=1
U(?prime)?lwl =
0, which implies ?1 = ??? = ?r = 0 since the vi are linearly independent. Hence,
55
the w1,...,wr are also linearly independent and span an r-dimensional subspace
of Rm?k. Thus, I(?prime)U(?prime)wl = 0 for l = 1,...,r. This proves the implication,
namely, that nullity(UT(?prime)I(?prime)U(?prime))?nullity(M(?prime)).
To show the inverse, assume the vectors w1,...,wr are linearly independent
and span the null space of the UT(?prime)I(?prime)U(?prime). Define vl = U(?prime)wl for l =
1,...,r. Since I(?prime)vl = 0 and F(?prime)vl = 0, then M(?prime)v1 = 0 for l = 1,...,r.
Note for any?1,...,?r, then
rsummationdisplay
l=1
?lvl =
rsummationdisplay
l=1
U(?prime)?lwl = 0if and only if
rsummationdisplay
l=1
?lwl = 0
(sinceU(?prime) is full column rank) which is true if and only if ?l =???= ?r = 0. This
proves the converse, i.e., nullity(M(?prime))?nullity(UT(?prime)I(?prime)U(?prime)).
The theorem essentially states that in the local neighborhood where the im-
plicitfunctiongisdefined,M(?)has fullcolumnrank if and only ifUT(?)I(?)U(?)
does, for any ???f. As a consequence, we have the following corollary.
Corollary 3.26. If I(?) is regular, then UT(?)I(?)U(?) is also regular. And if ?
is locally identifiable in ?, then ? is locally identifiable in ?f.
Proof. If ? is locally identifiable, then nullity(I(?)) = 0 by theorem 2.5. And if
nullity(I(?)) = 0, then nullity(UT(?)I(?)U(?)) = 0 by theorem 3.25. Finally, if
nullity(UT(?)I(?)U(?)) = 0, then ? is locally identifiable in ?f by theorem 3.24.
56
3.2.1.1 Local identifiability in the Aitchison-Silvey-Crowder CCRB
formula
In subsection 3.1.2.2, an alternative form of the CCRB is presented where a
loaded Fisher information is used in place of the FIM to resolve issues of singularity
of the FIM. The Aitchison-Silvey loaded FIM isI(?)+FT(?)F(?) [4], whereas the
Crowder loaded FIM I(?) +FT(?)KF(?) [18], where K is chosen such that the
loaded FIM is full rank. The following theorem, which connects local identifiability
and the Aitchison-Silvey-Crowder CCRB, shows that under certain conditions, the
matrix K is unnecessary. This result is hinted at, but not clearly stated in [18,
lemma 6].
Theorem 3.27. The Aitchison-Silvey-Crowder loaded FIM I(?) +FT(?)F(?) is
nonsingular if and only if M(?) is full column rank. Hence, if I(?prime)+FT(?prime)F(?prime)
is constant locally about ?, then ? is identifiable if and only if I(?) +FT(?)F(?)
is nonsingular.
Proof. To show the contrapositive, assume M(?) is not full column rank and v
is a nontrivial vector such that M(?)v = 0. Then I(?)v = 0 and F(?)v = 0.
Therefore, v is in the null space of the Gram matrix I(?) +FT(?)F(?). To show
the inverse, assume parenleftbigI(?) +FT(?)F(?)parenrightbigv = 0 for some nontrivial vector v?Rm.
Then since
I(?) +FT(?)F(?) =bracketleftbigI1/2(?) FT(?)bracketrightbig?
bracketleftbiggI1/2(?)
F(?)
bracketrightbigg
,
we must have thatI1/2(?)v= 0 andF(?)v = 0. Hence,I(?)v =I1/2(?)I1/2(?)v=
0, which impliesM(?)v= 0 and M(?) is not full column rank.
57
3.2.2 Strong Identifiability
Restricting the discussion to normal distributions in this section we can extend
the equivalence criterion between regularity and strong identifiability as defined in
section 2.2.2. That is, assume x ? N(?(?),?(?)) with ?(?) ? Rp, with the
elements of the mean and variance explicitly defined by a map ? : ??Rq where
q?p+p(p+ 1)/2 and assume m?q.
Theorem 3.28. If ? is strongly identifiable on ? and f(?) = 0, then ? is also
strongly identifiable on ?f.
Proof. If ? is strongly identifiable, then there exists a representative mapping ??,
which is injective on ?. Since ?f ? ?, ?? is still injective and a representative
mapping on ?f.
This theorem is complementary to Corollary 3.26. Essentially, the imposition
of constraints does not take away existing identifiability (local or strong) or Fisher
information regularity that already exists in a model. However, it is not always the
case that the original (unconstrained) model is information regular or identifiable.
The following theorem, an extension of Theorem 2.6, connects the notion of strong
identifiability with regularity of UT(?)I(?)U(?).
Theorem 3.29. Assume ? is a holomorphic mapping of z??????A?? into Cq,
where ?f ???Cm and ?? is open in Cm for each ?. Then
(a) if UT(z)I(z)U(z) is regular, there exists a strongly identifiable open neigh-
borhood about z, and
58
(b) if there exists a representative mapping ??? : ?? ?Cq for each ?, then the
matrix UT(z)I(z)U(z) is regular for everyz??.
Proof. By the implicit function theorem (Theorem 3.3), then for any matrix U(?)
whose columns form a basis for the null space of the Jacobian of f(?), there exists
an open set Vowner?, an open set W?Rm?k, and some transformation g? : W?Rm
such that? =g?(?) for some??W. In particular, in this reduced parameter space,
there exists a FIM such that ?I(?) = UT(?)I(?)U(?). Since Theorem 2.6 applies
to ?I(?), the result for UT(?)I(?)U(?) is proven.
Regardless of the regularity of the FIM, only regularity of UT(?)I(?)U(?)
determines strict identifiability under constraints for normal distributions, given a
proper holomorphic function(s).
3.3 Linear Model
Assume the observations x model a linear function of the parameters ?, as in
x=H?+w, (3.23)
whereH is afull columnrankn?mobservation matrix consisting of known elements
and w is a random noise vector with mean zero and known variance C. As noted
in section 2.3.1, the (weighted) LSE
??LS(x) =parenleftbigHTC?1Hparenrightbig?1HTC?1x
59
is the BLUE. The LSE has a variance of Q?1 whereQ=parenleftbigHTC?1Hparenrightbig. In addition,
we assume a linear constraint
f(?) =F?+v = 0,
where F is a known full row rank k?m projection matrix and v is a known shift
vector. For a linear constraint, the Jacobian F(?) = F does not depend on the
parameter. As the linear problem is well-studied, many of the results in this section
are known (e.g., see [61, section 3.8] and [57, section 11.3.3]) but are presented here
from a different perspective.
3.3.1 Best Linear Unbiased Estimation
The constrained (weighted) LSE (CLSE) is most often given by [36, p. 252]
??CLS(x) = ??LS(x)?Q?1FT parenleftbigFQ?1FTparenrightbig?1parenleftBigF ??LS(x) +vparenrightBig. (3.24)
Simple calculation confirms that this CLSE exists in ker(f) = ?f (i.e., it satisfies
the constraint), is unbiased, and has variance
Var(??CLS(x)) = Q?1?Q?1FT parenleftbigFQ?1FTparenrightbig?1FQ?1. (3.25)
The BLUE property of the LSE is preserved for the CLSE. This is not a surprising
result since a linear constraint on a linear model is still a linear model, as seen in
figure 3.2.
In the context of the null space approach of the CCRB, an alternative CLSE
may be developed. One particular advantage of this approach is the avoidance of
60
H?
?x = H??LSE
x
?
H?f
H??CLSE
Figure 3.2: Projection of the observationsxonto the linear spaceH? and the linear
constraint space H?f.
the need for directly using and solving for Lagrange multipliers. Note that zero
solutions of f(?) are of the form
? =?FT parenleftbigFFTparenrightbig?1v+U? =g?(?)
whereU satisfies the equations in (3.6), i.e., the columns ofU form an orthonormal
basis for the null space of the row vectors of F, and ? ? Rm?k is a parameter
representing the projection of? to the constraint space ?f. Since the JacobianF is
independent of the parameter, thenU(?) =U and, hence,g? are as well. Moreover,
the typical local properties for the implicit function hold globally. Substituting this
solution for ? the linear model is reformulated
y =HU?+w (3.26)
where y = x+HFT parenleftbigFFTparenrightbig?1v. Following the least squares result for the CRB,
61
we desire a solution that minimizes a quadratic objective function
??LS = argmin
?
parenleftbigy?HU?parenrightbigT C?1parenleftbigy?HU?parenrightbig.
This solution must satisfy the normal equations given by
parenleftbigUTQUparenrightbig ??
LS(y) =UTHTC?1y.
Provided the normalizing matrix parenleftbigUTQUparenrightbig is full rank, the LSE of ? is given by
??LS(y) = parenleftbigUTQUparenrightbig?1UTHTC?1y
and is the BLUE ((2.4) in section 2.3.1). The corresponding LSE of ? based on this
null space approach is
??CLS(x) = ?FT parenleftbigFFTparenrightbig?1v+U??LS(y)
= ?FT parenleftbigFFTparenrightbig?1v+UparenleftbigUTQUparenrightbig?1UTHTC?1
parenleftBig
x+HFT parenleftbigFFTparenrightbig?1v
parenrightBig
= UparenleftbigUTQUparenrightbig?1UTQ
parenleftBig
?1 +FT parenleftbigFFTparenrightbig?1v
parenrightBig
?FT parenleftbigFFTparenrightbig?1v
+UparenleftbigUTQUparenrightbig?1UTHTC?1parenleftbigx?H?1parenrightbig
= ?1 +UparenleftbigUTQUparenrightbig?1UTHTC?1parenleftbigx?H?1parenrightbig (3.27)
where ?1 =?FT parenleftbigFFTparenrightbig?1v+U?1 can be any arbitrary point satisfying the linear
constraint (?1 is unrestricted). This alternative CLSE is more general than the prior
formula as it is applicable in scenarios whenH andF are not necessarily full column
rank and full row rank, respectively. As such, it has the more general expression for
its variance
UparenleftbigUTQUparenrightbig?1UT,
62
which is equivalent to (3.25) whenH andF are full column rank and full row rank,
respectively. Replacing the rank conditions on H and F, the weaker necessary
conditions for this CLSE are thatHU be full columnrank. If the stronger necessary
conditions on H and F exist, then the CLSE can be reformulated in terms of the
LSE as before, e.g.,
??CLS(x) =?1 +UparenleftbigUTQUparenrightbig?1UTQparenleftBig??LS(x)??1parenrightBig.
It is easy to confirm that ??CLS(x) ? ker(f) by recalling that FU = 0 and ?1 ?
ker(f).
Also, as in section 2.3.1, if HU is not full column rank, then for estimable
functions dT?, i.e., for vectors d in the column space of UTHT, the LSE is BLUE
and is given by
dT ??CLS(x) =dT?1 +dTUparenleftbigUTQUparenrightbig?UTHTC?1parenleftbigx?H?1parenrightbig (3.28)
similar to (3.27) with variance dTUparenleftbigUTQUparenrightbig?UTd.
3.3.2 Uniform Minimum Variance Estimation under Gaussian noise
Under the assumption that the noise is normally distributed, the (uncon-
strained) LSE is also the MLE. The FIM12 is I(?) = Q = parenleftbigHTC?1Hparenrightbig, and the
LSE/MLE is the MVUE being efficient with respect to CRB = I?1(?). Given this
general principle that for linear models with additive Gaussian noise, the LSE is
the MLE, then since a linear constraint is essentially a reduced dimensional linear
12Note that neither the Fisher information, the Jacobian of the constraints, nor the null space
matrix depend on the parameters in the linear model with linear constraints.
63
model as evidenced in (3.26), the CLSE should be the constrained MLE (CMLE).
As we shall see, this is indeed the case.
First, for the Gaussian linear model in (3.26), the pdf is
q(y;?) = 1(2pi)(m?k)/2(detC)(m?k)/2 exp
braceleftBig
?12parenleftbigy?HU?parenrightbigT C?1parenleftbigy?HU?parenrightbig
bracerightBig
.
The Fisher score ?s(y;?) = ?UTHTC?1parenleftbigy?HU?parenrightbig has a variance, or Fisher
information, of ?I(?) =UTHTC?1HU. Hence the CCRB is
CCRB(?) = G(?)?I?1(?)GT(?)
= UparenleftbigUTHTC?1HUparenrightbig?1UT
(see example 3.20).
Maximizing the likelihood (or pdf) is equivalent to minimizing the quadratic,
therefore the LSE of? is also the MLE of?. And by the invariance property [36, 14]
of the MLE, then the CMLE of ? is
??CML(x) =g(??ML(y)) = ??CLS(x).
Theorem 3.30. The CMLE is optimal for the linear model under linear constraints.
That is, if the observations obey the linear model in (3.23), where H is a known
matrix, ? is an unknown parameter vector subject to the linear constraint f(?) =
F?+v = 0, and w is a zero-mean normal random vector with known variance C,
then provided HU is full column rank, where U is defined by (3.6), the CMLE
??CML(x) =?1 +UparenleftbigUTHTC?1HUparenrightbig?1UTHTC?1parenleftbigx?H?1parenrightbig (3.29)
is unbiased and efficient.
64
While the formula for the CMLE based on the MLE might seem preferable,
analogous to the formula for the CLSE based on the LSE, i.e.,
??CML(x) =?1 +UparenleftbigUTQUparenrightbig?1UTQparenleftBig??ML(x)??1parenrightBig, (3.30)
with Q = HTC?1H, this formulation requires the existence of a full rank FIM
I(?) = Q in the MLE. The benefit of using the CMLE in (3.29) versus the CMLE
in (3.30) is that the following proof does not require this assumption.
Proof. First note for any ?,?1 satisfying the constraints
???1 = ?FT parenleftbigFFTparenrightbig?1v+U?+FT parenleftbigFFTparenrightbig?1v?U?1
= Uparenleftbig???1parenrightbig
for some ?,?1?Rm?k. Therefore, the expected value of the CMLE is
E? ??CML(x) = ?1 +UparenleftbigUTQUparenrightbig?1UTQparenleftbig???1parenrightbig
= ?1 +UparenleftbigUTQUparenrightbig?1UTQUparenleftbig???1parenrightbig
= ?1 +Uparenleftbig???1parenrightbig
= ?1 +???1
= ?.
65
Finally, the variance of the CMLE is
Var?(??CML(x)) = Var?
parenleftBig
?1 +UparenleftbigUTQUparenrightbig?1UTHTC?1parenleftbigx?H?1parenrightbig
parenrightBig
= Var?
parenleftBig
UparenleftbigUTQUparenrightbig?1UTHTC?1parenleftbigx?H?+H??H?1parenrightbig
parenrightBig
= Var?
parenleftBig
UparenleftbigUTQUparenrightbig?1UTHTC?1parenleftbigx?H?parenrightbig
parenrightBig
= UparenleftbigUTQUparenrightbig?1UTHTC?1CC?1HUparenleftbigUTQUparenrightbig?1UT
= UparenleftbigUTQUparenrightbig?1UTQUparenleftbigUTQUparenrightbig?1UT
= UparenleftbigUTQUparenrightbig?1UT,
i.e., the CCRB of ?.
Thus, the CMLE is the MVUE for the linear model with linear constraints
under a Gaussian assumption.
Additionally, whenHU is not full column rank, then whendis in the column
space of UTHT, the MLE of dT? is still the MVUE and is given by dT ??CML(x) =
dT ??CLS(x) from (3.28), with variance dTCCRB(?)d.
3.4 Constrained Maximum Likelihood Estimation
The constrained MLE (CMLE) of the parameter vector ? constrained to the
manifold ?f = {?prime : f(?prime) = 0} is the estimator in ?f, which given the observa-
tions x, that maximizes the likelihood distribution p(x;?), i.e. it is the maximum
likelihood in ?f. Since log(?) is concave, it is convenient to equivalently maximize
the log-likelihood logp(x;?) since then the Jacobian of the objective is the Fisher
score. In an optimization context, the CMLE, which will be denoted ??CML(x), is
66
the solution to the following constrained optimization problem:
max
?prime
logp(x;?prime)
s.t. f(?prime) = 0.
(3.31)
Analogous to the method of maximum likelihoodapproach of (2.7), solutions ??(x) =
g??(x)( ??(x)) satisfying
?
??prime logp(x,g?prime(?
prime))
vextendsinglevextendsingle
vextendsinglevextendsingle
?prime= ??(x)
= 0,
whereg?prime is defined by (3.7), are candidates to be the CMLE. More formally, a solu-
tion to this optimization problem must satisfy the Karush-Kuhn-Tucker conditions
[45], i.e.,
s(x;?prime)??TF(?prime) = 0 (stationarity) (3.32)
f(?prime) = 0. (feasibility) (3.33)
Any point satisfying these conditions is a stationary, feasible point.
Since ??CML(x) ??f, then the implicit function theorem implies there exists
an open set O??f containing ??CML(x), an open set P?Rm?k, and a continuously
differentiable bijectiong??(x) : P?O such that ??(x) =g??(x)(??(x)) for some ??(x)?
P. If ??(x) is a maximizer of the likelihood function p(x;?) in the constraint set ?f,
then the likelihood q(x;?prime) = p(x;g??(?prime)) has a maximum at ??(x) in P (i.e., a local
maximum at ??(x) inRm?k). It cannot be said thatq(x;?) has a global maximum at
??(x), since ?prime = g??(x)(?prime) is only guaranteed to exist in ?f when ?prime ?P, i.e., there
may exist a point ?prime ?Rm?k\P such that q(x;?prime) >q(x; ??(x)) and g??(x)(?prime)negationslash??f.
67
3.4.1 Efficient estimation
In Section 2.4.1, it was explained that when an efficient estimator exists, the
method of maximum likelihood finds the estimator [36, 62]. It is useful to note the
connection between efficiency and the method of constrained maximum likelihood
since Stoica and Ng ignored this extension in their paper [68], despite Marzetta
having showed that this result extends to the constrained case when the FIM is
non-singular [47, theorem 3]. What follows is the general extension of this result,
including the case for singular FIMs.
Theorem 3.31. If t(x) is a constrained estimator of ?, required to satisfy the
constraint f(t(x)) = 0, which is also efficient with respect to the CCRB, then the
estimator is a stationary point for the constrained optimization problem in (3.31).
Proof. This is perhaps more easily proven strictly from the constrained parameter
perspective, since the global maximum of the likelihood relative to the implicit
reparameterization may not correspond to global maximum in ?f relative to the
constrained parameterization. Since t(x) is efficient then in the mean-square sense
we have t(x)?? = CCRB(?)s(x;?) as a function of ?. Then as ??t(x) (this
assumes the observations are consistent with ?) we have
0?t(x)?? = CCRB(?)s(x;?).
The continuity of the CCRB and the Fisher score impliess(x;t(x)) =FT(t(x))??
for some ??Rk or
s(x;t(x))??TF(t(x)) = 0,
68
which defines the stationarity condition (3.32) of the constrained optimization prob-
lem with ? being the vector of Lagrange multipliers.
3.4.2 Asymptotic Normality
The asymptotic properties of the MLE can be found in section 2.4.2. Therein,
it was mentioned that the maximum likelihood estimator was asymptotically un-
biased and efficient with variance asymptotically equivalent to the CRB. A corre-
sponding relationship exists between the CMLE and the CCRB. As before, let the
samples x1,x2,...,xn, be iid as x from the likelihood p(x;?), where ? is assumed
to exist in ?f. Denote yn = (x1,x2,...,xn) to be the collection of these samples,
so that the likelihood will be p(yn;?) =
nproductdisplay
i=1
p(xi;?). Hence, the asymptotic CMLE
will be denoted ??(yn).
Theorem 3.32. Assuming the pdf satisfies the regularity conditions (see (3.19) and
discussion after proof), then the CMLE is asymptotically distributed according to
?nparenleftBig??(y
n)??
parenrightBig d
?Nparenleftbig0,CCRB(?)parenrightbig.
There exists a number of results in the literature regarding the asymptotic
characteristics of the CMLE (e.g., the works of Aitchison and Silvey [3, 63, 4, 2],
and of Crowder [18]). For example, Crowder shows that
?nparenleftBig??(y
n)??
parenrightBig d
?
N
parenleftBig
0,D?1(?)?D?1(?)FT(?)parenleftbigF(?)D?1(?)FT(?)parenrightbig?1F(?)D?1(?)
parenrightBig
where D(?) = I(?) +FT(?)KF(?) for any positive semi-definite matrix K that
ensures the nonsingularity of D(?). And while it would be sufficient to use these
69
existing results to verify the connection between the CMLE and the CCRB, it is
also insightful (and the point of this treatise) to examine the problem entirely from
the perspective of the reduced parameter space, i.e., using the implicit function or
a null space approach.13
Proof. By the implicit function theorem (Theorem 3.3), there exists an open set
O ? ?f containing ?, an open set P ? Rm?k, and a continuously differentiable
bijection g? : P?O such that ? = g?(?) for some ??P. The likelihood for ? is
given by q(yn;?) = p(yn;g?(?)).
Let ??(yn) be the MLE of ? based on the likelihood q(yn;?). Since the MLE
is consistent and asymptotically efficient, then
?nparenleftBig??(y
n)??
parenrightBig d
?Nparenleftbig0, ?I?1(?)parenrightbig. (3.34)
In particular, since ??(yn)?? as n??, then for n sufficiently large, say n >N,
??(yn)?P. Let ??(yn) be the CMLE of ? based on the likelihood p(yn;?) and the
constraint f(?) = 0. By the invariance property [36, 62], for n > N, ??(yn) =
g?(??(yn)) and ??(yn)?O. Therefore, since ??(yn)?? as n??, ??(yn)?? also
and the CMLE is consistent.
The Taylor seriesexpansion (see section 3.1.5) ofg?(?prime) can be truncated using
a Lagrange remainder term [38, p. 232] as in
g?(??(yn)) =g?(?) +G?(?, ??(yn))
parenleftBig
??(yn)??
parenrightBig
where G?(?, ??(yn)) is meant to represent a matrix of the form ofG?(?) where each
13Nevertheless, a proof directly from Crowder?s asymptotic normality result is detailed in ap-
pendix A.3.
70
row is evaluated at possibly different points ?prime(i), i= 1,...,m?k, each existing on
the line segment starting at ? and ending with ??(yn). From the invariance property
of the MLE, this can be rewritten as
??(yn)?? =G?(?, ??(yn))parenleftBig??(yn)??parenrightBig.
Since the MLE ??(yn) is consistent, then G?(?, ??(yn)) p?G?(?). Given this and
(3.34), then by Slutsky?s theorem [62, p. 60]
?nparenleftBig??(y
n)??
parenrightBig d
?Nparenleftbig0,G?(?)?I?1(?)GT?(?)parenrightbig,
which by theorem 3.5 shows ??(yn) is asymptotically efficient with respect to the
CCRB.
The conditions for asymptotic normality with respect to the CCRB are the
conditions that ??(yn) be asymptotically normal [14, p. 516]. For the MLE these
include (a) differentiabilityof the Fisher score, (b) the Fisher information continuous
with respect to the parameter and nonzero at?, and (c) consistency. For the CMLE
and theorem 3.32, this translates to (a) differentiability of the Fisher score and the
existence of first and second derivatives of any implicit function (or equivalently, the
constraint f), (b) UT(?prime)I(?prime)U(?prime) continuous with respect to ?prime and regular at
?prime =?, and (c) consistency of the CMLE.
3.4.3 The Method of Scoring Under Parametric Constraints
The methodof scoringfor unconstrained parametersisdetailedinsection2.4.3.
Here, we examine scoring with constraints. Assume we have an iterate ??(k)??f and
71
we wish to improve this iterate in the sense of the optimization problem expressed
in (3.31). The method of scoring does not directly apply, since any projection step
will not take into account the constraint, i.e., it is likely the direction of steepest
ascent is not the appropriate path in terms of maximizing the likelihood subject to
the functional equality constraints. Thus, it is desirable to have projected direction
and restoration steps that take the constraints into consideration.
Given an initial estimate ??(k), there exists a set Oowner ??(1) open in ?f, a set P
open in Rm?k, and a continuously differentiable function g??(k) : Rm?k ?Rm such
that g??(k) is a diffeomorphism on P, g??(k)(P) = O, and in particular there exists a
??(k) ? P such that g??(k)( ??(k)) = ??(k). Scoring can now be applied in the reduced
parameter space of Rm?k.
For the given set of observationsxand this corresponding initial estimate ??(k),
the method of scoring suggests the projection step
??(k+1) = ??(k) + ?I?1( ??(k))?s(x; ??(k))
to generate a better estimate ??(k+1) in the sense of maximizingthe likelihoodq(x;?prime)
= p(x;g??(k)(?prime)). As with many iterativeprocedures, convergence is only guaranteed
under certaininitialconditions. If the projectionstepor shift istoo large, then ??(k+1)
may not be a usable point, i.e., an iterate that increases the value of the likelihood
function. To add stability to the procedure, often a step-size rule or shift-cutting
is employed. This amounts to the inclusion of a multiplicative factor ?(k) ?[0,1],
modifying the projection step to
??(k+1) = ??(k) +?(k) ?I?1( ??(k))?s(x; ??(k)).
72
Choosing an appropriate step-size rule for ?(k) will guarantee convergence, although
typically at a cost to the rate of convergence.
The Taylor series expansion (see section3.1.5) ofg??(k) about ??(k) and evaluated
at ??(k+1) is given by
g??(k)( ??(k+1)) =g??(k)( ??(k)) +G??(k)( ??(k))?( ??(k+1)? ??(k)) +o(||??(k+1)? ??(k)||)
where o(||??(k+1)? ??(k)||) is a term that shrinks faster than ||??(k+1)? ??(k)|| as k?
?. Ignoring this error term, this generates an iteration in the larger dimensional
parameter space ??Rm by defining the next iterate ??(k+1) = g??(k)( ??(k+1)). That
is,
??(k+1) = g??(k)( ??(k)) +G??(k)( ??(k))?( ??(k+1)? ??(k))
= ??(k) +?(k)G??(k)( ??(k))?I?1( ??(k))?s(x; ??(k))
= ??(k) +?(k)G??(k)( ??(k))parenleftbigGT??(k)( ??(k))I( ??(k))G??(k)( ??(k))parenrightbig?1GT??(k)( ??(k))s(x; ??(k))
= ??(k) +?(k)U( ??(k))parenleftbigUT( ??(k))I( ??(k))U( ??(k))parenrightbig?1UT( ??(k))s(x; ??(k)).
In comparison with the classical method of scoring, this iteration
??(k+1) = ??(k) +?(k)CCRB( ??(k))s(x; ??(k)) (3.35)
is essentially a replacement of the CRB with the CCRB. This is the projection step
of the method of scoring with parametric equality constraints.14 Intuitively, this
should seem appropriate since the CCRB is a generalization of the CRB. However,
14Osborne [55] used a Lagrangian multiplier approach to develop the method of scoring. But his
scenario wasrestricted to linearconstraints and, hence, lacked the restoration step. Additionally,he
makes no mention that the matrix projecting the negative Jacobian of the objective is a constrained
Cram?er-Rao bound. Note the structure of this projection step is well-known as a nonstatistical
formulation exists for the conventional optimization problem in [25, p. 178].
73
even with an appropriate step-size rule to generate usable iterates, since there is no
certainty that ??(k+1) ?P, then it is likely that ??(k+1) will not be a feasible point.
To correct this, an encompassing restoration step is required to produce the next
iterate, i.e.,
??(k+1) =pibracketleftbig??(k) +?(k)CCRB( ??(k))s(x; ??(k))bracketrightbig (3.36)
where pi[?] is the natural projection of Rm onto ?f. This is the method of scoring
with parametric equality constraints. With this additional restoration step, the
constraint
projection
level surfaces in the
 constraint space
?(5)
?(3)
?(2)
?(4)
?(1)
linear gradient projection
?(k+1) = pi[?(k) + ?(k) CCRB(?(k)) J(x;?(k))]
Figure 3.3: Path created by iterates from the method of scoring with constraints.
usability of the iterate would be tested and accepted or rejected after (3.36) as
opposed to after (3.35). For a convex set, the natural projection is well-defined. In
general, though, some other rule will likely need to be applied in cases for which
there does not exist a unique shortest distance to ?f, e.g., reducing the step size
74
?(k). Simple projections, e.g., onto planar or spherical constraints, are relatively
simple operations, however, it might be more commonlythe case that the restoration
cannot be expressed analytically. To ensure the iterates satisfy the constraints
approximately, one approach is to apply an additional iterative process [25]
??(k,l+1) = ??(k,l)?FT( ??(k,l))parenleftbigF( ??(k,l))FT( ??(k,l))parenrightbig?1f( ??(k,l)),
where ??(k,1) = ??(k) and ??(k+1) = pibracketleftbig??(k)bracketrightbig = ??(k,l) when f( ??(k,l)) ? 0 to a desired
degree of accuracy and provided the iterate is still usable. Alternatively, a penalty
can be added to the cost (objective) function, e.g., as in
logp(x;?prime) +?
ksummationdisplay
i=1
|fi(?prime)|
for some positive?and wherefi istheithconstraint equation, to limitthe divergence
of the iterations away from ?f.
3.4.3.1 Convergence Properties
There is a large class of conditions that guarantee convergence in fixed point
theorems, some of which can be found in [64, 25, 10]. The most general statement
is that given an initialization ?sufficiently close? to the maximum value ??(x), the
sequence {??(k)} generated by the algorithm will converge to this CMLE. As the
method of scoring with parametric equality constraints is a Newton-type method,
convergence properties that already exist for these methods can be adapted here.
As it is impossible to cover all the potential approaches to developing properties for
this constrained scoring algorithm, this section focuses on properties similar to those
75
found in Goldstein [22]. To ease reading of this section, the proofs of the theorems
in this section are presented in appendix B.
First, define ???(k) ={?prime ??f : p(x;?prime)?p(x; ??(k))}as the set of all feasible
and usable iterates after the kth iterate ??(k). The step rule for the properties in this
section is as follows: for a fixed??(0,1) choose the least positive integerm(k) such
that ?(k) =?m(k) satisfies the inequality
?(k)parenleftbiglogp(x; ??(k+1))?logp(x; ??(k))parenrightbig??
vextenddoublevextenddouble
vextenddouble??(k+1)? ??(k)
vextenddoublevextenddouble
vextenddouble
2
I( ??(k))
(3.37)
where ??(k+1) is defined by (3.36). If no finite m(k) exists, then choose ?(k) = 0. This
type of step-size rule enforces a stepwise Lipschitz condition. For theorems 3.33 and
3.37, we require that ?f be convex.15
Theorem 3.33. If for any iterate ??(k) ??f there does not exist an ?(k) > 0 that
satisfies (3.37), then ??(k) is a stationary point.
Therefore, when the step rule forces the choice of ?(k) = 0 then the method
of scoring with parametric equality constraints has converged. The next theorem
details a property on the sequence of likelihood functions generated by the iterates.
Theorem 3.34. The sequence{p(x; ??(k))}is a monotone increasing sequence. Fur-
thermore, if p(x;?) is bounded above, then{p(x; ??(k))}converges.
15For any nonlinear equality constraint, ?f will not be convex. However, locally, the restoration
of the linear projection onto the tangent space of the CCRB has the appearance of the restoration
onto a convex set forsufficiently small?(k), i.e., ?f appears locallyconvex. Whileitmay be possible
to restrict the step size to this local convexity, such an enhancement is beyond the scope of the
work presented here. For inequality constraints, ?f may be convex; although such constraints, as
in example 3.15, will not inform the projection update in (3.35), they might inform the restoration
update in (3.36).
76
Thus, ???(k) is a decreasing sequence of closed (nested) sets, i.e., ???(k+1) ????(k)
or for any given sequence, ??(i) ????(j) provided i?j. That is, using a proper step
size rule will guarantee usable iterates. The monotonicity of {p(x; ??(k))}, even if
bounded above, does not imply monotonicity in the sequence {logp(x; ??(k+1))?
logp(x; ??(k))}. However, this does guarantee convergence.
Theorem 3.35. If the likelihood p(x;?) is bounded above, then the sequence
{logp(x; ??(k+1))?logp(x; ??(k))}
vanishes.
Hence, a bounded likelihood function guarantees the existence of a maximum
likelihood solution(s). This can also be shown by the nested interval theorem [38,
problem 2-1-12]. These previous properties would be a consequence of any rule that
chooses feasible, usable iterates. The value of the rule in (3.37) is that it allows for
statements to be made on the sequence{??(k)}.
Theorem 3.36. If the likelihood p(x;?) is bounded above, then the sequence
{bardbl??(k+1)? ??(k)bardblI(??(k))}
vanishes as k??.
This theorem does not guarantee that the sequence {??(k)} will converge or
even have a limit point.16 (The series
ksummationdisplay
j=1
1
j is an example of a sequence satisfying
the above theorem with no real-valued limit points.) Although, if the set ???(k) is
16A point a is a limit point of a sequence {an} if for any integer K and any epsilon1 > 0, there exists
an k > K such that|ak?a|< epsilon1.
77
bounded for somek, then the Bolzano-Weierstrass theorem [38, p. 52, theorem 2-12]
implies the existence of a limit point of the sequence.
Theorem 3.37. If ???(1) is compact and convex, then limit points of the sequence
{??(k)}are also stationary points.
The theorem remains true if ???(k) is compact and convex for any k.
Theorem 3.38. If ???(1) is compact for all sequences in a closed set of ?f and if
there is a unique limit point ?star for all such sequences then lim
k??
??(k) = ?star for every
sequence{??(k)}. Also, ?star is the maximum of p(x;?).
3.4.3.2 Linear constraints
Linear constraints on the parameter typically restrict the parameter to a set
of the form ?f = {?prime ? ? : F?prime +v = 0} (see section 3.3). Under this linear
constraint, the restoration operationpi[?] is redundant since any step remains in the
constraint space, i.e., since ??(k)??f and F?CCRB( ??(k)) = 0. Thus the method of
scoring with parametric equality constraints in (3.36) simplifies to the iteration
??(k+1) = ??(k) +?(k)CCRB( ??(k))s(x; ??(k)).
Example 3.39 (Linear model with linear constraints). Inthe linear model withnor-
mal noise case of section3.3.2, we havex=H?+wwithw?N(0,C). Inthiscase,
the negative Hessian is the FIM and the optimization problem becomes a null space
quadratic exercise [21], i.e., the minimization of a quadratic objective subject to a
linear constraint. The CCRB =UparenleftbigUTHTC?1HUparenrightbig?1UT, which is constant with
78
respect to the parameter; and the Fisher score is s(x;?prime) = HTC?1parenleftbigx?H?primeparenrightbig.
Hence, if ??(1) is any feasible vector, e.g., ??(1) =?FT parenleftbigFFTparenrightbig?1v, then the method
of scoring with constraints finds the CMLE in one step to be
??(2) = ??(1) + CCRB?HTC?1parenleftbigx?H ??(1)parenrightbig
which is exactly the formula in (3.29) from theorem 3.30. The next iterate ??(3) =
??(2) +CCRB?HTC?1parenleftbigx?H ??(2)parenrightbigreveals that the procedure reaches a fixed point
since
CCRBHTC?1parenleftbigx?H ??(2)parenrightbig
= CCRBHTC?1
parenleftBig
x?Hparenleftbig??(1) + CCRBHTC?1parenleftbigx?H ??(1)parenrightbigparenrightbig
parenrightBig
= CCRBHTC?1parenleftbigx?H ??(1)parenrightbig?CCRBHTC?1HCCRBHTC?1parenleftbigx?H ??(1)parenrightbig
= 0
Therefore, ??CML(x) = ??(2).
Example 3.40 (Jamshidian?sGP algorithm). Jamshidian[33] developedaGradient
Projection (GP) algorithm for maximizingthe likelihoodsubject to linear parameter
constraints using the iteration
??(k+1) = ??(k) +?(k)parenleftBigW?1?W?1FT parenleftbigFW?1FTparenrightbig?1FW?1parenrightBigs(x; ??(k)) (3.38)
for some positive definite matrix W. An optimal choice with regard to the algo-
rithm?s rate of convergence, Jamshidian suggests, is a possibly diagonally loaded
Hessian of the log-likelihood
W(x, ??(k)) = ???Ts(x; ??(k)) +?(k)Im?m,
79
where ?(k) ? 0 is chosen to be sufficiently large enough to ensure the positive
definiteness of the matrix W. This formulation is closely connected to the method
of scoring with constraints. The GP iteration is equivalent to scoring by choosing
W(x, ??(k)) to be the FIM, when it is nonsingular. Indeed, the projecting matrix
is similar to the Marzetta form of the CCRB in [47] with W(x, ??(k)) replacing the
FIM. This fact produces a slight generalization of the GP iteration, given by
??(k+1) = ??(k) +?(k)UparenleftbigUTW(x, ??(k))Uparenrightbig?1UTs(x; ??(k))
whereU is defined as in (3.6). In this formulation, the projecting metricW(x, ??(k))
only needs to be positive semidefinite. Alternatively, in (3.38) the Aitchison and
Silvey [4] substitution for the FIM, I( ??(k)) +FT( ??(k))KF( ??(k)), instead of a di-
agonally loaded Hessian of the log-likelihood (or even a diagonally loaded Fisher
information matrix).
In this sense, the two iterations are equivalent for the linear model, when the
FIM is simply the negative Hessian. This occurs when the log-likelihood is quadratic
(normal). This also suggests the adaption of Jamshidian?s GP algorithm to cases of
nonlinear constraints. Likewise, asymptotically, where yn is denoted as in section
3.4.2, then by the law of large numbers the Jamshidian projection matrix
W(yn, ??(k))?nI( ??(k)) +?(k)Im?m
for an arbitrarily small (possibly zero) ?(k) > 0, as I( ??(k)) is positive semidefinite.
The projection step update then becomes
??(k+1) = ??(k) +?(k)CCRB( ??(k))s(yn; ??(k))??(k)?(k)s(yn; ??(k)),
80
i.e., essentially equivalent to the method of scoring with parametric equality con-
straints in (3.35).
3.5 Hypothesis testing
In section 2.5, hypothesis testing (the Rao and Wald tests) using the CRB
was reviewed. In this section, hypothesis testing is considered under a constrained
alternative. Assume h : Rm ?Rr is a consistent and nonredundant differentiable
function, which is also consistent and nonredundant with the differentiable function
f : Rm?Rk. Hence, ?h ={?prime :h(?prime) = 0}??f, where ?f is defined as in (3.4),
and also rank(
bracketleftbiggH(?)
F(?)
bracketrightbigg
) =r+k?m and r<m?k. Then the hypothesis test can
be stated as
H0 :h(?) = 0 vs. H1 :f(?) = 0. (3.39)
Naturally, f(?prime) = 0 under these conditions defines an implicit function locally, so
assume g?prime : Rm?k ?Rk is such a function satisfying theorem 3.3 for any ?prime ??f.
Then, a locally (or asymptotically) equivalent hypothesis can be stated as
H0 :h(g?(?)) = 0 vs. H1 :h(g?(?))negationslash= 0. (3.40)
In this formulation, the well-known Rao and Wald statistics were shown in section
2.5.
81
3.5.1 The Rao statistic
For the hypothesis testing scenario in (3.40), the Rao test statistic presented
in section 2.5.1 is given by
?(yn) = ?sT(yn; ??h(g)(yn))?I?1n (??h(g)(yn))?s(yn; ??h(g)(yn))
where ?s(yn; ??h(g)(yn)) is the Fisher score of the observations yn (as defined in sec-
tion3.4.2) and evaluatedat theH0-constrained maximumlikelihoodestimate or con-
strained root of the likelihoodestimate (CRLE) ??h(g)(yn) of the likelihoodq(yn;?) =
p(yn;g?(?)), and ?I?1n (??h(g)(yn)) is the n-sample Fisher information evaluated at the
CMLE. In this context, the CMLE is the solution to the optimization problem
max
?prime
logq(x;?prime) s.t. h?g?prime(?prime) = 0. As in theorem 3.5, ?s(yn;?prime) =G?prime(?prime)s(yn;?prime)
and ?In(?prime) = n GT(?prime)I(g?prime(?prime))G(?prime). Also, recall that for sufficiently large n,
??h(yn) = g?(??h(g)(yn)). Therefore, the locally (or asymptotically) equivalent Rao
test statistic for the hypothesis in (3.39) is
?(yn) = 1nsT(yn; ??h(yn))CCRB(??h(yn))s(yn; ??h(yn)), (3.41)
which is analogous to (2.9) with the CCRB replacing the CRB. Under H0, ?(yn)
is still asymptotically ?2r in distribution. The corresponding Lagrange-multiplier
variant of this statistic is given by
?(yn) = 1n??h(yn)H(??h(yn))CCRB(??h(yn))HT(??h(yn))??h(yn), (3.42)
where the Lagrange multiplier estimates ??h(yn) are based on the first order condi-
tions relating to the constraint h (not h and f).
82
The result in (3.41) is consistent with the classical results in [63], although not
explicitly in this form. For the hypothesis scenario in (3.39) the Lagrange multiplier
statistic should be ??ThR?1?? ??h, where Rh,?? is defined by
?
?
P? ? ?
? ? ?
? ? Rh,?
?
?=
?
?
I(?) +FT(?)F(?) ?FT(?) HT(?)
F(?) 0 0
H(?) 0 0
?
?
?1
,
which is a variant of what appears in [63, equation (6.5)]. Finding the inverse using
the Schur complement, it is clear that
Rh,? =HT(?)
bracketleftBig
D?1(?)?D?1(?)FT(?)parenleftbigF(?)D?1(?)FT(?)parenrightbig?1F(?)D?1(?)
bracketrightBig
H(?)
with D(?) = I(?) +FT(?)F(?). Recognizing the inner matrix as the Aitchison-
Silvey-Crowder variant of the CCRB formula in (3.14) and substituting the CMLE
for the parameter obtains (3.42).
3.5.2 The Wald statistic
Similarly, the Wald test statistic presented in section 2.5.2 is
hT(g(??(yn)))
parenleftBig
H(g(??(yn)))GT(??(yn))?I?1n (??(yn))G(??(yn))HT(g(??(yn)))
parenrightBig?1
h(g(??(yn)))
for the testing problem in (3.40), where g is localized about ?. Following the steps
in section 3.5.1, then to test (3.39), the corresponding Wald test statistic is
?(yn) = nhT(??(yn))
parenleftBig
H(??(yn))CCRB(??(yn))HT(??(yn))
parenrightBig?1
h(??(yn)).
As with the Rao statistic, this general Wald statistic replaces the CRB in (2.10)
with the CCRB, and under H0, ?(yn) is asymptotically ?2r in distribution.
83
Thisagreeswiththe classicalresultin[2, ??21(?)?on p. 240] wherethe Gorman-
Hero-Aitchison-Silvey variant of the CCRB formula in (3.12) is used instead. (The
general scenario when the FIM is singular is discussed in [2, section 3.9].)
A requirement for the existence of this statistic is that H(?)CCRB(?)HT(?)
be regular. This is not an additional requirement, but a necessity in testing that
the hypothesis testing function itself be identifiable for the hypothesis to be valid.
3.6 Discussion
The previous sections have established that the theory of the constrained CRB
is equivalent to that of the CRB. The majority of the proofs, besides their novelty in
the recent literature on the CCRB, essentially rely on the generation of an implicit
function that translates the constrained parametric problem into an unconstrained
parametric problem, for which the theory of the CRB is well-established.
This theory has been extended to the CCRB, in particular, for identifiabil-
ity under constraints; for the linear model with linear constraints (already well-
documented in the literature but perhaps not with reference to this CCRB); for the
constrained maximum likelihood problem, its asymptotic normality and the method
of scoring; as well as for the Wald and Rao hypothesis tests. This list is by no means
exhaustive. There exists recent research, for example, related to biased estimation
with the CCRB [9], and an extension of the CCRB for complex-valued parameters
[32]. In addition, there are other areas in mathematical statistics that, as far as I
know, have not yet been connected to the CCRB, including a geometric interpreta-
84
tion of the CCRB (possibly in the manner of [60] or and extension of [5]), biasedness
issues with constrained estimation, confidence intervals or sets, and the plausibility
of a Bayesian version of the CCRB.
While the primary contributions of this thesis are the theoretical results and
their proofs existing in this chapter, this should not discount the practical applica-
tions of these ideas. From the practitioners viewpoint, this research has produced a
number of useful tools. For example, to test local identifiability, in addition to the
classic result of Rothenberg (theorem 3.23), theorem 3.24 may be used. Likewise,
for strict identifiability, theorem 3.29 is useful. And the method of scoring in (3.36)
adds to the list of constrained maximum likelihood methods. These and others will
be applied in a communications context, in the next chapter.
85
Chapter 4
Applications of the CCRB in Communications Models
Communications models in statistical signal processing often have the struc-
ture
y(n) = ?(n,?n) +w(n) , n = 0,...,N, (4.1)
wherey(n) are the receivedobservations of some model?(n,?n) affected (additively)
by noise w(n) over a series of time samples n. The parameters may or may not
be dependent on the number of time samples and the noise may or may not be
independent (or even normally distributed). This general model encompasses a
number of areas of signal processing, e.g., communications, sonar, radar, speech,
imaging, control, sensing, networks, etc. In every one of these areas, there exist
models where parametric equality constraints are of interest to practitioners in the
field. The CCRB has proven useful as a performance analysis tool in localization [6],
watermarking security [15], tomography [28], source bearing and symbol estimation
[59], space-time block coding [42], and even a variant of least squares [46].
In this chapter, we shall detail just two rather general signal processing com-
munications models: the convolutive channel and the calibrated array, each with
unknown signal and channel components. Because the source and channel interact
multiplicatively in the models, numerous variations of these two basic models are
possible (and necessary). We shall formulate constraints for just a few of these vari-
86
ations and connect the results of this chapter to the theory developed in Chapter
3. For the sake of coherence in the presentation, the lengthier proofs of theorems in
this chapter are relegated to appendix C.
In section 4.1, the convolutive mixture model with deterministic parameters is
presentedusing avarietyof descriptions. Inaddition, severalusefultermsare defined
in section 4.1.1.2 that are useful to characterize conditions on the a notion of near-
identifiability presented in section 4.1.2 and conditions on the Fisher information
derived in 4.1.3. The corresponding complex-valued FIM (CFIM) is defined in
4.1.3.2 and important properties are given in 4.1.3.3. These properties are critical to
understanding the particular inherent ambiguities in the basic convolutive mixture
model and determining the class of constraints are necessary to obtain a CCRB, as
discussed in 4.1.4.1. Two constraint models, the norm constraint (in 4.1.4.2) and the
semiblind constraint (in 4.1.4.3) serve as further validation of the CCRB method
by obtaining prior results in the literature; while another constraint model, the
combination of the semiblind and unit modulus constraints in 4.1.4.4, demonstrates
an important constraint model that did not previously exist in the literature.
In section 4.2, a special case of the convolutive mixture model, called the cal-
ibrated array model, is considered. The FIM for this (sub)model and its properties
are detailed in section 4.2.1 and various constraints are considered in section 4.2.2.
As before, a number of the constraint models are validations of the CCRB method
compared with previous results in the literature (in 4.2.2.1, 4.2.2.2 and 4.2.2.5).
The CCRB approach is also used to detail constraint models where the constraints
are inappropriate either because the constraints are over-determined (in 4.2.2.3) or
87
because they are under-determined (in 4.2.2.4). As with the more general convo-
lutive mixture model, a constraint model case is presented in 4.2.2.6 that did not
previously exist in the literature.
4.1 Convolutive Mixture Model
The complex baseband representation of a multi-input, multi-output (MIMO)
finite impulse response (FIR) system, or the convolutive mixture model, may be
written as
ym(n) =
Ksummationdisplay
k=1
s(k)(n)?h(k)m (n) +wm(n)
=
Ksummationdisplay
k=1
Lksummationdisplay
l=0
s(k)(n?l)h(k)m (l) +wm(n) (4.2)
for the nth observation of the mth channel where the model consists of K sources,
M channels (1?m?M) with maximal order Lk for the kth source (or Lk +1 taps
for the kth source), N output samples (0?n?N?1) per channel and N +Lk
input samples per channel per source. For each source, the model can be seen in
figure 4.1. The dimensions K, M, Lk for k = 1,...,K, and N are all assumed
known. The elements s(k)(n) denote the scalar, complex value of the kth input
source at time n; the elements of h(k)m (l) denote the scalar, complex value of the lth
filter coefficient (or lag) of the kth source processed by the mth channel. Both the
signal inputs and channel coefficients are treated as deterministic unknowns, i.e.,
parameters having true values that can be estimated. The noise elements wm(n)
are commonly assumed to be zero-mean, complex-valued, circular Gaussian iid over
88
space and time (and channel) with known variance ?2.1
s[n] z?1 z?1 z?1
Lsummationdisplay
l=0
h[l]s[n?l]
?
? ? ?h[0] h[1] h[L]
Figure 4.1: Finite Impulse Response (tapped delay line) model.
The convolution aspect of this model is often used to characterize the inter-
symbol interference in direct sequence CDMA (code-division multiple-access) over
dispersive channels [44]. This occurs, for example, when the propagation time of
the signal is shorter than the coherence time of the channel in wideband signals.
The varying channel order lengths Lk represent the different coherence times of
the different source-channel links. This convolution also applies to scenarios where
reverberation of the transmitted signal is present, which occurs when the commu-
nications is echoed. Additionally, the additive aspect over the sources represents
the multiuser interference (e.g., the cocktail problem) common in communications
systems today. Moreover, this convolutive mixture model incorporates a number of
important model subclasses:
1. the convolutive single-input, multi-output (SIMO) model when K = 1,
1Under the Gaussian assumption, an unknown variance parameter decouples from the unknown
mean parameters in the Fisher information. So, while the noise power will affect the performance
potential, whether ?2 is known or not does not affect how the CRB (CCRB) depends on the
parameters. It does, of course, affect estimation.
89
2. the convolutive single-input, single-output (SISO) model when K = M = 1,
3. the memoryless, instantaneous mixingmodel whenLk = 0 for allk = 1,...,K,
and
4. the calibrated model (with constraints and a transformation of parameters).
For generality, we consider the full (MIMO) model, but the results contained in this
section also apply to the preceding subclasses.
4.1.1 Equivalent Convolutive Mixture Models
4.1.1.1 The Vector-Matrix Model
In vector-matrix notation, the model may be written as
y =
Ksummationdisplay
k=1
H(k)M s(k) +w =HMs+w (4.3)
where the observations are contained in yT = bracketleftbigyT1 ,???,yTMbracketrightbig? CNM and the ob-
servations for each channel contained in yTm = bracketleftbigym(0),???,ym(N?1)bracketrightbig? CN for
m = 1,...,M, s(k)T = bracketleftbigs(k)(?Lk),...,s(k)(N?1)bracketrightbig? CN+Lk represents the input
sequence of the kth source, the channel matrix is represented by the NM?N +Lk
matrix
H(k)M =
?
??
?
H(k)(1)
...
H(k)(M)
?
??
? (4.4)
90
where the mth channel submatrix over the kth source is given by the N?N +Lk
matrix
H(k)(m) =
?
??
??
??
?
h(k)m (Lk) h(k)m (Lk?1) ??? h(k)m (0)
h(k)m (Lk) h(k)m (Lk?1) ??? h(k)m (0)
... ...
h(k)m (Lk) h(k)m (Lk?1) ??? h(k)m (0)
?
??
??
??
?
(4.5)
for m = 1,...,M, and the noise vector is wT = bracketleftbigwT1,???,wTMbracketrightbig ? CMN with
the noise for each channel given by wTm = bracketleftbigwm(0),...,wm(N?1)bracketrightbig? CN. This
particular vector-matrix ordering of the model can also be represented as
y =
Ksummationdisplay
k=1
parenleftbigI
M?M ?S(k)
parenrightbigh(k) +w=S
Mh+w (4.6)
where the input samples are now organized into an N?Lk + 1 Toeplitz matrix
S(k) =
?
??
??
s(k)(0) s(k)(?1) ??? s(k)(?Lk)
s(k)(1) s(k)(0) ??? s(k)(?Lk + 1)
... ... ... ...
s(k)(N?1) s(k)(N?2) ??? s(k)(N?Lk?1)
?
??
??
with ? being the Kronecker product, and the channel elements are vectorized as
h(k)T =
bracketleftBig
h(k)T1 ,...,h(k)TM
bracketrightBig
?CM(Lk+1) with h(k)Tm =
bracketleftBig
h(k)m (0),...,h(k)m (Lk)
bracketrightBig
?CLk+1
for each k = 1,...,K and m = 1,...,M. The purpose of these alternative vector-
matrix methods will become evident in the development of the Fisher information
(see section 4.1.3) for the original model in (4.2).
4.1.1.2 The Z transform model
Yet another alternative representation of this model, necessary to introduce
relevant characteristics of the model parameters, and often referred to as being in
91
the Z-transform domain, is the M-variate stationary process
y(n) =bracketleftbigH(z)bracketrightbig?s(n) +w(n) (4.7)
where H(z) is a M?K (global) transfer function (polynomial) defined by
H(z) =bracketleftbigH(1)(z),...,H(K)(z)bracketrightbig
for nonzeroz?C? (including?) withH(k)(z) beingthekth source transfer function
H(k)(z) =
Lksummationdisplay
l=0
h(k)(l)z?l
and with h(k)(l)T =
bracketleftBig
h(k)1 (l),h(k)2 (l),...,h(k)M (l)
bracketrightBig
? CM for l = 0,...,Lk and k =
1,...,K, s(n)T = bracketleftbigs(1)(n),s(2)(n),...,s(K)(n)bracketrightbig? CK for n = 0,...,N?1, and
y(n) and w(n) correspond to match the models in (4.3) and (4.6).
Hence, H(z) is a polynomial matrix of the backward shift z?1. The kth
source transfer function is said to have a common zero if there exists a nonzero
z0 ?C? such that H(k)(z0) = 0. If the transfer function does not have a common
zero, then the polynomials H(k)(m)(z) =
Lksummationdisplay
l=0
h(k)m (l)z?l, for m = 1,...,M, are said
to be coprime. The global transfer function is said to be reducible if there exists a
nonzero z0 ? C? such that rank(H(z0)) < K. If not, it is said to be irreducible.
The global transfer function is said to be column-reduced if lim
z?0
rank(H(z)Z) = K
where Z = diag{zL1,...,zLK}. This necessarily implies for an irreducible and
column-reduced transfer function, called a minimum polynomial basis [35], that
M ?K. The connection between irreducibility in the multi-source case and not
having common zeros in the single source case is made clear in the following result.
92
Theorem 4.1. H(z) is irreducible if and only if
Ksummationdisplay
k=1
?kH(k)(z) has no common
zeros for any nontrivial complex-valued collection ?1,...,?K.
Proof. H(z) is reducible if and only if there exists some point z0 ?C? such that
rank(H(z0))<K, which follows if and only if there exists nontrivial?1,...,?K ?C
such that
Ksummationdisplay
k=1
?kH(k)(z0) = 0, i.e., if and only if
Ksummationdisplay
k=1
?kH(k)(z) has a common zero.
The connection between the model in (4.3) and (4.7) is that the K-source, M-
channel matrix HM is a column-rotated, generalized Sylvester matrix of the block
Toeplitz matrix
?
??
??
H(L) H(L?1) ??? H(0)
H(L) H(L?1) ??? H(0)
... ...
H(L) H(L?1) ??? H(0)
?
??
??
where H(l) = bracketleftbigh(1)(l),...,h(K)(l)bracketrightbig with h(k)(l) a null vector when l > Lk and
L = max
k
Lk. Furthermore, under certain conditions, the rank of HM is determined
by the characterization of the basis H(z).
Theorem 4.2. Assume M >K and N?
Ksummationdisplay
k=1
Lk. Then HM is full column rank if
and only if H(z) is a minimum polynomial basis.
Proof. This can be shown by a variant of the proof in Loubaton and Moulines [44,
theorem 1].
The input sequence s(k) is said to have pk modes2 if it can be written as a
2There are a number of alternative definition of modes [30, 43], [35, p. 168].
93
linear combination
s(k)(n) =
pksummationdisplay
i=1
cimn+Lki ,
where ci, i = 1,...,pk are complex-valued weights and mi, i = 1,...,pk are the
C?-valued roots of the polynomial
a(0) +a(1)z?1 +a(2)z?2 +???+a(pk)z?pk,
whose coefficients satisfy
pksummationdisplay
j=0
s(k)(i+j)a(j) = 0
for i=?Lk,...,N?pk?1. Hence, the Toeplitz matrix
S(k)(n) =
?
??
??
s(k)(n) s(k)(n?1) ??? s(k)(?Lk)
s(k)(n+ 1) s(k)(n) ??? s(k)(?Lk + 1)
... ... ...
s(k)(N?1) s(k)(N?2) ??? s(k)(N?n?Lk?1)
?
??
?? (4.8)
has rank min{N ?n,p,n + Lk + 1} [76, lemma 1]. Note, S(k)(0) = S(k) and
S(k)(?Lk) = s(k) from section 4.1.1.1. If one of the modes of s(k) is a common
zero of the channel transfer function H(k)(z), say mv, then the channel makes no
distinction between s(k) and s(k)prime, defined by
s(k)prime(n) =
pksummationdisplay
i=1inegationslash=v
cimn+Lki ,
even though s(k) negationslash= s(k)prime. Ironically, if a channel lacks sufficient diversity, the more
modes an input has leads to greater risk of lost information on the input but poten-
tially less meaningful loss depending on the weights.
Theorem 4.3. The matrix
S(n) =bracketleftbigS(1)(n) S(2)(n) ??? S(K)(n)bracketrightbig (4.9)
94
is full column rank only if N?(K + 1)n+K +
Ksummationdisplay
j=1
Lj, ptotal ?Kn+K +
Ksummationdisplay
k=1
Lk,
and pk ?Lk+1+n for eachk = 1,...,K and ifN?(K+1)n+K+(K+2)
Ksummationdisplay
j=1
Lj,
ptotal?K?n+K + (K + 1)
Ksummationdisplay
k=1
Lk, and pk ?Lk + 1+n+
Ksummationdisplay
j=1
Lj.
Proof. This is a variation of the results in [76, 1, 48, 49].
In particular, forn = 0 thenS(0) requiresN?K+
Ksummationdisplay
j=1
Lj,ptotal?K+
Ksummationdisplay
j=1
Lj,
and pk ?Lk + 1, for each k = 1,...,K, to be full rank. Or conversely, S(0) is full
rank ifN?K+(K+2)
Ksummationdisplay
j=1
Lj, ptotal?K+(K+1)
Ksummationdisplay
k=1
Lk, andpk ?Lk+1+
Ksummationdisplay
j=1
Lj.
Before continuing to the development of the Fisher information for this con-
volutive mixture model, it is relevant to consider a notion of identifiability using the
concepts of the Z-transform model in (4.7). However, this next section is a brief
aside that may be skipped if not of interest to the reader.
4.1.2 Strict Identifiability
The K-user, M-channel FIR system in (4.2) or (4.3) is said to be strictly
identifiable (SI) if and only if
HMs=HprimeMsprime ??Hprime(z) =H(z)A and sprime(n) =A?1s(n)
for some nonsingular matrix A. The converse statement is always true, so strict
identifiability depends on the conditional statement. The term strict identifiability
is a misnomer. As is clear from the definition, the deterministic parameters are
not ?identifiable? when they are strictly identifiable. Instead, the parameters are
95
identifiable up to some minimal ambiguity. When this situation occurs, then it
is possible to treat the channel (signal) components statistically to truly identify
the deterministic signal (channel) parameters using stochastic approaches, e.g., the
subspace method [24, 44]. In the context of this thesis, where the parameters are
not treated as random but as deterministic, the reduction to a minimal ambiguity
set also reduces the number of necessary constraints to eliminate it.
Strict identifiability in the convolutive channel model, it shall be shown, has
some interesting connections with the Fisher information matrix. This is not sur-
prising, given the results in section 3.2. It is perhaps also intuitive to expect that as
strict identifiability is a notion of near-identifiability, then the corresponding Fisher
information will satisfy some notion of near-regularity.
Abed-Meraim and Hua [1] developed necessary and sufficient conditions for
strict identifiability in terms of characteristics in the Z-transform model, i.e., no
channel zeros, the number of signal modes, more sources than sensors, etc. The
following two theorems will be presented without proof, as they were in [1]. Proofs
I developed can be found in [48].
Theorem 4.4 (SI necessary conditions). The M-channel K-source FIR system is
strictly identifiable only if3
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K +
Ksummationdisplay
j=1
Lj,
3In [1], condition (a) omits the column-reducedness requirement, condition (c) did not include
the special case, and condition (d) is originally(and I think erroneously) stated N ?2K+summationtextKj=1 Lj.
96
(c) pk ?Lk + 2 for k = 1,...,K of pk ?1 if Lk = 0, and
(d) N?K +
Ksummationdisplay
j=1
Lj.
Theorem 4.5 (SI sufficiency conditions). The M-channel K-source FIR system is
strictly identifiable if
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K + (K + 1)
Ksummationdisplay
j=1
Lj,
(c) pk ?Lk + 1+
Ksummationdisplay
j=1
Lj for k = 1,...,K, and
(d) N?K + (K + 2)
Ksummationdisplay
j=1
Lj.
Yet other notions of near-identifiability in the convolutive channel model exist
in the literature, e.g., cross-relation-based identifiability [43], which have an equiv-
alence to strict identifiability in the SIMO case [30]. The necessary and sufficient
conditions presented here are independent of the number of channels, although one
should expect that increasing the number of channels would increase channel di-
versity thereby weakening the requirements on the other channel or source charac-
teristics. Hence, theorems 4.4 and 4.5 are in some sense conditions for the least
diversified, strictly identifiable scenario M = K + 1. Table 4.1 shows the growth
in necessary and sufficient conditions as the number of users increases for a fixed
channel size and for fixed channel orders.
97
Table 4.1: Necessary and sufficient data sizes for strict identifiability
# of sources K 1 2 3 4 5
necessary data size N 6 12 18 24 30
sufficient data size N 16 42 78 124 180
(M = 6 channels, Lk = 5 for all sources)
4.1.3 The Fisher information of the convolutive mixture model
4.1.3.1 Complex-valued Fisher information
Before presenting the Fisher information on the model in (4.2), certain details
about Fisher information matriceson complex-valuedparameters are necessary. The
structure of the FIM depends on the ordering of the parameters. For complex-valued
parameters, the FIM parameter vector consists of the real and imaginary parts of
the parameters. If the complex-valued parameters are collected in the vector?and
the (FIM) real parameter vector is defined to be ?T = bracketleftbigRe(?T),Im(?T)bracketrightbig and the
real-valued parameter FIM has the structure
I(?) =
bracketleftbiggE
1(?) ?E2(?)
E2(?) E1(?)
bracketrightbigg
, (4.10)
then the complex-valued parameter Fisher information matrix (CFIM) may be de-
fined as4
I(?) = E??(x;?)?H(x;?) = 12parenleftbigE1(?) +j?E2(?)parenrightbig
where ?(x;?) = ?logp(x;?)??? .5 A number of properties for the real-valued param-
eter FIM can be gleaned from properties on the CFIM.
4This CFIM is a submatrix of the complex-valued parameter FIM developed by van den Bos
[70], which would be the preferred FIM for use in a performance metric or in applying constraints
[32]. However, in this and the next sections, the CFIM presented here is only used to obtain
properties relevant to the real-valued parameter FIM.
5The complex derivative is defined to be ?
?? =
?
?Re??j?
?
?Im? in this thesis.
98
Theorem 4.6. The null space of the FIM of the form (4.10) has dimension exactly
twice the dimension of the null space of the corresponding CFIM, i.e.,
nullity(I(?)) = 2?nullity(I(?)).
Proof. Note
bracketleftbiggE
1 ?E2
E2 E1
bracketrightbiggbracketleftbigga
b
bracketrightbigg
= 0 if and only ifE1a?E2b= 0 andE2a+E1b= 0
if and only if parenleftbigE1 +jE2parenrightbigparenleftbiga+jbparenrightbig = parenleftbigE1a?E2bparenrightbig+ jparenleftbigE2a+E1bparenrightbig = 0. Also,
bracketleftbiggE
1 ?E2
E2 E1
bracketrightbiggbracketleftbigga
b
bracketrightbigg
= 0 if and only if
bracketleftbiggE
1 ?E2
E2 E1
bracketrightbiggbracketleftbigg?b
a
bracketrightbigg
= 0. Finally, the dimension
of
bracketleftbigga ?b
b a
bracketrightbigg
is 2 unlessa=b= 0; however, the dimension ofbracketleftbiga+jb, ?b+jabracketrightbigis
only 1 sinceparenleftbig?b+japarenrightbig=?j?parenleftbiga+jbparenrightbig.
A version of Bang?s formula [36, a variant of (15.52) on p. 525] can be devel-
oped for complex-valued parameters, i.e.,
Iij(?) = tr
bracketleftbigg
C?1(?)?C(?)???
i
C?1(?)?C(?)??
j
bracketrightbigg
+??
H(?)
???i C
?1(?)??(?)
??j +
??T(?)
???i C
?1(?)??
?(?)
??j
for any observations y?CN(?(?),C(?)).
4.1.3.2 CFIM for the convolutive mixture model
For the convolutive mixture model in (4.3) and (4.6), only the mean vector
depends on the unknown parameters, hence
I(?) = ??
H(?)
??? C
?1??(?)
??T +
??T(?)
??? C
?1??
?(?)
??T .
The mean vector is
?(?) =
Ksummationdisplay
k=1
H(k)M s(k) =
Ksummationdisplay
k=1
parenleftbigI
M ?S(k)
parenrightbigh(k).
99
Therefore, if the complex-valued parameter vector in the model in (4.3) is defined
by
?T =
?
??
??
??
h(1)T
s(1)T
...
h(K)T
s(K)T
?
??
??
?? (4.11)
then the complex-valued FIM of the model can be shown to be
I(?) = 1?2QHQ = 1?2
?
??
??
QH1 Q1 QH1 Q2 ??? QH1 QK
QH2 Q1 QH2 Q2 ??? QH2 QK
... ... ??? ...
QHKQ1 QHKQ2 ??? QHKQK
?
??
?? (4.12)
where Q =bracketleftbigQ1,...,QKbracketrightbig and Qk =
bracketleftBig
IM ?S(k) H(k)M
bracketrightBig
.
4.1.3.3 Properties of the CFIM
In this section, I develop properties of this CFIM. In particular, the singularity
of the CFIM is proven and a limit on the dimension of this singularity is detailed, as
well as necessary and sufficient conditions on the parameter characteristics to attain
this limit.
Given the inherent relationship between regularity of the FIM and identifi-
ability as detailed in section 3.2, it should not be surprising that the FIM, and
hence CFIM, is singular. The model presented in (4.3) or (4.6) has a multiplicative
ambiguity with any source interacting with its corresponding channel, i.e., for any
nonsingular matrix A?RLk+1?Lk+1, the input source matrix S(k) and the channel
vectorh(k)m are indistinguishable fromS(k)AandA?1h(k)m , respectively. Over all the
channels, this presumes a complex-valued multiplicative ambiguity of at least MLk.
Additionally, cross-source ambiguities exist between a source input and a different
100
source channel. The limit for the minimal degrees of freedom or the rank of the
ambiguity is given in the following theorem.
Theorem 4.7. The CFIM is singular and the dimension of its null space is lower
bounded as
nullity(I(?))?
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+, (4.13)
where (a)+ = a for a?0 and (a)+ = 0 for a< 0. This limit quantity is the nullity
lower bound (NLB).
The proof can be found in the appendix C. The proof constructs the following
matrix, whose linearly independent columns are a basis for the null space of I(?),
N =
?
??
??
??
??
?
h(1) H(2)(1) 0 0 ??? H(K)(1) 0 ??? 0 ??? 0 0
?s(1) 0 0 ?S(2)(1) ??? 0 0 ??? 0 ??? 0 ?S(K)(1)
0 0 h(2) H(1)(2) ??? 0 H(K)(2) ??? 0 ??? 0 0
0 ?S(1)(2) ?s(2) 0 ??? 0 0 ??? 0 ??? ?S(K)(2) 0
..
.
..
.
..
.
..
. ...
..
.
..
. ???
..
. ???
..
.
..
.
0 0 0 0 ??? 0 0 ??? h(K) ??? H(2)(K) H(1)(K)
0 0 0 0 ??? ?S(1)(K) ?S(2)(K) ??? ?s(K) ??? 0 0
?
??
??
??
??
?
,
(4.14)
where H(i)(j) and S(i)(j) are defined in (C.2) and (C.4), respectively.
The nullity lower bound (NLB) seems to be an unusual quantity. The term
(Li?Lj + 1)+ represents the ambiguity for the ?tap window? of the ith channel
masking the interaction of the jth channel with its corresponding source. This
quantity also reveals that an ambiguity exists only if the coherence-propagation
delay of the ith channel is at least as greats as for the jth channel. Intuitively,
the greater the spread between the channel orders increases the overlapping window
for the jth channel?s interaction to be masked and thereby increases the ambiguity.
Conversely, when the channel orders have narrow differences, this increases the
101
diversity of the channel by limiting the window where one channel can cover that of
another.
A simple corollary follows stating that the NLB or the dimension of the am-
biguity set is at least the square of the number of sources.
Corollary 4.8. nullity(I(?))?K2.
Proof. Note (Li ?Lj + 1)+ + (Lj ?Li + 1)+ ? 2 with equality if and only if
|Li?Lj|?1. Thus,
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+ =
Ksummationdisplay
i=1
Ksummationdisplay
j=1
jnegationslash=1
(Li?Lj + 1)+ +
Ksummationdisplay
i=1
(Li?Li + 1)+
=
Ksummationdisplay
i=1
Ksummationdisplay
j=i+1
(Li?Lj + 1)+ + (Lj?Li + 1)+ +K
? K(K?1)2 ?2 +K = K2.
The proof also reveals that thisK2 degree of uncertainty can only be attained
if any two orders differ by at most 1 tap, whichagrees with the intuitionthat channel
diversity is enhanced when the channel orders are not widely spread.
In addition to the ambiguity due to the channel order spread, the degrees of
freedom in the model depends on the number of parameters and the number of
observations.
Theorem 4.9. nullity(I(?))?
Ksummationdisplay
k=1
(N +Lk +M(Lk + 1))?MN.
Proof. Since I(?) = 1?2QHQ then rank(I(?)) = rank(Q). Since Q is a MN?
Ksummationdisplay
k=1
(N+Lk+M(Lk+1)), thenrank(I(?))?min
braceleftBigg
MN,
Ksummationdisplay
k=1
(N +Lk +M(Lk + 1))
bracerightBigg
102
and therefore
nullity(I(?)) ? colsize(I(?))?rank(I(?))
? max
braceleftBigg
0 ,
Ksummationdisplay
k=1
(N +Lk +M(Lk + 1))?MN
bracerightBigg
.
The number of columns ofQ(orI(?)) corresponds to the number of unknown
parameters. Likewise, the number of rows of Q corresponds to the number of equa-
tions (observations) in the model (4.2). In most scenarios, having more equations
(rows) than unknowns (columns) is a necessary requirement for the unknowns to be
solvable. Thus, the degrees of freedom of the ambiguity space is at least the number
of unknowns (parameters) less the number of equations. However, if the equations
are linearly dependent (redundant), as is the case in the convolutive mixture model,
then the degrees of freedom is potentially greater. The following corollary combines
theorems 4.7 and 4.9.
Corollary 4.10.
nullity(I(?))?max
braceleftBigg Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+,
Ksummationdisplay
k=1
(N +Lk +M(Lk + 1))?MN
bracerightBigg
.
The NLB depends only on the channel orders and number of sources, whereas
the ?equations vs. unknowns? nullity bound depends on the channel orders, the
sample size per source, the number of subchannels per source, and the number of
sources. Hence, it is of interest to determine under what conditions on the sample
size N and the number of channels M per source will this second bound be of
no consequence. Control over the dimensions K (the number of transmitters), M
103
(the number of receivers),N (the number of time snapshots or transmission length)
is possible in the design of many communications models. The next two results
determine conditions on the dimensions of the model which allow the NLB to be the
minimumpossible degrees of freedom. The first condition requiresmore subchannels
per source than sources (or more receivers than transmitters in a communications
context).
Theorem 4.11. The CFIM I(?) can attain the CFIM nullity lower bound only if
M >K.
Proof. If M ? K then HM, which is a MN?KN +
Ksummationdisplay
k=1
Lk matrix, cannot be
full column rank by theorem 4.2. This implies the existence of a null space of a
larger dimension than the space spanned by the columns of N in (4.14). Hence
nullity(I(?)) >
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+.
Under the assumption that more receivers than transmitters are in the model,
then corollary 4.10 implies a minimal requirement on the transmission snapshots.
Theorem 4.12. Provided M > K, the CFIM I(?) can attain the CFIM nullity
lower bound only if
N? K + 1M?K
Ksummationdisplay
j=1
Lj +
Ksummationdisplay
j=1
Lj +K+ 1M?K
parenleftBigg
K2?
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+
parenrightBigg
(4.15)
Proof. We desire
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj +1)+ ?
Ksummationdisplay
k=1
(N+Lk +M(Lk +1))?MN. Solving
for N shows the result.
As the channel diversity (in terms of the number of channels per source) in-
creases, the necessary size on the data to attain the CFIM NLB decreases. As
104
M??then the bound on the sample size becomes simplyN?K+
Ksummationdisplay
j=1
Lj, which
is comparable to theorem 4.4(d). However, for any finite M and nontrivial channel
orders, the bound is effectivelyN?K+
Ksummationdisplay
j=1
Lj +1 since?
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj +1)+ ?
?K
Ksummationdisplay
j=1
(Lj + 1), and hence a looser bound than (4.15) is
N ? K + 1M?K
Ksummationdisplay
j=1
Lj +
Ksummationdisplay
j=1
Lj +K + 1M?K
parenleftBigg
K2?K
Ksummationdisplay
j=1
(Lj + 1)
parenrightBigg
?
Ksummationdisplay
j=1
Lj +K + 1M?K
Ksummationdisplay
j=1
Lj.
In the scenario requiring the most data samples, M = K + 1, then the system
requires N?(K +1)
Ksummationdisplay
j=1
Lj +
Ksummationdisplay
j=1
Lj +K +
parenleftBigg
K2?
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+
parenrightBigg
. (This
last term is always nonpositive.) In the SIMO scenario (K = 1), this necessary
condition becomes N?3L+ 1 which agrees with the requirement in [31].6
Given that M >K and N satisfies (4.15), then the minimum possible nullity
of I(?) is the nullity lower bound,
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+,
i.e., it is possible to limit the ambiguity strict to the mixing of sources in the convo-
lutive channel. If it is possible to understand the conditions under which the CFIM
attains this nullity lower bound, then it is known that the null space is completely
characterized by the columns of N in (4.14). The importance of this is that when
6For the SIMO (K = 1) case, this nullity lower bound is exactly 1 and parameters for which
the CFIM attains this bound are said to be Fisher information identifiable in [30]. The notion
of identifiability for the SIMO case is sensible since by scaling a single parameter it is possible to
obtain a bound on all the remaining parameters relative to the scaled parameter. This naturally
connects with the notion of strict identifiability in section 4.1.2. This interpretation does not
extend simply to the MIMO (K > 1) case; however, it shall be shown that there does exist an
inherent connection between a CFIM attaining the NLB and the notion of strict identifiability.
105
the null space can be parameterized, then it is possible to use theorems 3.23 and
3.24 to determine constraints that lead to regularity of the CFIM (and FIM). Us-
ing the concepts of signal excitation (modes) and channel diversity (irreducibility
and column-reducedness) as defined in section 4.1.1.2, the following theorems also
establish a correlation of conditions between the idea of near-regularity when the
FIM attains the NLB and of near-identifiability (or strict identifiability) of section
4.1.2.
Theorem 4.13 (CFIM NLB necessary conditions). The M-channel K-source FIR
system Fisher information matrix has a nullity of exactly the NLB in (4.13) only if
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K +
Ksummationdisplay
j=1
Lj,
(c) pk ?Lk + 2 for k = 1,...,K or pk ?1 if Lk = 0,
(d) N?K +
Ksummationdisplay
j=1
Lj, and
(e) M >K.
Theorem 4.14 (CFIM NLB sufficiency conditions). The M-channel K-source FIR
system FIM has a nullity of exactly the NLB in (4.13) if
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K + (K + 1)
Ksummationdisplay
j=1
Lj,
(c) pk ?Lk + 1+
Ksummationdisplay
j=1
Lj for k = 1,...,K,
106
Table 4.2: Necessary and sufficient data sizes for the FIM to attain the NLB
# of sources K 1 2 3 4 5
necessary data size N 8 20 38 74 180
sufficient data size N 16 42 78 124 180
(M = 6 channels, Lk = 5 for all sources)
(d) N?K + (K + 2)
Ksummationdisplay
j=1
Lj, and
(e) M >K.
With the exception of condition (e), then theorems 4.13 and 4.14 are identical
to theorems 4.4 and 4.5, respectively. This is an expected result since the Fisher
information for identifiable parametric components within the model should meet
certain regularity conditions (see theorem 2.5). For theorem 4.13(d), the condition
is weaker than the condition in theorem 4.12, but it is left weaker to agree with the
necessary condition for strict identifiability in theorem 4.4(d). (It is unclear if that
conditionwouldchange if dependenceon channelsizeM isconsidered.) Table 4.1.3.3
show the growth in the necessary and sufficient data sizeN as the number of sources
increases for fixed channel size and channel orders, using the stronger condition in
theorem 4.12 instead of that in theorem 4.13(d). It is clear that in comparison,
as the number of sources increases the number of channels, the necessary data size
grows to match the sufficient data size, which lead to theorem 4.15 and corollary
4.16.
First, the special case when the CFIM attains the minimal ambiguity is con-
sidered. As stated earlier this occurs when the windows of the channels are, in some
sense, minimally spread.
107
Theorem 4.15. Given sufficient channel diversity and modes in the signals, e.g.,
the sufficient conditions of theorem 4.14, then nullity(I(?)) = K2 if and only if
Lj ?{L0,L0 + 1}for all j = 1,...,K, for some integer L0.
Proof. Under the conditions of theorem 4.14, then nullity(I(?)) = NLB and from
the proof of corollary 4.8, NLB = K2 if and only if (Li?Lj+1)++(Lj?Li+1)+ ?2
for eachi,j = 1,...,K. Either(Li?Lj+1)+ = (Lj?Li+1)+ = 1or (Li?Lj+1)+ = 2
and (Lj?Li + 1)+ = 0 or vice versa.
The next corollary details the necessary and sufficient data size for the sources
under the special case of equivalent channel orders.
Corollary 4.16. Given sufficient signal modes and channel diversity, e.g., as in
theorem 4.14, if Lk = L for each k and M = K+1, then the CFIM attains the NLB
if and only if
N?(K + 2)KL+K.
4.1.4 Constraints for the convolutive mixture model
In the previous section, necessary and sufficient conditions on characteristics
of the signal source and channel properties were detailed for the CFIM to have a
null space with minimal dimensions equaling the nullity lower bound
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?
Lj +1)+. In this section, pathways to regularity in the CFIM (and hence the FIM)
as well as several typical constraint sets are considered.
108
4.1.4.1 Pathways to regularity
Requiring constraints to attain regularity in the Fisher information is unnec-
essary to use this CCRB theory (theorem 3.17), but often without constraints the
identifiable parameters are not in the desired framework to be of use to the practi-
tioner. In the communications context, constraints on a model assumed a priori are
simply a common method used to maintain a particular parametric structure in the
model.
Guided by theorems 3.23, 3.24, 3.25, 3.27, and 3.29, then an objective of
communications system design is to discover constraints for which identifiability of
the parameter, as they are defined, is achieved, i.e., what properties can be imposed
on the signal or channel to guarantee parametric identifiability. For the convolutive
mixture model in (4.2), this involves an examination of the Fisher information in
(4.10) with the CFIM in (4.12). In some sense, this has already been done. In
section 4.1.3.3, conditions under which the CFIM has a null space spanned by the
columns of the matrix N, defined in (4.14), are derived.
If I(?) is a Fisher information with Cholesky factorization L(?)LT(?) where
V(?) is an orthogonal complement toLT(?) [20, p.194]. Then theorem 3.23 implies
for any constraint to achieve local identifiability, its Jacobian must satisfy F(?) =
ALT(?) +BVT(?) where BVT has full row rank. Similarly, theorem 3.24 implies
the constraints must have an orthonormal complement U(?) satisfying U(?) =
L(?)C+V(?)D where L(?)C if full column rank.
109
4.1.4.2 Norm channel + real-valued source constraint
The norm channel constraint
vextenddoublevextenddoubleh(1)vextenddoublevextenddouble2 = 1 (4.16)
is a common scaling ?trick? used in SIMO (K = 1 source) models to obtain chan-
nel estimates under second-order statistics assumptions. Combined with the (rota-
tional) constraint of restricting the source elements to be real-valued, i.e.,
Im(s(1)(n)) = 0 (4.17)
for n =?L1,?L1 +1,...,N?1, it is clear that the multiplicative ambiguity which
is the basis of N is eliminated. For the parameter vector ? = bracketleftbigRe(?T),Im(?T)bracketrightbigT
with ? defined in (4.11), the constraints are essentially separated as
F(?) =
bracketleftbiggF
Re(h) 0 FIm(h) 0
0 FRe(s) 0 FIm(s)
bracketrightbigg
where h = h(1) and s = s(1). An orthonormal complement, or a basis for the null
space, of bracketleftbigFRe(s) , FIm(s)bracketrightbig is simply
bracketleftbiggI
(N+L1)?(N+L1)
0
bracketrightbigg
, but an analytic formula for
the null space relating to the channel components is not so simple and requires
numerical programming for arbitrary sizes L1 and N.
Example 4.17. As a particular example of this constraint, consider the M = 2
SIMO convolutive channel model with L = 3 taps, where the channel is predefined
by
bracketleftBigg
h(1)T1
h(1)T2
bracketrightBigg
=
bracketleftbigg 0.3079+ j0.0698 0.1657+ j0.2304 0.0198?j0.3823 0.0929?j0.1853
?0.1841 + j0.3294 0.4484?j0.1689 0.0156+j0.1526 0.4750?j0.0952
bracketrightbigg
,
110
under a white noise assumption (with known variance ?2) and the constraints in
(4.16) and (4.17). The 54 (N = 50) signal elements are BPSK (?1) symbols
randomly generated with equal probability (i.e., Bernoulli(12)) and fixed for the
simulation. The subspace method [52, 68] is used with a smoothing factor of 8.
Additionally scoring with constraints (CSA) using (3.36) is applied on the subspace
method?s estimate ?h(1) for possible improvement. The results are shown in figure
4.2, where the mean-square error (MSE) per real channel coefficient is evaluated by
1
16R
Rsummationdisplay
r=1
vextenddoublevextenddouble
vextenddouble?h(1)?h(1)
vextenddoublevextenddouble
vextenddouble
2 overR = 100 runs or trialsandthe SNR isgivenby 10log
10
1
?2.
This example is also in [68] and shows how the subspace method, which had no prior
theory-based performance metric, tracks with the CCRB. The additional overlaying
estimation using the method of scoring demonstrates the local maximum likelihood
properties in the subspace method.
0 5 10 15 20 25 3010
?5
10?4
10?3
10?2
10?1
SNR (dB)
MSE per channel coeff
subspace method
CSA
CCRB
Figure 4.2: Norm-constrained channel estimation performance.
111
4.1.4.3 Semiblind constraints: s(k)(t) = p(t) for t?T
For multiplesources, designing communicationswith channel constraints, such
as in the previous section, is less tenable since it is not always possible to estab-
lish guarantees on functions of the channel elements. The sources, however, are
often entirely designable elements by restricting the class of signals which are to be
passed through the channels. The class only needs to be defined by some functional
constraint.
Perhaps, the simplest constraint is knowledge of a parameter element, i.e.,
?i =a. This constraint produces a row vectoreTi in the Jacobian F(?), where ei is
the unit vector with unity in the ith position and zero values in the other positions.
Therefore, the corresponding orthonormal complement U(?) for this unit vector
eliminates or nulls out the i row and column of the Fisher information I(?) while
preserving the other elements. Any other nonredundant constraint in f does not
change this.
Theorem 4.18. Assume the conditions of theorem 4.14, then nullity(I(?)) =
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li ?Lj + 1)+. Let I?(?) be denoted by I(?) with the rows {ip : p =
1,...,
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+} and the corresponding columns removed. Then the
nullity(I?(?)) = 0 if and only if N? is full column rank, where N? is the matrix
formed by taking the rows{ip}of N in (4.14).
Proof. Let F be a matrix that when multiplied by the matrix N selects the rows
{ip}, i.e., F is the matrix consisting of the row vectors {eTi : i ?{ip}}. Then
FN = N?, a square full rank matrix. Therefore F = AI(?) +BNT for some
112
A and some B where BNT is full row rank. As discussed in section 4.1.4.1, this
identifies the parameters and hence nullity(I?(?)) = 0 by theorem 3.24.
This theorem can be used to specify either channel or source parameters to
achieve identifiability and FIM regularity. For example, if only source signal param-
eters are specified, then it is necessary and sufficient to specify
Ksummationdisplay
j=1
(Li?Lj + 1)+
parameters of s(i) for each i = 1,...,K, under the conditions of theorem 4.18.
4.1.4.4 Unit Modulus constraint + Semiblind constraint
The unit or constant modulus constraint is a particularly powerful and rea-
sonable assumption. All the source elements are assumed to have unit modulus,
i.e.,
|s(k)(n)|2 = 1, (4.18)
for everyn =?Lk,...,N?1 and for eachk = 1,...,K. This constraint is useful for
modeling P-ary phase-shift keying (PSK) in communications models with unknown
P, where the signals are assumed to be derived from a finite constellation of P
equispaced points on the unit circle. While in practice, this assumption is often
viewed as a single constraint, it is ultimately
Ksummationdisplay
i=1
(N+Li) constraints. Nevertheless,
despite this, it is insufficient to identify the parameters by itself. For the parameter
vector in (4.11), the constraint has a gradient
F(?) =
?
??
0 2Re(S(1)d ) 0 0 ??? 0 2Im(S(1)d ) 0 0 ???
0 0 0 2Re(S(2)d ) ??? 0 0 0 2Im(S(2)d ) ???
... ...
?
??,
113
which has an orthonormal complement
U(?) =
?
??
??
??
??
??
??
??
??
??
0 IM(L1+1) 0 0 0 0 ???
?Im(S(1)d ) 0 0 0 0 0 ???
0 0 0 0 IM(L2+1) 0
0 0 0 ?Im(S(2)d ) 0 0
... ... ... ...
0 0 IM(L1+1) 0 0 0 ???
Re(S(1)d ) 0 0 0 0 0 ???
0 0 0 0 0 IM(L2+1)
0 0 0 Re(S(2)d ) 0 0
... ... ... ...
?
??
??
??
??
??
??
??
??
??
satisfying (3.6) and where S(i)d = diag(s(i)). Using this complement with (4.10) and
(4.12), we have that thei,j subblock ofUT(?)I(?)U(?) has the structureC(i,j) =
?
?? Re[S
(i)H
d H
(i)H
M H
(j)
M S
(j)
d ] Im[S
(i)H
d H
(i)H
M (Im?S
(j))] Re[S(i)H
d H
(i)H
M (Im?S
(j))]
?Im[(Im?S(i)H)H(j)M S(j)d ] Re[Im?S(i)HS(j)] Im[Im?S(i)HS(j)]
Re[H(i)HM H(j)M S(j)d ] Im[H(i)HM (Im?S(j))] Re[H(i)HM (Im?S(j))]
?
??.
Lack of identifiability is noted since
C(k,k)?
?
?
1N+Lk
Im(h(k)?)
Re(h(k)?)
?
?= 0
for each k = 1,...,K. Careful examination determines these are the only vectors
in the null space of UT(?I(?)U(?), resulting in the following theorem.
Theorem 4.19. Assume nullity(I(?)) = 2
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li ?Lj + 1)+. If all sources
are assumed to be unit modulus and one complex-valued parameter for each source
is assumed known, then UT(?)I(?)U(?) will be regular (and the model locally
identifiable).
Intuitively, the unit modulus constraint eliminates the convolutive mixture
and intra-source multiplicative amplitude ambiguity leaving only an intra-source
multiplicative phase ambiguity.
114
Example 4.20. Consider aK = 2 source-M = 2 sensor linear instantaneous mixing
model (Lk = 0 for all sources), with N = 30 samples per source, under a R = 3-ray
multipath subchannels, i.e., the spatial signature of the kth source is expressed as a
weighted sum of steering vectors, i.e.,hk =
Rsummationdisplay
r=1
?kra(?kr) where ?kr and ?kr are the
complex-valued amplitude and the real-valued AOA of the rth multipath of the kth
source (seeFigure 4.3). The AOAsandcorresponding amplitudesare{??1,?,?+4}
source
sensors
direct paths
side paths
Figure 4.3: Example of multipath channel.
and{?0.2?(?pi6),?0.5,?0.15?(?pi5 )}for sources(1) and{?+???5,?+??,?+
??+6}and{?0.15?(?pi5 ),?0.6,?0.25?(pi3)}for sources(2). (See section 4.2 for a
greater description on steering vectors.) The source elements come from an 8PSK
alphabet with signal powers SNR(s(1)) = 20dB and SNR(s(2)) = 15dB. The channel
elements are normalized so that SNR(s(k)) = bardblh(k)bardbl
2
M?2 with ?
2 = 1. The constraints
assumed are unit modulus (8PSK) as well as knowledge of the first T = 2 symbols
for each source, more than sufficient for identifiability and FIM regularity (theorem
4.19).
An initial estimate is obtained using the zero-forcing (bias reducing) variant
of the algebraic constant modulus algorithm (ZF-ACMA) [71]. This algorithm is a
useful tool for estimationof constant modulus source parameters inshort data length
115
experiments (only N?K2 or 2K required), but has no means of incorporating the
training side information. The ZF-ACMA estimate is projected onto ?f to establish
an initialization for the method of scoring with constraints. The step size rule chose
?(k) = 2?m for the least positive integer m satisfying (3.37). The average MSE (per
real parameter coefficient) at each iteration over 5000 trails is compared with the
CCRB for each source in figure 4.4.
0 10 20 30 40 5010
?3
10?2
10?1
100
??
MSE
Source Estimation with varying ??
 
 
S1: ZF?ACMA est
S1: Scoring est
S1: CCRB
S2: ZF?ACMA est
S2: Scoring est
S2: CCRB
Figure 4.4: Source estimation with varying ??.
The mean-square error improvementby utilizingthe complete side information
in scoring maintains efficiency with respect to the constrained CRB for moderately
separated angle of arrivals compared with ZF-ACMA. As should be expected, the
estimation performance degrades as the primary source angles overlap, but even in
116
the worst case scenario with ?? = 0? of separation, the approximate corresponding
8PSK bit-error-rate (BER) for ZF-ACMA and scoring with constraints is.2914 and
.0254, meaning the estimation schemes result in bit decision errors roughly 29% and
3% of the time, respectively. The departure of the estimation performance from
the CCRB as ?? approaches 0? is possibly due to a loss of unbiasedness in the
estimation.
4.2 Calibrated Array Model
The narrowband (calibrated) array model may be written as
ym(n) =
Ksummationdisplay
k=1
am(?k)?ks(k)(n) +wm(n) (4.19)
for n = 1,...,N and m = 1,...,M, where s(k)(n) is the value of the kth input
source at time index n, ?k is the complex-valued channel gain for the kth input,
am(?k) is the mth sensor response to the kth input source, ?k is the angle-of-arrival
(AOA) of the kth source, and wm(n) is the noise, modeled as zero-mean circular
Gaussian with variance ?2 iid in both time and space. In vector-matrix notation,
the model for each time slot can be written as
y(n) =
Ksummationdisplay
k=1
a(?k)?ks(k)(n) +w(n) =A(?)?s(n) +w(n) (4.20)
where the input is given by sT(n) =bracketleftbigs(1)(n),...,s(K)(n)bracketrightbig, the channel gain matrix
is ? = diagparenleftbig?1,...,?Kparenrightbig, and the response matrix is given by
A(?) =bracketleftbiga(?1) a(?2) ??? a(?K)bracketrightbig (4.21)
117
?
source
sensors
(a) uniform linear array
?
source sensors
(b) uniform circular array
Figure 4.5: Calibrated array model geometriesfor (a) uniform linear and (b) circular
arrays.
with the array response vectorsa(?k) depending on the physical design of the array
and the AOA. Calibration posits this known array geometry. The examples consid-
ered in this section consist of the uniform linear array (ULA) and uniform circular
array (UCA), as shown in Figure 4.5. For example, for the ULA the response vectors
typically have a Vandermonde vector structure with base e?ipi sin(?k).
4.2.1 The Fisher information of the calibrated array model
4.2.1.1 Indirect derivation of the FIM
As was mentioned in section 4.1, the calibrated array model can be a special
case of the convolutive mixture model using constraints. The method for transform-
118
ing the model is to keep the parameters for the instantaneous mixing model and to
the parameter vector add extraparameters for the calibratedarray and channel gain.
When evaluating the FIM, the elements in the rows and columns corresponding to
these extra parameters are zero. The constraint then represents the model reparam-
eterization, e.g., for this case, we choose the constraints which define H =A(?)?,
element by element. The resulting CCRB submatrix corresponding to the extra
parameters will be equivalent to the CRB of those parameters. This procedure is
made clear in the following example.
Example 4.21. Assume x?N(ab,1). The FIM for?T = [a,b] isI(?) =
bracketleftbiggb2 ab
ab a2
bracketrightbigg
,
which is singular. Suppose we wish to reparameterize the model replacing ab with
c. This is equivalent to the constraint f(?star) = ab?c for the expanded parameter
vector ??T = [a,b,c]. For this orthonormal complement
U(??) =
?
?
1 0
0 1
b a
?
?
of the Jacobian of the constraints, thenUT(??)I(??)U(??) =I(?), whichisstill sin-
gular. Hence CCRB(??) = U(??)I?(?)UT(??) where the pseudoinverse is I?(?) =
1
(a2+b2)2I(?), or
CCRB(??) =
?
?
b2 ab b3 +a2b
ab a2 ab2 +a3
b3 +a2b ab2 +a3 b4 + 2a2b2 +a4
?
?.
The component corresponding to CCRB(c) = 1(a2+b2)2(b4 + 2a2b2 +a4) = 1, which
agrees with the value of CRB(c) for the model x?N(c,1).
Note that the original model need not be identifiable, nor does the replace-
ment model. Additional constraints under either model can also be included in the
119
constraint function. The example verifies that even for the more difficult case it
is possible to find the CRB and CCRB for the calibrated array model indirectly
from the instantaneous mixing model using constraints. However, in the interests
of clarity, in the next section the FIM and CCRB will be derived for the calibrated
array model directly.
4.2.1.2 Direct derivation of the FIM
The calibrated model in (4.19) has a likelihood given by
p(y(1),...,y(N);?)
= 1(pi?2)MN exp
braceleftBigg
? 1?2
Nsummationdisplay
n=1
parenleftbigy(n)?A(?)?s(n)parenrightbigHparenleftbigy(n)?A(?)?s(n)parenrightbig
bracerightBigg
.(4.22)
For clarity, since there exists a mixture of complex and real parameters requiring
estimation, then using the parameter vector7
? =
?
??
??
??
??
??
Re(s(1))
Im(s(1))
...
Re(s(N))
Im(s(N))
?
?
?
??
??
??
??
??
, (4.23)
with ?T = bracketleftbig?1,...,?Kbracketrightbig and ?T = bracketleftbig?1,...,?Kbracketrightbig, the Fisher information matrix is
given by
I(?) =
?
??
??
??
??
M 0 ??? 0 M1 L1
0 M 0 M2 L2
... ... ... ... ...
0 0 ??? M MN LN
MT1 MT2 ??? MTN K? L
LT1 LT2 ??? LTN LT K?
?
??
??
??
??
(4.24)
7It is also possible to include the noise variance parameter ?2 in the parameter vector. However,
this parameter decouples from the other parameters resulting in an optimistic CRB of ?4MN [16, 59],
and so is uninteresting to the results herein.
120
where
M =
bracketleftbiggRe(M) ?Im(M)
Im(M) Re(M)
bracketrightbigg
, M = 2?2?HAH(?)A(?)?, (4.25)
Mn =
bracketleftbiggRe(M
n) ?Im(Mn)
Im(Mn) Re(Mn)
bracketrightbigg
, Mn = 2?2?HAH(?)A(?)S(n), (4.26)
Ln =
bracketleftbiggRe(L
n)
Im(Ln)
bracketrightbigg
, Ln = 2?2?HAH(?)D(?)?S(n), (4.27)
L=
bracketleftbiggRe(L)
Im(L)
bracketrightbigg
, L= 2?2
Nsummationdisplay
n=1
SH(n)AH(?)D(?)?S(n), (4.28)
K? =
bracketleftbiggRe(K
?) ?Im(K?)
Im(K?) Re(K?)
bracketrightbigg
, K? = 2?2
Nsummationdisplay
n=1
SH(n)AH(?)A(?)S(n), (4.29)
and K? = 2?2
Nsummationdisplay
n=1
ReparenleftbigSH(n)?HDH(?)D(?)?S(n)parenrightbig. (4.30)
In the equations above, we redefine S(n) = diagparenleftbigs(1)(n),...,s(K)(n)parenrightbig and
D(?) =
bracketleftbigg?a(?
1)
??1
?a(?2)
??2 ???
?a(?K)
??K
bracketrightbigg
.
4.2.1.3 Properties of the FIM
The model in (4.19) admits an ambiguity as ?ks(k)(n) is indistinguishable from
parenleftbigc
k?k
parenrightbigparenleftBigs(k)(n)
ck
parenrightBig
for any nonzerock ?C. From Section 2.2, then it should be expected
that the FIM in (4.24) is singular. This is indeed the case, e.g., note that
?
??
??
??
??
M 0 ??? 0 M1
0 M 0 M2
... ... ... ...
0 0 ??? M MN
MT1 MT2 ??? MTN K?
LT1 LT2 ??? LTN LT
?
??
??
??
??
?
?
??
??
??
S(1)
S(2)
...
S(N)
??
?
??
??
??= 0.
121
And since the FIM consists of real and imaginary parts of this matrix, it too has a
null space, namely, the columns of
N(?) =
?
??
??
??
??
??
??
Re(S(1)) ?Im(S(1))
Im(S(1)) Re(S(1))
... ...
Re(S(N)) ?Im(S(N))
Im(S(N)) Re(S(N))
?Re(?) Im(?)
?Im(?) ?Re(?)
0 0
?
??
??
??
??
??
??
are a basis for the null space (or at least a null subspace) of I(?). So the nullity,
or dimension of the null space, of I(?) is at least 2K. In fact, it is exactly 2K
(provided N? KM?K for reasons similar to that given in theorem 4.12).
4.2.2 Constraints for the calibrated array model
4.2.2.1 Constraints on the complex-valued channel gain: ? =IK?K
One approach to eliminating the ambiguity between the complex-valued gain
and the source input isto incorporate thisgain intothe signal. Insteadof remodeling
the mean of (4.19) to be
?(n,?) =
Ksummationdisplay
k=1
a(?k)s(k)(n) (4.31)
by eliminating the unknown gain, it is equivalent to impose the constraints ?k = 1
for k = 1,...,K. The model in (4.31) is a model presented in a paper on ?direction
finding with narrow-band sensor arrays? by Stoica and Nehorai [67]. Hence, if
the theory of Chapter 3 is to be trusted, then results found by imposing proper
constraints that ? =IK?K (or? = 1K) should be equivalent to the results of Stoica
122
and Nehorai. TheK constraints of acomplex-valuedparameter?are 2K constraints
of the corresponding real-valued parameters, i.e., the vector f of constraints can be
defined as
fk(?) = Re(?k)?1 = 0
fK+k(?) = Im(?k) = 0
for k = 1,...,K. For the parameter vector in (4.23), the Jacobian of these con-
straints is given by
F(?) =bracketleftbig02K?2KN I2K?2K 02K?Kbracketrightbig.
Since F(?)N(?) = I2K?2K, then by theorem 3.23, this constraint is sufficient to
(locally) identify the parameters and by theorem 3.24 the matrix UT(?)I(?)U(?)
will be regular for any matrix U(?) satisfying (3.6). An orthonormal basis for the
null space of the Jacobian would be the columns of
U(?) =
?
?
I2KN?2KN 02KN?K
02K?2KN 02K?K
0K?2KN IK?K
?
?,
and this generates a reduced ?FIM?
UT(?)I(?)U(?) =
?
??
??
??
M 0 ??? 0 L1
0 M 0 L2
... ... ... ...
0 0 ??? M LN
LT1 LT2 ??? LTN K?
?
??
??
??,
which is equivalent to the Fisher information in [67, equation (E.9)]. Since the ?
parameters are known and therefore it isunnecessary to understand the performance
potential (or CCRB) of a known parameter, it is only of interest to have a bound
123
on the performance of estimators of the transformation
?=k(?) =
?
??
??
??
??
Re(s(1))
Im(s(1))
...
Re(s(N))
Im(s(N))
?
?
??
??
??
??
,
which has the Jacobian ???T? = UT(?). Hence, CCRB(?) = parenleftbigUT(?)I(?)U(?)parenrightbig?1
is the same as the CRB found by Stoica and Nehorai. This equivalence serves as
further validation of the CCRB approach.
4.2.2.2 Semiblind constraints: s(t) =p(t) for t?T
An alternative approach to eliminating the ambiguity between the source and
coefficient is the have prior knowledge of some of the source signals. These known
elementsare oftenreferredto as training or pilot symbols incommunications. Knowl-
edge of any kth source element s(k)(t) = p(k)(t) at any time sample t resolves the
ambiguity between ?ks(k)(n) for all n since ?k is solvable in the observation corre-
sponding to time sample t and can thus be used to solve the unknown source values
when n /?T. This model can be written as
?(n,?) =
Ksummationdisplay
k=1
a(?k)parenleftbig?kp(k)(n)?n?T +?ks(k)(n)?n/?Tparenrightbig (4.32)
where ?statement = 1 when the statement is true and = 0 when the statement is false.
This model is equivalent to the model designed by Kozick and Sadler [39, equation
(9)] except with ?ks(k)(n) being simply s(k)(n), a distinction that still allows for a
match of results of the CCRB of a properly chosen transformation. The model is
also equivalent to the model designed by Li and Compton [41, equation (2)] when
124
T = {1,2,...,N}. Equivalence of the reparameterized CRBs in [39, 41] with the
CCRB can be found in [59]. As in the previous example, the CCRB on the unknown
parameters is the inverseof the (unconstrained) FIM after the eliminationof its rows
and columns corresponding to the known or specified parameters.
4.2.2.3 Finite alphabet constraint: s(k)(n)?S
This constraint derives from the assumption that the source elements exists in
a (finite) discrete set S. In communications models, this corresponds to digital mod-
ulation designs such as pulse amplitude modulation (PAM), quadrature amplitude
modulation (QAM), phase-shift keying (PSK), etc. As such, the model can also be
the same as any of the previous models in (4.19), (4.31), or (4.32), but the defining
characteristic is that the source samples only exist on a discrete set. There do not
exist any CRB-type bounds for this model in the literature due to the problem of
differentiability with respect to the parameters.8 It is certainly possible to constrain
any single real-valued parameter to a discrete set by creating a polynomial (or sine
function) whose zeros match the set values. But what information can be gained
from a constraint formulation of this discrete-alphabet model? This is perhaps best
answered with the following example.
Example 4.22. Reconsider the model in example 2.2, y?CN(?,?2). Suppose the
8A Chapman-Robbins or Barankin-type bound, which does not require differentiability with
respect to the parameters would be possible but even this approach does not seem to exist in
the literature. Many communications engineers also discount the importance of a mean-square
error bound for a digital signal and rely an alternative performance criteria, such as bit-error rate
(BER).
125
parameter is required to satisfy the constraints
f1(?) = (Re(?))2?12 = 0
f2(?) = (Im(?))2?12 = 0.
This constraint restricts the real and imaginary part of ? to reside in the discrete
set{
?2
2 ,?
?2
2 }, i.e.
??S =
braceleftBigg?
2
2 +i
?2
2 ,
?2
2 ?i
?2
2 ,?
?2
2 +i
?2
2 ,?
?2
2 ?i
?2
2
bracerightBigg
.
(In the communications vernacular, this is quadrature PSK or 4-QAM.) The Jaco-
bian for this constraint is
F(?) =
bracketleftbigg2Re(?) 0
0 2Im(?)
bracketrightbigg
,
which is full column rank for the possible set of values for ?. Hence, to satisfy (3.6),
U(?) = [ ] is a 2?0 null matrix.
This is analogous to the determinate case of example 3.14, therefore knowledge
that a parameter exists in a discrete set is equivalent to complete knowledge of the
parameter value in terms of mean-square error performance potential-the result
being a Cram?er-Rao bound of zero. This does not mean that the mean-square
error will be zero (the decision of which set value the parameter actually is can be
wrong given sufficient power in the noise), but it does mean that the mean-square
error bound is trivial and not particularly helpful. This result is not surprising
considering the degrees of freedom of the parameter that are restricted from such
constraints. The restriction of a real-valued parameter to satisfying a single root
equation eliminates its single degree of freedom.
126
4.2.2.4 Unit modulus constraints: |s(n)|= 1 for all n
Since knowledge of parameters from a discrete (finite) alphabet results in a
trivial bound, then to obtain a relevant and useful measure of performance potential
a relaxation of the side information is necessary. One such example of this approach
is using a constant or unit modulus constraint on the source elements as in section
4.1.4.4 for the convolutive mixture model. This approach remodels the mean as
?(n,?) =
Ksummationdisplay
k=1
a(?k)?kej?(k)(n). (4.33)
This is the constraint considered in example 3.4 applied to the communications
context. Therefore, imposing the constraints
f(n?1)K+k(?) =vextendsinglevextendsingles(k)(n)vextendsinglevextendsingle2?1 = 0,
fork = 1,...,K, n = 1,...,N, is an alternative approach than rederivingthe Fisher
information for the model in (4.33). The Jacobian for this constraint has the form
F(?) =
?
??
2Re(S(1)) 2Im(S(1)) 0K?3K
... ...
2Re(S(N)) 2Im(S(N)) 0K?3K
?
??, (4.34)
which has a null space generated by the columns of the matrix
U(?) =
?
??
??
??
??
?Im(S(1))
Re(S(1))
...
?Im(S(N))
Re(S(N))
I3K?3K
?
??
??
??
??
. (4.35)
From this we can check the (local) identifiability of this model under the (unit)
constant modulus constraint using theorem 3.24. The matrix UT(?)I(?)U(?) is
2
66
66
66
66
4
Re(S?(1)MS(1)) Im(S?(1)M1) Re(S?(1)M1) Im(S?(1)L1)
... ... ... ...
Re(S?(N)MS(N)) Im(S?(N)MN) Re(S?(N)MN) Im(S?(N)LN)
?Im(MH1 S(1)) ??? ?Im(MHNS(N)) Re(K?) ?Im(K?) Re(L)
Re(MH1 S(1)) ??? Re(MHNS(N)) Im(K?) Re(K?) Im(L)
?Im(LH1 S(1)) ??? ?Im(LHNS(N)) Re(LH) ?Im(LH) K?.
3
77
77
77
77
5
127
Since
UT(?)I(?)U(?)?
?
??
??
??
??
IK?K
...
IK?K
?Im(?)
Re(?)
0K?K
?
??
??
??
??
= 0,
then UT(?)I(?)U(?) is singular and by theorem 3.24 the (unit) constant modulus
constraints are not sufficient for identifiability. This was also the case for the convo-
lutive mixture model in section 4.1.4.4. Furthermore, reviewing the model in (4.33),
this result has more reason to be expected. The original identifiability issue in the
calibrated model (4.19) is the multiplicative ambiguity between the sources and the
channel gain. While the constant modulus constraint resolves any amplitude ambi-
guity of the channel gain, i.e.,|?k|= vextendsinglevextendsingle?ks(k)(n)vextendsinglevextendsingle for any n, there still exist a phase
rotation ambiguity. It is clear from both the model and from theorem 3.23, that for
each source k knowledge of an element for either the real or imaginary part of either
the channel gain or a source sample will be sufficient for identifiability and a regular
UT(?)I(?)U(?). Of course, given this constant unit modulus constraint, knowl-
edge of the real (imaginary) part of any source sample is equivalent to restricting
the imaginary (real) part of the source sample to a finite discrete alphabet, which
as discussed in section 4.2.2.3 is the same as knowledge of the parameter in regards
to the CCRB performance potential but not in regards to the estimation. Hence for
estimation performance it is a necessity to constrain the real and/or imaginary part
of the source sample.
128
4.2.2.5 Unit modulus constraint; real-valued channel gain: Im(?k) =
0 for all k
This model is that given in (4.33) except with ?k being real-valued. As seen
in sections 4.2.2.2 and 4.2.2.3, the constraint is knowledge of the imaginary parts of
? and the effect of the correspondingU(?) matrix is the elimination of the columns
and rows corresponding to the Im(?k) parameters. This model is equivalent to that
of Leshem and van der Veen [40]. Verification that the CCRB is equivalent can be
found in [59]. The bound is essentially the inverse of the reduced-parameter-space
Fisher informationUT(?)I(?)U(?) from section 4.2.2.4 after the elimination of the
rows and columns corresponding to the channel gain parameters.
4.2.2.6 Semi-blind and unit modulus constraint
This model merges the models in (4.33) and (4.32). Without loss of generality,
assume the source elementsare known for the firstT time slots for eachsource. Then
the constraint Jacobian is F(?) =
?
??
I2TK?2TK 02TK?3K
2Re(S(T + 1)) 2Im(S(T + 1)) 0K?3K
... ...
2Re(S(N)) 2Im(S(N)) 0K?3K
?
??,
(4.36)
which has an orthonormal complement
U(?) =
?
??
??
??
??
??
02TK?K
?Im(S(1))
Re(S(1))
...
?Im(S(N))
Re(S(N))
I3K?3K
?
??
??
??
??
??
. (4.37)
129
The reduced FIM is then
U(?)I(?)U(?) =
?
??
??
??
??
G(T + 1) C(T + 1) B(T + 1)
G(T + 2) C(T + 2) B(T + 2)
... ... ...
G(N) C(N) B(N)
CT(T + 1) CT(T + 2) ??? CT(N) K? L
BT(T + 1) BT(T + 2) ??? BT(N) LT K?
?
??
??
??
??
(4.38)
where G(t) = Re[S?(t)MS(t)] and M is defined as in (4.25), where C(t) =
bracketleftbigIm[S?(t)M
t] Re[S?(t)Mt]
bracketrightbig and M
n is defined in (4.26), B(t) = Im[S?(t)Lt]
and Lt is defined in (4.27). To analytically invert this matrix, the Schur comple-
ment,
? =
?
??
??
??
K??
Nsummationdisplay
t=T+1
CT(t)G?1(t)C(t) L?
Nsummationdisplay
t=T+1
CT(t)G?1(t)B(t)
LT ?
Nsummationdisplay
t=T+1
BT(t)G?1(t)C(t) K??
Nsummationdisplay
t=T+1
BT(t)G?1(t)B(t)
?
??
??
??, (4.39)
is useful. If ? is partitioned into corresponding subblocks
bracketleftbigg?
K? ?L
?LT ?K?
bracketrightbigg
, then the
CCRB subblocks for the unknown elements are given by
CCRB(?) =
bracketleftBig
?K? ??LT??1K??L
bracketrightBig?1
CCRB(?) = bracketleftbig?K? ??L??1K??LTbracketrightbig?1
CCRB(
bracketleftbiggRe(s(t))
Im(s(t))
bracketrightbigg
) =
bracketleftbigg?Im(S(t))
Re(S(t))
bracketrightbigg
X(t)bracketleftbig?Im(S(t)) Re(S(t))bracketrightbig
where X(t) =G?1(t) +G?1(t)bracketleftbigC(t) B(t)bracketrightbig??1
bracketleftbiggCT(t)
BT(t)
bracketrightbigg
G?1(t).
Example 4.23. A distinct advantage of the CCRB, as implementedin this treatise,
is the ability to compare the performance potential of a number of different models
seamlessly. Suppose we consider M = 5 omni-directional sensors with a beamwidth
130
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
SOURCE 2 AOA (DEGREES)
RMS ERROR (DEGREES)
CRB ON SOURCE 2 AOA
SNR2 =  5 dB
SNR2 = 15 dB
BLIND
CM SIGNAL         
KNOWN SIGNAL      
Figure 4.6: CCRBs on AOA for blind, constant modulus, and known signal models.
of?23.6? inauniform linear array receivingK = 2 source signals overN = 100 time
samples. Then, it is possible to compare various communications design scenarios,
e.g.,
(a) the ?blind? case9: sk(1) known for k = 1,2,
(b) the unit modulus case: |sk(t)|2 = 1 for k = 1,2 and t = 1,...,100,
(c) the semiblind case: sk(t) known for k = 1,2 and t = 1,...,T = 20,
(d) the unit modulus and semiblind case: sk(t) known for k = 1,2 and t =
1,...,T = 20, and|sk(t)|2 = 1 for k = 1,2 and t = T + 1,...,100, and
(e) the known signal case: sk(t) known for k = 1,2 and t= 1,...,100.
9This is not a truly blind scenario, but is often referred to as blind in the literature.
131
0 5 10 15 20 25 300
0.1
0.2
0.3
0.4
0.5
0.6
0.7
RMS ERROR (DEGREES)
CRB ON SOURCE 2 AOA
SNR2 =  5 dB
SNR2 = 15 dB
SEMI?BLIND
CM SIGNAL + SEMI?BLIND 
KNOWN SIGNAL           
Figure 4.7: CCRBs on AOA for semiblind, constant modulus + semiblind, and
known signal models.
132
Figure 4.6 displays a comparison between blind, unit modulus constraints, and
known signal model designs of the CCRBs on angle-of-arrival (AOA) estimation for
the second source signal over varying directions and different signal-to-noise ratios
(SNRs) with the first source signal arriving at 0?. Figure 4.7 displays a comparison
of CCRBs on AOA estimation between semiblind constraints, unit modulus with
semiblind constraints, and known signal model designs. And finally, figure 4.8 dis-
plays CCRBs on signal phase estimation between blind, unit modulus constraints,
semiblind constraints, and a mixture of unit modulus and semiblind constraints.
The known signal model is the best case scenario for AOA estimation potential
and is a useful guideline for more desirable scenarios where information (data or un-
known parameters) is included in the transmission. Figures 4.6 and 4.7 demonstrate
the characteristic loss of performance when the sources? AOAs differ by roughly the
beamwidth. The value of the unit modulus constraint is evident when the sources?
AOAs are closely spaced as the CCRB performance potential approximates that of
the known signal model. In figures 4.7 and 4.8, the estimation potential actually
improves for closely space sources with the semiblind and unit modulus constraint
mixture.
4.3 Discussion
This chapter includes extensions of only a brief sampling of my prior research
[59, 39, 48, 51, 39, 48, 59, 49] as it relates to the practical application of the CCRB.
In this chapter, the convolutive mixture model and the calibrated array model were
133
0 5 10 15 20 25 300
5
10
15
20
25
30
35
40
RMS ERROR (DEGREES)
MEAN CRB ON SOURCE 2 SIGNAL PHASE
SNR2 =  5 dB
20% TRAINING
BLIND    
CM SIGNAL             
SEMI?BLIND            
CM SIGNAL + SEMI?BLIND
Figure 4.8: CCRBs on signal phase for blind, constant modulus, semiblind, and
constant modulus + semiblind models.
134
treated as base models for which the Fisher information was derived a single time
and then a series of variations on the models in the form of differentiable parametric
constraints were considered. This approach presents a simple procedure to compare
and contrast a large class of constraints, essentially different models, in an efficient
manner to determine the value of particular formulation in terms of performance
potential as measured in the CCRB.
135
Appendix A
Appendices
A.1 A proof of the CCRB using the Chapman-Robbins version of
the Barankin bound
Gorman and Hero developed a CCRB using the multiparameter version of
the Hammersley-Chapman-Robbins bound (HCRB) [16, 26]. However, the result
produced a variant form of the CCRB
I?1(?)?I?1(?)FT(?)parenleftbigF(?)I?1(?)FT(?)parenrightbig?1F(?)I?1(?),
which requires a nonsingular FIM. What follows is a shorter variation of their ap-
proach that does not assume a nonsingular FIM, starting with a brief description of
the HCRB.
Rather than relying on the Fisher score, which is the derivative of the log-
likelihood, i.e.,
s(x;?) = ??? logp(x;?) = 1p(x;?)?p(x;?)??
the regularity conditions requiring a differentiable likelihood can be relaxed by con-
sidering finite differences, i.e., for each i= 1,...,m,
1
p(x;?)
p(x;?+epsilon1iei)?p(x;?)
epsilon1i
where the ei are canonical unit vectors. If the likelihood is differentiable then the
limit as each epsilon1i ? 0 is the Fisher score. Of course, the finite differences need not
136
be with respect to the canonical axis and the number of finite differences need not
be the same as the dimension of the parameters. This is the generalization of the
CRB known as the Hammersley-Chapman-Robbins (HCR) version of the Barankin
bound.
Theorem A.1. If t(x) be an unbiased estimate of h(?)?h(Rm)?Rt, and ? =
bracketleftbig?(1),...,?(p)bracketrightbig is a matrix whose columns are test points in Rm, all distinct from
each other as well as from ?, then the variance of t(x) is bounded below by the
inequality
Var(t(x))? sup
?(1),...,?(p),p
?(?,?)??1(?,?)?T(?,?)
where ?(?,?) is called a translation matrix defined by
?(?,?) =bracketleftbigh(?(1))?h(?), ???, h(?(p))?h(?)bracketrightbig
and ?(?,?) is called an HCR information matrix defined by
?ij(?,?) = E?p(x;?
(i))?p(x;?)
p(x;?)
p(x;?(j))?p(x;?)
p(x;?) .
This result encompasses the CRB result when the test points satisfy ?(i) =
?+epsilon1iei and the epsilon1i ?0 for i = 1,...,m = p, i.e., as a properly chosen set of test
points approach the parameter. If the set of vectors ?(1)??,...,?(m)?? span
an m-dimensional space, then the limit as the finite differences approach derivatives
will still obtain the CRB.
Since f(?) = 0 it makes sense to restrict the test points to also satisfy the
constraints,f(?(i)) = 0, and examine the limit of the HCRB as the finite differences
137
approach derivatives. The Taylor series approximation of f(?(i)) about ? is
f(?(i)) =f(?)F(?)parenleftbig?(i)??parenrightbig+o(||?(i)??||).
Since f(?(i)) = f(?) = 0, then ?(i)?? almost entirely resides in null(F(?)). So
without loss of generality we can allow
?(i) =?+?iui(?) +o(||?(i)??||)
where ui(?) is the ith column of the matrix U(?) satisfying (3.6). Then
p(x;?(i))?p(x;?)
?ip(x;?) =
p(x;?+?iui(?)+o(||?(i)??||))?p(x;?)
?ip(x;?)
? uTi (?)?p(x;?)?? 1p(x;?) =uTi (?)s(x;?)
and
?(i)??
?i =ui(?) +
1
?io(||?
(i)??||)?ui(?)
as ?i ? 0. This is true for any i, so if the test points are chosen such that each
column of U(?) is used, this gives us the CCRB
U(?)parenleftbigUT(?)I(?)U(?)parenrightbig?1UT(?)
as the limit of a constrained HCRB when the finite differences become derivatives.
A.2 A proofof theCCRB using themethodof implicit differentiation
Suppose ? is restricted to the zeros of f : Rm ?Rk with Jacobian F(?) =
?f(?)
??T having rank k whenever f(?) = 0. The method of implicit differentiation
assumes that parameters that would be eliminated under a reparameterization can
138
be written in terms of the remaining parameters, since the conditions satisfy the
implicit function theorem. The actual function generally remains unknown, but it?s
derivative is calculated by first taking partial derivatives of constraint function and
using linear algebra to solve for the partial of the eliminated parameters in terms of
the remaining parameters. This approach was also used by Marzetta [47, proof of
theorem 1] to prove the regularity conditions given in (3.20).
The parameter vector may be separated as ? =
bracketleftbigg?
1
?2
bracketrightbigg
and the constraint f
may be rewritten as fstar : Rm?k?Rk ?Rk via the mapping fstar(?1,?2) = f(
bracketleftbigg?
1
?2
bracketrightbigg
).
Then the Jacobian of f can be represented as
F(?) =bracketleftbigfstar?1(?1,?2) fstar?2(?1,?2)bracketrightbig
where fstar?i(?1,?2) = ???T
i
fstar(?1,?2) for each i= 1,2.
Without loss of generality, assume ?2 ?Rk is a function of ?1 ?Rm?k, i.e.,
?2 =?2(?1) is an implicit function. Therefore, fstar is implicitly only a parameter of
?1 and
?
??T1 f
star(?1,?2(?1)) =fstar
?1(?1,?2(?1)) +f
star
?2(?1,?2(?1))
??2(?1)
??T1 .
If this derivative is only taken where fstar(?1,?2) = 0, then in matrix form,
bracketleftbigf
?1(?1,?2) f?2(?1,?2)
bracketrightbig
bracketleftBigg
Im?k,m?k
??2(?1)
??1
bracketrightBigg
= 0.
The first matrix above is F(?); the second matrix above consists of m?k linearly
independent columns which exist in the null space of the row vectors of F(?), i.e.,
the second matrix is merely some transformation of some matrix U(?) defined as
in (3.6).
139
A.3 Alternative proof of asymptotic normality
Crowder [18] proved the following theorem.
Theorem A.2 (Crowder). If
1. there is a consistent solution ( ??n, ??n) of the likelihood equations,
2. s(x;?0) d?N(0,I(?0)),
3. D?1( ??n)
parenleftBig
I(?0) + ???Ts(x;?0, ??n)
parenrightBig p
?0,
4. Q( ??n)?Q(?0) p?0, and
5. detQ( ??n)?K <?
6. detD?1( ??n)FT( ??n)parenleftbigF(?, ??n)D?1( ??n)FT( ??n)parenrightbig?1 ?K <?
wheres(x;?0,?) is a matrix inthe form of the Fisher scores(x;?) but with eachrow
evaluated at possibly different points on the line between ?0 and ??n, and similarly
for F(?, ??n), and Q(?) =D?1(?)FT(?)parenleftbigF(?0,?)D?1(?)FT(?)parenrightbig?1. Then
?nparenleftBig????parenrightBig d?NparenleftBig0,D?1(?)?D?1(?)FT(?)parenleftbigF(?)D?1(?)FT(?)parenrightbig?1F(?)D?1(?)parenrightBig
where D(?) = I(?) +FT(?)KF(?) for an arbitrary positive semi-definite matrix
K.
Crowder?s asymptotic normality theorem shows that variance of the CMLE
satisfies
Var(?n??n) d?D?1(?0)?D?1(?0)FT(?0)parenleftbigF(?0)D?1(?0)FT(?0)parenrightbig?1F(?0)D?1(?0)
140
as n??, where D(?) =I(?) +FT(?)KF(?). This asymptotic variance has the
exact structure as the Marzetta form of the CCRB whenK = 0 and the FIM is full
rank. Applying the algebraic identity of Lemma 3.8, then
Var(?n??n) d?U(?0)parenleftbigUT(?0)D(?0)U(?0)parenrightbig?1UT(?0).
It only remains to note
UT(?0)D(?0)U(?0) =UT(?0)I(?0)U(?0)
since F(?)U(?) = 0.
141
Appendix B
Proofs of Convergence Properties of Constrained Scoring
Theorem (Theorem 3.33). If for any iterate ??(k) ? ?f there does not exist an
?(k) > 0 that satisfies (3.37), then ??(k) is a stationary point.
Proof. Let ????f and define ??(?) = pibracketleftbig??+?CCRB( ??)s(x; ??)bracketrightbig. By a property of
the natural projection of convex sets,
vextenddoublevextenddouble
vextenddouble??(?)? ??
vextenddoublevextenddouble
vextenddouble
I(??)
??
vextenddoublevextenddouble
vextenddoubleCCRB( ??)s(x; ??)
vextenddoublevextenddouble
vextenddouble
I( ??)
.
Hence it is sufficient to show there exists an ?> 0 such that
parenleftbiglogp(x; ??(?))?logp(x; ??)parenrightbig
? ??s
T(x; ??)CCRB( ??)s(x; ??).
To show this by contradiction, assume not and take the limit as ??0. Then
sT(x; ??)CCRB( ??)s(x; ??) ? ?sT(x; ??)CCRB( ??)s(x; ??)
0 ? (??1)sT(x; ??)CCRB( ??)s(x; ??).
This inequality implies CCRB( ??)s(x; ??) = 0 and s(x; ??) ? span(FT( ??)) since
?< 1, so ?? satisfies the stationarity condition (3.32).
Theorem (Theorem 3.34). The sequence{p(x; ??(k))}is a monotone increasing se-
quence. Furthermore, if p(x;?) is bounded above, then{p(x; ??(k))}converges.
Proof. Since ? ? 0 and bardbl??(k+1)? ??(k)bardbl2I( ??(k)) ? 0, then by the rule in (3.37), the
value of the likelihood function can only increase after each iteration. The second
142
statement is a consequence of the monotone convergence principle, i.e., a bounded
monotone sequence converges [38, p. 44, theorem 2-6].
Theorem (Theorem 3.35). If the likelihood p(x;?) is bounded above, then the
sequence
{logp(x; ??(k+1))?logp(x; ??(k))}
vanishes.
Proof. Since{p(x; ??(k))}converges, then p(x; ??(k+1))p(x; ??(k)) ?1.1 And since log(?) is contin-
uous, then log p(x; ??(k+1))p(x; ??(k)) ?0.
Theorem (Theorem 3.36). If the likelihood p(x;?) is bounded above, then the
sequence
{bardbl??(k+1)? ??(k)bardblI(??(k))}
vanishes as k??.
Proof. Again, by the rule in (3.37), the sequence{bardbl??(k+1)? ??(k)bardbl2I( ??(k))}is bounded
above by the product of a bounded sequence {?(k)} and a vanishing sequence
{logp(x; ??(k+1))?logp(x; ??(k))}, and clearly each element of the sequence is non-
negative. Hence, by the squeezing theorem2, the sequence vanishes as k ? ?.
Theorem (Theorem 3.37). If ???(1) is compact and convex, then limit points of the
sequence{??(k)}are also stationary points.
1If ak?a, bk?b, and bknegationslash= 0 for any k, then ak
bk ?
a
b [38, p. 41, theorem 2-4(d)].2
If 0?ak?bk and bk?0, then ak?0 [38, p. 43, theorem 2-5(c)].
143
Proof. Let ?star be a limit point of the sequence{??(k)}. Then by virtue of Bolzano-
Weierstrass [38, p. 54, theorem 2-14], there exists a convergent subsequence{??(ki)}
that converges to?star. Additionally, since{?(ki)}is a bounded sequence, it contains a
convergent subsequence{?(kij)}with a limit point we shall denote?star. It is still true
that ??(kij) ??star [38, p.49, theorem 2-10]. Then we can bound the norm-distance
between pibracketleftbig?star +?starCCRB(?star)s(x;?star)bracketrightbig and ?star using the triangle inequality, e.g.,
vextenddoublevextenddoublepibracketleftbig?star + ?starCCRB(?star)s(x;?star)bracketrightbig??starvextenddoublevextenddouble
I(?star)
?
vextenddoublevextenddouble
vextenddoublepibracketleftbig?star + ?starCCRB(?star)s(x;?star)bracketrightbig?pi
bracketleftBig
??(kij) + ?(kij)CCRB( ??(kij))s(x; ??(kij))
bracketrightBigvextenddoublevextenddouble
vextenddouble
I(?star)
+
vextenddoublevextenddouble
vextenddoublepi
bracketleftBig
??(kij) +?(kij)CCRB( ??(kij))s(x; ??(kij))
bracketrightBig
? ??(kij)
vextenddoublevextenddouble
vextenddoubleI(?star) +
vextenddoublevextenddouble
vextenddouble??(kij)??star
vextenddoublevextenddouble
vextenddoubleI(?star)
?
vextenddoublevextenddouble
vextenddouble?star + ?starCCRB(?star)s(x;?star)? ??(kij)??(kij)CCRB( ??(kij))s(x; ??(kij))
vextenddoublevextenddouble
vextenddoubleI(?star)
+
vextenddoublevextenddouble
vextenddouble??(kij+1)? ??(kij)
vextenddoublevextenddouble
vextenddouble
I(?star)
+
vextenddoublevextenddouble
vextenddouble??(kij)??star
vextenddoublevextenddouble
vextenddouble
I(?star)
.
The second inequality is a result of a property of projections on convex sets for the
first term and the definition of our method of scoring with constraints for the second
term. Note this second term will vanish by theorem 3.36 as kij ??. Also, the
third term will vanish as kij ??since?star is the limit of the sequence{??(kij)}. The
first term is bounded, using the triangle inequality again, as in
vextenddoublevextenddouble
vextenddouble?star +?starCCRB(?star)s(x;?star)? ??(kij)??(kij)CCRB( ??(kij))s(x; ??(kij))
vextenddoublevextenddouble
vextenddoubleI(?star)
?
vextenddoublevextenddouble
vextenddouble??(kij)??star
vextenddoublevextenddouble
vextenddouble
I(?star)
+
vextenddoublevextenddouble
vextenddouble?starCCRB(?star)s(x;?star)??(kij)CCRB( ??(kij))s(x; ??(kij))
vextenddoublevextenddouble
vextenddouble
I(?star)
.
Again, this first term will vanish as kij ??. This last term, using the triangle
144
inequality, satisfies
vextenddoublevextenddouble
vextenddouble?starCCRB(?star)s(x;?star)??(kij)CCRB( ??(kij))s(x; ??(kij))
vextenddoublevextenddouble
vextenddouble
I(?star)
?
vextenddoublevextenddouble
vextenddouble?starCCRB(?star)s(x;?star)??(kij)CCRB(?star)s(x;?star)
vextenddoublevextenddouble
vextenddouble
I(?star)
+
vextenddoublevextenddouble
vextenddouble?(kij)CCRB(?star)s(x;?star)??(kij)CCRB(?(kij))s(x;?(kij))
vextenddoublevextenddouble
vextenddouble
I(?star)
?
vextenddoublevextenddouble
vextenddouble?(kij)??star
vextenddoublevextenddouble
vextenddouble
I(?star)
bardblCCRB(?star)s(x;?star)bardblI(?star)
+
vextenddoublevextenddouble
vextenddouble?(kij)
vextenddoublevextenddouble
vextenddouble
I(?star)
vextenddoublevextenddouble
vextenddoubleCCRB(?(kij))s(x;?(kij))?CCRB(?star)s(x;?star)
vextenddoublevextenddouble
vextenddouble
I(?star)
.
The second inequality used the distributive property of norms [19, p.170, theorem
6.9.2]. By compactness, {bardblCCRB(?star)s(x;?star)bardblI(?star)} is bounded. So the first term
above will vanish as kij ??since?star is the limit of the sequence{?(kij)}. Similarly,
{bardbl?(kij)bardblI(?star)}is a bounded sequence, and so the second term above will vanish as
kij ??since CCRB(?(kij))?CCRB(?star) ands(x;?(kij))?s(x;?star) by continuity
[38, p.78, corollary 4-2]. Therefore,
pibracketleftbig?star +?starCCRB(?star)s(x;?star)bracketrightbig=?star
and one of the following holds:
(a) ?star = 0,
(b) CCRB(?star)s(x;?star) = 0, or
(c) the step projection ?star +?starCCRB(?star)s(x;?star) is perpendicular to ?f at ?star.
This last case (c) is impossible since the step is directed by linear combinations of
the vectors ofU(?star) which are tangent to the constraint space at?star. This first case
(a) implies stationarity by applying continuity on the step size rule condition (3.37)
145
and then theorem 3.33. And (b) impliess(x;?star) is some linear combination of the
columns of FT(?star), i.e.,?star satisfies (3.32). Therefore,?star is a stationary point.
Theorem (Theorem 3.38). If ???(1) is compact for all sequences in a closed set of
?f and if there is a unique limit point ?star for all such sequences then lim
k??
??(k) =?star
for every sequence{??(k)}. Also, ?star is the maximum of p(x;?).
Proof. Let ??(2) be any point in the compact set ???(1). Since {??(k)} resides in a
compact set it has a limit point (Bolzano-Weierstrass [38, p. 52, theorem 2-12]),
which must be unique and therefore lim
k??
??(k) =?star. By theorem 3.34, p(x; ??(k+d))?
p(x; ??(k)), and by continuity p(x; ??(k)) ?p(x;?star). Hence, p(x;?star) ?p(x;?prime) for
every ?prime ????(1).
146
Appendix C
Proofs of Theorems in Chapter 4
Theorem (Theorem 4.7). The CFIM is singular and the dimension of its null space
is lower bounded as
nullity(I(?))?
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+, (C.1)
where (a)+ = a for a?0 and (a)+ = 0 for a< 0. This limit quantity is the nullity
lower bound (NLB).
Proof. This proof is by construction. We shall develop a null subspace of the sub-
matrix bracketleftbigQi Qjbracketrightbig of Q. In particular, consider the submatrix of consisting of the
ith source elements and j channel elements corresponding to the mth channel, i.e.,
bracketleftBig
S(i) H(j)(m)
bracketrightBig
. There exists three case to consider: (1) Li = Lj, (2) Li > Lj, and
(3) Li <Lj. But first, for use in the proof, define the M(Lj + 1)?(Lj?Li + 1)+
matrix
H(i)(j) =
?
??
??
?
H(i)(j)1
H(i)(j)2
...
H(i)(j)M
?
??
??
?
(C.2)
147
where H(i)(j)m is the (Lj + 1)?(Lj?Li + 1)+ matrix
H(i)(j)m =
?
??
??
??
??
??
??
?
h(i)m (0) 0 ??? 0
h(i)m (1) h(i)m (0) ??? 0
... h(i)
m (1)
... 0
h(i)m (Li) ... ??? h(i)m (0)
0 h(i)m (Li) ??? h(i)m (1)
... ... ??? ...
0 0 ??? h(i)m (Li)
?
??
??
??
??
??
??
?
. (C.3)
ThenH(i)(j)m =h(i)m and H(i)(j) =h(i) from the secondary vector-matrix model in (4.6)
if Li = Lj. And also define the (N +Lj)?(Li?Lj + 1)+ matrix
S(i)(j) =
?
??
??
s(i)(?Lj) s(i)(?Lj?1) ??? s(i)(?Li)
s(i)(?Lj + 1) s(i)(?Lj) ??? s(i)(?Li + 1)
... ... ... ...
s(i)(N?1) s(i)(N?2) ??? s(i)(N?Li +Lj?1)
?
??
??. (C.4)
Then S(i)(j) =s(i) if Li = Lj. Now consider the cases:
(1) From the dual interpretation of the model in (4.3) and (4.6), it is clear that
H(j)(m)s(i) =S(i)h(j)m ; hence,
bracketleftBig
IM ?S(i) H(j)M
bracketrightBigbracketleftBiggH(j)
(i)
?S(i)(j)
bracketrightBigg
= 0.
Also, (Li?Lj +1)+ = dim(
bracketleftbiggh(j)
?s(i)
bracketrightbigg
) = 1 unlessh(j) = 0 ands(i) = 0, in which
case, nullity(
bracketleftBig
IM ?S(i) H(j)M
bracketrightBig
) > (Li?Lj + 1)+.
(2) H(j)(i)m in (C.3) has rank (Li?Lj + 1) unless h(j)m = 0, which is impossible
by definition. Likewise, S(i)(j) is full column rank unless s(i) has fewer than
(Li?Lj + 1) modes or N <Li?2Lj + 1 (see theorem 4.3). Therefore, since
bracketleftBig
IM ?S(i) H(j)M
bracketrightBigbracketleftBiggH(j)
(i)
?S(i)(j)
bracketrightBigg
= 0
then nullity(
bracketleftBig
IM ?S(i) H(j)M
bracketrightBig
)?(Li?Lj + 1)+.
148
(3) Finally, nullity(
bracketleftBig
IM ?S(i) H(j)M
bracketrightBig
)?0?(Li?Lj + 1).
Theorem (CFIM NLB necessary conditions, Theorem 4.13). The M-channel K-
source FIR system Fisher information matrix has a nullity of exactly the NLB in
(4.13) only if
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K +
Ksummationdisplay
j=1
Lj,
(c) pk ?Lk + 2 for k = 1,...,K or pk ?1 if Lk = 0,
(d) N?K +
Ksummationdisplay
j=1
Lj, and
(e) M >K.
Proof. If any of these conditions fail, then the null space of I(?) is greater than the
NLB.
(a) If H(z) is reducible or not column-reduced, then by theorem 4.2 HM is not
full column rank. Let v?null(HM) and partition v as
v =
?
??
??
v(1)
v(2)
...
v(K)
?
??
??
where v(k) is a N +Lk length vector. (It is assumed that v is independent of
s otherwise E?y = 0 and the model itself is not identifiable.) Then
v? =
?
??
??
??
0(1)
v(1)
...
0(K)
v(K)
?
??
??
???null(I(?))
149
where 0(k) is a M(Lk + 1) length zero vector. Assume v? is a linear com-
bination of the columns of N in (4.14). Then for some k the columns of
bracketleftBig
H(1)(k) H(2)(k) ??? H(K)(k)
bracketrightBig
must be linearly dependent. (Otherwise without
this assumption, then v? /?span(N). This corresponds to the 0(k) subvector
in v?.) Let
u(k) =
?
??
??
?
u(1)(k)
u(2)(k)
...
u(K)(k)
?
??
??
?
?null(
bracketleftBig
H(1)(k) H(2)(k) ??? H(K)(k)
bracketrightBig
)
with u(i)(k) a (Lk?Li + 1)+ length vector. Then
U(k) =
?
??
??
?
U(1)(k)
U(2)(k)
...
U(K)(k)
?
??
??
?
?null(HM)
where
U(i)(k) =
?
??
??
?
u(i)T(k) 0 ??? 0
0 u(i)T(k) ??? 0
... ... ...
0 ??? u(i)T(k)
?
??
??
?
N+Li?N+Lk
.
(If Li > Lk then U(i)(k) is a null matrix. Also, U(k)(k) = u(k)IN+Lk?N+Lk.) The
matrixU(k) can be arranged to create N+Lk linearly independent columns in
the null space of I(?), where the submatrices U(i)(k) correspond to the rows of
N containing
bracketleftBig
?S(1)(i) ?S(2)(i) ??? ?S(K)(i)
bracketrightBig
, which has rank at most
Ksummationdisplay
j=1
(Lj?
Li+1)+ ?
Ksummationdisplay
j=1
(Lj +1). For the columns ofN to be a basis of the null space of
I(?), it isneededthenthat
Ksummationdisplay
j=1
(Lj+1)?
Ksummationdisplay
j=1
(Lj?Lk+1)+ ?N+Lk ?N+Li,
150
which implies at most N = K +
Ksummationdisplay
j=1
Lj contradicting theorem 4.12 unless all
the channel orders are zero. In this latter case, then note that
N?
?
??
??
?
IK?K?
?
??
??
?
u(1)(k)
u(2)(k)
...
u(K)(k)
?
??
??
?
?
??
??
?
will not be full rank, and hence neither can N be so.
(b) If ptotal < K +
Ksummationdisplay
k=1
Lk, then there exists v? null(S) where the matrix S =
bracketleftbigS(1) S(2) ??? S(K)bracketrightbig. If v is partitioned as
v =
?
??
??
v(1)
v(2)
...
v(K)
?
??
??
where v(k) is a Lk + 1 length vector, then
v? =
?
??
??
??
1M?1?v(1)
0(1)
...
1M?1?v(K)
0(K)
?
??
??
???null(I(?))
with 0(j) is an N + Lj length zero vector. If v? ? span(N), then for some
k we have rank(
bracketleftBig
S(1)(k) S(2)(k) ??? S(K)(k)
bracketrightBig
) <
Ksummationdisplay
j=1
(Lj ?Lk + 1)+. Then by a
construction similar to part (a), nullity(S)?Lk + 1 and this contribution to
the null space of I(?) has rank at least M(Lk + 1), with submatrices that
coincide with
bracketleftBig
H(1)(k) H(2)(k) ??? H(K)(k)
bracketrightBig
, which has rank at most
Ksummationdisplay
j=1
(Lk ?
Lj + 1)+ ?
Ksummationdisplay
j=1
(Lk + 1) = K(Lk + 1) <M(Lk + 1) for any Lk since M >K.
Therefore, nullity(I(?)) >
Ksummationdisplay
i=1
Ksummationdisplay
j=1
(Li?Lj + 1)+.
151
(c) If pk < Lk + 1 then by [31, lemma 1], S(k) and, consequently S, has a null
space and the argument in (b) applies. So assume pk = Lk + 1. If N <
Lk + 1 (and Lk negationslash= 0) then S(k) has a null space [76, lemma 1], so assume
N ? Lk + 1. From [31, the proof of theorem 1], it is possible to construct
a v independent from s(k) such that span(V) = span(S(k)) for V defined
similarly asS(k). So for anyh(k) there exists ah? such thatH(k)M v = (IM?M?
V)h(k) = (IM?M ?S(k))h?. Therefore both
bracketleftbigg?v
h?
bracketrightbigg
and
bracketleftbigg?s(k)
h(k)
bracketrightbigg
reside in
null(
bracketleftBig
H(k)M (IM?M ?S(k))
bracketrightBig
), which increases the nullity lower bound by at
least one.
(d) This is a looser bound and hence required by theorem 4.12.
Theorem (CFIM NLB sufficiency conditions, theorem 4.14). The M-channel K-
source FIR system FIM has a nullity of exactly the NLB in (4.13) if
(a) H(z) is irreducible and column-reduced,
(b) ptotal?K + (K + 1)
Ksummationdisplay
j=1
Lj,
(c) pk ?Lk + 1+
Ksummationdisplay
j=1
Lj for k = 1,...,K,
(d) N?K + (K + 2)
Ksummationdisplay
j=1
Lj, and
(e) M >K.
This result is conceptually easier to prove from yet another alternative matrix
model from the ones in (4.3) or (4.6) using S(n) as defined in (4.9). Then define
152
the (Lk + 1+n)?(n+ 1) matrix
H(k)(m)(n) =
?
??
??
?
h(k)m (0)
... ... h(k)
m (0)
h(k)m (Lk) ... ...
h(k)m (Lk)
?
??
??
?
,
which is the impulse response for the ith subchannel of the kth source. Then define
H(k)(n) =
bracketleftBig
H(k)(1)(n),...,H(k)(M)(n)
bracketrightBig
and the (K(n+1)+
Ksummationdisplay
k=1
Lk)?M(n+1) matrix
H(n) =
?
?
H(1)(n)
???
H(K)(n)
?
?.
The observations for the mth channels (receiver) can be collected into the (N?n)?
(n+ 1) matrix
Y(m)(n) =
?
?? ym(n) ??? ym(0)... ...
ym(N?1) ??? ym(N?1?n)
?
??
with Y(n) = bracketleftbigY(1)(n),...,Y(M)(n)bracketrightbig. The noise matrix W(n) is defined similarly.
Then the alternative model is
Y(n) =S(n)H(n)+W(n). (C.5)
Before this theorem is proven, a lemma will be needed. This lemma is a
generalization of a result in [52, theorem 3]. The proof was originally shown in [49,
Appendix].
Lemma C.1. AssumeH(n) be full row rank andhprime(k) be any nontrivialM(Lk +1)
length vector, and defineHprime(k)(n?) to be theLk+1+n??M(1+n?) matrix composed
from hprime(k) as in (4.4) and (4.5). Then the following two statements are equivalent:
(i) corange{Hprime(k)(n?)}?corange{H(1)(n?),...,H(K)(n?)}= corange{H(n?)}.
153
(ii) hprime(k) ?range{H(1)(k),...,H(K)(k)}.
We shall prove a more general version of the lemma, which is essentially an
extension of the identifiability theorem of [52, theorem 3] from the SIMO to the
MIMO scenario. First, defineh(j)(l) =
bracketleftBig
h(j)(1)(l),...,h(j)(M)(l)
bracketrightBigT
and theMn?(n+Lj)
matrix
H(j)(n) =
?
??
??
h(j)(Lj) h(j)(Lj?1) ??? h(j)(0)
h(j)(Lj) h(j)(Lj?1) ??? h(j)(0)
... ... ... ...
h(j)(Lj) h(j)(Lj?1) ??? h(j)(0)
?
??
??
so that H(N) = bracketleftbigH(1)(N),...,H(K)(N)bracketrightbig is similar to HM in the original model
(4.3), except for the order of the rows. Let L satisfy min
j
{Lj}?L? max
j
{Lj}.
Also define HL as the M(L+ 1)?(L?Lj + 1)+ matrix
H(j)L =
?
??
h(j) 0M?1 ??? 0M(L?Lj?1)?1
0M?1 h(j) ... 0M?1
0M(L?Lj?1)?1 0M(L?Lj?1)?1 h(j)
?
??
where h(j) = bracketleftbigh(j)(0)T,...,h(j)(Lj)TbracketrightbigT, an M(Lj + 1)?1 matrix. (The matrix is
not to scale, i.e., 0M?1 does not align withh(j).) Note, that H(j)L is null for L<Lj.
Now, we restate the lemma as the following. Instead, we show the following are
equivalent:
(1) Range{Hprime(N)}?RangebraceleftbigH(1)(N),...,H(K)(N)bracerightbig= Range{H(N)}.
(2) hprime?Range
braceleftBig
H(1)L ,...,H(K)L
bracerightBig
.
Proof. Assume N >
Ksummationdisplay
j=1
Lj and H(N ?1) is full-column rank. Let hprime be any
M(L+ 1)?1 nonzero complex vector, and define Hprime(N) to be the MN?N +L
channel matrix composed from hprime. We?ll first assume (1), and show (2). Note that
H(j)(N) =
bracketleftbigg h(j)(0) p(j)(N?1)
0M(N?1)?1 H(j)(N?1)
bracketrightbigg
=
bracketleftbigg H(j)(N?1) 0
M(N?1)?1
q(j)(N?1) h(j)(Lj)
bracketrightbigg
154
where
p(j)(N?1) = bracketleftbigh(j)(1),...,h(j)(Lj),0M?N?1bracketrightbig, and
q(j)(N?1) = bracketleftbig0M?N?1,h(j)(0),...,h(j)(Lj?1)bracketrightbig.
Note statement (1) implies the first column of Hprime(N) satisfies
bracketleftbigg hprime(0)
0M(N?1)?1
bracketrightbigg
=
Ksummationdisplay
j=1
bracketleftbigg h(j)(0) p(j)(N?1)
0M(N?1)?1 H(j)(N?1)
bracketrightbigg
?
bracketleftBigg
?(j)0
a(j)0
bracketrightBigg
where ?(j)0 is a constant and a(j)0 is an N?1 +Lj ?1 vector. Hence, we have the
following two linear systems:
(A1) hprime(0) =
Ksummationdisplay
j=1
?(j)0 h(j)(0) +p(j)(N?1)a(j)0
(A2) 0M(N?1)?1 =
Ksummationdisplay
j=1
H(j)(N?1)a(j)0
But H(N?1) is full-column rank, thus a(j)0 = 0N+Lj?1?1 for all j. Therefore,
hprime(0) =
Ksummationdisplay
j=1
?(j)0 h(j)(0). (C.6)
Likewise, the next column of Hprime(N) is given by
?
?
hprime(1)
hprime(0)
0M(N?2)?1
?
?=
Ksummationdisplay
j=1
bracketleftbigg h(j)(0) p(j)(N?1)
0M(N?1)?1 H(j)(N?1)
bracketrightbigg
?
bracketleftBigg
?(j)1
a(j)1
bracketrightBigg
where, again, ?(j)1 is a constant and a(j)1 is an N +Lj?1?1 vector. Now, we have
the following two systems:
(B1) hprime(1) =
Ksummationdisplay
j=1
?(j)1 h(j)(0) +p(j)(N?1)a(j)1
(B2)
bracketleftbigg hprime(0)
0M(N?2)?1
bracketrightbigg
=
Ksummationdisplay
j=1
H(j)(N?1)a(j)1
155
Note (C.6) implies that
bracketleftbigg hprime(0)
0M(N?2)?1
bracketrightbigg
=
Ksummationdisplay
j=1
?(j)0 ?
bracketleftbigg h(j)(0)
0M(N?2)?1
bracketrightbigg
, i.e., it is a linear
combination of column vectors of H(N?1). Since H(N?1) is full-column rank,
then (B2) implies that a(j)1 =
bracketleftbigg ?(j)
0
0N+Lj?2?1
bracketrightbigg
. Thus, evaluating (B1), we have
hprime(1) =
Ksummationdisplay
j=1
?(j)1 h(j)(0) +?(j)0 h(j)(1). (C.7)
Continuing in this fashion, we arrive at the general expression
hprime(l) =
Ksummationdisplay
j=1
lsummationdisplay
i=0
?(j)l?ih(j)(i) (C.8)
for 0?l?L. For convenience, we define h(j)(i) to be null if i<Lj or i< 0. Now,
consider the (L+ 2)nd column of Hprime(N),
?
??
??
??
0M?1
hprime(L)
...
hprime(0)
0M(N?L?2)?1
?
??
??
??=
Ksummationdisplay
j=1
bracketleftbigg h(j)(0) p(j)(N?1)
0M(N?1)?1 H(j)(N?1)
bracketrightbigg
?
bracketleftBigg
?(j)L+1
a(j)L+1
bracketrightBigg
It follows that
(C1) 0M?1 =
Ksummationdisplay
j=1
?(j)L+1h(j)(0) +p(j)(N?1)a(j)L+1
(C2)
?
??
??
hprime(L)
...
hprime(0)
0M(N?L?2)?1
?
??
??=
Ksummationdisplay
j=1
H(j)(N)a(j)L+1
Since H(N?1) is full-column rank, equation (C.8) and (C2) imply that a(j)L+1 =
bracketleftBig
?(j)L ,...,?(j)0 ,01?M(N?L?2)
bracketrightBigT
. Applied to (C1) we have
0M?1 =
Ksummationdisplay
j=1
?(j)L+1h(j)(0) +?(j)L h(j)(1) +???+?(j)L?Lj+1h(j)(Lj)
156
Continuing in this fashion, we see that for L+ 1?l?N?1,
0M?1 =
Ksummationdisplay
j=1
Ljsummationdisplay
i=0
?(j)l?ih(j)(i) (C.9)
In particular, (C.8) and (C.9) show that the Nth column of Hprime(N) can be written
as ?
??
??
0M(N?L?1)?1
hprime(L)
...
hprime(0)
?
??
??=
Ksummationdisplay
j=1
H(j)(N)?
?
??
??
?(j)(N?1)
...
?(j)0
0Lj?1
?
??
?? (C.10)
Similarly, statement (1) implies the last column of Hprime(N) satisfies
bracketleftbigg0
M(N?1)?1
hprime(L)
bracketrightbigg
=
Ksummationdisplay
j=1
bracketleftbigg H(j)(N?1) 0
M(N?1)?1
q(j)(N?1) h(j)(Lj)
bracketrightbigg
?
bracketleftBigg
y(j)L
?(j)L
bracketrightBigg
where y(j)L is an N +Lj + 1?1 vector and ?(j)L a constant. Thus we have
(a1) 0M(N?1)?1 =
Ksummationdisplay
j=1
H(j)(N?1)y(j)L
(a2) hprime(L) =
Ksummationdisplay
j=1
q(j)(N?1)y(j)L +?(j)L h(j)(Lj)
Here, since H(N?1) is full rank, then (a1) implies that y(j)L = 0N+Lj?1?1, hence
hprime(L) =
Ksummationdisplay
j=1
?(j)L h(j)(Lj)
Proceeding as before, it is clear that
hprime(L?l) =
Ksummationdisplay
j=1
lsummationdisplay
i=0
?(j)L?lh(j)(Lj?l+i)
for 0?l?L. Thus, the Nth column of Hprime(N) can also be expressed as
?
??
??
0M(N?L?1)?1
hprime(L)
...
hprime(0)
?
??
??=
Ksummationdisplay
j=1
H(j)(N)?
?
??
??
0N+Lj?L?1?1
?(j)L
...
?(j)0
?
??
?? (C.11)
157
But H(N) is full-column rank, hence (C.10) and (C.11) imply that
?
??
??
?(j)(N?1)
...
?(j)0
0Lj?1
?
??
??=
?
??
??
0N+Lj?L?1?1
?(j)L
...
?(j)0
?
??
??
Thus, ?(j)l = 0 for all l?L+ 1. Now, consider the following three cases:
(i) L = Lj, then ?(j)0 = ?(j)L and ?(j)l ,?(j)L?l = 0 for 1?l?L.
(ii) L<Lj, then ?(j)l ,?(j)L?l = 0 for 0?l?L.
(iii) L>Lj, then?(j)l = ?(j)Lj+l for 0?l?L?Lj and?(j)l ,?(j)L?l = 0 forL?Lj+1?
l?L.
Define an (L?Lj + 1)+?1 vector ?(j) =
bracketleftBig
?(j)0 ,...,?(j)L?Lj
bracketrightBigT
for each j. If L<Lj,
let ? be null. Then,
hprime =
Ksummationdisplay
j=1
H(j)L ?(j)
or hprime?Range
braceleftBig
H(1)L ,...,H(K)L
bracerightBig
. This proves statement (2).
Now, conversely assume statement (2). Then hprime =
Ksummationdisplay
j=1
H(j)L ?(j) where ?(j) =
bracketleftBig
?(j)1 ,...,?(j)(L?Lj+1)+
bracketrightBigT
is an (L?Lj + 1)+?1 vector. Define
?(j)(N) =
?
??
??
??
?(j)1 ?(j)2 ??? ?(j)(L?Lj+1)+
?(j)1 ?(j)2 ??? ?(j)(L?Lj+1)+
... ... ... ...
?(j)1 ?(j)2 ??? ?(j)L?Lj+1)+
?
??
??
??
N+Lj?N+L
Note, if L<Lj, then ?(j)(N) = 0N+Lj?N+L. Then, it is simple to verify that
Hprime(N) =
Ksummationdisplay
j=1
H(j)(N)?(j)(N)
which proves statement (1).
158
Now having this lemma at hand, theorem 4.14 can be proven.
Proof. Let n? =
Ksummationdisplay
j=1
Lj. If (a)-(d) are satisfied, then we have column-rank(S(n?)) =
row-rank(H(n?) = K + (K + 1)n?. Let v?null(I(?)) where v is partitioned as
v =
?
??
??
??
hprime(1)
sprime(1)
...
hprime(K)
sprime(K)
?
??
??
??.
Then
Ksummationdisplay
k=1
(IM?M ?S(k))hprime(k) +H(k)M sprime(k) = 0, or in the alternative model in (C.5),
we have
Ksummationdisplay
k=1
S(k)(n?)Hprime(k)(n?) +
Ksummationdisplay
k=1
Sprime(k)(n?)H(k)(n?) = 0, or
bracketleftbigS(n?)Sprime(n?)bracketrightbigbracketleftbiggHprime(n?)
H(n?)
bracketrightbigg
= 0.
Therefore nullity(bracketleftbigS(n?)Sprime(n?)bracketrightbig)?rank(
bracketleftbiggHprime(n?)
H(n?)
bracketrightbigg
). Since
nullity(bracketleftbigS(n?)Sprime(n?)bracketrightbig) = columns(bracketleftbigS(n?)Sprime(n?)bracketrightbig)?rank(bracketleftbigS(n?)Sprime(n?)bracketrightbig)
? 2K + 2(K + 1)n??rank(S(n?))
= K + (K + 1)n?
and
rank(
bracketleftbiggHprime(n?)
H(n?)
bracketrightbigg
)?rank(H(n?)) =K + (K + 1)n?,
then nullity(bracketleftbigS(n?)Sprime(n?)bracketrightbig) = rank(
bracketleftbiggHprime(n?)
H(n?)
bracketrightbigg
) = K+(K+1)n?. Since rank(H(n?))
= K + (K + 1)n? then there exist some matrix T such that Hprime(n?) = TH(n?).
Thus, by the lemma, for each k thenhprime(k) =
Ksummationdisplay
j=1
H(j)(k)?(k,j) for some (Lk?Lj +1)+
length vector. In the alternative model of (C.5), Hprime(k)(n?) =
Ksummationdisplay
j=1
?(k,j)H(j)(n?)
159
where
?(k,j) =
?
??
??
??
??
??
??
??
?(k,j)(Lk?Lj+1)+
... ...
?(k,j)2 ?(k,j)(Lk?Lj+1)+
?(k,j)1 ... ?(k,j)(Lk?Lj+1)+
... ?(k,j)
2
...
?(k,j)1 ?(k,j)2
0 ?(k,j)1
?
??
??
??
??
??
??
??
Lk+1+n??Lj+1+n?
and this defines T as
T =
?
??
??
?(1,1) ?(1,2) ??? ?(1,K)
?(2,1) ?(2,2) ??? ?(2,K)
... ... ??? ...
?(K,1) ?(K,2) ??? ?(K,K)
?
??
??.
Then since
parenleftbigS(n?)T +Sprime(n?)parenrightbigH(n?) =bracketleftbigS(n?) Sprime(n?)bracketrightbigbracketleftbiggTH(n?)
H(n?)
bracketrightbigg
= 0
and H(n?) is full row rank, it can be seen that Sprime(n?) =?S(n?)T or Sprime(k)(n?) =
Ksummationdisplay
j=1
?S(j)(n?)?(j,k), which implies in the vector-matrix model in (4.3) that sprime(k) =
Ksummationdisplay
j=1
?S(j)(k)?(j,k). Let
? =
?
??
??
??
??
??
??
??
??
??
?(1,1)
?(1,2)
?(2,2)
?(2,1)
...
?(1,K)
...
?(K,K)
...
?(K,1)
?
??
??
??
??
??
??
??
??
??
.
Then v =N? and span(N)?null(I(?)).
160
Bibliography
[1] K. Abed-Meraim, Y. Hua, ?Strict identifiability of multichannel FIR systems:
further results and developments,? Proceedings of the International Conference
on Telecommunications, Melbourne, Australia, pp. 1029-1032, April 1997.
[2] John Aitchison, ?Large Sample Restricted Parametric Tests,? Journal of the
Royal Statistical Society, Series B (Methodological), Volume 24, Number 1, pp.
234-250, 1962.
[3] J. Aitchison, S.D. Silvey, ?Maximum likelihood estimation of parameters sub-
ject to restraints,? Ann. Math. Statist., Volume 29, pp. 813-828, 1958.
[4] J. Aitchison, S.D. Silvey, ?Maximum-likelihood estimation procedures and as-
sociated tests of significance,? Journal of the Royal Statistical Society, Series
B (Methodological), Volume 22, No. 1, pp. 154-171, 1960.
[5] Joshua N. Ash, ?On singular estimation problems in sensor localization sys-
tems,? Ph.D. Thesis, Ohio State University, 2007.
[6] Joshua N. Ash, Randolph L. Moses, ?On the relative error and absolute posi-
tioning errors in self-localization systems,? IEEE Transactions on Signal Pro-
cessing, Volume 56, Number 11, pp. 5668-5679, November 2008.
[7] W.J. Bangs, Array Processing with Generalized Beamformers, Ph.D. Thesis,
Yale University, New Haven, CT, 1971.
[8] E. W. Barankin, ?Locally best unbiased estimates,? Ann. Math. Statist., Vol-
ume 20, Number 4, pp. 477-501, 1949.
[9] Zvika Ben-Haim, Yonina Eldar, ?On the constrained Cram?er-Rao bound with
a singular Fisher information matrix,? IEEE Signal Processing Letters, Volume
16, Number 6, pp. 453-456, June 2009.
[10] Dimitri Bertsekas, Nonlinear Programming: 2nd Edition, Athena Scientific,
1999.
[11] A. Bhattacharyya, ?On some analogues of the amount of information and their
use in statistical estimation,? Sankhya, Volume 8, pp. 1-14, 1946.
[12] A.A. Borovkov, A. Moullagaliev, Mathematical Statistics. Boca-Raton, FL:
CRC, 1998.
161
[13] Roger Bowden, ?The theory of parametric identification,? Econometrica, Vol-
ume 41, Number 6, pp. 1069-1074, November 1973.
[14] George Casella, Roger L. Berger, Statistical Inference, 2nd Edition, Duxbury
Press, 2002.
[15] F. Cayre, C. Fontaine, T. Furon, ?Watermarkingsecurity: theory and practice,?
IEEE Transactions on Signal Processing, Volume 53, Number 10, pp. 3976-
3987, October 2005.
[16] D. G. Chapman, H. Robbins, ?Minimumvariance estimation without regularity
assumptions,? Ann. Math. Stat., Volume 22, pp. 581-586, 1951.
[17] Harald Cram?er, Mathematical Methods of Statistics, Princeton NJ: Princeton
University Press, 1946.
[18] Martin Crowder, ?On constrained maximum likelihood estimation with non-
i.i.d. observations,? Ann. Inst. Statist. Math., Volume 36, Part A, pp.239-249,
1984.
[19] J.N. Franklin, Matrix Theory. New York: Dover, 1993.
[20] James E. Gentle, Matrix Algebra: Theory, Computations, and Applications in
Statistics. New York: Springer, 2007.
[21] P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization. New York:
Academic, 1981.
[22] A.A. Goldstein, ?Convex programming in Hilbert space,? Bulletin of the Amer-
ican Mathematics Society, Volume 70, Number 5, pp. 709-710, 1964.
[23] John D. Gorman, Alfred O. Hero, ?Lower bounds for parametric estimation
with constraints,? IEEE Transactions on Information Theory, Volume 26,
Number 6, pp. 1285-1301, November 1990.
[24] A. Gorokhov, P. Loubaton, ?Subspace based techniques for blind separation of
convolutive mixtures with temporally correlated sources,? IEEE Transactions
on Circuits and Systems, Volume 44, pp. 813-820, September 1997.
[25] R.T. Haftka, Z. G?urdal, Elements of Structural Optimization. Norwell, MA:
Kluwer Academic, 1992.
162
[26] J. M. Hammersley,?On estimating restrictedparameters,? Journal of the Royal
Statistical Society, Series B, Volume 12, Number 2, pp. 192-240, 1950.
[27] Harrie Hendriks, ?A Cram?er-Rao type lower bound for estimators with values
in a manifold,? Journal of Multivariate Analysis, Volume 38, pp. 245-261, 1991.
[28] A.O. Hero, R. Piramuthu, J.A. Fessler, S.R. Titus, ?Minimax emission com-
puted tomography using high-resolution anatomical side information and B-
spline models,? IEEE Transactions on Information Theory, Volume 45, Num-
ber 3, pp. 920-938, April 1999.
[29] Bertrand Hochwald, Arye Nehorai, ?On identifiability and information-
regularity in parameterized normal distributions,? Circuits Systems Signal Pro-
cessing, Volume 16, Number 1, pp. 83-89, 1997.
[30] Yingbo Hua, ?Fast maximum-likelihood for blind identification of multiple FIR
channels,? IEEE Transactions on Signal Processing, Volume 44, Number 3, pp.
661-672, March 1996.
[31] Y. Hua, M. Wax, ?Strict identifiability of multiple FIR channels driven by an
unknown arbitrary sequence,?IEEE Transactions on Signal Processing, Volume
44, pp. 756-759, March 1996.
[32] Aditya K. Jagannatham, Bhaskar D. Rao, ?Cram?er-Rao lower bound for con-
strained complex parameters,? IEEE Signal Processing Letters, Volume 11,
Number 11, pp. 875-878, November 2004.
[33] Mortaza Jamshidian, ?On algorithms for restricted maximum likelihood esti-
mation,? Computational Statistics & Data Analysis, Volume 45, pp. 137-157,
2004.
[34] J?urgen Jost, Riemannian Geometry and Geometric Analysis, Third Edition,
Springer-Verlag, 2002.
[35] T. Kailath, Linear Systems. Englewood Cliffs, NH: Prentice-Hall, 1980.
[36] Steven M. Kay, Fundamentals of Statistical Signal Processing: Estimation The-
ory. Englewood Cliffs, NJ: Prentice Hall, 1993.
[37] C.G. Khatri, ?A note on a MANOVA model applied to problems in growth
curve,? Annals of the Institute of Statistical Mathematics, Volume 18, pp. 75-
86, 1966.
163
[38] J.R. Kirkwood, An Introduction to Analysis. Boston, MA: PWS Kent, 1995.
[39] Richard J. Kozick, Brian M. Sadler, ?Maximum likelihood array processing:
The semi-blind case,? Proceedings of the 2nd IEEE Workshop on Signal Pro-
cessing Advances in Wireless Communications (SPAWC), pp.70-73, 1999.
[40] Amir Leshem, Alle-Jan van der Veen, ?Direction-of-arrival estimation for con-
stant modulus signals,? IEEE Transactions on Signal Processing, Volume 47,
Number 11, pp. 3125-2319, November 1999.
[41] Jian Li, R.T. Compton, Jr., ?Maximum likelihood angle estimation for signals
with known waveforms,? IEEE Transactions on Signal Processing, Volume 41,
Number 9, pp. 2850-2863, September 1993.
[42] Z. Liu, G.B. Giannakis, S. Barbarossa, A. Scaglione, ?Transmit antennae space-
time block coding for generalized OFDM in the presence of unknown multi-
path,? IEEE Journal on Selected Areas in Communications, Volume 19, Num-
ber 7, pp. 1352-1364, July 2001.
[43] H. Liu, G. Xu, L. Tong, ?A deterministic approach to blind channel identifi-
cation of multi-channel FIR systems,? Proceedings of the IEEE International
Conference on Acoustics, Speech, and Signal Processing (ICASSP?94), Ade-
laide, Australia, Volume 4, pp. 581-584, April 1994.
[44] P. Loubaton, E. Moulines, ?On blind multiuser forward link channel estimation
by the subspace method: Identifiability results,? IEEE Transactions on Signal
Processing, Volume 48, pp. 2366-2376, August 2000.
[45] D.G. Luenberger, Introduction to Linear and Nonlinear Programming.Reading,
MA: Addison-Wesley, 1973.
[46] J.H. Manton, ?An improved least squares blind channel identification algo-
rithm for linear and affinely precoded communication systems,? IEEE Signal
Processing Letters, Volume 9, Number 9, pp. 282-285, September 2002.
[47] Thomas L. Marzetta, ?A simple derivation of the constrained multiple parame-
ter Cram?er-Rao bound,? IEEE Transactions on Signal Processing, Volume 41,
Number 6, pp. 2247-2249, June 1993.
[48] Terrence J. Moore, Brian M. Sadler, Richard J. Kozick, ?Regularity and strict
identifiability in MIMO systems,? IEEE Transactions on Signal Processing,
Volume 50, Number 8, pp. 1831-1842, August 2002.
164
[49] Terrence J. Moore, Brian M. Sadler, ?Sufficient conditions for regularity and
strict identifiability in MIMO systems,? IEEE Transactions on Signal Process-
ing, Volume 52, Number 9, pp. 2650-2655, September 2004.
[50] Terrence J. Moore, Richard J. Kozick, Brian M. Sadler, ?The constrained
Cram?er-Rao bound from the perspective of fitting a model,? IEEE Signal Pro-
cessing Letters, Volume 14, Number 8, pp. 564-567, August 2007.
[51] Terrence J. Moore, Brian M. Sadler, Richard J. Kozick, ?Maximum-likelihood
estimation, the Cram?er-Rao bound, and the method of scoring with parameter
constraints,? IEEE Transactions on Signal Processing, Volume 56, Number 3,
pp. 895-908, March 2008.
[52] E. Moulines, P. Duhamel, J. Cardoso, S. Mayrargue, ?Subspace methods for the
blind identification of multichannel FIR filters,? IEEE Transactions on Signal
Processing, Volume 43, pp. 516-525, February 1995.
[53] J.M. Oller, J.M. Corcuera, ?Intrinsic analysis of statistical estimation,? The
Annals of Statistics, Volume 23, Number 5, pp. 1562-1581, October 1995.
[54] M.R. Osborne, ?Fisher?s method of scoring,? Int. Stat. Rev., Volume 60, pp.
99-117, 1992.
[55] M.R.Osborne, ?Scoring withconstraints,? ANZIAM Journal, Volume42, Num-
ber 1, pp. 9-25, July 2000.
[56] Calyampudi Radakrishna Rao, ?Information and the accuracy attainable in
the estimation of statistical parameters,? Bulletin of the Calcutta Mathematical
Society, Volume 37, pp. 81-89, 1945.
[57] Alvin C. Rencher, Linear Models In Statistics, John Wiley & Sons, New York,
2000.
[58] Thomas J. Rothenberg, ?Identification in parametric models,? Econometrica,
Volume 39, Number 3, pp. 577-591, May 1971.
[59] Brian M. Sadler, Richard J. Kozick, Terrence Moore, ?Bounds on bearing and
symbol estimation with side information,? IEEE Transactions on Signal Pro-
cessing, Volume 49, Number 4, pp. 822-834, April 2001.
[60] L.L. Scharf, L.T. McWhorter, ?Geometry of the Cram?er-Rao bound,? Con-
ference Proceedings, IEEE Sixth SP Workshop on Statistical Signal and Array
Processing, pp. 5-8, October 1992.
165
[61] George A.F. Seber, Alan J. Lee, Linear Regression Analysis: 2nd Edition,
Wiley-Interscience, 2003.
[62] Jun Shao, Mathematical Statistics. New York, NY: Springer-Verlag, 2003.
[63] S.D. Silvey, ?The Lagrange multiplier test,? Ann. Math. Statist., Volume 30,
pp. 389-407, 1959.
[64] D.R. Smart, Fixed Point Theorems, Cambridge University Press, 1980.
[65] M. Spivak, Calculus on Manifolds. Reading, MA: Addison-Wesley, 1965.
[66] Petre Stoica, Thomas L. Marzetta, ?Parameter estimation problems with sin-
gular information matrices,? IEEE Transactions on Signal Processing, Volume
49, Number 1, pp. 87-90, January 2001.
[67] Petre Stoica, Arye Nehorai, ?MUSIC, maximum likelihood, and Cram?er-Rao
bound,? IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol-
ume 37, Number 5, pp. 720-741, May 1989.
[68] Petre Stoica, Boon Chong Ng, ?On the Cram?er-Rao bound under parametric
constraints,? IEEE Signal Processing Letters, Volume 5, Number 7, pp. 177-
179, July 1998.
[69] Petre Stoica, Torsten S?oderstr?om, ?On non-singular information matrices and
local identifiability,? International Journal on Control, Volume 36, Number 2,
pp. 323-329, 1982.
[70] A. van den Bos, ?A Cram?er-Rao lower bound for complex parameters,? IEEE
Transactions on Signal Processing, Volume 42, Number 10, p. 2859, October
1994.
[71] Alle-Jan van der Veen, ?Asymptotic properties of the algebraic constant mod-
ulus algorithm,? IEEE Transactions on Signal Processing, Volume 49, Number
8, pp. 1796-1807, August 2001.
[72] Harry L. Van Trees, Detection, Estimation, and Modulation Theory, Part I,
John Wiley and Sons, Inc. 1968.
[73] L.J. Waldorp, H.M. Huizenga, R.P.P.P. Grasman, ?The Wald test and Cram?er-
Rao bound for misspecified models in electromagnetic source analysis,? IEEE
Transactions on Signal Processing, Volume 53, Number 9, pp. 3427-3435,
September 2005.
166
[74] Jo?ao Xavier, Victor Barroso, ?Intrinsic distance lower bound for unbiased es-
timators on Riemannian manifolds,? Proceedings of the IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP?02), Volume
2, pp. 1141-1144, May 2002.
[75] Jo?ao Xavier, Victor Barroso, ?Intrinsic variance lower bound (IVLB): An ex-
tension of the Cram?er-Rao bound to Riemannian manifolds,? Proceedings of
the IEEE International Conference on Acoustics, Speech and Signal Processing
(ICASSP?05), Volume 5, pp. 1033-1036, March 2005.
[76] Guanghan Xu, Hui Liu, Lang Tong, Thomas Kailath, ?A least-squaresapproach
to blind channel identification,? IEEE Transactions on Signal Processing, Vol-
ume 43, Number 12, pp. 2982-2993, December 1995.
[77] Yingwei Yao, Georgios Giannakis, ?On regularity and identifiability of blind
source separation under constant-modulus constraints,? IEEE Transactions on
Signal Processing, Volume 53, Number 4, pp. 1272-1281, April 2005.
[78] J. Ziv, M. Zakai, ?Some lower bounds on signal parameter estimation,? IEEE
Transactions on Information Theory, Volume IT-15, pp. 386-391, May 1969.
167